2024-09-15T01:41:20.386 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T01:41:20.405 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T01:41:20.485 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_workunits} email: skanta@redhat.com first_in_suite: false job_id: '7905072' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMXm5r0SM8bmxVjX0Ciz83GQGz1IKMSgovUu0dBZC9AGSosx9koPM+UdFQ8ksA6247zid+jtWgi222Gy5uAK/A= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB2ccedPMpmbmUqpQv7FzibDf3Wm6PTsep+hZZE3613EYAF8hZOhwdnXMkq1pu0Fa+8wgXYLSEOB9AxDd+Ot82Q= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T01:41:20.485 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T01:41:20.486 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T01:41:20.486 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T01:41:20.487 INFO:teuthology.task.internal:Checking packages... 2024-09-15T01:41:20.509 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T01:41:20.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T01:41:20.509 INFO:teuthology.packaging:ref: None 2024-09-15T01:41:20.509 INFO:teuthology.packaging:tag: None 2024-09-15T01:41:20.509 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T01:41:20.509 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:41:20.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T01:41:20.654 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T01:41:20.656 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T01:41:20.661 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T01:41:20.662 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T01:41:20.697 INFO:teuthology.task.internal:Saving configuration 2024-09-15T01:41:20.711 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T01:41:20.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T01:41:20.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 01:36:19.212091', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMXm5r0SM8bmxVjX0Ciz83GQGz1IKMSgovUu0dBZC9AGSosx9koPM+UdFQ8ksA6247zid+jtWgi222Gy5uAK/A='} 2024-09-15T01:41:20.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 01:36:19.213263', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB2ccedPMpmbmUqpQv7FzibDf3Wm6PTsep+hZZE3613EYAF8hZOhwdnXMkq1pu0Fa+8wgXYLSEOB9AxDd+Ot82Q='} 2024-09-15T01:41:20.767 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T01:41:20.774 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-15T01:41:20.774 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-15T01:41:20.774 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T01:41:20.828 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68f4a63ac0>, signals=[15]) 2024-09-15T01:41:20.828 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T01:41:20.834 INFO:teuthology.task.internal:Opening connections... 2024-09-15T01:41:20.834 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T01:41:20.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:41:20.913 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T01:41:20.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:41:20.997 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T01:41:21.004 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-09-15T01:41:21.022 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-09-15T01:41:21.022 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-09-15T01:41:21.080 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-09-15T01:41:21.081 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T01:41:21.082 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T01:41:21.082 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T01:41:21.082 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-09-15T01:41:21.103 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-09-15T01:41:21.121 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-09-15T01:41:21.121 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-09-15T01:41:21.179 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-09-15T01:41:21.180 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T01:41:21.180 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-09-15T01:41:21.180 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T01:41:21.180 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T01:41:21.180 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T01:41:21.180 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-09-15T01:41:21.202 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T01:41:21.213 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T01:41:21.219 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T01:41:21.219 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-09-15T01:41:21.221 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-09-15T01:41:21.237 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T01:41:21.245 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T01:41:21.245 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-09-15T01:41:21.280 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-09-15T01:41:21.306 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T01:41:21.367 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T01:41:21.396 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T01:41:21.396 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T01:41:21.397 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T01:41:21.398 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T01:41:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:41:21.415 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-15T01:41:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:41:21.417 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-15T01:41:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-503.el9.x86_64 2024-09-15T01:41:21.472 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-503.el9.x86_64 2024-09-15T01:41:21.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-09-15T01:41:21.475 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-503.el9.x86_64 2024-09-15T01:41:21.476 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-503.el9.x86_64 2024-09-15T01:41:21.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-09-15T01:41:23.690 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-15T01:41:23.729 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-15T01:41:26.180 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-15T01:41:30.723 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T01:41:32.157 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-15T01:41:32.928 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-15T01:41:38.450 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-15T01:41:40.490 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T01:41:40.490 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T01:41:40.589 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:41:40.600 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:41:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-15T01:41:40.600 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 79 M 2024-09-15T01:41:40.601 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 126 M 2024-09-15T01:41:40.602 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:41:40.718 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 239 kB/s | 23 kB 00:00 2024-09-15T01:41:41.202 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 59 MB/s | 29 MB 00:00 2024-09-15T01:41:41.677 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 33 MB/s | 35 MB 00:01 2024-09-15T01:41:41.919 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-09-15T01:41:41.919 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:41:41.919 INFO:teuthology.orchestra.run.smithi017.stdout:Total 60 MB/s | 79 MB 00:01 2024-09-15T01:41:42.629 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:41:42.994 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:41:42.995 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:41:43.971 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:41:43.971 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:41:44.680 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 20 MB 00:19 2024-09-15T01:41:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:41:47.198 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:41:47.288 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:41:50.079 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:41:50.196 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:41:52.364 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.5 MB 00:03 2024-09-15T01:41:54.207 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-15T01:41:55.086 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T01:41:55.324 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:41:55.363 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:00.496 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:00.742 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T01:42:02.815 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T01:42:02.815 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T01:42:02.904 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:42:02.910 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:42:02.910 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T01:42:02.910 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-09-15T01:42:02.911 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:42:02.912 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 79 M 2024-09-15T01:42:02.912 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 126 M 2024-09-15T01:42:02.912 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:42:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 200 kB/s | 23 kB 00:00 2024-09-15T01:42:03.995 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-09-15T01:42:04.444 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 22 MB/s | 29 MB 00:01 2024-09-15T01:42:04.819 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-09-15T01:42:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:42:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Total 41 MB/s | 79 MB 00:01 2024-09-15T01:42:05.525 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:42:05.877 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:42:05.877 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:42:06.876 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:42:06.877 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:42:09.435 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:42:10.095 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:42:10.178 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:42:12.956 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:42:13.081 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:42:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:42:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:23.386 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:39.711 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:39.752 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:40.097 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:40.097 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:42:40.098 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:42:40.098 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T01:42:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:42:40.920 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:42:41.263 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:42:41.389 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:42:41.390 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:42:41.390 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T01:42:41.390 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T01:42:41.390 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T01:42:41.391 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T01:42:41.391 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T01:42:41.391 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:42:41.448 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:42:41.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:42:41.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-09-15T01:42:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-09-15T01:42:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-09-15T01:42:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-09-15T01:42:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 82.el9 2024-09-15T01:42:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8274670 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-09-15T01:42:41.490 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:42:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T01:42:41.492 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T01:42:41.492 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-505.el9.x86_64 2024-09-15T01:42:41.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T01:42:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-09-15T01:42:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T01:42:43.590 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-09-15T01:42:43.592 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-15T01:42:43.607 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.erPYPCRMCB 2024-09-15T01:42:43.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.erPYPCRMCB 2024-09-15T01:42:43.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.erPYPCRMCB 2024-09-15T01:42:43.819 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.erPYPCRMCB is 6KB 2024-09-15T01:42:43.869 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.erPYPCRMCB 2024-09-15T01:42:43.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T01:42:43.946 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T01:42:43.946 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T01:42:43.946 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T01:42:43.946 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T01:42:43.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T01:42:44.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-09-15T01:43:02.291 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:43:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:43:02.675 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:43:02.675 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:43:02.675 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:43:02.675 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:43:03.510 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:43:03.860 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:43:03.948 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:43:03.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:43:03.949 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T01:43:03.949 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T01:43:03.949 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T01:43:03.949 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T01:43:03.949 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T01:43:03.949 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:43:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:43:04.008 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:43:04.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 82.el9 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-09-15T01:43:04.045 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8274670 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-09-15T01:43:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:43:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T01:43:04.048 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T01:43:04.048 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-505.el9.x86_64 2024-09-15T01:43:04.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T01:43:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-09-15T01:43:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T01:43:06.060 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-09-15T01:43:06.062 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-15T01:43:06.076 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.diB6OplQkK 2024-09-15T01:43:06.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.diB6OplQkK 2024-09-15T01:43:06.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.diB6OplQkK 2024-09-15T01:43:06.289 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.diB6OplQkK is 6KB 2024-09-15T01:43:06.338 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.diB6OplQkK 2024-09-15T01:43:06.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T01:43:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T01:43:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T01:43:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T01:43:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T01:43:06.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T01:43:06.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-09-15T01:43:14.108 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T01:43:14.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-15T01:43:14.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:43:36.602 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T01:43:36.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:43:36.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:44:11.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-15T01:44:14.114 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T01:44:20.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:44:20.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:44:23.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-15T01:44:23.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:44:23.468 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-15T01:44:23.901 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-15T01:44:23.902 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T01:44:23.902 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-15T01:44:23.920 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-505.el9.x86_64 2024-09-15T01:44:23.920 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T01:44:23.920 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T01:44:23.920 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T01:44:24.921 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T01:44:24.921 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T01:44:25.148 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-09-15T01:44:25.164 DEBUG:teuthology.parallel:result is None 2024-09-15T01:44:38.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-15T01:44:50.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:44:50.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:44:50.881 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-15T01:44:51.248 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:44:51.248 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-15T01:44:51.248 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-15T01:44:51.263 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-505.el9.x86_64 2024-09-15T01:44:51.264 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T01:44:51.264 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T01:44:51.264 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T01:44:52.265 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T01:44:52.265 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T01:44:52.341 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-09-15T01:44:52.363 DEBUG:teuthology.parallel:result is None 2024-09-15T01:44:52.363 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T01:44:52.371 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T01:44:52.371 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T01:44:52.375 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T01:44:52.391 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T01:44:52.400 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T01:44:52.407 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T01:44:52.407 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T01:44:52.435 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T01:44:52.478 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T01:44:52.484 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T01:44:52.485 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T01:44:52.499 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T01:44:52.533 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:44:52.540 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:44:52.545 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:44:52.551 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:44:52.553 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T01:44:52.560 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T01:44:52.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T01:44:52.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T01:44:52.618 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T01:44:52.628 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T01:44:52.628 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T01:44:52.661 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T01:44:52.677 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:44:52.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:44:52.814 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:44:52.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:44:52.942 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:44:52.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T01:44:53.010 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:44:53.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:44:53.102 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:44:53.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:44:53.221 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:44:53.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T01:44:53.284 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-09-15T01:44:53.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-09-15T01:44:53.336 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T01:44:53.371 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T01:44:53.818 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T01:44:53.829 INFO:teuthology.task.internal:Starting timer... 2024-09-15T01:44:53.829 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T01:44:53.843 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T01:44:53.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-09-15T01:44:53.927 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-09-15T01:44:53.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-09-15T01:44:53.995 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-09-15T01:44:53.997 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T01:44:53.997 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-09-15T01:44:54.021 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-09-15T01:44:54.021 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-09-15T01:44:54.043 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-09-15T01:44:54.043 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2024-09-15T01:44:54.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T01:44:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:44:54.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T01:44:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:44:54.115 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T01:44:54.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-09-15T01:44:54.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-09-15T01:44:54.197 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T01:44:54.207 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T01:44:54.210 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T01:44:54.332 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T01:44:54.343 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T01:44:54.344 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T01:48:44.277 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2024-09-15T01:48:44.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-15T01:48:44.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:48:44.358 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-15T01:48:44.440 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-15T01:48:44.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:48:44.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:48:44.517 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-15T01:48:44.605 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T01:48:44.605 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T01:48:44.632 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T01:48:44.633 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T01:48:44.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T01:48:44.636 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T01:48:44.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T01:48:44.666 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T01:48:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T01:48:44.681 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T01:48:44.692 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T01:48:44.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-09-15T01:48:44.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-09-15T01:48:44.719 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-09-15T01:48:44.728 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-09-15T01:48:44.734 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T01:48:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T01:48:44.749 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T01:48:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T01:48:44.807 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.811 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T01:48:44.817 INFO:teuthology.run_tasks:Running task install... 2024-09-15T01:48:44.828 DEBUG:teuthology.task.install:project ceph 2024-09-15T01:48:44.828 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T01:48:44.829 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T01:48:44.829 INFO:teuthology.task.install:Using flavor: default 2024-09-15T01:48:44.834 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T01:48:44.834 INFO:teuthology.task.install:extra packages: [] 2024-09-15T01:48:44.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T01:48:44.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T01:48:45.436 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:45 ago on Sun 15 Sep 2024 01:48:00 AM UTC. 2024-09-15T01:48:45.438 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:45 ago on Sun 15 Sep 2024 01:48:00 AM UTC. 2024-09-15T01:48:45.527 INFO:teuthology.orchestra.run.smithi017.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T01:48:45.527 INFO:teuthology.orchestra.run.smithi067.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T01:48:45.571 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:48:45.572 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:48:45.573 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:48:45.573 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:48:45.573 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:48:45.573 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:48:45.637 DEBUG:teuthology.orchestra.run.smithi017:> sudo dnf -y copr enable ceph/el9 2024-09-15T01:48:45.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo dnf -y copr enable ceph/el9 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:of the main distribution, and quality may vary. 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:, 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:and packages are not held to any quality or security level. 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:48:45.826 INFO:teuthology.orchestra.run.smithi017.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T01:48:45.827 INFO:teuthology.orchestra.run.smithi017.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:of the main distribution, and quality may vary. 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:, 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:and packages are not held to any quality or security level. 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T01:48:45.829 INFO:teuthology.orchestra.run.smithi067.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T01:48:45.902 INFO:teuthology.orchestra.run.smithi017.stdout:Repository successfully enabled. 2024-09-15T01:48:45.902 INFO:teuthology.orchestra.run.smithi067.stdout:Repository successfully enabled. 2024-09-15T01:48:45.939 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T01:48:45.939 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:48:45.941 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T01:48:45.942 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:48:46.089 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T01:48:46.090 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T01:48:46.095 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T01:48:46.095 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T01:48:46.201 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T01:48:46.202 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:48:46.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T01:48:46.209 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T01:48:46.210 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:48:46.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T01:48:46.228 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T01:48:46.228 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T01:48:46.236 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T01:48:46.236 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T01:48:46.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T01:48:46.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T01:48:46.373 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T01:48:46.379 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T01:48:46.438 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2024-09-15T01:48:46.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2024-09-15T01:48:46.444 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-09-15T01:48:46.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-09-15T01:48:46.674 INFO:teuthology.orchestra.run.smithi017.stdout:45 files removed 2024-09-15T01:48:46.693 INFO:teuthology.orchestra.run.smithi067.stdout:45 files removed 2024-09-15T01:48:46.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw 2024-09-15T01:48:46.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-09-15T01:48:47.268 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 289 kB/s | 77 kB 00:00 2024-09-15T01:48:47.288 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 287 kB/s | 77 kB 00:00 2024-09-15T01:48:47.552 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T01:48:47.564 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-15T01:48:47.828 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T01:48:47.831 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 7.8 kB/s | 1.8 kB 00:00 2024-09-15T01:48:47.998 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-15T01:48:48.003 INFO:teuthology.orchestra.run.smithi017.stdout:Copr repo for el9 owned by ceph 55 kB/s | 7.6 kB 00:00 2024-09-15T01:48:48.520 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-15T01:48:49.323 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.2 MB 00:01 2024-09-15T01:48:51.308 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T01:48:51.429 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-15T01:48:55.997 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-15T01:48:57.084 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-15T01:48:57.768 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-15T01:48:58.601 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-15T01:48:58.818 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-15T01:48:59.505 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T01:49:04.558 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T01:49:05.170 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-15T01:49:07.086 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T01:49:07.094 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T01:49:07.095 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T01:49:07.096 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:Install 22 Packages 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 2 Packages 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 58 M 2024-09-15T01:49:07.097 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:49:07.626 INFO:teuthology.orchestra.run.smithi017.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 12 MB/s | 5.0 MB 00:00 2024-09-15T01:49:07.684 INFO:teuthology.orchestra.run.smithi017.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 433 kB/s | 25 kB 00:00 2024-09-15T01:49:07.714 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T01:49:07.726 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T01:49:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T01:49:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Install 22 Packages 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 58 M 2024-09-15T01:49:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:49:07.776 INFO:teuthology.orchestra.run.smithi017.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-15T01:49:07.860 INFO:teuthology.orchestra.run.smithi017.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-15T01:49:08.185 INFO:teuthology.orchestra.run.smithi017.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 15 MB/s | 4.9 MB 00:00 2024-09-15T01:49:08.411 INFO:teuthology.orchestra.run.smithi017.stdout:(6/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 19 MB/s | 22 MB 00:01 2024-09-15T01:49:08.436 INFO:teuthology.orchestra.run.smithi017.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 180 kB/s | 45 kB 00:00 2024-09-15T01:49:08.586 INFO:teuthology.orchestra.run.smithi017.stdout:(8/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 10 MB/s | 14 MB 00:01 2024-09-15T01:49:08.604 INFO:teuthology.orchestra.run.smithi067.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 9.3 MB/s | 5.0 MB 00:00 2024-09-15T01:49:08.611 INFO:teuthology.orchestra.run.smithi017.stdout:(9/24): python3-ceph-common-17.2.7-1643.gd75f23 600 kB/s | 120 kB 00:00 2024-09-15T01:49:08.637 INFO:teuthology.orchestra.run.smithi017.stdout:(10/24): python3-cephfs-17.2.7-1643.gd75f2382.e 777 kB/s | 156 kB 00:00 2024-09-15T01:49:08.662 INFO:teuthology.orchestra.run.smithi017.stdout:(11/24): python3-rados-17.2.7-1643.gd75f2382.el 4.1 MB/s | 320 kB 00:00 2024-09-15T01:49:08.679 INFO:teuthology.orchestra.run.smithi067.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 336 kB/s | 25 kB 00:00 2024-09-15T01:49:08.704 INFO:teuthology.orchestra.run.smithi017.stdout:(12/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 3.1 MB/s | 295 kB 00:00 2024-09-15T01:49:08.730 INFO:teuthology.orchestra.run.smithi017.stdout:(13/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.1 MB/s | 100 kB 00:00 2024-09-15T01:49:08.780 INFO:teuthology.orchestra.run.smithi067.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-15T01:49:08.838 INFO:teuthology.orchestra.run.smithi017.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-09-15T01:49:08.864 INFO:teuthology.orchestra.run.smithi017.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 285 kB/s | 45 kB 00:00 2024-09-15T01:49:08.888 INFO:teuthology.orchestra.run.smithi067.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.1 MB/s | 458 kB 00:00 2024-09-15T01:49:08.914 INFO:teuthology.orchestra.run.smithi017.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-15T01:49:08.939 INFO:teuthology.orchestra.run.smithi017.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-15T01:49:08.990 INFO:teuthology.orchestra.run.smithi017.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 662 kB 00:00 2024-09-15T01:49:09.023 INFO:teuthology.orchestra.run.smithi017.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-09-15T01:49:09.048 INFO:teuthology.orchestra.run.smithi017.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-09-15T01:49:09.073 INFO:teuthology.orchestra.run.smithi017.stdout:(21/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2024-09-15T01:49:09.088 INFO:teuthology.orchestra.run.smithi067.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 14 MB/s | 14 MB 00:01 2024-09-15T01:49:09.132 INFO:teuthology.orchestra.run.smithi017.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-15T01:49:09.164 INFO:teuthology.orchestra.run.smithi067.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 604 kB/s | 45 kB 00:00 2024-09-15T01:49:09.247 INFO:teuthology.orchestra.run.smithi067.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-15T01:49:09.266 INFO:teuthology.orchestra.run.smithi017.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 14 MB/s | 3.1 MB 00:00 2024-09-15T01:49:09.272 INFO:teuthology.orchestra.run.smithi067.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 1.1 MB/s | 120 kB 00:00 2024-09-15T01:49:09.298 INFO:teuthology.orchestra.run.smithi067.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.0 MB/s | 156 kB 00:00 2024-09-15T01:49:09.331 INFO:teuthology.orchestra.run.smithi067.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 5.3 MB/s | 320 kB 00:00 2024-09-15T01:49:09.332 INFO:teuthology.orchestra.run.smithi017.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-15T01:49:09.338 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:09.338 INFO:teuthology.orchestra.run.smithi017.stdout:Total 26 MB/s | 58 MB 00:02 2024-09-15T01:49:09.356 INFO:teuthology.orchestra.run.smithi067.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-15T01:49:09.382 INFO:teuthology.orchestra.run.smithi067.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T01:49:09.407 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:49:09.446 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:49:09.446 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:49:09.591 INFO:teuthology.orchestra.run.smithi067.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 15 MB/s | 22 MB 00:01 2024-09-15T01:49:09.657 INFO:teuthology.orchestra.run.smithi067.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 533 kB/s | 160 kB 00:00 2024-09-15T01:49:09.691 INFO:teuthology.orchestra.run.smithi067.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 146 kB/s | 45 kB 00:00 2024-09-15T01:49:09.775 INFO:teuthology.orchestra.run.smithi067.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-15T01:49:09.800 INFO:teuthology.orchestra.run.smithi067.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 389 kB/s | 42 kB 00:00 2024-09-15T01:49:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-15T01:49:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-09-15T01:49:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-15T01:49:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 790 kB/s | 67 kB 00:00 2024-09-15T01:49:09.977 INFO:teuthology.orchestra.run.smithi067.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T01:49:10.080 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:49:10.081 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:49:10.085 INFO:teuthology.orchestra.run.smithi067.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 17 MB/s | 3.1 MB 00:00 2024-09-15T01:49:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-09-15T01:49:10.148 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:10.148 INFO:teuthology.orchestra.run.smithi067.stdout:Total 24 MB/s | 58 MB 00:02 2024-09-15T01:49:10.211 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:49:10.248 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:49:10.248 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:49:10.911 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:49:10.912 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:49:11.326 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:49:11.718 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T01:49:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:11.878 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:11.958 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T01:49:11.986 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T01:49:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T01:49:12.197 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:49:12.577 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T01:49:12.616 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T01:49:12.648 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T01:49:12.679 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:12.711 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T01:49:12.748 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T01:49:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:12.791 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T01:49:12.828 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T01:49:12.861 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T01:49:12.889 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:12.891 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T01:49:12.917 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T01:49:12.978 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:12.982 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T01:49:13.023 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T01:49:13.052 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T01:49:13.061 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T01:49:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T01:49:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T01:49:13.436 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T01:49:13.479 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T01:49:13.514 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T01:49:13.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T01:49:13.571 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T01:49:13.604 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T01:49:13.641 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T01:49:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T01:49:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T01:49:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T01:49:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T01:49:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T01:49:13.827 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T01:49:13.893 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T01:49:13.949 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T01:49:13.986 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T01:49:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T01:49:14.327 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T01:49:14.381 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T01:49:14.472 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T01:49:14.548 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T01:49:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T01:49:16.263 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:16.320 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:16.871 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:16.914 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T01:49:17.029 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T01:49:17.029 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T01:49:17.029 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T01:49:17.029 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:17.083 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:17.141 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:17.180 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:17.740 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T01:49:17.774 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T01:49:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T01:49:17.871 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T01:49:17.871 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T01:49:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2024-09-15T01:49:27.466 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /root 2024-09-15T01:49:27.467 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2024-09-15T01:49:27.467 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-09-15T01:49:28.402 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-09-15T01:49:28.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:28.825 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T01:49:28.912 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:28.914 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T01:49:28.994 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T01:49:28.995 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:29.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T01:49:29.814 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T01:49:29.815 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:29.815 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T01:49:29.815 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T01:49:29.815 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T01:49:29.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:29.816 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T01:49:29.900 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T01:49:29.901 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:30.317 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:30.317 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T01:49:30.318 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T01:49:30.319 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T01:49:31.244 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:31.244 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:31.244 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2024-09-15T01:49:31.244 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T01:49:31.245 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:31.246 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T01:49:31.247 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:31.247 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:49:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T01:49:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T01:49:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T01:49:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T01:49:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T01:49:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T01:49:31.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-test 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.143 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:49:32.157 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:49:32.294 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:32.294 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:32.294 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:49:32.294 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-15T01:49:32.295 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:32.296 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 48 M 2024-09-15T01:49:32.296 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 193 M 2024-09-15T01:49:32.296 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:49:32.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-09-15T01:49:32.652 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 756 kB/s | 163 kB 00:00 2024-09-15T01:49:32.986 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 553 kB/s | 304 kB 00:00 2024-09-15T01:49:33.019 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 173 kB/s | 64 kB 00:00 2024-09-15T01:49:33.070 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:49:33.209 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:33.209 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-09-15T01:49:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:33.211 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 48 M 2024-09-15T01:49:33.211 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 193 M 2024-09-15T01:49:33.211 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:49:33.449 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-15T01:49:33.499 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-09-15T01:49:33.582 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 468 kB/s | 163 kB 00:00 2024-09-15T01:49:33.928 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-15T01:49:33.929 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:33.930 INFO:teuthology.orchestra.run.smithi017.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-15T01:49:33.958 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:49:33.979 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:49:33.979 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:49:34.364 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:49:34.364 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:49:34.849 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-15T01:49:34.851 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:34.852 INFO:teuthology.orchestra.run.smithi067.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-15T01:49:34.881 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:49:34.902 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:49:34.902 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:49:34.919 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:49:34.997 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T01:49:35.045 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T01:49:35.080 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T01:49:35.287 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:49:35.287 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:49:35.833 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:49:35.902 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T01:49:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T01:49:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T01:49:39.771 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T01:49:39.855 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T01:49:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T01:49:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T01:49:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T01:49:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T01:49:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T01:49:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T01:49:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T01:49:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:42.405 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:49:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T01:49:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T01:49:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:42.406 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:49:42.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph 2024-09-15T01:49:43.015 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T01:49:43.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T01:49:43.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T01:49:43.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T01:49:43.260 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:43.331 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:49:43.408 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:43.410 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:49:43.410 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:43.410 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:49:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T01:49:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T01:49:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:49:43.413 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:43.414 INFO:teuthology.orchestra.run.smithi017.stdout:Install 34 Packages 2024-09-15T01:49:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:43.414 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 27 M 2024-09-15T01:49:43.414 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 96 M 2024-09-15T01:49:43.414 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:49:43.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-09-15T01:49:43.920 INFO:teuthology.orchestra.run.smithi017.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-09-15T01:49:44.087 INFO:teuthology.orchestra.run.smithi017.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.6 MB/s | 2.1 MB 00:00 2024-09-15T01:49:44.129 INFO:teuthology.orchestra.run.smithi017.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-15T01:49:44.213 INFO:teuthology.orchestra.run.smithi017.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.8 MB/s | 240 kB 00:00 2024-09-15T01:49:44.231 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:49:44.347 INFO:teuthology.orchestra.run.smithi017.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-15T01:49:44.376 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T01:49:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T01:49:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout:Install 34 Packages 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 27 M 2024-09-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 96 M 2024-09-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:49:44.430 INFO:teuthology.orchestra.run.smithi017.stdout:(6/34): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-15T01:49:44.489 INFO:teuthology.orchestra.run.smithi017.stdout:(7/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 6.9 MB/s | 3.9 MB 00:00 2024-09-15T01:49:44.514 INFO:teuthology.orchestra.run.smithi017.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-15T01:49:44.673 INFO:teuthology.orchestra.run.smithi017.stdout:(9/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 26 MB/s | 15 MB 00:00 2024-09-15T01:49:44.691 INFO:teuthology.orchestra.run.smithi067.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-15T01:49:44.698 INFO:teuthology.orchestra.run.smithi017.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 680 kB/s | 125 kB 00:00 2024-09-15T01:49:44.740 INFO:teuthology.orchestra.run.smithi017.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 539 kB/s | 135 kB 00:00 2024-09-15T01:49:44.798 INFO:teuthology.orchestra.run.smithi017.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.7 MB/s | 215 kB 00:00 2024-09-15T01:49:44.849 INFO:teuthology.orchestra.run.smithi017.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-15T01:49:44.867 INFO:teuthology.orchestra.run.smithi067.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.7 MB/s | 2.1 MB 00:00 2024-09-15T01:49:44.899 INFO:teuthology.orchestra.run.smithi017.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-09-15T01:49:44.908 INFO:teuthology.orchestra.run.smithi067.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-15T01:49:44.924 INFO:teuthology.orchestra.run.smithi017.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-09-15T01:49:44.958 INFO:teuthology.orchestra.run.smithi017.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 485 kB/s | 77 kB 00:00 2024-09-15T01:49:44.983 INFO:teuthology.orchestra.run.smithi017.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-15T01:49:44.992 INFO:teuthology.orchestra.run.smithi067.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.8 MB/s | 240 kB 00:00 2024-09-15T01:49:45.017 INFO:teuthology.orchestra.run.smithi017.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-15T01:49:45.050 INFO:teuthology.orchestra.run.smithi017.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 441 kB/s | 11 kB 00:00 2024-09-15T01:49:45.076 INFO:teuthology.orchestra.run.smithi017.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 759 kB/s | 18 kB 00:00 2024-09-15T01:49:45.101 INFO:teuthology.orchestra.run.smithi017.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-15T01:49:45.126 INFO:teuthology.orchestra.run.smithi067.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-15T01:49:45.126 INFO:teuthology.orchestra.run.smithi017.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-15T01:49:45.151 INFO:teuthology.orchestra.run.smithi017.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-15T01:49:45.206 INFO:teuthology.orchestra.run.smithi017.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-15T01:49:45.206 INFO:teuthology.orchestra.run.smithi017.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-15T01:49:45.219 INFO:teuthology.orchestra.run.smithi017.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-15T01:49:45.234 INFO:teuthology.orchestra.run.smithi067.stdout:(6/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 7.2 MB/s | 3.9 MB 00:00 2024-09-15T01:49:45.244 INFO:teuthology.orchestra.run.smithi017.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 655 kB/s | 16 kB 00:00 2024-09-15T01:49:45.269 INFO:teuthology.orchestra.run.smithi017.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-15T01:49:45.276 INFO:teuthology.orchestra.run.smithi067.stdout:(7/34): python3-cryptography-36.0.1-4.el9.x86_6 8.2 MB/s | 1.2 MB 00:00 2024-09-15T01:49:45.295 INFO:teuthology.orchestra.run.smithi017.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-15T01:49:45.320 INFO:teuthology.orchestra.run.smithi017.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 707 kB/s | 36 kB 00:00 2024-09-15T01:49:45.345 INFO:teuthology.orchestra.run.smithi017.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T01:49:45.379 INFO:teuthology.orchestra.run.smithi017.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 599 kB/s | 20 kB 00:00 2024-09-15T01:49:45.404 INFO:teuthology.orchestra.run.smithi017.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-15T01:49:45.438 INFO:teuthology.orchestra.run.smithi017.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-15T01:49:45.441 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:45.441 INFO:teuthology.orchestra.run.smithi017.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-15T01:49:45.443 INFO:teuthology.orchestra.run.smithi067.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 27 MB/s | 15 MB 00:00 2024-09-15T01:49:45.468 INFO:teuthology.orchestra.run.smithi067.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 705 kB/s | 135 kB 00:00 2024-09-15T01:49:45.502 INFO:teuthology.orchestra.run.smithi067.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 399 kB/s | 106 kB 00:00 2024-09-15T01:49:45.527 INFO:teuthology.orchestra.run.smithi067.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-15T01:49:45.575 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:49:45.577 INFO:teuthology.orchestra.run.smithi067.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 932 kB/s | 125 kB 00:00 2024-09-15T01:49:45.615 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:49:45.616 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:49:45.661 INFO:teuthology.orchestra.run.smithi067.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-15T01:49:45.686 INFO:teuthology.orchestra.run.smithi067.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 935 kB/s | 172 kB 00:00 2024-09-15T01:49:45.712 INFO:teuthology.orchestra.run.smithi067.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 577 kB/s | 77 kB 00:00 2024-09-15T01:49:45.745 INFO:teuthology.orchestra.run.smithi067.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 608 kB/s | 51 kB 00:00 2024-09-15T01:49:45.771 INFO:teuthology.orchestra.run.smithi067.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-15T01:49:45.804 INFO:teuthology.orchestra.run.smithi067.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-15T01:49:45.838 INFO:teuthology.orchestra.run.smithi067.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 450 kB/s | 11 kB 00:00 2024-09-15T01:49:45.863 INFO:teuthology.orchestra.run.smithi067.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 771 kB/s | 18 kB 00:00 2024-09-15T01:49:45.891 INFO:teuthology.orchestra.run.smithi067.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-15T01:49:45.917 INFO:teuthology.orchestra.run.smithi067.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 433 kB/s | 23 kB 00:00 2024-09-15T01:49:45.942 INFO:teuthology.orchestra.run.smithi067.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-15T01:49:45.967 INFO:teuthology.orchestra.run.smithi067.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-15T01:49:45.993 INFO:teuthology.orchestra.run.smithi067.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-09-15T01:49:46.018 INFO:teuthology.orchestra.run.smithi067.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T01:49:46.043 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:49:46.044 INFO:teuthology.orchestra.run.smithi067.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-15T01:49:46.044 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:49:46.069 INFO:teuthology.orchestra.run.smithi067.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-15T01:49:46.094 INFO:teuthology.orchestra.run.smithi067.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-15T01:49:46.119 INFO:teuthology.orchestra.run.smithi067.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-15T01:49:46.145 INFO:teuthology.orchestra.run.smithi067.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-15T01:49:46.169 INFO:teuthology.orchestra.run.smithi067.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-15T01:49:46.195 INFO:teuthology.orchestra.run.smithi067.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T01:49:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-15T01:49:46.233 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:46.233 INFO:teuthology.orchestra.run.smithi067.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-15T01:49:46.364 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:49:46.405 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:49:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:49:46.809 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:49:46.809 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:49:47.444 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:49:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T01:49:47.540 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T01:49:47.573 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T01:49:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T01:49:47.746 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T01:49:47.793 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T01:49:47.828 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T01:49:47.857 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T01:49:47.892 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T01:49:47.925 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T01:49:47.966 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T01:49:48.021 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T01:49:48.055 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T01:49:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:49:48.104 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T01:49:48.140 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T01:49:48.144 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T01:49:48.173 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T01:49:48.205 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T01:49:48.221 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T01:49:48.310 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T01:49:48.374 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T01:49:48.387 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T01:49:48.437 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T01:49:48.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T01:49:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T01:49:48.514 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T01:49:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T01:49:48.565 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T01:49:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T01:49:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T01:49:48.611 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T01:49:48.658 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T01:49:48.669 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T01:49:48.692 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T01:49:48.733 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T01:49:48.741 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T01:49:48.781 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T01:49:48.857 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T01:49:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T01:49:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T01:49:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T01:49:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T01:49:49.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T01:49:49.151 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T01:49:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T01:49:49.197 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T01:49:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T01:49:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T01:49:49.318 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T01:49:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T01:49:49.475 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T01:49:49.515 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T01:49:49.542 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T01:49:49.543 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T01:49:49.543 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:49.544 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T01:49:49.544 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T01:49:49.544 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T01:49:49.544 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:49.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T01:49:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T01:49:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T01:49:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T01:49:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T01:49:50.101 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T01:49:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T01:49:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T01:49:50.160 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:50.160 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T01:49:50.160 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T01:49:50.161 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T01:49:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T01:49:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T01:49:50.978 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:50.979 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T01:49:50.979 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T01:49:50.979 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T01:49:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:51.383 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T01:49:51.409 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:51.559 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T01:49:51.589 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:51.634 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T01:49:51.659 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T01:49:51.660 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:51.660 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T01:49:51.660 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T01:49:51.660 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T01:49:51.660 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T01:49:51.994 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T01:49:52.021 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:52.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T01:49:52.266 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T01:49:52.266 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:52.266 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T01:49:52.266 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T01:49:52.267 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T01:49:52.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:52.353 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T01:49:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T01:49:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T01:49:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T01:49:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T01:49:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T01:49:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T01:49:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T01:49:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T01:49:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T01:49:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:54.583 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T01:49:54.584 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T01:49:54.585 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:54.586 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:49:54.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-base 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:49:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T01:49:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T01:49:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:55.081 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:49:55.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-09-15T01:49:55.532 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:49:55.627 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:49:55.671 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:55.672 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:49:55.672 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:49:55.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install cephadm 2024-09-15T01:49:56.031 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:49:56.124 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:49:56.168 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:56.169 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:49:56.169 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:49:56.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-09-15T01:49:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:49:56.465 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:56.466 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:49:56.467 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:56.467 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 83 k 2024-09-15T01:49:56.467 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 375 k 2024-09-15T01:49:56.467 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:49:56.660 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-15T01:49:56.661 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:56.661 INFO:teuthology.orchestra.run.smithi017.stdout:Total 424 kB/s | 83 kB 00:00 2024-09-15T01:49:56.661 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:49:56.667 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:49:56.667 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:49:56.669 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:49:56.669 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:49:56.751 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:49:56.829 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:49:56.967 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:49:56.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:56.969 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 83 k 2024-09-15T01:49:56.969 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 375 k 2024-09-15T01:49:56.969 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:49:57.159 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 435 kB/s | 83 kB 00:00 2024-09-15T01:49:57.160 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:57.160 INFO:teuthology.orchestra.run.smithi067.stdout:Total 431 kB/s | 83 kB 00:00 2024-09-15T01:49:57.160 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:49:57.166 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:49:57.166 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:49:57.167 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:49:57.168 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:49:57.205 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.245 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:49:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.660 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:57.879 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:49:58.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T01:49:58.110 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:58.370 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:49:58.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T01:49:58.673 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:49:58.812 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:49:58.812 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:49:58.813 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:58.814 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 137 k 2024-09-15T01:49:58.814 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 379 k 2024-09-15T01:49:58.814 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:49:59.027 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 645 kB/s | 137 kB 00:00 2024-09-15T01:49:59.027 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:59.028 INFO:teuthology.orchestra.run.smithi017.stdout:Total 641 kB/s | 137 kB 00:00 2024-09-15T01:49:59.028 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:49:59.036 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:49:59.036 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:49:59.096 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:49:59.096 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:49:59.162 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:49:59.244 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:49:59.299 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:49:59.300 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T01:49:59.300 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:59.300 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:49:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 137 k 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 379 k 2024-09-15T01:49:59.301 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:49:59.303 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:49:59.329 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:49:59.329 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:59.329 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T01:49:59.329 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:49:59.506 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 669 kB/s | 137 kB 00:00 2024-09-15T01:49:59.507 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:49:59.507 INFO:teuthology.orchestra.run.smithi067.stdout:Total 664 kB/s | 137 kB 00:00 2024-09-15T01:49:59.507 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:49:59.515 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:49:59.515 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:49:59.575 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:49:59.575 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:49:59.721 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:49:59.791 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:49:59.817 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:49:59.817 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:49:59.817 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T01:49:59.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:50:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:00.243 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:00.243 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:00.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr 2024-09-15T01:50:00.782 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T01:50:00.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:00.782 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:00.782 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:00.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:00.783 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:01.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-09-15T01:50:01.047 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:01.142 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:01.187 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:01.188 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:01.188 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:01.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T01:50:01.597 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:01.690 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:01.734 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:01.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T01:50:01.843 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:01.982 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:01.982 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:01.982 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout:Installing weak dependencies: 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T01:50:01.983 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout:Install 7 Packages 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.7 M 2024-09-15T01:50:01.984 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 80 M 2024-09-15T01:50:01.985 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:02.387 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:02.497 INFO:teuthology.orchestra.run.smithi017.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 179 kB/s | 24 kB 00:00 2024-09-15T01:50:02.522 INFO:teuthology.orchestra.run.smithi017.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 90 kB/s | 15 kB 00:00 2024-09-15T01:50:02.526 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:02.527 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:02.527 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:02.527 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T01:50:02.528 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout:Install 7 Packages 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:02.529 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.7 M 2024-09-15T01:50:02.530 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 80 M 2024-09-15T01:50:02.530 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:02.564 INFO:teuthology.orchestra.run.smithi017.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T01:50:02.589 INFO:teuthology.orchestra.run.smithi017.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T01:50:02.624 INFO:teuthology.orchestra.run.smithi017.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 405 kB/s | 41 kB 00:00 2024-09-15T01:50:02.649 INFO:teuthology.orchestra.run.smithi017.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-15T01:50:02.749 INFO:teuthology.orchestra.run.smithi017.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.6 MB/s | 3.4 MB 00:00 2024-09-15T01:50:02.753 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:02.753 INFO:teuthology.orchestra.run.smithi017.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-09-15T01:50:02.753 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:02.762 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:02.762 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:02.845 INFO:teuthology.orchestra.run.smithi067.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 161 kB/s | 24 kB 00:00 2024-09-15T01:50:02.850 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:02.850 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:02.861 INFO:teuthology.orchestra.run.smithi067.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 87 kB/s | 15 kB 00:00 2024-09-15T01:50:02.895 INFO:teuthology.orchestra.run.smithi067.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-15T01:50:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T01:50:02.954 INFO:teuthology.orchestra.run.smithi067.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 442 kB/s | 41 kB 00:00 2024-09-15T01:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-15T01:50:03.079 INFO:teuthology.orchestra.run.smithi067.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.7 MB/s | 3.4 MB 00:00 2024-09-15T01:50:03.083 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:03.083 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2024-09-15T01:50:03.083 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:03.092 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:03.092 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:03.186 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:03.187 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T01:50:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T01:50:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T01:50:03.354 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T01:50:03.384 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T01:50:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:03.581 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T01:50:03.615 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T01:50:03.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T01:50:03.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T01:50:03.731 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T01:50:04.432 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T01:50:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T01:50:04.781 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T01:50:04.825 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T01:50:05.482 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T01:50:05.482 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T01:50:05.482 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T01:50:05.482 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T01:50:05.483 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T01:50:05.483 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T01:50:05.483 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T01:50:05.754 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T01:50:05.754 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:05.754 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:05.755 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:05.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T01:50:05.945 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T01:50:05.945 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T01:50:05.945 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T01:50:05.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T01:50:05.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T01:50:05.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T01:50:05.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T01:50:06.235 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T01:50:06.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T01:50:06.236 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:06.236 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:06.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T01:50:06.525 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:06.665 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================= 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================= 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T01:50:06.666 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T01:50:06.667 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================= 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:Install 12 Packages 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 38 M 2024-09-15T01:50:06.669 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 212 M 2024-09-15T01:50:06.669 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:06.982 INFO:teuthology.orchestra.run.smithi017.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 892 kB/s | 191 kB 00:00 2024-09-15T01:50:07.007 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:07.040 INFO:teuthology.orchestra.run.smithi017.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 561 kB/s | 33 kB 00:00 2024-09-15T01:50:07.074 INFO:teuthology.orchestra.run.smithi017.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-15T01:50:07.132 INFO:teuthology.orchestra.run.smithi017.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-09-15T01:50:07.145 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:07.146 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T01:50:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:07.148 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T01:50:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-09-15T01:50:07.149 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 38 M 2024-09-15T01:50:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 212 M 2024-09-15T01:50:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:07.266 INFO:teuthology.orchestra.run.smithi017.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-15T01:50:07.283 INFO:teuthology.orchestra.run.smithi017.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 256 kB/s | 38 kB 00:00 2024-09-15T01:50:07.342 INFO:teuthology.orchestra.run.smithi017.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-09-15T01:50:07.475 INFO:teuthology.orchestra.run.smithi017.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-09-15T01:50:07.500 INFO:teuthology.orchestra.run.smithi017.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-15T01:50:07.526 INFO:teuthology.orchestra.run.smithi017.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-09-15T01:50:07.601 INFO:teuthology.orchestra.run.smithi017.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 15 MB/s | 4.9 MB 00:00 2024-09-15T01:50:07.909 INFO:teuthology.orchestra.run.smithi017.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-09-15T01:50:07.910 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:07.911 INFO:teuthology.orchestra.run.smithi017.stdout:Total 31 MB/s | 38 MB 00:01 2024-09-15T01:50:08.271 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 233 kB/s | 191 kB 00:00 2024-09-15T01:50:08.273 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:08.300 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:08.300 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:08.396 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-09-15T01:50:08.430 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 819 kB/s | 801 kB 00:00 2024-09-15T01:50:08.547 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-09-15T01:50:08.572 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 129 kB/s | 18 kB 00:00 2024-09-15T01:50:08.614 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 576 kB/s | 38 kB 00:00 2024-09-15T01:50:08.714 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-15T01:50:08.813 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:08.813 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:08.856 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-15T01:50:08.965 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-15T01:50:08.990 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-15T01:50:09.223 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 4.2 MB/s | 7.4 MB 00:01 2024-09-15T01:50:09.399 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-09-15T01:50:09.400 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:09.400 INFO:teuthology.orchestra.run.smithi067.stdout:Total 17 MB/s | 38 MB 00:02 2024-09-15T01:50:09.605 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:09.675 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T01:50:09.722 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T01:50:09.770 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:09.798 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:09.798 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T01:50:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T01:50:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T01:50:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T01:50:10.171 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T01:50:10.336 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:10.336 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:10.866 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T01:50:10.991 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T01:50:11.115 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:11.214 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T01:50:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T01:50:11.369 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T01:50:11.407 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T01:50:11.598 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T01:50:11.622 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T01:50:11.706 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T01:50:12.324 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T01:50:12.396 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T01:50:12.523 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T01:50:13.407 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T01:50:13.433 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T01:50:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T01:50:14.924 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T01:50:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T01:50:15.957 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T01:50:15.958 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T01:50:15.959 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T01:50:16.378 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:16.379 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:16.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-rook 2024-09-15T01:50:17.333 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:13 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:17.477 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:17.478 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:17.478 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:17.478 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:17.478 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:17.478 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T01:50:17.479 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout:Install 13 Packages 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:17.480 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 2.2 M 2024-09-15T01:50:17.481 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 26 M 2024-09-15T01:50:17.481 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T01:50:17.590 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T01:50:17.591 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T01:50:17.591 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T01:50:17.591 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T01:50:17.591 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T01:50:17.776 INFO:teuthology.orchestra.run.smithi017.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 205 kB/s | 19 kB 00:00 2024-09-15T01:50:17.801 INFO:teuthology.orchestra.run.smithi017.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 216 kB/s | 26 kB 00:00 2024-09-15T01:50:17.859 INFO:teuthology.orchestra.run.smithi017.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 268 kB/s | 47 kB 00:00 2024-09-15T01:50:17.910 INFO:teuthology.orchestra.run.smithi017.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-15T01:50:17.935 INFO:teuthology.orchestra.run.smithi017.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-15T01:50:17.969 INFO:teuthology.orchestra.run.smithi017.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-15T01:50:18.000 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T01:50:18.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:18.000 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:18.000 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:18.000 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:18.001 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:18.002 INFO:teuthology.orchestra.run.smithi017.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T01:50:18.028 INFO:teuthology.orchestra.run.smithi017.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-15T01:50:18.053 INFO:teuthology.orchestra.run.smithi017.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.5 MB/s | 220 kB 00:00 2024-09-15T01:50:18.078 INFO:teuthology.orchestra.run.smithi017.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-15T01:50:18.095 INFO:teuthology.orchestra.run.smithi017.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-15T01:50:18.145 INFO:teuthology.orchestra.run.smithi017.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-15T01:50:18.170 INFO:teuthology.orchestra.run.smithi017.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 975 kB/s | 90 kB 00:00 2024-09-15T01:50:18.173 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:18.173 INFO:teuthology.orchestra.run.smithi017.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-15T01:50:18.238 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:18.255 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:18.255 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:18.351 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:18.353 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:18.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-09-15T01:50:18.910 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:18.965 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:13 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:19.093 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T01:50:19.105 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T01:50:19.106 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T01:50:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.2 M 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-09-15T01:50:19.108 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T01:50:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T01:50:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T01:50:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T01:50:19.397 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T01:50:19.461 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 26 kB 00:00 2024-09-15T01:50:19.486 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 19 kB 00:00 2024-09-15T01:50:19.508 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T01:50:19.511 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 267 kB/s | 47 kB 00:00 2024-09-15T01:50:19.568 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T01:50:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-15T01:50:19.670 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-15T01:50:19.704 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 796 kB/s | 54 kB 00:00 2024-09-15T01:50:19.746 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-15T01:50:19.771 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 319 kB/s | 32 kB 00:00 2024-09-15T01:50:19.805 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-15T01:50:19.830 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-15T01:50:19.864 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-15T01:50:19.889 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-15T01:50:19.922 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-09-15T01:50:19.925 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:19.925 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-15T01:50:19.992 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:20.008 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:20.008 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:20.104 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:20.104 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T01:50:20.380 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T01:50:20.414 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T01:50:20.460 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T01:50:20.492 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T01:50:20.646 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T01:50:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T01:50:20.947 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T01:50:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T01:50:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T01:50:21.131 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T01:50:21.258 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T01:50:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T01:50:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T01:50:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T01:50:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T01:50:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T01:50:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T01:50:21.328 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T01:50:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T01:50:21.640 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T01:50:21.641 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T01:50:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T01:50:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:21.642 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:21.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T01:50:22.089 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T01:50:22.157 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T01:50:22.191 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T01:50:22.237 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T01:50:22.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T01:50:22.476 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:18 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:22.615 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:50:22.616 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2024-09-15T01:50:22.617 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:22.618 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 7.0 M 2024-09-15T01:50:22.618 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 33 M 2024-09-15T01:50:22.618 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T01:50:22.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T01:50:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T01:50:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 559 kB/s | 124 kB 00:00 2024-09-15T01:50:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-15T01:50:23.299 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T01:50:23.323 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:23.324 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-09-15T01:50:23.349 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-15T01:50:23.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T01:50:24.144 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:19 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:24.284 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:24.284 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:24.284 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:24.284 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:24.285 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-09-15T01:50:24.286 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:24.286 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 7.0 M 2024-09-15T01:50:24.286 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-09-15T01:50:24.286 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:24.616 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-15T01:50:24.666 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T01:50:24.691 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 572 kB/s | 124 kB 00:00 2024-09-15T01:50:24.717 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T01:50:24.817 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-15T01:50:24.842 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 496 kB/s | 74 kB 00:00 2024-09-15T01:50:24.844 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:24.845 INFO:teuthology.orchestra.run.smithi067.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-15T01:50:24.883 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-09-15T01:50:24.885 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:24.886 INFO:teuthology.orchestra.run.smithi017.stdout:Total 3.1 MB/s | 7.0 MB 00:02 2024-09-15T01:50:24.917 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:24.930 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:24.961 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:24.974 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:24.974 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:25.035 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:25.036 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:25.080 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:25.081 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:25.395 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T01:50:25.572 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T01:50:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T01:50:25.619 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T01:50:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T01:50:26.171 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T01:50:26.267 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T01:50:26.285 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T01:50:26.329 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T01:50:26.344 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T01:50:26.356 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T01:50:26.371 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T01:50:27.139 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T01:50:27.144 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T01:50:27.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T01:50:27.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T01:50:27.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T01:50:27.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T01:50:27.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T01:50:27.393 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T01:50:27.393 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:27.393 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:27.393 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:27.393 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:27.394 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:27.433 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:27.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-09-15T01:50:27.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-fuse 2024-09-15T01:50:28.242 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:28.262 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:28.379 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T01:50:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 893 k 2024-09-15T01:50:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.6 M 2024-09-15T01:50:28.382 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:28.408 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:28.409 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 893 k 2024-09-15T01:50:28.410 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 2.6 M 2024-09-15T01:50:28.410 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:28.743 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 761 kB/s | 80 kB 00:00 2024-09-15T01:50:28.819 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 527 kB/s | 80 kB 00:00 2024-09-15T01:50:28.944 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.6 MB/s | 813 kB 00:00 2024-09-15T01:50:28.944 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:28.944 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-15T01:50:28.952 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-15T01:50:28.952 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:28.953 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-15T01:50:28.957 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:28.965 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:28.981 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:28.981 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:28.987 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:28.987 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:29.065 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:29.066 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:29.071 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:29.071 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:29.258 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:29.400 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T01:50:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T01:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T01:50:29.479 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T01:50:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T01:50:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T01:50:30.180 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T01:50:30.181 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T01:50:30.446 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T01:50:30.446 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:30.446 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:30.447 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T01:50:30.447 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:30.447 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T01:50:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:30.479 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:30.480 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T01:50:30.480 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:30.480 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:30.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-09-15T01:50:30.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-volume 2024-09-15T01:50:31.326 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:31.336 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:27 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:31.465 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:31.466 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:31.466 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T01:50:31.466 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:31.466 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 270 k 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.3 M 2024-09-15T01:50:31.467 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:31.477 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:31.478 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:31.479 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 270 k 2024-09-15T01:50:31.479 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 1.3 M 2024-09-15T01:50:31.479 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:31.681 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T01:50:31.682 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:31.682 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T01:50:31.682 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:31.691 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:31.691 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:31.714 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-15T01:50:31.714 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:31.715 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T01:50:31.715 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:31.721 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:31.724 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:31.724 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:31.754 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:31.755 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:31.941 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:32.008 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:32.025 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:32.025 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:50:32.025 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T01:50:32.025 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:32.067 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:50:32.067 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T01:50:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:33.040 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:33.040 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:33.083 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T01:50:33.084 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:33.084 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:33.084 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T01:50:33.084 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:33.084 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:33.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-09-15T01:50:33.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados-devel 2024-09-15T01:50:33.907 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:33.964 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:34.046 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:34.046 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 126 k 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 448 k 2024-09-15T01:50:34.047 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:34.103 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:34.104 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:34.104 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-15T01:50:34.104 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:34.104 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 126 k 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 448 k 2024-09-15T01:50:34.105 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:34.386 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 373 kB/s | 126 kB 00:00 2024-09-15T01:50:34.386 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:34.386 INFO:teuthology.orchestra.run.smithi067.stdout:Total 372 kB/s | 126 kB 00:00 2024-09-15T01:50:34.387 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:34.393 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:34.393 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:34.427 INFO:teuthology.orchestra.run.smithi017.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 392 kB/s | 126 kB 00:00 2024-09-15T01:50:34.428 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:34.428 INFO:teuthology.orchestra.run.smithi017.stdout:Total 390 kB/s | 126 kB 00:00 2024-09-15T01:50:34.428 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:34.434 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:34.434 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:34.459 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:34.460 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:34.502 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:34.503 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:34.665 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:34.720 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:34.770 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:35.155 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:35.430 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:35.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:35.491 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:35.491 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:35.491 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:35.492 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:35.492 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:35.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-09-15T01:50:35.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs2 2024-09-15T01:50:36.240 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:36.334 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:36.349 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:36.381 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:36.382 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:36.382 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:36.443 INFO:teuthology.orchestra.run.smithi017.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:36.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-09-15T01:50:36.489 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:36.490 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:36.490 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:36.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs-devel 2024-09-15T01:50:37.037 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:37.156 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:37.175 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:37.176 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:37.176 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T01:50:37.176 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:37.176 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:37.176 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 25 k 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 110 k 2024-09-15T01:50:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:37.296 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:37.296 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:37.296 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 25 k 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 110 k 2024-09-15T01:50:37.297 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:37.331 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 162 kB/s | 25 kB 00:00 2024-09-15T01:50:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:Total 161 kB/s | 25 kB 00:00 2024-09-15T01:50:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:37.426 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:37.490 INFO:teuthology.orchestra.run.smithi017.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 129 kB/s | 25 kB 00:00 2024-09-15T01:50:37.491 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:37.491 INFO:teuthology.orchestra.run.smithi017.stdout:Total 129 kB/s | 25 kB 00:00 2024-09-15T01:50:37.491 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:37.494 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:37.494 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:37.496 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:37.496 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:37.539 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:37.581 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:37.698 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:37.898 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:38.148 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:38.299 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:38.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-09-15T01:50:38.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados2 2024-09-15T01:50:38.924 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:39.019 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:39.064 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:39.065 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:39.065 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:39.075 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:39.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-09-15T01:50:39.168 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:39.214 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:39.215 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:39.215 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:39.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librbd1 2024-09-15T01:50:39.715 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:39.808 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:39.854 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:39.855 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:39.855 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:39.871 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:39.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-09-15T01:50:39.964 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:40.010 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:40.011 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:40.011 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:40.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rados 2024-09-15T01:50:40.517 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:40.611 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:40.656 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:40.657 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:40.657 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:40.676 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:40.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-09-15T01:50:40.771 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:40.816 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:40.817 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:40.817 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:40.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rgw 2024-09-15T01:50:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:41.406 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:41.451 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:41.452 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:41.452 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:41.474 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:41.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-09-15T01:50:41.567 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:41.612 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:41.613 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:41.613 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:41.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-cephfs 2024-09-15T01:50:42.112 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:42.204 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:42.249 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:42.250 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:42.250 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:42.276 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:42.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-09-15T01:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:42.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rbd 2024-09-15T01:50:42.903 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:42.996 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:43.043 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:43.044 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T01:50:43.044 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:43.076 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:43.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-09-15T01:50:43.169 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T01:50:43.214 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:43.215 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-15T01:50:43.215 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:43.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-fuse 2024-09-15T01:50:43.700 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:43.839 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:43.840 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:43.841 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:43.841 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 82 k 2024-09-15T01:50:43.841 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 222 k 2024-09-15T01:50:43.841 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:43.887 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:44.017 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 463 kB/s | 82 kB 00:00 2024-09-15T01:50:44.018 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:44.018 INFO:teuthology.orchestra.run.smithi067.stdout:Total 460 kB/s | 82 kB 00:00 2024-09-15T01:50:44.018 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:44.025 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:44.026 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:44.026 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:44.027 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:44.028 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 82 k 2024-09-15T01:50:44.028 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 222 k 2024-09-15T01:50:44.028 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:44.092 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:44.092 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:44.226 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 412 kB/s | 82 kB 00:00 2024-09-15T01:50:44.227 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:44.227 INFO:teuthology.orchestra.run.smithi017.stdout:Total 410 kB/s | 82 kB 00:00 2024-09-15T01:50:44.227 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:44.235 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:44.235 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:44.302 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:44.302 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:44.450 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:44.552 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:44.918 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:45.038 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:45.039 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:45.039 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:45.039 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:45.039 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:45.039 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:45.170 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:45.170 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:45.171 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:45.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-09-15T01:50:45.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-mirror 2024-09-15T01:50:45.865 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:45.995 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:46.004 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:46.005 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.9 M 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 12 M 2024-09-15T01:50:46.006 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:46.134 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:46.135 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:46.136 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 2.9 M 2024-09-15T01:50:46.136 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 12 M 2024-09-15T01:50:46.136 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:46.354 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.4 MB/s | 2.9 MB 00:00 2024-09-15T01:50:46.355 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:46.355 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-15T01:50:46.355 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:46.365 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:46.365 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:46.449 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:46.449 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:46.472 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-15T01:50:46.473 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:46.473 INFO:teuthology.orchestra.run.smithi017.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-15T01:50:46.473 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:46.483 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:46.483 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:46.566 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:46.567 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:46.888 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:46.963 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:46.989 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:46.990 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:50:46.990 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T01:50:46.990 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T01:50:46.990 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T01:50:46.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:47.001 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:47.073 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:47.099 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:47.099 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T01:50:47.099 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T01:50:47.099 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T01:50:47.100 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T01:50:47.100 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:48.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:48.146 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:48.146 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:48.146 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:48.146 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:48.146 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:48.262 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:48.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-09-15T01:50:48.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2024-09-15T01:50:48.975 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 01:49:05 AM UTC. 2024-09-15T01:50:49.094 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 01:49:04 AM UTC. 2024-09-15T01:50:49.114 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T01:50:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:49.116 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T01:50:49.116 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T01:50:49.116 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T01:50:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:49.116 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 165 k 2024-09-15T01:50:49.117 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 477 k 2024-09-15T01:50:49.117 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T01:50:49.235 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-15T01:50:49.236 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:49.237 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-15T01:50:49.237 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:49.237 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-15T01:50:49.237 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T01:50:49.237 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 165 k 2024-09-15T01:50:49.238 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 477 k 2024-09-15T01:50:49.239 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-15T01:50:49.333 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 767 kB/s | 165 kB 00:00 2024-09-15T01:50:49.333 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:49.333 INFO:teuthology.orchestra.run.smithi067.stdout:Total 761 kB/s | 165 kB 00:00 2024-09-15T01:50:49.333 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T01:50:49.342 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T01:50:49.342 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T01:50:49.409 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T01:50:49.409 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T01:50:49.443 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 804 kB/s | 165 kB 00:00 2024-09-15T01:50:49.444 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:50:49.444 INFO:teuthology.orchestra.run.smithi017.stdout:Total 797 kB/s | 165 kB 00:00 2024-09-15T01:50:49.445 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-15T01:50:49.453 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-15T01:50:49.453 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-15T01:50:49.521 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-15T01:50:49.521 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-15T01:50:49.569 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T01:50:49.677 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-15T01:50:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:49.785 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:50.222 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:50.427 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:50.428 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T01:50:50.648 DEBUG:teuthology.parallel:result is None 2024-09-15T01:50:50.653 DEBUG:teuthology.parallel:result is None 2024-09-15T01:50:50.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:50:50.801 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T01:50:50.823 INFO:teuthology.orchestra.run.smithi017.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T01:50:50.824 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T01:50:50.824 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T01:50:50.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:50:50.976 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T01:50:50.999 INFO:teuthology.orchestra.run.smithi067.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T01:50:50.999 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T01:50:50.999 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T01:50:51.001 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T01:50:51.001 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:51.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T01:50:51.029 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:51.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T01:50:51.067 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T01:50:51.067 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:51.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T01:50:51.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T01:50:51.161 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:51.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T01:50:51.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T01:50:51.253 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T01:50:51.253 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:51.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T01:50:51.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T01:50:51.352 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:51.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T01:50:51.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T01:50:51.443 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T01:50:51.444 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:51.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T01:50:51.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T01:50:51.535 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:51.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T01:50:51.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T01:50:51.627 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T01:50:51.700 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T01:50:51.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2024-09-15T01:50:51.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-09-15T01:50:51.733 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T01:50:51.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T01:50:51.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T01:50:51.802 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T01:50:51.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T01:50:51.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T01:50:51.956 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T01:50:51.956 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T01:50:51.956 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_workunits}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905072', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMXm5r0SM8bmxVjX0Ciz83GQGz1IKMSgovUu0dBZC9AGSosx9koPM+UdFQ8ksA6247zid+jtWgi222Gy5uAK/A=', 'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB2ccedPMpmbmUqpQv7FzibDf3Wm6PTsep+hZZE3613EYAF8hZOhwdnXMkq1pu0Fa+8wgXYLSEOB9AxDd+Ot82Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T01:50:51.957 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T01:50:51.959 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T01:50:51.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T01:50:52.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T01:50:52.040 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:52.040 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T01:50:52.094 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T01:50:52.094 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-09-15T01:50:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-15 01:50:47.379034404 +0000 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-15 01:48:24.819620132 +0000 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-15 01:48:24.819620132 +0000 2024-09-15T01:50:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-15 01:48:24.819620132 +0000 2024-09-15T01:50:52.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T01:50:52.214 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-15T01:50:52.214 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-15T01:50:52.214 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00010009 s, 5.1 MB/s 2024-09-15T01:50:52.215 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T01:50:52.272 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-09-15T01:50:52.327 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T01:50:52.327 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:52.327 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-15 01:50:47.380034394 +0000 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-15 01:48:25.135616928 +0000 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-15 01:48:25.135616928 +0000 2024-09-15T01:50:52.328 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-15 01:48:25.135616928 +0000 2024-09-15T01:50:52.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T01:50:52.392 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-15T01:50:52.393 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-15T01:50:52.393 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00010046 s, 5.1 MB/s 2024-09-15T01:50:52.394 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T01:50:52.450 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-09-15T01:50:52.506 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-15 01:50:47.380034394 +0000 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-15 01:48:25.459613644 +0000 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-15 01:48:25.459613644 +0000 2024-09-15T01:50:52.507 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-15 01:48:25.459613644 +0000 2024-09-15T01:50:52.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T01:50:52.571 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-15T01:50:52.571 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-15T01:50:52.571 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000100914 s, 5.1 MB/s 2024-09-15T01:50:52.572 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T01:50:52.629 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-09-15T01:50:52.684 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T01:50:52.684 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:52.684 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T01:50:52.684 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:52.684 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:52.685 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-15 01:50:47.380034394 +0000 2024-09-15T01:50:52.685 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-15 01:48:25.778610411 +0000 2024-09-15T01:50:52.685 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-15 01:48:25.778610411 +0000 2024-09-15T01:50:52.685 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-15 01:48:25.778610411 +0000 2024-09-15T01:50:52.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T01:50:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-15T01:50:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-15T01:50:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 8.4542e-05 s, 6.1 MB/s 2024-09-15T01:50:52.750 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T01:50:52.807 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T01:50:52.808 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:52.808 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T01:50:52.830 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T01:50:52.830 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-09-15T01:50:52.886 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 01:50:47.251016235 +0000 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 01:48:24.851257758 +0000 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 01:48:24.851257758 +0000 2024-09-15T01:50:52.887 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 01:48:24.851257758 +0000 2024-09-15T01:50:52.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T01:50:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T01:50:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T01:50:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000348914 s, 1.5 MB/s 2024-09-15T01:50:52.954 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T01:50:53.011 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-09-15T01:50:53.068 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 01:50:47.251016235 +0000 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 01:48:25.201249763 +0000 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 01:48:25.201249763 +0000 2024-09-15T01:50:53.069 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 01:48:25.201249763 +0000 2024-09-15T01:50:53.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T01:50:53.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T01:50:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T01:50:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000193249 s, 2.6 MB/s 2024-09-15T01:50:53.135 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T01:50:53.192 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:53.248 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 01:50:47.252016212 +0000 2024-09-15T01:50:53.249 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 01:48:25.536242110 +0000 2024-09-15T01:50:53.249 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 01:48:25.536242110 +0000 2024-09-15T01:50:53.249 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 01:48:25.536242110 +0000 2024-09-15T01:50:53.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T01:50:53.312 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T01:50:53.312 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T01:50:53.312 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000207517 s, 2.5 MB/s 2024-09-15T01:50:53.313 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T01:50:53.370 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 01:50:47.252016212 +0000 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 01:48:25.862234664 +0000 2024-09-15T01:50:53.426 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 01:48:25.862234664 +0000 2024-09-15T01:50:53.427 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 01:48:25.862234664 +0000 2024-09-15T01:50:53.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T01:50:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T01:50:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T01:50:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000212753 s, 2.4 MB/s 2024-09-15T01:50:53.491 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T01:50:53.548 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T01:50:53.548 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T01:50:53.548 INFO:tasks.ceph:Generating config... 2024-09-15T01:50:53.553 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T01:50:53.553 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T01:50:53.553 INFO:tasks.ceph:[global] ms type = async 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-15T01:50:53.554 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T01:50:53.555 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T01:50:53.555 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T01:50:53.556 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T01:50:53.557 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T01:50:53.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T01:50:53.609 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T01:50:53.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T01:50:53.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T01:50:53.726 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.17'), ('mon.c', '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]'), ('mon.b', '172.21.15.67')] 2024-09-15T01:50:53.726 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.67', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T01:50:53.727 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:53.727 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T01:50:53.783 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.17 --addv c '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]' --add b 172.21.15.67 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = octopus 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 15 (octopus) 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:election_strategy: 1 2024-09-15T01:50:53.867 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:50:53.868 INFO:teuthology.orchestra.run.smithi017.stdout:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:50:53.868 INFO:teuthology.orchestra.run.smithi017.stdout:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:50:53.868 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T01:50:53.869 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T01:50:53.886 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e3089478-c52a-4174-94a2-258b1876359d... 2024-09-15T01:50:53.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T01:50:53.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T01:50:53.973 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-09-15T01:50:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2024-09-15T01:50:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T01:50:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2024-09-15T01:50:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T01:50:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2024-09-15T01:50:53.975 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2024-09-15T01:50:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.67" 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: mon client directed command retry = 5 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: mon election default strategy = 1 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject delay max = 1 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject delay probability = 0.005 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject delay type = mon 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject internal delays = 0.002 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject socket failures = 2500 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: ms type = async 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.977 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2024-09-15T01:50:53.978 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2024-09-15T01:50:53.979 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bdev async discard = True 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bdev enable discard = True 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore allocator = bitmap 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore block size = 96636764160 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore fsck on mount = True 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore zero block detection = True 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluefs = 20 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluestore = 20 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: debug rocksdb = 10 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: mon client ping interval = 4 2024-09-15T01:50:53.980 INFO:teuthology.orchestra.run.smithi017.stdout: mon client ping timeout = 12 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd full ratio = 0.9 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd nearfull ratio = 0.8 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify cached snaps = True 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify missing on start = True 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd failsafe full ratio = 0.95 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock override recovery settings = True 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd objectstore = bluestore 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue = debug_random 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue cut off = debug_random 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2024-09-15T01:50:53.981 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug monc = 10 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2024-09-15T01:50:53.982 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T01:50:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon min osdmap epochs = 25 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune interval = 2 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune min = 15 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon osdmap full prune txsize = 2 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon scrub inject missing keys = 0.05 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: mon scrub interval = 300 2024-09-15T01:50:53.984 INFO:teuthology.orchestra.run.smithi017.stdout: paxos service trim min = 5 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: debug monc = 20 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.c] 2024-09-15T01:50:53.985 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.b] 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-09-15T01:50:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.67" 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon election default strategy = 1 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay max = 1 2024-09-15T01:50:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay probability = 0.005 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay type = mon 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject internal delays = 0.002 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 2500 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: ms type = async 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-09-15T01:50:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-09-15T01:50:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore zero block detection = True 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon client ping interval = 4 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon client ping timeout = 12 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify cached snaps = True 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify missing on start = True 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-09-15T01:50:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock override recovery settings = True 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue = debug_random 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue cut off = debug_random 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-15T01:50:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 10 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-09-15T01:50:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon min osdmap epochs = 25 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune interval = 2 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune min = 15 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune txsize = 2 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub inject missing keys = 0.05 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub interval = 300 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: paxos service trim min = 5 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T01:50:53.997 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-09-15T01:50:53.998 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-09-15T01:50:53.999 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T01:50:54.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T01:50:54.090 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T01:50:54.091 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:54.091 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T01:50:54.106 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:54.106 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T01:50:54.162 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T01:50:54.162 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:54.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T01:50:54.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T01:50:54.237 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:50:54.237 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:54.294 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T01:50:54.294 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:54.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T01:50:54.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T01:50:54.332 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:50:54.332 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:54.389 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T01:50:54.389 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T01:50:54.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T01:50:54.485 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T01:50:54.488 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T01:50:54.488 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T01:50:54.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T01:50:54.527 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T01:50:54.539 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T01:50:54.540 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T01:50:54.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T01:50:54.566 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T01:50:54.566 INFO:tasks.ceph:role: osd.0 2024-09-15T01:50:54.566 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T01:50:54.566 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:50:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:50:54.665 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-15T01:50:54.667 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-15T01:50:54.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T01:50:54.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T01:50:54.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T01:50:54.878 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T01:50:54.878 INFO:tasks.ceph:role: osd.1 2024-09-15T01:50:54.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T01:50:54.878 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T01:50:54.943 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:50:54.943 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:50:54.943 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:50:54.943 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:50:54.944 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:50:54.976 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-15T01:50:54.977 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-15T01:50:54.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T01:50:55.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T01:50:55.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T01:50:55.184 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T01:50:55.184 INFO:tasks.ceph:role: osd.2 2024-09-15T01:50:55.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T01:50:55.184 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T01:50:55.249 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:50:55.249 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:50:55.249 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:50:55.249 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:50:55.249 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:50:55.250 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:50:55.250 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:50:55.250 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:50:55.250 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:50:55.250 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:50:55.281 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-15T01:50:55.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-15T01:50:55.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T01:50:55.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T01:50:55.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:55.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T01:50:55.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T01:50:55.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T01:50:55.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T01:50:55.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T01:50:55.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:55.519+0000 7f0342c0b440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T01:50:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:56.042+0000 7f0342c0b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:50:56.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:56.808+0000 7f0342c0b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:50:57.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:57.571+0000 7f0342c0b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:50:58.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T01:50:58.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.491+0000 7f4fe0e14440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.491+0000 7f4fe0e14440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.491+0000 7f4fe0e14440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.491+0000 7f4fe0e14440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.491+0000 7f4fe0e14440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T01:50:58.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:58.492+0000 7f4fe0e14440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T01:50:59.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:59.019+0000 7f4fe0e14440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:50:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:50:59.784+0000 7f4fe0e14440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:00.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:00.546+0000 7f4fe0e14440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:01.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T01:51:01.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.463+0000 7f2584610440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.464+0000 7f2584610440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.464+0000 7f2584610440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.464+0000 7f2584610440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.464+0000 7f2584610440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T01:51:01.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.464+0000 7f2584610440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T01:51:01.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:01.991+0000 7f2584610440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:02.753+0000 7f2584610440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:03.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:03.517+0000 7f2584610440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:04.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T01:51:04.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T01:51:04.376 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T01:51:04.376 INFO:tasks.ceph:role: osd.3 2024-09-15T01:51:04.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T01:51:04.376 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:51:04.453 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:51:04.454 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:51:04.454 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:51:04.968 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T01:51:04.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T01:51:04.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T01:51:05.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T01:51:05.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T01:51:05.239 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T01:51:05.239 INFO:tasks.ceph:role: osd.4 2024-09-15T01:51:05.240 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T01:51:05.240 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T01:51:05.315 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:51:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:51:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:51:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:51:05.766 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T01:51:05.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T01:51:05.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T01:51:05.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T01:51:05.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T01:51:05.966 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T01:51:05.966 INFO:tasks.ceph:role: osd.5 2024-09-15T01:51:05.966 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T01:51:05.967 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T01:51:06.040 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T01:51:06.040 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T01:51:06.040 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T01:51:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T01:51:06.526 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T01:51:06.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T01:51:06.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-15T01:51:06.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T01:51:06.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.801+0000 7f04eb40a440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T01:51:06.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.806+0000 7f04eb40a440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T01:51:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.806+0000 7f04eb40a440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T01:51:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.806+0000 7f04eb40a440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T01:51:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.806+0000 7f04eb40a440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T01:51:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:06.811+0000 7f04eb40a440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T01:51:07.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:07.340+0000 7f04eb40a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:08.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:08.106+0000 7f04eb40a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:08.872+0000 7f04eb40a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:09.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T01:51:09.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.784+0000 7fe68c77a440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T01:51:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.785+0000 7fe68c77a440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T01:51:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.785+0000 7fe68c77a440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T01:51:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.785+0000 7fe68c77a440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T01:51:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.785+0000 7fe68c77a440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T01:51:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:09.785+0000 7fe68c77a440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T01:51:10.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:10.310+0000 7fe68c77a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:11.072+0000 7fe68c77a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:11.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:11.839+0000 7fe68c77a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:12.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T01:51:12.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.752+0000 7feec9208440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T01:51:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.753+0000 7feec9208440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T01:51:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.753+0000 7feec9208440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T01:51:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.753+0000 7feec9208440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T01:51:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.753+0000 7feec9208440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T01:51:12.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:12.753+0000 7feec9208440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T01:51:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:13.284+0000 7feec9208440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:14.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:14.049+0000 7feec9208440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:14.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:14.812+0000 7feec9208440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T01:51:15.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T01:51:15.638 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T01:51:15.639 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:15.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T01:51:15.671 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:15.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T01:51:15.742 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:15.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T01:51:15.811 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:15.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T01:51:15.843 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:15.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T01:51:15.913 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:15.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T01:51:15.983 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:15.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T01:51:16.051 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:16.051 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T01:51:16.108 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T01:51:16.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T01:51:16.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T01:51:16.151 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2024-09-15T01:51:16.151 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB/PeZm7csMHxAAiaGi81zExsfdkl+G2GLv6w== 2024-09-15T01:51:16.151 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2024-09-15T01:51:16.151 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCCPeZmJfRnHRAA2bMSvAg5jCXBagUEWV+5bQ== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCFPeZmKGfDGxAAIHQW1X9UCi3y1P9SAM5yfQ== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+PeZm8hfjHBAACRxFptqjgiixR4RufHWSIQ== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.3] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCKPeZmZXLWLxAAJQ7tjufR8svfSBm7B9xpbw== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.4] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCNPeZmRuDZLhAA85STo1EBPIQO9UmFAk8cSQ== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.5] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCQPeZmC9fuLBAApaN5hdZFGJ5oTXAbsqqjXQ== 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-09-15T01:51:16.152 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQB+PeZmZjBbHxAA94Weq0bE34BjuvAcUliEcw== 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB/PeZm7csMHxAAiaGi81zExsfdkl+G2GLv6w== 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCCPeZmJfRnHRAA2bMSvAg5jCXBagUEWV+5bQ== 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-09-15T01:51:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCFPeZmKGfDGxAAIHQW1X9UCi3y1P9SAM5yfQ== 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+PeZm8hfjHBAACRxFptqjgiixR4RufHWSIQ== 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKPeZmZXLWLxAAJQ7tjufR8svfSBm7B9xpbw== 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCNPeZmRuDZLhAA85STo1EBPIQO9UmFAk8cSQ== 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCQPeZmC9fuLBAApaN5hdZFGJ5oTXAbsqqjXQ== 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-09-15T01:51:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+PeZmZjBbHxAA94Weq0bE34BjuvAcUliEcw== 2024-09-15T01:51:16.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T01:51:16.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T01:51:16.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T01:51:16.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T01:51:16.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T01:51:16.618 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T01:51:16.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T01:51:16.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T01:51:16.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T01:51:16.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T01:51:16.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T01:51:16.920 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T01:51:16.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T01:51:16.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T01:51:17.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T01:51:17.093 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:17.097 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T01:51:17.149 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T01:51:17.149 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T01:51:17.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T01:51:17.152 INFO:tasks.ceph.mon.a:Started 2024-09-15T01:51:17.152 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:51:17.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:51:17.154 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:51:17.154 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T01:51:17.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T01:51:17.192 INFO:tasks.ceph.mon.b:Started 2024-09-15T01:51:17.192 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T01:51:17.192 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T01:51:17.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T01:51:17.195 INFO:tasks.ceph.mgr.x:Started 2024-09-15T01:51:17.195 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T01:51:17.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 10 monclient: build_initial_monmap 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 10 monclient: monmap: 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:17.291916+0000 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:17.291916+0000 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:17.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 1 Processor -- start 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 1 -- start start 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.291+0000 7f6af164f640 10 monclient: init 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec131910 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 0x7f6aec09ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec09eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6aec068e80 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec131ce0 con 0x7f6aec068e80 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0a15e0 con 0x7f6aec131910 2024-09-15T01:51:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec09fef0 con 0x7f6aec09eb30 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6af164f640 10 monclient(hunting): authenticate will time out at 737.288289s 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40104/0 (socket says 172.21.15.17:40104) 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 -- 172.21.15.17:0/2721081946 learned_addr learned my addr 172.21.15.17:0/2721081946 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request con 0x7f6aec131910 auth_method 0 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 0x7f6aec09ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request con 0x7f6aec09eb30 auth_method 0 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 msgr2=0x7f6aec131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 msgr2=0x7f6aec131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aeaffd640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 msgr2=0x7f6aec09ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:51:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 msgr2=0x7f6aec09ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 0x7f6aec09ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.292+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 0x7f6aec09ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.294+0000 7f6aeb7fe640 1 -- 172.21.15.17:0/2721081946 >> v1:172.21.15.67:6789/0 conn(0x7f6aec068e80 legacy=0x7f6aec069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:17.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.468+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398552666 0 0) 0x7f6aec131ce0 con 0x7f6aec068e80 2024-09-15T01:51:17.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.468+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.468+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.468+0000 7f6ae9ffb640 10 monclient(hunting): my global_id is 4099 2024-09-15T01:51:17.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.468+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad80034e0 con 0x7f6aec068e80 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099443130 0 0) 0x7f6ad80034e0 con 0x7f6aec068e80 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 msgr2=0x7f6aec09ef30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.67:3300/0 conn(0x7f6aec09eb30 0x7f6aec09ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 msgr2=0x7f6aec131fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 --2- 172.21.15.17:0/2721081946 >> v2:172.21.15.17:3300/0 conn(0x7f6aec131910 0x7f6aec131fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: _finish_hunting 0 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0a0120 con 0x7f6aec068e80 2024-09-15T01:51:17.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f6ae4001770 con 0x7f6aec068e80 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: dump: 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:17.478 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 >> v1:172.21.15.67:6789/0 conn(0x7f6aec068e80 legacy=0x7f6aec069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ad8003b20 0x7f6ad8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): picked mon.c con 0x7f6ad8003b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6ad8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): picked mon.a con 0x7f6ad8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad80052f0 0x7f6ad800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): picked mon.b con 0x7f6ad80052f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0a15e0 con 0x7f6ad8004780 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec09fef0 con 0x7f6ad80052f0 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad800c330 con 0x7f6ad8003b20 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6ae9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ad8003b20 0x7f6ad8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6ad8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request con 0x7f6ad8003b20 auth_method 0 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request con 0x7f6ad8004780 auth_method 0 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad80052f0 0x7f6ad800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f6ad80052f0 auth_method 0 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ad8003b20 msgr2=0x7f6ad8003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ad8003b20 0x7f6ad8003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad80052f0 msgr2=0x7f6ad800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad80052f0 0x7f6ad800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 monclient: found mon.a 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80034e0 con 0x7f6ad8004780 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeb7fe640 1 -- 172.21.15.17:0/2721081946 reap_dead start 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6af164f640 5 monclient: authenticate success, global_id 4099 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6aeaffd640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ad8003b20 0x7f6ad8003f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:17.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.473+0000 7f6af164f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:17.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6aea7fc640 1 -- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 msgr2=0x7f6ad8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:17.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6ad8004bf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4006fd0 tx=0x7f6ad4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f53a562bbd8811 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad402add0 con 0x7f6ad8004780 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: dump: 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad402e070 con 0x7f6ad8004780 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6af164f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6af164f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6af164f640 4 set_mon_vals no callback set 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 1 -- 172.21.15.17:0/2721081946 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4034660 con 0x7f6ad8004780 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.476+0000 7f6ae9ffb640 10 monclient: dump: 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.490 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 10 monclient: shutdown 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 -- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 msgr2=0x7f6ad8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6ad8004bf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4006fd0 tx=0x7f6ad4008040 comp rx=0 tx=0).stop 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 -- 172.21.15.17:0/2721081946 shutdown_connections 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 --2- 172.21.15.17:0/2721081946 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6ad8004bf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 -- 172.21.15.17:0/2721081946 >> 172.21.15.17:0/2721081946 conn(0x7f6aec071470 msgr2=0x7f6aec0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 -- 172.21.15.17:0/2721081946 shutdown_connections 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.477+0000 7f6af164f640 1 -- 172.21.15.17:0/2721081946 wait complete. 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient: build_initial_monmap 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient: monmap: 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:17.492 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 1 Processor -- start 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 1 -- start start 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient: init 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.478+0000 7f6af164f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec11bbb0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6aec1310c0 0x7f6aec141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): picked mon.noname-b con 0x7f6aec1310c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec11e560 con 0x7f6ad8004780 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec11c000 con 0x7f6aec1310c0 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec11cd60 con 0x7f6aec11bbb0 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6af164f640 10 monclient(hunting): authenticate will time out at 737.475031s 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44028/0 (socket says 172.21.15.17:44028) 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40108/0 (socket says 172.21.15.17:40108) 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 -- 172.21.15.17:0/557491134 learned_addr learned my addr 172.21.15.17:0/557491134 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:17.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request con 0x7f6aec11bbb0 auth_method 0 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request con 0x7f6ad8004780 auth_method 0 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6aec1310c0 0x7f6aec141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f6aec1310c0 auth_method 0 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 -- 172.21.15.17:0/557491134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6aec1310c0 msgr2=0x7f6aec141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6aec1310c0 0x7f6aec141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 -- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 msgr2=0x7f6aec09ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 monclient: found mon.noname-c 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aea7fc640 1 -- 172.21.15.17:0/557491134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec11dcc0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.479+0000 7f6aeaffd640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6aea7fc640 1 -- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 msgr2=0x7f6aec130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6aea7fc640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ad402a890 tx=0x7f6ad4035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40335c0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6ae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6ae2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6ae2ffd640 10 monclient: dump: 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:17.497 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.481+0000 7f6ae2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4033720 con 0x7f6aec11bbb0 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 5 monclient: authenticate success, global_id 4103 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:17.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _renew_subs 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4033ae0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec11dfa0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 10 monclient: dump: 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:17.499 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae97fa640 4 set_mon_vals no callback set 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _renew_subs 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec11c780 con 0x7f6aec11bbb0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6ad40496f0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6ad403f790 con 0x7f6aec11bbb0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.482+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec0724a0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.485+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f6ad4039040 con 0x7f6aec11bbb0 2024-09-15T01:51:17.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.485+0000 7f6ae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:17.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.485+0000 7f6ae2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:17.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.547+0000 7f6af164f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T01:51:17.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.547+0000 7f6af164f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T01:51:17.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.547+0000 7f6af164f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:17.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.547+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6aec0724a0 con 0x7f6aec11bbb0 2024-09-15T01:51:17.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6ae2ffd640 1 -- 172.21.15.17:0/557491134 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6ad4030070 con 0x7f6aec11bbb0 2024-09-15T01:51:17.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6ae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T01:51:17.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6ae2ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T01:51:17.559 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 10 monclient: shutdown 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 msgr2=0x7f6aec130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ad402a890 tx=0x7f6ad4035000 comp rx=0 tx=0).stop 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 shutdown_connections 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6aec1310c0 0x7f6aec141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.557+0000 7f6af164f640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6aec11bbb0 0x7f6aec130b80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.558+0000 7f6af164f640 1 --2- 172.21.15.17:0/557491134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ad8004780 0x7f6aec09ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:17.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.558+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 >> 172.21.15.17:0/557491134 conn(0x7f6aec071470 msgr2=0x7f6aec134c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:17.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.558+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 shutdown_connections 2024-09-15T01:51:17.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:17.558+0000 7f6af164f640 1 -- 172.21.15.17:0/557491134 wait complete. 2024-09-15T01:51:17.568 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:17.565+0000 7f0b1f1c2ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T01:51:17.569 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T01:51:17.570 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:17.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T01:51:17.595 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:17.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T01:51:17.608 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:17.606+0000 7f0b1f1c2ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T01:51:17.660 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:17.658+0000 7f0b1f1c2ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T01:51:17.661 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-15T01:51:17.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T01:51:17.727 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:17.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T01:51:17.754 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:17.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T01:51:17.806 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:17.803+0000 7f0b1f1c2ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T01:51:17.820 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:51:17.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T01:51:17.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 4a708981-55e8-4fe3-ab03-03a86c8340f0 0 2024-09-15T01:51:17.919 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:17.917+0000 7f0b1f1c2ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T01:51:18.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 10 monclient: monmap: 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:18.014255+0000 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:18.014255+0000 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 1 Processor -- start 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 1 -- start start 2024-09-15T01:51:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.013+0000 7fc7dd79d640 10 monclient: init 2024-09-15T01:51:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8131910 0x7fc7d8131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d8131910 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7d809eb30 0x7fc7d809ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d809eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7d8068e80 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d8131ce0 con 0x7fc7d8068e80 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80a15e0 con 0x7fc7d809eb30 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d809fef0 con 0x7fc7d8131910 2024-09-15T01:51:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7dd79d640 10 monclient(hunting): authenticate will time out at 710.483341s 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7d809eb30 0x7fc7d809ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7d809eb30 0x7fc7d809ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54210/0 (socket says 172.21.15.67:54210) 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc7d8068e80 0x7fc7d8069300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45146/0 (socket says 172.21.15.67:45146) 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 -- 172.21.15.67:0/1721150887 learned_addr learned my addr 172.21.15.67:0/1721150887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): get_auth_request con 0x7fc7d809eb30 auth_method 0 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d6ffd640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8131910 0x7fc7d8131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7d8131910 auth_method 0 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 -- 172.21.15.67:0/1721150887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8131910 msgr2=0x7fc7d8131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8131910 0x7fc7d8131fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 -- 172.21.15.67:0/1721150887 >> v1:172.21.15.67:6789/0 conn(0x7fc7d8068e80 legacy=0x7fc7d8069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 monclient: found mon.noname-c 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.014+0000 7fc7d67fc640 1 -- 172.21.15.67:0/1721150887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d80a0120 con 0x7fc7d809eb30 2024-09-15T01:51:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d67fc640 1 -- 172.21.15.67:0/1721150887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 msgr2=0x7fc7d809ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d67fc640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d809ef00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009910 tx=0x7fc7cc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c026322216b5396a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 1 -- 172.21.15.67:0/1721150887 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc02aea0 con 0x7fc7d809eb30 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: dump: 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.024 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7dd79d640 5 monclient: authenticate success, global_id 4108 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 1 -- 172.21.15.67:0/1721150887 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc032440 con 0x7fc7d809eb30 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7dd79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7dd79d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7dd79d640 4 set_mon_vals no callback set 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 1 -- 172.21.15.67:0/1721150887 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0327e0 con 0x7fc7d809eb30 2024-09-15T01:51:18.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.017+0000 7fc7d5ffb640 10 monclient: dump: 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.026 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 10 monclient: shutdown 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 -- 172.21.15.67:0/1721150887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 msgr2=0x7fc7d809ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d809ef00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009910 tx=0x7fc7cc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 -- 172.21.15.67:0/1721150887 shutdown_connections 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d809ef00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/1721150887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8131910 0x7fc7d8131fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 -- 172.21.15.67:0/1721150887 >> 172.21.15.67:0/1721150887 conn(0x7fc7d8071470 msgr2=0x7fc7d80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 -- 172.21.15.67:0/1721150887 shutdown_connections 2024-09-15T01:51:18.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.018+0000 7fc7dd79d640 1 -- 172.21.15.67:0/1721150887 wait complete. 2024-09-15T01:51:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 10 monclient: monmap: 2024-09-15T01:51:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 1 Processor -- start 2024-09-15T01:51:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 1 -- start start 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 10 monclient: init 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.019+0000 7fc7dd79d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8068e80 0x7fc7d809a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d8068e80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d809eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 0x7fc7d811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7d8131910 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d809bf70 con 0x7fc7d8131910 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d811dec0 con 0x7fc7d809eb30 2024-09-15T01:51:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8141f70 con 0x7fc7d8068e80 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7dd79d640 10 monclient(hunting): authenticate will time out at 710.489065s 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54224/0 (socket says 172.21.15.67:54224) 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 -- 172.21.15.67:0/473451716 learned_addr learned my addr 172.21.15.67:0/473451716 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): get_auth_request con 0x7fc7d809eb30 auth_method 0 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 0x7fc7d811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d6ffd640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8068e80 0x7fc7d809a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 10 monclient(hunting): get_auth_request con 0x7fc7d8131910 auth_method 0 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7d8068e80 auth_method 0 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 -- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8068e80 msgr2=0x7fc7d809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8068e80 0x7fc7d809a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 -- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 msgr2=0x7fc7d811b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 0x7fc7d811b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 monclient: found mon.noname-b 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d67fc640 1 -- 172.21.15.67:0/473451716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc009450 con 0x7fc7d809eb30 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.020+0000 7fc7d77fe640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 0x7fc7d811b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.022+0000 7fc7d67fc640 1 -- 172.21.15.67:0/473451716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 msgr2=0x7fc7d8130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.022+0000 7fc7d67fc640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc035040 tx=0x7fc7cc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc02aea0 con 0x7fc7d809eb30 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: dump: 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc031c70 con 0x7fc7d809eb30 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 5 monclient: authenticate success, global_id 4111 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc03d440 con 0x7fc7d809eb30 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 10 monclient: dump: 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 10 monclient: _renew_subs 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d811dbe0 con 0x7fc7d809eb30 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7d57fa640 4 set_mon_vals no callback set 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 10 monclient: _renew_subs 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d809b440 con 0x7fc7d809eb30 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fc7cc039040 con 0x7fc7d809eb30 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.023+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc7cc046740 con 0x7fc7d809eb30 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.024+0000 7fc7dd79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.024+0000 7fc7dd79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.024+0000 7fc7dd79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.024+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d811e030 con 0x7fc7d809eb30 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.026+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fc7cc042070 con 0x7fc7d809eb30 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.026+0000 7fc7beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.026+0000 7fc7beffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.095+0000 7fc7dd79d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4a708981-55e8-4fe3-ab03-03a86c8340f0", "id": 0}] 2024-09-15T01:51:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.095+0000 7fc7dd79d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4a708981-55e8-4fe3-ab03-03a86c8340f0", "id": 0}] 2024-09-15T01:51:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.095+0000 7fc7dd79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.095+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a708981-55e8-4fe3-ab03-03a86c8340f0", "id": 0} v 0) v1 -- 0x7fc7d80724a0 con 0x7fc7d809eb30 2024-09-15T01:51:18.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.101+0000 7fc7beffd640 1 -- 172.21.15.67:0/473451716 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a708981-55e8-4fe3-ab03-03a86c8340f0", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7cc037070 con 0x7fc7d809eb30 2024-09-15T01:51:18.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.101+0000 7fc7beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4a708981-55e8-4fe3-ab03-03a86c8340f0", "id": 0}] 2024-09-15T01:51:18.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.101+0000 7fc7beffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:18.104 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 10 monclient: shutdown 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 msgr2=0x7fc7d8130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc035040 tx=0x7fc7cc032000 comp rx=0 tx=0).stop 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 shutdown_connections 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7d8131910 0x7fc7d811b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d809eb30 0x7fc7d8130ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 --2- 172.21.15.67:0/473451716 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7d8068e80 0x7fc7d809a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 >> 172.21.15.67:0/473451716 conn(0x7fc7d8071470 msgr2=0x7fc7d80a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 shutdown_connections 2024-09-15T01:51:18.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.102+0000 7fc7dd79d640 1 -- 172.21.15.67:0/473451716 wait complete. 2024-09-15T01:51:18.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new a9c20296-137e-4c23-9966-a98fd2ec1eec 1 2024-09-15T01:51:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.208+0000 7f130d00f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.208+0000 7f130d00f640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.208+0000 7f130d00f640 10 monclient: monmap: 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:18.209647+0000 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:18.209647+0000 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:18.212 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 Processor -- start 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 -- start start 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient: init 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1308131910 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f130809eb30 0x7f130809ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-c con 0x7f130809eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1308136c80 0x7f1308139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-b con 0x7f1308136c80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13081326d0 con 0x7f1308131910 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13080a15e0 con 0x7f130809eb30 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308131ce0 con 0x7f1308136c80 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130d00f640 10 monclient(hunting): authenticate will time out at 710.678636s 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f130809eb30 0x7f130809ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f130809eb30 0x7f130809ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54238/0 (socket says 172.21.15.67:54238) 2024-09-15T01:51:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 1 -- 172.21.15.67:0/2237230311 learned_addr learned my addr 172.21.15.67:0/2237230311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): get_auth_request con 0x7f130809eb30 auth_method 0 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.209+0000 7f130759e640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1308136c80 0x7f1308139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 1 -- 172.21.15.67:0/2237230311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1308136c80 msgr2=0x7f1308139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1308136c80 0x7f1308139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 1 -- 172.21.15.67:0/2237230311 >> v1:172.21.15.17:6789/0 conn(0x7f1308131910 legacy=0x7f1308131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 1 monclient: found mon.noname-c 2024-09-15T01:51:18.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.210+0000 7f130659c640 1 -- 172.21.15.67:0/2237230311 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130809fef0 con 0x7f130809eb30 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.212+0000 7f130659c640 1 -- 172.21.15.67:0/2237230311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f130809eb30 msgr2=0x7f130809ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.212+0000 7f130659c640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f130809eb30 0x7f130809ef30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009910 tx=0x7f12fc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=740c892af5d266b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 1 -- 172.21.15.67:0/2237230311 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc032440 con 0x7f130809eb30 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: dump: 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.225 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 1 -- 172.21.15.67:0/2237230311 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc0325a0 con 0x7f130809eb30 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 5 monclient: authenticate success, global_id 4114 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 1 -- 172.21.15.67:0/2237230311 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc032980 con 0x7f130809eb30 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 4 set_mon_vals no callback set 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f1305d9b640 10 monclient: dump: 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 10 monclient: shutdown 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 -- 172.21.15.67:0/2237230311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f130809eb30 msgr2=0x7f130809ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f130809eb30 0x7f130809ef30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009910 tx=0x7f12fc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 -- 172.21.15.67:0/2237230311 shutdown_connections 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1308136c80 0x7f1308139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 --2- 172.21.15.67:0/2237230311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f130809eb30 0x7f130809ef30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 -- 172.21.15.67:0/2237230311 >> 172.21.15.67:0/2237230311 conn(0x7f1308071470 msgr2=0x7f13080a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 -- 172.21.15.67:0/2237230311 shutdown_connections 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.213+0000 7f130d00f640 1 -- 172.21.15.67:0/2237230311 wait complete. 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.214+0000 7f130d00f640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.214+0000 7f130d00f640 10 monclient: monmap: 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.214+0000 7f130d00f640 1 Processor -- start 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.214+0000 7f130d00f640 1 -- start start 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.214+0000 7f130d00f640 10 monclient: init 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f1308141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-c con 0x7f130809eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f1308141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-a con 0x7f1308131910 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): picked mon.noname-b con 0x7f1308136c80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130809db20 con 0x7f1308131910 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1308099e70 con 0x7f1308136c80 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130809ade0 con 0x7f130809eb30 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130d00f640 10 monclient(hunting): authenticate will time out at 710.684298s 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54242/0 (socket says 172.21.15.67:54242) 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 learned_addr learned my addr 172.21.15.67:0/3538916629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): get_auth_request con 0x7f1308136c80 auth_method 0 2024-09-15T01:51:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f1308141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f1306d9d640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f1308141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 10 monclient(hunting): get_auth_request con 0x7f1308131910 auth_method 0 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f1306d9d640 10 monclient(hunting): get_auth_request con 0x7f130809eb30 auth_method 0 2024-09-15T01:51:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f1306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f1306d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f1306d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 msgr2=0x7f1308141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 msgr2=0x7f1308141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f1308141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130659c640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f1308141eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 msgr2=0x7f1308141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f1308141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 msgr2=0x7f1308141eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f1308141eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 monclient: found mon.noname-b 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.215+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009450 con 0x7f1308136c80 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.217+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 msgr2=0x7f13080975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.217+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f130000ce80 tx=0x7f1300007840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 msgr2=0x7f13080975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130000e610 con 0x7f1308136c80 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 msgr2=0x7f13080975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f130000ce80 tx=0x7f1300007840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f130000ce80 tx=0x7f1300007840 comp rx=0 tx=0).stop 2024-09-15T01:51:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: dump: 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 msgr2=0x7f13080975e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1308136c80 0x7f13080975e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f12f40056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): picked mon.a con 0x7f1308131910 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f12f40080a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): picked mon.c con 0x7f130809eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 0x7f12f400aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): picked mon.b con 0x7f12f4008a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f400b950 con 0x7f1308131910 2024-09-15T01:51:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f400ba90 con 0x7f12f4008a20 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f400bbd0 con 0x7f130809eb30 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f12eeffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 0x7f12f400aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): get_auth_request con 0x7f12f4008a20 auth_method 0 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f12f40056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): get_auth_request con 0x7f1308131910 auth_method 0 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f12f40080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 10 monclient(hunting): get_auth_request con 0x7f130809eb30 auth_method 0 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f1306d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient(hunting): handle_auth_done global_id 4117 payload 293 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 msgr2=0x7f12f40080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f12f40080a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 msgr2=0x7f12f40056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f12f40056b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 monclient: found mon.b 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f12f400be00 con 0x7f12f4008a20 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130759e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 5 monclient: authenticate success, global_id 4117 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 10 monclient: _renew_subs 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130659c640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f12f40080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130809b120 con 0x7f12f4008a20 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 10 monclient: _renew_subs 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.218+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130809b7c0 con 0x7f12f4008a20 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.219+0000 7f130d00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.219+0000 7f130d00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.219+0000 7f130d00f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.219+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1308099030 con 0x7f12f4008a20 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.220+0000 7f130759e640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 msgr2=0x7f12f400aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.220+0000 7f130759e640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 0x7f12f400aea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1300016000 tx=0x7f1300019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300007500 con 0x7f12f4008a20 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 10 monclient: dump: 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130000ee00 con 0x7f12f4008a20 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f130559a640 4 set_mon_vals no callback set 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1300003020 con 0x7f12f4008a20 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.221+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 4 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f1300012070 con 0x7f12f4008a20 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.223+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f130002b460 con 0x7f12f4008a20 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.223+0000 7f12eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.223+0000 7f12eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:18.243 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:18.228+0000 7f0b1f1c2ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T01:51:18.289 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:18.286+0000 7f0b1f1c2ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T01:51:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.290+0000 7f130d00f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a9c20296-137e-4c23-9966-a98fd2ec1eec", "id": 1}] 2024-09-15T01:51:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.290+0000 7f130d00f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a9c20296-137e-4c23-9966-a98fd2ec1eec", "id": 1}] 2024-09-15T01:51:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.290+0000 7f130d00f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.290+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a9c20296-137e-4c23-9966-a98fd2ec1eec", "id": 1} v 0) v1 -- 0x7f130809ab00 con 0x7f12f4008a20 2024-09-15T01:51:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.298+0000 7f12eeffd640 1 -- 172.21.15.67:0/3538916629 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a9c20296-137e-4c23-9966-a98fd2ec1eec", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f130002bcc0 con 0x7f12f4008a20 2024-09-15T01:51:18.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.298+0000 7f12eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a9c20296-137e-4c23-9966-a98fd2ec1eec", "id": 1}] 2024-09-15T01:51:18.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.298+0000 7f12eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:18.301 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 10 monclient: shutdown 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 msgr2=0x7f12f400aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 0x7f12f400aea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1300016000 tx=0x7f1300019000 comp rx=0 tx=0).stop 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 shutdown_connections 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1308131910 0x7f12f40056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f4008a20 0x7f12f400aea0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 --2- 172.21.15.67:0/3538916629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f130809eb30 0x7f12f40080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 >> 172.21.15.67:0/3538916629 conn(0x7f1308071470 msgr2=0x7f13080a3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 shutdown_connections 2024-09-15T01:51:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.299+0000 7f130d00f640 1 -- 172.21.15.67:0/3538916629 wait complete. 2024-09-15T01:51:18.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 4e77b2f6-442a-4b8e-a4f9-3243120ad5fa 2 2024-09-15T01:51:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 10 monclient: monmap: 2024-09-15T01:51:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:18.405219+0000 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:18.405219+0000 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 1 Processor -- start 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 1 -- start start 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.404+0000 7f9921f5d640 10 monclient: init 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f991c131910 0x7f991c131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f991c131910 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c09eb30 0x7f991c09ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f991c09eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f991c068e80 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c131ce0 con 0x7f991c068e80 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991c0a15e0 con 0x7f991c131910 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c09fef0 con 0x7f991c09eb30 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9921f5d640 10 monclient(hunting): authenticate will time out at 710.874390s 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f991c068e80 0x7f991c069300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45156/0 (socket says 172.21.15.67:45156) 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991bfff640 1 -- 172.21.15.67:0/4081966828 learned_addr learned my addr 172.21.15.67:0/4081966828 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:51:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4081966828 >> v2:172.21.15.17:3300/0 conn(0x7f991c131910 0x7f991c131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): get_auth_request con 0x7f991c131910 auth_method 0 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c09eb30 0x7f991c09ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): get_auth_request con 0x7f991c09eb30 auth_method 0 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f9912ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 1 -- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c09eb30 msgr2=0x7f991c09ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.405+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c09eb30 0x7f991c09ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 1 -- 172.21.15.67:0/4081966828 >> v1:172.21.15.67:6789/0 conn(0x7f991c068e80 legacy=0x7f991c069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.406+0000 7f991b7fe640 1 -- 172.21.15.67:0/4081966828 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0a0120 con 0x7f991c131910 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.408+0000 7f991b7fe640 1 -- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c131910 msgr2=0x7f991c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.408+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c131910 0x7f991c131fc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f990c009850 tx=0x7f990c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faea58c5f32ecebf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 1 -- 172.21.15.67:0/4081966828 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c02adf0 con 0x7f991c131910 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: dump: 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 1 -- 172.21.15.67:0/4081966828 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c032440 con 0x7f991c131910 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 5 monclient: authenticate success, global_id 4113 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 4 set_mon_vals no callback set 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 1 -- 172.21.15.67:0/4081966828 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c0327e0 con 0x7f991c131910 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f991affd640 10 monclient: dump: 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 10 monclient: shutdown 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 -- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c131910 msgr2=0x7f991c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c131910 0x7f991c131fc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f990c009850 tx=0x7f990c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 -- 172.21.15.67:0/4081966828 shutdown_connections 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c09eb30 0x7f991c09ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4081966828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c131910 0x7f991c131fc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 -- 172.21.15.67:0/4081966828 >> 172.21.15.67:0/4081966828 conn(0x7f991c071470 msgr2=0x7f991c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 -- 172.21.15.67:0/4081966828 shutdown_connections 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.409+0000 7f9921f5d640 1 -- 172.21.15.67:0/4081966828 wait complete. 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.410+0000 7f9921f5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.410+0000 7f9921f5d640 10 monclient: monmap: 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.410+0000 7f9921f5d640 1 Processor -- start 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.410+0000 7f9921f5d640 1 -- start start 2024-09-15T01:51:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.410+0000 7f9921f5d640 10 monclient: init 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f991c068e80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 0x7f991c12fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f991c09eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 0x7f991c099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f991c131910 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c11b860 con 0x7f991c09eb30 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c09bf70 con 0x7f991c068e80 2024-09-15T01:51:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c130ac0 con 0x7f991c131910 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9921f5d640 10 monclient(hunting): authenticate will time out at 710.880370s 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54258/0 (socket says 172.21.15.67:54258) 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 -- 172.21.15.67:0/4281141404 learned_addr learned my addr 172.21.15.67:0/4281141404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): get_auth_request con 0x7f991c068e80 auth_method 0 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 0x7f991c12fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991bfff640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 0x7f991c099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 10 monclient(hunting): get_auth_request con 0x7f991c09eb30 auth_method 0 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 -- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 msgr2=0x7f991c099be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 0x7f991c099be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 -- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 msgr2=0x7f991c12fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 0x7f991c12fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:51:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991b7fe640 1 -- 172.21.15.67:0/4281141404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c009390 con 0x7f991c068e80 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991bfff640 10 monclient: get_auth_request con 0x7f991c131910 auth_method 0 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f991bfff640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 0x7f991c099be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.411+0000 7f9912ffd640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 0x7f991c12fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f991b7fe640 1 -- 172.21.15.67:0/4281141404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 msgr2=0x7f991c1412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f991b7fe640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f990c0098b0 tx=0x7f990c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c031a10 con 0x7f991c068e80 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: dump: 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 5 monclient: authenticate success, global_id 4120 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 10 monclient: _renew_subs 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c031b70 con 0x7f991c068e80 2024-09-15T01:51:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c11cda0 con 0x7f991c068e80 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c03c400 con 0x7f991c068e80 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 10 monclient: dump: 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.428 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f991a7fc640 4 set_mon_vals no callback set 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 10 monclient: _renew_subs 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c11beb0 con 0x7f991c068e80 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f990c03c830 con 0x7f991c068e80 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.414+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f990c044820 con 0x7f991c068e80 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.415+0000 7f9921f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.415+0000 7f9921f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.415+0000 7f9921f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.415+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c0724a0 con 0x7f991c068e80 2024-09-15T01:51:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.417+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f990c0637a0 con 0x7f991c068e80 2024-09-15T01:51:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.417+0000 7f9913fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.417+0000 7f9913fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:18.430 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:18.428+0000 7f0b1f1c2ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T01:51:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.486+0000 7f9921f5d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e77b2f6-442a-4b8e-a4f9-3243120ad5fa", "id": 2}] 2024-09-15T01:51:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.487+0000 7f9921f5d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e77b2f6-442a-4b8e-a4f9-3243120ad5fa", "id": 2}] 2024-09-15T01:51:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.487+0000 7f9921f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.487+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e77b2f6-442a-4b8e-a4f9-3243120ad5fa", "id": 2} v 0) v1 -- 0x7f991c0a2320 con 0x7f991c068e80 2024-09-15T01:51:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.492+0000 7f9913fff640 1 -- 172.21.15.67:0/4281141404 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e77b2f6-442a-4b8e-a4f9-3243120ad5fa", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f990c052030 con 0x7f991c068e80 2024-09-15T01:51:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.492+0000 7f9913fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e77b2f6-442a-4b8e-a4f9-3243120ad5fa", "id": 2}] 2024-09-15T01:51:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.492+0000 7f9913fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:18.495 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 10 monclient: shutdown 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 msgr2=0x7f991c1412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f990c0098b0 tx=0x7f990c032000 comp rx=0 tx=0).stop 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 shutdown_connections 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f991c09eb30 0x7f991c12fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f991c131910 0x7f991c099be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 --2- 172.21.15.67:0/4281141404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f991c068e80 0x7f991c1412f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 >> 172.21.15.67:0/4281141404 conn(0x7f991c071470 msgr2=0x7f991c134d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 shutdown_connections 2024-09-15T01:51:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.493+0000 7f9921f5d640 1 -- 172.21.15.67:0/4281141404 wait complete. 2024-09-15T01:51:18.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 83be4a40-aa29-4bbb-817a-d31d9bff815e 3 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.599+0000 7f264e62b640 10 monclient: get_monmap_and_config 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.599+0000 7f264e62b640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.599+0000 7f264e62b640 10 monclient: monmap: 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:18.601115+0000 2024-09-15T01:51:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:18.601115+0000 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:18.603 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.600+0000 7f264e62b640 1 Processor -- start 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.600+0000 7f264e62b640 1 -- start start 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.600+0000 7f264e62b640 10 monclient: init 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2648131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-b con 0x7f2648131910 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f264809eb30 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2648136c80 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2648131ce0 con 0x7f2648136c80 2024-09-15T01:51:18.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264809fef0 con 0x7f264809eb30 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26480a15e0 con 0x7f2648131910 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f264e62b640 10 monclient(hunting): authenticate will time out at 711.070299s 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f26477fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f264809eb30 0x7f264809ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45168/0 (socket says 172.21.15.67:45168) 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f26477fe640 1 -- 172.21.15.67:0/3287177615 learned_addr learned my addr 172.21.15.67:0/3287177615 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2648131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 10 monclient(hunting): get_auth_request con 0x7f2648131910 auth_method 0 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 msgr2=0x7f2648131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 msgr2=0x7f2648131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2648131fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:18.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.601+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2648131fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f26477fe640 1 -- 172.21.15.67:0/3287177615 >> v1:172.21.15.67:6789/0 conn(0x7f264809eb30 legacy=0x7f264809ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267167769 0 0) 0x7f264809fef0 con 0x7f264809eb30 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 10 monclient(hunting): my global_id is 4123 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26300034e0 con 0x7f264809eb30 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f264cbc8640 1 -- 172.21.15.67:0/3287177615 >> v1:172.21.15.17:6789/0 conn(0x7f2648136c80 legacy=0x7f2648139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.603+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480396577 0 0) 0x7f2648131ce0 con 0x7f2648136c80 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): my global_id is 4119 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264809fef0 con 0x7f2648136c80 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361090384 0 0) 0x7f26300034e0 con 0x7f264809eb30 2024-09-15T01:51:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 msgr2=0x7f2648131fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2648131fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 >> v1:172.21.15.17:6789/0 conn(0x7f2648136c80 legacy=0x7f2648139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:51:18.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480a0120 con 0x7f264809eb30 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f263c0017e0 con 0x7f264809eb30 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264e62b640 5 monclient: authenticate success, global_id 4123 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264e62b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: dump: 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.619 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:51:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 >> v1:172.21.15.67:6789/0 conn(0x7f264809eb30 legacy=0x7f264809ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f2630003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): picked mon.b con 0x7f2630003b60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2630004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): picked mon.c con 0x7f2648131910 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2630004cc0 0x7f2630005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): picked mon.a con 0x7f2630004cc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480a15e0 con 0x7f2630004cc0 2024-09-15T01:51:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263000bbe0 con 0x7f2630003b60 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263000bda0 con 0x7f2648131910 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2646ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264e62b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f2630003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): get_auth_request con 0x7f2630003b60 auth_method 0 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2630004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2630004cc0 0x7f2630005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): get_auth_request con 0x7f2648131910 auth_method 0 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): get_auth_request con 0x7f2630004cc0 auth_method 0 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f264cbc8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 293 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 msgr2=0x7f2630004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648131910 0x7f2630004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2630004cc0 msgr2=0x7f2630005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2630004cc0 0x7f2630005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 monclient: found mon.b 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26300034e0 con 0x7f2630003b60 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.604+0000 7f2647fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2647fff640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 msgr2=0x7f2630003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2647fff640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f2630003f50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2638006a70 tx=0x7f2638008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f162592e9a145cad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263802ada0 con 0x7f2630003b60 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: dump: 2024-09-15T01:51:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f264e62b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263802e070 con 0x7f2630003b60 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f264e62b640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f264e62b640 4 set_mon_vals no callback set 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 1 -- 172.21.15.67:0/3287177615 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638034660 con 0x7f2630003b60 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.607+0000 7f2646ffd640 10 monclient: dump: 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 10 monclient: shutdown 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 -- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 msgr2=0x7f2630003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 --2- 172.21.15.67:0/3287177615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f2630003f50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2638006a70 tx=0x7f2638008040 comp rx=0 tx=0).stop 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264cbc8640 1 -- 172.21.15.67:0/3287177615 reap_dead start 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 -- 172.21.15.67:0/3287177615 shutdown_connections 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 -- 172.21.15.67:0/3287177615 >> 172.21.15.67:0/3287177615 conn(0x7f2648071470 msgr2=0x7f26480a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 -- 172.21.15.67:0/3287177615 shutdown_connections 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 1 -- 172.21.15.67:0/3287177615 wait complete. 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.608+0000 7f264e62b640 10 monclient: monmap: 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 Processor -- start 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 -- start start 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient: init 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-b con 0x7f2630003b60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 0x7f26480999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-a con 0x7f2648130f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648099f30 0x7f264809a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): picked mon.noname-c con 0x7f2648099f30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2648131310 con 0x7f2648130f20 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264809dbe0 con 0x7f2630003b60 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264809b8c0 con 0x7f2648099f30 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.609+0000 7f264e62b640 10 monclient(hunting): authenticate will time out at 711.078933s 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54280/0 (socket says 172.21.15.67:54280) 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 -- 172.21.15.67:0/4222454473 learned_addr learned my addr 172.21.15.67:0/4222454473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): get_auth_request con 0x7f2630003b60 auth_method 0 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 0x7f26480999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f264cbc8640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648099f30 0x7f264809a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 10 monclient(hunting): get_auth_request con 0x7f2648130f20 auth_method 0 2024-09-15T01:51:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f264cbc8640 10 monclient(hunting): get_auth_request con 0x7f2648099f30 auth_method 0 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f264cbc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f264cbc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f264cbc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 -- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648099f30 msgr2=0x7f264809a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648099f30 0x7f264809a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 -- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 msgr2=0x7f26480999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 0x7f26480999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 monclient: found mon.noname-b 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f2647fff640 1 -- 172.21.15.67:0/4222454473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264811cdd0 con 0x7f2630003b60 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.610+0000 7f26477fe640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 0x7f26480999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f2647fff640 1 -- 172.21.15.67:0/4222454473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 msgr2=0x7f26481309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f2647fff640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2638006ea0 tx=0x7f2638035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380335c0 con 0x7f2630003b60 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: dump: 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2638033720 con 0x7f2630003b60 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 5 monclient: authenticate success, global_id 4126 2024-09-15T01:51:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 10 monclient: _renew_subs 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2648141f30 con 0x7f2630003b60 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638033ae0 con 0x7f2630003b60 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f262ffff640 10 monclient: dump: 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 10 monclient: _renew_subs 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264811bb90 con 0x7f2630003b60 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.612+0000 7f26467fc640 4 set_mon_vals no callback set 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f26380496f0 con 0x7f2630003b60 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f263803f8e0 con 0x7f2630003b60 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f264e62b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f264e62b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f264e62b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.613+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26480724a0 con 0x7f2630003b60 2024-09-15T01:51:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.615+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f2638039040 con 0x7f2630003b60 2024-09-15T01:51:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.615+0000 7f262ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.615+0000 7f262ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.684+0000 7f264e62b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "83be4a40-aa29-4bbb-817a-d31d9bff815e", "id": 3}] 2024-09-15T01:51:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.684+0000 7f264e62b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "83be4a40-aa29-4bbb-817a-d31d9bff815e", "id": 3}] 2024-09-15T01:51:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.684+0000 7f264e62b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.684+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "83be4a40-aa29-4bbb-817a-d31d9bff815e", "id": 3} v 0) v1 -- 0x7f264809b3b0 con 0x7f2630003b60 2024-09-15T01:51:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.689+0000 7f262ffff640 1 -- 172.21.15.67:0/4222454473 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83be4a40-aa29-4bbb-817a-d31d9bff815e", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f264809b3b0 con 0x7f2630003b60 2024-09-15T01:51:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.689+0000 7f262ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "83be4a40-aa29-4bbb-817a-d31d9bff815e", "id": 3}] 2024-09-15T01:51:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.689+0000 7f262ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:18.692 INFO:teuthology.orchestra.run.smithi067.stdout:3 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 10 monclient: shutdown 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 msgr2=0x7f26481309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2638006ea0 tx=0x7f2638035000 comp rx=0 tx=0).stop 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 shutdown_connections 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2648130f20 0x7f26480999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2648099f30 0x7f264809a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 --2- 172.21.15.67:0/4222454473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630003b60 0x7f26481309e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 >> 172.21.15.67:0/4222454473 conn(0x7f2648071470 msgr2=0x7f26480a3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 shutdown_connections 2024-09-15T01:51:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.690+0000 7f264e62b640 1 -- 172.21.15.67:0/4222454473 wait complete. 2024-09-15T01:51:18.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 27d9cb97-3b36-4002-8665-cf5170791748 4 2024-09-15T01:51:18.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.796+0000 7f53eaaef640 10 monclient: get_monmap_and_config 2024-09-15T01:51:18.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.796+0000 7f53eaaef640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.797+0000 7f53eaaef640 10 monclient: monmap: 2024-09-15T01:51:18.799 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.799 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:18.798162+0000 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:18.798162+0000 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:18.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.797+0000 7f53eaaef640 1 Processor -- start 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.797+0000 7f53eaaef640 1 -- start start 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.797+0000 7f53eaaef640 10 monclient: init 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f53e4131910 0x7f53e4131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-a con 0x7f53e4131910 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:18.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e409ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-b con 0x7f53e409eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53e4068e80 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e4131ce0 con 0x7f53e4068e80 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e40a15e0 con 0x7f53e4131910 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e409fef0 con 0x7f53e409eb30 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53eaaef640 10 monclient(hunting): authenticate will time out at 711.267360s 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e908c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53e4068e80 0x7f53e4069300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45176/0 (socket says 172.21.15.67:45176) 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e908c640 1 -- 172.21.15.67:0/1588131352 learned_addr learned my addr 172.21.15.67:0/1588131352 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:51:18.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 --2- 172.21.15.67:0/1588131352 >> v2:172.21.15.17:3300/0 conn(0x7f53e4131910 0x7f53e4131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e409ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): get_auth_request con 0x7f53e4131910 auth_method 0 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): get_auth_request con 0x7f53e409eb30 auth_method 0 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 -- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 msgr2=0x7f53e409ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e409ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 -- 172.21.15.67:0/1588131352 >> v1:172.21.15.67:6789/0 conn(0x7f53e4068e80 legacy=0x7f53e4069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 monclient: found mon.noname-a 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:18.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.798+0000 7f53e888b640 1 -- 172.21.15.67:0/1588131352 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40a0120 con 0x7f53e4131910 2024-09-15T01:51:18.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.801+0000 7f53e888b640 1 -- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 msgr2=0x7f53e4131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.801+0000 7f53e888b640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4131fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009850 tx=0x7f53d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c009dff310776ab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 1 -- 172.21.15.67:0/1588131352 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d802adf0 con 0x7f53e4131910 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: dump: 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.813 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 1 -- 172.21.15.67:0/1588131352 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8032440 con 0x7f53e4131910 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 5 monclient: authenticate success, global_id 4125 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 4 set_mon_vals no callback set 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 1 -- 172.21.15.67:0/1588131352 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d80327e0 con 0x7f53e4131910 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:18.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53e37fe640 10 monclient: dump: 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 10 monclient: shutdown 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:18.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 -- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 msgr2=0x7f53e4131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4131fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009850 tx=0x7f53d80090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 -- 172.21.15.67:0/1588131352 shutdown_connections 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e409ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 --2- 172.21.15.67:0/1588131352 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4131fc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 -- 172.21.15.67:0/1588131352 >> 172.21.15.67:0/1588131352 conn(0x7f53e4071470 msgr2=0x7f53e40a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 -- 172.21.15.67:0/1588131352 shutdown_connections 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.802+0000 7f53eaaef640 1 -- 172.21.15.67:0/1588131352 wait complete. 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.803+0000 7f53eaaef640 10 monclient: build_initial_monmap 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.803+0000 7f53eaaef640 10 monclient: monmap: 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:18.816 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.803+0000 7f53eaaef640 1 Processor -- start 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.803+0000 7f53eaaef640 1 -- start start 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.803+0000 7f53eaaef640 10 monclient: init 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:18.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-b con 0x7f53e4068e80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e412fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-c con 0x7f53e409eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): picked mon.noname-a con 0x7f53e4131910 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e411b860 con 0x7f53e4131910 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e409bf70 con 0x7f53e4068e80 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e4130ac0 con 0x7f53e409eb30 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:18.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53eaaef640 10 monclient(hunting): authenticate will time out at 711.273340s 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54288/0 (socket says 172.21.15.67:54288) 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 -- 172.21.15.67:0/835064266 learned_addr learned my addr 172.21.15.67:0/835064266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): get_auth_request con 0x7f53e4068e80 auth_method 0 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e412fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): get_auth_request con 0x7f53e4131910 auth_method 0 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): get_auth_request con 0x7f53e409eb30 auth_method 0 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e908c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 -- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 msgr2=0x7f53e412fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e412fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 -- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 msgr2=0x7f53e4099be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4099be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 monclient: found mon.noname-b 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.804+0000 7f53e888b640 1 -- 172.21.15.67:0/835064266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d8009390 con 0x7f53e4068e80 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e888b640 1 -- 172.21.15.67:0/835064266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 msgr2=0x7f53e41412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e888b640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f53d80098b0 tx=0x7f53d8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8031a10 con 0x7f53e4068e80 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: dump: 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8031b70 con 0x7f53e4068e80 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 5 monclient: authenticate success, global_id 4129 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 10 monclient: _renew_subs 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e411cda0 con 0x7f53e4068e80 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d803c400 con 0x7f53e4068e80 2024-09-15T01:51:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 10 monclient: dump: 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:18.824 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e2ffd640 4 set_mon_vals no callback set 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 10 monclient: _renew_subs 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e411beb0 con 0x7f53e4068e80 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f53d803c830 con 0x7f53e4068e80 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.807+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f53d803b8b0 con 0x7f53e4068e80 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.808+0000 7f53eaaef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.808+0000 7f53eaaef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.808+0000 7f53eaaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.808+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e40724a0 con 0x7f53e4068e80 2024-09-15T01:51:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.810+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f53d8037070 con 0x7f53e4068e80 2024-09-15T01:51:18.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.810+0000 7f53e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:18.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.810+0000 7f53e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.877+0000 7f53eaaef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27d9cb97-3b36-4002-8665-cf5170791748", "id": 4}] 2024-09-15T01:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.877+0000 7f53eaaef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27d9cb97-3b36-4002-8665-cf5170791748", "id": 4}] 2024-09-15T01:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.877+0000 7f53eaaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:18.877+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "27d9cb97-3b36-4002-8665-cf5170791748", "id": 4} v 0) v1 -- 0x7f53e40a2320 con 0x7f53e4068e80 2024-09-15T01:51:19.427 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:19.425+0000 7f0b1f1c2ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T01:51:19.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.446+0000 7f53e0ff9640 1 -- 172.21.15.67:0/835064266 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27d9cb97-3b36-4002-8665-cf5170791748", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f53d803ccc0 con 0x7f53e4068e80 2024-09-15T01:51:19.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.446+0000 7f53e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27d9cb97-3b36-4002-8665-cf5170791748", "id": 4}] 2024-09-15T01:51:19.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.446+0000 7f53e0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:19.449 INFO:teuthology.orchestra.run.smithi067.stdout:4 2024-09-15T01:51:19.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 10 monclient: shutdown 2024-09-15T01:51:19.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:19.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 msgr2=0x7f53e41412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:19.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f53d80098b0 tx=0x7f53d8032000 comp rx=0 tx=0).stop 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 shutdown_connections 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53e4131910 0x7f53e4099be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53e409eb30 0x7f53e412fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 --2- 172.21.15.67:0/835064266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53e4068e80 0x7f53e41412f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.447+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 >> 172.21.15.67:0/835064266 conn(0x7f53e4071470 msgr2=0x7f53e4134d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.448+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 shutdown_connections 2024-09-15T01:51:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.448+0000 7f53eaaef640 1 -- 172.21.15.67:0/835064266 wait complete. 2024-09-15T01:51:19.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 142778f7-2757-4349-8124-1946d5d29c78 5 2024-09-15T01:51:19.481 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:19.479+0000 7f0b1f1c2ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 10 monclient: get_monmap_and_config 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 10 monclient: build_initial_monmap 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 10 monclient: monmap: 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:51:19.554201+0000 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:51:19.554201+0000 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 1 Processor -- start 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 1 -- start start 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.553+0000 7fae6fac9640 10 monclient: init 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae68131910 0x7fae68131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-c con 0x7fae68131910 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fae6809eb30 0x7fae6809ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-a con 0x7fae6809eb30 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68068e80 0x7fae680692f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-b con 0x7fae68068e80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae680a15e0 con 0x7fae6809eb30 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae68131ce0 con 0x7fae68131910 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809fef0 con 0x7fae68068e80 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6fac9640 10 monclient(hunting): authenticate will time out at 712.023340s 2024-09-15T01:51:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae68131910 0x7fae68131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae68131910 0x7fae68131fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54298/0 (socket says 172.21.15.67:54298) 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 -- 172.21.15.67:0/1350383603 learned_addr learned my addr 172.21.15.67:0/1350383603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): get_auth_request con 0x7fae68131910 auth_method 0 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6e066640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68068e80 0x7fae680692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 -- 172.21.15.67:0/1350383603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68068e80 msgr2=0x7fae680692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d064640 1 --2- 172.21.15.67:0/1350383603 >> v2:172.21.15.17:3300/0 conn(0x7fae6809eb30 0x7fae6809ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68068e80 0x7fae680692f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 -- 172.21.15.67:0/1350383603 >> v2:172.21.15.17:3300/0 conn(0x7fae6809eb30 msgr2=0x7fae6809ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 --2- 172.21.15.67:0/1350383603 >> v2:172.21.15.17:3300/0 conn(0x7fae6809eb30 0x7fae6809ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient: _finish_hunting 0 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 monclient: found mon.noname-c 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.554+0000 7fae6d865640 1 -- 172.21.15.67:0/1350383603 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680a0120 con 0x7fae68131910 2024-09-15T01:51:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6d865640 1 -- 172.21.15.67:0/1350383603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68131910 msgr2=0x7fae68131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6d865640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68131910 0x7fae68131fc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae680a5fa0 tx=0x7fae5c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706cbf4fa556b70b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 1 -- 172.21.15.67:0/1350383603 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c032440 con 0x7fae68131910 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: dump: 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: _finish_auth 0 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 1 -- 172.21.15.67:0/1350383603 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c0325a0 con 0x7fae68131910 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6fac9640 5 monclient: authenticate success, global_id 4132 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6fac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6fac9640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 1 -- 172.21.15.67:0/1350383603 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c032940 con 0x7fae68131910 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6fac9640 4 set_mon_vals no callback set 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.557+0000 7fae6c863640 10 monclient: dump: 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 10 monclient: shutdown 2024-09-15T01:51:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 -- 172.21.15.67:0/1350383603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68131910 msgr2=0x7fae68131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68131910 0x7fae68131fc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae680a5fa0 tx=0x7fae5c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 -- 172.21.15.67:0/1350383603 shutdown_connections 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68068e80 0x7fae680692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 --2- 172.21.15.67:0/1350383603 >> v2:172.21.15.17:3300/0 conn(0x7fae6809eb30 0x7fae6809ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 --2- 172.21.15.67:0/1350383603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68131910 0x7fae68131fc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 -- 172.21.15.67:0/1350383603 >> 172.21.15.67:0/1350383603 conn(0x7fae68071470 msgr2=0x7fae680a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 -- 172.21.15.67:0/1350383603 shutdown_connections 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.558+0000 7fae6fac9640 1 -- 172.21.15.67:0/1350383603 wait complete. 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient: build_initial_monmap 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient: monmap: 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 Processor -- start 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 -- start start 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient: init 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-b con 0x7fae68068e80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae6809eb30 0x7fae68140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-a con 0x7fae6809eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 0x7fae681414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): picked mon.noname-c con 0x7fae68131910 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6811cbe0 con 0x7fae6809eb30 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6811d100 con 0x7fae68068e80 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6811f160 con 0x7fae68131910 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.559+0000 7fae6fac9640 10 monclient(hunting): authenticate will time out at 712.028980s 2024-09-15T01:51:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54314/0 (socket says 172.21.15.67:54314) 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 -- 172.21.15.67:0/3050281578 learned_addr learned my addr 172.21.15.67:0/3050281578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): get_auth_request con 0x7fae68068e80 auth_method 0 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae6809eb30 0x7fae68140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): get_auth_request con 0x7fae6809eb30 auth_method 0 2024-09-15T01:51:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 0x7fae681414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 10 monclient(hunting): get_auth_request con 0x7fae68131910 auth_method 0 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 -- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 msgr2=0x7fae681414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 0x7fae681414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 -- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae6809eb30 msgr2=0x7fae68140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae6809eb30 0x7fae68140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient: _finish_hunting 0 2024-09-15T01:51:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 monclient: found mon.noname-b 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6d865640 1 -- 172.21.15.67:0/3050281578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c009450 con 0x7fae68068e80 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.560+0000 7fae6e066640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 0x7fae681414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6d865640 1 -- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 msgr2=0x7fae68130a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6d865640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009210 tx=0x7fae5c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c006b30 con 0x7fae68068e80 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: dump: 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.581 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:19.581 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:19.581 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:19.581 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c031c70 con 0x7fae68068e80 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c03c400 con 0x7fae68068e80 2024-09-15T01:51:19.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae59ffb640 10 monclient: dump: 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.583 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-09-15T01:51:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6fac9640 5 monclient: authenticate success, global_id 4135 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6fac9640 10 monclient: _renew_subs 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6811f440 con 0x7fae68068e80 2024-09-15T01:51:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.562+0000 7fae5bfff640 4 set_mon_vals no callback set 2024-09-15T01:51:19.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae6fac9640 10 monclient: _renew_subs 2024-09-15T01:51:19.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae6fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae68130fb0 con 0x7fae68068e80 2024-09-15T01:51:19.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fae5c03c830 con 0x7fae68068e80 2024-09-15T01:51:19.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fae5c03cae0 con 0x7fae68068e80 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.563+0000 7fae6fac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.564+0000 7fae6fac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.564+0000 7fae6fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.564+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae680724a0 con 0x7fae68068e80 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.566+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fae5c049060 con 0x7fae68068e80 2024-09-15T01:51:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.566+0000 7fae59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.566+0000 7fae59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.633+0000 7fae6fac9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5}] 2024-09-15T01:51:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.633+0000 7fae6fac9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5}] 2024-09-15T01:51:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.633+0000 7fae6fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.633+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5} v 0) v1 -- 0x7fae6806efc0 con 0x7fae68068e80 2024-09-15T01:51:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:19.637+0000 7fae6d865640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009210 tx=0x7fae5c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fae5c037070 mon_command_ack([{"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5}]=0 v8) v1 2024-09-15T01:51:19.867 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:19.865+0000 7f0b1f1c2ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T01:51:19.952 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:19.950+0000 7f0b1f1c2ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T01:51:20.044 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.042+0000 7f0b1f1c2ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T01:51:20.091 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.088+0000 7f0b1f1c2ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T01:51:20.189 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.187+0000 7f0b1f1c2ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T01:51:20.244 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.241+0000 7f0b1f1c2ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T01:51:20.289 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.286+0000 7f0b1f1c2ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T01:51:20.385 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.383+0000 7f0b1f1c2ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T01:51:20.436 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.434+0000 7f0b1f1c2ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T01:51:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.465+0000 7fae59ffb640 1 -- 172.21.15.67:0/3050281578 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fae5c037070 con 0x7fae68068e80 2024-09-15T01:51:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.465+0000 7fae59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "142778f7-2757-4349-8124-1946d5d29c78", "id": 5}] 2024-09-15T01:51:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.465+0000 7fae59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:20.468 INFO:teuthology.orchestra.run.smithi067.stdout:5 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 10 monclient: shutdown 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 msgr2=0x7fae68130a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009210 tx=0x7fae5c032000 comp rx=0 tx=0).stop 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 shutdown_connections 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae6809eb30 0x7fae68140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae68131910 0x7fae681414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 --2- 172.21.15.67:0/3050281578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae68068e80 0x7fae68130a70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 >> 172.21.15.67:0/3050281578 conn(0x7fae68071470 msgr2=0x7fae680a3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 shutdown_connections 2024-09-15T01:51:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T01:51:20.466+0000 7fae6fac9640 1 -- 172.21.15.67:0/3050281578 wait complete. 2024-09-15T01:51:20.479 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T01:51:20.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T01:51:20.482 INFO:tasks.ceph.osd.0:Started 2024-09-15T01:51:20.482 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T01:51:20.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T01:51:20.485 INFO:tasks.ceph.osd.1:Started 2024-09-15T01:51:20.485 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T01:51:20.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T01:51:20.487 INFO:tasks.ceph.osd.2:Started 2024-09-15T01:51:20.487 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T01:51:20.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T01:51:20.489 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.487+0000 7f0b1f1c2ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T01:51:20.491 INFO:tasks.ceph.osd.3:Started 2024-09-15T01:51:20.491 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T01:51:20.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T01:51:20.493 INFO:tasks.ceph.osd.4:Started 2024-09-15T01:51:20.493 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T01:51:20.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T01:51:20.496 INFO:tasks.ceph.osd.5:Started 2024-09-15T01:51:20.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:20.566 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.564+0000 7f0b1f1c2ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T01:51:20.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.607+0000 7f02e0d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:20.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.607+0000 7f02e0d9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.607+0000 7f02e0d9d640 10 monclient: monmap: 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:20.608379+0000 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:20.608379+0000 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:20.611 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:20.611 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:20.611 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:20.611 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:20.611 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.617 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.610+0000 7f0b1f1c2ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 Processor -- start 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 -- start start 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient: init 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T01:51:20.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f02dc131b20 0x7f02dc09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f02dc131b20 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 0x7f02dc131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 0x7f02dc132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc09f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc0a0040 con 0x7f02dc131b20 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc09bff0 con 0x7f02dc09f1e0 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0a14a0 con 0x7f02dc0a0280 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02e0d9d640 10 monclient(hunting): authenticate will time out at 740.604874s 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02da59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f02dc131b20 0x7f02dc09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.608+0000 7f02d1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 0x7f02dc131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f02dc131b20 0x7f02dc09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40190/0 (socket says 172.21.15.17:40190) 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 0x7f02dc131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44050/0 (socket says 172.21.15.17:44050) 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 -- 172.21.15.17:0/861832519 learned_addr learned my addr 172.21.15.17:0/861832519 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): get_auth_request con 0x7f02dc131b20 auth_method 0 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 1 --2- 172.21.15.17:0/861832519 >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 0x7f02dc132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request con 0x7f02dc0a0280 auth_method 0 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): get_auth_request con 0x7f02dc09f1e0 auth_method 0 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient(hunting): handle_auth_done global_id 4146 payload 274 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 -- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 msgr2=0x7f02dc131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 --2- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 0x7f02dc131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 -- 172.21.15.17:0/861832519 >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 msgr2=0x7f02dc132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:20.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 --2- 172.21.15.17:0/861832519 >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 0x7f02dc132430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 monclient: found mon.noname-a 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02da59c640 1 -- 172.21.15.17:0/861832519 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc068e80 con 0x7f02dc131b20 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.609+0000 7f02dad9d640 1 --2- 172.21.15.17:0/861832519 >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 0x7f02dc132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:20.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02da59c640 1 -- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc131b20 msgr2=0x7f02dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02da59c640 1 --2- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc131b20 0x7f02dc09eca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0a2f90 tx=0x7f02cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e9876e81d1772d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 1 -- 172.21.15.17:0/861832519 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032440 con 0x7f02dc131b20 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: dump: 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:20.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: _finish_auth 0 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 1 -- 172.21.15.17:0/861832519 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc0325a0 con 0x7f02dc131b20 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02e0d9d640 5 monclient: authenticate success, global_id 4146 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02e0d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:20.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02e0d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02e0d9d640 4 set_mon_vals no callback set 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 1 -- 172.21.15.17:0/861832519 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032980 con 0x7f02dc131b20 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.612+0000 7f02d9d9b640 10 monclient: dump: 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:20.631 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 10 monclient: shutdown 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 -- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc131b20 msgr2=0x7f02dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc131b20 0x7f02dc09eca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0a2f90 tx=0x7f02cc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 -- 172.21.15.17:0/861832519 shutdown_connections 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/861832519 >> v2:172.21.15.67:3300/0 conn(0x7f02dc09f1e0 0x7f02dc132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc0a0280 0x7f02dc131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/861832519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc131b20 0x7f02dc09eca0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 -- 172.21.15.17:0/861832519 >> 172.21.15.17:0/861832519 conn(0x7f02dc071270 msgr2=0x7f02dc0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 -- 172.21.15.17:0/861832519 shutdown_connections 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.613+0000 7f02e0d9d640 1 -- 172.21.15.17:0/861832519 wait complete. 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.614+0000 7f02e0d9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.614+0000 7f02e0d9d640 10 monclient: monmap: 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:20.633 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.614+0000 7f02e0d9d640 1 Processor -- start 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.614+0000 7f02e0d9d640 1 -- start start 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.614+0000 7f02e0d9d640 10 monclient: init 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 0x7f02dc130340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f02dc0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 0x7f02dc11ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0a0a20 con 0x7f02dc0a0280 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc121780 con 0x7f02dc131b20 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc09e7b0 con 0x7f02dc09f1e0 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02e0d9d640 10 monclient(hunting): authenticate will time out at 740.611417s 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 0x7f02dc130340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44062/0 (socket says 172.21.15.17:44062) 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 1 -- 172.21.15.17:0/851915779 learned_addr learned my addr 172.21.15.17:0/851915779 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): get_auth_request con 0x7f02dc09f1e0 auth_method 0 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request con 0x7f02dc0a0280 auth_method 0 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 0x7f02dc11ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): get_auth_request con 0x7f02dc131b20 auth_method 0 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 -- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 msgr2=0x7f02dc141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:20.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 -- 172.21.15.17:0/851915779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 msgr2=0x7f02dc11ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 0x7f02dc11ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 monclient: found mon.noname-a 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02d1d9b640 1 -- 172.21.15.17:0/851915779 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4009a20 con 0x7f02dc0a0280 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02da59c640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.615+0000 7f02dad9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 0x7f02dc11ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d1d9b640 1 -- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 msgr2=0x7f02dc130340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d1d9b640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 0x7f02dc130340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e960 tx=0x7f02c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4009540 con 0x7f02dc0a0280 2024-09-15T01:51:20.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: dump: 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c400cb50 con 0x7f02dc0a0280 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4013470 con 0x7f02dc0a0280 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 10 monclient: dump: 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 5 monclient: authenticate success, global_id 4149 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 10 monclient: _renew_subs 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d959a640 4 set_mon_vals no callback set 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc009450 con 0x7f02dc0a0280 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 10 monclient: _renew_subs 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc141590 con 0x7f02dc0a0280 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f02c4005970 con 0x7f02dc0a0280 2024-09-15T01:51:20.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.618+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f02c4006620 con 0x7f02dc0a0280 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.619+0000 7f02e0d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.619+0000 7f02e0d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.619+0000 7f02e0d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.619+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc0724a0 con 0x7f02dc0a0280 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.621+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f02c4021060 con 0x7f02dc0a0280 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.621+0000 7f02d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:20.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.621+0000 7f02d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:20.656 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:20.654+0000 7f0b1f1c2ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T01:51:20.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.678+0000 7f02e0d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:20.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.678+0000 7f02e0d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:20.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.678+0000 7f02e0d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:20.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.678+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02dc0a3ce0 con 0x7f02dc0a0280 2024-09-15T01:51:20.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.679+0000 7f02d2ffd640 1 -- 172.21.15.17:0/851915779 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f02c4017070 con 0x7f02dc0a0280 2024-09-15T01:51:20.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.679+0000 7f02d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:20.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.679+0000 7f02d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:20.682 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:20.682 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":8,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:19.635512+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 10 monclient: shutdown 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 msgr2=0x7f02dc130340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 0x7f02dc130340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e960 tx=0x7f02c40090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 shutdown_connections 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f02dc0a0280 0x7f02dc130340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02dc131b20 0x7f02dc11ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 --2- 172.21.15.17:0/851915779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f02dc09f1e0 0x7f02dc141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 >> 172.21.15.17:0/851915779 conn(0x7f02dc071270 msgr2=0x7f02dc09cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 shutdown_connections 2024-09-15T01:51:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:20.680+0000 7f02e0d9d640 1 -- 172.21.15.17:0/851915779 wait complete. 2024-09-15T01:51:20.692 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T01:51:20.692 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T01:51:21.090 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T01:51:21.087+0000 7f9532234440 -1 Falling back to public interface 2024-09-15T01:51:21.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T01:51:21.087+0000 7f933c809440 -1 Falling back to public interface 2024-09-15T01:51:21.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T01:51:21.087+0000 7f933a7fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-09-15T01:51:21.091 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-15T01:51:21.088+0000 7fd9aa20f440 -1 Falling back to public interface 2024-09-15T01:51:21.091 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-15T01:51:21.089+0000 7f651c521440 -1 Falling back to public interface 2024-09-15T01:51:21.095 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-09-15T01:51:21.092+0000 7ff30064f440 -1 Falling back to public interface 2024-09-15T01:51:21.096 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-15T01:51:21.094+0000 7f6064213440 -1 Falling back to public interface 2024-09-15T01:51:21.296 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:21.298 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:21.296+0000 7f0b1f1c2ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T01:51:21.397 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.395+0000 7fb5b275d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.395+0000 7fb5b275d640 10 monclient: build_initial_monmap 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.395+0000 7fb5b275d640 10 monclient: monmap: 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:51:21.396730+0000 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:51:21.396730+0000 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:21.398 INFO:teuthology.misc.health.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:21.399 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.396+0000 7fb5b275d640 1 Processor -- start 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.396+0000 7fb5b275d640 1 -- start start 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.396+0000 7fb5b275d640 10 monclient: init 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb5ac131c50 0x7fb5ac09ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:21.401 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac131c50 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac09f4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5ac132560 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0a1ac0 con 0x7fb5ac132560 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a0250 con 0x7fb5ac131c50 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac068bf0 con 0x7fb5ac09f4d0 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b275d640 10 monclient(hunting): authenticate will time out at 741.393313s 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb5ac131c50 0x7fb5ac09ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:21.402 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb5ac131c50 0x7fb5ac09ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40204/0 (socket says 172.21.15.17:40204) 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44074/0 (socket says 172.21.15.17:44074) 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5b0cfa640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb5ac132560 0x7fb5ac067780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47030/0 (socket says 172.21.15.17:47030) 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 1 -- 172.21.15.17:0/1853462571 learned_addr learned my addr 172.21.15.17:0/1853462571 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb5ac09f4d0 auth_method 0 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): get_auth_request con 0x7fb5ac131c50 auth_method 0 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:21.403 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient(hunting): handle_auth_done global_id 4170 payload 274 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 -- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 msgr2=0x7fb5ac132020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 -- 172.21.15.17:0/1853462571 >> v1:172.21.15.17:6789/0 conn(0x7fb5ac132560 legacy=0x7fb5ac067780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 monclient: found mon.noname-a 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:21.404 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5abfff640 1 -- 172.21.15.17:0/1853462571 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0a0690 con 0x7fb5ac131c50 2024-09-15T01:51:21.405 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.397+0000 7fb5ab7fe640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:21.409 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5abfff640 1 -- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac131c50 msgr2=0x7fb5ac09ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:21.409 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5abfff640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac131c50 0x7fb5ac09ef90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009850 tx=0x7fb59c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b732c6bdf0637607 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:21.409 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 1 -- 172.21.15.17:0/1853462571 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c02adf0 con 0x7fb5ac131c50 2024-09-15T01:51:21.409 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:21.409 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: dump: 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:21.410 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 1 -- 172.21.15.17:0/1853462571 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c032440 con 0x7fb5ac131c50 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5b275d640 5 monclient: authenticate success, global_id 4170 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5b275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5b275d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 1 -- 172.21.15.17:0/1853462571 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0327e0 con 0x7fb5ac131c50 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5b275d640 4 set_mon_vals no callback set 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.400+0000 7fb5aaffd640 10 monclient: dump: 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:21.411 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 10 monclient: shutdown 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 -- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac131c50 msgr2=0x7fb5ac09ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac131c50 0x7fb5ac09ef90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009850 tx=0x7fb59c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 -- 172.21.15.17:0/1853462571 shutdown_connections 2024-09-15T01:51:21.412 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac09f4d0 0x7fb5ac132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1853462571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac131c50 0x7fb5ac09ef90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 -- 172.21.15.17:0/1853462571 >> 172.21.15.17:0/1853462571 conn(0x7fb5ac0711d0 msgr2=0x7fb5ac0a5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 -- 172.21.15.17:0/1853462571 shutdown_connections 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.401+0000 7fb5b275d640 1 -- 172.21.15.17:0/1853462571 wait complete. 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.402+0000 7fb5b275d640 10 monclient: build_initial_monmap 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.402+0000 7fb5b275d640 10 monclient: monmap: 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:21.413 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.402+0000 7fb5b275d640 1 Processor -- start 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.402+0000 7fb5b275d640 1 -- start start 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.402+0000 7fb5b275d640 10 monclient: init 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac09f4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:21.414 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5ac131c50 0x7fb5ac130f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac131c50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 0x7fb5ac11b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ac132560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac11ca40 con 0x7fb5ac09f4d0 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac09c100 con 0x7fb5ac131c50 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a4f20 con 0x7fb5ac132560 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b275d640 10 monclient(hunting): authenticate will time out at 741.399255s 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:21.415 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 0x7fb5ac11b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40212/0 (socket says 172.21.15.17:40212) 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 1 -- 172.21.15.17:0/1343635433 learned_addr learned my addr 172.21.15.17:0/1343635433 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): get_auth_request con 0x7fb5ac09f4d0 auth_method 0 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): get_auth_request con 0x7fb5ac132560 auth_method 0 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5ac131c50 0x7fb5ac130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb5ac131c50 auth_method 0 2024-09-15T01:51:21.416 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:21.417 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 -- 172.21.15.17:0/1343635433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5ac131c50 msgr2=0x7fb5ac130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5ac131c50 0x7fb5ac130f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 -- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 msgr2=0x7fb5ac09ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient: _finish_hunting 0 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 monclient: found mon.noname-c 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5b0cfa640 1 -- 172.21.15.17:0/1343635433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0009450 con 0x7fb5ac132560 2024-09-15T01:51:21.418 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.403+0000 7fb5abfff640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.405+0000 7fb5b0cfa640 1 -- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 msgr2=0x7fb5ac11b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.405+0000 7fb5b0cfa640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 0x7fb5ac11b980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c790 tx=0x7fb5a4004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4005d60 con 0x7fb5ac132560 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: dump: 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.419 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 5 monclient: authenticate success, global_id 4163 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 10 monclient: _renew_subs 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a400a640 con 0x7fb5ac132560 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c009390 con 0x7fb5ac132560 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400aa00 con 0x7fb5ac132560 2024-09-15T01:51:21.420 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb593fff640 10 monclient: dump: 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 10 monclient: _renew_subs 2024-09-15T01:51:21.421 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5aa7fc640 4 set_mon_vals no callback set 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.406+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac11e060 con 0x7fb5ac132560 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 2) v1 ==== 44721+0+0 (secure 0 0 0) 0x7fb5a40070f0 con 0x7fb5ac132560 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb5b275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb5b275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb5b275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac064900 con 0x7fb5ac132560 2024-09-15T01:51:21.422 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.407+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fb5a404a570 con 0x7fb5ac132560 2024-09-15T01:51:21.423 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.409+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fb5a4010950 con 0x7fb5ac132560 2024-09-15T01:51:21.423 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.409+0000 7fb593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:21.423 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.409+0000 7fb593fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:21.470 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.467+0000 7fb5b275d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:21.470 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.467+0000 7fb5b275d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:21.470 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.467+0000 7fb5b275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:21.470 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.467+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5ac0008d0 con 0x7fb5ac132560 2024-09-15T01:51:21.472 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.470+0000 7fb593fff640 1 -- 172.21.15.17:0/1343635433 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7fb5a4010770 con 0x7fb5ac132560 2024-09-15T01:51:21.472 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.470+0000 7fb593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:21.473 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.470+0000 7fb593fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:21.473 INFO:teuthology.misc.health.smithi017.stdout: 2024-09-15T01:51:21.473 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":8,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:19.635512+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:21.474 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 10 monclient: shutdown 2024-09-15T01:51:21.474 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:21.474 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 msgr2=0x7fb5ac11b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 0x7fb5ac11b980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c790 tx=0x7fb5a4004e20 comp rx=0 tx=0).stop 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 shutdown_connections 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb5ac132560 0x7fb5ac11b980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5ac131c50 0x7fb5ac130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 --2- 172.21.15.17:0/1343635433 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5ac09f4d0 0x7fb5ac09ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 >> 172.21.15.17:0/1343635433 conn(0x7fb5ac0711d0 msgr2=0x7fb5ac09e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 shutdown_connections 2024-09-15T01:51:21.475 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:21.471+0000 7fb5b275d640 1 -- 172.21.15.17:0/1343635433 wait complete. 2024-09-15T01:51:21.482 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-15T01:51:22.316 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:22.314+0000 7f0b03720640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T01:51:22.318 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:51:22.316+0000 7f0b03720640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T01:51:24.553 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-15T01:51:24.551+0000 7fd9aa20f440 -1 osd.0 0 log_to_monitors true 2024-09-15T01:51:24.590 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T01:51:24.588+0000 7f933c809440 -1 osd.4 0 log_to_monitors true 2024-09-15T01:51:24.599 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T01:51:24.597+0000 7f9532234440 -1 osd.5 0 log_to_monitors true 2024-09-15T01:51:24.605 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-15T01:51:24.603+0000 7f6064213440 -1 osd.2 0 log_to_monitors true 2024-09-15T01:51:24.614 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-15T01:51:24.612+0000 7f651c521440 -1 osd.1 0 log_to_monitors true 2024-09-15T01:51:24.619 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-09-15T01:51:24.616+0000 7ff30064f440 -1 osd.3 0 log_to_monitors true 2024-09-15T01:51:26.486 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-15T01:51:26.484+0000 7fd9a49f6640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T01:51:26.486 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-09-15T01:51:26.485+0000 7ff2fb7f6640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T01:51:26.488 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-15T01:51:26.486+0000 7f605e9f6640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T01:51:26.488 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T01:51:26.487+0000 7f952d3f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T01:51:26.491 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-15T01:51:26.489+0000 7fd99f9ec640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:26.492 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T01:51:26.491+0000 7f9336ff6640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T01:51:26.492 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-09-15T01:51:26.491+0000 7ff2f6fdc640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:26.493 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T01:51:26.491+0000 7f9528288640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:26.493 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-15T01:51:26.491+0000 7f60599ec640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:26.497 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T01:51:26.496+0000 7f9331fec640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:26.500 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-15T01:51:26.498+0000 7f6516ff6640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T01:51:26.504 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-15T01:51:26.502+0000 7f6511fdb640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T01:51:28.088 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.184+0000 7fb6cab5d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.184+0000 7fb6cab5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.185+0000 7fb6cab5d640 10 monclient: monmap: 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:51:28.185825+0000 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:51:28.185825+0000 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:28.187 INFO:teuthology.misc.health.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:28.188 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:28.188 INFO:teuthology.misc.health.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:28.188 INFO:teuthology.misc.health.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:28.188 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.185+0000 7fb6cab5d640 1 Processor -- start 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.185+0000 7fb6cab5d640 1 -- start start 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.185+0000 7fb6cab5d640 10 monclient: init 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c4098f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.190 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c40a37c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6c4099c80 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb6c4099490 0x7fb6c409e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c4099490 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c409f610 con 0x7fb6c4099c80 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c4099910 con 0x7fb6c4099490 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4068310 con 0x7fb6c40a37c0 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6cab5d640 10 monclient(hunting): authenticate will time out at 748.182429s 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c4098f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb6c4099490 0x7fb6c409e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.191 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c4098f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33756/0 (socket says 172.21.15.17:33756) 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 1 -- 172.21.15.17:0/2737193746 learned_addr learned my addr 172.21.15.17:0/2737193746 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb6c4099490 0x7fb6c409e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34642/0 (socket says 172.21.15.17:34642) 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): get_auth_request con 0x7fb6c40a37c0 auth_method 0 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): get_auth_request con 0x7fb6c4099490 auth_method 0 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.192 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 -- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 msgr2=0x7fb6c4098f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 --2- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c4098f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 -- 172.21.15.17:0/2737193746 >> v1:172.21.15.17:6789/0 conn(0x7fb6c4099c80 legacy=0x7fb6c409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient: _finish_hunting 0 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 monclient: found mon.noname-a 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.193 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.186+0000 7fb6c90fa640 1 -- 172.21.15.17:0/2737193746 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40a3b90 con 0x7fb6c4099490 2024-09-15T01:51:28.198 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c90fa640 1 -- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099490 msgr2=0x7fb6c409e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:28.198 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c90fa640 1 --2- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099490 0x7fb6c409e100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009e60 tx=0x7fb6bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156af2056ac31db6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 1 -- 172.21.15.17:0/2737193746 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc007760 con 0x7fb6c4099490 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: dump: 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.199 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6cab5d640 5 monclient: authenticate success, global_id 4194 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 1 -- 172.21.15.17:0/2737193746 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc02d050 con 0x7fb6c4099490 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6cab5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6cab5d640 4 set_mon_vals no callback set 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 1 -- 172.21.15.17:0/2737193746 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc007af0 con 0x7fb6c4099490 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.189+0000 7fb6c37fe640 10 monclient: dump: 2024-09-15T01:51:28.200 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 10 monclient: shutdown 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 -- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099490 msgr2=0x7fb6c409e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.201 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099490 0x7fb6c409e100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009e60 tx=0x7fb6bc009780 comp rx=0 tx=0).stop 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 -- 172.21.15.17:0/2737193746 shutdown_connections 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099490 0x7fb6c409e100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/2737193746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c4098f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 -- 172.21.15.17:0/2737193746 >> 172.21.15.17:0/2737193746 conn(0x7fb6c4070e60 msgr2=0x7fb6c40a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 -- 172.21.15.17:0/2737193746 shutdown_connections 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.190+0000 7fb6cab5d640 1 -- 172.21.15.17:0/2737193746 wait complete. 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.191+0000 7fb6cab5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:28.202 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.191+0000 7fb6cab5d640 10 monclient: monmap: 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:epoch 0 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:created 0.000000 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.191+0000 7fb6cab5d640 1 Processor -- start 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.191+0000 7fb6cab5d640 1 -- start start 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.191+0000 7fb6cab5d640 10 monclient: init 2024-09-15T01:51:28.203 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4099490 0x7fb6c4134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c4099490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 0x7fb6c411e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c4099c80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c40a37c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4141d80 con 0x7fb6c4099c80 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4120800 con 0x7fb6c4099490 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.204 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4099160 con 0x7fb6c40a37c0 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6cab5d640 10 monclient(hunting): authenticate will time out at 748.188279s 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 0x7fb6c411e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33764/0 (socket says 172.21.15.17:33764) 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request con 0x7fb6c4099c80 auth_method 0 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 1 -- 172.21.15.17:0/1815373803 learned_addr learned my addr 172.21.15.17:0/1815373803 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): get_auth_request con 0x7fb6c40a37c0 auth_method 0 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.205 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4099490 0x7fb6c4134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): get_auth_request con 0x7fb6c4099490 auth_method 0 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.206 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 -- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 msgr2=0x7fb6c411ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 -- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4099490 msgr2=0x7fb6c4134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4099490 0x7fb6c4134c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 monclient: found mon.noname-a 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c3fff640 1 -- 172.21.15.17:0/1815373803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc009a20 con 0x7fb6c4099c80 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.192+0000 7fb6c90fa640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c3fff640 1 -- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 msgr2=0x7fb6c411e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:28.207 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c3fff640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 0x7fb6c411e7e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c4e0 tx=0x7fb6b8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80117a0 con 0x7fb6c4099c80 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: dump: 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.208 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b8011920 con 0x7fb6c4099c80 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 5 monclient: authenticate success, global_id 4197 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 10 monclient: _renew_subs 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c406ec90 con 0x7fb6c4099c80 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8011ce0 con 0x7fb6c4099c80 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c0ff9640 10 monclient: dump: 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:epoch 1 2024-09-15T01:51:28.209 INFO:teuthology.misc.health.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr: 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 10 monclient: _renew_subs 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6c2ffd640 4 set_mon_vals no callback set 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.195+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c4120280 con 0x7fb6c4099c80 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7fb6b8012630 con 0x7fb6c4099c80 2024-09-15T01:51:28.210 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6cab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6cab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6cab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4064900 con 0x7fb6c4099c80 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6c0ff9640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 0x7fb6b003ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7fb6b804b110 con 0x7fb6c4099c80 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6c88f9640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 0x7fb6b003ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.211 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.196+0000 7fb6c88f9640 10 monclient: get_auth_request con 0x7fb6b00386e0 auth_method 0 2024-09-15T01:51:28.212 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.199+0000 7fb6c88f9640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 0x7fb6b003ab90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009d20 tx=0x7fb6b40091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.212 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.201+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6b80128e0 con 0x7fb6c4099c80 2024-09-15T01:51:28.212 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.201+0000 7fb6c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.212 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.201+0000 7fb6c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:28.370 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6cab5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:28.370 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6cab5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:28.370 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6cab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:28.370 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6c4120e50 con 0x7fb6c4099c80 2024-09-15T01:51:28.371 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6c0ff9640 1 -- 172.21.15.17:0/1815373803 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8068 (secure 0 0 0) 0x7fb6b8010750 con 0x7fb6c4099c80 2024-09-15T01:51:28.371 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:28.371 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.368+0000 7fb6c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:28.371 INFO:teuthology.misc.health.smithi017.stdout: 2024-09-15T01:51:28.371 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:27.483335+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 msgr2=0x7fb6b003ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 0x7fb6b003ab90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009d20 tx=0x7fb6b40091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 10 monclient: shutdown 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 msgr2=0x7fb6c411e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 0x7fb6c411e7e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c4e0 tx=0x7fb6b8007e10 comp rx=0 tx=0).stop 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 shutdown_connections 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386e0 0x7fb6b003ab90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4099c80 0x7fb6c411e7e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.373 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a37c0 0x7fb6c411ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.374 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 --2- 172.21.15.17:0/1815373803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4099490 0x7fb6c4134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.374 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.370+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 >> 172.21.15.17:0/1815373803 conn(0x7fb6c4070e60 msgr2=0x7fb6c409c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:28.374 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.371+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 shutdown_connections 2024-09-15T01:51:28.374 INFO:teuthology.misc.health.smithi017.stderr:2024-09-15T01:51:28.371+0000 7fb6cab5d640 1 -- 172.21.15.17:0/1815373803 wait complete. 2024-09-15T01:51:28.384 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-15T01:51:28.384 INFO:tasks.ceph:Creating RBD pool 2024-09-15T01:51:28.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T01:51:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.478+0000 7f1bc519d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.478+0000 7f1bc519d640 10 monclient: build_initial_monmap 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.478+0000 7f1bc519d640 10 monclient: monmap: 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:28.479648+0000 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:28.479648+0000 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:28.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 Processor -- start 2024-09-15T01:51:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 -- start start 2024-09-15T01:51:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient: init 2024-09-15T01:51:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bc0131a10 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bc009eba0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 0x7f1bc013a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc0137d50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc009fd50 con 0x7f1bc0131a10 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc00a0190 con 0x7f1bc009eba0 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc009c760 con 0x7f1bc0137d50 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.479+0000 7f1bc519d640 10 monclient(hunting): authenticate will time out at 748.475995s 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 0x7f1bc013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbed9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1bc0131a10 0x7f1bc0131de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60768/0 (socket says 172.21.15.17:60768) 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbed9d640 1 -- 172.21.15.17:0/962642952 learned_addr learned my addr 172.21.15.17:0/962642952 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): get_auth_request con 0x7f1bc0137d50 auth_method 0 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 1 -- 172.21.15.17:0/962642952 >> v1:172.21.15.67:6789/0 conn(0x7f1bc009eba0 legacy=0x7f1bc009ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 1 -- 172.21.15.17:0/962642952 >> v1:172.21.15.17:6789/0 conn(0x7f1bc0131a10 legacy=0x7f1bc0131de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient: _finish_hunting 0 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 1 monclient: found mon.noname-b 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.480+0000 7f1bbf59e640 1 -- 172.21.15.17:0/962642952 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00a1300 con 0x7f1bc0137d50 2024-09-15T01:51:28.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbf59e640 1 -- 172.21.15.17:0/962642952 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 msgr2=0x7f1bc013a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:28.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbf59e640 1 --2- 172.21.15.17:0/962642952 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 0x7f1bc013a130 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009880 tx=0x7f1bb80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=641d36ede38f1816 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 1 -- 172.21.15.17:0/962642952 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8032440 con 0x7f1bc0137d50 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: dump: 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.491 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: _finish_auth 0 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 1 -- 172.21.15.17:0/962642952 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb80325a0 con 0x7f1bc0137d50 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 5 monclient: authenticate success, global_id 4190 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 4 set_mon_vals no callback set 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 1 -- 172.21.15.17:0/962642952 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8032940 con 0x7f1bc0137d50 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bbdd9b640 10 monclient: dump: 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 10 monclient: shutdown 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 1 -- 172.21.15.17:0/962642952 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 msgr2=0x7f1bc013a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.483+0000 7f1bc519d640 1 --2- 172.21.15.17:0/962642952 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 0x7f1bc013a130 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009880 tx=0x7f1bb80090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 1 -- 172.21.15.17:0/962642952 shutdown_connections 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 1 --2- 172.21.15.17:0/962642952 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0137d50 0x7f1bc013a130 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 1 -- 172.21.15.17:0/962642952 >> 172.21.15.17:0/962642952 conn(0x7f1bc0070fe0 msgr2=0x7f1bc00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 1 -- 172.21.15.17:0/962642952 shutdown_connections 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 1 -- 172.21.15.17:0/962642952 wait complete. 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 10 monclient: build_initial_monmap 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.484+0000 7f1bc519d640 10 monclient: monmap: 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:28.494 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 Processor -- start 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 -- start start 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient: init 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc009eba0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 0x7f1bc0132a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc0131a10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 0x7f1bc0132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc0137d50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00a2500 con 0x7f1bc009eba0 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc0133f50 con 0x7f1bc0137d50 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc01359a0 con 0x7f1bc0131a10 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bc519d640 10 monclient(hunting): authenticate will time out at 748.481508s 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 0x7f1bc0132a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34660/0 (socket says 172.21.15.17:34660) 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 1 -- 172.21.15.17:0/320259797 learned_addr learned my addr 172.21.15.17:0/320259797 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): get_auth_request con 0x7f1bc009eba0 auth_method 0 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): get_auth_request con 0x7f1bc0131a10 auth_method 0 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbf59e640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 0x7f1bc0132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbf59e640 10 monclient(hunting): get_auth_request con 0x7f1bc0137d50 auth_method 0 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbf59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 1 -- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 msgr2=0x7f1bc0132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 0x7f1bc0132fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 1 -- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 msgr2=0x7f1bc009b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.485+0000 7f1bbe59c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:28.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbe59c640 1 monclient: found mon.noname-c 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbe59c640 1 -- 172.21.15.17:0/320259797 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8009390 con 0x7f1bc0131a10 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbf59e640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 0x7f1bc0132fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.486+0000 7f1bbed9d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bbe59c640 1 -- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 msgr2=0x7f1bc0132a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bbe59c640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 0x7f1bc0132a90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c4e0 tx=0x7f1ba8007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80117a0 con 0x7f1bc0131a10 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: dump: 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.499 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba8011920 con 0x7f1bc0131a10 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 5 monclient: authenticate success, global_id 4193 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:28.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8011ce0 con 0x7f1bc0131a10 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bb6ffd640 10 monclient: dump: 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bbd59a640 4 set_mon_vals no callback set 2024-09-15T01:51:28.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 10 monclient: _renew_subs 2024-09-15T01:51:28.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc0135c80 con 0x7f1bc0131a10 2024-09-15T01:51:28.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 10 monclient: _renew_subs 2024-09-15T01:51:28.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.488+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc0134c50 con 0x7f1bc0131a10 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bc519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bc519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7f1ba8012630 con 0x7f1bc0131a10 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bc519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc00724a0 con 0x7f1bc0131a10 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bb6ffd640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 0x7f1ba003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.489+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7f1ba804b180 con 0x7f1bc0131a10 2024-09-15T01:51:28.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.490+0000 7f1bbed9d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 0x7f1ba003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.490+0000 7f1bbed9d640 10 monclient: get_auth_request con 0x7f1ba0038700 auth_method 0 2024-09-15T01:51:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.492+0000 7f1bbed9d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 0x7f1ba003abb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000d7e0 tx=0x7f1bb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.494+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1ba80128e0 con 0x7f1bc0131a10 2024-09-15T01:51:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.494+0000 7f1bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.494+0000 7f1bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:28.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.657+0000 7f1bc519d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T01:51:28.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.657+0000 7f1bc519d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T01:51:28.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.657+0000 7f1bc519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:28.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:28.657+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f1bc01321c0 con 0x7f1bc0131a10 2024-09-15T01:51:29.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.511+0000 7f1bb6ffd640 1 -- 172.21.15.17:0/320259797 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f1ba801beb0 con 0x7f1bc0131a10 2024-09-15T01:51:29.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.511+0000 7f1bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T01:51:29.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.511+0000 7f1bb6ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T01:51:29.514 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 msgr2=0x7f1ba003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 0x7f1ba003abb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000d7e0 tx=0x7f1bb0009040 comp rx=0 tx=0).stop 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 10 monclient: shutdown 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 msgr2=0x7f1bc0132a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 0x7f1bc0132a90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c4e0 tx=0x7f1ba8007e10 comp rx=0 tx=0).stop 2024-09-15T01:51:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 shutdown_connections 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba0038700 0x7f1ba003abb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0137d50 0x7f1bc0132fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc0131a10 0x7f1bc0132a90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 --2- 172.21.15.17:0/320259797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc009eba0 0x7f1bc009b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 >> 172.21.15.17:0/320259797 conn(0x7f1bc0070fe0 msgr2=0x7f1bc009cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 shutdown_connections 2024-09-15T01:51:29.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.514+0000 7f1bc519d640 1 -- 172.21.15.17:0/320259797 wait complete. 2024-09-15T01:51:29.528 DEBUG:teuthology.orchestra.run.smithi017:> rbd --cluster ceph pool init rbd 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.593+0000 7f2eafa3ab80 10 monclient: get_monmap_and_config 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.593+0000 7f2eafa3ab80 10 monclient: build_initial_monmap 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.593+0000 7f2eafa3ab80 10 monclient: monmap: 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:29.594044+0000 2024-09-15T01:51:29.595 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:29.594044+0000 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:29.596 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.593+0000 7f2eafa3ab80 1 Processor -- start 2024-09-15T01:51:29.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 -- start start 2024-09-15T01:51:29.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient: init 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient: _reopen_session rank -1 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-c-legacy con 0x55db3c565580 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 --2- >> v2:172.21.15.67:3300/0 conn(0x55db3c565ee0 0x55db3c5673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-c con 0x55db3c565ee0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-b con 0x55db3c601ee0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55db3c4890e0 con 0x55db3c565580 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x55db3c4c2260 con 0x55db3c565ee0 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db3c4bdf40 con 0x55db3c601ee0 2024-09-15T01:51:29.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): _renew_subs 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eafa3ab80 10 monclient(hunting): authenticate will time out at 749.590602s 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33820/0 (socket says 172.21.15.17:33820) 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 1 -- 172.21.15.17:0/195806123 learned_addr learned my addr 172.21.15.17:0/195806123 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): get_auth_request con 0x55db3c601ee0 auth_method 0 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.594+0000 7f2eac9a1640 1 --2- 172.21.15.17:0/195806123 >> v2:172.21.15.67:3300/0 conn(0x55db3c565ee0 0x55db3c5673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2eac9a1640 10 monclient(hunting): get_auth_request con 0x55db3c565ee0 auth_method 0 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2eac9a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2eac9a1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2eac9a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 1 -- 172.21.15.17:0/195806123 >> v2:172.21.15.67:3300/0 conn(0x55db3c565ee0 msgr2=0x55db3c5673e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 1 --2- 172.21.15.17:0/195806123 >> v2:172.21.15.67:3300/0 conn(0x55db3c565ee0 0x55db3c5673e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 1 -- 172.21.15.17:0/195806123 >> v1:172.21.15.67:6789/0 conn(0x55db3c565580 legacy=0x55db3c566cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 10 monclient: _finish_hunting 0 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 1 monclient: found mon.noname-b 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:29.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.595+0000 7f2ead9a3640 1 -- 172.21.15.17:0/195806123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3c462270 con 0x55db3c601ee0 2024-09-15T01:51:29.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2ead9a3640 1 -- 172.21.15.17:0/195806123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 msgr2=0x55db3c6042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:29.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2ead9a3640 1 --2- 172.21.15.17:0/195806123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000a640 tx=0x7f2ea0045a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef159d17aba636ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 1 -- 172.21.15.17:0/195806123 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00514b0 con 0x55db3c601ee0 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: dump: 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: _finish_auth 0 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 1 -- 172.21.15.17:0/195806123 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0051610 con 0x55db3c601ee0 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 5 monclient: authenticate success, global_id 4196 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 10 monclient: get_monmap_and_config success 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 1 -- 172.21.15.17:0/195806123 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00517a0 con 0x55db3c601ee0 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 4 set_mon_vals no callback set 2024-09-15T01:51:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2e9ffff640 10 monclient: dump: 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 10 monclient: shutdown 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/195806123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 msgr2=0x55db3c6042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.599+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/195806123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000a640 tx=0x7f2ea0045a50 comp rx=0 tx=0).stop 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/195806123 shutdown_connections 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/195806123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c6042c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/195806123 >> v2:172.21.15.67:3300/0 conn(0x55db3c565ee0 0x55db3c5673e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/195806123 >> 172.21.15.17:0/195806123 conn(0x55db3c568dd0 msgr2=0x55db3c5691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/195806123 shutdown_connections 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/195806123 wait complete. 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 10 monclient: build_initial_monmap 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 10 monclient: monmap: 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:29.609 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 Processor -- start 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 1 -- start start 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.600+0000 7f2eafa3ab80 10 monclient: init 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient: _reopen_session rank -1 2024-09-15T01:51:29.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db3c565580 0x55db3c5e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-b con 0x55db3c565580 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x55db3c565ee0 0x55db3c5e8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-a con 0x55db3c565ee0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): picked mon.noname-c con 0x55db3c601ee0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3c480ba0 con 0x55db3c565ee0 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3c480840 con 0x55db3c565580 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db3c4437a0 con 0x55db3c601ee0 2024-09-15T01:51:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): _renew_subs 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eafa3ab80 10 monclient(hunting): authenticate will time out at 749.597465s 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x55db3c565ee0 0x55db3c5e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33834/0 (socket says 172.21.15.17:33834) 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 learned_addr learned my addr 172.21.15.17:0/4083590452 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): get_auth_request con 0x55db3c601ee0 auth_method 0 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): get_auth_request con 0x55db3c565ee0 auth_method 0 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:29.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db3c565580 0x55db3c5e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): get_auth_request con 0x55db3c565580 auth_method 0 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2eac9a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:29.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead1a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db3c565580 msgr2=0x55db3c5e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db3c565580 0x55db3c5e7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.601+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x55db3c565ee0 msgr2=0x55db3c5e8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x55db3c565ee0 0x55db3c5e8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 10 monclient: _finish_hunting 0 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 1 monclient: found mon.noname-c 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea000a180 con 0x55db3c601ee0 2024-09-15T01:51:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.602+0000 7f2eac9a1640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x55db3c565ee0 0x55db3c5e8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:29.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 msgr2=0x55db3c5e8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:29.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2ead9a3640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000a640 tx=0x7f2ea0057000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:29.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00528a0 con 0x55db3c601ee0 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: dump: 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 5 monclient: authenticate success, global_id 4199 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0052a00 con 0x55db3c601ee0 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 10 monclient: _renew_subs 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55db3c46f0e0 con 0x55db3c601ee0 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0052dc0 con 0x55db3c601ee0 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9d7fa640 10 monclient: dump: 2024-09-15T01:51:29.618 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2e9f7fe640 4 set_mon_vals no callback set 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 10 monclient: _renew_subs 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.604+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db3c462600 con 0x55db3c601ee0 2024-09-15T01:51:29.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7f2ea005d620 con 0x55db3c601ee0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2e9d7fa640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e8c036fe0 0x7f2e8c039490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4320+0+0 (secure 0 0 0) 0x7f2ea004e040 con 0x55db3c601ee0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2eafa3ab80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2eafa3ab80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2eafa3ab80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55db3c462060 con 0x55db3c601ee0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2ead1a2640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e8c036fe0 0x7f2e8c039490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.605+0000 7f2ead1a2640 10 monclient: get_auth_request con 0x7f2e8c036fe0 auth_method 0 2024-09-15T01:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:29.608+0000 7f2ead1a2640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e8c036fe0 0x7f2e8c039490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4003f10 tx=0x7f2ea40027f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:30.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.511+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2ea0050ae0 con 0x55db3c601ee0 2024-09-15T01:51:30.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.511+0000 7f2e9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T01:51:30.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.511+0000 7f2e9d7fa640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T01:51:30.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.511+0000 7f2eafa3ab80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:30.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.511+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55db3c480630 con 0x55db3c601ee0 2024-09-15T01:51:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.516+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ea0050ae0 con 0x55db3c601ee0 2024-09-15T01:51:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.516+0000 7f2e9d7fa640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-15T01:51:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.516+0000 7f2e9f7fe640 10 monclient: _renew_subs 2024-09-15T01:51:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.516+0000 7f2e9f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.516+0000 7f2e9f7fe640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=14}) v3 -- 0x7f2e84002990 con 0x55db3c601ee0 2024-09-15T01:51:30.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.517+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1020+0+0 (secure 0 0 0) 0x7f2ea005d8d0 con 0x55db3c601ee0 2024-09-15T01:51:30.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.517+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x55db3c60b020 0x55db3c6139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:30.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.517+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] -- osd_op(unknown.0.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x55db3c613ef0 con 0x55db3c60b020 2024-09-15T01:51:30.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.518+0000 7f2eac9a1640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x55db3c60b020 0x55db3c6139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:30.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.518+0000 7f2eac9a1640 10 monclient: get_auth_request con 0x55db3c60b020 auth_method 0 2024-09-15T01:51:30.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.524+0000 7f2eac9a1640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x55db3c60b020 0x55db3c6139b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:30.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.525+0000 7f2eac9a1640 1 -- 172.21.15.17:0/4083590452 <== osd.1 v2:172.21.15.17:6808/31931 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f2e98012040 con 0x55db3c60b020 2024-09-15T01:51:30.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.527+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x55db3c6617e0 0x55db3c663c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:30.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.527+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- osd_op(unknown.0.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55db3c6641c0 con 0x55db3c6617e0 2024-09-15T01:51:30.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.528+0000 7f2ead1a2640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x55db3c6617e0 0x55db3c663c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:30.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.528+0000 7f2ead1a2640 10 monclient: get_auth_request con 0x55db3c6617e0 auth_method 0 2024-09-15T01:51:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.532+0000 7f2ead1a2640 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x55db3c6617e0 0x55db3c663c60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.533+0000 7f2ead1a2640 1 -- 172.21.15.17:0/4083590452 <== osd.4 v2:172.21.15.67:6802/31881 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2ea4002ab0 con 0x55db3c6617e0 2024-09-15T01:51:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.533+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- osd_op(unknown.0.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55db3c6602c0 con 0x55db3c6617e0 2024-09-15T01:51:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.533+0000 7f2ead1a2640 1 -- 172.21.15.17:0/4083590452 <== osd.4 v2:172.21.15.67:6802/31881 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2ea4002ab0 con 0x55db3c6617e0 2024-09-15T01:51:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.533+0000 7f2e9effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:30.533+0000 7f2e9effd640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- pool_op(create unmanaged snap pool 1 tid 4 name v0) v4 -- 0x7f2e88001ee0 con 0x55db3c601ee0 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.559+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (secure 0 0 0) 0x7f2ea00506c0 con 0x55db3c601ee0 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.559+0000 7f2e9d7fa640 10 monclient: _renew_subs 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.559+0000 7f2e9d7fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.559+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=15}) v3 -- 0x7f2e8c03d4a0 con 0x55db3c601ee0 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.560+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 10 ==== osd_map(15..15 src has 1..15) v4 ==== 988+0+0 (secure 0 0 0) 0x7f2ea0050d10 con 0x55db3c601ee0 2024-09-15T01:51:31.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.560+0000 7f2e9f7fe640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- osd_op(unknown.0.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f2e84005450 con 0x55db3c6617e0 2024-09-15T01:51:31.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.568+0000 7f2ead1a2640 1 -- 172.21.15.17:0/4083590452 <== osd.4 v2:172.21.15.67:6802/31881 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f2ea4002ab0 con 0x55db3c6617e0 2024-09-15T01:51:31.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.568+0000 7f2e9effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:31.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:31.568+0000 7f2e9effd640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- pool_op(delete unmanaged snap pool 1 tid 6 name v15) v4 -- 0x7f2e880033b0 con 0x55db3c601ee0 2024-09-15T01:51:32.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.073+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 11 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2ea008e270 con 0x55db3c601ee0 2024-09-15T01:51:32.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (secure 0 0 0) 0x7f2ea006f8f0 con 0x55db3c601ee0 2024-09-15T01:51:32.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9d7fa640 10 monclient: _renew_subs 2024-09-15T01:51:32.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9d7fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:32.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f2e8c03d620 con 0x55db3c601ee0 2024-09-15T01:51:32.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9d7fa640 1 -- 172.21.15.17:0/4083590452 <== mon.2 v2:172.21.15.17:3301/0 13 ==== osd_map(16..16 src has 1..16) v4 ==== 646+0+0 (secure 0 0 0) 0x7f2ea00067d0 con 0x55db3c601ee0 2024-09-15T01:51:32.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.567+0000 7f2e9f7fe640 1 -- 172.21.15.17:0/4083590452 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- osd_op(unknown.0.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f2e84005880 con 0x55db3c6617e0 2024-09-15T01:51:32.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2ead1a2640 1 -- 172.21.15.17:0/4083590452 <== osd.4 v2:172.21.15.67:6802/31881 4 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2ea4002ab0 con 0x55db3c6617e0 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x55db3c60b020 msgr2=0x55db3c6139b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x55db3c60b020 0x55db3c6139b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x55db3c6617e0 msgr2=0x55db3c663c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x55db3c6617e0 0x55db3c663c60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e8c036fe0 msgr2=0x7f2e8c039490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e8c036fe0 0x7f2e8c039490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4003f10 tx=0x7f2ea40027f0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 10 monclient: shutdown 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 msgr2=0x55db3c5e8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000a640 tx=0x7f2ea0057000 comp rx=0 tx=0).stop 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2ead9a3640 1 -- 172.21.15.17:0/4083590452 reap_dead start 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 shutdown_connections 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 --2- 172.21.15.17:0/4083590452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x55db3c601ee0 0x55db3c5e8890 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 >> 172.21.15.17:0/4083590452 conn(0x55db3c568dd0 msgr2=0x55db3c564320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 shutdown_connections 2024-09-15T01:51:32.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.569+0000 7f2eafa3ab80 1 -- 172.21.15.17:0/4083590452 wait complete. 2024-09-15T01:51:32.576 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T01:51:32.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T01:51:32.577 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T01:51:32.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T01:51:32.577 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T01:51:32.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.681+0000 7f58a860f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.681+0000 7f58a860f640 10 monclient: build_initial_monmap 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.681+0000 7f58a860f640 10 monclient: monmap: 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:32.682593+0000 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:32.682593+0000 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.684 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:32.685 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:32.685 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:32.685 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:32.685 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:32.685 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.682+0000 7f58a860f640 1 Processor -- start 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.682+0000 7f58a860f640 1 -- start start 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.682+0000 7f58a860f640 10 monclient: init 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:51:32.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 0x7f58a0132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-c con 0x7f58a0067090 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58a009ce00 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f58a0067b70 0x7f58a0139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-a con 0x7f58a0067b70 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a009e1e0 con 0x7f58a009ce00 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a009e4c0 con 0x7f58a0067b70 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a009cbf0 con 0x7f58a0067090 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a860f640 10 monclient(hunting): authenticate will time out at 752.679108s 2024-09-15T01:51:32.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f58a0067b70 0x7f58a0139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f58a0067b70 0x7f58a0139010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34690/0 (socket says 172.21.15.17:34690) 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 -- 172.21.15.17:0/376708534 learned_addr learned my addr 172.21.15.17:0/376708534 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): get_auth_request con 0x7f58a0067b70 auth_method 0 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 1 --2- 172.21.15.17:0/376708534 >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 0x7f58a0132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 10 monclient(hunting): get_auth_request con 0x7f58a0067090 auth_method 0 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:32.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 -- 172.21.15.17:0/376708534 >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 msgr2=0x7f58a0132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 --2- 172.21.15.17:0/376708534 >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 0x7f58a0132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 -- 172.21.15.17:0/376708534 >> v1:172.21.15.67:6789/0 conn(0x7f58a009ce00 legacy=0x7f58a0067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient: _finish_hunting 0 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 monclient: found mon.noname-a 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a6bac640 1 -- 172.21.15.17:0/376708534 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a006efb0 con 0x7f58a0067b70 2024-09-15T01:51:32.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.683+0000 7f58a63ab640 1 --2- 172.21.15.17:0/376708534 >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 0x7f58a0132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:32.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.686+0000 7f58a6bac640 1 -- 172.21.15.17:0/376708534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067b70 msgr2=0x7f58a0139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:32.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.686+0000 7f58a6bac640 1 --2- 172.21.15.17:0/376708534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067b70 0x7f58a0139010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58a004b6a0 tx=0x7f5898009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40152af4db1ad081 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:32.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 1 -- 172.21.15.17:0/376708534 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898007610 con 0x7f58a0067b70 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: dump: 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:32.698 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: _finish_auth 0 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 1 -- 172.21.15.17:0/376708534 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589802d050 con 0x7f58a0067b70 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a860f640 5 monclient: authenticate success, global_id 4215 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a860f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 1 -- 172.21.15.17:0/376708534 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58980079a0 con 0x7f58a0067b70 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a860f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a860f640 4 set_mon_vals no callback set 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.688+0000 7f58a53a9640 10 monclient: dump: 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:32.699 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 10 monclient: shutdown 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 -- 172.21.15.17:0/376708534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067b70 msgr2=0x7f58a0139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 --2- 172.21.15.17:0/376708534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067b70 0x7f58a0139010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58a004b6a0 tx=0x7f5898009840 comp rx=0 tx=0).stop 2024-09-15T01:51:32.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 -- 172.21.15.17:0/376708534 shutdown_connections 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 --2- 172.21.15.17:0/376708534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067b70 0x7f58a0139010 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 --2- 172.21.15.17:0/376708534 >> v2:172.21.15.67:3300/0 conn(0x7f58a0067090 0x7f58a0132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 -- 172.21.15.17:0/376708534 >> 172.21.15.17:0/376708534 conn(0x7f58a00a1ee0 msgr2=0x7f58a00a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 -- 172.21.15.17:0/376708534 shutdown_connections 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 1 -- 172.21.15.17:0/376708534 wait complete. 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 10 monclient: build_initial_monmap 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.689+0000 7f58a860f640 10 monclient: monmap: 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:32.701 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 Processor -- start 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 -- start start 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient: init 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-a con 0x7f58a0067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 0x7f58a0137520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-b con 0x7f58a0067b70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58a009ce00 0x7f58a0137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): picked mon.noname-c con 0x7f58a009ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0119a20 con 0x7f58a0067090 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a011a050 con 0x7f58a0067b70 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01188f0 con 0x7f58a009ce00 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a860f640 10 monclient(hunting): authenticate will time out at 752.686864s 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34698/0 (socket says 172.21.15.17:34698) 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 1 -- 172.21.15.17:0/4165844828 learned_addr learned my addr 172.21.15.17:0/4165844828 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:32.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 10 monclient(hunting): get_auth_request con 0x7f58a0067090 auth_method 0 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.690+0000 7f58a63ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58a009ce00 0x7f58a0137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): get_auth_request con 0x7f58a009ce00 auth_method 0 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:32.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a6bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 -- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58a009ce00 msgr2=0x7f58a0137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58a009ce00 0x7f58a0137a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a5baa640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 0x7f58a0137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 -- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 msgr2=0x7f58a0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 0x7f58a0137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient: _finish_hunting 0 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 monclient: found mon.noname-a 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a63ab640 1 -- 172.21.15.17:0/4165844828 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898009ae0 con 0x7f58a0067090 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a5baa640 10 monclient: get_auth_request con 0x7f58a0067b70 auth_method 0 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.691+0000 7f58a5baa640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 0x7f58a0137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a63ab640 1 -- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 msgr2=0x7f58a0130ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a63ab640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d840 tx=0x7f589c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00b8d0 con 0x7f58a0067090 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: dump: 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.706 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00ba30 con 0x7f58a0067090 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 5 monclient: authenticate success, global_id 4218 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 10 monclient: _renew_subs 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a011a6e0 con 0x7f58a0067090 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:32.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00bdf0 con 0x7f58a0067090 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58967fc640 10 monclient: dump: 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:32.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a4ba8640 4 set_mon_vals no callback set 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 10 monclient: _renew_subs 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.693+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a012ffa0 con 0x7f58a0067090 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.694+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f589c014640 con 0x7f58a0067090 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.694+0000 7f58a860f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.694+0000 7f58a860f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.694+0000 7f58a860f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.694+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a00724a0 con 0x7f58a0067090 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.695+0000 7f58967fc640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 0x7f588403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.695+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f589c04bed0 con 0x7f58a0067090 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.695+0000 7f58a5baa640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 0x7f588403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.695+0000 7f58a5baa640 10 monclient: get_auth_request con 0x7f5884038790 auth_method 0 2024-09-15T01:51:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.698+0000 7f58a5baa640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 0x7f588403ac40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58900027e0 tx=0x7f5890009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.699+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f589c01dde0 con 0x7f58a0067090 2024-09-15T01:51:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.700+0000 7f58967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.700+0000 7f58967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:32.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.906+0000 7f58a860f640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T01:51:32.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.906+0000 7f58a860f640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T01:51:32.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.906+0000 7f58a860f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:32.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.906+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f58a009d5c0 con 0x7f58a0067090 2024-09-15T01:51:32.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.908+0000 7f58967fc640 1 -- 172.21.15.17:0/4165844828 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+87655 (secure 0 0 0) 0x7f589c01d7e0 con 0x7f58a0067090 2024-09-15T01:51:32.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.908+0000 7f58967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T01:51:32.911 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:32.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.908+0000 7f58967fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 msgr2=0x7f588403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 0x7f588403ac40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58900027e0 tx=0x7f5890009040 comp rx=0 tx=0).stop 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 10 monclient: shutdown 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 msgr2=0x7f58a0130ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d840 tx=0x7f589c008040 comp rx=0 tx=0).stop 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 shutdown_connections 2024-09-15T01:51:32.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.910+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5884038790 0x7f588403ac40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58a009ce00 0x7f58a0137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a0067b70 0x7f58a0137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 --2- 172.21.15.17:0/4165844828 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58a0067090 0x7f58a0130ab0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 >> 172.21.15.17:0/4165844828 conn(0x7f58a00a1ee0 msgr2=0x7f58a0097ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 shutdown_connections 2024-09-15T01:51:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:32.911+0000 7f58a860f640 1 -- 172.21.15.17:0/4165844828 wait complete. 2024-09-15T01:51:32.922 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":31455},{"type":"v1","addr":"172.21.15.67:6825","nonce":31455}]},"active_addr":"172.21.15.67:6825/31455","active_change":"2024-09-15T01:51:21.299972+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":97638287}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1912331251}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1051794396}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":144260737}]}]} 2024-09-15T01:51:32.924 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T01:51:32.925 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T01:51:32.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:33.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 10 monclient: monmap: 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.019901+0000 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.019901+0000 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 1 Processor -- start 2024-09-15T01:51:33.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 1 -- start start 2024-09-15T01:51:33.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.019+0000 7f3a2d309640 10 monclient: init 2024-09-15T01:51:33.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a28131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a280a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3a28137e60 0x7f3a2813a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-a con 0x7f3a28137e60 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a280a0040 con 0x7f3a28131b20 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2809bff0 con 0x7f3a280a0280 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a280a14a0 con 0x7f3a28137e60 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a2d309640 10 monclient(hunting): authenticate will time out at 753.016213s 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3a28137e60 0x7f3a2813a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a26ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3a28131b20 0x7f3a2809eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33614/0 (socket says 172.21.15.17:33614) 2024-09-15T01:51:33.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a26ffd640 1 -- 172.21.15.17:0/4003286471 learned_addr learned my addr 172.21.15.17:0/4003286471 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): get_auth_request con 0x7f3a28137e60 auth_method 0 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 1 -- 172.21.15.17:0/4003286471 >> v1:172.21.15.67:6789/0 conn(0x7f3a280a0280 legacy=0x7f3a28131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 1 -- 172.21.15.17:0/4003286471 >> v1:172.21.15.17:6789/0 conn(0x7f3a28131b20 legacy=0x7f3a2809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.020+0000 7f3a277fe640 1 -- 172.21.15.17:0/4003286471 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2809f3b0 con 0x7f3a28137e60 2024-09-15T01:51:33.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a277fe640 1 -- 172.21.15.17:0/4003286471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28137e60 msgr2=0x7f3a2813a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a277fe640 1 --2- 172.21.15.17:0/4003286471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28137e60 0x7f3a2813a240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009850 tx=0x7f3a200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8ccde422117d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 1 -- 172.21.15.17:0/4003286471 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20032440 con 0x7f3a28137e60 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: dump: 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.033 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a2d309640 5 monclient: authenticate success, global_id 4221 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 1 -- 172.21.15.17:0/4003286471 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a200325a0 con 0x7f3a28137e60 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a2d309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a2d309640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a2d309640 4 set_mon_vals no callback set 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 1 -- 172.21.15.17:0/4003286471 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20032940 con 0x7f3a28137e60 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.023+0000 7f3a25ffb640 10 monclient: dump: 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 10 monclient: shutdown 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 -- 172.21.15.17:0/4003286471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28137e60 msgr2=0x7f3a2813a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 --2- 172.21.15.17:0/4003286471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28137e60 0x7f3a2813a240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009850 tx=0x7f3a200090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 -- 172.21.15.17:0/4003286471 shutdown_connections 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 --2- 172.21.15.17:0/4003286471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28137e60 0x7f3a2813a240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 -- 172.21.15.17:0/4003286471 >> 172.21.15.17:0/4003286471 conn(0x7f3a28071270 msgr2=0x7f3a280a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 -- 172.21.15.17:0/4003286471 shutdown_connections 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 1 -- 172.21.15.17:0/4003286471 wait complete. 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.024+0000 7f3a2d309640 10 monclient: monmap: 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.036 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 Processor -- start 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 -- start start 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient: init 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a280a0280 0x7f3a2809b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-c con 0x7f3a280a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-a con 0x7f3a28131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28137e60 0x7f3a281341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): picked mon.noname-b con 0x7f3a28137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2809eed0 con 0x7f3a28131b20 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28135f30 con 0x7f3a28137e60 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2809b7c0 con 0x7f3a280a0280 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a2d309640 10 monclient(hunting): authenticate will time out at 753.021757s 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38850/0 (socket says 172.21.15.17:38850) 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 1 -- 172.21.15.17:0/532064625 learned_addr learned my addr 172.21.15.17:0/532064625 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): get_auth_request con 0x7f3a28131b20 auth_method 0 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a280a0280 0x7f3a2809b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): get_auth_request con 0x7f3a280a0280 auth_method 0 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.025+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28137e60 0x7f3a281341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): get_auth_request con 0x7f3a28137e60 auth_method 0 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 -- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a280a0280 msgr2=0x7f3a2809b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a280a0280 0x7f3a2809b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 -- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28137e60 msgr2=0x7f3a281341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28137e60 0x7f3a281341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.026+0000 7f3a267fc640 1 -- 172.21.15.17:0/532064625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a20009390 con 0x7f3a28131b20 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a267fc640 1 -- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 msgr2=0x7f3a28133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a267fc640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000c610 tx=0x7f3a10007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012840 con 0x7f3a28131b20 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: dump: 2024-09-15T01:51:33.041 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a100129e0 con 0x7f3a28131b20 2024-09-15T01:51:33.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 5 monclient: authenticate success, global_id 4224 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 10 monclient: _renew_subs 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a280a09b0 con 0x7f3a28131b20 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012da0 con 0x7f3a28131b20 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a1effd640 10 monclient: dump: 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.043 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a257fa640 4 set_mon_vals no callback set 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 10 monclient: _renew_subs 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.028+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a28135c80 con 0x7f3a28131b20 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.029+0000 7f3a2d309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.029+0000 7f3a2d309640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.029+0000 7f3a2d309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.029+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28136160 con 0x7f3a28131b20 2024-09-15T01:51:33.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.029+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a10011590 con 0x7f3a28131b20 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.030+0000 7f3a1effd640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 0x7f3a0803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.030+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f3a1004b470 con 0x7f3a28131b20 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.030+0000 7f3a26ffd640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 0x7f3a0803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.030+0000 7f3a26ffd640 10 monclient: get_auth_request con 0x7f3a08038740 auth_method 0 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.032+0000 7f3a26ffd640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 0x7f3a0803abf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a18006df0 tx=0x7f3a18005330 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.034+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3a10019910 con 0x7f3a28131b20 2024-09-15T01:51:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.034+0000 7f3a1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.034+0000 7f3a1effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:33.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a2d309640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a2d309640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a2d309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a280724a0 con 0x7f3a28131b20 2024-09-15T01:51:33.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a1effd640 1 -- 172.21.15.17:0/532064625 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11380 (secure 0 0 0) 0x7f3a10019310 con 0x7f3a28131b20 2024-09-15T01:51:33.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.197+0000 7f3a1effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:33.200 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:33.200 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":16,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:32.561543+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T01:51:28.688141+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T01:51:29.355809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.593725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.627101+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.602809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.641423+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.589740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.592626+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:33.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 msgr2=0x7f3a0803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 0x7f3a0803abf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a18006df0 tx=0x7f3a18005330 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 10 monclient: shutdown 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 msgr2=0x7f3a28133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000c610 tx=0x7f3a10007ed0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 shutdown_connections 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a08038740 0x7f3a0803abf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a28131b20 0x7f3a28133c90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28137e60 0x7f3a281341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 --2- 172.21.15.17:0/532064625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a280a0280 0x7f3a2809b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 >> 172.21.15.17:0/532064625 conn(0x7f3a28071270 msgr2=0x7f3a280a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 shutdown_connections 2024-09-15T01:51:33.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.199+0000 7f3a2d309640 1 -- 172.21.15.17:0/532064625 wait complete. 2024-09-15T01:51:33.212 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T01:51:33.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.306+0000 7fa907268640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.306+0000 7fa907268640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.306+0000 7fa907268640 10 monclient: monmap: 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.307125+0000 2024-09-15T01:51:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.307125+0000 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.306+0000 7fa907268640 1 Processor -- start 2024-09-15T01:51:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 -- start start 2024-09-15T01:51:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient: init 2024-09-15T01:51:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9000a5780 0x7fa90009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): picked mon.noname-c con 0x7fa9000a5780 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa90009ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa900069c90 0x7fa900139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): picked mon.noname-a con 0x7fa900069c90 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9000a5b50 con 0x7fa90009ccf0 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9000a18b0 con 0x7fa900069c90 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90009c5f0 con 0x7fa9000a5780 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa907268640 10 monclient(hunting): authenticate will time out at 753.303601s 2024-09-15T01:51:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa900069c90 0x7fa900139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa900069c90 0x7fa900139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38860/0 (socket says 172.21.15.17:38860) 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 1 -- 172.21.15.17:0/2815258025 learned_addr learned my addr 172.21.15.17:0/2815258025 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): get_auth_request con 0x7fa900069c90 auth_method 0 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905004640 1 --2- 172.21.15.17:0/2815258025 >> v2:172.21.15.67:3300/0 conn(0x7fa9000a5780 0x7fa90009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905004640 10 monclient(hunting): get_auth_request con 0x7fa9000a5780 auth_method 0 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905004640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.307+0000 7fa905004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 1 -- 172.21.15.17:0/2815258025 >> v2:172.21.15.67:3300/0 conn(0x7fa9000a5780 msgr2=0x7fa90009bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 1 --2- 172.21.15.17:0/2815258025 >> v2:172.21.15.67:3300/0 conn(0x7fa9000a5780 0x7fa90009bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 1 -- 172.21.15.17:0/2815258025 >> v1:172.21.15.67:6789/0 conn(0x7fa90009ccf0 legacy=0x7fa900069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.308+0000 7fa905805640 1 -- 172.21.15.17:0/2815258025 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90009d210 con 0x7fa900069c90 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.310+0000 7fa905805640 1 -- 172.21.15.17:0/2815258025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa900069c90 msgr2=0x7fa900139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.310+0000 7fa905805640 1 --2- 172.21.15.17:0/2815258025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa900069c90 0x7fa900139880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa90006a5b0 tx=0x7fa8f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771fe6bb424c6723 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 1 -- 172.21.15.17:0/2815258025 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8007610 con 0x7fa900069c90 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: dump: 2024-09-15T01:51:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 1 -- 172.21.15.17:0/2815258025 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f802d050 con 0x7fa900069c90 2024-09-15T01:51:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 1 -- 172.21.15.17:0/2815258025 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80079a0 con 0x7fa900069c90 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 5 monclient: authenticate success, global_id 4227 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 4 set_mon_vals no callback set 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa8f7fff640 10 monclient: dump: 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.321 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 10 monclient: shutdown 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 -- 172.21.15.17:0/2815258025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa900069c90 msgr2=0x7fa900139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 --2- 172.21.15.17:0/2815258025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa900069c90 0x7fa900139880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa90006a5b0 tx=0x7fa8f8009840 comp rx=0 tx=0).stop 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 -- 172.21.15.17:0/2815258025 shutdown_connections 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 --2- 172.21.15.17:0/2815258025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa900069c90 0x7fa900139880 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 --2- 172.21.15.17:0/2815258025 >> v2:172.21.15.67:3300/0 conn(0x7fa9000a5780 0x7fa90009bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 -- 172.21.15.17:0/2815258025 >> 172.21.15.17:0/2815258025 conn(0x7fa900070e10 msgr2=0x7fa9000a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 -- 172.21.15.17:0/2815258025 shutdown_connections 2024-09-15T01:51:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.311+0000 7fa907268640 1 -- 172.21.15.17:0/2815258025 wait complete. 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.312+0000 7fa907268640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.312+0000 7fa907268640 10 monclient: monmap: 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.312+0000 7fa907268640 1 Processor -- start 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.312+0000 7fa907268640 1 -- start start 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.312+0000 7fa907268640 10 monclient: init 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 0x7fa9001410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): picked mon.noname-c con 0x7fa900069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa90009ccf0 0x7fa900131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): picked mon.noname-b con 0x7fa90009ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): picked mon.noname-a con 0x7fa9000a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90011ec90 con 0x7fa9000a5780 2024-09-15T01:51:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9000a0920 con 0x7fa90009ccf0 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90009b5d0 con 0x7fa900069c90 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa907268640 10 monclient(hunting): authenticate will time out at 753.309348s 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 0x7fa9001410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38866/0 (socket says 172.21.15.17:38866) 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 -- 172.21.15.17:0/4289459720 learned_addr learned my addr 172.21.15.17:0/4289459720 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): get_auth_request con 0x7fa9000a5780 auth_method 0 2024-09-15T01:51:33.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): get_auth_request con 0x7fa900069c90 auth_method 0 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa90009ccf0 0x7fa900131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): get_auth_request con 0x7fa90009ccf0 auth_method 0 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa904803640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 -- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 msgr2=0x7fa9001410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 0x7fa9001410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 -- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa90009ccf0 msgr2=0x7fa900131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa90009ccf0 0x7fa900131b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905805640 1 -- 172.21.15.17:0/4289459720 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8009ae0 con 0x7fa9000a5780 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.313+0000 7fa905004640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 0x7fa9001410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa905805640 1 -- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 msgr2=0x7fa90011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa905805640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009f50 tx=0x7fa8f8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8007170 con 0x7fa9000a5780 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: dump: 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f8032cc0 con 0x7fa9000a5780 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 5 monclient: authenticate success, global_id 4230 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 10 monclient: _renew_subs 2024-09-15T01:51:33.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90011d670 con 0x7fa9000a5780 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f803b680 con 0x7fa9000a5780 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f57fa640 10 monclient: dump: 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.330 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 10 monclient: _renew_subs 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa8f77fe640 4 set_mon_vals no callback set 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.316+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90011d160 con 0x7fa9000a5780 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa8f8038050 con 0x7fa9000a5780 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa907268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa907268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa907268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90011efd0 con 0x7fa9000a5780 2024-09-15T01:51:33.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa8f57fa640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 0x7fa8e403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fa8f806c470 con 0x7fa9000a5780 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa905004640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 0x7fa8e403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.317+0000 7fa905004640 10 monclient: get_auth_request con 0x7fa8e4038740 auth_method 0 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.320+0000 7fa905004640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 0x7fa8e403abf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0113e0 tx=0x7fa8fc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.321+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa90011efd0 con 0x7fa9000a5780 2024-09-15T01:51:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.321+0000 7fa8f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:33.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.321+0000 7fa8f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:33.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.485+0000 7fa907268640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.485+0000 7fa907268640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.485+0000 7fa907268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.485+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa90011efd0 con 0x7fa9000a5780 2024-09-15T01:51:33.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.485+0000 7fa8f57fa640 1 -- 172.21.15.17:0/4289459720 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11380 (secure 0 0 0) 0x7fa90011efd0 con 0x7fa9000a5780 2024-09-15T01:51:33.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.486+0000 7fa8f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:33.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.486+0000 7fa8f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:33.488 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:33.488 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":16,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:32.561543+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T01:51:28.688141+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T01:51:29.355809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.593725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.627101+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.602809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.641423+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.589740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.592626+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.487+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 msgr2=0x7fa8e403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.487+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 0x7fa8e403abf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0113e0 tx=0x7fa8fc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 10 monclient: shutdown 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 msgr2=0x7fa90011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009f50 tx=0x7fa8f8036000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 shutdown_connections 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8e4038740 0x7fa8e403abf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa9000a5780 0x7fa90011b6d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa90009ccf0 0x7fa900131b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 --2- 172.21.15.17:0/4289459720 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa900069c90 0x7fa9001410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 >> 172.21.15.17:0/4289459720 conn(0x7fa900070e10 msgr2=0x7fa90009d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 shutdown_connections 2024-09-15T01:51:33.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.488+0000 7fa907268640 1 -- 172.21.15.17:0/4289459720 wait complete. 2024-09-15T01:51:33.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T01:51:33.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T01:51:33.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T01:51:33.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T01:51:33.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T01:51:33.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T01:51:33.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.614+0000 7ff77959d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.614+0000 7ff77959d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.614+0000 7ff77959d640 10 monclient: monmap: 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.615326+0000 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.615326+0000 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.617 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.618 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.618 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 1 Processor -- start 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 1 -- start start 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 10 monclient: init 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T01:51:33.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff774131b20 0x7ff77409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-c con 0x7ff774131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.615+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7740a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59528/0 (socket says 172.21.15.17:59528) 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 learned_addr learned my addr 172.21.15.17:0/2181386125 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff77409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7740a0040 con 0x7ff77409f1e0 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77409bff0 con 0x7ff774131b20 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7740a14a0 con 0x7ff7740a0280 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff77959d640 10 monclient(hunting): authenticate will time out at 753.612237s 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): get_auth_request con 0x7ff7740a0280 auth_method 0 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7f6cabe34640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7f6cabe34640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7f6cabe34640 10 monclient: monmap: 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.617070+0000 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.617070+0000 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.622 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff772ffd640 1 --2- 172.21.15.17:0/2181386125 >> v2:172.21.15.67:3300/0 conn(0x7ff774131b20 0x7ff77409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff772ffd640 10 monclient(hunting): get_auth_request con 0x7ff774131b20 auth_method 0 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff772ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff772ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> v2:172.21.15.67:3300/0 conn(0x7ff774131b20 msgr2=0x7ff77409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> v2:172.21.15.67:3300/0 conn(0x7ff774131b20 0x7ff77409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> v1:172.21.15.17:6789/0 conn(0x7ff77409f1e0 legacy=0x7ff774068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 monclient: found mon.noname-b 2024-09-15T01:51:33.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.616+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff774132430 con 0x7ff7740a0280 2024-09-15T01:51:33.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 1 Processor -- start 2024-09-15T01:51:33.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 1 -- start start 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient: init 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ca409b180 0x7f6ca409b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca409b180 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ca406eca0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6ca409d260 0x7f6ca4137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca409d260 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.617+0000 7f6cabe34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca4067970 con 0x7f6ca406eca0 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca4099460 con 0x7f6ca409d260 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca4067ff0 con 0x7f6ca409b180 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6cabe34640 10 monclient(hunting): authenticate will time out at 753.614073s 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6ca409d260 0x7f6ca4137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6ca409d260 0x7f6ca4137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38872/0 (socket says 172.21.15.17:38872) 2024-09-15T01:51:33.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 -- 172.21.15.17:0/2821479520 learned_addr learned my addr 172.21.15.17:0/2821479520 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): get_auth_request con 0x7f6ca409d260 auth_method 0 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 1 --2- 172.21.15.17:0/2821479520 >> v2:172.21.15.67:3300/0 conn(0x7f6ca409b180 0x7f6ca409b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): get_auth_request con 0x7f6ca409b180 auth_method 0 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-15T01:51:33.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 -- 172.21.15.17:0/2821479520 >> v2:172.21.15.67:3300/0 conn(0x7f6ca409b180 msgr2=0x7f6ca409b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/2821479520 >> v2:172.21.15.67:3300/0 conn(0x7f6ca409b180 0x7f6ca409b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 -- 172.21.15.17:0/2821479520 >> v1:172.21.15.67:6789/0 conn(0x7f6ca406eca0 legacy=0x7f6ca409cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.618+0000 7f6caa3d1640 1 -- 172.21.15.17:0/2821479520 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4099c50 con 0x7f6ca409d260 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 msgr2=0x7ff774131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff768009910 tx=0x7ff7680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ed35a4b7076984 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 msgr2=0x7ff774131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 msgr2=0x7ff774131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff768009910 tx=0x7ff7680091d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff768009910 tx=0x7ff7680091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 msgr2=0x7ff774131ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7740a0280 0x7ff774131ef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient: _add_conns ranks=[1,3] 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff75c0034c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 1 --2- 172.21.15.17:0/2181386125 >> v2:172.21.15.17:3300/0 conn(0x7ff75c007440 0x7ff75c0098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient(hunting): picked mon.noname-a con 0x7ff75c007440 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c00a1c0 con 0x7ff75c0034c0 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75c00b3c0 con 0x7ff75c007440 2024-09-15T01:51:33.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.619+0000 7ff771ffb640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> v2:172.21.15.17:3300/0 conn(0x7ff75c007440 0x7ff75c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): get_auth_request con 0x7ff75c007440 auth_method 0 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient(hunting): handle_auth_done global_id 4220 payload 293 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> v1:172.21.15.67:6789/0 conn(0x7ff75c0034c0 legacy=0x7ff75c005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.620+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c00b5f0 con 0x7ff75c007440 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6caa3d1640 1 -- 172.21.15.17:0/2821479520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409d260 msgr2=0x7f6ca4137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/2821479520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409d260 0x7f6ca4137780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ca409e9b0 tx=0x7f6c9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad165a1278bd201 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 1 -- 172.21.15.17:0/2821479520 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c007610 con 0x7f6ca409d260 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: dump: 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6cabe34640 5 monclient: authenticate success, global_id 4233 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6cabe34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 1 -- 172.21.15.17:0/2821479520 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c02d050 con 0x7f6ca409d260 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 1 -- 172.21.15.17:0/2821479520 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0079e0 con 0x7f6ca409d260 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6ca8bce640 10 monclient: dump: 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6cabe34640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.621+0000 7f6cabe34640 4 set_mon_vals no callback set 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 10 monclient: shutdown 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 -- 172.21.15.17:0/2821479520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409d260 msgr2=0x7f6ca4137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 --2- 172.21.15.17:0/2821479520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409d260 0x7f6ca4137780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ca409e9b0 tx=0x7f6c9c009840 comp rx=0 tx=0).stop 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 -- 172.21.15.17:0/2821479520 shutdown_connections 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 --2- 172.21.15.17:0/2821479520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409d260 0x7f6ca4137780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 --2- 172.21.15.17:0/2821479520 >> v2:172.21.15.67:3300/0 conn(0x7f6ca409b180 0x7f6ca409b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 -- 172.21.15.17:0/2821479520 >> 172.21.15.17:0/2821479520 conn(0x7f6ca40a2100 msgr2=0x7f6ca40a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 -- 172.21.15.17:0/2821479520 shutdown_connections 2024-09-15T01:51:33.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7f6cabe34640 1 -- 172.21.15.17:0/2821479520 wait complete. 2024-09-15T01:51:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff7727fc640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff75c007440 msgr2=0x7ff75c0098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff7727fc640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff75c007440 0x7ff75c0098f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7740a2f90 tx=0x7ff768032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa64f4980a85985 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768033a60 con 0x7ff75c007440 2024-09-15T01:51:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff771ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff771ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.622+0000 7ff771ffb640 10 monclient: dump: 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff768033bc0 con 0x7ff75c007440 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 5 monclient: authenticate success, global_id 4220 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 1 -- 172.21.15.17:0/2181386125 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76803d420 con 0x7ff75c007440 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff771ffb640 10 monclient: dump: 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 4 set_mon_vals no callback set 2024-09-15T01:51:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7f6cabe34640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7f6cabe34640 10 monclient: monmap: 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 10 monclient: shutdown 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff75c007440 msgr2=0x7ff75c0098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff77959d640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff75c007440 0x7ff75c0098f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7740a2f90 tx=0x7ff768032000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.623+0000 7ff7737fe640 1 -- 172.21.15.17:0/2181386125 reap_dead start 2024-09-15T01:51:33.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 shutdown_connections 2024-09-15T01:51:33.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7ff77959d640 1 --2- 172.21.15.17:0/2181386125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff75c007440 0x7ff75c0098f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 >> 172.21.15.17:0/2181386125 conn(0x7ff774071270 msgr2=0x7ff7740a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 Processor -- start 2024-09-15T01:51:33.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 shutdown_connections 2024-09-15T01:51:33.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 -- start start 2024-09-15T01:51:33.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient: init 2024-09-15T01:51:33.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7ff77959d640 1 -- 172.21.15.17:0/2181386125 wait complete. 2024-09-15T01:51:33.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:33.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ca406eca0 0x7f6ca4140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca406eca0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409b180 0x7f6ca4131ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca409b180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca409d260 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4121250 con 0x7f6ca409b180 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41327e0 con 0x7f6ca406eca0 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4120ee0 con 0x7f6ca409d260 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6cabe34640 10 monclient(hunting): authenticate will time out at 753.620895s 2024-09-15T01:51:33.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59538/0 (socket says 172.21.15.17:59538) 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 learned_addr learned my addr 172.21.15.17:0/3097117810 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 10 monclient(hunting): get_auth_request con 0x7f6ca409d260 auth_method 0 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.624+0000 7f6caa3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ca406eca0 0x7f6ca4140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409b180 0x7f6ca4131ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): get_auth_request con 0x7f6ca409b180 auth_method 0 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): get_auth_request con 0x7f6ca406eca0 auth_method 0 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca93cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6ca9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7ff77959d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7ff77959d640 10 monclient: monmap: 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.656 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ca406eca0 msgr2=0x7f6ca4140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ca406eca0 0x7f6ca4140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409b180 msgr2=0x7f6ca4131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ca409b180 0x7f6ca4131ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 monclient: found mon.noname-c 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.625+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009ae0 con 0x7f6ca409d260 2024-09-15T01:51:33.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 Processor -- start 2024-09-15T01:51:33.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 -- start start 2024-09-15T01:51:33.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient: init 2024-09-15T01:51:33.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff75c007440 0x7ff77411c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-c con 0x7ff75c007440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 0x7ff77411c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 0x7ff77411c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-a con 0x7ff77411c610 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 0x7ff77411c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38898/0 (socket says 172.21.15.17:38898) 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77411cf20 0x7ff7741478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): picked mon.noname-b con 0x7ff77411cf20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7740a0a70 con 0x7ff77411c610 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774131760 con 0x7ff77411cf20 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77411f980 con 0x7ff75c007440 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff77959d640 10 monclient(hunting): authenticate will time out at 753.622828s 2024-09-15T01:51:33.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 1 -- 172.21.15.17:0/598649929 learned_addr learned my addr 172.21.15.17:0/598649929 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): get_auth_request con 0x7ff77411c610 auth_method 0 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.626+0000 7ff7727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7737fe640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77411cf20 0x7ff7741478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7737fe640 10 monclient(hunting): get_auth_request con 0x7ff77411cf20 auth_method 0 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff75c007440 msgr2=0x7ff77411c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff75c007440 0x7ff77411c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77411cf20 msgr2=0x7ff7741478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77411cf20 0x7ff7741478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7ff7727fc640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77411e7b0 con 0x7ff77411c610 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 msgr2=0x7f6ca4096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.627+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009f50 tx=0x7f6c9c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c007170 con 0x7f6ca409d260 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: dump: 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c032cc0 con 0x7f6ca409d260 2024-09-15T01:51:33.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c03b640 con 0x7f6ca409d260 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c99ffb640 10 monclient: dump: 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6c9bfff640 4 set_mon_vals no callback set 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 5 monclient: authenticate success, global_id 4223 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 10 monclient: _renew_subs 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca4098570 con 0x7f6ca409d260 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 10 monclient: _renew_subs 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.628+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca411fd10 con 0x7f6ca409d260 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6c9c038050 con 0x7f6ca409d260 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6cabe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ca40724a0 con 0x7f6ca409d260 2024-09-15T01:51:33.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6caa3d1640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009f50 tx=0x7f6c9c036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f6c9c06e990 mon_get_version_reply(handle=1 version=16) v2 2024-09-15T01:51:33.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6c99ffb640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c88038790 0x7f6c8803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f6c9c06c470 con 0x7f6ca409d260 2024-09-15T01:51:33.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7ff7727fc640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 msgr2=0x7ff77411c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7ff7727fc640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 0x7ff77411c9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7680097e0 tx=0x7ff76803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6c99ffb640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7f6c8803e4a0 0x7f6c880408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.629+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c88040fa0 con 0x7f6c8803e4a0 2024-09-15T01:51:33.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768033a60 con 0x7ff77411c610 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: dump: 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.675 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff768006b30 con 0x7ff77411c610 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7680468b0 con 0x7ff77411c610 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff75affd640 10 monclient: dump: 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff7717fa640 4 set_mon_vals no callback set 2024-09-15T01:51:33.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7f6ca9bd0640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c88038790 0x7f6c8803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7f6ca9bd0640 10 monclient: get_auth_request con 0x7f6c88038790 auth_method 0 2024-09-15T01:51:33.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7f6ca93cf640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7f6c8803e4a0 0x7f6c880408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7f6ca93cf640 10 monclient: get_auth_request con 0x7f6c8803e4a0 auth_method 0 2024-09-15T01:51:33.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 5 monclient: authenticate success, global_id 4239 2024-09-15T01:51:33.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 10 monclient: _renew_subs 2024-09-15T01:51:33.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77411e7b0 con 0x7ff77411c610 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 10 monclient: _renew_subs 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.630+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77411e7b0 con 0x7ff77411c610 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff77959d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff77411e7b0 con 0x7ff77411c610 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff76804e070 con 0x7ff77411c610 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff760038790 0x7ff76003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7ff76806dca0 con 0x7ff77411c610 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7ff76003e690 0x7ff760040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff760041190 con 0x7ff76003e690 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff772ffd640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff760038790 0x7ff76003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7680731d0 con 0x7ff77411c610 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff75affd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.631+0000 7ff772ffd640 10 monclient: get_auth_request con 0x7ff760038790 auth_method 0 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.632+0000 7ff7737fe640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7ff76003e690 0x7ff760040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.632+0000 7ff7737fe640 10 monclient: get_auth_request con 0x7ff76003e690 auth_method 0 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.632+0000 7f6ca9bd0640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c88038790 0x7f6c8803ac40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d20 tx=0x7f6ca00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.634+0000 7ff772ffd640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff760038790 0x7ff76003ac40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.635+0000 7f6ca93cf640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7f6c8803e4a0 0x7f6c880408d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.636+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== osd.4 v2:172.21.15.67:6802/31881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6c88040fa0 con 0x7f6c8803e4a0 2024-09-15T01:51:33.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.636+0000 7ff7737fe640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7ff76003e690 0x7ff760040ac0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.637+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== osd.2 v2:172.21.15.17:6816/31929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff760041190 con 0x7ff76003e690 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.652+0000 7f15000ef640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.652+0000 7f15000ef640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.652+0000 7f15000ef640 10 monclient: monmap: 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.653514+0000 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.653514+0000 2024-09-15T01:51:33.683 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.652+0000 7fd74331d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.652+0000 7fd74331d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7fd74331d640 10 monclient: monmap: 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.653827+0000 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.653827+0000 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.685 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ca41416c0 con 0x7f6c8803e4a0 2024-09-15T01:51:33.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7f15000ef640 1 Processor -- start 2024-09-15T01:51:33.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7fd74331d640 1 Processor -- start 2024-09-15T01:51:33.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7f15000ef640 1 -- start start 2024-09-15T01:51:33.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7f15000ef640 10 monclient: init 2024-09-15T01:51:33.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7f6c99ffb640 1 -- 172.21.15.17:0/3097117810 <== osd.4 v2:172.21.15.67:6802/31881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ca41416c0 con 0x7f6c8803e4a0 2024-09-15T01:51:33.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7fd74331d640 1 -- start start 2024-09-15T01:51:33.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.653+0000 7fd74331d640 10 monclient: init 2024-09-15T01:51:33.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7f6c8803e4a0 msgr2=0x7f6c880408d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7f6c8803e4a0 0x7f6c880408d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T01:51:33.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f14f8131b20 0x7f14f809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c88038790 msgr2=0x7f6c8803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c88038790 0x7f6c8803ac40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d20 tx=0x7f6ca00090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 10 monclient: shutdown 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 20 monclient: shutdown canceling and discarding version request 1 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 msgr2=0x7f6ca4096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 --2- 172.21.15.17:0/3097117810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ca409d260 0x7f6ca4096a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009f50 tx=0x7f6c9c036000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6caa3d1640 1 -- 172.21.15.17:0/3097117810 reap_dead start 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-a con 0x7f14f8131b20 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 0x7f14f8131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-c con 0x7f14f80a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f809f1e0 0x7f14f8132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-b con 0x7f14f809f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f80a0040 con 0x7f14f8131b20 2024-09-15T01:51:33.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f809bff0 con 0x7f14f80a0280 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80a14a0 con 0x7f14f809f1e0 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f15000ef640 10 monclient(hunting): authenticate will time out at 753.650848s 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 shutdown_connections 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 >> 172.21.15.17:0/3097117810 conn(0x7f6ca40a2100 msgr2=0x7f6ca41329f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 shutdown_connections 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c09c5b0 0x7fd73c09c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c09c5b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd73c06eda0 0x7fd73c06f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-c con 0x7fd73c06eda0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f6cabe34640 1 -- 172.21.15.17:0/3097117810 wait complete. 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd73c09d8e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c09bad0 con 0x7fd73c09d8e0 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd74331d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c06f6b0 con 0x7fd73c06eda0 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7fd7408b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd73c06eda0 0x7fd73c06f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.654+0000 7f14fd68a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 0x7f14f8131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 0x7f14f8131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:58810/0 (socket says 172.21.15.17:58810) 2024-09-15T01:51:33.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 1 -- 172.21.15.17:0/3121396470 learned_addr learned my addr 172.21.15.17:0/3121396470 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 10 monclient(hunting): get_auth_request con 0x7f14f80a0280 auth_method 0 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fd68a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd73c06eda0 0x7fd73c06f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:58822/0 (socket says 172.21.15.17:58822) 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 -- 172.21.15.17:0/2799018295 learned_addr learned my addr 172.21.15.17:0/2799018295 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd74331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c09a990 con 0x7fd73c09c5b0 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd74331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd74331d640 10 monclient(hunting): authenticate will time out at 753.651090s 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 --2- 172.21.15.17:0/3121396470 >> v2:172.21.15.17:3300/0 conn(0x7f14f8131b20 0x7f14f809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f809f1e0 0x7f14f8132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): get_auth_request con 0x7f14f809f1e0 auth_method 0 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): get_auth_request con 0x7f14f8131b20 auth_method 0 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): get_auth_request con 0x7fd73c06eda0 auth_method 0 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c09c5b0 0x7fd73c09c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): get_auth_request con 0x7fd73c09c5b0 auth_method 0 2024-09-15T01:51:33.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fe68c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 -- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f809f1e0 msgr2=0x7f14f8132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f809f1e0 0x7f14f8132430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 -- 172.21.15.17:0/3121396470 >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 msgr2=0x7f14f8131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 --2- 172.21.15.17:0/3121396470 >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 0x7f14f8131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7f14fde8b640 1 -- 172.21.15.17:0/3121396470 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8068e80 con 0x7f14f8131b20 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 -- 172.21.15.17:0/2799018295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c09c5b0 msgr2=0x7fd73c09c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c09c5b0 0x7fd73c09c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 -- 172.21.15.17:0/2799018295 >> v1:172.21.15.67:6789/0 conn(0x7fd73c09d8e0 legacy=0x7fd73c09dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 monclient: found mon.noname-c 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:33.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.655+0000 7fd7408b8640 1 -- 172.21.15.17:0/2799018295 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c09b7f0 con 0x7fd73c06eda0 2024-09-15T01:51:33.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.656+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 --> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7740724a0 con 0x7ff76003e690 2024-09-15T01:51:33.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff75affd640 1 -- 172.21.15.17:0/598649929 <== osd.2 v2:172.21.15.17:6816/31929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff7740724a0 con 0x7ff76003e690 2024-09-15T01:51:33.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7fa8694af640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7fa8694af640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7ff76003e690 msgr2=0x7ff760040ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7ff76003e690 0x7ff760040ac0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7fa8694af640 10 monclient: monmap: 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.658274+0000 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.658274+0000 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff760038790 msgr2=0x7ff76003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff760038790 0x7ff76003ac40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764008040 comp rx=0 tx=0).stop 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 10 monclient: shutdown 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 msgr2=0x7ff77411c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 --2- 172.21.15.17:0/598649929 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff77411c610 0x7ff77411c9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7680097e0 tx=0x7ff76803d000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff7737fe640 1 -- 172.21.15.17:0/598649929 reap_dead start 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 shutdown_connections 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.657+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 >> 172.21.15.17:0/598649929 conn(0x7ff774071270 msgr2=0x7ff77409e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 shutdown_connections 2024-09-15T01:51:33.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 Processor -- start 2024-09-15T01:51:33.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7ff77959d640 1 -- 172.21.15.17:0/598649929 wait complete. 2024-09-15T01:51:33.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 -- start start 2024-09-15T01:51:33.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient: init 2024-09-15T01:51:33.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7f14fde8b640 1 -- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f8131b20 msgr2=0x7f14f809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7f14fde8b640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f8131b20 0x7f14f809eca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14f80a5010 tx=0x7f14f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6537dba0da35f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa864131b20 0x7fa86409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-a con 0x7fa864131b20 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8640a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa86409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8640a0040 con 0x7fa86409f1e0 2024-09-15T01:51:33.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86409bff0 con 0x7fa8640a0280 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.658+0000 7fa8694af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8640a14a0 con 0x7fa864131b20 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa8694af640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa8694af640 10 monclient(hunting): authenticate will time out at 753.655016s 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa8637fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa86409f1e0 0x7fa864138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33630/0 (socket says 172.21.15.17:33630) 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa8637fe640 1 -- 172.21.15.17:0/491199026 learned_addr learned my addr 172.21.15.17:0/491199026 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 1 --2- 172.21.15.17:0/491199026 >> v2:172.21.15.17:3300/0 conn(0x7fa864131b20 0x7fa86409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): get_auth_request con 0x7fa864131b20 auth_method 0 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd7408b8640 1 -- 172.21.15.17:0/2799018295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c06eda0 msgr2=0x7fd73c06f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd7408b8640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c06eda0 0x7fd73c06f170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009910 tx=0x7fd72c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed60059a40a33223 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fbfdef09640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fbfdef09640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fbfdef09640 10 monclient: monmap: 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.660034+0000 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.660034+0000 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.710 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 1 -- 172.21.15.17:0/3121396470 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f402adf0 con 0x7f14f8131b20 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: dump: 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.712 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 1 -- 172.21.15.17:0/3121396470 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4032440 con 0x7f14f8131b20 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 1 -- 172.21.15.17:0/3121396470 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f40327e0 con 0x7f14f8131b20 2024-09-15T01:51:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f14fce89640 10 monclient: dump: 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.714 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f15000ef640 5 monclient: authenticate success, global_id 4242 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f15000ef640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7f15000ef640 4 set_mon_vals no callback set 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 1 -- 172.21.15.17:0/2799018295 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c032440 con 0x7fd73c06eda0 2024-09-15T01:51:33.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: dump: 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.716 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd74331d640 5 monclient: authenticate success, global_id 4243 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd74331d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 1 -- 172.21.15.17:0/2799018295 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c0325a0 con 0x7fd73c06eda0 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 1 -- 172.21.15.17:0/2799018295 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c032940 con 0x7fd73c06eda0 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd74331d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd74331d640 4 set_mon_vals no callback set 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fd733fff640 10 monclient: dump: 2024-09-15T01:51:33.717 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 1 -- 172.21.15.17:0/491199026 >> v1:172.21.15.67:6789/0 conn(0x7fa8640a0280 legacy=0x7fa864131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 1 -- 172.21.15.17:0/491199026 >> v1:172.21.15.17:6789/0 conn(0x7fa86409f1e0 legacy=0x7fa864138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.659+0000 7fa862ffd640 1 -- 172.21.15.17:0/491199026 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864132600 con 0x7fa864131b20 2024-09-15T01:51:33.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fbfdef09640 1 Processor -- start 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 10 monclient: shutdown 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 -- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f8131b20 msgr2=0x7f14f809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f8131b20 0x7f14f809eca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14f80a5010 tx=0x7f14f40090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 10 monclient: shutdown 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c06eda0 msgr2=0x7fd73c06f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c06eda0 0x7fd73c06f170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009910 tx=0x7fd72c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fbfdef09640 1 -- start start 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fbfdef09640 10 monclient: init 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 -- 172.21.15.17:0/3121396470 shutdown_connections 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 --2- 172.21.15.17:0/3121396470 >> v2:172.21.15.67:3300/0 conn(0x7f14f80a0280 0x7f14f8131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f809f1e0 0x7f14f8132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 --2- 172.21.15.17:0/3121396470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f8131b20 0x7f14f809eca0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 -- 172.21.15.17:0/3121396470 >> 172.21.15.17:0/3121396470 conn(0x7f14f8071270 msgr2=0x7f14f80a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 shutdown_connections 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c06eda0 0x7fd73c06f170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 --2- 172.21.15.17:0/2799018295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c09c5b0 0x7fd73c09c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 >> 172.21.15.17:0/2799018295 conn(0x7fd73c0714c0 msgr2=0x7fd73c0a2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 -- 172.21.15.17:0/3121396470 shutdown_connections 2024-09-15T01:51:33.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7f15000ef640 1 -- 172.21.15.17:0/3121396470 wait complete. 2024-09-15T01:51:33.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfd81319a0 0x7fbfd8131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd81319a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfd81322b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd809f240 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd80a1300 con 0x7fbfd81322b0 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd81326b0 con 0x7fbfd81319a0 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd806a260 con 0x7fbfd809f240 2024-09-15T01:51:33.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdef09640 10 monclient(hunting): authenticate will time out at 753.657172s 2024-09-15T01:51:33.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.660+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 shutdown_connections 2024-09-15T01:51:33.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fd74331d640 1 -- 172.21.15.17:0/2799018295 wait complete. 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59574/0 (socket says 172.21.15.17:59574) 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfd7fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbfd81322b0 0x7fbfd809eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33646/0 (socket says 172.21.15.17:33646) 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2347441135 learned_addr learned my addr 172.21.15.17:0/2347441135 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): get_auth_request con 0x7fbfd809f240 auth_method 0 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 1 --2- 172.21.15.17:0/2347441135 >> v2:172.21.15.67:3300/0 conn(0x7fbfd81319a0 0x7fbfd8131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): get_auth_request con 0x7fbfd81319a0 auth_method 0 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2347441135 >> v2:172.21.15.67:3300/0 conn(0x7fbfd81319a0 msgr2=0x7fbfd8131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2347441135 >> v2:172.21.15.67:3300/0 conn(0x7fbfd81319a0 0x7fbfd8131d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2347441135 >> v1:172.21.15.17:6789/0 conn(0x7fbfd81322b0 legacy=0x7fbfd809eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 monclient: found mon.noname-b 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.661+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2347441135 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80a0750 con 0x7fbfd809f240 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient: monmap: 2024-09-15T01:51:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7fd74331d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7fd74331d640 10 monclient: monmap: 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 1 Processor -- start 2024-09-15T01:51:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 1 -- start start 2024-09-15T01:51:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient: init 2024-09-15T01:51:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7fa862ffd640 1 -- 172.21.15.17:0/491199026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa864131b20 msgr2=0x7fa86409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7fa862ffd640 1 --2- 172.21.15.17:0/491199026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa864131b20 0x7fa86409eca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa854009850 tx=0x7fa8540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f242f2d24abcdd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 0x7f14f8140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-a con 0x7f14f809f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f80a0280 0x7f14f8141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-b con 0x7f14f80a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7f15000ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f8131b20 0x7f14f8141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.662+0000 7fa861ffb640 1 -- 172.21.15.17:0/491199026 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854032440 con 0x7fa864131b20 2024-09-15T01:51:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: dump: 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 5 monclient: authenticate success, global_id 4245 2024-09-15T01:51:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 1 -- 172.21.15.17:0/491199026 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8540325a0 con 0x7fa864131b20 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 1 -- 172.21.15.17:0/491199026 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854032940 con 0x7fa864131b20 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 4 set_mon_vals no callback set 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa861ffb640 10 monclient: dump: 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.737 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.737 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.737 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.737 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.737 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): picked mon.noname-c con 0x7f14f8131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8120930 con 0x7f14f809f1e0 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f811fae0 con 0x7f14f80a0280 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80a09b0 con 0x7f14f8131b20 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f15000ef640 10 monclient(hunting): authenticate will time out at 753.659092s 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 0x7f14f8140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 0x7f14f8140fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38916/0 (socket says 172.21.15.17:38916) 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 -- 172.21.15.17:0/217176789 learned_addr learned my addr 172.21.15.17:0/217176789 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): get_auth_request con 0x7f14f809f1e0 auth_method 0 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f80a0280 0x7f14f8141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): get_auth_request con 0x7f14f80a0280 auth_method 0 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fd74331d640 1 Processor -- start 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fd74331d640 1 -- start start 2024-09-15T01:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fd74331d640 10 monclient: init 2024-09-15T01:51:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 10 monclient: shutdown 2024-09-15T01:51:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 1 -- 172.21.15.17:0/491199026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa864131b20 msgr2=0x7fa86409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7fa8694af640 1 --2- 172.21.15.17:0/491199026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa864131b20 0x7fa86409eca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa854009850 tx=0x7fa8540090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fd68a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f8131b20 msgr2=0x7f14f8141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14f8131b20 0x7f14f8141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f80a0280 msgr2=0x7f14f8141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f80a0280 0x7f14f8141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.663+0000 7f14fde8b640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f4009390 con 0x7f14f809f1e0 2024-09-15T01:51:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 1 -- 172.21.15.17:0/491199026 shutdown_connections 2024-09-15T01:51:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 1 --2- 172.21.15.17:0/491199026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa864131b20 0x7fa86409eca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 1 -- 172.21.15.17:0/491199026 >> 172.21.15.17:0/491199026 conn(0x7fa864071270 msgr2=0x7fa8640a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 1 -- 172.21.15.17:0/491199026 shutdown_connections 2024-09-15T01:51:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 1 -- 172.21.15.17:0/491199026 wait complete. 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 0x7fd73c0689a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-c con 0x7fd73c06eda0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd73c09c5b0 0x7fd73c068ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2347441135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 msgr2=0x7fbfd8138ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2347441135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009f20 tx=0x7fbfd0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2ea77ff1334782 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fa8694af640 10 monclient: monmap: 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.747 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-a con 0x7fd73c09c5b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd74331d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c09d8e0 0x7fd73c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd7410b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 0x7fd73c0689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.664+0000 7fd7410b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 0x7fd73c0689a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59590/0 (socket says 172.21.15.17:59590) 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c09d8e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 -- 172.21.15.17:0/2339974698 learned_addr learned my addr 172.21.15.17:0/2339974698 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c098c30 con 0x7fd73c09c5b0 2024-09-15T01:51:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 1 -- 172.21.15.17:0/2347441135 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0007610 con 0x7fbfd809f240 2024-09-15T01:51:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c11c7f0 con 0x7fd73c09d8e0 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: dump: 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.750 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c06a470 con 0x7fd73c06eda0 2024-09-15T01:51:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 1 -- 172.21.15.17:0/2347441135 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd002d050 con 0x7fbfd809f240 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 1 -- 172.21.15.17:0/2347441135 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd00079a0 con 0x7fbfd809f240 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfd77fe640 10 monclient: dump: 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfdef09640 5 monclient: authenticate success, global_id 4232 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfdef09640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fbfdef09640 4 set_mon_vals no callback set 2024-09-15T01:51:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fa8694af640 1 Processor -- start 2024-09-15T01:51:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c09d8e0 0x7fd73c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd74331d640 10 monclient(hunting): authenticate will time out at 753.661318s 2024-09-15T01:51:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fa8694af640 1 -- start start 2024-09-15T01:51:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fa8694af640 10 monclient: init 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): get_auth_request con 0x7fd73c06eda0 auth_method 0 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): get_auth_request con 0x7fd73c09d8e0 auth_method 0 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7418ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c09d8e0 msgr2=0x7fd73c069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd73c09d8e0 0x7fd73c069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd73c09c5b0 msgr2=0x7fd73c068ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd73c09c5b0 0x7fd73c068ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 monclient: found mon.noname-c 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.665+0000 7fd7410b9640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c009450 con 0x7fd73c06eda0 2024-09-15T01:51:33.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 0x7fa864099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-c con 0x7fa86409f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 0x7fa86409a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-a con 0x7fa8640a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa864131b20 0x7fa86409a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): picked mon.noname-b con 0x7fa864131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86412b6d0 con 0x7fa8640a0280 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86412c630 con 0x7fa864131b20 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86412cbf0 con 0x7fa86409f1e0 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8694af640 10 monclient(hunting): authenticate will time out at 753.662109s 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 0x7fa864099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 0x7fa864099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59602/0 (socket says 172.21.15.17:59602) 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 1 -- 172.21.15.17:0/3087506178 learned_addr learned my addr 172.21.15.17:0/3087506178 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): get_auth_request con 0x7fa86409f1e0 auth_method 0 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 0x7fa86409a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa864131b20 0x7fa86409a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): get_auth_request con 0x7fa8640a0280 auth_method 0 2024-09-15T01:51:33.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 10 monclient: shutdown 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 -- 172.21.15.17:0/2347441135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 msgr2=0x7fbfd8138ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2347441135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009f20 tx=0x7fbfd0009840 comp rx=0 tx=0).stop 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 -- 172.21.15.17:0/2347441135 shutdown_connections 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2347441135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd809f240 0x7fbfd8138ce0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2347441135 >> v2:172.21.15.67:3300/0 conn(0x7fbfd81319a0 0x7fbfd8131d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 -- 172.21.15.17:0/2347441135 >> 172.21.15.17:0/2347441135 conn(0x7fbfd80714e0 msgr2=0x7fbfd80a50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): get_auth_request con 0x7fa864131b20 auth_method 0 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 -- 172.21.15.17:0/2347441135 shutdown_connections 2024-09-15T01:51:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fbfdef09640 1 -- 172.21.15.17:0/2347441135 wait complete. 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7fa8627fc640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 msgr2=0x7fa864099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 0x7fa864099ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14fde8b640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 msgr2=0x7f14f8140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14fde8b640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 0x7f14f8140fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009880 tx=0x7f14f4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4031c70 con 0x7f14f809f1e0 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa864131b20 msgr2=0x7fa86409a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa864131b20 0x7fa86409a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8627fc640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854009390 con 0x7fa8640a0280 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa8637fe640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa864131b20 0x7fa86409a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fa862ffd640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa86409f1e0 0x7fa864099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 10 monclient: dump: 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f15000ef640 5 monclient: authenticate success, global_id 4248 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f15000ef640 10 monclient: _renew_subs 2024-09-15T01:51:33.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f15000ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f80a3180 con 0x7f14f809f1e0 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4031dd0 con 0x7f14f809f1e0 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.666+0000 7f14effff640 4 set_mon_vals no callback set 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fbfdef09640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.667+0000 7fbfdef09640 10 monclient: monmap: 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.770 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fbfdef09640 1 Processor -- start 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f403c650 con 0x7f14f809f1e0 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f14edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f14edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f14edffb640 10 monclient: dump: 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f15000ef640 10 monclient: _renew_subs 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f15000ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f811ee70 con 0x7f14f809f1e0 2024-09-15T01:51:33.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7410b9640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 msgr2=0x7fd73c0689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7410b9640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 0x7fd73c0689a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd73800ea20 tx=0x7fd7380090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738009600 con 0x7fd73c06eda0 2024-09-15T01:51:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fbfdef09640 1 -- start start 2024-09-15T01:51:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 10 monclient: dump: 2024-09-15T01:51:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd7317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 5 monclient: authenticate success, global_id 4235 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 10 monclient: _renew_subs 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c11cb00 con 0x7fd73c06eda0 2024-09-15T01:51:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 10 monclient: _renew_subs 2024-09-15T01:51:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f14f403ca40 con 0x7f14f809f1e0 2024-09-15T01:51:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c11d0c0 con 0x7fd73c06eda0 2024-09-15T01:51:33.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.668+0000 7fbfdef09640 10 monclient: init 2024-09-15T01:51:33.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f15000ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f14edffb640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14dc038740 0x7f14dc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f14f406dcf0 con 0x7f14f809f1e0 2024-09-15T01:51:33.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fa8627fc640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 msgr2=0x7fa86409a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fa8627fc640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 0x7fa86409a410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c4e0 tx=0x7fa858007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14f8121d20 con 0x7f14f809f1e0 2024-09-15T01:51:33.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd74331d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:33.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 0x7fbfd8131370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd809f240 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f14edffb640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7f14dc03e540 0x7f14dc040970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 0x7fbfd8133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd81319a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdef09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd81322b0 0x7fbfd8134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fbfdcca5640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 0x7fbfd8131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14dc041040 con 0x7f14dc03e540 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 0x7fbfd8131370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38952/0 (socket says 172.21.15.17:38952) 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd81322b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd8135510 con 0x7fbfd809f240 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80a4250 con 0x7fbfd81319a0 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd81360a0 con 0x7fbfd81322b0 2024-09-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdef09640 10 monclient(hunting): authenticate will time out at 753.666109s 2024-09-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 1 -- 172.21.15.17:0/2478120119 learned_addr learned my addr 172.21.15.17:0/2478120119 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14f403b3a0 con 0x7f14f809f1e0 2024-09-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7f14edffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): get_auth_request con 0x7fbfd809f240 auth_method 0 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 0x7fbfd8133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd73c11bc60 con 0x7fd73c06eda0 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73800cc20 con 0x7fd73c06eda0 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738013470 con 0x7fd73c06eda0 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 10 monclient: dump: 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd7380137e0 con 0x7fd73c06eda0 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.669+0000 7fd7337fe640 4 set_mon_vals no callback set 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd720038740 0x7fd72003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): get_auth_request con 0x7fbfd81319a0 auth_method 0 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd81322b0 0x7fbfd8134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 10 monclient(hunting): get_auth_request con 0x7fbfd81322b0 auth_method 0 2024-09-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd73804c0e0 con 0x7fd73c06eda0 2024-09-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7fd72003e5c0 0x7fd7200409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7200410c0 con 0x7fd72003e5c0 2024-09-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd73801aac0 con 0x7fd73c06eda0 2024-09-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7317fa640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580117a0 con 0x7fa8640a0280 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: dump: 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.790 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858011920 con 0x7fa8640a0280 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7408b8640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd720038740 0x7fd72003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-09-15T01:51:33.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd81322b0 msgr2=0x7fbfd8134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd81322b0 0x7fbfd8134500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fd7408b8640 10 monclient: get_auth_request con 0x7fd720038740 auth_method 0 2024-09-15T01:51:33.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 5 monclient: authenticate success, global_id 4251 2024-09-15T01:51:33.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 10 monclient: _renew_subs 2024-09-15T01:51:33.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 msgr2=0x7fbfd8131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86412c910 con 0x7fa8640a0280 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858011ce0 con 0x7fa8640a0280 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8617fa640 4 set_mon_vals no callback set 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 0x7fbfd8131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 monclient: found mon.noname-b 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfd7fff640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0009ae0 con 0x7fbfd81319a0 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa84affd640 10 monclient: dump: 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 10 monclient: _renew_subs 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86412b1b0 con 0x7fa8640a0280 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd81322b0 0x7fbfd8134500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.670+0000 7fbfdcca5640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd809f240 0x7fbfd8131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7f14fd68a640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14dc038740 0x7f14dc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7f14fd68a640 10 monclient: get_auth_request con 0x7f14dc038740 auth_method 0 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7f14fe68c640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7f14dc03e540 0x7f14dc040970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7f14fe68c640 10 monclient: get_auth_request con 0x7f14dc03e540 auth_method 0 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7fd7418ba640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7fd72003e5c0 0x7fd7200409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7fd7418ba640 10 monclient: get_auth_request con 0x7fd72003e5c0 auth_method 0 2024-09-15T01:51:33.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.671+0000 7fa8694af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa8640724a0 con 0x7fa8640a0280 2024-09-15T01:51:33.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa858012630 con 0x7fa8640a0280 2024-09-15T01:51:33.799 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:33.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T01:51:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa850038790 0x7fa85003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fa85804b520 con 0x7fa8640a0280 2024-09-15T01:51:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7fa85003e690 0x7fa850040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa850041190 con 0x7fa85003e690 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa85804dab0 con 0x7fa8640a0280 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa84affd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa8637fe640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7fa85003e690 0x7fa850040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.672+0000 7fa8637fe640 10 monclient: get_auth_request con 0x7fa85003e690 auth_method 0 2024-09-15T01:51:33.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd7fff640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 msgr2=0x7fbfd8133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd7fff640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 0x7fbfd8133fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c490 tx=0x7fbfc4007e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40074f0 con 0x7fbfd81319a0 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: dump: 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc4007670 con 0x7fbfd81319a0 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4005a20 con 0x7fbfd81319a0 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd4ff9640 10 monclient: dump: 2024-09-15T01:51:33.803 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 5 monclient: authenticate success, global_id 4258 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 10 monclient: _renew_subs 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd8137c10 con 0x7fbfd81319a0 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfd6ffd640 4 set_mon_vals no callback set 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 10 monclient: _renew_subs 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd81364c0 con 0x7fbfd81319a0 2024-09-15T01:51:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fd7408b8640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd720038740 0x7fd72003abf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0a0da0 tx=0x7fd72c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.806 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:33.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T01:51:33.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fa862ffd640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa850038790 0x7fa85003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7fa862ffd640 10 monclient: get_auth_request con 0x7fa850038790 auth_method 0 2024-09-15T01:51:33.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.673+0000 7f14fd68a640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14dc038740 0x7f14dc03abf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14f811ea70 tx=0x7f14e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfdef09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbfd80724a0 con 0x7fbfd81319a0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbfc401b3b0 con 0x7fbfd81319a0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfbc038740 0x7fbfbc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fbfc404ae60 con 0x7fbfd81319a0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7fbfbc03e5c0 0x7fbfbc0409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfbc0410c0 con 0x7fbfbc03e5c0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfc404d390 con 0x7fbfd81319a0 2024-09-15T01:51:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfd4ff9640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T01:51:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.674+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7fbfbc03e5c0 0x7fbfbc0409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.675+0000 7fbfdd4a6640 10 monclient: get_auth_request con 0x7fbfbc03e5c0 auth_method 0 2024-09-15T01:51:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.676+0000 7fa862ffd640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa850038790 0x7fa85003ac40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa854032b30 tx=0x7fa854032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.676+0000 7f14fe68c640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7f14dc03e540 0x7f14dc040970 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.676+0000 7fd7418ba640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7fd72003e5c0 0x7fd7200409f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.677+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== osd.3 v2:172.21.15.67:6816/31880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f14dc041040 con 0x7f14dc03e540 2024-09-15T01:51:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.677+0000 7fa8637fe640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7fa85003e690 0x7fa850040ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.677+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== osd.5 v2:172.21.15.67:6800/31882 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd7200410c0 con 0x7fd72003e5c0 2024-09-15T01:51:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.678+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== osd.0 v2:172.21.15.17:6800/31930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa850041190 con 0x7fa85003e690 2024-09-15T01:51:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.679+0000 7fbfdd4a6640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7fbfbc03e5c0 0x7fbfbc0409f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.680+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== osd.1 v2:172.21.15.17:6808/31931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbfbc0410c0 con 0x7fbfbc03e5c0 2024-09-15T01:51:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.680+0000 7fbfdcca5640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfbc038740 0x7fbfbc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.680+0000 7fbfdcca5640 10 monclient: get_auth_request con 0x7fbfbc038740 auth_method 0 2024-09-15T01:51:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.682+0000 7fbfdcca5640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfbc038740 0x7fbfbc03abf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0114a0 tx=0x7fbfcc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.703+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 --> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd73c0724a0 con 0x7fd72003e5c0 2024-09-15T01:51:33.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.703+0000 7fd7317fa640 1 -- 172.21.15.17:0/2339974698 <== osd.5 v2:172.21.15.67:6800/31882 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd73c0724a0 con 0x7fd72003e5c0 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7fd72003e5c0 msgr2=0x7fd7200409f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7fd72003e5c0 0x7fd7200409f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd720038740 msgr2=0x7fd72003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd720038740 0x7fd72003abf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0a0da0 tx=0x7fd72c032000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 10 monclient: shutdown 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 msgr2=0x7fd73c0689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 --2- 172.21.15.17:0/2339974698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd73c06eda0 0x7fd73c0689a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd73800ea20 tx=0x7fd7380090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd7418ba640 1 -- 172.21.15.17:0/2339974698 reap_dead start 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 shutdown_connections 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 >> 172.21.15.17:0/2339974698 conn(0x7fd73c0714c0 msgr2=0x7fd73c0a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 shutdown_connections 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.704+0000 7fd74331d640 1 -- 172.21.15.17:0/2339974698 wait complete. 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.705+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 --> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14f80724a0 con 0x7f14dc03e540 2024-09-15T01:51:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.705+0000 7f14edffb640 1 -- 172.21.15.17:0/217176789 <== osd.3 v2:172.21.15.67:6816/31880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f14f80724a0 con 0x7f14dc03e540 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7f14dc03e540 msgr2=0x7f14dc040970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7f14dc03e540 0x7f14dc040970 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14dc038740 msgr2=0x7f14dc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14dc038740 0x7f14dc03abf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14f811ea70 tx=0x7f14e8008040 comp rx=0 tx=0).stop 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 10 monclient: shutdown 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 msgr2=0x7f14f8140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 --2- 172.21.15.17:0/217176789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f14f809f1e0 0x7f14f8140fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009880 tx=0x7f14f4032000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 --> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa86412d590 con 0x7fa85003e690 2024-09-15T01:51:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f14fe68c640 1 -- 172.21.15.17:0/217176789 reap_dead start 2024-09-15T01:51:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 shutdown_connections 2024-09-15T01:51:33.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 >> 172.21.15.17:0/217176789 conn(0x7f14f8071270 msgr2=0x7f14f809cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 shutdown_connections 2024-09-15T01:51:33.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7f15000ef640 1 -- 172.21.15.17:0/217176789 wait complete. 2024-09-15T01:51:33.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.706+0000 7fa84affd640 1 -- 172.21.15.17:0/3087506178 <== osd.0 v2:172.21.15.17:6800/31930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa86412d590 con 0x7fa85003e690 2024-09-15T01:51:33.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 --> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbfd8136760 con 0x7fbfbc03e5c0 2024-09-15T01:51:33.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7fa85003e690 msgr2=0x7fa850040ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7fa85003e690 0x7fa850040ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfd4ff9640 1 -- 172.21.15.17:0/2478120119 <== osd.1 v2:172.21.15.17:6808/31931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbfd8136760 con 0x7fbfbc03e5c0 2024-09-15T01:51:33.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa850038790 msgr2=0x7fa85003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa850038790 0x7fa85003ac40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa854032b30 tx=0x7fa854032000 comp rx=0 tx=0).stop 2024-09-15T01:51:33.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 10 monclient: shutdown 2024-09-15T01:51:33.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 msgr2=0x7fa86409a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 --2- 172.21.15.17:0/3087506178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8640a0280 0x7fa86409a410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c4e0 tx=0x7fa858007e10 comp rx=0 tx=0).stop 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8637fe640 1 -- 172.21.15.17:0/3087506178 reap_dead start 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 shutdown_connections 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7fbfbc03e5c0 msgr2=0x7fbfbc0409f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7fbfbc03e5c0 0x7fbfbc0409f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 >> 172.21.15.17:0/3087506178 conn(0x7fa864071270 msgr2=0x7fa8640a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfbc038740 msgr2=0x7fbfbc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 shutdown_connections 2024-09-15T01:51:33.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fa8694af640 1 -- 172.21.15.17:0/3087506178 wait complete. 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfbc038740 0x7fbfbc03abf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0114a0 tx=0x7fbfcc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 10 monclient: shutdown 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 msgr2=0x7fbfd8133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.707+0000 7fbfdef09640 1 --2- 172.21.15.17:0/2478120119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd81319a0 0x7fbfd8133fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c490 tx=0x7fbfc4007e40 comp rx=0 tx=0).stop 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fbfdd4a6640 1 -- 172.21.15.17:0/2478120119 reap_dead start 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 shutdown_connections 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 >> 172.21.15.17:0/2478120119 conn(0x7fbfd80714e0 msgr2=0x7fbfd809ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 shutdown_connections 2024-09-15T01:51:33.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.708+0000 7fbfdef09640 1 -- 172.21.15.17:0/2478120119 wait complete. 2024-09-15T01:51:33.823 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-15T01:51:33.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T01:51:33.824 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:33.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T01:51:33.825 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:33.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T01:51:33.826 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-15T01:51:33.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T01:51:33.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7f8d1cadd640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7fd81759d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7fd81759d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7fd81759d640 10 monclient: monmap: 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.958221+0000 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.958221+0000 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.961 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.961 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.961 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.961 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7f8d1cadd640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.957+0000 7f8d1cadd640 10 monclient: monmap: 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.958212+0000 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.958212+0000 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:33.962 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:33.963 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 1 Processor -- start 2024-09-15T01:51:33.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 1 -- start start 2024-09-15T01:51:33.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 10 monclient: init 2024-09-15T01:51:33.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7f8d1cadd640 1 Processor -- start 2024-09-15T01:51:33.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7f8d1cadd640 1 -- start start 2024-09-15T01:51:33.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7f8d1cadd640 10 monclient: init 2024-09-15T01:51:33.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:51:33.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.958+0000 7fd81759d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8100a58b0 0x7fd810132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8100a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8100980a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 0x7f8d1809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-c con 0x7f8d18131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d180a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd810136dd0 0x7fd8101391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-a con 0x7fd810136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8100a01e0 con 0x7fd8100980a0 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8100a1580 con 0x7fd810136dd0 2024-09-15T01:51:33.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8100a5c80 con 0x7fd8100a58b0 2024-09-15T01:51:33.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd81759d640 10 monclient(hunting): authenticate will time out at 753.955258s 2024-09-15T01:51:33.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 0x7f8d18138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1809f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180a0040 con 0x7f8d180a0280 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1809bff0 con 0x7f8d18131b20 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180a14a0 con 0x7f8d1809f1e0 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d0ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d180a0280 0x7f8d18131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52392/0 (socket says 172.21.15.17:52392) 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 0x7f8d1809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d0ffff640 1 -- 172.21.15.17:0/460625713 learned_addr learned my addr 172.21.15.17:0/460625713 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1cadd640 10 monclient(hunting): authenticate will time out at 753.955602s 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd814b38640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8100980a0 0x7fd8100984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52376/0 (socket says 172.21.15.17:52376) 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd814b38640 1 -- 172.21.15.17:0/2557958901 learned_addr learned my addr 172.21.15.17:0/2557958901 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 1 --2- 172.21.15.17:0/2557958901 >> v2:172.21.15.17:3300/0 conn(0x7fd810136dd0 0x7fd8101391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): get_auth_request con 0x7fd810136dd0 auth_method 0 2024-09-15T01:51:33.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): get_auth_request con 0x7f8d18131b20 auth_method 0 2024-09-15T01:51:33.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 1 --2- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 0x7f8d18138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): get_auth_request con 0x7f8d1809f1e0 auth_method 0 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815339640 1 --2- 172.21.15.17:0/2557958901 >> v2:172.21.15.67:3300/0 conn(0x7fd8100a58b0 0x7fd810132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:33.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:33.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815339640 10 monclient(hunting): get_auth_request con 0x7fd8100a58b0 auth_method 0 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.959+0000 7fd815b3a640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 1 -- 172.21.15.17:0/2557958901 >> v2:172.21.15.67:3300/0 conn(0x7fd8100a58b0 msgr2=0x7fd810132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 1 --2- 172.21.15.17:0/2557958901 >> v2:172.21.15.67:3300/0 conn(0x7fd8100a58b0 0x7fd810132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 1 -- 172.21.15.17:0/2557958901 >> v1:172.21.15.67:6789/0 conn(0x7fd8100980a0 legacy=0x7fd8100984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 1 monclient: found mon.noname-a 2024-09-15T01:51:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7fd815b3a640 1 -- 172.21.15.17:0/2557958901 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810098bb0 con 0x7fd810136dd0 2024-09-15T01:51:33.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-15T01:51:33.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 1 -- 172.21.15.17:0/460625713 >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 msgr2=0x7f8d1809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 1 --2- 172.21.15.17:0/460625713 >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 0x7f8d1809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 1 -- 172.21.15.17:0/460625713 >> v1:172.21.15.67:6789/0 conn(0x7f8d180a0280 legacy=0x7f8d18131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:33.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 1 monclient: found mon.noname-b 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d16d9d640 1 -- 172.21.15.17:0/460625713 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18132600 con 0x7f8d1809f1e0 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.960+0000 7f8d1659c640 1 --2- 172.21.15.17:0/460625713 >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 0x7f8d1809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d16d9d640 1 -- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 msgr2=0x7f8d18138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d16d9d640 1 --2- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 0x7f8d18138e60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009e60 tx=0x7f8d10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b79438d2e9ef2f7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd815b3a640 1 -- 172.21.15.17:0/2557958901 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd810136dd0 msgr2=0x7fd8101391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:33.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd815b3a640 1 --2- 172.21.15.17:0/2557958901 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd810136dd0 0x7fd8101391b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8100a60e0 tx=0x7fd808009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32e6e5eede9fdfb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:33.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 1 -- 172.21.15.17:0/460625713 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10007760 con 0x7f8d1809f1e0 2024-09-15T01:51:33.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 1 -- 172.21.15.17:0/2557958901 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808007760 con 0x7fd810136dd0 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: dump: 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.979 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 1 -- 172.21.15.17:0/460625713 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1002d050 con 0x7f8d1809f1e0 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d1cadd640 5 monclient: authenticate success, global_id 4244 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d1cadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: dump: 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.981 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 1 -- 172.21.15.17:0/460625713 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10007af0 con 0x7f8d1809f1e0 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd81759d640 5 monclient: authenticate success, global_id 4257 2024-09-15T01:51:33.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd81759d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 1 -- 172.21.15.17:0/2557958901 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80802d050 con 0x7fd810136dd0 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd81759d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd81759d640 4 set_mon_vals no callback set 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 1 -- 172.21.15.17:0/2557958901 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808007af0 con 0x7fd810136dd0 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7fd807fff640 10 monclient: dump: 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d15d9b640 10 monclient: dump: 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:33.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d1cadd640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.963+0000 7f8d1cadd640 4 set_mon_vals no callback set 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 10 monclient: shutdown 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 -- 172.21.15.17:0/2557958901 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd810136dd0 msgr2=0x7fd8101391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 --2- 172.21.15.17:0/2557958901 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd810136dd0 0x7fd8101391b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8100a60e0 tx=0x7fd808009780 comp rx=0 tx=0).stop 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 10 monclient: shutdown 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 -- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 msgr2=0x7f8d18138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 0x7f8d18138e60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009e60 tx=0x7f8d10009780 comp rx=0 tx=0).stop 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 -- 172.21.15.17:0/460625713 shutdown_connections 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 -- 172.21.15.17:0/2557958901 shutdown_connections 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 --2- 172.21.15.17:0/2557958901 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd810136dd0 0x7fd8101391b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 --2- 172.21.15.17:0/2557958901 >> v2:172.21.15.67:3300/0 conn(0x7fd8100a58b0 0x7fd810132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 -- 172.21.15.17:0/2557958901 >> 172.21.15.17:0/2557958901 conn(0x7fd810070ed0 msgr2=0x7fd8100a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/460625713 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d1809f1e0 0x7f8d18138e60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/460625713 >> v2:172.21.15.67:3300/0 conn(0x7f8d18131b20 0x7f8d1809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 -- 172.21.15.17:0/460625713 >> 172.21.15.17:0/460625713 conn(0x7f8d18071270 msgr2=0x7f8d180a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 -- 172.21.15.17:0/460625713 shutdown_connections 2024-09-15T01:51:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 -- 172.21.15.17:0/2557958901 shutdown_connections 2024-09-15T01:51:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7f8d1cadd640 1 -- 172.21.15.17:0/460625713 wait complete. 2024-09-15T01:51:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.964+0000 7fd81759d640 1 -- 172.21.15.17:0/2557958901 wait complete. 2024-09-15T01:51:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.965+0000 7f8d1cadd640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.965+0000 7fd81759d640 10 monclient: build_initial_monmap 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.965+0000 7fd81759d640 10 monclient: monmap: 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.965+0000 7f8d1cadd640 10 monclient: monmap: 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:33.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7f8d1cadd640 1 Processor -- start 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7fd81759d640 1 Processor -- start 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7f8d1cadd640 1 -- start start 2024-09-15T01:51:33.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7f8d1cadd640 10 monclient: init 2024-09-15T01:51:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7fd81759d640 1 -- start start 2024-09-15T01:51:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.966+0000 7fd81759d640 10 monclient: init 2024-09-15T01:51:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7f8d1cadd640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7f8d1cadd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 0x7fd810132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7f8d1cadd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 0x7f8d1809a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8100980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-a con 0x7fd8100a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 0x7fd8101337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd814b38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd815339640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 0x7fd810132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd815339640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 0x7fd810132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:58872/0 (socket says 172.21.15.17:58872) 2024-09-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7f8d1659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 0x7f8d1809a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd814b38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38982/0 (socket says 172.21.15.17:38982) 2024-09-15T01:51:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd815b3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 0x7fd8101337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd815b3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 0x7fd8101337d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59638/0 (socket says 172.21.15.17:59638) 2024-09-15T01:51:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient(hunting): picked mon.noname-c con 0x7fd810136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd815339640 1 -- 172.21.15.17:0/2661159285 learned_addr learned my addr 172.21.15.17:0/2661159285 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810136140 con 0x7fd8100a58b0 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.967+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810098dc0 con 0x7fd8100980a0 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd81759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100a3070 con 0x7fd810136dd0 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd81759d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd81759d640 10 monclient(hunting): authenticate will time out at 753.964106s 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): get_auth_request con 0x7fd8100980a0 auth_method 0 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): get_auth_request con 0x7fd8100a58b0 auth_method 0 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815b3a640 10 monclient(hunting): get_auth_request con 0x7fd810136dd0 auth_method 0 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1809f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-b con 0x7f8d180a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 0x7f8d1812d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 0x7f8d1809a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38974/0 (socket says 172.21.15.17:38974) 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:58884/0 (socket says 172.21.15.17:58884) 2024-09-15T01:51:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 1 -- 172.21.15.17:0/4280889798 learned_addr learned my addr 172.21.15.17:0/4280889798 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): picked mon.noname-c con 0x7f8d18131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1812ed10 con 0x7f8d1809f1e0 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1812f8a0 con 0x7f8d180a0280 2024-09-15T01:51:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18097850 con 0x7f8d18131b20 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1cadd640 10 monclient(hunting): authenticate will time out at 753.964584s 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): get_auth_request con 0x7f8d1809f1e0 auth_method 0 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): get_auth_request con 0x7f8d180a0280 auth_method 0 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 0x7f8d1812d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): get_auth_request con 0x7f8d18131b20 auth_method 0 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd815339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-15T01:51:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 -- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 msgr2=0x7fd8101337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 0x7fd8101337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 -- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 msgr2=0x7fd810132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7f8d16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-09-15T01:51:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 -- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 msgr2=0x7f8d1812d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 0x7f8d1812d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 -- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 msgr2=0x7f8d1809a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 0x7f8d1809a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 0x7fd810132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 monclient: found mon.noname-a 2024-09-15T01:51:34.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.968+0000 7fd814b38640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808009a20 con 0x7fd8100a58b0 2024-09-15T01:51:34.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 monclient: found mon.noname-b 2024-09-15T01:51:34.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d0ffff640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009390 con 0x7f8d180a0280 2024-09-15T01:51:34.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.969+0000 7f8d16d9d640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 0x7f8d1812d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:34.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.970+0000 7fd814b38640 1 -- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 msgr2=0x7fd810133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.970+0000 7fd814b38640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b490 tx=0x7fd800006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800011840 con 0x7fd8100a58b0 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: dump: 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.012 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8000119c0 con 0x7fd8100a58b0 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 5 monclient: authenticate success, global_id 4260 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 10 monclient: _renew_subs 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd810135ed0 con 0x7fd8100a58b0 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800011d80 con 0x7fd8100a58b0 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8057fa640 10 monclient: dump: 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.014 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 10 monclient: _renew_subs 2024-09-15T01:51:34.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd8077fe640 4 set_mon_vals no callback set 2024-09-15T01:51:34.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd810134a10 con 0x7fd8100a58b0 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0ffff640 1 -- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 msgr2=0x7f8d1812efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0ffff640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c4f0 tx=0x7f8d00007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00005cb0 con 0x7f8d180a0280 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: dump: 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d00005e30 con 0x7f8d180a0280 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00007320 con 0x7f8d180a0280 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d0e7fc640 10 monclient: dump: 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 5 monclient: authenticate success, global_id 4270 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.971+0000 7f8d1559a640 4 set_mon_vals no callback set 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 10 monclient: _renew_subs 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d10009a20 con 0x7f8d180a0280 2024-09-15T01:51:34.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd800010590 con 0x7fd8100a58b0 2024-09-15T01:51:34.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd8057fa640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 0x7fd7f403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd80004b600 con 0x7fd8100a58b0 2024-09-15T01:51:34.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 10 monclient: _renew_subs 2024-09-15T01:51:34.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.972+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d18097570 con 0x7f8d180a0280 2024-09-15T01:51:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7f8d1cadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7f8d1cadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7f8d1cadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d18097b90 con 0x7f8d180a0280 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd81759d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd815339640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 0x7fd7f403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd81759d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd81759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd810135b80 con 0x7fd8100a58b0 2024-09-15T01:51:34.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.973+0000 7fd815339640 10 monclient: get_auth_request con 0x7fd7f4038790 auth_method 0 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.974+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d0001b3b0 con 0x7f8d180a0280 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.974+0000 7f8d0e7fc640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 0x7f8cf803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.974+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8d0004ad80 con 0x7f8d180a0280 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.974+0000 7f8d1659c640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 0x7f8cf803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.974+0000 7f8d1659c640 10 monclient: get_auth_request con 0x7f8cf8038790 auth_method 0 2024-09-15T01:51:34.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.976+0000 7fd815339640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 0x7fd7f403ac40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd80c007c70 tx=0x7fd80c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.977+0000 7f8d1659c640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 0x7f8cf803ac40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d080099a0 tx=0x7f8d08016040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.978+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd80004b030 con 0x7fd8100a58b0 2024-09-15T01:51:34.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.978+0000 7fd8057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.978+0000 7fd8057fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.979+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d0001a360 con 0x7f8d180a0280 2024-09-15T01:51:34.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.979+0000 7f8d0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.979+0000 7f8d0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 10 monclient: get_monmap_and_config 2024-09-15T01:51:34.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 10 monclient: monmap: 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.998049+0000 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.998049+0000 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 1 Processor -- start 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 1 -- start start 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.997+0000 7f0c0128b640 10 monclient: init 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f8e2719d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:34.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f8e2719d640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f8e2719d640 10 monclient: monmap: 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:33.999210+0000 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:33.999210+0000 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:34.029 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:34.030 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bfc131b20 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bfc0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bfc132430 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.998+0000 7f0c0128b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc0a0040 con 0x7f0bfc132430 2024-09-15T01:51:34.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0c0128b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc09bff0 con 0x7f0bfc131b20 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0c0128b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0c0128b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc0a14a0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0c0128b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0c0128b640 10 monclient(hunting): authenticate will time out at 753.995021s 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0bfc131b20 0x7f0bfc09eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52398/0 (socket says 172.21.15.17:52398) 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfaffd640 1 -- 172.21.15.17:0/315538880 learned_addr learned my addr 172.21.15.17:0/315538880 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 1 --2- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): get_auth_request con 0x7f0bfc0a0280 auth_method 0 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f8e2719d640 1 Processor -- start 2024-09-15T01:51:34.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f8e2719d640 1 -- start start 2024-09-15T01:51:34.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f8e2719d640 10 monclient: init 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 1 -- 172.21.15.17:0/315538880 >> v1:172.21.15.67:6789/0 conn(0x7f0bfc131b20 legacy=0x7f0bfc09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 1 -- 172.21.15.17:0/315538880 >> v1:172.21.15.17:6789/0 conn(0x7f0bfc132430 legacy=0x7f0bfc138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:33.999+0000 7f0bfa7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f0bfa7fc640 1 monclient: found mon.noname-b 2024-09-15T01:51:34.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f0bfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f0bfa7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f0bfa7fc640 1 -- 172.21.15.17:0/315538880 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc09f3b0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e20131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e200a0280 0x7f8e20131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e200a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e20132430 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200a0040 con 0x7f8e20131b20 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2009bff0 con 0x7f8e20132430 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e200a14a0 con 0x7f8e200a0280 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e2719d640 10 monclient(hunting): authenticate will time out at 753.996292s 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e24f39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8e20131b20 0x7f8e2009eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33670/0 (socket says 172.21.15.17:33670) 2024-09-15T01:51:34.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7f8e24f39640 1 -- 172.21.15.17:0/669117089 learned_addr learned my addr 172.21.15.17:0/669117089 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.000+0000 7fbe46434640 10 monclient: get_monmap_and_config 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7fbe46434640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7fbe46434640 10 monclient: monmap: 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:34.001853+0000 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:34.001853+0000 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:34.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:34.039 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:34.039 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:34.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 1 --2- 172.21.15.17:0/669117089 >> v2:172.21.15.67:3300/0 conn(0x7f8e200a0280 0x7f8e20131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): get_auth_request con 0x7f8e200a0280 auth_method 0 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 1 -- 172.21.15.17:0/669117089 >> v1:172.21.15.67:6789/0 conn(0x7f8e20132430 legacy=0x7f8e20138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 1 -- 172.21.15.17:0/669117089 >> v1:172.21.15.17:6789/0 conn(0x7f8e20131b20 legacy=0x7f8e2009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 1 monclient: found mon.noname-c 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.001+0000 7f8e1ffff640 1 -- 172.21.15.17:0/669117089 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2009f3b0 con 0x7f8e200a0280 2024-09-15T01:51:34.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.002+0000 7fbe46434640 1 Processor -- start 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.002+0000 7fbe46434640 1 -- start start 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.002+0000 7fbe46434640 10 monclient: init 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe40131b20 0x7fbe4009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-c con 0x7fbe40131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe400a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 0x7fbe40138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400a0040 con 0x7fbe400a0280 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4009bff0 con 0x7fbe40131b20 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400a14a0 con 0x7fbe4009f1e0 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe46434640 10 monclient(hunting): authenticate will time out at 753.999536s 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbe400a0280 0x7fbe40131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33682/0 (socket says 172.21.15.17:33682) 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 0x7fbe40138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe40131b20 0x7fbe4009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3f7fe640 1 -- 172.21.15.17:0/1588514421 learned_addr learned my addr 172.21.15.17:0/1588514421 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): get_auth_request con 0x7fbe4009f1e0 auth_method 0 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): get_auth_request con 0x7fbe40131b20 auth_method 0 2024-09-15T01:51:34.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bfa7fc640 1 -- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 msgr2=0x7f0bfc131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bfa7fc640 1 --2- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc131ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009880 tx=0x7f0bf00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed89a9de9bbbbb9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe449d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-09-15T01:51:34.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 1 -- 172.21.15.17:0/315538880 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0032440 con 0x7f0bfc0a0280 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 10 monclient: dump: 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.047 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.003+0000 7f0bf9ffb640 1 -- 172.21.15.17:0/315538880 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00325a0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0bf9ffb640 1 -- 172.21.15.17:0/315538880 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0032940 con 0x7f0bfc0a0280 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0c0128b640 5 monclient: authenticate success, global_id 4253 2024-09-15T01:51:34.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 1 -- 172.21.15.17:0/1588514421 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 msgr2=0x7fbe40138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0bf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0bf9ffb640 10 monclient: dump: 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.049 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0c0128b640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:34.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0c0128b640 4 set_mon_vals no callback set 2024-09-15T01:51:34.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 0x7fbe40138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 1 -- 172.21.15.17:0/1588514421 >> v1:172.21.15.17:6789/0 conn(0x7fbe400a0280 legacy=0x7fbe40131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 1 monclient: found mon.noname-c 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe3ffff640 1 -- 172.21.15.17:0/1588514421 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe40132600 con 0x7fbe40131b20 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7fbe449d1640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 0x7fbe40138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f8e1ffff640 1 -- 172.21.15.17:0/669117089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e200a0280 msgr2=0x7f8e20131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0c0128b640 10 monclient: shutdown 2024-09-15T01:51:34.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f0c0128b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 -- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 msgr2=0x7f0bfc131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.004+0000 7f8e1ffff640 1 --2- 172.21.15.17:0/669117089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e200a0280 0x7f8e20131ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009850 tx=0x7f8e100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b1c09960f04e198 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 --2- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc131ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009880 tx=0x7f0bf00090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 -- 172.21.15.17:0/315538880 shutdown_connections 2024-09-15T01:51:34.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 --2- 172.21.15.17:0/315538880 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc131ef0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 -- 172.21.15.17:0/315538880 >> 172.21.15.17:0/315538880 conn(0x7f0bfc071270 msgr2=0x7f0bfc0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 -- 172.21.15.17:0/315538880 shutdown_connections 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 1 -- 172.21.15.17:0/669117089 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10032440 con 0x7f8e200a0280 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: dump: 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e2719d640 5 monclient: authenticate success, global_id 4273 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e2719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 1 -- 172.21.15.17:0/669117089 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e100325a0 con 0x7f8e200a0280 2024-09-15T01:51:34.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f0c0128b640 1 -- 172.21.15.17:0/315538880 wait complete. 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e2719d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e2719d640 4 set_mon_vals no callback set 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 1 -- 172.21.15.17:0/669117089 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10032940 con 0x7f8e200a0280 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8e1f7fe640 10 monclient: dump: 2024-09-15T01:51:34.056 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.057 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8ca4a0f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8ca4a0f640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.005+0000 7f8ca4a0f640 10 monclient: monmap: 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:34.006417+0000 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:34.006417+0000 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:34.058 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8ca4a0f640 1 Processor -- start 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 10 monclient: shutdown 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8ca4a0f640 1 -- start start 2024-09-15T01:51:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8ca4a0f640 10 monclient: init 2024-09-15T01:51:34.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 -- 172.21.15.17:0/669117089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e200a0280 msgr2=0x7f8e20131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 --2- 172.21.15.17:0/669117089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e200a0280 0x7f8e20131ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009850 tx=0x7f8e100090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 -- 172.21.15.17:0/669117089 shutdown_connections 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 --2- 172.21.15.17:0/669117089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e200a0280 0x7f8e20131ef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 -- 172.21.15.17:0/669117089 >> 172.21.15.17:0/669117089 conn(0x7f8e20071270 msgr2=0x7f8e200a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 -- 172.21.15.17:0/669117089 shutdown_connections 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f8e2719d640 1 -- 172.21.15.17:0/669117089 wait complete. 2024-09-15T01:51:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f0c0128b640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.006+0000 7f0c0128b640 10 monclient: monmap: 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:34.062 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3ffff640 1 -- 172.21.15.17:0/1588514421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe40131b20 msgr2=0x7fbe4009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3ffff640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe40131b20 0x7fbe4009eca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009850 tx=0x7fbe300090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af06440e4ca5d86a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ca0131b20 0x7f8ca009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ca0131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ca00a0280 0x7f8ca0131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8e2719d640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8e2719d640 10 monclient: monmap: 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:34.064 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca00a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:34.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f0c0128b640 1 Processor -- start 2024-09-15T01:51:34.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ca00a0280 0x7f8ca0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ca0131b20 0x7f8ca009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ca00a0280 0x7f8ca0131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38996/0 (socket says 172.21.15.17:38996) 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 1 -- 172.21.15.17:0/2668472973 learned_addr learned my addr 172.21.15.17:0/2668472973 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ca009f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f0c0128b640 1 -- start start 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca00a0040 con 0x7f8ca009f1e0 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca009bff0 con 0x7f8ca00a0280 2024-09-15T01:51:34.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f0c0128b640 10 monclient: init 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca00a14a0 con 0x7f8ca0131b20 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8ca4a0f640 10 monclient(hunting): authenticate will time out at 754.003701s 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): get_auth_request con 0x7f8ca00a0280 auth_method 0 2024-09-15T01:51:34.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9e59c640 10 monclient(hunting): get_auth_request con 0x7f8ca0131b20 auth_method 0 2024-09-15T01:51:34.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8e2719d640 1 Processor -- start 2024-09-15T01:51:34.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 1 -- 172.21.15.17:0/1588514421 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3002adf0 con 0x7fbe40131b20 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: dump: 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.073 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 1 -- 172.21.15.17:0/1588514421 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30032440 con 0x7fbe40131b20 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 1 -- 172.21.15.17:0/1588514421 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300327e0 con 0x7fbe40131b20 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7fbe3effd640 10 monclient: dump: 2024-09-15T01:51:34.074 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7fbe46434640 5 monclient: authenticate success, global_id 4276 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7fbe46434640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:34.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7fbe46434640 4 set_mon_vals no callback set 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0c0128b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.007+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 1 -- 172.21.15.17:0/2668472973 >> v2:172.21.15.67:3300/0 conn(0x7f8ca0131b20 msgr2=0x7f8ca009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 1 --2- 172.21.15.17:0/2668472973 >> v2:172.21.15.67:3300/0 conn(0x7f8ca0131b20 0x7f8ca009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 1 -- 172.21.15.17:0/2668472973 >> v1:172.21.15.17:6789/0 conn(0x7f8ca009f1e0 legacy=0x7f8ca0068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 1 monclient: found mon.noname-a 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8c9dd9b640 1 -- 172.21.15.17:0/2668472973 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0132430 con 0x7f8ca00a0280 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0c0128b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:34.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0c0128b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 -- start start 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient: init 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e200a0280 0x7f8e20145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e200a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e20131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e20132430 0x7f8e20097c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0bfaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0bfaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59664/0 (socket says 172.21.15.17:59664) 2024-09-15T01:51:34.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7fbe46434640 10 monclient: shutdown 2024-09-15T01:51:34.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0bfaffd640 1 -- 172.21.15.17:0/664502989 learned_addr learned my addr 172.21.15.17:0/664502989 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-c con 0x7f0bfc0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 -- 172.21.15.17:0/1588514421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe40131b20 msgr2=0x7fbe4009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bfc131b20 0x7f0bfc132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe40131b20 0x7fbe4009eca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009850 tx=0x7fbe300090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e20132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200a09b0 con 0x7f8e200a0280 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200a4a10 con 0x7f8e20132430 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2009b450 con 0x7f8e20131b20 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-a con 0x7f0bfc131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.008+0000 7f8e2719d640 10 monclient(hunting): authenticate will time out at 754.004829s 2024-09-15T01:51:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 -- 172.21.15.17:0/1588514421 shutdown_connections 2024-09-15T01:51:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bfc132430 0x7f0bfc132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4009f1e0 0x7fbe40138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 --2- 172.21.15.17:0/1588514421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe40131b20 0x7fbe4009eca0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59668/0 (socket says 172.21.15.17:59668) 2024-09-15T01:51:34.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 -- 172.21.15.17:0/2894735761 learned_addr learned my addr 172.21.15.17:0/2894735761 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 -- 172.21.15.17:0/1588514421 >> 172.21.15.17:0/1588514421 conn(0x7fbe40071270 msgr2=0x7fbe400a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bfc132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): get_auth_request con 0x7f8e20131b20 auth_method 0 2024-09-15T01:51:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc136100 con 0x7f0bfc131b20 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 -- 172.21.15.17:0/1588514421 shutdown_connections 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e20132430 0x7f8e20097c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): get_auth_request con 0x7f8e20132430 auth_method 0 2024-09-15T01:51:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bfc132430 0x7f0bfc132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7fbe46434640 1 -- 172.21.15.17:0/1588514421 wait complete. 2024-09-15T01:51:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc0a4a30 con 0x7f0bfc132430 2024-09-15T01:51:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc136830 con 0x7f0bfc0a0280 2024-09-15T01:51:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0c0128b640 10 monclient(hunting): authenticate will time out at 754.005549s 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e2573a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): get_auth_request con 0x7f0bfc0a0280 auth_method 0 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f0bfc132430 auth_method 0 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 -- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e20132430 msgr2=0x7f8e20097c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e20132430 0x7f8e20097c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 -- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e200a0280 msgr2=0x7f8e20145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e200a0280 0x7f8e20145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 monclient: found mon.noname-c 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f8e1ffff640 1 -- 172.21.15.17:0/2894735761 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10009390 con 0x7f8e20131b20 2024-09-15T01:51:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-15T01:51:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 -- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bfc132430 msgr2=0x7f0bfc132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bfc132430 0x7f0bfc132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 -- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bfc131b20 msgr2=0x7f0bfc132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bfc131b20 0x7f0bfc132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 monclient: found mon.noname-c 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.010+0000 7fbe46434640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.010+0000 7fbe46434640 10 monclient: monmap: 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:34.094 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.009+0000 7f0bfaffd640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0009390 con 0x7f0bfc0a0280 2024-09-15T01:51:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.010+0000 7fbe46434640 1 Processor -- start 2024-09-15T01:51:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 -- start start 2024-09-15T01:51:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient: init 2024-09-15T01:51:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7f8c9dd9b640 1 -- 172.21.15.17:0/2668472973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 msgr2=0x7f8ca0131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7f8c9dd9b640 1 --2- 172.21.15.17:0/2668472973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca0131ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ca009f710 tx=0x7f8c940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4fce967bc8e76e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39022/0 (socket says 172.21.15.17:39022) 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe400a0280 0x7fbe40141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 1 -- 172.21.15.17:0/767789243 learned_addr learned my addr 172.21.15.17:0/767789243 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-b con 0x7fbe400a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe40131b20 0x7fbe40132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): picked mon.noname-c con 0x7fbe40131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40136520 con 0x7fbe4009f1e0 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400a3cf0 con 0x7fbe400a0280 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400a09b0 con 0x7fbe40131b20 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe46434640 10 monclient(hunting): authenticate will time out at 754.007974s 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 10 monclient(hunting): get_auth_request con 0x7fbe4009f1e0 auth_method 0 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7fbe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7f8c9d59a640 1 -- 172.21.15.17:0/2668472973 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94032440 con 0x7f8ca00a0280 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.011+0000 7f8c9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: dump: 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe400a0280 0x7fbe40141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 5 monclient: authenticate success, global_id 4266 2024-09-15T01:51:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 1 -- 172.21.15.17:0/2668472973 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c940325a0 con 0x7f8ca00a0280 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 1 -- 172.21.15.17:0/2668472973 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94032940 con 0x7f8ca00a0280 2024-09-15T01:51:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8c9d59a640 10 monclient: dump: 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 4 set_mon_vals no callback set 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): get_auth_request con 0x7fbe400a0280 auth_method 0 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1ffff640 1 -- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 msgr2=0x7f8e201460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1ffff640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009850 tx=0x7f8e10032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-15T01:51:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0bfaffd640 1 -- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 msgr2=0x7f0bfc09b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0bfaffd640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00c640 tx=0x7f0bec004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 -- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe40131b20 msgr2=0x7fbe40132950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe40131b20 0x7fbe40132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 -- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe400a0280 msgr2=0x7fbe40141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe400a0280 0x7fbe40141f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 monclient: found mon.noname-a 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7fbe3ffff640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38009a20 con 0x7fbe4009f1e0 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec005cb0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 10 monclient: dump: 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0c0128b640 5 monclient: authenticate success, global_id 4262 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0c0128b640 10 monclient: _renew_subs 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10031a60 con 0x7f8e20131b20 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 10 monclient: dump: 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10031bc0 con 0x7f8e20131b20 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 5 monclient: authenticate success, global_id 4259 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 10 monclient: _renew_subs 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0c0128b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc134ab0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e20133120 con 0x7f8e20131b20 2024-09-15T01:51:34.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 10 monclient: shutdown 2024-09-15T01:51:34.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec005e30 con 0x7f0bfc0a0280 2024-09-15T01:51:34.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f0c0128b640 10 monclient: _renew_subs 2024-09-15T01:51:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 msgr2=0x7f8ca0131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0c0128b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/2668472973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca0131ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ca009f710 tx=0x7f8c940091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc133ca0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0be2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec007340 con 0x7f0bfc0a0280 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0bf97fa640 4 set_mon_vals no callback set 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f0be2ffd640 10 monclient: dump: 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 10 monclient: _renew_subs 2024-09-15T01:51:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2009b170 con 0x7f8e20131b20 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.012+0000 7f8e1effd640 4 set_mon_vals no callback set 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1003c410 con 0x7f8e20131b20 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f8e1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f8e1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.013+0000 7f8e1cff9640 10 monclient: dump: 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 shutdown_connections 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/2668472973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca0131ef0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/2668472973 >> v2:172.21.15.67:3300/0 conn(0x7f8ca0131b20 0x7f8ca009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 >> 172.21.15.17:0/2668472973 conn(0x7f8ca0071270 msgr2=0x7f8ca00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0bec01b400 con 0x7f0bfc0a0280 2024-09-15T01:51:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8e2719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8e2719d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8e2719d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2009a270 con 0x7f8e20131b20 2024-09-15T01:51:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 shutdown_connections 2024-09-15T01:51:34.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/2668472973 wait complete. 2024-09-15T01:51:34.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7fbe3ffff640 1 -- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 msgr2=0x7fbe401419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7fbe3ffff640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098b0 tx=0x7fbe30032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.014+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8e10039070 con 0x7f8e20131b20 2024-09-15T01:51:34.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30031a10 con 0x7fbe4009f1e0 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 10 monclient: dump: 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.120 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 5 monclient: authenticate success, global_id 4269 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 10 monclient: _renew_subs 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe30009390 con 0x7fbe4009f1e0 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 10 monclient: _renew_subs 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe40135ca0 con 0x7fbe4009f1e0 2024-09-15T01:51:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0be2ffd640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8e1cff9640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 0x7f8e0403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f0bec04b4a0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8e1006c4b0 con 0x7f8e20131b20 2024-09-15T01:51:34.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0bfa7fc640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0c0128b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0c0128b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0c0128b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bfc135eb0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f0bfa7fc640 10 monclient: get_auth_request con 0x7f0be8038790 auth_method 0 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8e24f39640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 0x7f8e0403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8e24f39640 10 monclient: get_auth_request con 0x7f8e04038790 auth_method 0 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8ca4a0f640 10 monclient: build_initial_monmap 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8ca4a0f640 10 monclient: monmap: 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:34.124 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.015+0000 7f8ca4a0f640 1 Processor -- start 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 -- start start 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient: init 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ca009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:34.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca012f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca00a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca0131b20 0x7f8ca0140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f8ca0131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca009aec0 con 0x7f8ca00a0280 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca011c0a0 con 0x7f8ca0131b20 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00a09b0 con 0x7f8ca009f1e0 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8ca4a0f640 10 monclient(hunting): authenticate will time out at 754.012784s 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59692/0 (socket says 172.21.15.17:59692) 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 1 -- 172.21.15.17:0/4292328617 learned_addr learned my addr 172.21.15.17:0/4292328617 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 10 monclient(hunting): get_auth_request con 0x7f8ca009f1e0 auth_method 0 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.016+0000 7f8c9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca0131b20 0x7f8ca0140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): get_auth_request con 0x7f8ca0131b20 auth_method 0 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca012f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): get_auth_request con 0x7f8ca00a0280 auth_method 0 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:34.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 -- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca0131b20 msgr2=0x7f8ca0140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca0131b20 0x7f8ca0140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 -- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 msgr2=0x7f8ca012f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca012f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 monclient: found mon.noname-c 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7f8c9e59c640 1 -- 172.21.15.17:0/4292328617 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94009450 con 0x7f8ca009f1e0 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30031d80 con 0x7fbe4009f1e0 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3003c400 con 0x7fbe4009f1e0 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 10 monclient: dump: 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe3e7fc640 4 set_mon_vals no callback set 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe3003c5a0 con 0x7fbe4009f1e0 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 0x7fbe2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.017+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fbe3006c050 con 0x7fbe4009f1e0 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe46434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe3f7fe640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 0x7fbe2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7f8e24f39640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 0x7f8e0403ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005a50 tx=0x7f8e18004fb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe46434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe46434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe20005010 con 0x7fbe4009f1e0 2024-09-15T01:51:34.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7fbe3f7fe640 10 monclient: get_auth_request con 0x7fbe2c038740 auth_method 0 2024-09-15T01:51:34.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.018+0000 7f0bfa7fc640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 0x7f0be803ac40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0031000 tx=0x7f0bf0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.019+0000 7f8c9e59c640 1 -- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 msgr2=0x7f8ca012ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:34.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.019+0000 7f8c9e59c640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000d890 tx=0x7f8c90008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.021+0000 7fbe3f7fe640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 0x7fbe2c03abf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe34007fd0 tx=0x7fbe34007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.021+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8e1003c880 con 0x7f8e20131b20 2024-09-15T01:51:34.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.021+0000 7f8e1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.021+0000 7f8e1cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.024+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbe30037080 con 0x7fbe4009f1e0 2024-09-15T01:51:34.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.024+0000 7fbe27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.024+0000 7fbe27fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9000de20 con 0x7f8ca009f1e0 2024-09-15T01:51:34.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 10 monclient: dump: 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8ca4a0f640 5 monclient: authenticate success, global_id 4265 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8ca4a0f640 10 monclient: _renew_subs 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9000bae0 con 0x7f8ca009f1e0 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8ca4a0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0bec01ab40 con 0x7f0bfc0a0280 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f0be2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f0be2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.025+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca00a51c0 con 0x7f8ca009f1e0 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8ca4a0f640 10 monclient: _renew_subs 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8ca4a0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca0141440 con 0x7f8ca009f1e0 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9000be80 con 0x7f8ca009f1e0 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c8e7fc640 10 monclient: dump: 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.026+0000 7f8c9cd99640 4 set_mon_vals no callback set 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.028+0000 7f8ca4a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.028+0000 7f8ca4a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.028+0000 7f8ca4a0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.028+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84005010 con 0x7f8ca009f1e0 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.028+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8c90014640 con 0x7f8ca009f1e0 2024-09-15T01:51:34.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.029+0000 7f8c8e7fc640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 0x7f8c8003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.029+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8c9004bf90 con 0x7f8ca009f1e0 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.029+0000 7f8c9dd9b640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 0x7f8c8003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.029+0000 7f8c9dd9b640 10 monclient: get_auth_request con 0x7f8c80038740 auth_method 0 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.032+0000 7f8c9dd9b640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 0x7f8c8003abf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c94031000 tx=0x7f8c94032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.034+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8c90011db0 con 0x7f8ca009f1e0 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.034+0000 7f8c8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:34.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.034+0000 7f8c8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:34.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e2719d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:34.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e2719d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:34.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e2719d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8e2009a270 con 0x7f8e20131b20 2024-09-15T01:51:34.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e1cff9640 1 -- 172.21.15.17:0/2894735761 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e1004a020 con 0x7f8e20131b20 2024-09-15T01:51:34.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:34.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.191+0000 7f8e1cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.194 INFO:teuthology.orchestra.run.smithi017.stdout:47244640258 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 msgr2=0x7f8e0403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 0x7f8e0403ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005a50 tx=0x7f8e18004fb0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 10 monclient: shutdown 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 msgr2=0x7f8e201460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009850 tx=0x7f8e10032000 comp rx=0 tx=0).stop 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 shutdown_connections 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e04038790 0x7f8e0403ac40 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005a50 tx=0x7f8e18004fb0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e20132430 0x7f8e20097c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e20131b20 0x7f8e201460c0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009850 tx=0x7f8e10032000 comp rx=0 tx=0).stop 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 --2- 172.21.15.17:0/2894735761 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e200a0280 0x7f8e20145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 >> 172.21.15.17:0/2894735761 conn(0x7f8e20071270 msgr2=0x7f8e20130660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 shutdown_connections 2024-09-15T01:51:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.194+0000 7f8e2719d640 1 -- 172.21.15.17:0/2894735761 wait complete. 2024-09-15T01:51:34.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8ca4a0f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:34.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8ca4a0f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:34.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8ca4a0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8c840052c0 con 0x7f8ca009f1e0 2024-09-15T01:51:34.203 INFO:teuthology.orchestra.run.smithi017.stdout:47244640258 2024-09-15T01:51:34.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8c8e7fc640 1 -- 172.21.15.17:0/4292328617 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c90013e40 con 0x7f8ca009f1e0 2024-09-15T01:51:34.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8c8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:34.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.200+0000 7f8c8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 msgr2=0x7f8c8003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 0x7f8c8003abf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c94031000 tx=0x7f8c94032000 comp rx=0 tx=0).stop 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 10 monclient: shutdown 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 msgr2=0x7f8ca012ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000d890 tx=0x7f8c90008040 comp rx=0 tx=0).stop 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 shutdown_connections 2024-09-15T01:51:34.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c80038740 0x7f8c8003abf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8ca00a0280 0x7f8ca012f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca0131b20 0x7f8ca0140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.202+0000 7f8ca4a0f640 1 --2- 172.21.15.17:0/4292328617 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ca009f1e0 0x7f8ca012ef80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.203+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 >> 172.21.15.17:0/4292328617 conn(0x7f8ca0071270 msgr2=0x7f8ca009e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.203+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 shutdown_connections 2024-09-15T01:51:34.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.203+0000 7f8ca4a0f640 1 -- 172.21.15.17:0/4292328617 wait complete. 2024-09-15T01:51:34.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-15T01:51:34.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.277+0000 7fd81759d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.277+0000 7fd81759d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.277+0000 7fd81759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.277+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd810131940 con 0x7fd8100a58b0 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.278+0000 7fd8057fa640 1 -- 172.21.15.17:0/2661159285 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd80004b020 con 0x7fd8100a58b0 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.278+0000 7fd8057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.278+0000 7fd8057fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.280 INFO:teuthology.orchestra.run.smithi017.stdout:47244640258 2024-09-15T01:51:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 msgr2=0x7fd7f403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 0x7fd7f403ac40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd80c007c70 tx=0x7fd80c008040 comp rx=0 tx=0).stop 2024-09-15T01:51:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 10 monclient: shutdown 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 msgr2=0x7fd810133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b490 tx=0x7fd800006f50 comp rx=0 tx=0).stop 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 shutdown_connections 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7f4038790 0x7fd7f403ac40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd8100a58b0 0x7fd810133290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd810136dd0 0x7fd8101337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 --2- 172.21.15.17:0/2661159285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8100980a0 0x7fd810132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 >> 172.21.15.17:0/2661159285 conn(0x7fd810070ed0 msgr2=0x7fd810068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 shutdown_connections 2024-09-15T01:51:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.280+0000 7fd81759d640 1 -- 172.21.15.17:0/2661159285 wait complete. 2024-09-15T01:51:34.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d1cadd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:34.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d1cadd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:34.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d1cadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:34.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8d180724a0 con 0x7f8d180a0280 2024-09-15T01:51:34.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d0e7fc640 1 -- 172.21.15.17:0/4280889798 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d00017020 con 0x7f8d180a0280 2024-09-15T01:51:34.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:34.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.281+0000 7f8d0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.285 INFO:teuthology.orchestra.run.smithi017.stdout:47244640257 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 msgr2=0x7f8cf803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 0x7f8cf803ac40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d080099a0 tx=0x7f8d08016040 comp rx=0 tx=0).stop 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 10 monclient: shutdown 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 msgr2=0x7f8d1812efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c4f0 tx=0x7f8d00007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 shutdown_connections 2024-09-15T01:51:34.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8cf8038790 0x7f8cf803ac40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d18131b20 0x7f8d1812d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d180a0280 0x7f8d1812efe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 --2- 172.21.15.17:0/4280889798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d1809f1e0 0x7f8d1809a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.283+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 >> 172.21.15.17:0/4280889798 conn(0x7f8d18071270 msgr2=0x7f8d1812aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.284+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 shutdown_connections 2024-09-15T01:51:34.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.284+0000 7f8d1cadd640 1 -- 172.21.15.17:0/4280889798 wait complete. 2024-09-15T01:51:34.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-15T01:51:34.296 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.1 2024-09-15T01:51:34.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe46434640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:34.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe46434640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe46434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbe20005590 con 0x7fbe4009f1e0 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe27fff640 1 -- 172.21.15.17:0/767789243 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbe3004bae0 con 0x7fbe4009f1e0 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.312+0000 7fbe27fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.315 INFO:teuthology.orchestra.run.smithi017.stdout:47244640258 2024-09-15T01:51:34.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 msgr2=0x7fbe2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 0x7fbe2c03abf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe34007fd0 tx=0x7fbe34007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:34.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 10 monclient: shutdown 2024-09-15T01:51:34.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 msgr2=0x7fbe401419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098b0 tx=0x7fbe30032000 comp rx=0 tx=0).stop 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 shutdown_connections 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe2c038740 0x7fbe2c03abf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe40131b20 0x7fbe40132950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe400a0280 0x7fbe40141f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 --2- 172.21.15.17:0/767789243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe4009f1e0 0x7fbe401419f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 >> 172.21.15.17:0/767789243 conn(0x7fbe40071270 msgr2=0x7fbe40130560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 shutdown_connections 2024-09-15T01:51:34.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.315+0000 7fbe46434640 1 -- 172.21.15.17:0/767789243 wait complete. 2024-09-15T01:51:34.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0c0128b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:34.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0c0128b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:34.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0c0128b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:34.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0bfc0724a0 con 0x7f0bfc0a0280 2024-09-15T01:51:34.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0be2ffd640 1 -- 172.21.15.17:0/664502989 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0bec01a360 con 0x7f0bfc0a0280 2024-09-15T01:51:34.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0be2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:34.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.317+0000 7f0be2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:34.320 INFO:teuthology.orchestra.run.smithi017.stdout:47244640257 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 msgr2=0x7f0be803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 0x7f0be803ac40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0031000 tx=0x7f0bf0032000 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 10 monclient: shutdown 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 msgr2=0x7f0bfc09b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00c640 tx=0x7f0bec004960 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 shutdown_connections 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bfc131b20 0x7f0bfc132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bfc132430 0x7f0bfc132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 --2- 172.21.15.17:0/664502989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bfc0a0280 0x7f0bfc09b290 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:34.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 >> 172.21.15.17:0/664502989 conn(0x7f0bfc071270 msgr2=0x7f0bfc139e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:34.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 shutdown_connections 2024-09-15T01:51:34.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:34.319+0000 7f0c0128b640 1 -- 172.21.15.17:0/664502989 wait complete. 2024-09-15T01:51:34.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-15T01:51:34.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.4 2024-09-15T01:51:35.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T01:51:35.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T01:51:35.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T01:51:35.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.319+0000 7fc7aab5d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.319+0000 7fc7aab5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.319+0000 7fc7aab5d640 10 monclient: monmap: 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.320786+0000 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.320786+0000 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.324 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T01:51:35.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.320+0000 7fc7aab5d640 1 Processor -- start 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 -- start start 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient: init 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T01:51:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a4131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7a40a0280 0x7fc7a4131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a40a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc7a409f1e0 0x7fc7a4132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a409f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40a0040 con 0x7fc7a409f1e0 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a409bff0 con 0x7fc7a40a0280 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40a14a0 con 0x7fc7a4131b20 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7aab5d640 10 monclient(hunting): authenticate will time out at 755.317652s 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc7a409f1e0 0x7fc7a4132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a88f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc7a409f1e0 0x7fc7a4132430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39050/0 (socket says 172.21.15.17:39050) 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 1 -- 172.21.15.17:0/1542354118 learned_addr learned my addr 172.21.15.17:0/1542354118 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 10 monclient(hunting): get_auth_request con 0x7fc7a409f1e0 auth_method 0 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a88f9640 10 monclient(hunting): get_auth_request con 0x7fc7a4131b20 auth_method 0 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.321+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/1542354118 >> v2:172.21.15.67:3300/0 conn(0x7fc7a40a0280 0x7fc7a4131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request con 0x7fc7a40a0280 auth_method 0 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 -- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 msgr2=0x7fc7a409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 -- 172.21.15.17:0/1542354118 >> v2:172.21.15.67:3300/0 conn(0x7fc7a40a0280 msgr2=0x7fc7a4131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 --2- 172.21.15.17:0/1542354118 >> v2:172.21.15.67:3300/0 conn(0x7fc7a40a0280 0x7fc7a4131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 monclient: found mon.noname-a 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a90fa640 1 -- 172.21.15.17:0/1542354118 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a4068e80 con 0x7fc7a409f1e0 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.322+0000 7fc7a88f9640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a90fa640 1 -- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 msgr2=0x7fc7a4132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a90fa640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4132430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7a406ab60 tx=0x7fc79c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d0dc4fc720694e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 1 -- 172.21.15.17:0/1542354118 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c007610 con 0x7fc7a409f1e0 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: dump: 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.343 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 1 -- 172.21.15.17:0/1542354118 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c02d050 con 0x7fc7a409f1e0 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7aab5d640 5 monclient: authenticate success, global_id 4275 2024-09-15T01:51:35.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7aab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7aab5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7aab5d640 4 set_mon_vals no callback set 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 1 -- 172.21.15.17:0/1542354118 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0079a0 con 0x7fc7a409f1e0 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.325+0000 7fc7a37fe640 10 monclient: dump: 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.345 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.326+0000 7fc7aab5d640 10 monclient: shutdown 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.326+0000 7fc7aab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.326+0000 7fc7aab5d640 1 -- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 msgr2=0x7fc7a4132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.326+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4132430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7a406ab60 tx=0x7fc79c009840 comp rx=0 tx=0).stop 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 -- 172.21.15.17:0/1542354118 shutdown_connections 2024-09-15T01:51:35.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/1542354118 >> v2:172.21.15.67:3300/0 conn(0x7fc7a40a0280 0x7fc7a4131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4132430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/1542354118 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 -- 172.21.15.17:0/1542354118 >> 172.21.15.17:0/1542354118 conn(0x7fc7a4071270 msgr2=0x7fc7a40a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 -- 172.21.15.17:0/1542354118 shutdown_connections 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 1 -- 172.21.15.17:0/1542354118 wait complete. 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.327+0000 7fc7aab5d640 10 monclient: monmap: 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 1 Processor -- start 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 1 -- start start 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 10 monclient: init 2024-09-15T01:51:35.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 0x7fc7a4141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a40a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.328+0000 7fc7aab5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a4131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a411dff0 con 0x7fc7a409f1e0 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4120100 con 0x7fc7a40a0280 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a412fd30 con 0x7fc7a4131b20 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7aab5d640 10 monclient(hunting): authenticate will time out at 755.325120s 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 0x7fc7a4141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59708/0 (socket says 172.21.15.17:59708) 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 1 -- 172.21.15.17:0/3729308009 learned_addr learned my addr 172.21.15.17:0/3729308009 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): get_auth_request con 0x7fc7a4131b20 auth_method 0 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request con 0x7fc7a40a0280 auth_method 0 2024-09-15T01:51:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): get_auth_request con 0x7fc7a409f1e0 auth_method 0 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 -- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 msgr2=0x7fc7a4141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 -- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 msgr2=0x7fc7a4140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 monclient: found mon.noname-b 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a3fff640 1 -- 172.21.15.17:0/3729308009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009390 con 0x7fc7a40a0280 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a90fa640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.329+0000 7fc7a88f9640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.332+0000 7fc7a3fff640 1 -- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 msgr2=0x7fc7a4141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.332+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 0x7fc7a4141520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9c0 tx=0x7fc7900090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000cb50 con 0x7fc7a40a0280 2024-09-15T01:51:35.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 10 monclient: dump: 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79000ccb0 con 0x7fc7a40a0280 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7a2ffd640 4 set_mon_vals no callback set 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7aab5d640 5 monclient: authenticate success, global_id 4297 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7aab5d640 10 monclient: _renew_subs 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7aab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c009ae0 con 0x7fc7a40a0280 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.333+0000 7fc7aab5d640 10 monclient: _renew_subs 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7aab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790013470 con 0x7fc7a40a0280 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a411e940 con 0x7fc7a40a0280 2024-09-15T01:51:35.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.334+0000 7fc7a0ff9640 10 monclient: dump: 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc790013610 con 0x7fc7a40a0280 2024-09-15T01:51:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7aab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7aab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7aab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a40724a0 con 0x7fc7a40a0280 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7a0ff9640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 0x7fc78803ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.335+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc79004bb00 con 0x7fc7a40a0280 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.336+0000 7fc7a88f9640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 0x7fc78803ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.336+0000 7fc7a88f9640 10 monclient: get_auth_request con 0x7fc7880387a0 auth_method 0 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.338+0000 7fc7a88f9640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 0x7fc78803ac50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc798011580 tx=0x7fc7980090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.340+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc790023ad0 con 0x7fc7a40a0280 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.340+0000 7fc7a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.340+0000 7fc7a0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.364+0000 7fb864500640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.364+0000 7fb864500640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.364+0000 7fb864500640 10 monclient: monmap: 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.364981+0000 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.364981+0000 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.366 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.367 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.365+0000 7fb864500640 1 Processor -- start 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.365+0000 7fb864500640 1 -- start start 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.365+0000 7fb864500640 10 monclient: init 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb85c131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb85c0a0280 0x7fb85c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c0a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 0x7fb85c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0a0040 con 0x7fb85c131b20 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c09bff0 con 0x7fb85c0a0280 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0a14a0 con 0x7fb85c132430 2024-09-15T01:51:35.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb864500640 10 monclient(hunting): authenticate will time out at 755.362346s 2024-09-15T01:51:35.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb86229c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb85c131b20 0x7fb85c09eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33702/0 (socket says 172.21.15.17:33702) 2024-09-15T01:51:35.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb86229c640 1 -- 172.21.15.17:0/1013468827 learned_addr learned my addr 172.21.15.17:0/1013468827 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:35.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 1 --2- 172.21.15.17:0/1013468827 >> v2:172.21.15.67:3300/0 conn(0x7fb85c0a0280 0x7fb85c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): get_auth_request con 0x7fb85c0a0280 auth_method 0 2024-09-15T01:51:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb862a9d640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 0x7fb85c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb862a9d640 10 monclient(hunting): get_auth_request con 0x7fb85c132430 auth_method 0 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb862a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb862a9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb862a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.366+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 1 -- 172.21.15.17:0/1013468827 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 msgr2=0x7fb85c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 0x7fb85c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 1 -- 172.21.15.17:0/1013468827 >> v1:172.21.15.17:6789/0 conn(0x7fb85c131b20 legacy=0x7fb85c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 1 monclient: found mon.noname-c 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb861a9b640 1 -- 172.21.15.17:0/1013468827 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c09f3b0 con 0x7fb85c0a0280 2024-09-15T01:51:35.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.367+0000 7fb862a9d640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 0x7fb85c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.370+0000 7fb861a9b640 1 -- 172.21.15.17:0/1013468827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c0a0280 msgr2=0x7fb85c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb861a9b640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c0a0280 0x7fb85c131ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009850 tx=0x7fb84c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=775719af9153fde3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 1 -- 172.21.15.17:0/1013468827 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c02ade0 con 0x7fb85c0a0280 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: dump: 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.389 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 1 -- 172.21.15.17:0/1013468827 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c032440 con 0x7fb85c0a0280 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 1 -- 172.21.15.17:0/1013468827 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c032820 con 0x7fb85c0a0280 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb86129a640 10 monclient: dump: 2024-09-15T01:51:35.390 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb864500640 5 monclient: authenticate success, global_id 4300 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb864500640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.371+0000 7fb864500640 4 set_mon_vals no callback set 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 10 monclient: shutdown 2024-09-15T01:51:35.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 -- 172.21.15.17:0/1013468827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c0a0280 msgr2=0x7fb85c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c0a0280 0x7fb85c131ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009850 tx=0x7fb84c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 -- 172.21.15.17:0/1013468827 shutdown_connections 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c132430 0x7fb85c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 --2- 172.21.15.17:0/1013468827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c0a0280 0x7fb85c131ef0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 -- 172.21.15.17:0/1013468827 >> 172.21.15.17:0/1013468827 conn(0x7fb85c071270 msgr2=0x7fb85c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 -- 172.21.15.17:0/1013468827 shutdown_connections 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.372+0000 7fb864500640 1 -- 172.21.15.17:0/1013468827 wait complete. 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.373+0000 7fb864500640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.373+0000 7fb864500640 10 monclient: monmap: 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.373+0000 7fb864500640 1 Processor -- start 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 1 -- start start 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 10 monclient: init 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 0x7fb85c141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb85c131b20 0x7fb85c141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 10 monclient(hunting): picked mon.noname-a con 0x7fb85c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.374+0000 7fb864500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 0x7fb85c1422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c134630 con 0x7fb85c131b20 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 0x7fb85c141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c135b70 con 0x7fb85c132430 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1340b0 con 0x7fb85c0a0280 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb864500640 10 monclient(hunting): authenticate will time out at 755.371109s 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 0x7fb85c141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59734/0 (socket says 172.21.15.17:59734) 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 1 -- 172.21.15.17:0/3221881463 learned_addr learned my addr 172.21.15.17:0/3221881463 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): get_auth_request con 0x7fb85c0a0280 auth_method 0 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb85c131b20 0x7fb85c141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): get_auth_request con 0x7fb85c131b20 auth_method 0 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb86229c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 0x7fb85c1422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb861a9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): get_auth_request con 0x7fb85c132430 auth_method 0 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 1 -- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 msgr2=0x7fb85c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 0x7fb85c141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 1 -- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb85c131b20 msgr2=0x7fb85c141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.375+0000 7fb862a9d640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb85c131b20 0x7fb85c141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.376+0000 7fb862a9d640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.376+0000 7fb862a9d640 1 monclient: found mon.noname-b 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.376+0000 7fb862a9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.376+0000 7fb862a9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.376+0000 7fb862a9d640 1 -- 172.21.15.17:0/3221881463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854009a20 con 0x7fb85c132430 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb862a9d640 1 -- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 msgr2=0x7fb85c1422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb862a9d640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 0x7fb85c1422d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb854009f10 tx=0x7fb85400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854014480 con 0x7fb85c132430 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.378+0000 7fb8527fc640 10 monclient: dump: 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.398 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85400f040 con 0x7fb85c132430 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854014850 con 0x7fb85c132430 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb8527fc640 10 monclient: dump: 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 5 monclient: authenticate success, global_id 4303 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 10 monclient: _renew_subs 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb860a99640 4 set_mon_vals no callback set 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c009390 con 0x7fb85c132430 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 10 monclient: _renew_subs 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.379+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c138eb0 con 0x7fb85c132430 2024-09-15T01:51:35.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.380+0000 7fb864500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.380+0000 7fb864500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.380+0000 7fb864500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.380+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb854003020 con 0x7fb85c132430 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.380+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c0724a0 con 0x7fb85c132430 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.381+0000 7fb8527fc640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 0x7fb84003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.381+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fb85404c440 con 0x7fb85c132430 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.381+0000 7fb86229c640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 0x7fb84003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.381+0000 7fb86229c640 10 monclient: get_auth_request con 0x7fb840038740 auth_method 0 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.384+0000 7fb86229c640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 0x7fb84003abf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb858004ca0 tx=0x7fb858012900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.386+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb8540112c0 con 0x7fb85c132430 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.386+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.386+0000 7fb8527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.410+0000 7f901766f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.410+0000 7f901766f640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.410+0000 7f901766f640 10 monclient: monmap: 2024-09-15T01:51:35.412 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.411120+0000 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.411120+0000 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.410+0000 7f901766f640 1 Processor -- start 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 1 -- start start 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient: init 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9010131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90100a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9010137e60 0x7f901013a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): picked mon.noname-a con 0x7f9010137e60 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100a0040 con 0x7f9010131b20 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901009bff0 con 0x7f90100a0280 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90100a14a0 con 0x7f9010137e60 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.411+0000 7f901766f640 10 monclient(hunting): authenticate will time out at 755.407931s 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9010137e60 0x7f901013a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9010137e60 0x7f901013a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39080/0 (socket says 172.21.15.17:39080) 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 -- 172.21.15.17:0/2602829737 learned_addr learned my addr 172.21.15.17:0/2602829737 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9014c0a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f90100a0280 0x7f9010131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52408/0 (socket says 172.21.15.17:52408) 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): get_auth_request con 0x7f9010137e60 auth_method 0 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 -- 172.21.15.17:0/2602829737 >> v1:172.21.15.67:6789/0 conn(0x7f90100a0280 legacy=0x7f9010131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 -- 172.21.15.17:0/2602829737 >> v1:172.21.15.17:6789/0 conn(0x7f9010131b20 legacy=0x7f901009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 monclient: found mon.noname-a 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.412+0000 7f9015c0c640 1 -- 172.21.15.17:0/2602829737 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901009f3b0 con 0x7f9010137e60 2024-09-15T01:51:35.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9015c0c640 1 -- 172.21.15.17:0/2602829737 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010137e60 msgr2=0x7f901013a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9015c0c640 1 --2- 172.21.15.17:0/2602829737 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010137e60 0x7f901013a240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9008009f20 tx=0x7f9008009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea4ca3b6f1fab41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 1 -- 172.21.15.17:0/2602829737 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900802aea0 con 0x7f9010137e60 2024-09-15T01:51:35.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: dump: 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 1 -- 172.21.15.17:0/2602829737 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900802d050 con 0x7f9010137e60 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 1 -- 172.21.15.17:0/2602829737 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008007840 con 0x7f9010137e60 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f9007fff640 10 monclient: dump: 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f901766f640 5 monclient: authenticate success, global_id 4284 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f901766f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.416+0000 7f901766f640 4 set_mon_vals no callback set 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 10 monclient: shutdown 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 -- 172.21.15.17:0/2602829737 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010137e60 msgr2=0x7f901013a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 --2- 172.21.15.17:0/2602829737 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010137e60 0x7f901013a240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9008009f20 tx=0x7f9008009840 comp rx=0 tx=0).stop 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 -- 172.21.15.17:0/2602829737 shutdown_connections 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 --2- 172.21.15.17:0/2602829737 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010137e60 0x7f901013a240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 -- 172.21.15.17:0/2602829737 >> 172.21.15.17:0/2602829737 conn(0x7f9010071270 msgr2=0x7f90100a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 -- 172.21.15.17:0/2602829737 shutdown_connections 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.417+0000 7f901766f640 1 -- 172.21.15.17:0/2602829737 wait complete. 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.418+0000 7f901766f640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.418+0000 7f901766f640 10 monclient: monmap: 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.419+0000 7f901766f640 1 Processor -- start 2024-09-15T01:51:35.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.419+0000 7f901766f640 1 -- start start 2024-09-15T01:51:35.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.419+0000 7f901766f640 10 monclient: init 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901766f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901766f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901766f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901766f640 10 monclient(hunting): picked mon.noname-c con 0x7f90100a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901766f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901540b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.420+0000 7f901540b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59738/0 (socket says 172.21.15.17:59738) 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39086/0 (socket says 172.21.15.17:39086) 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): picked mon.noname-a con 0x7f9010131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 1 -- 172.21.15.17:0/1097507319 learned_addr learned my addr 172.21.15.17:0/1097507319 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9010137e60 0x7f90101341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): picked mon.noname-b con 0x7f9010137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901009eed0 con 0x7f9010131b20 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9010135f30 con 0x7f9010137e60 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901009b7c0 con 0x7f90100a0280 2024-09-15T01:51:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901766f640 10 monclient(hunting): authenticate will time out at 755.417485s 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9015c0c640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9010137e60 0x7f90101341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): get_auth_request con 0x7f90100a0280 auth_method 0 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): get_auth_request con 0x7f9010131b20 auth_method 0 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9015c0c640 10 monclient(hunting): get_auth_request con 0x7f9010137e60 auth_method 0 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9015c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9015c0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9015c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f9014c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.421+0000 7f901540b640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 -- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9010137e60 msgr2=0x7f90101341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9010137e60 0x7f90101341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 -- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 msgr2=0x7f9010133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 monclient: found mon.noname-c 2024-09-15T01:51:35.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f901540b640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008009ae0 con 0x7f90100a0280 2024-09-15T01:51:35.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.422+0000 7f9014c0a640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.423+0000 7fc4914d4640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.423+0000 7fc4914d4640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.423+0000 7fc4914d4640 10 monclient: monmap: 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.424287+0000 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.424287+0000 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.435 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 1 Processor -- start 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 1 -- start start 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient: init 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7f901540b640 1 -- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 msgr2=0x7f901009b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7f901540b640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f900c00c560 tx=0x7f900c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c09c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-b con 0x7fc48c09c5c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc48c09a500 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc48c06f440 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.424+0000 7fc4914d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48c09bae0 con 0x7fc48c06f440 2024-09-15T01:51:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc4914d4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48c09a9a0 con 0x7fc48c09a500 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc4914d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc4914d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c09b800 con 0x7fc48c09c5c0 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc4914d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc4914d4640 10 monclient(hunting): authenticate will time out at 755.421070s 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc48c06f440 0x7fc48c136770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33726/0 (socket says 172.21.15.17:33726) 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48b7fe640 1 -- 172.21.15.17:0/3714567951 learned_addr learned my addr 172.21.15.17:0/3714567951 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 1 --2- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c09c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): get_auth_request con 0x7fc48c09c5c0 auth_method 0 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0117a0 con 0x7f90100a0280 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: dump: 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c011900 con 0x7f90100a0280 2024-09-15T01:51:35.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c011cc0 con 0x7f90100a0280 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90057fa640 10 monclient: dump: 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.442 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 5 monclient: authenticate success, global_id 4280 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 10 monclient: _renew_subs 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90100a09b0 con 0x7f90100a0280 2024-09-15T01:51:35.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 1 -- 172.21.15.17:0/3714567951 >> v1:172.21.15.67:6789/0 conn(0x7fc48c09a500 legacy=0x7fc48c06ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 1 -- 172.21.15.17:0/3714567951 >> v1:172.21.15.17:6789/0 conn(0x7fc48c06f440 legacy=0x7fc48c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7fc48affd640 1 monclient: found mon.noname-b 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7fc48affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7fc48affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7fc48affd640 1 -- 172.21.15.17:0/3714567951 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c139eb0 con 0x7fc48c09c5c0 2024-09-15T01:51:35.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f90077fe640 4 set_mon_vals no callback set 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 10 monclient: _renew_subs 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.425+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9010135c80 con 0x7f90100a0280 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7f901766f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7f901766f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7f901766f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.426+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9010136160 con 0x7f90100a0280 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.427+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f900c012630 con 0x7f90100a0280 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.427+0000 7f90057fa640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 0x7f8ff403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.427+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f900c04b670 con 0x7f90100a0280 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.428+0000 7f9014c0a640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 0x7f8ff403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.428+0000 7f9014c0a640 10 monclient: get_auth_request con 0x7f8ff4038790 auth_method 0 2024-09-15T01:51:35.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.429+0000 7fc48affd640 1 -- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 msgr2=0x7fc48c09c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.429+0000 7fc48affd640 1 --2- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c09c990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc48c071750 tx=0x7fc47c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bae5aecd8ee3f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.430+0000 7f9014c0a640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 0x7f8ff403ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9000006830 tx=0x7f9000007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 1 -- 172.21.15.17:0/3714567951 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c032440 con 0x7fc48c09c5c0 2024-09-15T01:51:35.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f900c002090 con 0x7f90100a0280 2024-09-15T01:51:35.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7f90057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7f90057fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: dump: 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.459 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 1 -- 172.21.15.17:0/3714567951 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c0325a0 con 0x7fc48c09c5c0 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 1 -- 172.21.15.17:0/3714567951 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c032940 con 0x7fc48c09c5c0 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc489ffb640 10 monclient: dump: 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 5 monclient: authenticate success, global_id 4283 2024-09-15T01:51:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 4 set_mon_vals no callback set 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 10 monclient: shutdown 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 1 -- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 msgr2=0x7fc48c09c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.432+0000 7fc4914d4640 1 --2- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c09c990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc48c071750 tx=0x7fc47c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 1 -- 172.21.15.17:0/3714567951 shutdown_connections 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 1 --2- 172.21.15.17:0/3714567951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c09c990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 1 -- 172.21.15.17:0/3714567951 >> 172.21.15.17:0/3714567951 conn(0x7fc48c071320 msgr2=0x7fc48c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 1 -- 172.21.15.17:0/3714567951 shutdown_connections 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 1 -- 172.21.15.17:0/3714567951 wait complete. 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.433+0000 7fc4914d4640 10 monclient: monmap: 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.462 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 Processor -- start 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 -- start start 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient: init 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc48c06f440 0x7fc48c13daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-b con 0x7fc48c06f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-a con 0x7fc48c09a500 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c12fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): picked mon.noname-c con 0x7fc48c09c5c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c132540 con 0x7fc48c09a500 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c0a33d0 con 0x7fc48c06f440 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c06e6c0 con 0x7fc48c09c5c0 2024-09-15T01:51:35.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc4914d4640 10 monclient(hunting): authenticate will time out at 755.430824s 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39090/0 (socket says 172.21.15.17:39090) 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 1 -- 172.21.15.17:0/950354727 learned_addr learned my addr 172.21.15.17:0/950354727 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 10 monclient(hunting): get_auth_request con 0x7fc48c09a500 auth_method 0 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.434+0000 7fc48a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c12fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc48c06f440 0x7fc48c13daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): get_auth_request con 0x7fc48c06f440 auth_method 0 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): get_auth_request con 0x7fc48c09c5c0 auth_method 0 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 -- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 msgr2=0x7fc48c12fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c12fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 -- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc48c06f440 msgr2=0x7fc48c13daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc48c06f440 0x7fc48c13daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.435+0000 7fc48a7fc640 1 -- 172.21.15.17:0/950354727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c009390 con 0x7fc48c09a500 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.437+0000 7fc48a7fc640 1 -- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 msgr2=0x7fc48c136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.437+0000 7fc48a7fc640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c5e0 tx=0x7fc480007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480012840 con 0x7fc48c09a500 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: dump: 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.468 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4800129c0 con 0x7fc48c09a500 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480012d80 con 0x7fc48c09a500 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc4914d4640 5 monclient: authenticate success, global_id 4290 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc472ffd640 10 monclient: dump: 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.469 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc4914d4640 10 monclient: _renew_subs 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc4914d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c131f80 con 0x7fc48c09a500 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.438+0000 7fc4897fa640 4 set_mon_vals no callback set 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.439+0000 7fc4914d4640 10 monclient: _renew_subs 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.439+0000 7fc4914d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.439+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c131c70 con 0x7fc48c09a500 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc480011590 con 0x7fc48c09a500 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc472ffd640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 0x7fc47803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc48004e220 con 0x7fc48c09a500 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc4914d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc48affd640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 0x7fc47803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc4914d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc4914d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c0724a0 con 0x7fc48c09a500 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.440+0000 7fc48affd640 10 monclient: get_auth_request con 0x7fc478038790 auth_method 0 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.443+0000 7fc48affd640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 0x7fc47803ac40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc47c031000 tx=0x7fc47c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.445+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc480016030 con 0x7fc48c09a500 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.445+0000 7fc472ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.445+0000 7fc472ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.483+0000 7f9a4fc80640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.483+0000 7f9a4fc80640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.484+0000 7f9a4fc80640 10 monclient: monmap: 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.484856+0000 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.484856+0000 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.486 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.487 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.484+0000 7f9a4fc80640 1 Processor -- start 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.485+0000 7f9a4fc80640 1 -- start start 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.485+0000 7f9a4fc80640 10 monclient: init 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a48131b20 0x7f9a4809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-c con 0x7f9a48131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:35.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a480a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a4809f1e0 0x7f9a48138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-b con 0x7f9a4809f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480a0040 con 0x7f9a480a0280 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4da1c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a48131b20 0x7f9a4809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4da1c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a48131b20 0x7f9a4809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59014/0 (socket says 172.21.15.17:59014) 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4da1c640 1 -- 172.21.15.17:0/1070336805 learned_addr learned my addr 172.21.15.17:0/1070336805 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4e21d640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a4809f1e0 0x7f9a48138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.486+0000 7f9a4fc80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4809bff0 con 0x7f9a48131b20 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4fc80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480a14a0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4fc80640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4fc80640 10 monclient(hunting): authenticate will time out at 755.483337s 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): get_auth_request con 0x7f9a48131b20 auth_method 0 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): get_auth_request con 0x7f9a4809f1e0 auth_method 0 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 1 -- 172.21.15.17:0/1070336805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a4809f1e0 msgr2=0x7f9a48138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a4809f1e0 0x7f9a48138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 1 -- 172.21.15.17:0/1070336805 >> v1:172.21.15.67:6789/0 conn(0x7f9a480a0280 legacy=0x7f9a48131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 1 monclient: found mon.noname-c 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.487+0000 7f9a4da1c640 1 -- 172.21.15.17:0/1070336805 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48132600 con 0x7f9a48131b20 2024-09-15T01:51:35.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4da1c640 1 -- 172.21.15.17:0/1070336805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a48131b20 msgr2=0x7f9a4809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4da1c640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a48131b20 0x7f9a4809eca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009910 tx=0x7f9a440091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706c4892e871ef0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 1 -- 172.21.15.17:0/1070336805 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44032440 con 0x7f9a48131b20 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 10 monclient: dump: 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.509 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4fc80640 5 monclient: authenticate success, global_id 4312 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4fc80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.491+0000 7f9a4ca1a640 1 -- 172.21.15.17:0/1070336805 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a440325a0 con 0x7f9a48131b20 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4ca1a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4fc80640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4fc80640 4 set_mon_vals no callback set 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4ca1a640 1 -- 172.21.15.17:0/1070336805 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44032940 con 0x7f9a48131b20 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4ca1a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4ca1a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:35.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.492+0000 7f9a4ca1a640 10 monclient: dump: 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 10 monclient: shutdown 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a48131b20 msgr2=0x7f9a4809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a48131b20 0x7f9a4809eca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009910 tx=0x7f9a440091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 shutdown_connections 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a4809f1e0 0x7f9a48138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 --2- 172.21.15.17:0/1070336805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a48131b20 0x7f9a4809eca0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 >> 172.21.15.17:0/1070336805 conn(0x7f9a48071270 msgr2=0x7f9a480a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 shutdown_connections 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.493+0000 7f9a4fc80640 1 -- 172.21.15.17:0/1070336805 wait complete. 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.494+0000 7f9a4fc80640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.494+0000 7f9a4fc80640 10 monclient: monmap: 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.512 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.494+0000 7f9a4fc80640 1 Processor -- start 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.494+0000 7f9a4fc80640 1 -- start start 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.494+0000 7f9a4fc80640 10 monclient: init 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-b con 0x7f9a4809f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-c con 0x7f9a480a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4fc80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4da1c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4da1c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59018/0 (socket says 172.21.15.17:59018) 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4d21b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4d21b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59780/0 (socket says 172.21.15.17:59780) 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.495+0000 7f9a4e21d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39100/0 (socket says 172.21.15.17:39100) 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 -- 172.21.15.17:0/2287387095 learned_addr learned my addr 172.21.15.17:0/2287387095 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): picked mon.noname-a con 0x7f9a48131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a48097600 con 0x7f9a48131b20 2024-09-15T01:51:35.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a48097b30 con 0x7f9a4809f1e0 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a48139040 con 0x7f9a480a0280 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4fc80640 10 monclient(hunting): authenticate will time out at 755.492071s 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): get_auth_request con 0x7f9a4809f1e0 auth_method 0 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): get_auth_request con 0x7f9a480a0280 auth_method 0 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): get_auth_request con 0x7f9a48131b20 auth_method 0 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-09-15T01:51:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 -- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 msgr2=0x7f9a4812d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 -- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 msgr2=0x7f9a4812d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 monclient: found mon.noname-b 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4da1c640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a44009450 con 0x7f9a4809f1e0 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4d21b640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.496+0000 7f9a4e21d640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4da1c640 1 -- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 msgr2=0x7f9a4809a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4da1c640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009210 tx=0x7f9a44032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44031c70 con 0x7f9a4809f1e0 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: dump: 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a44031df0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4403c650 con 0x7f9a4809f1e0 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3dffb640 10 monclient: dump: 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 5 monclient: authenticate success, global_id 4315 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 10 monclient: _renew_subs 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a480981c0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 10 monclient: _renew_subs 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a4fc80640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4809b700 con 0x7f9a4809f1e0 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.499+0000 7f9a3ffff640 4 set_mon_vals no callback set 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9a4403ca80 con 0x7f9a4809f1e0 2024-09-15T01:51:35.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a337fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a337fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a337fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4809b8f0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a3dffb640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 0x7f9a2c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f9a4406bff0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a4d21b640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 0x7f9a2c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.501+0000 7f9a4d21b640 10 monclient: get_auth_request con 0x7f9a2c0387c0 auth_method 0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.504+0000 7f9a4d21b640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 0x7f9a2c03ac70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a4812ecf0 tx=0x7f9a38016040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.506+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a4402f2a0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.506+0000 7f9a3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.506+0000 7f9a3dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.508+0000 7fa30fcaf640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.508+0000 7fa30fcaf640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.508+0000 7fa30fcaf640 10 monclient: monmap: 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.509058+0000 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.509058+0000 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.508+0000 7fa30fcaf640 1 Processor -- start 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 -- start start 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient: init 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa30809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-b con 0x7fa308131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3080a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 0x7fa308138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-a con 0x7fa30809f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080a0040 con 0x7fa3080a0280 2024-09-15T01:51:35.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30809bff0 con 0x7fa30809f1e0 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080a14a0 con 0x7fa308131b20 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.509+0000 7fa30fcaf640 10 monclient(hunting): authenticate will time out at 755.505966s 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30d24a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3080a0280 0x7fa308131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52444/0 (socket says 172.21.15.17:52444) 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30d24a640 1 -- 172.21.15.17:0/580134448 learned_addr learned my addr 172.21.15.17:0/580134448 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 --2- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa30809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): get_auth_request con 0x7fa308131b20 auth_method 0 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 1 --2- 172.21.15.17:0/580134448 >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 0x7fa308138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): get_auth_request con 0x7fa30809f1e0 auth_method 0 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 -- 172.21.15.17:0/580134448 >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 msgr2=0x7fa308138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 --2- 172.21.15.17:0/580134448 >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 0x7fa308138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 -- 172.21.15.17:0/580134448 >> v1:172.21.15.67:6789/0 conn(0x7fa3080a0280 legacy=0x7fa308131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 monclient: found mon.noname-b 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30da4b640 1 -- 172.21.15.17:0/580134448 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308132600 con 0x7fa308131b20 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.510+0000 7fa30e24c640 1 --2- 172.21.15.17:0/580134448 >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 0x7fa308138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.513+0000 7fa30da4b640 1 -- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 msgr2=0x7fa30809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.513+0000 7fa30da4b640 1 --2- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa30809eca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa304009850 tx=0x7fa3040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a112005ac586b41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 1 -- 172.21.15.17:0/580134448 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30402ade0 con 0x7fa308131b20 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: dump: 2024-09-15T01:51:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 1 -- 172.21.15.17:0/580134448 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa304032440 con 0x7fa308131b20 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30fcaf640 5 monclient: authenticate success, global_id 4295 2024-09-15T01:51:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30fcaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 1 -- 172.21.15.17:0/580134448 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3040327e0 con 0x7fa308131b20 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30fcaf640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30fcaf640 4 set_mon_vals no callback set 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.514+0000 7fa30ca49640 10 monclient: dump: 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.539 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 10 monclient: shutdown 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 -- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 msgr2=0x7fa30809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa30809eca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa304009850 tx=0x7fa3040090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 -- 172.21.15.17:0/580134448 shutdown_connections 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/580134448 >> v2:172.21.15.17:3300/0 conn(0x7fa30809f1e0 0x7fa308138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/580134448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa30809eca0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 -- 172.21.15.17:0/580134448 >> 172.21.15.17:0/580134448 conn(0x7fa308071270 msgr2=0x7fa3080a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 -- 172.21.15.17:0/580134448 shutdown_connections 2024-09-15T01:51:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.515+0000 7fa30fcaf640 1 -- 172.21.15.17:0/580134448 wait complete. 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.516+0000 7fa30fcaf640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.516+0000 7fa30fcaf640 10 monclient: monmap: 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.516+0000 7fa30fcaf640 1 Processor -- start 2024-09-15T01:51:35.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.516+0000 7fa30fcaf640 1 -- start start 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient: init 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-a con 0x7fa30809f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 0x7fa308133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3080a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa308134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): picked mon.noname-c con 0x7fa308131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308135ad0 con 0x7fa30809f1e0 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080a09b0 con 0x7fa3080a0280 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa308138f30 con 0x7fa308131b20 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30fcaf640 10 monclient(hunting): authenticate will time out at 755.513550s 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30e24c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa308134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39118/0 (socket says 172.21.15.17:39118) 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30e24c640 10 monclient(hunting): get_auth_request con 0x7fa308131b20 auth_method 0 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 1 -- 172.21.15.17:0/4265476138 learned_addr learned my addr 172.21.15.17:0/4265476138 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30e24c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30e24c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30e24c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 10 monclient(hunting): get_auth_request con 0x7fa30809f1e0 auth_method 0 2024-09-15T01:51:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.517+0000 7fa30da4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 0x7fa308133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 10 monclient(hunting): get_auth_request con 0x7fa3080a0280 auth_method 0 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30da4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 -- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 msgr2=0x7fa308133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 0x7fa308133c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 -- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 msgr2=0x7fa3081336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 monclient: found mon.noname-c 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30e24c640 1 -- 172.21.15.17:0/4265476138 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa300009a20 con 0x7fa308131b20 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30d24a640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 0x7fa308133c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.518+0000 7fa30da4b640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.520+0000 7fa30e24c640 1 -- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 msgr2=0x7fa308134140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.520+0000 7fa30e24c640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa308134140 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa30000c470 tx=0x7fa30000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300014480 con 0x7fa308131b20 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: dump: 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.546 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30000f040 con 0x7fa308131b20 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fffff640 4 set_mon_vals no callback set 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300014830 con 0x7fa308131b20 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa2fdffb640 10 monclient: dump: 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 5 monclient: authenticate success, global_id 4298 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 10 monclient: _renew_subs 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa304009390 con 0x7fa308131b20 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 10 monclient: _renew_subs 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.521+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa308135ca0 con 0x7fa308131b20 2024-09-15T01:51:35.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.522+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa300003020 con 0x7fa308131b20 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30fcaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa2fdffb640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 0x7fa2ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30fcaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30fcaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fa30004c5d0 con 0x7fa308131b20 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30809edb0 con 0x7fa308131b20 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30da4b640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 0x7fa2ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.523+0000 7fa30da4b640 10 monclient: get_auth_request con 0x7fa2ec038740 auth_method 0 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.526+0000 7fa30da4b640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 0x7fa2ec03abf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa304000e50 tx=0x7fa304032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.527+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa30809edb0 con 0x7fa308131b20 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.527+0000 7fa2fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.527+0000 7fa2fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:35.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.596+0000 7fc7aab5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:35.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.596+0000 7fc7aab5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:35.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.596+0000 7fc7aab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.596+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc7a411fe20 con 0x7fc7a40a0280 2024-09-15T01:51:35.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.597+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3729308009 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc7900232f0 con 0x7fc7a40a0280 2024-09-15T01:51:35.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.597+0000 7fc7a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T01:51:35.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.597+0000 7fc7a0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.599 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 msgr2=0x7fc78803ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 0x7fc78803ac50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc798011580 tx=0x7fc7980090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 10 monclient: shutdown 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 msgr2=0x7fc7a4141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 0x7fc7a4141520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9c0 tx=0x7fc7900090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 shutdown_connections 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc7880387a0 0x7fc78803ac50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4131b20 0x7fc7a4141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40a0280 0x7fc7a4141520 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 --2- 172.21.15.17:0/3729308009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a409f1e0 0x7fc7a4140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 >> 172.21.15.17:0/3729308009 conn(0x7fc7a4071270 msgr2=0x7fc7a40a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 shutdown_connections 2024-09-15T01:51:35.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.600+0000 7fc7aab5d640 1 -- 172.21.15.17:0/3729308009 wait complete. 2024-09-15T01:51:35.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-15T01:51:35.620 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc4914d4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc4914d4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc4914d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc48c1326b0 con 0x7fc48c09a500 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc472ffd640 1 -- 172.21.15.17:0/950354727 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc480016020 con 0x7fc48c09a500 2024-09-15T01:51:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc472ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T01:51:35.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.628+0000 7fc472ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.631 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:35.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 msgr2=0x7fc47803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 0x7fc47803ac40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc47c031000 tx=0x7fc47c032000 comp rx=0 tx=0).stop 2024-09-15T01:51:35.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 10 monclient: shutdown 2024-09-15T01:51:35.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 msgr2=0x7fc48c136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c5e0 tx=0x7fc480007ed0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 shutdown_connections 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc478038790 0x7fc47803ac40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc48c09a500 0x7fc48c136d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc48c09c5c0 0x7fc48c12fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 --2- 172.21.15.17:0/950354727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc48c06f440 0x7fc48c13daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 >> 172.21.15.17:0/950354727 conn(0x7fc48c071320 msgr2=0x7fc48c138ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 shutdown_connections 2024-09-15T01:51:35.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.631+0000 7fc4914d4640 1 -- 172.21.15.17:0/950354727 wait complete. 2024-09-15T01:51:35.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-15T01:51:35.648 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.701+0000 7fa30fcaf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:35.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.701+0000 7fa30fcaf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:35.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.701+0000 7fa30fcaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.701+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa30809edb0 con 0x7fa308131b20 2024-09-15T01:51:35.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.702+0000 7fa2fdffb640 1 -- 172.21.15.17:0/4265476138 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa30809edb0 con 0x7fa308131b20 2024-09-15T01:51:35.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.702+0000 7fa2fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T01:51:35.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.702+0000 7fa2fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.707 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 msgr2=0x7fa2ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 0x7fa2ec03abf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa304000e50 tx=0x7fa304032000 comp rx=0 tx=0).stop 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 10 monclient: shutdown 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 msgr2=0x7fa308134140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa308134140 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa30000c470 tx=0x7fa30000b040 comp rx=0 tx=0).stop 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 shutdown_connections 2024-09-15T01:51:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa2ec038740 0x7fa2ec03abf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa308131b20 0x7fa308134140 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3080a0280 0x7fa308133c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 --2- 172.21.15.17:0/4265476138 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa30809f1e0 0x7fa3081336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 >> 172.21.15.17:0/4265476138 conn(0x7fa308071270 msgr2=0x7fa30809d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 shutdown_connections 2024-09-15T01:51:35.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.704+0000 7fa30fcaf640 1 -- 172.21.15.17:0/4265476138 wait complete. 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb864500640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb864500640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb864500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb85c0a3eb0 con 0x7fb85c132430 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb8527fc640 1 -- 172.21.15.17:0/3221881463 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb854018090 con 0x7fb85c132430 2024-09-15T01:51:35.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T01:51:35.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.714+0000 7fb8527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-15T01:51:35.718 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-15T01:51:35.718 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 msgr2=0x7fb84003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 0x7fb84003abf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb858004ca0 tx=0x7fb858012900 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 10 monclient: shutdown 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 msgr2=0x7fb85c1422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 0x7fb85c1422d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb854009f10 tx=0x7fb85400b040 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 shutdown_connections 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb840038740 0x7fb84003abf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb85c131b20 0x7fb85c141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85c132430 0x7fb85c1422d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 --2- 172.21.15.17:0/3221881463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb85c0a0280 0x7fb85c141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 >> 172.21.15.17:0/3221881463 conn(0x7fb85c071270 msgr2=0x7fb85c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 shutdown_connections 2024-09-15T01:51:35.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.716+0000 7fb864500640 1 -- 172.21.15.17:0/3221881463 wait complete. 2024-09-15T01:51:35.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-15T01:51:35.729 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.737+0000 7f9a337fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.737+0000 7f9a337fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.737+0000 7f9a337fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.737+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9a480978e0 con 0x7f9a4809f1e0 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.738+0000 7f9a3dffb640 1 -- 172.21.15.17:0/2287387095 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a44037030 con 0x7f9a4809f1e0 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.738+0000 7f9a3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T01:51:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.738+0000 7f9a3dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.741 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 msgr2=0x7f9a2c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 0x7f9a2c03ac70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a4812ecf0 tx=0x7f9a38016040 comp rx=0 tx=0).stop 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 10 monclient: shutdown 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 msgr2=0x7f9a4809a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009210 tx=0x7f9a44032000 comp rx=0 tx=0).stop 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 shutdown_connections 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0387c0 0x7f9a2c03ac70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a48131b20 0x7f9a4812d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a480a0280 0x7f9a4812d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 --2- 172.21.15.17:0/2287387095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4809f1e0 0x7f9a4809a5f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 >> 172.21.15.17:0/2287387095 conn(0x7f9a48071270 msgr2=0x7f9a48130a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 shutdown_connections 2024-09-15T01:51:35.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.740+0000 7f9a337fe640 1 -- 172.21.15.17:0/2287387095 wait complete. 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f901766f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f901766f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f901766f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f90100724a0 con 0x7f90100a0280 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f90057fa640 1 -- 172.21.15.17:0/1097507319 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f900c0102e0 con 0x7f90100a0280 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f90057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.742+0000 7f90057fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:35.744 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-15T01:51:35.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 msgr2=0x7f8ff403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 0x7f8ff403ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9000006830 tx=0x7f9000007e40 comp rx=0 tx=0).stop 2024-09-15T01:51:35.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 10 monclient: shutdown 2024-09-15T01:51:35.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 msgr2=0x7f901009b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f900c00c560 tx=0x7f900c007ed0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 shutdown_connections 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ff4038790 0x7f8ff403ac40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9010131b20 0x7f9010133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9010137e60 0x7f90101341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 --2- 172.21.15.17:0/1097507319 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f90100a0280 0x7f901009b280 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 >> 172.21.15.17:0/1097507319 conn(0x7f9010071270 msgr2=0x7f90100a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 shutdown_connections 2024-09-15T01:51:35.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.744+0000 7f901766f640 1 -- 172.21.15.17:0/1097507319 wait complete. 2024-09-15T01:51:35.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-09-15T01:51:35.754 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-15T01:51:35.757 DEBUG:teuthology.parallel:result is None 2024-09-15T01:51:35.757 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T01:51:35.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T01:51:35.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 10 monclient: get_monmap_and_config 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 10 monclient: monmap: 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:35.892847+0000 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:35.892847+0000 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:35.894 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:35.895 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:35.895 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:35.895 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 1 Processor -- start 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 1 -- start start 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.892+0000 7fcad7880640 10 monclient: init 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcad00a58b0 0x7fcad0132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-a con 0x7fcad00a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcad00980a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcad0136dd0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00a01e0 con 0x7fcad0136dd0 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00a1580 con 0x7fcad00980a0 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad00a5c80 con 0x7fcad00a58b0 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad7880640 10 monclient(hunting): authenticate will time out at 755.889287s 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcad00a58b0 0x7fcad0132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad5e1d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fcad0136dd0 0x7fcad01391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33730/0 (socket says 172.21.15.17:33730) 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad5e1d640 1 -- 172.21.15.17:0/21066974 learned_addr learned my addr 172.21.15.17:0/21066974 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcad00a58b0 0x7fcad0132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39132/0 (socket says 172.21.15.17:39132) 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): get_auth_request con 0x7fcad00a58b0 auth_method 0 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 -- 172.21.15.17:0/21066974 >> v1:172.21.15.67:6789/0 conn(0x7fcad00980a0 legacy=0x7fcad00984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 -- 172.21.15.17:0/21066974 >> v1:172.21.15.17:6789/0 conn(0x7fcad0136dd0 legacy=0x7fcad01391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 monclient: found mon.noname-a 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.893+0000 7fcad561c640 1 -- 172.21.15.17:0/21066974 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0098bb0 con 0x7fcad00a58b0 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad561c640 1 -- 172.21.15.17:0/21066974 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00a58b0 msgr2=0x7fcad0132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad561c640 1 --2- 172.21.15.17:0/21066974 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00a58b0 0x7fcad0132040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009850 tx=0x7fcacc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7e97b7f83e677d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 1 -- 172.21.15.17:0/21066974 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc032440 con 0x7fcad00a58b0 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: dump: 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad7880640 5 monclient: authenticate success, global_id 4302 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 1 -- 172.21.15.17:0/21066974 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc0325a0 con 0x7fcad00a58b0 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad7880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad7880640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcad7880640 4 set_mon_vals no callback set 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 1 -- 172.21.15.17:0/21066974 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc032940 con 0x7fcad00a58b0 2024-09-15T01:51:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:35.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.896+0000 7fcac7fff640 10 monclient: dump: 2024-09-15T01:51:35.905 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.905 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 10 monclient: shutdown 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 -- 172.21.15.17:0/21066974 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00a58b0 msgr2=0x7fcad0132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 --2- 172.21.15.17:0/21066974 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00a58b0 0x7fcad0132040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009850 tx=0x7fcacc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 -- 172.21.15.17:0/21066974 shutdown_connections 2024-09-15T01:51:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 --2- 172.21.15.17:0/21066974 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00a58b0 0x7fcad0132040 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 -- 172.21.15.17:0/21066974 >> 172.21.15.17:0/21066974 conn(0x7fcad0070ed0 msgr2=0x7fcad00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 -- 172.21.15.17:0/21066974 shutdown_connections 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 1 -- 172.21.15.17:0/21066974 wait complete. 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 10 monclient: build_initial_monmap 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.897+0000 7fcad7880640 10 monclient: monmap: 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:35.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 Processor -- start 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 -- start start 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient: init 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-a con 0x7fcad00980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 0x7fcad0133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-b con 0x7fcad00a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 0x7fcad01335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): picked mon.noname-c con 0x7fcad0136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:35.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad01350d0 con 0x7fcad00980a0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00a3330 con 0x7fcad00a58b0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0134df0 con 0x7fcad0136dd0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad7880640 10 monclient(hunting): authenticate will time out at 755.894918s 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.898+0000 7fcad561c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 0x7fcad01335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39140/0 (socket says 172.21.15.17:39140) 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 -- 172.21.15.17:0/2503394158 learned_addr learned my addr 172.21.15.17:0/2503394158 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): get_auth_request con 0x7fcad00980a0 auth_method 0 2024-09-15T01:51:35.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): get_auth_request con 0x7fcad0136dd0 auth_method 0 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 0x7fcad0133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 10 monclient(hunting): get_auth_request con 0x7fcad00a58b0 auth_method 0 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 -- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 msgr2=0x7fcad01335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 0x7fcad01335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 -- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 msgr2=0x7fcad0133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 0x7fcad0133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 monclient: found mon.noname-a 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad561c640 1 -- 172.21.15.17:0/2503394158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc009390 con 0x7fcad00980a0 2024-09-15T01:51:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad5e1d640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 0x7fcad01335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.899+0000 7fcad4e1b640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 0x7fcad0133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad561c640 1 -- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 msgr2=0x7fcad0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad561c640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009360 tx=0x7fcacc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc03c960 con 0x7fcad00980a0 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: dump: 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc03cac0 con 0x7fcad00980a0 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 5 monclient: authenticate success, global_id 4305 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 10 monclient: _renew_subs 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad0133e90 con 0x7fcad00980a0 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc03ce60 con 0x7fcad00980a0 2024-09-15T01:51:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac57fa640 10 monclient: dump: 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcac77fe640 4 set_mon_vals no callback set 2024-09-15T01:51:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 10 monclient: _renew_subs 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.901+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad0134a50 con 0x7fcad00980a0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcacc039040 con 0x7fcad00980a0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcad7880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcad7880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcad7880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad0098e30 con 0x7fcad00980a0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.902+0000 7fcac57fa640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 0x7fcab403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.903+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fcacc06c020 con 0x7fcad00980a0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.903+0000 7fcad4e1b640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 0x7fcab403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.903+0000 7fcad4e1b640 10 monclient: get_auth_request con 0x7fcab4038790 auth_method 0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.905+0000 7fcad4e1b640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 0x7fcab403ac40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006fd0 tx=0x7fcac0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.907+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcacc09de40 con 0x7fcad00980a0 2024-09-15T01:51:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.907+0000 7fcac57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:35.907+0000 7fcac57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:36.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.068+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcad00724a0 con 0x7fcab4038790 2024-09-15T01:51:36.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.070+0000 7fcac57fa640 1 -- 172.21.15.17:0/2503394158 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40986 (secure 0 0 0) 0x7fcad00724a0 con 0x7fcab4038790 2024-09-15T01:51:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:36.073 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T01:51:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 msgr2=0x7fcab403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 0x7fcab403ac40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006fd0 tx=0x7fcac0008040 comp rx=0 tx=0).stop 2024-09-15T01:51:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 10 monclient: shutdown 2024-09-15T01:51:36.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 msgr2=0x7fcad0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009360 tx=0x7fcacc032000 comp rx=0 tx=0).stop 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 shutdown_connections 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcab4038790 0x7fcab403ac40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcad0136dd0 0x7fcad01335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcad00a58b0 0x7fcad0133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 --2- 172.21.15.17:0/2503394158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcad00980a0 0x7fcad0132b20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 >> 172.21.15.17:0/2503394158 conn(0x7fcad0070ed0 msgr2=0x7fcad01391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 shutdown_connections 2024-09-15T01:51:36.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.072+0000 7fcad7880640 1 -- 172.21.15.17:0/2503394158 wait complete. 2024-09-15T01:51:36.084 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-15T01:51:35.307288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786352,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444688,"statfs":{"total":579820584960,"available":577991360512,"internally_reserved":0,"allocated":1994752,"data_stored":1339600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.741032"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717899+0000","last_change":"2024-09-15T01:51:32.718180+0000","last_active":"2024-09-15T01:51:32.717899+0000","last_peered":"2024-09-15T01:51:32.717899+0000","last_clean":"2024-09-15T01:51:32.717899+0000","last_became_active":"2024-09-15T01:51:30.517374+0000","last_became_peered":"2024-09-15T01:51:30.517374+0000","last_unstale":"2024-09-15T01:51:32.717899+0000","last_undegraded":"2024-09-15T01:51:32.717899+0000","last_fullsized":"2024-09-15T01:51:32.717899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:49.285042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664627+0000","last_change":"2024-09-15T01:51:32.664815+0000","last_active":"2024-09-15T01:51:32.664627+0000","last_peered":"2024-09-15T01:51:32.664627+0000","last_clean":"2024-09-15T01:51:32.664627+0000","last_became_active":"2024-09-15T01:51:30.519255+0000","last_became_peered":"2024-09-15T01:51:30.519255+0000","last_unstale":"2024-09-15T01:51:32.664627+0000","last_undegraded":"2024-09-15T01:51:32.664627+0000","last_fullsized":"2024-09-15T01:51:32.664627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:46.999182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568082+0000","last_change":"2024-09-15T01:51:32.568282+0000","last_active":"2024-09-15T01:51:32.568082+0000","last_peered":"2024-09-15T01:51:32.568082+0000","last_clean":"2024-09-15T01:51:32.568082+0000","last_became_active":"2024-09-15T01:51:30.516534+0000","last_became_peered":"2024-09-15T01:51:30.516534+0000","last_unstale":"2024-09-15T01:51:32.568082+0000","last_undegraded":"2024-09-15T01:51:32.568082+0000","last_fullsized":"2024-09-15T01:51:32.568082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.027860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568144+0000","last_change":"2024-09-15T01:51:32.568412+0000","last_active":"2024-09-15T01:51:32.568144+0000","last_peered":"2024-09-15T01:51:32.568144+0000","last_clean":"2024-09-15T01:51:32.568144+0000","last_became_active":"2024-09-15T01:51:30.518571+0000","last_became_peered":"2024-09-15T01:51:30.518571+0000","last_unstale":"2024-09-15T01:51:32.568144+0000","last_undegraded":"2024-09-15T01:51:32.568144+0000","last_fullsized":"2024-09-15T01:51:32.568144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:13:16.913092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T01:51:31.575561+0000","last_change":"2024-09-15T01:51:30.519037+0000","last_active":"2024-09-15T01:51:31.575561+0000","last_peered":"2024-09-15T01:51:31.575561+0000","last_clean":"2024-09-15T01:51:31.575561+0000","last_became_active":"2024-09-15T01:51:30.518418+0000","last_became_peered":"2024-09-15T01:51:30.518418+0000","last_unstale":"2024-09-15T01:51:31.575561+0000","last_undegraded":"2024-09-15T01:51:31.575561+0000","last_fullsized":"2024-09-15T01:51:31.575561+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:24:49.620721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568092+0000","last_change":"2024-09-15T01:51:32.568425+0000","last_active":"2024-09-15T01:51:32.568092+0000","last_peered":"2024-09-15T01:51:32.568092+0000","last_clean":"2024-09-15T01:51:32.568092+0000","last_became_active":"2024-09-15T01:51:30.519207+0000","last_became_peered":"2024-09-15T01:51:30.519207+0000","last_unstale":"2024-09-15T01:51:32.568092+0000","last_undegraded":"2024-09-15T01:51:32.568092+0000","last_fullsized":"2024-09-15T01:51:32.568092+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:11.690964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568265+0000","last_change":"2024-09-15T01:51:32.568265+0000","last_active":"2024-09-15T01:51:32.568265+0000","last_peered":"2024-09-15T01:51:32.568265+0000","last_clean":"2024-09-15T01:51:32.568265+0000","last_became_active":"2024-09-15T01:51:30.519209+0000","last_became_peered":"2024-09-15T01:51:30.519209+0000","last_unstale":"2024-09-15T01:51:32.568265+0000","last_undegraded":"2024-09-15T01:51:32.568265+0000","last_fullsized":"2024-09-15T01:51:32.568265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:32.888198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717928+0000","last_change":"2024-09-15T01:51:32.718152+0000","last_active":"2024-09-15T01:51:32.717928+0000","last_peered":"2024-09-15T01:51:32.717928+0000","last_clean":"2024-09-15T01:51:32.717928+0000","last_became_active":"2024-09-15T01:51:30.518721+0000","last_became_peered":"2024-09-15T01:51:30.518721+0000","last_unstale":"2024-09-15T01:51:32.717928+0000","last_undegraded":"2024-09-15T01:51:32.717928+0000","last_fullsized":"2024-09-15T01:51:32.717928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:49.882031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664674+0000","last_change":"2024-09-15T01:51:32.664834+0000","last_active":"2024-09-15T01:51:32.664674+0000","last_peered":"2024-09-15T01:51:32.664674+0000","last_clean":"2024-09-15T01:51:32.664674+0000","last_became_active":"2024-09-15T01:51:30.520465+0000","last_became_peered":"2024-09-15T01:51:30.520465+0000","last_unstale":"2024-09-15T01:51:32.664674+0000","last_undegraded":"2024-09-15T01:51:32.664674+0000","last_fullsized":"2024-09-15T01:51:32.664674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:51:25.079633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":47777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":42821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:51:36.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T01:51:36.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.179+0000 7f9825c2b640 10 monclient: get_monmap_and_config 2024-09-15T01:51:36.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.179+0000 7f9825c2b640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.179+0000 7f9825c2b640 10 monclient: monmap: 2024-09-15T01:51:36.181 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.181 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:36.180146+0000 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:36.180146+0000 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:36.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 Processor -- start 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 -- start start 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient: init 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9820131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98200a0280 0x7f9820131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f98200a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f982009f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200a0040 con 0x7f982009f1e0 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982009bff0 con 0x7f98200a0280 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200a14a0 con 0x7f9820131b20 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f9825c2b640 10 monclient(hunting): authenticate will time out at 756.176690s 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59808/0 (socket says 172.21.15.17:59808) 2024-09-15T01:51:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 1 -- 172.21.15.17:0/3844363916 learned_addr learned my addr 172.21.15.17:0/3844363916 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): get_auth_request con 0x7f9820131b20 auth_method 0 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981effd640 1 --2- 172.21.15.17:0/3844363916 >> v2:172.21.15.67:3300/0 conn(0x7f98200a0280 0x7f9820131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981effd640 10 monclient(hunting): get_auth_request con 0x7f98200a0280 auth_method 0 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.180+0000 7f981effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 1 -- 172.21.15.17:0/3844363916 >> v2:172.21.15.67:3300/0 conn(0x7f98200a0280 msgr2=0x7f9820131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 1 --2- 172.21.15.17:0/3844363916 >> v2:172.21.15.67:3300/0 conn(0x7f98200a0280 0x7f9820131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 1 -- 172.21.15.17:0/3844363916 >> v1:172.21.15.67:6789/0 conn(0x7f982009f1e0 legacy=0x7f9820068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 1 monclient: found mon.noname-b 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.181+0000 7f981f7fe640 1 -- 172.21.15.17:0/3844363916 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820132430 con 0x7f9820131b20 2024-09-15T01:51:36.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.183+0000 7f981f7fe640 1 -- 172.21.15.17:0/3844363916 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 msgr2=0x7f982009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.183+0000 7f981f7fe640 1 --2- 172.21.15.17:0/3844363916 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9810009f20 tx=0x7f9810009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db2fb1234371d015 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 1 -- 172.21.15.17:0/3844363916 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810007610 con 0x7f9820131b20 2024-09-15T01:51:36.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: dump: 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 1 -- 172.21.15.17:0/3844363916 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981002d050 con 0x7f9820131b20 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 5 monclient: authenticate success, global_id 4304 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 1 -- 172.21.15.17:0/3844363916 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100079e0 con 0x7f9820131b20 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 4 set_mon_vals no callback set 2024-09-15T01:51:36.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f981e7fc640 10 monclient: dump: 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 10 monclient: shutdown 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 -- 172.21.15.17:0/3844363916 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 msgr2=0x7f982009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 --2- 172.21.15.17:0/3844363916 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9810009f20 tx=0x7f9810009840 comp rx=0 tx=0).stop 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 -- 172.21.15.17:0/3844363916 shutdown_connections 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 --2- 172.21.15.17:0/3844363916 >> v2:172.21.15.67:3300/0 conn(0x7f98200a0280 0x7f9820131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 --2- 172.21.15.17:0/3844363916 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9820131b20 0x7f982009eca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 -- 172.21.15.17:0/3844363916 >> 172.21.15.17:0/3844363916 conn(0x7f9820071270 msgr2=0x7f98200a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 -- 172.21.15.17:0/3844363916 shutdown_connections 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.184+0000 7f9825c2b640 1 -- 172.21.15.17:0/3844363916 wait complete. 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.185+0000 7f9825c2b640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.185+0000 7f9825c2b640 10 monclient: monmap: 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.197 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.185+0000 7f9825c2b640 1 Processor -- start 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.185+0000 7f9825c2b640 1 -- start start 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.185+0000 7f9825c2b640 10 monclient: init 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f982009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98200a0280 0x7f982012ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f98200a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9820131b20 0x7f9820098bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9820131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982011bd80 con 0x7f98200a0280 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200a09b0 con 0x7f9820131b20 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982009ac00 con 0x7f982009f1e0 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f9825c2b640 10 monclient(hunting): authenticate will time out at 756.182354s 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59822/0 (socket says 172.21.15.17:59822) 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 -- 172.21.15.17:0/1873058160 learned_addr learned my addr 172.21.15.17:0/1873058160 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): get_auth_request con 0x7f982009f1e0 auth_method 0 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98200a0280 0x7f982012ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): get_auth_request con 0x7f98200a0280 auth_method 0 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9820131b20 0x7f9820098bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): get_auth_request con 0x7f9820131b20 auth_method 0 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 -- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9820131b20 msgr2=0x7f9820098bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9820131b20 0x7f9820098bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 -- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98200a0280 msgr2=0x7f982012ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98200a0280 0x7f982012ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 monclient: found mon.noname-c 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.186+0000 7f981f7fe640 1 -- 172.21.15.17:0/1873058160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009ae0 con 0x7f982009f1e0 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f981f7fe640 1 -- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 msgr2=0x7f9820141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f981f7fe640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9810009f80 tx=0x7f9810036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810007e10 con 0x7f982009f1e0 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: dump: 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981003c840 con 0x7f982009f1e0 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 5 monclient: authenticate success, global_id 4307 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 10 monclient: _renew_subs 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98200a53d0 con 0x7f982009f1e0 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981003cbe0 con 0x7f982009f1e0 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f98077fe640 10 monclient: dump: 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.204 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 10 monclient: _renew_subs 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982009b2c0 con 0x7f982009f1e0 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.189+0000 7f981dffb640 4 set_mon_vals no callback set 2024-09-15T01:51:36.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9810031400 con 0x7f982009f1e0 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f98077fe640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 0x7f980c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f981002ae40 con 0x7f982009f1e0 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f9825c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f9825c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f9825c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982011c1f0 con 0x7f982009f1e0 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f981effd640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 0x7f980c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.190+0000 7f981effd640 10 monclient: get_auth_request con 0x7f980c038740 auth_method 0 2024-09-15T01:51:36.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.193+0000 7f981effd640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 0x7f980c03abf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f982009a380 tx=0x7f98140090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.194+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9810040020 con 0x7f982009f1e0 2024-09-15T01:51:36.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.194+0000 7f98077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.194+0000 7f98077fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:36.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.356+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f982011bfa0 con 0x7f980c038740 2024-09-15T01:51:36.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.358+0000 7f98077fe640 1 -- 172.21.15.17:0/1873058160 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40986 (secure 0 0 0) 0x7f982011bfa0 con 0x7f980c038740 2024-09-15T01:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:36.361 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T01:51:36.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 msgr2=0x7f980c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 0x7f980c03abf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f982009a380 tx=0x7f98140090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 10 monclient: shutdown 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 msgr2=0x7f9820141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9810009f80 tx=0x7f9810036000 comp rx=0 tx=0).stop 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 shutdown_connections 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f980c038740 0x7f980c03abf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98200a0280 0x7f982012ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9820131b20 0x7f9820098bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 --2- 172.21.15.17:0/1873058160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f982009f1e0 0x7f9820141160 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 >> 172.21.15.17:0/1873058160 conn(0x7f9820071270 msgr2=0x7f982009d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 shutdown_connections 2024-09-15T01:51:36.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.360+0000 7f9825c2b640 1 -- 172.21.15.17:0/1873058160 wait complete. 2024-09-15T01:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-15T01:51:35.307288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786352,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444688,"statfs":{"total":579820584960,"available":577991360512,"internally_reserved":0,"allocated":1994752,"data_stored":1339600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.741032"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717899+0000","last_change":"2024-09-15T01:51:32.718180+0000","last_active":"2024-09-15T01:51:32.717899+0000","last_peered":"2024-09-15T01:51:32.717899+0000","last_clean":"2024-09-15T01:51:32.717899+0000","last_became_active":"2024-09-15T01:51:30.517374+0000","last_became_peered":"2024-09-15T01:51:30.517374+0000","last_unstale":"2024-09-15T01:51:32.717899+0000","last_undegraded":"2024-09-15T01:51:32.717899+0000","last_fullsized":"2024-09-15T01:51:32.717899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:49.285042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664627+0000","last_change":"2024-09-15T01:51:32.664815+0000","last_active":"2024-09-15T01:51:32.664627+0000","last_peered":"2024-09-15T01:51:32.664627+0000","last_clean":"2024-09-15T01:51:32.664627+0000","last_became_active":"2024-09-15T01:51:30.519255+0000","last_became_peered":"2024-09-15T01:51:30.519255+0000","last_unstale":"2024-09-15T01:51:32.664627+0000","last_undegraded":"2024-09-15T01:51:32.664627+0000","last_fullsized":"2024-09-15T01:51:32.664627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:46.999182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568082+0000","last_change":"2024-09-15T01:51:32.568282+0000","last_active":"2024-09-15T01:51:32.568082+0000","last_peered":"2024-09-15T01:51:32.568082+0000","last_clean":"2024-09-15T01:51:32.568082+0000","last_became_active":"2024-09-15T01:51:30.516534+0000","last_became_peered":"2024-09-15T01:51:30.516534+0000","last_unstale":"2024-09-15T01:51:32.568082+0000","last_undegraded":"2024-09-15T01:51:32.568082+0000","last_fullsized":"2024-09-15T01:51:32.568082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.027860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568144+0000","last_change":"2024-09-15T01:51:32.568412+0000","last_active":"2024-09-15T01:51:32.568144+0000","last_peered":"2024-09-15T01:51:32.568144+0000","last_clean":"2024-09-15T01:51:32.568144+0000","last_became_active":"2024-09-15T01:51:30.518571+0000","last_became_peered":"2024-09-15T01:51:30.518571+0000","last_unstale":"2024-09-15T01:51:32.568144+0000","last_undegraded":"2024-09-15T01:51:32.568144+0000","last_fullsized":"2024-09-15T01:51:32.568144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:13:16.913092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T01:51:31.575561+0000","last_change":"2024-09-15T01:51:30.519037+0000","last_active":"2024-09-15T01:51:31.575561+0000","last_peered":"2024-09-15T01:51:31.575561+0000","last_clean":"2024-09-15T01:51:31.575561+0000","last_became_active":"2024-09-15T01:51:30.518418+0000","last_became_peered":"2024-09-15T01:51:30.518418+0000","last_unstale":"2024-09-15T01:51:31.575561+0000","last_undegraded":"2024-09-15T01:51:31.575561+0000","last_fullsized":"2024-09-15T01:51:31.575561+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:24:49.620721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568092+0000","last_change":"2024-09-15T01:51:32.568425+0000","last_active":"2024-09-15T01:51:32.568092+0000","last_peered":"2024-09-15T01:51:32.568092+0000","last_clean":"2024-09-15T01:51:32.568092+0000","last_became_active":"2024-09-15T01:51:30.519207+0000","last_became_peered":"2024-09-15T01:51:30.519207+0000","last_unstale":"2024-09-15T01:51:32.568092+0000","last_undegraded":"2024-09-15T01:51:32.568092+0000","last_fullsized":"2024-09-15T01:51:32.568092+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:11.690964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568265+0000","last_change":"2024-09-15T01:51:32.568265+0000","last_active":"2024-09-15T01:51:32.568265+0000","last_peered":"2024-09-15T01:51:32.568265+0000","last_clean":"2024-09-15T01:51:32.568265+0000","last_became_active":"2024-09-15T01:51:30.519209+0000","last_became_peered":"2024-09-15T01:51:30.519209+0000","last_unstale":"2024-09-15T01:51:32.568265+0000","last_undegraded":"2024-09-15T01:51:32.568265+0000","last_fullsized":"2024-09-15T01:51:32.568265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:32.888198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717928+0000","last_change":"2024-09-15T01:51:32.718152+0000","last_active":"2024-09-15T01:51:32.717928+0000","last_peered":"2024-09-15T01:51:32.717928+0000","last_clean":"2024-09-15T01:51:32.717928+0000","last_became_active":"2024-09-15T01:51:30.518721+0000","last_became_peered":"2024-09-15T01:51:30.518721+0000","last_unstale":"2024-09-15T01:51:32.717928+0000","last_undegraded":"2024-09-15T01:51:32.717928+0000","last_fullsized":"2024-09-15T01:51:32.717928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:49.882031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664674+0000","last_change":"2024-09-15T01:51:32.664834+0000","last_active":"2024-09-15T01:51:32.664674+0000","last_peered":"2024-09-15T01:51:32.664674+0000","last_clean":"2024-09-15T01:51:32.664674+0000","last_became_active":"2024-09-15T01:51:30.520465+0000","last_became_peered":"2024-09-15T01:51:30.520465+0000","last_unstale":"2024-09-15T01:51:32.664674+0000","last_undegraded":"2024-09-15T01:51:32.664674+0000","last_fullsized":"2024-09-15T01:51:32.664674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:51:25.079633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":47777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":42821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:51:36.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T01:51:36.375 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T01:51:36.375 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T01:51:36.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.469+0000 7f5649c80640 10 monclient: get_monmap_and_config 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.469+0000 7f5649c80640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.469+0000 7f5649c80640 10 monclient: monmap: 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:36.470062+0000 2024-09-15T01:51:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:36.470062+0000 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:36.472 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.469+0000 7f5649c80640 1 Processor -- start 2024-09-15T01:51:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 -- start start 2024-09-15T01:51:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient: init 2024-09-15T01:51:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:51:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5644131b20 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 0x7f5644131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-c con 0x7f56440a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-b con 0x7f5644132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56440a0040 con 0x7f5644131b20 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564409bff0 con 0x7f56440a0280 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56440a14a0 con 0x7f5644132430 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5649c80640 10 monclient(hunting): authenticate will time out at 756.466539s 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59838/0 (socket says 172.21.15.17:59838) 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 -- 172.21.15.17:0/2100227305 learned_addr learned my addr 172.21.15.17:0/2100227305 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): get_auth_request con 0x7f5644132430 auth_method 0 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 1 --2- 172.21.15.17:0/2100227305 >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 0x7f5644131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 10 monclient(hunting): get_auth_request con 0x7f56440a0280 auth_method 0 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 -- 172.21.15.17:0/2100227305 >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 msgr2=0x7f5644131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 --2- 172.21.15.17:0/2100227305 >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 0x7f5644131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 -- 172.21.15.17:0/2100227305 >> v1:172.21.15.67:6789/0 conn(0x7f5644131b20 legacy=0x7f564409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 monclient: found mon.noname-b 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5643fff640 1 -- 172.21.15.17:0/2100227305 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564409f3b0 con 0x7f5644132430 2024-09-15T01:51:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.470+0000 7f5642ffd640 1 --2- 172.21.15.17:0/2100227305 >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 0x7f5644131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:36.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.473+0000 7f5643fff640 1 -- 172.21.15.17:0/2100227305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 msgr2=0x7f5644138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.473+0000 7f5643fff640 1 --2- 172.21.15.17:0/2100227305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f563c009f20 tx=0x7f563c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e34cb12ff2da832a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 1 -- 172.21.15.17:0/2100227305 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c007610 con 0x7f5644132430 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: dump: 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.488 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 1 -- 172.21.15.17:0/2100227305 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c02d050 con 0x7f5644132430 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 5 monclient: authenticate success, global_id 4310 2024-09-15T01:51:36.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 1 -- 172.21.15.17:0/2100227305 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0079a0 con 0x7f5644132430 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 4 set_mon_vals no callback set 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f56427fc640 10 monclient: dump: 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.490 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 10 monclient: shutdown 2024-09-15T01:51:36.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 -- 172.21.15.17:0/2100227305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 msgr2=0x7f5644138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 --2- 172.21.15.17:0/2100227305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f563c009f20 tx=0x7f563c009840 comp rx=0 tx=0).stop 2024-09-15T01:51:36.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 -- 172.21.15.17:0/2100227305 shutdown_connections 2024-09-15T01:51:36.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 --2- 172.21.15.17:0/2100227305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644138e60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 --2- 172.21.15.17:0/2100227305 >> v2:172.21.15.67:3300/0 conn(0x7f56440a0280 0x7f5644131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 -- 172.21.15.17:0/2100227305 >> 172.21.15.17:0/2100227305 conn(0x7f5644071270 msgr2=0x7f56440a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 -- 172.21.15.17:0/2100227305 shutdown_connections 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.474+0000 7f5649c80640 1 -- 172.21.15.17:0/2100227305 wait complete. 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.475+0000 7f5649c80640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.475+0000 7f5649c80640 10 monclient: monmap: 2024-09-15T01:51:36.493 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:36.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:36.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.475+0000 7f5649c80640 1 Processor -- start 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.475+0000 7f5649c80640 1 -- start start 2024-09-15T01:51:36.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.475+0000 7f5649c80640 10 monclient: init 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 0x7f5644138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-b con 0x7f56440a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-a con 0x7f5644131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644133640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): picked mon.noname-c con 0x7f5644132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5644136cb0 con 0x7f5644131b20 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440a09b0 con 0x7f56440a0280 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5644131580 con 0x7f5644132430 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5649c80640 10 monclient(hunting): authenticate will time out at 756.472419s 2024-09-15T01:51:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644133640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39158/0 (socket says 172.21.15.17:39158) 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 -- 172.21.15.17:0/2599561926 learned_addr learned my addr 172.21.15.17:0/2599561926 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): get_auth_request con 0x7f5644131b20 auth_method 0 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): get_auth_request con 0x7f5644132430 auth_method 0 2024-09-15T01:51:36.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5643fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 0x7f5644138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 10 monclient(hunting): get_auth_request con 0x7f56440a0280 auth_method 0 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 -- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 msgr2=0x7f5644133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644133640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 -- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 msgr2=0x7f5644138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 0x7f5644138260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 monclient: found mon.noname-a 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f5642ffd640 1 -- 172.21.15.17:0/2599561926 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c009ae0 con 0x7f5644131b20 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.476+0000 7f56437fe640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 0x7f5644138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5642ffd640 1 -- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 msgr2=0x7f5644133100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5642ffd640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f563800d8c0 tx=0x7f5638008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800deb0 con 0x7f5644131b20 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: dump: 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.501 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800bae0 con 0x7f5644131b20 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800bea0 con 0x7f5644131b20 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 5 monclient: authenticate success, global_id 4311 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 10 monclient: _renew_subs 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5644135020 con 0x7f5644131b20 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5641ffb640 4 set_mon_vals no callback set 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f562b7fe640 10 monclient: dump: 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 10 monclient: _renew_subs 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.479+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5644137340 con 0x7f5644131b20 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f5649c80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f5649c80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5638014640 con 0x7f5644131b20 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f5649c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56440724a0 con 0x7f5644131b20 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f562b7fe640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 0x7f563003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f563804c030 con 0x7f5644131b20 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.480+0000 7f56437fe640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 0x7f563003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.481+0000 7f56437fe640 10 monclient: get_auth_request con 0x7f5630038740 auth_method 0 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.483+0000 7f56437fe640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 0x7f563003abf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5634002860 tx=0x7f5634009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.484+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f563807de40 con 0x7f5644131b20 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.484+0000 7f562b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.484+0000 7f562b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f5649c80640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f5649c80640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f5649c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56441362f0 con 0x7f5644131b20 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f562b7fe640 1 -- 172.21.15.17:0/2599561926 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5638011570 con 0x7f5644131b20 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f562b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T01:51:36.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.679+0000 7f562b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:36.682 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:36.682 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T01:51:36.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 msgr2=0x7f563003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 0x7f563003abf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5634002860 tx=0x7f5634009040 comp rx=0 tx=0).stop 2024-09-15T01:51:36.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 10 monclient: shutdown 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 msgr2=0x7f5644133100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f563800d8c0 tx=0x7f5638008040 comp rx=0 tx=0).stop 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 shutdown_connections 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5630038740 0x7f563003abf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5644131b20 0x7f5644133100 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5644132430 0x7f5644133640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 --2- 172.21.15.17:0/2599561926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56440a0280 0x7f5644138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 >> 172.21.15.17:0/2599561926 conn(0x7f5644071270 msgr2=0x7f56440a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 shutdown_connections 2024-09-15T01:51:36.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.681+0000 7f5649c80640 1 -- 172.21.15.17:0/2599561926 wait complete. 2024-09-15T01:51:36.694 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T01:51:36.695 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-15T01:51:36.711 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-15T01:51:36.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 10 monclient: get_monmap_and_config 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 10 monclient: monmap: 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:36.806903+0000 2024-09-15T01:51:36.808 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:36.806903+0000 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:36.809 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 1 Processor -- start 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 1 -- start start 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.806+0000 7ff7d4a34640 10 monclient: init 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7d0131b20 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d0132430 0x7ff7d0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d0132430 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00a0040 con 0x7ff7d0131b20 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d009bff0 con 0x7ff7d0132430 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00a14a0 con 0x7ff7d00a0280 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7d4a34640 10 monclient(hunting): authenticate will time out at 756.803518s 2024-09-15T01:51:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59866/0 (socket says 172.21.15.17:59866) 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/75114460 learned_addr learned my addr 172.21.15.17:0/75114460 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): get_auth_request con 0x7ff7d00a0280 auth_method 0 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7ced9d640 1 --2- 172.21.15.17:0/75114460 >> v2:172.21.15.67:3300/0 conn(0x7ff7d0132430 0x7ff7d0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7ced9d640 10 monclient(hunting): get_auth_request con 0x7ff7d0132430 auth_method 0 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/75114460 >> v2:172.21.15.67:3300/0 conn(0x7ff7d0132430 msgr2=0x7ff7d0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 --2- 172.21.15.17:0/75114460 >> v2:172.21.15.67:3300/0 conn(0x7ff7d0132430 0x7ff7d0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/75114460 >> v1:172.21.15.67:6789/0 conn(0x7ff7d0131b20 legacy=0x7ff7d009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 monclient: found mon.noname-b 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.807+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/75114460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d009f3b0 con 0x7ff7d00a0280 2024-09-15T01:51:36.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/75114460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 msgr2=0x7ff7d0131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cdd9b640 1 --2- 172.21.15.17:0/75114460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009910 tx=0x7ff7b80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b74a8c1221530d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 1 -- 172.21.15.17:0/75114460 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b802aea0 con 0x7ff7d00a0280 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: dump: 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.822 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 1 -- 172.21.15.17:0/75114460 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8032440 con 0x7ff7d00a0280 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 1 -- 172.21.15.17:0/75114460 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b80327e0 con 0x7ff7d00a0280 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7d4a34640 5 monclient: authenticate success, global_id 4316 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7d4a34640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:36.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.810+0000 7ff7d4a34640 4 set_mon_vals no callback set 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7cd59a640 10 monclient: dump: 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.824 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 10 monclient: shutdown 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 -- 172.21.15.17:0/75114460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 msgr2=0x7ff7d0131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/75114460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009910 tx=0x7ff7b80091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 -- 172.21.15.17:0/75114460 shutdown_connections 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/75114460 >> v2:172.21.15.67:3300/0 conn(0x7ff7d0132430 0x7ff7d0138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/75114460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d00a0280 0x7ff7d0131ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 -- 172.21.15.17:0/75114460 >> 172.21.15.17:0/75114460 conn(0x7ff7d0071270 msgr2=0x7ff7d00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 -- 172.21.15.17:0/75114460 shutdown_connections 2024-09-15T01:51:36.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.811+0000 7ff7d4a34640 1 -- 172.21.15.17:0/75114460 wait complete. 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient: build_initial_monmap 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient: monmap: 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:36.826 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 1 Processor -- start 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 1 -- start start 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient: init 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00a0280 0x7ff7d0132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d00a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d0131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.812+0000 7ff7d4a34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7d0132430 0x7ff7d0133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d0132430 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0135c20 con 0x7ff7d0132430 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00a09b0 con 0x7ff7d00a0280 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0138f60 con 0x7ff7d0131b20 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7d4a34640 10 monclient(hunting): authenticate will time out at 756.809038s 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59868/0 (socket says 172.21.15.17:59868) 2024-09-15T01:51:36.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/3970201496 learned_addr learned my addr 172.21.15.17:0/3970201496 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): get_auth_request con 0x7ff7d0131b20 auth_method 0 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7d0132430 0x7ff7d0133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): get_auth_request con 0x7ff7d0132430 auth_method 0 2024-09-15T01:51:36.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00a0280 0x7ff7d0132900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): get_auth_request con 0x7ff7d00a0280 auth_method 0 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00a0280 msgr2=0x7ff7d0132900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00a0280 0x7ff7d0132900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7d0132430 msgr2=0x7ff7d0133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7d0132430 0x7ff7d0133380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 monclient: found mon.noname-c 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.813+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/3970201496 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8009450 con 0x7ff7d0131b20 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7cdd9b640 1 -- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 msgr2=0x7ff7d0132e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7cdd9b640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009970 tx=0x7ff7b8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b802aea0 con 0x7ff7d0131b20 2024-09-15T01:51:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: dump: 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 5 monclient: authenticate success, global_id 4319 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8031c70 con 0x7ff7d0131b20 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 10 monclient: _renew_subs 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d0135ed0 con 0x7ff7d0131b20 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b803d440 con 0x7ff7d0131b20 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7c67fc640 10 monclient: dump: 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:36.834 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 10 monclient: _renew_subs 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d0134950 con 0x7ff7d0131b20 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.815+0000 7ff7ccd99640 4 set_mon_vals no callback set 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.816+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff7b8039040 con 0x7ff7d0131b20 2024-09-15T01:51:36.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7d4a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7d4a34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7d4a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d00724a0 con 0x7ff7d0131b20 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7c67fc640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 0x7ff7b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7ff7b806b780 con 0x7ff7d0131b20 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7ce59c640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 0x7ff7b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.817+0000 7ff7ce59c640 10 monclient: get_auth_request con 0x7ff7b0038740 auth_method 0 2024-09-15T01:51:36.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.819+0000 7ff7ce59c640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 0x7ff7b003abf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007fd0 tx=0x7ff7c8007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:36.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.821+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff7b802f2a0 con 0x7ff7d0131b20 2024-09-15T01:51:36.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.821+0000 7ff7c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:36.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.821+0000 7ff7c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:36.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.987+0000 7ff7d4a34640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:36.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.987+0000 7ff7d4a34640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:36.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.987+0000 7ff7d4a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:36.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.987+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d0135920 con 0x7ff7d0131b20 2024-09-15T01:51:36.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.988+0000 7ff7c67fc640 1 -- 172.21.15.17:0/3970201496 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11380 (secure 0 0 0) 0x7ff7b8037090 con 0x7ff7d0131b20 2024-09-15T01:51:36.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.988+0000 7ff7c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T01:51:36.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.988+0000 7ff7c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:36.990 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:36.990 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":16,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T01:51:32.561543+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:51:19.635512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T01:51:28.688141+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T01:51:29.355809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.593725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.627101+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.602809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.641423+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.589740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.592626+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T01:51:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 msgr2=0x7ff7b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 0x7ff7b003abf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007fd0 tx=0x7ff7c8007e10 comp rx=0 tx=0).stop 2024-09-15T01:51:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 10 monclient: shutdown 2024-09-15T01:51:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 msgr2=0x7ff7d0132e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009970 tx=0x7ff7b8032000 comp rx=0 tx=0).stop 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 shutdown_connections 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7b0038740 0x7ff7b003abf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7d0132430 0x7ff7d0133380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7d0131b20 0x7ff7d0132e40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 --2- 172.21.15.17:0/3970201496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00a0280 0x7ff7d0132900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 >> 172.21.15.17:0/3970201496 conn(0x7ff7d0071270 msgr2=0x7ff7d0131410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 shutdown_connections 2024-09-15T01:51:36.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:36.990+0000 7ff7d4a34640 1 -- 172.21.15.17:0/3970201496 wait complete. 2024-09-15T01:51:37.002 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-15T01:51:28.688141+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-15T01:51:29.355809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-15T01:51:37.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-15T01:51:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 10 monclient: monmap: 2024-09-15T01:51:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:37.101000+0000 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:37.101000+0000 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 1 Processor -- start 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 1 -- start start 2024-09-15T01:51:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.100+0000 7f8d5126f640 10 monclient: init 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d4c131910 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 0x7f8d4c09ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d4c09eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d4c136c80 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c1326d0 con 0x7f8d4c131910 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c131ce0 con 0x7f8d4c136c80 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0a15e0 con 0x7f8d4c09eb30 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d5126f640 10 monclient(hunting): authenticate will time out at 757.097325s 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4affd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8d4c131910 0x7f8d4c131fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33742/0 (socket says 172.21.15.17:33742) 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 0x7f8d4c09ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4affd640 1 -- 172.21.15.17:0/3826977054 learned_addr learned my addr 172.21.15.17:0/3826977054 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): get_auth_request con 0x7f8d4c09eb30 auth_method 0 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 1 -- 172.21.15.17:0/3826977054 >> v1:172.21.15.67:6789/0 conn(0x7f8d4c136c80 legacy=0x7f8d4c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 1 -- 172.21.15.17:0/3826977054 >> v1:172.21.15.17:6789/0 conn(0x7f8d4c131910 legacy=0x7f8d4c131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 1 monclient: found mon.noname-b 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.101+0000 7f8d4a7fc640 1 -- 172.21.15.17:0/3826977054 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c09fef0 con 0x7f8d4c09eb30 2024-09-15T01:51:37.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.104+0000 7f8d4a7fc640 1 -- 172.21.15.17:0/3826977054 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 msgr2=0x7f8d4c09ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.104+0000 7f8d4a7fc640 1 --2- 172.21.15.17:0/3826977054 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 0x7f8d4c09ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0a2eb0 tx=0x7f8d340090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b52f646b20188ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.104+0000 7f8d49ffb640 1 -- 172.21.15.17:0/3826977054 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34032440 con 0x7f8d4c09eb30 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.104+0000 7f8d49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.104+0000 7f8d49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: dump: 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.111 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 1 -- 172.21.15.17:0/3826977054 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d340325a0 con 0x7f8d4c09eb30 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 5 monclient: authenticate success, global_id 4322 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:37.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 4 set_mon_vals no callback set 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 1 -- 172.21.15.17:0/3826977054 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34032940 con 0x7f8d4c09eb30 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d49ffb640 10 monclient: dump: 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.113 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 10 monclient: shutdown 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 -- 172.21.15.17:0/3826977054 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 msgr2=0x7f8d4c09ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3826977054 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 0x7f8d4c09ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0a2eb0 tx=0x7f8d340090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 -- 172.21.15.17:0/3826977054 shutdown_connections 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3826977054 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c09eb30 0x7f8d4c09ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 -- 172.21.15.17:0/3826977054 >> 172.21.15.17:0/3826977054 conn(0x7f8d4c071470 msgr2=0x7f8d4c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 -- 172.21.15.17:0/3826977054 shutdown_connections 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.105+0000 7f8d5126f640 1 -- 172.21.15.17:0/3826977054 wait complete. 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.106+0000 7f8d5126f640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.106+0000 7f8d5126f640 10 monclient: monmap: 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.106+0000 7f8d5126f640 1 Processor -- start 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.106+0000 7f8d5126f640 1 -- start start 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.106+0000 7f8d5126f640 10 monclient: init 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-a con 0x7f8d4c09eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 0x7f8d4c096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d4c131910 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 0x7f8d4c096f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): picked mon.noname-c con 0x7f8d4c136c80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c09bf70 con 0x7f8d4c09eb30 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0980f0 con 0x7f8d4c131910 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c13a400 con 0x7f8d4c136c80 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d5126f640 10 monclient(hunting): authenticate will time out at 757.103131s 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39182/0 (socket says 172.21.15.17:39182) 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 1 -- 172.21.15.17:0/3407174460 learned_addr learned my addr 172.21.15.17:0/3407174460 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): get_auth_request con 0x7f8d4c09eb30 auth_method 0 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 0x7f8d4c096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d4c136c80 auth_method 0 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4a7fc640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 0x7f8d4c096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 -- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 msgr2=0x7f8d4c096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 0x7f8d4c096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 -- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 msgr2=0x7f8d4c09ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:51:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4b7fe640 1 -- 172.21.15.17:0/3407174460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34009390 con 0x7f8d4c136c80 2024-09-15T01:51:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4a7fc640 10 monclient: get_auth_request con 0x7f8d4c131910 auth_method 0 2024-09-15T01:51:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4affd640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.107+0000 7f8d4a7fc640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 0x7f8d4c096a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.109+0000 7f8d4b7fe640 1 -- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 msgr2=0x7f8d4c096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.109+0000 7f8d4b7fe640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 0x7f8d4c096f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00c460 tx=0x7f8d3c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.109+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0117a0 con 0x7f8d4c136c80 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.109+0000 7f8d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: dump: 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c011900 con 0x7f8d4c136c80 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 5 monclient: authenticate success, global_id 4325 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 10 monclient: _renew_subs 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c099e00 con 0x7f8d4c136c80 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c011ca0 con 0x7f8d4c136c80 2024-09-15T01:51:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 10 monclient: dump: 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 10 monclient: _renew_subs 2024-09-15T01:51:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d48ff9640 4 set_mon_vals no callback set 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c098bc0 con 0x7f8d4c136c80 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.110+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d3c012630 con 0x7f8d4c136c80 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d5126f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d5126f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d5126f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c0724a0 con 0x7f8d4c136c80 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d42ffd640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 0x7f8d2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8d3c04b550 con 0x7f8d4c136c80 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d4affd640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 0x7f8d2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.111+0000 7f8d4affd640 10 monclient: get_auth_request con 0x7f8d2c038740 auth_method 0 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.114+0000 7f8d4affd640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 0x7f8d2c03abf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d7e0 tx=0x7f8d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.115+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d3c0128e0 con 0x7f8d4c136c80 2024-09-15T01:51:37.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.115+0000 7f8d42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.115+0000 7f8d42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.278+0000 7f8d5126f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.278+0000 7f8d5126f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.278+0000 7f8d5126f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.278+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8d4c1324b0 con 0x7f8d4c136c80 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.279+0000 7f8d42ffd640 1 -- 172.21.15.17:0/3407174460 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8d3c01beb0 con 0x7f8d4c136c80 2024-09-15T01:51:37.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.279+0000 7f8d42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T01:51:37.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.279+0000 7f8d42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:37.282 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 8 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 msgr2=0x7f8d2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 0x7f8d2c03abf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d7e0 tx=0x7f8d44009040 comp rx=0 tx=0).stop 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 10 monclient: shutdown 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 msgr2=0x7f8d4c096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 0x7f8d4c096f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00c460 tx=0x7f8d3c007e10 comp rx=0 tx=0).stop 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 shutdown_connections 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d2c038740 0x7f8d2c03abf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d4c136c80 0x7f8d4c096f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d4c131910 0x7f8d4c096a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 --2- 172.21.15.17:0/3407174460 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d4c09eb30 0x7f8d4c09ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 >> 172.21.15.17:0/3407174460 conn(0x7f8d4c071470 msgr2=0x7f8d4c09e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 shutdown_connections 2024-09-15T01:51:37.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.281+0000 7f8d5126f640 1 -- 172.21.15.17:0/3407174460 wait complete. 2024-09-15T01:51:37.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T01:51:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.390+0000 7ff6ffb9d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.390+0000 7ff6ffb9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.390+0000 7ff6ffb9d640 10 monclient: monmap: 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:37.391648+0000 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:37.391648+0000 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:37.394 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.391+0000 7ff6ffb9d640 1 Processor -- start 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.391+0000 7ff6ffb9d640 1 -- start start 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.391+0000 7ff6ffb9d640 10 monclient: init 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f80a5aa0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff6f809eba0 0x7ff6f809ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f809eba0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 0x7ff6f80692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f8068e80 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f809fdf0 con 0x7ff6f809eba0 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80a18c0 con 0x7ff6f8068e80 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f809c660 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6ffb9d640 10 monclient(hunting): authenticate will time out at 757.388138s 2024-09-15T01:51:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff6f809eba0 0x7ff6f809ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff6f809eba0 0x7ff6f809ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39188/0 (socket says 172.21.15.17:39188) 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 -- 172.21.15.17:0/2672667068 learned_addr learned my addr 172.21.15.17:0/2672667068 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59894/0 (socket says 172.21.15.17:59894) 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): get_auth_request con 0x7ff6f80a5aa0 auth_method 0 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): get_auth_request con 0x7ff6f809eba0 auth_method 0 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 1 --2- 172.21.15.17:0/2672667068 >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 0x7ff6f80692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 10 monclient(hunting): get_auth_request con 0x7ff6f8068e80 auth_method 0 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 -- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 msgr2=0x7ff6f8131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 -- 172.21.15.17:0/2672667068 >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 msgr2=0x7ff6f80692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 --2- 172.21.15.17:0/2672667068 >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 0x7ff6f80692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 monclient: found mon.noname-a 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd138640 1 -- 172.21.15.17:0/2672667068 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f8132650 con 0x7ff6f809eba0 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fd939640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.392+0000 7ff6fe13a640 1 --2- 172.21.15.17:0/2672667068 >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 0x7ff6f80692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fd138640 1 -- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f809eba0 msgr2=0x7ff6f809ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fd138640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f809eba0 0x7ff6f809ef70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80a0800 tx=0x7ff6e80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed2253811ce314b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 1 -- 172.21.15.17:0/2672667068 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8032440 con 0x7ff6f809eba0 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: dump: 2024-09-15T01:51:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 1 -- 172.21.15.17:0/2672667068 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e80325a0 con 0x7ff6f809eba0 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6ffb9d640 5 monclient: authenticate success, global_id 4320 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6ffb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6ffb9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 1 -- 172.21.15.17:0/2672667068 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8032940 con 0x7ff6f809eba0 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6ffb9d640 4 set_mon_vals no callback set 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.395+0000 7ff6fc937640 10 monclient: dump: 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 10 monclient: shutdown 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f809eba0 msgr2=0x7ff6f809ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f809eba0 0x7ff6f809ef70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80a0800 tx=0x7ff6e80091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/2672667068 shutdown_connections 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/2672667068 >> v2:172.21.15.67:3300/0 conn(0x7ff6f8068e80 0x7ff6f80692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f809eba0 0x7ff6f809ef70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/2672667068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f80a5aa0 0x7ff6f8131f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/2672667068 >> 172.21.15.17:0/2672667068 conn(0x7ff6f8071010 msgr2=0x7ff6f80a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/2672667068 shutdown_connections 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/2672667068 wait complete. 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 10 monclient: monmap: 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.411 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.396+0000 7ff6ffb9d640 1 Processor -- start 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 -- start start 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient: init 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f8068e80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 0x7ff6f8130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f809eba0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 0x7ff6f8140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f80a5aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f811dc50 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f811df30 con 0x7ff6f809eba0 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f811e210 con 0x7ff6f8068e80 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6ffb9d640 10 monclient(hunting): authenticate will time out at 757.393506s 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 0x7ff6f8140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59906/0 (socket says 172.21.15.17:59906) 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 1 -- 172.21.15.17:0/982946793 learned_addr learned my addr 172.21.15.17:0/982946793 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): get_auth_request con 0x7ff6f8068e80 auth_method 0 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): get_auth_request con 0x7ff6f80a5aa0 auth_method 0 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd138640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 0x7ff6f8130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 -- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 msgr2=0x7ff6f809a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 -- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 msgr2=0x7ff6f8130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 0x7ff6f8130b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 monclient: found mon.noname-a 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fe13a640 1 -- 172.21.15.17:0/982946793 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4009390 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd138640 10 monclient: get_auth_request con 0x7ff6f809eba0 auth_method 0 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd939640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.397+0000 7ff6fd138640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 0x7ff6f8130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:51:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6fe13a640 1 -- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 msgr2=0x7ff6f8140f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6fe13a640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 0x7ff6f8140f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0007f60 tx=0x7ff6f00098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000de20 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: dump: 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 5 monclient: authenticate success, global_id 4323 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 10 monclient: _renew_subs 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f000f040 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e8009450 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f00127b0 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6edffb640 10 monclient: dump: 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.420 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6effff640 4 set_mon_vals no callback set 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 10 monclient: _renew_subs 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.400+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f811bc00 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff6f0012be0 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6ffb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6ffb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6ffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f811e5d0 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6edffb640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 0x7ff6dc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.401+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7ff6f004bf50 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.402+0000 7ff6fd939640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 0x7ff6dc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.402+0000 7ff6fd939640 10 monclient: get_auth_request con 0x7ff6dc038790 auth_method 0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.404+0000 7ff6fd939640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 0x7ff6dc03ac40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4011640 tx=0x7ff6f40090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.406+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff6f0002880 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.406+0000 7ff6edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.406+0000 7ff6edffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:37.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6ffb9d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T01:51:37.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6ffb9d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T01:51:37.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6ffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:37.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6f80724a0 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6edffb640 1 -- 172.21.15.17:0/982946793 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6f001a020 con 0x7ff6f80a5aa0 2024-09-15T01:51:37.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T01:51:37.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.571+0000 7ff6edffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:37.574 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2024-09-15T01:51:37.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 msgr2=0x7ff6dc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 0x7ff6dc03ac40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4011640 tx=0x7ff6f40090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 10 monclient: shutdown 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 msgr2=0x7ff6f8140f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 0x7ff6f8140f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0007f60 tx=0x7ff6f00098f0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 shutdown_connections 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff6dc038790 0x7ff6dc03ac40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6f80a5aa0 0x7ff6f8140f70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f809eba0 0x7ff6f8130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 --2- 172.21.15.17:0/982946793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6f8068e80 0x7ff6f809a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 >> 172.21.15.17:0/982946793 conn(0x7ff6f8071010 msgr2=0x7ff6f809cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 shutdown_connections 2024-09-15T01:51:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.573+0000 7ff6ffb9d640 1 -- 172.21.15.17:0/982946793 wait complete. 2024-09-15T01:51:37.586 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-15T01:51:37.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T01:51:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.680+0000 7f26f8c3f640 10 monclient: get_monmap_and_config 2024-09-15T01:51:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.680+0000 7f26f8c3f640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.680+0000 7f26f8c3f640 10 monclient: monmap: 2024-09-15T01:51:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:37.681279+0000 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:37.681279+0000 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:37.683 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 Processor -- start 2024-09-15T01:51:37.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 -- start start 2024-09-15T01:51:37.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient: init 2024-09-15T01:51:37.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26f40a5780 0x7f26f409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f26f40a5780 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f4069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f26f409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26f4069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f40a5b50 con 0x7f26f4069ac0 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f40a18b0 con 0x7f26f40a5780 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f409c5f0 con 0x7f26f409ccf0 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f8c3f640 10 monclient(hunting): authenticate will time out at 757.677806s 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f4069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f2d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f26f4069ac0 0x7f26f4138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33752/0 (socket says 172.21.15.17:33752) 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f2d9d640 1 -- 172.21.15.17:0/2261991045 learned_addr learned my addr 172.21.15.17:0/2261991045 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f1d9b640 10 monclient(hunting): get_auth_request con 0x7f26f409ccf0 auth_method 0 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.681+0000 7f26f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f259c640 1 --2- 172.21.15.17:0/2261991045 >> v2:172.21.15.67:3300/0 conn(0x7f26f40a5780 0x7f26f409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f259c640 10 monclient(hunting): get_auth_request con 0x7f26f40a5780 auth_method 0 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2261991045 >> v2:172.21.15.67:3300/0 conn(0x7f26f40a5780 msgr2=0x7f26f409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 1 --2- 172.21.15.17:0/2261991045 >> v2:172.21.15.67:3300/0 conn(0x7f26f40a5780 0x7f26f409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2261991045 >> v1:172.21.15.17:6789/0 conn(0x7f26f4069ac0 legacy=0x7f26f4138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 1 monclient: found mon.noname-b 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.682+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2261991045 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f409d210 con 0x7f26f409ccf0 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2261991045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 msgr2=0x7f26f4069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f1d9b640 1 --2- 172.21.15.17:0/2261991045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f4069580 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009850 tx=0x7f26e40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c82213f1ca11b615 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 1 -- 172.21.15.17:0/2261991045 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e402adf0 con 0x7f26f409ccf0 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: dump: 2024-09-15T01:51:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 1 -- 172.21.15.17:0/2261991045 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4032440 con 0x7f26f409ccf0 2024-09-15T01:51:37.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 1 -- 172.21.15.17:0/2261991045 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4032820 con 0x7f26f409ccf0 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 5 monclient: authenticate success, global_id 4334 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 4 set_mon_vals no callback set 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f159a640 10 monclient: dump: 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 10 monclient: shutdown 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2261991045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 msgr2=0x7f26f4069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2261991045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f4069580 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009850 tx=0x7f26e40090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2261991045 shutdown_connections 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2261991045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f4069580 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2261991045 >> v2:172.21.15.67:3300/0 conn(0x7f26f40a5780 0x7f26f409bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2261991045 >> 172.21.15.17:0/2261991045 conn(0x7f26f4070e10 msgr2=0x7f26f40a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2261991045 shutdown_connections 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.685+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2261991045 wait complete. 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.686+0000 7f26f8c3f640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.686+0000 7f26f8c3f640 10 monclient: monmap: 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.686+0000 7f26f8c3f640 1 Processor -- start 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.686+0000 7f26f8c3f640 1 -- start start 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.686+0000 7f26f8c3f640 10 monclient: init 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f4069ac0 0x7f26f413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f26f4069ac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f26f409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f40a5780 0x7f26f4141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f26f40a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4122e90 con 0x7f26f40a5780 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f41228d0 con 0x7f26f4069ac0 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f41203b0 con 0x7f26f409ccf0 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f8c3f640 10 monclient(hunting): authenticate will time out at 757.683325s 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59924/0 (socket says 172.21.15.17:59924) 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2298382274 learned_addr learned my addr 172.21.15.17:0/2298382274 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): get_auth_request con 0x7f26f409ccf0 auth_method 0 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f40a5780 0x7f26f4141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): get_auth_request con 0x7f26f40a5780 auth_method 0 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f259c640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f4069ac0 0x7f26f413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f259c640 10 monclient(hunting): get_auth_request con 0x7f26f4069ac0 auth_method 0 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f4069ac0 msgr2=0x7f26f413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f4069ac0 0x7f26f413dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f40a5780 msgr2=0x7f26f4141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f40a5780 0x7f26f4141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 monclient: found mon.noname-c 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.687+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2298382274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4009390 con 0x7f26f409ccf0 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f1d9b640 1 -- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 msgr2=0x7f26f413d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f1d9b640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f26e40098b0 tx=0x7f26e4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4031a40 con 0x7f26f409ccf0 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: dump: 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.702 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 5 monclient: authenticate success, global_id 4337 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4031ba0 con 0x7f26f409ccf0 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 10 monclient: _renew_subs 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e403c400 con 0x7f26f409ccf0 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41225f0 con 0x7f26f409ccf0 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f0d99640 4 set_mon_vals no callback set 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26ea7fc640 10 monclient: dump: 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 10 monclient: _renew_subs 2024-09-15T01:51:37.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.690+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f413da90 con 0x7f26f409ccf0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f26e403c830 con 0x7f26f409ccf0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f8c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f8c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f8c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f413dc80 con 0x7f26f409ccf0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26ea7fc640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 0x7f26d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f26e406c350 con 0x7f26f409ccf0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f259c640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 0x7f26d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.691+0000 7f26f259c640 10 monclient: get_auth_request con 0x7f26d4038740 auth_method 0 2024-09-15T01:51:37.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.694+0000 7f26f259c640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 0x7f26d403abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0075a0 tx=0x7f26dc0097a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.695+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f26e40426c0 con 0x7f26f409ccf0 2024-09-15T01:51:37.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.695+0000 7f26ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.695+0000 7f26ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:37.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.854+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26f4123000 con 0x7f26d4038740 2024-09-15T01:51:37.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.856+0000 7f26ea7fc640 1 -- 172.21.15.17:0/2298382274 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40986 (secure 0 0 0) 0x7f26f4123000 con 0x7f26d4038740 2024-09-15T01:51:37.858 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:37.858 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 msgr2=0x7f26d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 0x7f26d403abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0075a0 tx=0x7f26dc0097a0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 10 monclient: shutdown 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 msgr2=0x7f26f413d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f26e40098b0 tx=0x7f26e4032000 comp rx=0 tx=0).stop 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 shutdown_connections 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26d4038740 0x7f26d403abf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f40a5780 0x7f26f4141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f409ccf0 0x7f26f413d550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 --2- 172.21.15.17:0/2298382274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f4069ac0 0x7f26f413dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 >> 172.21.15.17:0/2298382274 conn(0x7f26f4070e10 msgr2=0x7f26f4098d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 shutdown_connections 2024-09-15T01:51:37.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.857+0000 7f26f8c3f640 1 -- 172.21.15.17:0/2298382274 wait complete. 2024-09-15T01:51:37.871 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-15T01:51:37.307759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786352,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444688,"statfs":{"total":579820584960,"available":577991360512,"internally_reserved":0,"allocated":1994752,"data_stored":1339600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.741504"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717899+0000","last_change":"2024-09-15T01:51:32.718180+0000","last_active":"2024-09-15T01:51:32.717899+0000","last_peered":"2024-09-15T01:51:32.717899+0000","last_clean":"2024-09-15T01:51:32.717899+0000","last_became_active":"2024-09-15T01:51:30.517374+0000","last_became_peered":"2024-09-15T01:51:30.517374+0000","last_unstale":"2024-09-15T01:51:32.717899+0000","last_undegraded":"2024-09-15T01:51:32.717899+0000","last_fullsized":"2024-09-15T01:51:32.717899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:49.285042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664627+0000","last_change":"2024-09-15T01:51:32.664815+0000","last_active":"2024-09-15T01:51:32.664627+0000","last_peered":"2024-09-15T01:51:32.664627+0000","last_clean":"2024-09-15T01:51:32.664627+0000","last_became_active":"2024-09-15T01:51:30.519255+0000","last_became_peered":"2024-09-15T01:51:30.519255+0000","last_unstale":"2024-09-15T01:51:32.664627+0000","last_undegraded":"2024-09-15T01:51:32.664627+0000","last_fullsized":"2024-09-15T01:51:32.664627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:46.999182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568082+0000","last_change":"2024-09-15T01:51:32.568282+0000","last_active":"2024-09-15T01:51:32.568082+0000","last_peered":"2024-09-15T01:51:32.568082+0000","last_clean":"2024-09-15T01:51:32.568082+0000","last_became_active":"2024-09-15T01:51:30.516534+0000","last_became_peered":"2024-09-15T01:51:30.516534+0000","last_unstale":"2024-09-15T01:51:32.568082+0000","last_undegraded":"2024-09-15T01:51:32.568082+0000","last_fullsized":"2024-09-15T01:51:32.568082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.027860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568144+0000","last_change":"2024-09-15T01:51:32.568412+0000","last_active":"2024-09-15T01:51:32.568144+0000","last_peered":"2024-09-15T01:51:32.568144+0000","last_clean":"2024-09-15T01:51:32.568144+0000","last_became_active":"2024-09-15T01:51:30.518571+0000","last_became_peered":"2024-09-15T01:51:30.518571+0000","last_unstale":"2024-09-15T01:51:32.568144+0000","last_undegraded":"2024-09-15T01:51:32.568144+0000","last_fullsized":"2024-09-15T01:51:32.568144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:13:16.913092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T01:51:31.575561+0000","last_change":"2024-09-15T01:51:30.519037+0000","last_active":"2024-09-15T01:51:31.575561+0000","last_peered":"2024-09-15T01:51:31.575561+0000","last_clean":"2024-09-15T01:51:31.575561+0000","last_became_active":"2024-09-15T01:51:30.518418+0000","last_became_peered":"2024-09-15T01:51:30.518418+0000","last_unstale":"2024-09-15T01:51:31.575561+0000","last_undegraded":"2024-09-15T01:51:31.575561+0000","last_fullsized":"2024-09-15T01:51:31.575561+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:24:49.620721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568092+0000","last_change":"2024-09-15T01:51:32.568425+0000","last_active":"2024-09-15T01:51:32.568092+0000","last_peered":"2024-09-15T01:51:32.568092+0000","last_clean":"2024-09-15T01:51:32.568092+0000","last_became_active":"2024-09-15T01:51:30.519207+0000","last_became_peered":"2024-09-15T01:51:30.519207+0000","last_unstale":"2024-09-15T01:51:32.568092+0000","last_undegraded":"2024-09-15T01:51:32.568092+0000","last_fullsized":"2024-09-15T01:51:32.568092+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:11.690964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568265+0000","last_change":"2024-09-15T01:51:32.568265+0000","last_active":"2024-09-15T01:51:32.568265+0000","last_peered":"2024-09-15T01:51:32.568265+0000","last_clean":"2024-09-15T01:51:32.568265+0000","last_became_active":"2024-09-15T01:51:30.519209+0000","last_became_peered":"2024-09-15T01:51:30.519209+0000","last_unstale":"2024-09-15T01:51:32.568265+0000","last_undegraded":"2024-09-15T01:51:32.568265+0000","last_fullsized":"2024-09-15T01:51:32.568265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:32.888198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717928+0000","last_change":"2024-09-15T01:51:32.718152+0000","last_active":"2024-09-15T01:51:32.717928+0000","last_peered":"2024-09-15T01:51:32.717928+0000","last_clean":"2024-09-15T01:51:32.717928+0000","last_became_active":"2024-09-15T01:51:30.518721+0000","last_became_peered":"2024-09-15T01:51:30.518721+0000","last_unstale":"2024-09-15T01:51:32.717928+0000","last_undegraded":"2024-09-15T01:51:32.717928+0000","last_fullsized":"2024-09-15T01:51:32.717928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:49.882031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664674+0000","last_change":"2024-09-15T01:51:32.664834+0000","last_active":"2024-09-15T01:51:32.664674+0000","last_peered":"2024-09-15T01:51:32.664674+0000","last_clean":"2024-09-15T01:51:32.664674+0000","last_became_active":"2024-09-15T01:51:30.520465+0000","last_became_peered":"2024-09-15T01:51:30.520465+0000","last_unstale":"2024-09-15T01:51:32.664674+0000","last_undegraded":"2024-09-15T01:51:32.664674+0000","last_fullsized":"2024-09-15T01:51:32.664674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:51:25.079633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":47777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":42821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:51:37.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 10 monclient: get_monmap_and_config 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 10 monclient: monmap: 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:37.968855+0000 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:37.968855+0000 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:37.971 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 1 Processor -- start 2024-09-15T01:51:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 1 -- start start 2024-09-15T01:51:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.968+0000 7f74b1c34640 10 monclient: init 2024-09-15T01:51:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74ac09ec30 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74ac09f000 0x7f74ac069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-c con 0x7f74ac09f000 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-b con 0x7f74ac136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac132460 con 0x7f74ac09ec30 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac09c6f0 con 0x7f74ac09f000 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac06a650 con 0x7f74ac136e50 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74b1c34640 10 monclient(hunting): authenticate will time out at 757.965335s 2024-09-15T01:51:37.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59938/0 (socket says 172.21.15.17:59938) 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 -- 172.21.15.17:0/1153485630 learned_addr learned my addr 172.21.15.17:0/1153485630 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): get_auth_request con 0x7f74ac136e50 auth_method 0 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74aaffd640 1 --2- 172.21.15.17:0/1153485630 >> v2:172.21.15.67:3300/0 conn(0x7f74ac09f000 0x7f74ac069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 -- 172.21.15.17:0/1153485630 >> v2:172.21.15.67:3300/0 conn(0x7f74ac09f000 msgr2=0x7f74ac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 --2- 172.21.15.17:0/1153485630 >> v2:172.21.15.67:3300/0 conn(0x7f74ac09f000 0x7f74ac069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 -- 172.21.15.17:0/1153485630 >> v1:172.21.15.67:6789/0 conn(0x7f74ac09ec30 legacy=0x7f74ac131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 monclient: found mon.noname-b 2024-09-15T01:51:37.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.969+0000 7f74abfff640 1 -- 172.21.15.17:0/1153485630 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0a13a0 con 0x7f74ac136e50 2024-09-15T01:51:37.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74abfff640 1 -- 172.21.15.17:0/1153485630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 msgr2=0x7f74ac139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74abfff640 1 --2- 172.21.15.17:0/1153485630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009f20 tx=0x7f74a4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec5963160fa1635 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 1 -- 172.21.15.17:0/1153485630 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4007610 con 0x7f74ac136e50 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: dump: 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74b1c34640 5 monclient: authenticate success, global_id 4340 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74b1c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 1 -- 172.21.15.17:0/1153485630 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a402d050 con 0x7f74ac136e50 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74b1c34640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 1 -- 172.21.15.17:0/1153485630 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40079a0 con 0x7f74ac136e50 2024-09-15T01:51:37.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74b1c34640 4 set_mon_vals no callback set 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.972+0000 7f74aa7fc640 10 monclient: dump: 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 10 monclient: shutdown 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 -- 172.21.15.17:0/1153485630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 msgr2=0x7f74ac139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1153485630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009f20 tx=0x7f74a4009840 comp rx=0 tx=0).stop 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 -- 172.21.15.17:0/1153485630 shutdown_connections 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1153485630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac136e50 0x7f74ac139230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1153485630 >> v2:172.21.15.67:3300/0 conn(0x7f74ac09f000 0x7f74ac069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 -- 172.21.15.17:0/1153485630 >> 172.21.15.17:0/1153485630 conn(0x7f74ac071320 msgr2=0x7f74ac0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 -- 172.21.15.17:0/1153485630 shutdown_connections 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 1 -- 172.21.15.17:0/1153485630 wait complete. 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 10 monclient: build_initial_monmap 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.973+0000 7f74b1c34640 10 monclient: monmap: 2024-09-15T01:51:37.985 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 Processor -- start 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 -- start start 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient: init 2024-09-15T01:51:37.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-c con 0x7f74ac09ec30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 0x7f74ac097d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-b con 0x7f74ac09f000 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): picked mon.noname-a con 0x7f74ac136e50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac09dc20 con 0x7f74ac136e50 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac09bec0 con 0x7f74ac09f000 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac09a930 con 0x7f74ac09ec30 2024-09-15T01:51:37.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74b1c34640 10 monclient(hunting): authenticate will time out at 757.970908s 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.974+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59950/0 (socket says 172.21.15.17:59950) 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 -- 172.21.15.17:0/1271912140 learned_addr learned my addr 172.21.15.17:0/1271912140 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39212/0 (socket says 172.21.15.17:39212) 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): get_auth_request con 0x7f74ac09ec30 auth_method 0 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): get_auth_request con 0x7f74ac136e50 auth_method 0 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 0x7f74ac097d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 10 monclient(hunting): get_auth_request con 0x7f74ac09f000 auth_method 0 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:37.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 -- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 msgr2=0x7f74ac097d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 0x7f74ac097d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 -- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 msgr2=0x7f74ac098260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 monclient: found mon.noname-c 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74ab7fe640 1 -- 172.21.15.17:0/1271912140 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a4009ae0 con 0x7f74ac09ec30 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74aaffd640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 0x7f74ac097d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.975+0000 7f74abfff640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74ab7fe640 1 -- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 msgr2=0x7f74ac0977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74ab7fe640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f749c00b5e0 tx=0x7f749c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00f7a0 con 0x7f74ac09ec30 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: dump: 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c00f920 con 0x7f74ac09ec30 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74b1c34640 5 monclient: authenticate success, global_id 4343 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74b1c34640 10 monclient: _renew_subs 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00fce0 con 0x7f74ac09ec30 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74b1c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac0995b0 con 0x7f74ac09ec30 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74937fe640 10 monclient: dump: 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:37.992 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.977+0000 7f74a97fa640 4 set_mon_vals no callback set 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.978+0000 7f74b1c34640 10 monclient: _renew_subs 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.978+0000 7f74b1c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.978+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac099c70 con 0x7f74ac09ec30 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.978+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f749c010630 con 0x7f74ac09ec30 2024-09-15T01:51:37.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74937fe640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 0x7f749803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74b1c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74b1c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74b1c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f749c04b4f0 con 0x7f74ac09ec30 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac099100 con 0x7f74ac09ec30 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74aaffd640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 0x7f749803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.979+0000 7f74aaffd640 10 monclient: get_auth_request con 0x7f7498038740 auth_method 0 2024-09-15T01:51:37.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.982+0000 7f74aaffd640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 0x7f749803abf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0686f0 tx=0x7f74a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:37.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.983+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f749c01a460 con 0x7f74ac09ec30 2024-09-15T01:51:37.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.983+0000 7f74937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:37.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:37.983+0000 7f74937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.144+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74ac0724a0 con 0x7f7498038740 2024-09-15T01:51:38.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.146+0000 7f74937fe640 1 -- 172.21.15.17:0/1271912140 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40986 (secure 0 0 0) 0x7f74ac0724a0 con 0x7f7498038740 2024-09-15T01:51:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T01:51:38.148 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 msgr2=0x7f749803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 0x7f749803abf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0686f0 tx=0x7f74a0008040 comp rx=0 tx=0).stop 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 10 monclient: shutdown 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 msgr2=0x7f74ac0977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.147+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f749c00b5e0 tx=0x7f749c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 shutdown_connections 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7498038740 0x7f749803abf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74ac136e50 0x7f74ac098260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac09f000 0x7f74ac097d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 --2- 172.21.15.17:0/1271912140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74ac09ec30 0x7f74ac0977e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 >> 172.21.15.17:0/1271912140 conn(0x7f74ac071320 msgr2=0x7f74ac1305a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 shutdown_connections 2024-09-15T01:51:38.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.148+0000 7f74b1c34640 1 -- 172.21.15.17:0/1271912140 wait complete. 2024-09-15T01:51:38.160 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-15T01:51:37.307759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786352,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444688,"statfs":{"total":579820584960,"available":577991360512,"internally_reserved":0,"allocated":1994752,"data_stored":1339600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.741504"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717899+0000","last_change":"2024-09-15T01:51:32.718180+0000","last_active":"2024-09-15T01:51:32.717899+0000","last_peered":"2024-09-15T01:51:32.717899+0000","last_clean":"2024-09-15T01:51:32.717899+0000","last_became_active":"2024-09-15T01:51:30.517374+0000","last_became_peered":"2024-09-15T01:51:30.517374+0000","last_unstale":"2024-09-15T01:51:32.717899+0000","last_undegraded":"2024-09-15T01:51:32.717899+0000","last_fullsized":"2024-09-15T01:51:32.717899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:49.285042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664627+0000","last_change":"2024-09-15T01:51:32.664815+0000","last_active":"2024-09-15T01:51:32.664627+0000","last_peered":"2024-09-15T01:51:32.664627+0000","last_clean":"2024-09-15T01:51:32.664627+0000","last_became_active":"2024-09-15T01:51:30.519255+0000","last_became_peered":"2024-09-15T01:51:30.519255+0000","last_unstale":"2024-09-15T01:51:32.664627+0000","last_undegraded":"2024-09-15T01:51:32.664627+0000","last_fullsized":"2024-09-15T01:51:32.664627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:46.999182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568082+0000","last_change":"2024-09-15T01:51:32.568282+0000","last_active":"2024-09-15T01:51:32.568082+0000","last_peered":"2024-09-15T01:51:32.568082+0000","last_clean":"2024-09-15T01:51:32.568082+0000","last_became_active":"2024-09-15T01:51:30.516534+0000","last_became_peered":"2024-09-15T01:51:30.516534+0000","last_unstale":"2024-09-15T01:51:32.568082+0000","last_undegraded":"2024-09-15T01:51:32.568082+0000","last_fullsized":"2024-09-15T01:51:32.568082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.027860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568144+0000","last_change":"2024-09-15T01:51:32.568412+0000","last_active":"2024-09-15T01:51:32.568144+0000","last_peered":"2024-09-15T01:51:32.568144+0000","last_clean":"2024-09-15T01:51:32.568144+0000","last_became_active":"2024-09-15T01:51:30.518571+0000","last_became_peered":"2024-09-15T01:51:30.518571+0000","last_unstale":"2024-09-15T01:51:32.568144+0000","last_undegraded":"2024-09-15T01:51:32.568144+0000","last_fullsized":"2024-09-15T01:51:32.568144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:13:16.913092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T01:51:31.575561+0000","last_change":"2024-09-15T01:51:30.519037+0000","last_active":"2024-09-15T01:51:31.575561+0000","last_peered":"2024-09-15T01:51:31.575561+0000","last_clean":"2024-09-15T01:51:31.575561+0000","last_became_active":"2024-09-15T01:51:30.518418+0000","last_became_peered":"2024-09-15T01:51:30.518418+0000","last_unstale":"2024-09-15T01:51:31.575561+0000","last_undegraded":"2024-09-15T01:51:31.575561+0000","last_fullsized":"2024-09-15T01:51:31.575561+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:24:49.620721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568092+0000","last_change":"2024-09-15T01:51:32.568425+0000","last_active":"2024-09-15T01:51:32.568092+0000","last_peered":"2024-09-15T01:51:32.568092+0000","last_clean":"2024-09-15T01:51:32.568092+0000","last_became_active":"2024-09-15T01:51:30.519207+0000","last_became_peered":"2024-09-15T01:51:30.519207+0000","last_unstale":"2024-09-15T01:51:32.568092+0000","last_undegraded":"2024-09-15T01:51:32.568092+0000","last_fullsized":"2024-09-15T01:51:32.568092+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:11.690964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.568265+0000","last_change":"2024-09-15T01:51:32.568265+0000","last_active":"2024-09-15T01:51:32.568265+0000","last_peered":"2024-09-15T01:51:32.568265+0000","last_clean":"2024-09-15T01:51:32.568265+0000","last_became_active":"2024-09-15T01:51:30.519209+0000","last_became_peered":"2024-09-15T01:51:30.519209+0000","last_unstale":"2024-09-15T01:51:32.568265+0000","last_undegraded":"2024-09-15T01:51:32.568265+0000","last_fullsized":"2024-09-15T01:51:32.568265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:32.888198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.717928+0000","last_change":"2024-09-15T01:51:32.718152+0000","last_active":"2024-09-15T01:51:32.717928+0000","last_peered":"2024-09-15T01:51:32.717928+0000","last_clean":"2024-09-15T01:51:32.717928+0000","last_became_active":"2024-09-15T01:51:30.518721+0000","last_became_peered":"2024-09-15T01:51:30.518721+0000","last_unstale":"2024-09-15T01:51:32.717928+0000","last_undegraded":"2024-09-15T01:51:32.717928+0000","last_fullsized":"2024-09-15T01:51:32.717928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:10:49.882031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T01:51:32.664674+0000","last_change":"2024-09-15T01:51:32.664834+0000","last_active":"2024-09-15T01:51:32.664674+0000","last_peered":"2024-09-15T01:51:32.664674+0000","last_clean":"2024-09-15T01:51:32.664674+0000","last_became_active":"2024-09-15T01:51:30.520465+0000","last_became_peered":"2024-09-15T01:51:30.520465+0000","last_unstale":"2024-09-15T01:51:32.664674+0000","last_undegraded":"2024-09-15T01:51:32.664674+0000","last_fullsized":"2024-09-15T01:51:32.664674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:51:25.079633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":47777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":143360,"data_stored":42821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298156,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073684,"statfs":{"total":96636764160,"available":96331452416,"internally_reserved":0,"allocated":688128,"data_stored":576705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":47796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T01:51:38.161 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-15T01:51:38.161 DEBUG:tasks.mon_thrash:Yielding 2024-09-15T01:51:38.162 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T01:51:38.174 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:51:38.174 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T01:51:38.175 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T01:51:38.175 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-15T01:51:38.175 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726365098, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-09-15T01:51:38.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:51:38.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:51:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:51:38.195 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T01:51:38.196 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T01:51:38.254 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T01:51:38.255 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T01:51:38.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.269+0000 7f91ea509640 10 monclient: get_monmap_and_config 2024-09-15T01:51:38.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.269+0000 7f91ea509640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.269+0000 7f91ea509640 10 monclient: monmap: 2024-09-15T01:51:38.271 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.271 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:38.270129+0000 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:38.270129+0000 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:38.272 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 Processor -- start 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 -- start start 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient: init 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-b con 0x7f91e40a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f91e40980a0 0x7f91e4098470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-a con 0x7f91e40980a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91e40989b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e40a01e0 con 0x7f91e40989b0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e40a1580 con 0x7f91e40980a0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40a5c80 con 0x7f91e40a58b0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91ea509640 10 monclient(hunting): authenticate will time out at 758.266644s 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59962/0 (socket says 172.21.15.17:59962) 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f91e40980a0 0x7f91e4098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 1 -- 172.21.15.17:0/1317379970 learned_addr learned my addr 172.21.15.17:0/1317379970 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): get_auth_request con 0x7f91e40a58b0 auth_method 0 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): get_auth_request con 0x7f91e40980a0 auth_method 0 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.270+0000 7f91e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 1 -- 172.21.15.17:0/1317379970 >> v2:172.21.15.17:3300/0 conn(0x7f91e40980a0 msgr2=0x7f91e4098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 1 --2- 172.21.15.17:0/1317379970 >> v2:172.21.15.17:3300/0 conn(0x7f91e40980a0 0x7f91e4098470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 1 -- 172.21.15.17:0/1317379970 >> v1:172.21.15.17:6789/0 conn(0x7f91e40989b0 legacy=0x7f91e4068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 1 monclient: found mon.noname-b 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.271+0000 7f91e3fff640 1 -- 172.21.15.17:0/1317379970 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e41326c0 con 0x7f91e40a58b0 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.273+0000 7f91e3fff640 1 -- 172.21.15.17:0/1317379970 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 msgr2=0x7f91e4132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.273+0000 7f91e3fff640 1 --2- 172.21.15.17:0/1317379970 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009910 tx=0x7f91d40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2ca8dd53524b956 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.273+0000 7f91e2ffd640 1 -- 172.21.15.17:0/1317379970 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4032440 con 0x7f91e40a58b0 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: dump: 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.285 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 5 monclient: authenticate success, global_id 4346 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 1 -- 172.21.15.17:0/1317379970 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d40325a0 con 0x7f91e40a58b0 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 4 set_mon_vals no callback set 2024-09-15T01:51:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 1 -- 172.21.15.17:0/1317379970 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4032940 con 0x7f91e40a58b0 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91e2ffd640 10 monclient: dump: 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.287 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 10 monclient: shutdown 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 -- 172.21.15.17:0/1317379970 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 msgr2=0x7f91e4132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 --2- 172.21.15.17:0/1317379970 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009910 tx=0x7f91d40091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 -- 172.21.15.17:0/1317379970 shutdown_connections 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 --2- 172.21.15.17:0/1317379970 >> v2:172.21.15.17:3300/0 conn(0x7f91e40980a0 0x7f91e4098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 --2- 172.21.15.17:0/1317379970 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40a58b0 0x7f91e4132040 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 -- 172.21.15.17:0/1317379970 >> 172.21.15.17:0/1317379970 conn(0x7f91e4070ed0 msgr2=0x7f91e40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 -- 172.21.15.17:0/1317379970 shutdown_connections 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.274+0000 7f91ea509640 1 -- 172.21.15.17:0/1317379970 wait complete. 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 10 monclient: monmap: 2024-09-15T01:51:38.288 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 1 Processor -- start 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 1 -- start start 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 10 monclient: init 2024-09-15T01:51:38.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.275+0000 7f91ea509640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 0x7f91e409ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-c con 0x7f91e40980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 0x7f91e4130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-b con 0x7f91e40989b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 0x7f91e411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): picked mon.noname-a con 0x7f91e40a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e411e330 con 0x7f91e40a58b0 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e409eb00 con 0x7f91e40989b0 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e409b9a0 con 0x7f91e40980a0 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91ea509640 10 monclient(hunting): authenticate will time out at 758.272060s 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 0x7f91e409ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 0x7f91e411b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 0x7f91e409ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59966/0 (socket says 172.21.15.17:59966) 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 1 -- 172.21.15.17:0/2472830169 learned_addr learned my addr 172.21.15.17:0/2472830169 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): get_auth_request con 0x7f91e40980a0 auth_method 0 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): get_auth_request con 0x7f91e40a58b0 auth_method 0 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 0x7f91e4130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 10 monclient(hunting): get_auth_request con 0x7f91e40989b0 auth_method 0 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 -- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 msgr2=0x7f91e409ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 0x7f91e409ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 -- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 msgr2=0x7f91e4130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 0x7f91e4130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 monclient: found mon.noname-a 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e8aa6640 1 -- 172.21.15.17:0/2472830169 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009450 con 0x7f91e40a58b0 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.276+0000 7f91e37fe640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 0x7f91e4130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91e8aa6640 1 -- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 msgr2=0x7f91e411b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91e8aa6640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 0x7f91e411b7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f91e409f7a0 tx=0x7f91dc0025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00a850 con 0x7f91e40a58b0 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: dump: 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc00a9b0 con 0x7f91e40a58b0 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 5 monclient: authenticate success, global_id 4335 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 10 monclient: _renew_subs 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e411dc70 con 0x7f91e40a58b0 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00adb0 con 0x7f91e40a58b0 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91cbfff640 10 monclient: dump: 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.295 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91e27fc640 4 set_mon_vals no callback set 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 10 monclient: _renew_subs 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.278+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e4141730 con 0x7f91e40a58b0 2024-09-15T01:51:38.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.279+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f91dc012630 con 0x7f91e40a58b0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.279+0000 7f91ea509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.279+0000 7f91ea509640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.279+0000 7f91ea509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.279+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e40724a0 con 0x7f91e40a58b0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.280+0000 7f91cbfff640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 0x7f91d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.280+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f91dc04bb90 con 0x7f91e40a58b0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.280+0000 7f91e3fff640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 0x7f91d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.280+0000 7f91e3fff640 10 monclient: get_auth_request con 0x7f91d0038740 auth_method 0 2024-09-15T01:51:38.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.283+0000 7f91e3fff640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 0x7f91d003abf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009420 tx=0x7f91d4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.284+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f91dc022d70 con 0x7f91e40a58b0 2024-09-15T01:51:38.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.284+0000 7f91cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.284+0000 7f91cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:38.315 INFO:tasks.workunit:timeout=3h 2024-09-15T01:51:38.315 INFO:tasks.workunit:cleanup=True 2024-09-15T01:51:38.315 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:51:38.374 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T01:51:38.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.508+0000 7f91ea509640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:51:38.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.508+0000 7f91ea509640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:38.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.508+0000 7f91ea509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.508+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f91e409fc90 con 0x7f91e40a58b0 2024-09-15T01:51:38.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.509+0000 7f91cbfff640 1 -- 172.21.15.17:0/2472830169 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f91dc016030 con 0x7f91e40a58b0 2024-09-15T01:51:38.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.509+0000 7f91cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:38.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.509+0000 7f91cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:38.511 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 msgr2=0x7f91d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 0x7f91d003abf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009420 tx=0x7f91d4032000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 10 monclient: shutdown 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 msgr2=0x7f91e411b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.510+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 0x7f91e411b7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f91e409f7a0 tx=0x7f91dc0025b0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 shutdown_connections 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f91d0038740 0x7f91d003abf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91e40a58b0 0x7f91e411b7d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e40989b0 0x7f91e4130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 --2- 172.21.15.17:0/2472830169 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91e40980a0 0x7f91e409ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 >> 172.21.15.17:0/2472830169 conn(0x7f91e4070ed0 msgr2=0x7f91e40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 shutdown_connections 2024-09-15T01:51:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.511+0000 7f91ea509640 1 -- 172.21.15.17:0/2472830169 wait complete. 2024-09-15T01:51:38.523 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:51:38.523 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:51:38.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:51:38.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:51:38.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.618+0000 7f0e5d19d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:38.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.618+0000 7f0e5d19d640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.618+0000 7f0e5d19d640 10 monclient: monmap: 2024-09-15T01:51:38.623 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:38.619328+0000 2024-09-15T01:51:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:38.619328+0000 2024-09-15T01:51:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.624 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:38.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 Processor -- start 2024-09-15T01:51:38.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 -- start start 2024-09-15T01:51:38.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient: init 2024-09-15T01:51:38.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T01:51:38.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e580a5bd0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:38.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e58131f40 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:38.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0e58098720 0x7f0e5813a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e58098720 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:38.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580a14d0 con 0x7f0e580a5bd0 2024-09-15T01:51:38.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580a5fa0 con 0x7f0e58131f40 2024-09-15T01:51:38.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5809fef0 con 0x7f0e58098720 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5d19d640 10 monclient(hunting): authenticate will time out at 758.615658s 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e56d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f0e580a5bd0 0x7f0e58098010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33770/0 (socket says 172.21.15.17:33770) 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2663016933 learned_addr learned my addr 172.21.15.17:0/2663016933 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5759e640 1 --2- 172.21.15.17:0/2663016933 >> v2:172.21.15.17:3300/0 conn(0x7f0e58098720 0x7f0e5813a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5759e640 10 monclient(hunting): get_auth_request con 0x7f0e58098720 auth_method 0 2024-09-15T01:51:38.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.619+0000 7f0e5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2663016933 >> v1:172.21.15.17:6789/0 conn(0x7f0e580a5bd0 legacy=0x7f0e58098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5659c640 1 -- 172.21.15.17:0/2663016933 >> v1:172.21.15.67:6789/0 conn(0x7f0e58131f40 legacy=0x7f0e58136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 1 -- 172.21.15.17:0/2663016933 >> v1:172.21.15.67:6789/0 conn(0x7f0e58131f40 legacy=0x7f0e58136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 1 -- 172.21.15.17:0/2663016933 >> v1:172.21.15.17:6789/0 conn(0x7f0e580a5bd0 legacy=0x7f0e58098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 1 monclient: found mon.noname-a 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.622+0000 7f0e5759e640 1 -- 172.21.15.17:0/2663016933 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58098b40 con 0x7f0e58098720 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5759e640 1 -- 172.21.15.17:0/2663016933 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58098720 msgr2=0x7f0e5813a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5759e640 1 --2- 172.21.15.17:0/2663016933 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58098720 0x7f0e5813a400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009f60 tx=0x7f0e50009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185e1cbc33e91ea3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 1 -- 172.21.15.17:0/2663016933 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50007840 con 0x7f0e58098720 2024-09-15T01:51:38.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: dump: 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 1 -- 172.21.15.17:0/2663016933 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5002d050 con 0x7f0e58098720 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 5 monclient: authenticate success, global_id 4338 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 1 -- 172.21.15.17:0/2663016933 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50007bd0 con 0x7f0e58098720 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e55d9b640 10 monclient: dump: 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 4 set_mon_vals no callback set 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 10 monclient: shutdown 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2663016933 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58098720 msgr2=0x7f0e5813a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2663016933 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58098720 0x7f0e5813a400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009f60 tx=0x7f0e50009780 comp rx=0 tx=0).stop 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2663016933 shutdown_connections 2024-09-15T01:51:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2663016933 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58098720 0x7f0e5813a400 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.625+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2663016933 >> 172.21.15.17:0/2663016933 conn(0x7f0e580712c0 msgr2=0x7f0e580a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.626+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2663016933 shutdown_connections 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.626+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2663016933 wait complete. 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.626+0000 7f0e5d19d640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.626+0000 7f0e5d19d640 10 monclient: monmap: 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.626+0000 7f0e5d19d640 1 Processor -- start 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 -- start start 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient: init 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58098720 0x7f0e5809af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e58098720 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 0x7f0e58130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e580a5bd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 0x7f0e58140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e58131f40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5811c6c0 con 0x7f0e58131f40 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5809ba50 con 0x7f0e58098720 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5811dd50 con 0x7f0e580a5bd0 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5d19d640 10 monclient(hunting): authenticate will time out at 758.623531s 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 0x7f0e58140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 0x7f0e58130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 0x7f0e58140f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39236/0 (socket says 172.21.15.17:39236) 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): get_auth_request con 0x7f0e580a5bd0 auth_method 0 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 1 -- 172.21.15.17:0/2392814313 learned_addr learned my addr 172.21.15.17:0/2392814313 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58098720 0x7f0e5809af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): get_auth_request con 0x7f0e58131f40 auth_method 0 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): get_auth_request con 0x7f0e58098720 auth_method 0 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-15T01:51:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58098720 msgr2=0x7f0e5809af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58098720 0x7f0e5809af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 msgr2=0x7f0e58140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 0x7f0e58140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 monclient: found mon.noname-c 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.627+0000 7f0e5659c640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50009a20 con 0x7f0e580a5bd0 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.628+0000 7f0e5759e640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131f40 0x7f0e58140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5659c640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 msgr2=0x7f0e58130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5659c640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 0x7f0e58130c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c4a0 tx=0x7f0e4c007c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0074f0 con 0x7f0e580a5bd0 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: dump: 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c005780 con 0x7f0e580a5bd0 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 5 monclient: authenticate success, global_id 4352 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 10 monclient: _renew_subs 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5811e030 con 0x7f0e580a5bd0 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c005b80 con 0x7f0e580a5bd0 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e3effd640 10 monclient: dump: 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5559a640 4 set_mon_vals no callback set 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 10 monclient: _renew_subs 2024-09-15T01:51:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.630+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5811cdf0 con 0x7f0e580a5bd0 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0e4c01b3b0 con 0x7f0e580a5bd0 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5d19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 0x7f0e580298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58063a70 con 0x7f0e5811bd00 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 0x7f0e580298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e3effd640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e44038740 0x7f0e4403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient: get_auth_request con 0x7f0e5811bd00 auth_method 0 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f0e4c04ad00 con 0x7f0e580a5bd0 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5759e640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e44038740 0x7f0e4403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e5811e510 con 0x7f0e5811bd00 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.631+0000 7f0e5759e640 10 monclient: get_auth_request con 0x7f0e44038740 auth_method 0 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.633+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 msgr2=0x7f0e580298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.633+0000 7f0e56d9d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 0x7f0e580298c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800f780 tx=0x7f0e48009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48005a80 con 0x7f0e5811bd00 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 10 monclient: dump: 2024-09-15T01:51:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e5759e640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e44038740 0x7f0e4403abf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097c0 tx=0x7f0e50037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0e48005be0 con 0x7f0e5811bd00 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 msgr2=0x7f0e580298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 0x7f0e580298c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800f780 tx=0x7f0e48009040 comp rx=0 tx=0).stop 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 msgr2=0x7f0e580298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.634+0000 7f0e3effd640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e5811bd00 0x7f0e580298c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e5d19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 0x7f0e58029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e5d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580724a0 con 0x7f0e58131170 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 0x7f0e58029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient: get_auth_request con 0x7f0e58131170 auth_method 0 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-15T01:51:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.640+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2392814313 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e581340f0 con 0x7f0e58131170 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e56d9d640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 msgr2=0x7f0e58029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e56d9d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 0x7f0e58029e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800f780 tx=0x7f0e48018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48021830 con 0x7f0e58131170 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.642+0000 7f0e3effd640 10 monclient: dump: 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f0e48005870 con 0x7f0e58131170 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 msgr2=0x7f0e58029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 0x7f0e58029e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800f780 tx=0x7f0e48018000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 msgr2=0x7f0e58029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e3effd640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e58131170 0x7f0e58029e00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 4, 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:51:38.664 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 21, 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.666 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:51:38.667 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.668 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.669 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.674 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.675 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.677 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:51:38.677 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:51:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e44038740 msgr2=0x7f0e4403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e44038740 0x7f0e4403abf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097c0 tx=0x7f0e50037000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 10 monclient: shutdown 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 msgr2=0x7f0e58130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 --2- 172.21.15.17:0/2392814313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e580a5bd0 0x7f0e58130c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c4a0 tx=0x7f0e4c007c30 comp rx=0 tx=0).stop 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5759e640 1 -- 172.21.15.17:0/2392814313 reap_dead start 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5759e640 1 -- 172.21.15.17:0/2392814313 reap_dead start 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 shutdown_connections 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 >> 172.21.15.17:0/2392814313 conn(0x7f0e580712c0 msgr2=0x7f0e5809eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 shutdown_connections 2024-09-15T01:51:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.643+0000 7f0e5d19d640 1 -- 172.21.15.17:0/2392814313 wait complete. 2024-09-15T01:51:38.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:51:38.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:51:38.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:51:38.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.775+0000 7f5747ca8640 10 monclient: get_monmap_and_config 2024-09-15T01:51:38.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.775+0000 7f5747ca8640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.775+0000 7f5747ca8640 10 monclient: monmap: 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:38.776092+0000 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:38.776092+0000 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:38.781 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.775+0000 7f5747ca8640 1 Processor -- start 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 -- start start 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient: init 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f574009c1b0 0x7f574009a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f574009c1b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f574009c580 0x7f574006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f574009c580 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f574006f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f574009c9c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5740069d40 con 0x7f574009c580 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574009f790 con 0x7f574009c1b0 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5740068b60 con 0x7f574009c9c0 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5747ca8640 10 monclient(hunting): authenticate will time out at 758.772539s 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f574009c580 0x7f574006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f574009c580 0x7f574006eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39266/0 (socket says 172.21.15.17:39266) 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 -- 172.21.15.17:0/1894635773 learned_addr learned my addr 172.21.15.17:0/1894635773 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): get_auth_request con 0x7f574009c580 auth_method 0 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f574006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): get_auth_request con 0x7f574009c9c0 auth_method 0 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 1 --2- 172.21.15.17:0/1894635773 >> v2:172.21.15.67:3300/0 conn(0x7f574009c1b0 0x7f574009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): get_auth_request con 0x7f574009c1b0 auth_method 0 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 -- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 msgr2=0x7f574006f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f574006f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.776+0000 7f5745243640 1 -- 172.21.15.17:0/1894635773 >> v2:172.21.15.67:3300/0 conn(0x7f574009c1b0 msgr2=0x7f574009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 1 --2- 172.21.15.17:0/1894635773 >> v2:172.21.15.67:3300/0 conn(0x7f574009c1b0 0x7f574009a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 1 monclient: found mon.noname-a 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5745243640 1 -- 172.21.15.17:0/1894635773 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740069aa0 con 0x7f574009c580 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.777+0000 7f5746245640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f574006f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5745243640 1 -- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c580 msgr2=0x7f574006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5745243640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c580 0x7f574006eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5740068b30 tx=0x7f573c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42e78c35d3ebf9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 1 -- 172.21.15.17:0/1894635773 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c032440 con 0x7f574009c580 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 10 monclient: dump: 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.779+0000 7f5744a42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 5 monclient: authenticate success, global_id 4353 2024-09-15T01:51:38.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 1 -- 172.21.15.17:0/1894635773 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c0325a0 con 0x7f574009c580 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 1 -- 172.21.15.17:0/1894635773 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c032940 con 0x7f574009c580 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 4 set_mon_vals no callback set 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5744a42640 10 monclient: dump: 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 10 monclient: shutdown 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 -- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c580 msgr2=0x7f574006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c580 0x7f574006eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5740068b30 tx=0x7f573c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 -- 172.21.15.17:0/1894635773 shutdown_connections 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f574006f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 --2- 172.21.15.17:0/1894635773 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c580 0x7f574006eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 --2- 172.21.15.17:0/1894635773 >> v2:172.21.15.67:3300/0 conn(0x7f574009c1b0 0x7f574009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 -- 172.21.15.17:0/1894635773 >> 172.21.15.17:0/1894635773 conn(0x7f5740071300 msgr2=0x7f57400a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 -- 172.21.15.17:0/1894635773 shutdown_connections 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.780+0000 7f5747ca8640 1 -- 172.21.15.17:0/1894635773 wait complete. 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient: monmap: 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 Processor -- start 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 -- start start 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient: init 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 0x7f5740141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f574009c1b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574009c580 0x7f57401350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f574009c580 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f5740135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f574009c9c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574009cdb0 con 0x7f574009c1b0 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574006f070 con 0x7f574009c580 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5740120750 con 0x7f574009c9c0 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.781+0000 7f5747ca8640 10 monclient(hunting): authenticate will time out at 758.777947s 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 0x7f5740141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f5740135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 0x7f5740141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39274/0 (socket says 172.21.15.17:39274) 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 1 -- 172.21.15.17:0/881893328 learned_addr learned my addr 172.21.15.17:0/881893328 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): get_auth_request con 0x7f574009c9c0 auth_method 0 2024-09-15T01:51:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): get_auth_request con 0x7f574009c1b0 auth_method 0 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574009c580 0x7f57401350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): get_auth_request con 0x7f574009c580 auth_method 0 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745243640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574009c580 msgr2=0x7f57401350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574009c580 0x7f57401350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 msgr2=0x7f5740141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 0x7f5740141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 monclient: found mon.noname-c 2024-09-15T01:51:38.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738009a20 con 0x7f574009c9c0 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.782+0000 7f5745a44640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f574009c1b0 0x7f5740141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 msgr2=0x7f5740135630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5746245640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f5740135630 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5738009ed0 tx=0x7f573800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800ce10 con 0x7f574009c9c0 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: dump: 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.803 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5747ca8640 5 monclient: authenticate success, global_id 4358 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800f040 con 0x7f574009c9c0 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5747ca8640 10 monclient: _renew_subs 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5747ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c009450 con 0x7f574009c9c0 2024-09-15T01:51:38.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380148e0 con 0x7f574009c9c0 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5731ffb640 10 monclient: dump: 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.805 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 10 monclient: _renew_subs 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.784+0000 7f5733fff640 4 set_mon_vals no callback set 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57401222c0 con 0x7f574009c9c0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5738014c50 con 0x7f574009c9c0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 0x7f574011eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5740063bb0 con 0x7f57401417c0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5745a44640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 0x7f574011eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5745a44640 10 monclient: get_auth_request con 0x7f57401417c0 auth_method 0 2024-09-15T01:51:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5745a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5745a44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.785+0000 7f5745a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5731ffb640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5724038740 0x7f572403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f573804c8e0 con 0x7f574009c9c0 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745a44640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745a44640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5740121380 con 0x7f57401417c0 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745243640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5724038740 0x7f572403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.786+0000 7f5745243640 10 monclient: get_auth_request con 0x7f5724038740 auth_method 0 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5745a44640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 msgr2=0x7f574011eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5745a44640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 0x7f574011eb70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5734009f60 tx=0x7f57340090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734014860 con 0x7f57401417c0 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 10 monclient: dump: 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5745243640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5724038740 0x7f572403abf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5740060cf0 tx=0x7f573c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f57340149c0 con 0x7f57401417c0 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 msgr2=0x7f574011eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 0x7f574011eb70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5734009f60 tx=0x7f57340090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 msgr2=0x7f574011eb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.788+0000 7f5731ffb640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57401417c0 0x7f574011eb70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5747ca8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5747ca8640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 0x7f574011f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5747ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574011f9f0 con 0x7f574011f0b0 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 0x7f574011f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient: get_auth_request con 0x7f574011f0b0 auth_method 0 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.794+0000 7f5745a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.795+0000 7f5745a44640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.795+0000 7f5745a44640 1 -- 172.21.15.17:0/881893328 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57400724a0 con 0x7f574011f0b0 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5745a44640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 msgr2=0x7f574011f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5745a44640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 0x7f574011f4b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f573401b000 tx=0x7f573401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734024040 con 0x7f574011f0b0 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 10 monclient: dump: 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f5734014650 con 0x7f574011f0b0 2024-09-15T01:51:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 msgr2=0x7f574011f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 0x7f574011f4b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f573401b000 tx=0x7f573401a000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 msgr2=0x7f574011f4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.797+0000 7f5731ffb640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574011f0b0 0x7f574011f4b0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f573401b000 tx=0x7f573401a000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:51:38.815 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 4, 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 21, 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:51:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:51:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:51:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:51:38.828 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:51:38.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5724038740 msgr2=0x7f572403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5724038740 0x7f572403abf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5740060cf0 tx=0x7f573c032000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 10 monclient: shutdown 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 msgr2=0x7f5740135630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 --2- 172.21.15.17:0/881893328 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f574009c9c0 0x7f5740135630 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5738009ed0 tx=0x7f573800b040 comp rx=0 tx=0).stop 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 reap_dead start 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5746245640 1 -- 172.21.15.17:0/881893328 reap_dead start 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 shutdown_connections 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 >> 172.21.15.17:0/881893328 conn(0x7f5740071300 msgr2=0x7f57400a1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 shutdown_connections 2024-09-15T01:51:38.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.798+0000 7f5747ca8640 1 -- 172.21.15.17:0/881893328 wait complete. 2024-09-15T01:51:38.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:51:38.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:51:38.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:51:38.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.925+0000 7fc5cd15d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:38.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.925+0000 7fc5cd15d640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.925+0000 7fc5cd15d640 10 monclient: monmap: 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:38.926441+0000 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:38.926441+0000 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:38.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 Processor -- start 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 -- start start 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient: init 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5c806eb30 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5c809e970 0x7fc5c809a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c809e970 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc5c809bfb0 0x7fc5c809c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c809bfb0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c809ed70 con 0x7fc5c806eb30 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c8099f40 con 0x7fc5c809bfb0 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c8069f20 con 0x7fc5c809e970 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.926+0000 7fc5cd15d640 10 monclient(hunting): authenticate will time out at 758.922884s 2024-09-15T01:51:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc5c809bfb0 0x7fc5c809c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc5c809bfb0 0x7fc5c809c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39278/0 (socket says 172.21.15.17:39278) 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 -- 172.21.15.17:0/2790926157 learned_addr learned my addr 172.21.15.17:0/2790926157 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): get_auth_request con 0x7fc5c809bfb0 auth_method 0 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5bffff640 1 --2- 172.21.15.17:0/2790926157 >> v2:172.21.15.67:3300/0 conn(0x7fc5c809e970 0x7fc5c809a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 -- 172.21.15.17:0/2790926157 >> v2:172.21.15.67:3300/0 conn(0x7fc5c809e970 msgr2=0x7fc5c809a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 --2- 172.21.15.17:0/2790926157 >> v2:172.21.15.67:3300/0 conn(0x7fc5c809e970 0x7fc5c809a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 -- 172.21.15.17:0/2790926157 >> v1:172.21.15.67:6789/0 conn(0x7fc5c806eb30 legacy=0x7fc5c806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 monclient: found mon.noname-a 2024-09-15T01:51:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.927+0000 7fc5c759e640 1 -- 172.21.15.17:0/2790926157 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c806a550 con 0x7fc5c809bfb0 2024-09-15T01:51:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c759e640 1 -- 172.21.15.17:0/2790926157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 msgr2=0x7fc5c809c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c759e640 1 --2- 172.21.15.17:0/2790926157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c809c3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009f20 tx=0x7fc5c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678ac46e70a305e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 1 -- 172.21.15.17:0/2790926157 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c002aea0 con 0x7fc5c809bfb0 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: dump: 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 1 -- 172.21.15.17:0/2790926157 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c002d050 con 0x7fc5c809bfb0 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 5 monclient: authenticate success, global_id 4359 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 1 -- 172.21.15.17:0/2790926157 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0007840 con 0x7fc5c809bfb0 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 4 set_mon_vals no callback set 2024-09-15T01:51:38.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5c659c640 10 monclient: dump: 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 10 monclient: shutdown 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 -- 172.21.15.17:0/2790926157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 msgr2=0x7fc5c809c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/2790926157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c809c3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009f20 tx=0x7fc5c0009840 comp rx=0 tx=0).stop 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 -- 172.21.15.17:0/2790926157 shutdown_connections 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/2790926157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c809c3f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/2790926157 >> v2:172.21.15.67:3300/0 conn(0x7fc5c809e970 0x7fc5c809a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 -- 172.21.15.17:0/2790926157 >> 172.21.15.17:0/2790926157 conn(0x7fc5c80712f0 msgr2=0x7fc5c80a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 -- 172.21.15.17:0/2790926157 shutdown_connections 2024-09-15T01:51:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.930+0000 7fc5cd15d640 1 -- 172.21.15.17:0/2790926157 wait complete. 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 10 monclient: build_initial_monmap 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 10 monclient: monmap: 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 1 Processor -- start 2024-09-15T01:51:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 1 -- start start 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 10 monclient: init 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.931+0000 7fc5cd15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c806eb30 0x7fc5c8140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c806eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c809bfb0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 0x7fc5c811e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8120470 con 0x7fc5c809bfb0 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c809ccd0 con 0x7fc5c806eb30 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80a0480 con 0x7fc5c809e970 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5cd15d640 10 monclient(hunting): authenticate will time out at 758.928065s 2024-09-15T01:51:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 0x7fc5c811e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39284/0 (socket says 172.21.15.17:39284) 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 0x7fc5c811e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60020/0 (socket says 172.21.15.17:60020) 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 -- 172.21.15.17:0/3326235195 learned_addr learned my addr 172.21.15.17:0/3326235195 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): get_auth_request con 0x7fc5c809bfb0 auth_method 0 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): get_auth_request con 0x7fc5c809e970 auth_method 0 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c806eb30 0x7fc5c8140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 10 monclient(hunting): get_auth_request con 0x7fc5c806eb30 auth_method 0 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 msgr2=0x7fc5c811e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 0x7fc5c811e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c806eb30 msgr2=0x7fc5c8140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c806eb30 0x7fc5c8140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient: _finish_hunting 0 2024-09-15T01:51:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 monclient: found mon.noname-a 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5bffff640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0009ae0 con 0x7fc5c809bfb0 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c759e640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5c809e970 0x7fc5c811e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.932+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c806eb30 0x7fc5c8140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5bffff640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 msgr2=0x7fc5c8134d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5bffff640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c8e0 tx=0x7fc5b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000ce70 con 0x7fc5c809bfb0 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 10 monclient: dump: 2024-09-15T01:51:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0007a40 con 0x7fc5c809bfb0 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5cd15d640 5 monclient: authenticate success, global_id 4362 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5cd15d640 10 monclient: _renew_subs 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5cd15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.934+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c8121ea0 con 0x7fc5c809bfb0 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007de0 con 0x7fc5c809bfb0 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5beffd640 10 monclient: dump: 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5c5d9b640 4 set_mon_vals no callback set 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 10 monclient: _renew_subs 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c8141d00 con 0x7fc5c809bfb0 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 0x7fc5c811ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.935+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8063b90 con 0x7fc5c8120fb0 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc5b0011600 con 0x7fc5c809bfb0 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 0x7fc5c811ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient: get_auth_request con 0x7fc5c8120fb0 auth_method 0 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5beffd640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5a8038770 0x7fc5a803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fc5b004b800 con 0x7fc5c809bfb0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c6d9d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5c8141670 con 0x7fc5c8120fb0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c759e640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5a8038770 0x7fc5a803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.936+0000 7fc5c759e640 10 monclient: get_auth_request con 0x7fc5a8038770 auth_method 0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5c6d9d640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 msgr2=0x7fc5c811ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 0x7fc5c811ee20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800d740 tx=0x7fc5b8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5c759e640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5a8038770 0x7fc5a803ac20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0033000 tx=0x7fc5c0036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8009950 con 0x7fc5c8120fb0 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.938+0000 7fc5beffd640 10 monclient: dump: 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc5b8009ab0 con 0x7fc5c8120fb0 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 msgr2=0x7fc5c811ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 0x7fc5c811ee20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800d740 tx=0x7fc5b8008040 comp rx=0 tx=0).stop 2024-09-15T01:51:38.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 msgr2=0x7fc5c811ee20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.939+0000 7fc5beffd640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c8120fb0 0x7fc5c811ee20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5cd15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 0x7fc5c811f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5cd15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80724a0 con 0x7fc5c81352c0 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 0x7fc5c811f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient: get_auth_request con 0x7fc5c81352c0 auth_method 0 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:38.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.945+0000 7fc5c6d9d640 1 -- 172.21.15.17:0/3326235195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc5c8067220 con 0x7fc5c81352c0 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.947+0000 7fc5c6d9d640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 msgr2=0x7fc5c811f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.947+0000 7fc5c6d9d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 0x7fc5c811f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8017000 tx=0x7fc5b801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8012c20 con 0x7fc5c81352c0 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 10 monclient: dump: 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:38.967 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7fc5b8009740 con 0x7fc5c81352c0 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:51:38.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 msgr2=0x7fc5c811f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 0x7fc5c811f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8017000 tx=0x7fc5b801a000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 msgr2=0x7fc5c811f360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5beffd640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c81352c0 0x7fc5c811f360 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8017000 tx=0x7fc5b801a000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 4, 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 21, 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:51:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:51:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:51:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:51:38.986 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5a8038770 msgr2=0x7fc5a803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5a8038770 0x7fc5a803ac20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0033000 tx=0x7fc5c0036000 comp rx=0 tx=0).stop 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 10 monclient: shutdown 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 msgr2=0x7fc5c8134d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c8e0 tx=0x7fc5b0008040 comp rx=0 tx=0).stop 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5c759e640 1 -- 172.21.15.17:0/3326235195 reap_dead start 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 shutdown_connections 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 --2- 172.21.15.17:0/3326235195 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5c809bfb0 0x7fc5c8134d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:38.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 >> 172.21.15.17:0/3326235195 conn(0x7fc5c80712f0 msgr2=0x7fc5c806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:38.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 shutdown_connections 2024-09-15T01:51:38.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:38.948+0000 7fc5cd15d640 1 -- 172.21.15.17:0/3326235195 wait complete. 2024-09-15T01:51:38.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:51:38.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T01:51:38.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:51:38.989 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T01:51:38.989 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:51:38.990 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T01:51:38.990 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:51:39.000 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T01:51:39.000 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:51:39.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 10 monclient: get_monmap_and_config 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 10 monclient: build_initial_monmap 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 10 monclient: monmap: 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:39.136 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:39.135112+0000 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:39.135112+0000 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:39.137 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 1 Processor -- start 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 1 -- start start 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.134+0000 7f763d680640 10 monclient: init 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76380a5780 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:39.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f763809ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f763809bc90 0x7f763813a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): picked mon.noname-a con 0x7f763809bc90 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76380a5b50 con 0x7f763809ccf0 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763809c5f0 con 0x7f76380a5780 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76380a18b0 con 0x7f763809bc90 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f763d680640 10 monclient(hunting): authenticate will time out at 759.131561s 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f763809bc90 0x7f763813a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f763809bc90 0x7f763813a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39298/0 (socket says 172.21.15.17:39298) 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76367fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f763809ccf0 0x7f7638069c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33774/0 (socket says 172.21.15.17:33774) 2024-09-15T01:51:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 -- 172.21.15.17:0/3213180331 learned_addr learned my addr 172.21.15.17:0/3213180331 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): get_auth_request con 0x7f763809bc90 auth_method 0 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f7636ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76380a5780 0x7f7638069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52558/0 (socket says 172.21.15.17:52558) 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 -- 172.21.15.17:0/3213180331 >> v1:172.21.15.67:6789/0 conn(0x7f76380a5780 legacy=0x7f7638069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 -- 172.21.15.17:0/3213180331 >> v1:172.21.15.17:6789/0 conn(0x7f763809ccf0 legacy=0x7f7638069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 monclient: found mon.noname-a 2024-09-15T01:51:39.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:39.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.135+0000 7f76377fe640 1 -- 172.21.15.17:0/3213180331 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763806a3a0 con 0x7f763809bc90 2024-09-15T01:51:39.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.138+0000 7f76377fe640 1 -- 172.21.15.17:0/3213180331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809bc90 msgr2=0x7f763813a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:39.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.138+0000 7f76377fe640 1 --2- 172.21.15.17:0/3213180331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809bc90 0x7f763813a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7630009f20 tx=0x7f7630009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b290b44daa2f50d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 1 -- 172.21.15.17:0/3213180331 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763002aea0 con 0x7f763809bc90 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: dump: 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:39.147 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 1 -- 172.21.15.17:0/3213180331 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763002d050 con 0x7f763809bc90 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 5 monclient: authenticate success, global_id 4365 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 4 set_mon_vals no callback set 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 1 -- 172.21.15.17:0/3213180331 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630007840 con 0x7f763809bc90 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:39.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f7635ffb640 10 monclient: dump: 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 10 monclient: shutdown 2024-09-15T01:51:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 -- 172.21.15.17:0/3213180331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809bc90 msgr2=0x7f763813a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 --2- 172.21.15.17:0/3213180331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809bc90 0x7f763813a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7630009f20 tx=0x7f7630009840 comp rx=0 tx=0).stop 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 -- 172.21.15.17:0/3213180331 shutdown_connections 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 --2- 172.21.15.17:0/3213180331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809bc90 0x7f763813a890 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 -- 172.21.15.17:0/3213180331 >> 172.21.15.17:0/3213180331 conn(0x7f7638070e10 msgr2=0x7f76380a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 -- 172.21.15.17:0/3213180331 shutdown_connections 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.139+0000 7f763d680640 1 -- 172.21.15.17:0/3213180331 wait complete. 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.140+0000 7f763d680640 10 monclient: build_initial_monmap 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.140+0000 7f763d680640 10 monclient: monmap: 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.140+0000 7f763d680640 1 Processor -- start 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.140+0000 7f763d680640 1 -- start start 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.140+0000 7f763d680640 10 monclient: init 2024-09-15T01:51:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f763809bc90 0x7f763813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): picked mon.noname-b con 0x7f763809bc90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): picked mon.noname-a con 0x7f763809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 0x7f7638141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): picked mon.noname-c con 0x7f76380a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638120a30 con 0x7f763809ccf0 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76380a0920 con 0x7f763809bc90 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763809b120 con 0x7f76380a5780 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f763d680640 10 monclient(hunting): authenticate will time out at 759.137316s 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76377fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 msgr2=0x7f7638141290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76377fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 0x7f7638141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39314/0 (socket says 172.21.15.17:39314) 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 -- 172.21.15.17:0/2167203019 learned_addr learned my addr 172.21.15.17:0/2167203019 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): get_auth_request con 0x7f763809ccf0 auth_method 0 2024-09-15T01:51:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f763809bc90 0x7f763813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): get_auth_request con 0x7f763809bc90 auth_method 0 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f7636ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 -- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 msgr2=0x7f7638141290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 0x7f7638141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 -- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f763809bc90 msgr2=0x7f763813dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f763809bc90 0x7f763813dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient: _finish_hunting 0 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 monclient: found mon.noname-a 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.141+0000 7f76367fc640 1 -- 172.21.15.17:0/2167203019 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630009ae0 con 0x7f763809ccf0 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f76367fc640 1 -- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 msgr2=0x7f763813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f76367fc640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c5e0 tx=0x7f762c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c007510 con 0x7f763809ccf0 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: dump: 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:39.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: _finish_auth 0 2024-09-15T01:51:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c0057d0 con 0x7f763809ccf0 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c007690 con 0x7f763809ccf0 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 5 monclient: authenticate success, global_id 4368 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 10 monclient: _renew_subs 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7638122490 con 0x7f763809ccf0 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f76357fa640 4 set_mon_vals no callback set 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f761effd640 10 monclient: dump: 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 10 monclient: _renew_subs 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.144+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76381216f0 con 0x7f763809ccf0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f763d680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f763d680640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f763d680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76380724a0 con 0x7f763809ccf0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f762c01b400 con 0x7f763809ccf0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f761effd640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 0x7f762403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.145+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f762c04b3b0 con 0x7f763809ccf0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.146+0000 7f7636ffd640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 0x7f762403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:39.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.146+0000 7f7636ffd640 10 monclient: get_auth_request con 0x7f7624038740 auth_method 0 2024-09-15T01:51:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.148+0000 7f7636ffd640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 0x7f762403abf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7628007310 tx=0x7f7628007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.149+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f762c01ab40 con 0x7f763809ccf0 2024-09-15T01:51:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.149+0000 7f761effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.149+0000 7f761effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f763d680640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:51:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f763d680640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f763d680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f763806f510 con 0x7f763809ccf0 2024-09-15T01:51:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f761effd640 1 -- 172.21.15.17:0/2167203019 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f762c01a360 con 0x7f763809ccf0 2024-09-15T01:51:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f761effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.379+0000 7f761effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:39.382 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 msgr2=0x7f762403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 0x7f762403abf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7628007310 tx=0x7f7628007e40 comp rx=0 tx=0).stop 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 10 monclient: shutdown 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 msgr2=0x7f763813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c5e0 tx=0x7f762c007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 shutdown_connections 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7624038740 0x7f762403abf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f763809ccf0 0x7f763813d3f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76380a5780 0x7f7638141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 --2- 172.21.15.17:0/2167203019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f763809bc90 0x7f763813dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:39.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 >> 172.21.15.17:0/2167203019 conn(0x7f7638070e10 msgr2=0x7f763809a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:39.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 shutdown_connections 2024-09-15T01:51:39.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:39.381+0000 7f763d680640 1 -- 172.21.15.17:0/2167203019 wait complete. 2024-09-15T01:51:42.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.501+0000 7f4f0b59d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.501+0000 7f4f0b59d640 10 monclient: build_initial_monmap 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.501+0000 7f4f0b59d640 10 monclient: monmap: 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:42.502773+0000 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:42.502773+0000 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:42.504 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:42.505 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:42.505 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:42.505 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:42.505 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.502+0000 7f4f0b59d640 1 Processor -- start 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.502+0000 7f4f0b59d640 1 -- start start 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.502+0000 7f4f0b59d640 10 monclient: init 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:51:42.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f040a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f04132040 0x7f4f040987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f04132040 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 0x7f4f041391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f04136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040a01e0 con 0x7f4f040a58b0 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f040a1580 con 0x7f4f04132040 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040a5c80 con 0x7f4f04136dd0 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:42.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f0b59d640 10 monclient(hunting): authenticate will time out at 762.499381s 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f09b3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 msgr2=0x7f4f041391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 0x7f4f041391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f04132040 0x7f4f040987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f04132040 0x7f4f040987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59260/0 (socket says 172.21.15.17:59260) 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 1 -- 172.21.15.17:0/1795202077 learned_addr learned my addr 172.21.15.17:0/1795202077 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): get_auth_request con 0x7f4f04132040 auth_method 0 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f09339640 1 --1- 172.21.15.17:0/1795202077 >> v1:172.21.15.67:6789/0 conn(0x7f4f040a58b0 0x7f4f040980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52572/0 (socket says 172.21.15.17:52572) 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:42.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.503+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 1 -- 172.21.15.17:0/1795202077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 msgr2=0x7f4f041391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 1 --2- 172.21.15.17:0/1795202077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 0x7f4f041391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 1 -- 172.21.15.17:0/1795202077 >> v1:172.21.15.67:6789/0 conn(0x7f4f040a58b0 legacy=0x7f4f040980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 10 monclient: _finish_hunting 0 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 1 monclient: found mon.noname-c 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.504+0000 7f4f08b38640 1 -- 172.21.15.17:0/1795202077 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04132440 con 0x7f4f04132040 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f08b38640 1 -- 172.21.15.17:0/1795202077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f04132040 msgr2=0x7f4f040987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f08b38640 1 --2- 172.21.15.17:0/1795202077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f04132040 0x7f4f040987b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009910 tx=0x7f4ef40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39624943e1e7b7d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 1 -- 172.21.15.17:0/1795202077 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4032440 con 0x7f4f04132040 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: dump: 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:42.516 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: _finish_auth 0 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 1 -- 172.21.15.17:0/1795202077 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef40325a0 con 0x7f4f04132040 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:42.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 1 -- 172.21.15.17:0/1795202077 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4032940 con 0x7f4f04132040 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 5 monclient: authenticate success, global_id 4378 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 4 set_mon_vals no callback set 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4efbfff640 10 monclient: dump: 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 10 monclient: shutdown 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 -- 172.21.15.17:0/1795202077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f04132040 msgr2=0x7f4f040987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/1795202077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f04132040 0x7f4f040987b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009910 tx=0x7f4ef40091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 -- 172.21.15.17:0/1795202077 shutdown_connections 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/1795202077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04136dd0 0x7f4f041391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/1795202077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f04132040 0x7f4f040987b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.507+0000 7f4f0b59d640 1 -- 172.21.15.17:0/1795202077 >> 172.21.15.17:0/1795202077 conn(0x7f4f04070ed0 msgr2=0x7f4f040a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 1 -- 172.21.15.17:0/1795202077 shutdown_connections 2024-09-15T01:51:42.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 1 -- 172.21.15.17:0/1795202077 wait complete. 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 10 monclient: build_initial_monmap 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 10 monclient: monmap: 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 1 Processor -- start 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 1 -- start start 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.508+0000 7f4f0b59d640 10 monclient: init 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f040a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 0x7f4f04133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f04132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f08b38640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 msgr2=0x7f4f04133290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f08b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 0x7f4f04133290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 0x7f4f041337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f04136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:42.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04098dc0 con 0x7f4f04136dd0 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040a3410 con 0x7f4f040a58b0 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04135e90 con 0x7f4f04132040 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f0b59d640 10 monclient(hunting): authenticate will time out at 762.505711s 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 0x7f4f041337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59272/0 (socket says 172.21.15.17:59272) 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 1 -- 172.21.15.17:0/516952205 learned_addr learned my addr 172.21.15.17:0/516952205 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): get_auth_request con 0x7f4f040a58b0 auth_method 0 2024-09-15T01:51:42.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4f04136dd0 auth_method 0 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:42.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.509+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 -- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 msgr2=0x7f4f04133290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 0x7f4f04133290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 -- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 msgr2=0x7f4f041337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 0x7f4f041337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 10 monclient: _finish_hunting 0 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 monclient: found mon.noname-b 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:42.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09339640 1 -- 172.21.15.17:0/516952205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef4009450 con 0x7f4f040a58b0 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.510+0000 7f4f09b3a640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 0x7f4f041337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f09339640 1 -- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 msgr2=0x7f4f04132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f09339640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000b540 tx=0x7f4f000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00011840 con 0x7f4f040a58b0 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 10 monclient: dump: 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.525 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 10 monclient: _finish_auth 0 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f000119a0 con 0x7f4f040a58b0 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:42.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00011d60 con 0x7f4f040a58b0 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f0b59d640 5 monclient: authenticate success, global_id 4381 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f0b59d640 10 monclient: _renew_subs 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f041363c0 con 0x7f4f040a58b0 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4ef97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4ef97fa640 10 monclient: dump: 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.512+0000 7f4efaffd640 4 set_mon_vals no callback set 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 10 monclient: _renew_subs 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f04133d10 con 0x7f4f040a58b0 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.513+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f04133f30 con 0x7f4f040a58b0 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.514+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4f00010590 con 0x7f4f040a58b0 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.514+0000 7f4ef97fa640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 0x7f4ee803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.514+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4f0004b650 con 0x7f4f040a58b0 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.514+0000 7f4f08b38640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 0x7f4ee803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.514+0000 7f4f08b38640 10 monclient: get_auth_request con 0x7f4ee8038740 auth_method 0 2024-09-15T01:51:42.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.517+0000 7f4f08b38640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 0x7f4ee803abf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f040a60e0 tx=0x7f4ef4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:42.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.519+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4f00020880 con 0x7f4f040a58b0 2024-09-15T01:51:42.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.519+0000 7f4ef97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:42.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.519+0000 7f4ef97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:42.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4f0b59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:51:42.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4f0b59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:42.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:51:42.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f04134850 con 0x7f4f040a58b0 2024-09-15T01:51:42.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4ef97fa640 1 -- 172.21.15.17:0/516952205 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f4f00015030 con 0x7f4f040a58b0 2024-09-15T01:51:42.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4ef97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:42.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.744+0000 7f4ef97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:42.747 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 msgr2=0x7f4ee803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 0x7f4ee803abf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f040a60e0 tx=0x7f4ef4032000 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 10 monclient: shutdown 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 msgr2=0x7f4f04132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000b540 tx=0x7f4f000090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 shutdown_connections 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ee8038740 0x7f4ee803abf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f04136dd0 0x7f4f041337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f04132040 0x7f4f04133290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 --2- 172.21.15.17:0/516952205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f040a58b0 0x7f4f04132d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:42.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 >> 172.21.15.17:0/516952205 conn(0x7f4f04070ed0 msgr2=0x7f4f04096e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:42.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 shutdown_connections 2024-09-15T01:51:42.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:42.746+0000 7f4f0b59d640 1 -- 172.21.15.17:0/516952205 wait complete. 2024-09-15T01:51:45.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.866+0000 7fe6ee79d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.866+0000 7fe6ee79d640 10 monclient: build_initial_monmap 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.866+0000 7fe6ee79d640 10 monclient: monmap: 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:45.867033+0000 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:45.867033+0000 2024-09-15T01:51:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:45.869 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.866+0000 7fe6ee79d640 1 Processor -- start 2024-09-15T01:51:45.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 -- start start 2024-09-15T01:51:45.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient: init 2024-09-15T01:51:45.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:45.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:51:45.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6e8068f20 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:45.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe6e80692f0 0x7fe6e80696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e80692f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e81393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e8136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80a1950 con 0x7fe6e8068f20 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80a1670 con 0x7fe6e80692f0 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81328d0 con 0x7fe6e8136fe0 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ee79d640 10 monclient(hunting): authenticate will time out at 765.863623s 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ecd3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 msgr2=0x7fe6e81393c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6ecd3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e81393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:45.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe6e80692f0 0x7fe6e80696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe6e8068f20 0x7fe6e81321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:58690/0 (socket says 172.21.15.17:58690) 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe6e80692f0 0x7fe6e80696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39128/0 (socket says 172.21.15.17:39128) 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e7fff640 1 -- 172.21.15.17:0/1537673120 learned_addr learned my addr 172.21.15.17:0/1537673120 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request con 0x7fe6e80692f0 auth_method 0 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:45.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.867+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 1 -- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 msgr2=0x7fe6e81393c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 1 --2- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e81393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 1 -- 172.21.15.17:0/1537673120 >> v1:172.21.15.67:6789/0 conn(0x7fe6e8068f20 legacy=0x7fe6e81321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 1 monclient: found mon.noname-a 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:45.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.868+0000 7fe6e77fe640 1 -- 172.21.15.17:0/1537673120 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80a0020 con 0x7fe6e80692f0 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e77fe640 1 -- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 msgr2=0x7fe6e80696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e77fe640 1 --2- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e80696f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009880 tx=0x7fe6dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b99fae445e5d184 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 1 -- 172.21.15.17:0/1537673120 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc032440 con 0x7fe6e80692f0 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: dump: 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.883 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 1 -- 172.21.15.17:0/1537673120 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc0325a0 con 0x7fe6e80692f0 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6ee79d640 5 monclient: authenticate success, global_id 4374 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6ee79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6ee79d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 1 -- 172.21.15.17:0/1537673120 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc032940 con 0x7fe6e80692f0 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6ee79d640 4 set_mon_vals no callback set 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.871+0000 7fe6e6ffd640 10 monclient: dump: 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 10 monclient: shutdown 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 -- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 msgr2=0x7fe6e80696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e80696f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009880 tx=0x7fe6dc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 -- 172.21.15.17:0/1537673120 shutdown_connections 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e81393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/1537673120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e80696f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 -- 172.21.15.17:0/1537673120 >> 172.21.15.17:0/1537673120 conn(0x7fe6e8070fb0 msgr2=0x7fe6e80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 -- 172.21.15.17:0/1537673120 shutdown_connections 2024-09-15T01:51:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 -- 172.21.15.17:0/1537673120 wait complete. 2024-09-15T01:51:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 10 monclient: build_initial_monmap 2024-09-15T01:51:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 10 monclient: monmap: 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.872+0000 7fe6ee79d640 1 Processor -- start 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 -- start start 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient: init 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:51:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e8068f20 0x7fe6e8130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e8068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e80692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e809a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e8136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e809cbe0 con 0x7fe6e80692f0 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e811d000 con 0x7fe6e8068f20 2024-09-15T01:51:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e809bab0 con 0x7fe6e8136fe0 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ee79d640 10 monclient(hunting): authenticate will time out at 765.869669s 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ecd3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 msgr2=0x7fe6e809a5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6ecd3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e809a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e8068f20 0x7fe6e8130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39130/0 (socket says 172.21.15.17:39130) 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): get_auth_request con 0x7fe6e8068f20 auth_method 0 2024-09-15T01:51:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 1 -- 172.21.15.17:0/3628748041 learned_addr learned my addr 172.21.15.17:0/3628748041 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request con 0x7fe6e80692f0 auth_method 0 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:45.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.873+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 -- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 msgr2=0x7fe6e809a5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e809a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 -- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e8068f20 msgr2=0x7fe6e8130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e8068f20 0x7fe6e8130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 monclient: found mon.noname-a 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:45.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.874+0000 7fe6e77fe640 1 -- 172.21.15.17:0/3628748041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc009390 con 0x7fe6e80692f0 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6e77fe640 1 -- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 msgr2=0x7fe6e809a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6e77fe640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0098e0 tx=0x7fe6dc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0318c0 con 0x7fe6e80692f0 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: dump: 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.894 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: _finish_auth 0 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 5 monclient: authenticate success, global_id 4377 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc031a20 con 0x7fe6e80692f0 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc031dc0 con 0x7fe6e80692f0 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 10 monclient: _renew_subs 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8142290 con 0x7fe6e80692f0 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:45.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6cffff640 10 monclient: dump: 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6e67fc640 4 set_mon_vals no callback set 2024-09-15T01:51:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 10 monclient: _renew_subs 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.876+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e8137880 con 0x7fe6e80692f0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe6dc039040 con 0x7fe6e80692f0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6ee79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6ee79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6ee79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e80724a0 con 0x7fe6e80692f0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.877+0000 7fe6cffff640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 0x7fe6d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.878+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe6dc06bd30 con 0x7fe6e80692f0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.878+0000 7fe6e7fff640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 0x7fe6d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.878+0000 7fe6e7fff640 10 monclient: get_auth_request con 0x7fe6d40386f0 auth_method 0 2024-09-15T01:51:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.880+0000 7fe6e7fff640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 0x7fe6d403aba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8007fd0 tx=0x7fe6d8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.882+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe6dc03c680 con 0x7fe6e80692f0 2024-09-15T01:51:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.882+0000 7fe6cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:45.882+0000 7fe6cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.110+0000 7fe6ee79d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:51:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.110+0000 7fe6ee79d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.110+0000 7fe6ee79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.110+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6e806f150 con 0x7fe6e80692f0 2024-09-15T01:51:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.111+0000 7fe6cffff640 1 -- 172.21.15.17:0/3628748041 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fe6dc040080 con 0x7fe6e80692f0 2024-09-15T01:51:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.111+0000 7fe6cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.111+0000 7fe6cffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:46.113 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 msgr2=0x7fe6d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 0x7fe6d403aba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8007fd0 tx=0x7fe6d8007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 10 monclient: shutdown 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 msgr2=0x7fe6e809a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0098e0 tx=0x7fe6dc032000 comp rx=0 tx=0).stop 2024-09-15T01:51:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 shutdown_connections 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6d40386f0 0x7fe6d403aba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6e80692f0 0x7fe6e809a060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe6e8136fe0 0x7fe6e809a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 --2- 172.21.15.17:0/3628748041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e8068f20 0x7fe6e8130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 >> 172.21.15.17:0/3628748041 conn(0x7fe6e8070fb0 msgr2=0x7fe6e809dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 shutdown_connections 2024-09-15T01:51:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:46.113+0000 7fe6ee79d640 1 -- 172.21.15.17:0/3628748041 wait complete. 2024-09-15T01:51:49.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:51:49.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.233+0000 7fe7f5846640 10 monclient: get_monmap_and_config 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.233+0000 7fe7f5846640 10 monclient: build_initial_monmap 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.233+0000 7fe7f5846640 10 monclient: monmap: 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:49.234655+0000 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:49.234655+0000 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:49.236 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:49.237 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:49.237 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:49.237 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.234+0000 7fe7f5846640 1 Processor -- start 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.234+0000 7fe7f5846640 1 -- start start 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.234+0000 7fe7f5846640 10 monclient: init 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f0131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7f00a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:49.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f009f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f00a0040 con 0x7fe7f00a0280 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f009bff0 con 0x7fe7f009f1e0 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f00a14a0 con 0x7fe7f0131b20 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7f5846640 10 monclient(hunting): authenticate will time out at 769.231147s 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7eeffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 msgr2=0x7fe7f009eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7eeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ee7fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe7f00a0280 0x7fe7f0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46540/0 (socket says 172.21.15.17:46540) 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ee7fc640 1 -- 172.21.15.17:0/2434477483 learned_addr learned my addr 172.21.15.17:0/2434477483 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7f009f1e0 auth_method 0 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:49.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.235+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:49.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.237+0000 7fe7ee7fc640 1 -- 172.21.15.17:0/2434477483 >> v1:172.21.15.17:6789/0 conn(0x7fe7f00a0280 legacy=0x7fe7f0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:49.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 msgr2=0x7fe7f009eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:49.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:51:49.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:49.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7f009f1e0 auth_method 0 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:49.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.436+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:51:49.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 msgr2=0x7fe7f009eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:49.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:51:49.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7f009f1e0 auth_method 0 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:49.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:49.837+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:51:50.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 msgr2=0x7fe7f009eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:50.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:51:50.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:50.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7f009f1e0 auth_method 0 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:50.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:50.638+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): tick 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 monclient(hunting): continuing hunt 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 msgr2=0x7fe7f009eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7f0131b20 0x7fe7f009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 msgr2=0x7fe7f0138e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.67:3300/0 conn(0x7fe7f009f1e0 0x7fe7f0138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 -- 172.21.15.17:0/2434477483 >> v1:172.21.15.17:6789/0 conn(0x7fe7f00a0280 legacy=0x7fe7f0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient: _add_conns ranks=[3,1] 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7dc003720 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7dc006260 addr v1:172.21.15.67:6789/0 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 -- 172.21.15.17:0/2434477483 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f0132600 con 0x7fe7dc006260 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 1 -- 172.21.15.17:0/2434477483 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f009bff0 con 0x7fe7dc003720 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7ecff9640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request con 0x7fe7dc003720 auth_method 0 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.235+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:52.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.237+0000 7fe7ee7fc640 1 -- 172.21.15.17:0/2434477483 >> v1:172.21.15.67:6789/0 conn(0x7fe7dc006260 legacy=0x7fe7dc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:52.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:52.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request con 0x7fe7dc003720 auth_method 0 2024-09-15T01:51:52.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:52.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:52.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:52.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:52.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:52.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:52.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.436+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request con 0x7fe7dc003720 auth_method 0 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:51:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:51:52.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:52.837+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286722264 0 0) 0x7fe7f0132600 con 0x7fe7dc006260 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient(hunting): my global_id is 4387 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d80034e0 con 0x7fe7dc006260 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912526778 0 0) 0x7fe7d80034e0 con 0x7fe7dc006260 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 msgr2=0x7fe7dc005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 --2- 172.21.15.17:0/2434477483 >> v2:172.21.15.17:3300/0 conn(0x7fe7dc003720 0x7fe7dc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc00b000 con 0x7fe7dc006260 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.243+0000 7fe7edffb640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe7e40016e0 con 0x7fe7dc006260 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient: dump: 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.249 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 >> v1:172.21.15.67:6789/0 conn(0x7fe7dc006260 legacy=0x7fe7dc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 reap_dead start 2024-09-15T01:51:53.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 0x7fe7d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): picked mon.c con 0x7fe7d80039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8004780 0x7fe7d8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): picked mon.b con 0x7fe7d8004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 0x7fe7d800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): picked mon.a con 0x7fe7d8005180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f009bff0 con 0x7fe7d8005180 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d800c120 con 0x7fe7d8004780 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d800c2e0 con 0x7fe7d80039d0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7edffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7eeffd640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 msgr2=0x7fe7d8003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 0x7fe7d8003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 0x7fe7d800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8004780 0x7fe7d8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7d8005180 auth_method 0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): get_auth_request con 0x7fe7d8004780 auth_method 0 2024-09-15T01:51:53.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:51:53.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_done global_id 4387 payload 293 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 msgr2=0x7fe7d8003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 0x7fe7d8003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8004780 msgr2=0x7fe7d8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8004780 0x7fe7d8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 monclient: found mon.a 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80034e0 con 0x7fe7d8005180 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7f5846640 5 monclient: authenticate success, global_id 4387 2024-09-15T01:51:53.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.244+0000 7fe7f5846640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.247+0000 7fe7ef7fe640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 msgr2=0x7fe7d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.247+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 0x7fe7d800b9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7f009f710 tx=0x7fe7e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13bcc3dd4cf86e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8031620 con 0x7fe7d8005180 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: dump: 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.261 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.262 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.262 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7f5846640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800f040 con 0x7fe7d8005180 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 1 -- 172.21.15.17:0/2434477483 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80319f0 con 0x7fe7d8005180 2024-09-15T01:51:53.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7edffb640 10 monclient: dump: 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.264 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7f5846640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.248+0000 7fe7f5846640 4 set_mon_vals no callback set 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 10 monclient: shutdown 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 -- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 msgr2=0x7fe7d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 0x7fe7d800b9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7f009f710 tx=0x7fe7e800b040 comp rx=0 tx=0).stop 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 -- 172.21.15.17:0/2434477483 shutdown_connections 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d8005180 0x7fe7d800b9d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8004780 0x7fe7d8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 --2- 172.21.15.17:0/2434477483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d80039d0 0x7fe7d8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 -- 172.21.15.17:0/2434477483 >> 172.21.15.17:0/2434477483 conn(0x7fe7f0071270 msgr2=0x7fe7f00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 -- 172.21.15.17:0/2434477483 shutdown_connections 2024-09-15T01:51:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.249+0000 7fe7f5846640 1 -- 172.21.15.17:0/2434477483 wait complete. 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.250+0000 7fe7f5846640 10 monclient: build_initial_monmap 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.250+0000 7fe7f5846640 10 monclient: monmap: 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.250+0000 7fe7f5846640 1 Processor -- start 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.250+0000 7fe7f5846640 1 -- start start 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.250+0000 7fe7f5846640 10 monclient: init 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 0x7fe7f011d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d8004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8005180 0x7fe7f011dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d8005180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f011fa50 con 0x7fe7d80039d0 2024-09-15T01:51:53.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f00a09b0 con 0x7fe7d8005180 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f00a45c0 con 0x7fe7d8004780 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7f5846640 10 monclient(hunting): authenticate will time out at 773.247458s 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ee7fc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 msgr2=0x7fe7f011d820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ee7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 0x7fe7f011d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57696/0 (socket says 172.21.15.17:57696) 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 -- 172.21.15.17:0/1255609414 learned_addr learned my addr 172.21.15.17:0/1255609414 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request con 0x7fe7d80039d0 auth_method 0 2024-09-15T01:51:53.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8005180 0x7fe7f011dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe7d8005180 auth_method 0 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 -- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 msgr2=0x7fe7f011d820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 0x7fe7f011d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 -- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8005180 msgr2=0x7fe7f011dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8005180 0x7fe7f011dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 monclient: found mon.noname-a 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.251+0000 7fe7eeffd640 1 -- 172.21.15.17:0/1255609414 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f0120b80 con 0x7fe7d80039d0 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7eeffd640 1 -- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 msgr2=0x7fe7f011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7eeffd640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007090 tx=0x7fe7e0009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0007e90 con 0x7fe7d80039d0 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: dump: 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0015420 con 0x7fe7d80039d0 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 5 monclient: authenticate success, global_id 4380 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 10 monclient: _renew_subs 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f01208a0 con 0x7fe7d80039d0 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00157e0 con 0x7fe7d80039d0 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7d6ffd640 10 monclient: dump: 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 10 monclient: _renew_subs 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f0130ad0 con 0x7fe7d80039d0 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.254+0000 7fe7ed7fa640 4 set_mon_vals no callback set 2024-09-15T01:51:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7f5846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7f5846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7f5846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe7e0015b90 con 0x7fe7d80039d0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f0120dc0 con 0x7fe7d80039d0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.255+0000 7fe7d6ffd640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 0x7fe7dc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.256+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe7e004c2f0 con 0x7fe7d80039d0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.256+0000 7fe7ee7fc640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 0x7fe7dc03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.256+0000 7fe7ee7fc640 10 monclient: get_auth_request con 0x7fe7dc038fe0 auth_method 0 2024-09-15T01:51:53.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.258+0000 7fe7ee7fc640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 0x7fe7dc03b490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4000e10 tx=0x7fe7e4007940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.260+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe7e00138e0 con 0x7fe7d80039d0 2024-09-15T01:51:53.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.260+0000 7fe7d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.260+0000 7fe7d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.488+0000 7fe7f5846640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.488+0000 7fe7f5846640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.488+0000 7fe7f5846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.488+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe7f00724a0 con 0x7fe7d80039d0 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.489+0000 7fe7d6ffd640 1 -- 172.21.15.17:0/1255609414 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1520 (secure 0 0 0) 0x7fe7e001c310 con 0x7fe7d80039d0 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.489+0000 7fe7d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:51:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.489+0000 7fe7d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:53.492 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:51:53.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 msgr2=0x7fe7dc03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 0x7fe7dc03b490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4000e10 tx=0x7fe7e4007940 comp rx=0 tx=0).stop 2024-09-15T01:51:53.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 10 monclient: shutdown 2024-09-15T01:51:53.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 msgr2=0x7fe7f011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007090 tx=0x7fe7e0009770 comp rx=0 tx=0).stop 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 shutdown_connections 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7dc038fe0 0x7fe7dc03b490 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d8005180 0x7fe7f011dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7d8004780 0x7fe7f011d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 --2- 172.21.15.17:0/1255609414 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7d80039d0 0x7fe7f011d2e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 >> 172.21.15.17:0/1255609414 conn(0x7fe7f0071270 msgr2=0x7fe7f00a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 shutdown_connections 2024-09-15T01:51:53.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.491+0000 7fe7f5846640 1 -- 172.21.15.17:0/1255609414 wait complete. 2024-09-15T01:51:53.504 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:51:53.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:51:53.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:51:53.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.599+0000 7fae26f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:51:53.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.599+0000 7fae26f9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.599+0000 7fae26f9d640 10 monclient: monmap: 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:53.600460+0000 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:53.600460+0000 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:53.602 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 Processor -- start 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 -- start start 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient: init 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae20068f20 addr v1:172.21.15.17:6789/0 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae200696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fae200692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae20136fe0 0x7fae201393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fae20136fe0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae200a1950 con 0x7fae20068f20 2024-09-15T01:51:53.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae200a1670 con 0x7fae20136fe0 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae201328d0 con 0x7fae200692f0 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae26f9d640 10 monclient(hunting): authenticate will time out at 773.596900s 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae1ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 msgr2=0x7fae200696c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.600+0000 7fae1ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae200696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae24d39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fae20068f20 0x7fae201321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55734/0 (socket says 172.21.15.17:55734) 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae24d39640 1 -- 172.21.15.17:0/1773094723 learned_addr learned my addr 172.21.15.17:0/1773094723 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 --2- 172.21.15.17:0/1773094723 >> v2:172.21.15.67:3300/0 conn(0x7fae20136fe0 0x7fae201393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): get_auth_request con 0x7fae20136fe0 auth_method 0 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 -- 172.21.15.17:0/1773094723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 msgr2=0x7fae200696c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 --2- 172.21.15.17:0/1773094723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae200696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 -- 172.21.15.17:0/1773094723 >> v1:172.21.15.17:6789/0 conn(0x7fae20068f20 legacy=0x7fae201321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 monclient: found mon.noname-c 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:51:53.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.601+0000 7fae2553a640 1 -- 172.21.15.17:0/1773094723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae200a0020 con 0x7fae20136fe0 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae2553a640 1 -- 172.21.15.17:0/1773094723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20136fe0 msgr2=0x7fae201393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae2553a640 1 --2- 172.21.15.17:0/1773094723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20136fe0 0x7fae201393c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fae18009e60 tx=0x7fae18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51eae57fe15831a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 1 -- 172.21.15.17:0/1773094723 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18007840 con 0x7fae20136fe0 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: dump: 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.613 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 1 -- 172.21.15.17:0/1773094723 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1802d050 con 0x7fae20136fe0 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 1 -- 172.21.15.17:0/1773094723 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18007bd0 con 0x7fae20136fe0 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae1f7fe640 10 monclient: dump: 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae26f9d640 5 monclient: authenticate success, global_id 4393 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae26f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae26f9d640 4 set_mon_vals no callback set 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.604+0000 7fae26f9d640 10 monclient: shutdown 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 -- 172.21.15.17:0/1773094723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20136fe0 msgr2=0x7fae201393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1773094723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20136fe0 0x7fae201393c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fae18009e60 tx=0x7fae18009780 comp rx=0 tx=0).stop 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 -- 172.21.15.17:0/1773094723 shutdown_connections 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1773094723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20136fe0 0x7fae201393c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1773094723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae200696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 -- 172.21.15.17:0/1773094723 >> 172.21.15.17:0/1773094723 conn(0x7fae20070fb0 msgr2=0x7fae200a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 -- 172.21.15.17:0/1773094723 shutdown_connections 2024-09-15T01:51:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 -- 172.21.15.17:0/1773094723 wait complete. 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 10 monclient: build_initial_monmap 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 10 monclient: monmap: 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.605+0000 7fae26f9d640 1 Processor -- start 2024-09-15T01:51:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 -- start start 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient: init 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20068f20 0x7fae20099e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fae20068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae2009a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fae200692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 0x7fae2009a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fae20136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae200a0750 con 0x7fae20136fe0 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2009c550 con 0x7fae20068f20 2024-09-15T01:51:53.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2011ceb0 con 0x7fae200692f0 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae26f9d640 10 monclient(hunting): authenticate will time out at 773.602530s 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae1ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 msgr2=0x7fae2009a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae1ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae2009a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 0x7fae2009a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20068f20 0x7fae20099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 0x7fae2009a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57700/0 (socket says 172.21.15.17:57700) 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 learned_addr learned my addr 172.21.15.17:0/1881740461 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): get_auth_request con 0x7fae20136fe0 auth_method 0 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): get_auth_request con 0x7fae20068f20 auth_method 0 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 msgr2=0x7fae2009a350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:53.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae200692f0 0x7fae2009a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20068f20 msgr2=0x7fae20099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae20068f20 0x7fae20099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 monclient: found mon.noname-a 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.606+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18009a20 con 0x7fae20136fe0 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 msgr2=0x7fae2009a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae2553a640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 0x7fae2009a890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae180099f0 tx=0x7fae18037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18031cd0 con 0x7fae20136fe0 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: dump: 2024-09-15T01:51:53.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 5 monclient: authenticate success, global_id 4383 2024-09-15T01:51:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae18040070 con 0x7fae20136fe0 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 10 monclient: _renew_subs 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2011db90 con 0x7fae20136fe0 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18045640 con 0x7fae20136fe0 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1cff9640 10 monclient: dump: 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.623 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae1effd640 4 set_mon_vals no callback set 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 10 monclient: _renew_subs 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.609+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2011c8c0 con 0x7fae20136fe0 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae26f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 0x7fae2002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20063b90 con 0x7fae2009add0 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fae18003020 con 0x7fae20136fe0 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 0x7fae2002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient: get_auth_request con 0x7fae2009add0 auth_method 0 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae1cff9640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae04038720 0x7fae0403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae24d39640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae200724a0 con 0x7fae2009add0 2024-09-15T01:51:53.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fae1802ae40 con 0x7fae20136fe0 2024-09-15T01:51:53.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae1ffff640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae04038720 0x7fae0403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.610+0000 7fae1ffff640 10 monclient: get_auth_request con 0x7fae04038720 auth_method 0 2024-09-15T01:51:53.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.612+0000 7fae24d39640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 msgr2=0x7fae2002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.612+0000 7fae24d39640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 0x7fae2002aa40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14004b50 con 0x7fae2009add0 2024-09-15T01:51:53.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 10 monclient: dump: 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fae14004cb0 con 0x7fae2009add0 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 msgr2=0x7fae2002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 0x7fae2002aa40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 msgr2=0x7fae2002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1cff9640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2009add0 0x7fae2002aa40 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14007f00 comp rx=0 tx=0).stop 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.613+0000 7fae1ffff640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae04038720 0x7fae0403abd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae0c006fd0 tx=0x7fae0c006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.618+0000 7fae26f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.618+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 0x7fae2005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae26f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2011cab0 con 0x7fae2005e310 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 0x7fae2005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient: get_auth_request con 0x7fae2005e310 auth_method 0 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.619+0000 7fae24d39640 1 -- 172.21.15.17:0/1881740461 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fae2009b6d0 con 0x7fae2005e310 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae24d39640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 msgr2=0x7fae2005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae24d39640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 0x7fae2005e710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14015d30 con 0x7fae2005e310 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.621+0000 7fae1cff9640 10 monclient: dump: 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7fae14015e90 con 0x7fae2005e310 2024-09-15T01:51:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 msgr2=0x7fae2005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 0x7fae2005e710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14019000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 msgr2=0x7fae2005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae1cff9640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae2005e310 0x7fae2005e710 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae1400e710 tx=0x7fae14019000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:53.633 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 8, 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:51:53.636 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.637 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:53.638 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.644 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:51:53.645 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae04038720 msgr2=0x7fae0403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae04038720 0x7fae0403abd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae0c006fd0 tx=0x7fae0c006d30 comp rx=0 tx=0).stop 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 10 monclient: shutdown 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 msgr2=0x7fae2009a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 --2- 172.21.15.17:0/1881740461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae20136fe0 0x7fae2009a890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae180099f0 tx=0x7fae18037000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 reap_dead start 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae2553a640 1 -- 172.21.15.17:0/1881740461 reap_dead start 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 shutdown_connections 2024-09-15T01:51:53.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 >> 172.21.15.17:0/1881740461 conn(0x7fae20070fb0 msgr2=0x7fae2009ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 shutdown_connections 2024-09-15T01:51:53.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.622+0000 7fae26f9d640 1 -- 172.21.15.17:0/1881740461 wait complete. 2024-09-15T01:51:53.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:51:53.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:51:53.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:51:53.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.743+0000 7fa714834640 10 monclient: get_monmap_and_config 2024-09-15T01:51:53.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.743+0000 7fa714834640 10 monclient: build_initial_monmap 2024-09-15T01:51:53.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.743+0000 7fa714834640 10 monclient: monmap: 2024-09-15T01:51:53.745 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:53.745 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:51:53.744296+0000 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:51:53.744296+0000 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:51:53.746 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 Processor -- start 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 -- start start 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient: init 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa7100a5780 0x7fa71009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): picked mon.noname-a con 0x7fa7100a5780 addr v2:172.21.15.17:3300/0 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 0x7fa710069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): picked mon.noname-c con 0x7fa71009ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa710069f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): picked mon.noname-b con 0x7fa710069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa71006a460 con 0x7fa7100a5780 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7100a18b0 con 0x7fa71009ccf0 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100a5b50 con 0x7fa710069ac0 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa714834640 10 monclient(hunting): authenticate will time out at 773.740835s 2024-09-15T01:51:53.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa70ed9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 msgr2=0x7fa710069f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa70ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa710069f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa70e59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa7100a5780 0x7fa71009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.744+0000 7fa70e59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa7100a5780 0x7fa71009bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57724/0 (socket says 172.21.15.17:57724) 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 -- 172.21.15.17:0/2569646153 learned_addr learned my addr 172.21.15.17:0/2569646153 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): get_auth_request con 0x7fa7100a5780 auth_method 0 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 1 --2- 172.21.15.17:0/2569646153 >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 0x7fa710069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 10 monclient(hunting): get_auth_request con 0x7fa71009ccf0 auth_method 0 2024-09-15T01:51:53.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 -- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 msgr2=0x7fa710069f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 --2- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa710069f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 -- 172.21.15.17:0/2569646153 >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 msgr2=0x7fa710069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 --2- 172.21.15.17:0/2569646153 >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 0x7fa710069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 monclient: found mon.noname-a 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70e59c640 1 -- 172.21.15.17:0/2569646153 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71009c5f0 con 0x7fa7100a5780 2024-09-15T01:51:53.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.745+0000 7fa70dd9b640 1 --2- 172.21.15.17:0/2569646153 >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 0x7fa710069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:51:53.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70e59c640 1 -- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 msgr2=0x7fa71009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70e59c640 1 --2- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa71009bc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa710068e70 tx=0x7fa7080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7be710cb0e9707d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 1 -- 172.21.15.17:0/2569646153 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708032440 con 0x7fa7100a5780 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: dump: 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 5 monclient: authenticate success, global_id 4392 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 1 -- 172.21.15.17:0/2569646153 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7080325a0 con 0x7fa7100a5780 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 10 monclient: get_monmap_and_config success 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 4 set_mon_vals no callback set 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 1 -- 172.21.15.17:0/2569646153 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708032940 con 0x7fa7100a5780 2024-09-15T01:51:53.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa70d59a640 10 monclient: dump: 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 10 monclient: shutdown 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 -- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 msgr2=0x7fa71009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 --2- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa71009bc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa710068e70 tx=0x7fa7080091d0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 -- 172.21.15.17:0/2569646153 shutdown_connections 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 --2- 172.21.15.17:0/2569646153 >> v2:172.21.15.67:3300/0 conn(0x7fa71009ccf0 0x7fa710069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 --2- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa710069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 --2- 172.21.15.17:0/2569646153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa71009bc90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 -- 172.21.15.17:0/2569646153 >> 172.21.15.17:0/2569646153 conn(0x7fa710070e10 msgr2=0x7fa7100a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 -- 172.21.15.17:0/2569646153 shutdown_connections 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.748+0000 7fa714834640 1 -- 172.21.15.17:0/2569646153 wait complete. 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.749+0000 7fa714834640 10 monclient: build_initial_monmap 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.749+0000 7fa714834640 10 monclient: monmap: 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:51:53.763 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.749+0000 7fa714834640 1 Processor -- start 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.749+0000 7fa714834640 1 -- start start 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.749+0000 7fa714834640 10 monclient: init 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient: _reopen_session rank -1 2024-09-15T01:51:53.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa71013dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): picked mon.noname-c con 0x7fa710069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71009ccf0 0x7fa71013d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): picked mon.noname-b con 0x7fa71009ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa710141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): picked mon.noname-a con 0x7fa7100a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710120a60 con 0x7fa7100a5780 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710122750 con 0x7fa71009ccf0 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa710120230 con 0x7fa710069ac0 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): _renew_subs 2024-09-15T01:51:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa714834640 10 monclient(hunting): authenticate will time out at 773.746145s 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70e59c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 msgr2=0x7fa71013dd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa71013dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa710141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa710141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57726/0 (socket says 172.21.15.17:57726) 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 learned_addr learned my addr 172.21.15.17:0/1855457859 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): get_auth_request con 0x7fa7100a5780 auth_method 0 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71009ccf0 0x7fa71013d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): get_auth_request con 0x7fa71009ccf0 auth_method 0 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 msgr2=0x7fa71013dd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa710069ac0 0x7fa71013dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71009ccf0 msgr2=0x7fa71013d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71009ccf0 0x7fa71013d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 monclient: found mon.noname-a 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.750+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708009450 con 0x7fa7100a5780 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.752+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 msgr2=0x7fa710141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.752+0000 7fa70ed9d640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa710141290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa70000beb0 tx=0x7fa7000096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.752+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700016430 con 0x7fa7100a5780 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.752+0000 7fa6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: dump: 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: _finish_auth 0 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7000165d0 con 0x7fa7100a5780 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 5 monclient: authenticate success, global_id 4395 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:51:53.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 monclient: _renew_subs 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700016780 con 0x7fa7100a5780 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa710122470 con 0x7fa7100a5780 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa6fe7fc640 10 monclient: dump: 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa70cd99640 4 set_mon_vals no callback set 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 monclient: _renew_subs 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71013d910 con 0x7fa7100a5780 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 0x7fa7101417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.753+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710063b20 con 0x7fa7101215a0 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa700012020 con 0x7fa7100a5780 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 0x7fa7101417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient: get_auth_request con 0x7fa7101215a0 auth_method 0 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa6fe7fc640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6f0038740 0x7fa6f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fa70004bdc0 con 0x7fa7100a5780 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70e59c640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa71009b7d0 con 0x7fa7101215a0 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70dd9b640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6f0038740 0x7fa6f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.754+0000 7fa70dd9b640 10 monclient: get_auth_request con 0x7fa6f0038740 auth_method 0 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa70e59c640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 msgr2=0x7fa7101417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa70e59c640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 0x7fa7101417d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa71006a5b0 tx=0x7fa708032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708032440 con 0x7fa7101215a0 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa6fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa70dd9b640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6f0038740 0x7fa6f003abf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa710120e00 tx=0x7fa704008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.756+0000 7fa6fe7fc640 10 monclient: dump: 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.773 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa708031e90 con 0x7fa7101215a0 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 msgr2=0x7fa7101417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 0x7fa7101417d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa71006a5b0 tx=0x7fa708032000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 msgr2=0x7fa7101417d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:53.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.757+0000 7fa6fe7fc640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7101215a0 0x7fa7101417d0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa71006a5b0 tx=0x7fa708032000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa714834640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:51:53.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa714834640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 0x7fa71011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:51:53.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa714834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:51:53.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71013e940 con 0x7fa710141d10 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 0x7fa71011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient: get_auth_request con 0x7fa710141d10 auth_method 0 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.763+0000 7fa70e59c640 1 -- 172.21.15.17:0/1855457859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa710122b50 con 0x7fa710141d10 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.765+0000 7fa70e59c640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 msgr2=0x7fa71011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.765+0000 7fa70e59c640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 0x7fa71011b6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa7100638f0 tx=0x7fa70803c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:51:53.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70804a070 con 0x7fa710141d10 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 10 monclient: dump: 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:51:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4723 (secure 0 0 0) 0x7fa708032920 con 0x7fa710141d10 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 msgr2=0x7fa71011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 0x7fa71011b6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa7100638f0 tx=0x7fa70803c000 comp rx=0 tx=0).stop 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 msgr2=0x7fa71011b6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa6fe7fc640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa710141d10 0x7fa71011b6d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:53.779 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 8, 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.780 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.781 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:53.782 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:51:53.783 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:51:53.784 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.786 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:51:53.787 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:51:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:51:53.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.790 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:51:53.791 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:51:53.792 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:51:53.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6f0038740 msgr2=0x7fa6f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6f0038740 0x7fa6f003abf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa710120e00 tx=0x7fa704008040 comp rx=0 tx=0).stop 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 10 monclient: shutdown 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 msgr2=0x7fa710141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 --2- 172.21.15.17:0/1855457859 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa7100a5780 0x7fa710141290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa70000beb0 tx=0x7fa7000096e0 comp rx=0 tx=0).stop 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 reap_dead start 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa70ed9d640 1 -- 172.21.15.17:0/1855457859 reap_dead start 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 shutdown_connections 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 >> 172.21.15.17:0/1855457859 conn(0x7fa710070e10 msgr2=0x7fa71009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 shutdown_connections 2024-09-15T01:51:53.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:51:53.766+0000 7fa714834640 1 -- 172.21.15.17:0/1855457859 wait complete. 2024-09-15T01:51:53.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:51:53.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:52:13.796 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:52:13.796 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T01:52:13.796 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:52:13.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:52:13.800 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:52:13.800 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:52:13.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.907+0000 7f4b7379d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.907+0000 7f4b7379d640 10 monclient: build_initial_monmap 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.907+0000 7f4b7379d640 10 monclient: monmap: 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:13.907938+0000 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:13.907938+0000 2024-09-15T01:52:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:13.910 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.907+0000 7f4b7379d640 1 Processor -- start 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.907+0000 7f4b7379d640 1 -- start start 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient: init 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-b con 0x7f4b6c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b6c0a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b6c09f1e0 0x7f4b6c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6c09f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0a0040 con 0x7f4b6c0a0280 2024-09-15T01:52:13.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c09bff0 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a14a0 con 0x7f4b6c131b20 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b7379d640 10 monclient(hunting): authenticate will time out at 793.904536s 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41526/0 (socket says 172.21.15.17:41526) 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 -- 172.21.15.17:0/4085312684 learned_addr learned my addr 172.21.15.17:0/4085312684 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 10 monclient(hunting): get_auth_request con 0x7f4b6c131b20 auth_method 0 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:13.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/4085312684 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c09f1e0 0x7f4b6c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): get_auth_request con 0x7f4b6c09f1e0 auth_method 0 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 -- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 msgr2=0x7f4b6c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 -- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 msgr2=0x7f4b6c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71539640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.908+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_done global_id 4411 payload 274 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 1 -- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 msgr2=0x7f4b6c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 1 -- 172.21.15.17:0/4085312684 >> v1:172.21.15.67:6789/0 conn(0x7f4b6c0a0280 legacy=0x7f4b6c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 1 monclient: found mon.noname-c 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:13.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.909+0000 7f4b71d3a640 1 -- 172.21.15.17:0/4085312684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c132600 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.911+0000 7f4b71d3a640 1 -- 172.21.15.17:0/4085312684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c09f1e0 msgr2=0x7f4b6c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.911+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c09f1e0 0x7f4b6c138e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009e60 tx=0x7f4b64009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c0dca45caab999 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 1 -- 172.21.15.17:0/4085312684 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64007760 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: dump: 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 1 -- 172.21.15.17:0/4085312684 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6402d050 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 5 monclient: authenticate success, global_id 4411 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 4 set_mon_vals no callback set 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 1 -- 172.21.15.17:0/4085312684 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64007af0 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:13.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b63fff640 10 monclient: dump: 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 10 monclient: shutdown 2024-09-15T01:52:13.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 -- 172.21.15.17:0/4085312684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c09f1e0 msgr2=0x7f4b6c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c09f1e0 0x7f4b6c138e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009e60 tx=0x7f4b64009780 comp rx=0 tx=0).stop 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 -- 172.21.15.17:0/4085312684 shutdown_connections 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c09f1e0 0x7f4b6c138e60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 --2- 172.21.15.17:0/4085312684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c131b20 0x7f4b6c09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 -- 172.21.15.17:0/4085312684 >> 172.21.15.17:0/4085312684 conn(0x7f4b6c071270 msgr2=0x7f4b6c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 -- 172.21.15.17:0/4085312684 shutdown_connections 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.912+0000 7f4b7379d640 1 -- 172.21.15.17:0/4085312684 wait complete. 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.913+0000 7f4b7379d640 10 monclient: build_initial_monmap 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.913+0000 7f4b7379d640 10 monclient: monmap: 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:13.925 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.913+0000 7f4b7379d640 1 Processor -- start 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.913+0000 7f4b7379d640 1 -- start start 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.913+0000 7f4b7379d640 10 monclient: init 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:13.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 0x7f4b6c12d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-b con 0x7f4b6c0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 0x7f4b6c12dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a09b0 con 0x7f4b6c131b20 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c139060 con 0x7f4b6c0a0280 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c097a10 con 0x7f4b6c09f1e0 2024-09-15T01:52:13.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b7379d640 10 monclient(hunting): authenticate will time out at 793.910285s 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41540/0 (socket says 172.21.15.17:41540) 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 -- 172.21.15.17:0/3279628735 learned_addr learned my addr 172.21.15.17:0/3279628735 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 10 monclient(hunting): get_auth_request con 0x7f4b6c09f1e0 auth_method 0 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 0x7f4b6c12dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 0x7f4b6c12d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): get_auth_request con 0x7f4b6c131b20 auth_method 0 2024-09-15T01:52:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 msgr2=0x7f4b6c12d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 msgr2=0x7f4b6c12d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): get_auth_request con 0x7f4b6c0a0280 auth_method 0 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71539640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:13.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 msgr2=0x7f4b6c12d2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 msgr2=0x7f4b6c12d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 0x7f4b6c12d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 monclient: found mon.noname-a 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b71d3a640 1 -- 172.21.15.17:0/3279628735 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64009a20 con 0x7f4b6c131b20 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.914+0000 7f4b70d38640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 0x7f4b6c12d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b71d3a640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 msgr2=0x7f4b6c12dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b71d3a640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 0x7f4b6c12dd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4b64031000 tx=0x7f4b64037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64031c10 con 0x7f4b6c131b20 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: dump: 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64040070 con 0x7f4b6c131b20 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 5 monclient: authenticate success, global_id 4398 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 10 monclient: _renew_subs 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c097790 con 0x7f4b6c131b20 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:13.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64045430 con 0x7f4b6c131b20 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b617fa640 10 monclient: dump: 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:13.933 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 10 monclient: _renew_subs 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b637fe640 4 set_mon_vals no callback set 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.917+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c0983f0 con 0x7f4b6c131b20 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4b64042030 con 0x7f4b6c131b20 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b7379d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b7379d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b7379d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c0724a0 con 0x7f4b6c131b20 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b617fa640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 0x7f4b5003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4b6406d3d0 con 0x7f4b6c131b20 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b71539640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 0x7f4b5003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.918+0000 7f4b71539640 10 monclient: get_auth_request con 0x7f4b50038740 auth_method 0 2024-09-15T01:52:13.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.921+0000 7f4b71539640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 0x7f4b5003abf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006f30 tx=0x7f4b68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:13.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.923+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4b6403c420 con 0x7f4b6c131b20 2024-09-15T01:52:13.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.923+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:13.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:13.923+0000 7f4b617fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:14.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:14.151+0000 7f4b7379d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:14.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:14.151+0000 7f4b7379d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:14.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:14.151+0000 7f4b7379d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:14.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:14.151+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b6c06efc0 con 0x7f4b6c131b20 2024-09-15T01:52:16.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:16.914+0000 7f4b57fff640 10 monclient: tick 2024-09-15T01:52:16.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:16.914+0000 7f4b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:16.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:16.914+0000 7f4b57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:17.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:17.914+0000 7f4b57fff640 10 monclient: tick 2024-09-15T01:52:17.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:17.914+0000 7f4b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:18.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:18.915+0000 7f4b57fff640 10 monclient: tick 2024-09-15T01:52:18.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:18.915+0000 7f4b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:19.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.113+0000 7f4b617fa640 1 -- 172.21.15.17:0/3279628735 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f4b640303a0 con 0x7f4b6c131b20 2024-09-15T01:52:19.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.113+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:19.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.113+0000 7f4b617fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:19.115 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:19.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 msgr2=0x7f4b5003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:19.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 0x7f4b5003abf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006f30 tx=0x7f4b68008040 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 10 monclient: shutdown 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 msgr2=0x7f4b6c12dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 0x7f4b6c12dd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4b64031000 tx=0x7f4b64037000 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 shutdown_connections 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b50038740 0x7f4b5003abf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b6c131b20 0x7f4b6c12dd30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0a0280 0x7f4b6c12d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 --2- 172.21.15.17:0/3279628735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c09f1e0 0x7f4b6c12d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 >> 172.21.15.17:0/3279628735 conn(0x7f4b6c071270 msgr2=0x7f4b6c0a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 shutdown_connections 2024-09-15T01:52:19.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:19.116+0000 7f4b7379d640 1 -- 172.21.15.17:0/3279628735 wait complete. 2024-09-15T01:52:22.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:22.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.239+0000 7fd594b9d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.239+0000 7fd594b9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.239+0000 7fd594b9d640 10 monclient: monmap: 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:22.240588+0000 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:22.240588+0000 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:22.242 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:22.243 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:22.243 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:22.243 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.240+0000 7fd594b9d640 1 Processor -- start 2024-09-15T01:52:22.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.240+0000 7fd594b9d640 1 -- start start 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.240+0000 7fd594b9d640 10 monclient: init 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5901325e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd5900a0700 0x7fd59009ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd5900a0700 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd590136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59006a650 con 0x7fd5901325e0 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd590132460 con 0x7fd5900a0700 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5901329b0 con 0x7fd590136e50 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd594b9d640 10 monclient(hunting): authenticate will time out at 802.237236s 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58e59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd5901325e0 0x7fd590131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46042/0 (socket says 172.21.15.17:46042) 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58e59c640 1 -- 172.21.15.17:0/1831184977 learned_addr learned my addr 172.21.15.17:0/1831184977 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd587fff640 1 --2- 172.21.15.17:0/1831184977 >> v2:172.21.15.17:3300/0 conn(0x7fd5900a0700 0x7fd59009ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd587fff640 10 monclient(hunting): get_auth_request con 0x7fd5900a0700 auth_method 0 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 10 monclient(hunting): get_auth_request con 0x7fd590136e50 auth_method 0 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 1 -- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:52:22.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 1 -- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:22.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.241+0000 7fd58ed9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd58e59c640 1 -- 172.21.15.17:0/1831184977 >> v1:172.21.15.17:6789/0 conn(0x7fd5901325e0 legacy=0x7fd590131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 1 -- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590139230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 1 -- 172.21.15.17:0/1831184977 >> v1:172.21.15.17:6789/0 conn(0x7fd5901325e0 legacy=0x7fd590131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 1 monclient: found mon.noname-a 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.243+0000 7fd587fff640 1 -- 172.21.15.17:0/1831184977 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5900a19b0 con 0x7fd5900a0700 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd587fff640 1 -- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5900a0700 msgr2=0x7fd59009ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd587fff640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5900a0700 0x7fd59009ec30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd5780098d0 tx=0x7fd5780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5201e0292a412191 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 1 -- 172.21.15.17:0/1831184977 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578032440 con 0x7fd5900a0700 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: dump: 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.259 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 1 -- 172.21.15.17:0/1831184977 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5780325a0 con 0x7fd5900a0700 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 1 -- 172.21.15.17:0/1831184977 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578032940 con 0x7fd5900a0700 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd594b9d640 5 monclient: authenticate success, global_id 4401 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd594b9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd594b9d640 4 set_mon_vals no callback set 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.246+0000 7fd58dd9b640 10 monclient: dump: 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.261 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 10 monclient: shutdown 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 -- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5900a0700 msgr2=0x7fd59009ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5900a0700 0x7fd59009ec30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd5780098d0 tx=0x7fd5780090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 -- 172.21.15.17:0/1831184977 shutdown_connections 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 --2- 172.21.15.17:0/1831184977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5900a0700 0x7fd59009ec30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 -- 172.21.15.17:0/1831184977 >> 172.21.15.17:0/1831184977 conn(0x7fd590071320 msgr2=0x7fd5900a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 -- 172.21.15.17:0/1831184977 shutdown_connections 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 1 -- 172.21.15.17:0/1831184977 wait complete. 2024-09-15T01:52:22.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.247+0000 7fd594b9d640 10 monclient: monmap: 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 Processor -- start 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 -- start start 2024-09-15T01:52:22.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient: init 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5900a0700 0x7fd59012e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd5900a0700 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd5901325e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd590136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59009b4a0 con 0x7fd5901325e0 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590099850 con 0x7fd5900a0700 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59009e0c0 con 0x7fd590136e50 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd594b9d640 10 monclient(hunting): authenticate will time out at 802.244745s 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41780/0 (socket says 172.21.15.17:41780) 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 1 -- 172.21.15.17:0/2744427295 learned_addr learned my addr 172.21.15.17:0/2744427295 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): get_auth_request con 0x7fd5901325e0 auth_method 0 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58ed9d640 10 monclient(hunting): get_auth_request con 0x7fd590136e50 auth_method 0 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.248+0000 7fd58e59c640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5900a0700 0x7fd59012e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58ed9d640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590097e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58ed9d640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590097e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): get_auth_request con 0x7fd5900a0700 auth_method 0 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58ed9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58ed9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd58e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 msgr2=0x7fd590097e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5900a0700 msgr2=0x7fd59012e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5900a0700 0x7fd59012e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 monclient: found mon.noname-a 2024-09-15T01:52:22.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.249+0000 7fd587fff640 1 -- 172.21.15.17:0/2744427295 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578009390 con 0x7fd5901325e0 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd587fff640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 msgr2=0x7fd5900978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd587fff640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd578009360 tx=0x7fd578032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578000d10 con 0x7fd5901325e0 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: dump: 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.268 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578031c70 con 0x7fd5901325e0 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 5 monclient: authenticate success, global_id 4407 2024-09-15T01:52:22.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57803d440 con 0x7fd5901325e0 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 10 monclient: _renew_subs 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59009ab10 con 0x7fd5901325e0 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd58d59a640 4 set_mon_vals no callback set 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd5867fc640 10 monclient: dump: 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.270 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 10 monclient: _renew_subs 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.251+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59009ae00 con 0x7fd5901325e0 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd578039040 con 0x7fd5901325e0 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd594b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd594b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd594b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5900724a0 con 0x7fd5901325e0 2024-09-15T01:52:22.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd5867fc640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.252+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd57806b960 con 0x7fd5901325e0 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.253+0000 7fd58e59c640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.253+0000 7fd58e59c640 10 monclient: get_auth_request con 0x7fd570038790 auth_method 0 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.255+0000 7fd58e59c640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd580005840 tx=0x7fd580009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.257+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd57802f2a0 con 0x7fd5901325e0 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.257+0000 7fd5867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.257+0000 7fd5867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.486+0000 7fd594b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.486+0000 7fd594b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.486+0000 7fd594b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.486+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd59009b880 con 0x7fd5901325e0 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.487+0000 7fd5867fc640 1 -- 172.21.15.17:0/2744427295 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fd578042080 con 0x7fd5901325e0 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.487+0000 7fd5867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.487+0000 7fd5867fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:22.489 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 msgr2=0x7fd57003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd580005840 tx=0x7fd580009040 comp rx=0 tx=0).stop 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 10 monclient: shutdown 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 msgr2=0x7fd5900978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd578009360 tx=0x7fd578032000 comp rx=0 tx=0).stop 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 shutdown_connections 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5901325e0 0x7fd5900978c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd590136e50 0x7fd590097e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 --2- 172.21.15.17:0/2744427295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5900a0700 0x7fd59012e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 >> 172.21.15.17:0/2744427295 conn(0x7fd590071320 msgr2=0x7fd590130710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 shutdown_connections 2024-09-15T01:52:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.489+0000 7fd594b9d640 1 -- 172.21.15.17:0/2744427295 wait complete. 2024-09-15T01:52:22.502 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:52:22.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:52:22.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.597+0000 7f6733660640 10 monclient: get_monmap_and_config 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.597+0000 7f6733660640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.597+0000 7f6733660640 10 monclient: monmap: 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:22.598651+0000 2024-09-15T01:52:22.600 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:22.598651+0000 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.598+0000 7f6733660640 1 Processor -- start 2024-09-15T01:52:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.598+0000 7f6733660640 1 -- start start 2024-09-15T01:52:22.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.598+0000 7f6733660640 10 monclient: init 2024-09-15T01:52:22.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f672c09c050 0x7f672c09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): picked mon.noname-c con 0x7f672c09c050 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f672c09c420 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f672c09c830 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c06a040 con 0x7f672c09c830 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c09f740 con 0x7f672c09c420 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c09aed0 con 0x7f672c09c050 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6733660640 10 monclient(hunting): authenticate will time out at 802.595176s 2024-09-15T01:52:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6731bfd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f672c09c830 0x7f672c139be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46046/0 (socket says 172.21.15.17:46046) 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f6731bfd640 1 -- 172.21.15.17:0/213666344 learned_addr learned my addr 172.21.15.17:0/213666344 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 1 --2- 172.21.15.17:0/213666344 >> v2:172.21.15.67:3300/0 conn(0x7f672c09c050 0x7f672c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): get_auth_request con 0x7f672c09c050 auth_method 0 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient(hunting): handle_auth_done global_id 4420 payload 274 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 1 -- 172.21.15.17:0/213666344 >> v1:172.21.15.67:6789/0 conn(0x7f672c09c420 legacy=0x7f672c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 1 -- 172.21.15.17:0/213666344 >> v1:172.21.15.17:6789/0 conn(0x7f672c09c830 legacy=0x7f672c139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 1 monclient: found mon.noname-c 2024-09-15T01:52:22.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:22.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.599+0000 7f67313fc640 1 -- 172.21.15.17:0/213666344 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c069cd0 con 0x7f672c09c050 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.602+0000 7f67313fc640 1 -- 172.21.15.17:0/213666344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c050 msgr2=0x7f672c09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.602+0000 7f67313fc640 1 --2- 172.21.15.17:0/213666344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c050 0x7f672c09a990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6724009850 tx=0x7f67240090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71401f2c8c7e84c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 1 -- 172.21.15.17:0/213666344 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724032440 con 0x7f672c09c050 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: dump: 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.609 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 1 -- 172.21.15.17:0/213666344 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67240325a0 con 0x7f672c09c050 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 1 -- 172.21.15.17:0/213666344 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724032940 con 0x7f672c09c050 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 5 monclient: authenticate success, global_id 4420 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 4 set_mon_vals no callback set 2024-09-15T01:52:22.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f672bfff640 10 monclient: dump: 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 10 monclient: shutdown 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 -- 172.21.15.17:0/213666344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c050 msgr2=0x7f672c09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 --2- 172.21.15.17:0/213666344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c050 0x7f672c09a990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6724009850 tx=0x7f67240090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 -- 172.21.15.17:0/213666344 shutdown_connections 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 --2- 172.21.15.17:0/213666344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c050 0x7f672c09a990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 -- 172.21.15.17:0/213666344 >> 172.21.15.17:0/213666344 conn(0x7f672c071300 msgr2=0x7f672c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 -- 172.21.15.17:0/213666344 shutdown_connections 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.603+0000 7f6733660640 1 -- 172.21.15.17:0/213666344 wait complete. 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.604+0000 7f6733660640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.604+0000 7f6733660640 10 monclient: monmap: 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:22.612 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.604+0000 7f6733660640 1 Processor -- start 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.604+0000 7f6733660640 1 -- start start 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.604+0000 7f6733660640 10 monclient: init 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): picked mon.noname-c con 0x7f672c09c050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c09c420 0x7f672c13d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): picked mon.noname-a con 0x7f672c09c420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 0x7f672c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): picked mon.noname-b con 0x7f672c09c830 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c120860 con 0x7f672c09c420 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c122550 con 0x7f672c09c830 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c13e610 con 0x7f672c09c050 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6733660640 10 monclient(hunting): authenticate will time out at 802.601217s 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c09c420 0x7f672c13d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41580/0 (socket says 172.21.15.17:41580) 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 learned_addr learned my addr 172.21.15.17:0/4031869664 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 10 monclient(hunting): get_auth_request con 0x7f672c09c050 auth_method 0 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 0x7f672c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): get_auth_request con 0x7f672c09c420 auth_method 0 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): get_auth_request con 0x7f672c09c830 auth_method 0 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 msgr2=0x7f672c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 msgr2=0x7f672c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 msgr2=0x7f672c13db20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c09c050 0x7f672c13db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c09c420 msgr2=0x7f672c13d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c09c420 0x7f672c13d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 monclient: found mon.noname-b 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724009390 con 0x7f672c09c830 2024-09-15T01:52:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.605+0000 7f6730bfb640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c09c420 0x7f672c13d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.607+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 msgr2=0x7f672c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.607+0000 7f6731bfd640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 0x7f672c141090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c4c0 tx=0x7f671c007ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c005cb0 con 0x7f672c09c830 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: dump: 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c005e10 con 0x7f672c09c830 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 5 monclient: authenticate success, global_id 4423 2024-09-15T01:52:22.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 monclient: _renew_subs 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c122270 con 0x7f672c09c830 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c007320 con 0x7f672c09c830 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f67297fa640 10 monclient: dump: 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.625 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 monclient: _renew_subs 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c13d8b0 con 0x7f672c09c830 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f672b7fe640 4 set_mon_vals no callback set 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 0x7f672c1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.608+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c063bb0 con 0x7f672c1213a0 2024-09-15T01:52:22.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 0x7f672c1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient: get_auth_request con 0x7f672c1213a0 auth_method 0 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f671c01b3b0 con 0x7f672c09c830 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f672c122b60 con 0x7f672c1213a0 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67297fa640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038740 0x7f671803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f671c04ad10 con 0x7f672c09c830 2024-09-15T01:52:22.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f6730bfb640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038740 0x7f671803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.609+0000 7f6730bfb640 10 monclient: get_auth_request con 0x7f6718038740 auth_method 0 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 msgr2=0x7f672c1415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 0x7f672c1415d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6724009130 tx=0x7f6724032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724031840 con 0x7f672c1213a0 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.611+0000 7f67297fa640 10 monclient: dump: 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.628 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f67240319a0 con 0x7f672c1213a0 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 msgr2=0x7f672c1415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 0x7f672c1415d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6724009130 tx=0x7f6724032000 comp rx=0 tx=0).stop 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 msgr2=0x7f672c1415d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f67297fa640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c1213a0 0x7f672c1415d0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6724009130 tx=0x7f6724032000 comp rx=0 tx=0).stop 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.612+0000 7f6730bfb640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038740 0x7f671803abf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6720003260 tx=0x7f6720002630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.617+0000 7f6733660640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.617+0000 7f6733660640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 0x7f672c141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.617+0000 7f6733660640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.617+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c122710 con 0x7f672c11fe00 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.617+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 0x7f672c141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient: get_auth_request con 0x7f672c11fe00 auth_method 0 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2024-09-15T01:52:22.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.618+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f672c0724a0 con 0x7f672c11fe00 2024-09-15T01:52:22.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67313fc640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 msgr2=0x7f672c141b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67313fc640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 0x7f672c141b10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f672c063930 tx=0x7f672403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724031e70 con 0x7f672c11fe00 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67297fa640 10 monclient: dump: 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.632 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.620+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f672403d810 con 0x7f672c11fe00 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 msgr2=0x7f672c141b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 0x7f672c141b10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f672c063930 tx=0x7f672403e000 comp rx=0 tx=0).stop 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 msgr2=0x7f672c141b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f67297fa640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c11fe00 0x7f672c141b10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.633 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:22.634 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:22.635 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:22.636 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:22.640 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:22.643 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.644 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:22.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:22.646 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:22.646 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038740 msgr2=0x7f671803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038740 0x7f671803abf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6720003260 tx=0x7f6720002630 comp rx=0 tx=0).stop 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 10 monclient: shutdown 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 msgr2=0x7f672c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 --2- 172.21.15.17:0/4031869664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f672c09c830 0x7f672c141090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c4c0 tx=0x7f671c007ee0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 reap_dead start 2024-09-15T01:52:22.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6731bfd640 1 -- 172.21.15.17:0/4031869664 reap_dead start 2024-09-15T01:52:22.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 shutdown_connections 2024-09-15T01:52:22.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 >> 172.21.15.17:0/4031869664 conn(0x7f672c071300 msgr2=0x7f672c0a1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:22.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 shutdown_connections 2024-09-15T01:52:22.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.621+0000 7f6733660640 1 -- 172.21.15.17:0/4031869664 wait complete. 2024-09-15T01:52:22.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:22.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:52:22.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:52:22.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.743+0000 7fe4df86f640 10 monclient: get_monmap_and_config 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.743+0000 7fe4df86f640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.743+0000 7fe4df86f640 10 monclient: monmap: 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:22.744534+0000 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:22.744534+0000 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:22.746 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:22.747 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:22.747 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 Processor -- start 2024-09-15T01:52:22.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 -- start start 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient: init 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4d809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe4d8069c90 0x7fe4d8139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d8069c90 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80a5b50 con 0x7fe4d809ccf0 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d80a18b0 con 0x7fe4d8069c90 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d809c5f0 con 0x7fe4d80a5780 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.744+0000 7fe4df86f640 10 monclient(hunting): authenticate will time out at 802.740968s 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe4d8069c90 0x7fe4d8139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41588/0 (socket says 172.21.15.17:41588) 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 -- 172.21.15.17:0/214800593 learned_addr learned my addr 172.21.15.17:0/214800593 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): get_auth_request con 0x7fe4d8069c90 auth_method 0 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request con 0x7fe4d80a5780 auth_method 0 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 -- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 -- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 -- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d809bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 -- 172.21.15.17:0/214800593 >> v1:172.21.15.67:6789/0 conn(0x7fe4d809ccf0 legacy=0x7fe4d8069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:22.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 monclient: found mon.noname-a 2024-09-15T01:52:22.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.745+0000 7fe4dde0c640 1 -- 172.21.15.17:0/214800593 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d809d210 con 0x7fe4d8069c90 2024-09-15T01:52:22.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4dde0c640 1 -- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d8069c90 msgr2=0x7fe4d8139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d8069c90 0x7fe4d8139880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e60 tx=0x7fe4d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd07f72e37f1fb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 1 -- 172.21.15.17:0/214800593 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0007760 con 0x7fe4d8069c90 2024-09-15T01:52:22.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: dump: 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 1 -- 172.21.15.17:0/214800593 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d002d050 con 0x7fe4d8069c90 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 5 monclient: authenticate success, global_id 4419 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 1 -- 172.21.15.17:0/214800593 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0007af0 con 0x7fe4d8069c90 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4cffff640 10 monclient: dump: 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 4 set_mon_vals no callback set 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 10 monclient: shutdown 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 -- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d8069c90 msgr2=0x7fe4d8139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d8069c90 0x7fe4d8139880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e60 tx=0x7fe4d0009780 comp rx=0 tx=0).stop 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 -- 172.21.15.17:0/214800593 shutdown_connections 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d8069c90 0x7fe4d8139880 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 --2- 172.21.15.17:0/214800593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.748+0000 7fe4df86f640 1 -- 172.21.15.17:0/214800593 >> 172.21.15.17:0/214800593 conn(0x7fe4d8070e10 msgr2=0x7fe4d80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 1 -- 172.21.15.17:0/214800593 shutdown_connections 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 1 -- 172.21.15.17:0/214800593 wait complete. 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 10 monclient: build_initial_monmap 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 10 monclient: monmap: 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.761 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 1 Processor -- start 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 1 -- start start 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.749+0000 7fe4df86f640 10 monclient: init 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8069c90 0x7fe4d809a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d8069c90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 0x7fe4d8130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d811e130 con 0x7fe4d809ccf0 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d809b5f0 con 0x7fe4d8069c90 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d811db70 con 0x7fe4d80a5780 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4df86f640 10 monclient(hunting): authenticate will time out at 802.746424s 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 0x7fe4d8130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41598/0 (socket says 172.21.15.17:41598) 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 -- 172.21.15.17:0/3801558353 learned_addr learned my addr 172.21.15.17:0/3801558353 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 0x7fe4d8130a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41832/0 (socket says 172.21.15.17:41832) 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 10 monclient(hunting): get_auth_request con 0x7fe4d80a5780 auth_method 0 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): get_auth_request con 0x7fe4d809ccf0 auth_method 0 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8069c90 0x7fe4d809a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request con 0x7fe4d8069c90 auth_method 0 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d811b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d811b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 msgr2=0x7fe4d811b6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:22.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d80a5780 0x7fe4d811b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8069c90 msgr2=0x7fe4d809a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8069c90 0x7fe4d809a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 monclient: found mon.noname-a 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.750+0000 7fe4dce0a640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0009a20 con 0x7fe4d809ccf0 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4dce0a640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 msgr2=0x7fe4d8130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4dce0a640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 0x7fe4d8130a50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800c4e0 tx=0x7fe4c8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80117a0 con 0x7fe4d809ccf0 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: dump: 2024-09-15T01:52:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8011920 con 0x7fe4d809ccf0 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 5 monclient: authenticate success, global_id 4422 2024-09-15T01:52:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 10 monclient: _renew_subs 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d811deb0 con 0x7fe4d809ccf0 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011ce0 con 0x7fe4d809ccf0 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cd7fa640 10 monclient: dump: 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:22.768 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 10 monclient: _renew_subs 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8141900 con 0x7fe4d809ccf0 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.753+0000 7fe4cf7fe640 4 set_mon_vals no callback set 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4df86f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4df86f640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8063b20 con 0x7fe4d811cc10 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 10 monclient: get_auth_request con 0x7fe4d811cc10 auth_method 0 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe4c8012630 con 0x7fe4d809ccf0 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4cd7fa640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe4c804b520 con 0x7fe4d809ccf0 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.754+0000 7fe4dde0c640 10 monclient: get_auth_request con 0x7fe4bc038790 auth_method 0 2024-09-15T01:52:22.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.757+0000 7fe4dde0c640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00097c0 tx=0x7fe4d0037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 10 monclient: get_auth_request con 0x7fe4d811cc10 auth_method 0 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:22.955+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:52:23.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 10 monclient: get_auth_request con 0x7fe4d811cc10 auth_method 0 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:23.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:23.356+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient: get_auth_request con 0x7fe4d811cc10 auth_method 0 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-09-15T01:52:24.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.157+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4d80724a0 con 0x7fe4d811cc10 2024-09-15T01:52:24.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.159+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.159+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006fd0 tx=0x7fe4d4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4015eb0 con 0x7fe4d811cc10 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 10 monclient: dump: 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.162 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.163 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe4d401c070 con 0x7fe4d811cc10 2024-09-15T01:52:24.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006fd0 tx=0x7fe4d4008040 comp rx=0 tx=0).stop 2024-09-15T01:52:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 msgr2=0x7fe4d811bc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.160+0000 7fe4cd7fa640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d811cc10 0x7fe4d811bc10 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006fd0 tx=0x7fe4d4008040 comp rx=0 tx=0).stop 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4df86f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4df86f640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 0x7fe4d811c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8142040 con 0x7fe4d8130f90 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 0x7fe4d811c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient: get_auth_request con 0x7fe4d8130f90 auth_method 0 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-15T01:52:24.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.167+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe4d806f4f0 con 0x7fe4d8130f90 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4dd60b640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 msgr2=0x7fe4d811c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4dd60b640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 0x7fe4d811c150 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4016010 tx=0x7fe4d4023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4018a10 con 0x7fe4d8130f90 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 10 monclient: dump: 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7fe4d400c050 con 0x7fe4d8130f90 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:24.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 msgr2=0x7fe4d811c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 0x7fe4d811c150 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4016010 tx=0x7fe4d4023000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 msgr2=0x7fe4d811c150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.170+0000 7fe4cd7fa640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4d8130f90 0x7fe4d811c150 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4016010 tx=0x7fe4d4023000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:24.176 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:24.177 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:24.178 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.179 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:24.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:24.183 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.185 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:24.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.187 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.187 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:24.187 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:24.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 msgr2=0x7fe4bc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00097c0 tx=0x7fe4d0037000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 10 monclient: shutdown 2024-09-15T01:52:24.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:24.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 msgr2=0x7fe4d8130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 --2- 172.21.15.17:0/3801558353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4d809ccf0 0x7fe4d8130a50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800c4e0 tx=0x7fe4c8007e10 comp rx=0 tx=0).stop 2024-09-15T01:52:24.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4dde0c640 1 -- 172.21.15.17:0/3801558353 reap_dead start 2024-09-15T01:52:24.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4dde0c640 1 -- 172.21.15.17:0/3801558353 reap_dead start 2024-09-15T01:52:24.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 shutdown_connections 2024-09-15T01:52:24.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 >> 172.21.15.17:0/3801558353 conn(0x7fe4d8070e10 msgr2=0x7fe4d809d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:24.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 shutdown_connections 2024-09-15T01:52:24.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.171+0000 7fe4df86f640 1 -- 172.21.15.17:0/3801558353 wait complete. 2024-09-15T01:52:24.190 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:24.191 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:52:24.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:52:24.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.295+0000 7f5e7731d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.295+0000 7f5e7731d640 10 monclient: build_initial_monmap 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.295+0000 7f5e7731d640 10 monclient: monmap: 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:24.296774+0000 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:24.296774+0000 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:24.298 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:24.299 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:24.299 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:24.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.296+0000 7f5e7731d640 1 Processor -- start 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.296+0000 7f5e7731d640 1 -- start start 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.296+0000 7f5e7731d640 10 monclient: init 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5e700a5780 0x7f5e7009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e700a5780 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e7009ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e70069c90 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700a5b50 con 0x7f5e7009ccf0 2024-09-15T01:52:24.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7009c5f0 con 0x7f5e70069c90 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e700a18b0 con 0x7f5e700a5780 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e7731d640 10 monclient(hunting): authenticate will time out at 804.293331s 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5e700a5780 0x7f5e7009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5e700a5780 0x7f5e7009bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47550/0 (socket says 172.21.15.17:47550) 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 1 -- 172.21.15.17:0/2086304580 learned_addr learned my addr 172.21.15.17:0/2086304580 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): get_auth_request con 0x7f5e700a5780 auth_method 0 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e748b8640 1 --1- 172.21.15.17:0/2086304580 >> v1:172.21.15.17:6789/0 conn(0x7f5e7009ccf0 0x7f5e70069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56202/0 (socket says 172.21.15.17:56202) 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.297+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e748b8640 1 -- 172.21.15.17:0/2086304580 >> v1:172.21.15.17:6789/0 conn(0x7f5e7009ccf0 legacy=0x7f5e70069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e758ba640 1 -- 172.21.15.17:0/2086304580 >> v1:172.21.15.67:6789/0 conn(0x7f5e70069c90 legacy=0x7f5e70139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2024-09-15T01:52:24.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 1 -- 172.21.15.17:0/2086304580 >> v1:172.21.15.67:6789/0 conn(0x7f5e70069c90 legacy=0x7f5e70139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 1 -- 172.21.15.17:0/2086304580 >> v1:172.21.15.17:6789/0 conn(0x7f5e7009ccf0 legacy=0x7f5e70069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 10 monclient: _finish_hunting 0 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 1 monclient: found mon.noname-a 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:24.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.299+0000 7f5e750b9640 1 -- 172.21.15.17:0/2086304580 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7009d210 con 0x7f5e700a5780 2024-09-15T01:52:24.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.302+0000 7f5e750b9640 1 -- 172.21.15.17:0/2086304580 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 msgr2=0x7f5e7009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.302+0000 7f5e750b9640 1 --2- 172.21.15.17:0/2086304580 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e7009bc90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009a10 tx=0x7f5e600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c895809ca1e02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 1 -- 172.21.15.17:0/2086304580 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002f4b0 con 0x7f5e700a5780 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: dump: 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.306 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 1 -- 172.21.15.17:0/2086304580 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6002f610 con 0x7f5e700a5780 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 5 monclient: authenticate success, global_id 4425 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 1 -- 172.21.15.17:0/2086304580 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002f9f0 con 0x7f5e700a5780 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 4 set_mon_vals no callback set 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e67fff640 10 monclient: dump: 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 10 monclient: shutdown 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 -- 172.21.15.17:0/2086304580 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 msgr2=0x7f5e7009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 --2- 172.21.15.17:0/2086304580 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e7009bc90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009a10 tx=0x7f5e600091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 -- 172.21.15.17:0/2086304580 shutdown_connections 2024-09-15T01:52:24.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 --2- 172.21.15.17:0/2086304580 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e7009bc90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 -- 172.21.15.17:0/2086304580 >> 172.21.15.17:0/2086304580 conn(0x7f5e70070e10 msgr2=0x7f5e700a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:24.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 -- 172.21.15.17:0/2086304580 shutdown_connections 2024-09-15T01:52:24.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.303+0000 7f5e7731d640 1 -- 172.21.15.17:0/2086304580 wait complete. 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.304+0000 7f5e7731d640 10 monclient: build_initial_monmap 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.304+0000 7f5e7731d640 10 monclient: monmap: 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:24.314 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.304+0000 7f5e7731d640 1 Processor -- start 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.304+0000 7f5e7731d640 1 -- start start 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.304+0000 7f5e7731d640 10 monclient: init 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70069c90 0x7f5e7013dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e70069c90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 0x7f5e7013d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7009ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e700a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70122f90 con 0x7f5e700a5780 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e701229d0 con 0x7f5e70069c90 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70120460 con 0x7f5e7009ccf0 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e7731d640 10 monclient(hunting): authenticate will time out at 804.301329s 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 0x7f5e7013d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 0x7f5e7013d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59416/0 (socket says 172.21.15.17:59416) 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 1 -- 172.21.15.17:0/3890854376 learned_addr learned my addr 172.21.15.17:0/3890854376 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): get_auth_request con 0x7f5e7009ccf0 auth_method 0 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70069c90 0x7f5e7013dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): get_auth_request con 0x7f5e700a5780 auth_method 0 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): get_auth_request con 0x7f5e70069c90 auth_method 0 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 msgr2=0x7f5e7013d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 0x7f5e7013d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70069c90 msgr2=0x7f5e7013dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70069c90 0x7f5e7013dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient: _finish_hunting 0 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 monclient: found mon.noname-a 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e758ba640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60009450 con 0x7f5e700a5780 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.305+0000 7f5e748b8640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e7009ccf0 0x7f5e7013d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e758ba640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 msgr2=0x7f5e70141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e758ba640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c710 tx=0x7f5e68001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680070f0 con 0x7f5e700a5780 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: dump: 2024-09-15T01:52:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.320 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68007250 con 0x7f5e700a5780 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680075f0 con 0x7f5e700a5780 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e657fa640 10 monclient: dump: 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 5 monclient: authenticate success, global_id 4428 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 10 monclient: _renew_subs 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e701226f0 con 0x7f5e700a5780 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e677fe640 4 set_mon_vals no callback set 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 10 monclient: _renew_subs 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.308+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7013dd30 con 0x7f5e700a5780 2024-09-15T01:52:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e7731d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e7731d640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 0x7f5e70141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70063b20 con 0x7f5e70121820 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e758ba640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c710 tx=0x7f5e68001730 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f5e6801b400 mgrmap(e 5) v1 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5e6801b400 con 0x7f5e700a5780 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 0x7f5e70141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient: get_auth_request con 0x7f5e70121820 auth_method 0 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e657fa640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e54038790 0x7f5e5403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 10 monclient(hunting): handle_auth_done global_id 4432 payload 824 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e750b9640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e70123100 con 0x7f5e70121820 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e748b8640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e54038790 0x7f5e5403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.309+0000 7f5e748b8640 10 monclient: get_auth_request con 0x7f5e54038790 auth_method 0 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.311+0000 7f5e750b9640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 msgr2=0x7f5e70141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.311+0000 7f5e750b9640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 0x7f5e70141a50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5e70068e70 tx=0x7f5e60032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e748b8640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e54038790 0x7f5e5403ac40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e700a1ad0 tx=0x7f5e6c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6003c940 con 0x7f5e70121820 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 10 monclient: dump: 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.325 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5e60040070 con 0x7f5e70121820 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 msgr2=0x7f5e70141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 0x7f5e70141a50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5e70068e70 tx=0x7f5e60032000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 msgr2=0x7f5e70141a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.312+0000 7f5e657fa640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e70121820 0x7f5e70141a50 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5e70068e70 tx=0x7f5e60032000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e7731d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e7731d640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 0x7f5e70141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e7731d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70122e30 con 0x7f5e7013e900 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 0x7f5e70141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient: get_auth_request con 0x7f5e7013e900 auth_method 0 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.318+0000 7f5e750b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.319+0000 7f5e750b9640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.319+0000 7f5e750b9640 1 -- 172.21.15.17:0/3890854376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5e700724a0 con 0x7f5e7013e900 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e750b9640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 msgr2=0x7f5e70141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e750b9640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 0x7f5e70141f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006d90 tx=0x7f5e6003e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60032650 con 0x7f5e7013e900 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 10 monclient: dump: 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.329 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f5e6004f070 con 0x7f5e7013e900 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 msgr2=0x7f5e70141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 0x7f5e70141f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006d90 tx=0x7f5e6003e070 comp rx=0 tx=0).stop 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 msgr2=0x7f5e70141f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.321+0000 7f5e657fa640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7013e900 0x7f5e70141f90 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006d90 tx=0x7f5e6003e070 comp rx=0 tx=0).stop 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:52:24.331 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.332 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:24.334 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:24.335 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:24.338 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:24.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:24.341 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.342 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:24.343 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e54038790 msgr2=0x7f5e5403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e54038790 0x7f5e5403ac40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e700a1ad0 tx=0x7f5e6c007de0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 10 monclient: shutdown 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 msgr2=0x7f5e70141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e758ba640 1 -- 172.21.15.17:0/3890854376 reap_dead start 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c710 tx=0x7f5e68001730 comp rx=0 tx=0).stop 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e657fa640 1 -- 172.21.15.17:0/3890854376 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7f5e6804b490 con 0x7f5e700a5780 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 shutdown_connections 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 --2- 172.21.15.17:0/3890854376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e700a5780 0x7f5e70141510 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 >> 172.21.15.17:0/3890854376 conn(0x7f5e70070e10 msgr2=0x7f5e70098ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:24.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 shutdown_connections 2024-09-15T01:52:24.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.322+0000 7f5e7731d640 1 -- 172.21.15.17:0/3890854376 wait complete. 2024-09-15T01:52:24.346 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:24.346 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:52:24.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:52:24.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.441+0000 7fe251a20640 10 monclient: get_monmap_and_config 2024-09-15T01:52:24.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.441+0000 7fe251a20640 10 monclient: build_initial_monmap 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.441+0000 7fe251a20640 10 monclient: monmap: 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:24.442362+0000 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:24.442362+0000 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:24.444 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:24.445 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 Processor -- start 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 -- start start 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient: init 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe24c097320 0x7fe24c0976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-a con 0x7fe24c097320 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe24c09a590 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-b con 0x7fe24c12edb0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c0985d0 con 0x7fe24c09a590 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c09d200 con 0x7fe24c097320 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c097c30 con 0x7fe24c12edb0 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.442+0000 7fe251a20640 10 monclient(hunting): authenticate will time out at 804.438934s 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe24c097320 0x7fe24c0976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe24c097320 0x7fe24c0976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47578/0 (socket says 172.21.15.17:47578) 2024-09-15T01:52:24.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59446/0 (socket says 172.21.15.17:59446) 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 -- 172.21.15.17:0/2445238173 learned_addr learned my addr 172.21.15.17:0/2445238173 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): get_auth_request con 0x7fe24c097320 auth_method 0 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): get_auth_request con 0x7fe24c12edb0 auth_method 0 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient(hunting): handle_auth_done global_id 4431 payload 274 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 -- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 msgr2=0x7fe24c1369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 -- 172.21.15.17:0/2445238173 >> v1:172.21.15.67:6789/0 conn(0x7fe24c09a590 legacy=0x7fe24c12e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 monclient: found mon.noname-a 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24affd640 1 -- 172.21.15.17:0/2445238173 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c0992f0 con 0x7fe24c097320 2024-09-15T01:52:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.443+0000 7fe24b7fe640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:24.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24affd640 1 -- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 msgr2=0x7fe24c0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24affd640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c0976f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009910 tx=0x7fe23c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d953cc3eb6f44c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 1 -- 172.21.15.17:0/2445238173 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c02aea0 con 0x7fe24c097320 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: dump: 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe251a20640 5 monclient: authenticate success, global_id 4431 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe251a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 1 -- 172.21.15.17:0/2445238173 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c032440 con 0x7fe24c097320 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe251a20640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 1 -- 172.21.15.17:0/2445238173 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c032820 con 0x7fe24c097320 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe251a20640 4 set_mon_vals no callback set 2024-09-15T01:52:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.446+0000 7fe24a7fc640 10 monclient: dump: 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.458 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 10 monclient: shutdown 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 -- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 msgr2=0x7fe24c0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c0976f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009910 tx=0x7fe23c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 -- 172.21.15.17:0/2445238173 shutdown_connections 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c12edb0 0x7fe24c1369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 --2- 172.21.15.17:0/2445238173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c0976f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 -- 172.21.15.17:0/2445238173 >> 172.21.15.17:0/2445238173 conn(0x7fe24c0a1000 msgr2=0x7fe24c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 -- 172.21.15.17:0/2445238173 shutdown_connections 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 1 -- 172.21.15.17:0/2445238173 wait complete. 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 10 monclient: build_initial_monmap 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.447+0000 7fe251a20640 10 monclient: monmap: 2024-09-15T01:52:24.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 Processor -- start 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 -- start start 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient: init 2024-09-15T01:52:24.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c09b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-a con 0x7fe24c097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 0x7fe24c11d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-c con 0x7fe24c09a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe24c12edb0 0x7fe24c11d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): picked mon.noname-b con 0x7fe24c12edb0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c09ad50 con 0x7fe24c097320 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c12e630 con 0x7fe24c12edb0 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c11f920 con 0x7fe24c09a590 2024-09-15T01:52:24.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe251a20640 10 monclient(hunting): authenticate will time out at 804.444801s 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c09b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 0x7fe24c11d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c09b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47592/0 (socket says 172.21.15.17:47592) 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 1 -- 172.21.15.17:0/2714466883 learned_addr learned my addr 172.21.15.17:0/2714466883 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 10 monclient(hunting): get_auth_request con 0x7fe24c097320 auth_method 0 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): get_auth_request con 0x7fe24c09a590 auth_method 0 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24b7fe640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe24c12edb0 0x7fe24c11d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24b7fe640 10 monclient(hunting): get_auth_request con 0x7fe24c12edb0 auth_method 0 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.448+0000 7fe243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:24.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe24b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 10 monclient(hunting): handle_auth_done global_id 34112 payload 824 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 -- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe24c12edb0 msgr2=0x7fe24c11d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe24c12edb0 0x7fe24c11d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 -- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 msgr2=0x7fe24c09b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c09b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 monclient: found mon.noname-c 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.449+0000 7fe243fff640 1 -- 172.21.15.17:0/2714466883 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244009a20 con 0x7fe24c09a590 2024-09-15T01:52:24.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe243fff640 1 -- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 msgr2=0x7fe24c11d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe243fff640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 0x7fe24c11d480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c510 tx=0x7fe234007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340117a0 con 0x7fe24c09a590 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: dump: 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.465 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 5 monclient: authenticate success, global_id 34112 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234011920 con 0x7fe24c09a590 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 10 monclient: _renew_subs 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234011ce0 con 0x7fe24c09a590 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c009450 con 0x7fe24c09a590 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:24.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe249ffb640 4 set_mon_vals no callback set 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe242ffd640 10 monclient: dump: 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:24.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 10 monclient: _renew_subs 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.451+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c11ea30 con 0x7fe24c09a590 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe234012630 con 0x7fe24c09a590 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe242ffd640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 0x7fe22c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe23404b490 con 0x7fe24c09a590 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe251a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe251a20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe251a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c11fe40 con 0x7fe24c09a590 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.452+0000 7fe24affd640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 0x7fe22c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:24.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.453+0000 7fe24affd640 10 monclient: get_auth_request con 0x7fe22c038740 auth_method 0 2024-09-15T01:52:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.455+0000 7fe24affd640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 0x7fe22c03abf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe23c006d90 tx=0x7fe23c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.457+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe23401c460 con 0x7fe24c09a590 2024-09-15T01:52:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.457+0000 7fe242ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.457+0000 7fe242ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:24.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.688+0000 7fe251a20640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:52:24.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.688+0000 7fe251a20640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:52:24.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.688+0000 7fe251a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:24.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.688+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe24c0724a0 con 0x7fe24c09a590 2024-09-15T01:52:24.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fe242ffd640 1 -- 172.21.15.17:0/2714466883 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fe2340102e0 con 0x7fe24c09a590 2024-09-15T01:52:24.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fe242ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:52:24.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fe242ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2166120901,config=3140514009,health=861809967,kv=1529054990,logm=4085647665}) 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2166120901,config=3140514009,health=861809967,kv=1529054990,logm=2169443489}) 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2166120901,config=3140514009,health=861809967,kv=1529054990,logm=4085647665}) 2024-09-15T01:52:24.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.690+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=1,health=12,kv=7,logm=71} crc {auth=4032955431,config=1228700967,health=861809967,kv=769410566,logm=679847440}) 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133120325}) 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858869782}) 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133120325}) 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.691+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813502627}) 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.694 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500572696}) 2024-09-15T01:52:24.695 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209727112}) 2024-09-15T01:52:24.695 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 msgr2=0x7fe22c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 0x7fe22c03abf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe23c006d90 tx=0x7fe23c032000 comp rx=0 tx=0).stop 2024-09-15T01:52:24.695 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500572696}) 2024-09-15T01:52:24.695 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393350862}) 2024-09-15T01:52:24.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 10 monclient: shutdown 2024-09-15T01:52:24.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:24.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 msgr2=0x7fe24c11d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 0x7fe24c11d480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c510 tx=0x7fe234007e10 comp rx=0 tx=0).stop 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 shutdown_connections 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe22c038740 0x7fe22c03abf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe24c12edb0 0x7fe24c11d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe24c09a590 0x7fe24c11d480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 --2- 172.21.15.17:0/2714466883 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe24c097320 0x7fe24c09b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 >> 172.21.15.17:0/2714466883 conn(0x7fe24c0a1000 msgr2=0x7fe24c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 shutdown_connections 2024-09-15T01:52:24.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:24.692+0000 7fe251a20640 1 -- 172.21.15.17:0/2714466883 wait complete. 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=51991159,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2909244028,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2927750135,osd_snap=1063953654,osdmap=693176761}) 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=25} crc {logm=548184581,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2909244028,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2927750135,osd_snap=1063953654,osdmap=3036438625}) 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=51991159,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2909244028,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2927750135,osd_snap=1063953654,osdmap=693176761}) 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.693+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=1036166805,mdsmap=1531151339,mgr=2936826139,mgr_command_descs=3545737492,mgrstat=413285755,mon_config_key=1383706935,monmap=3178194974,osd_metadata=3003250900,osd_pg_creating=2927750135,osd_snap=1063953654,osdmap=2452241027}) 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.694+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.694+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=3114453383}) 2024-09-15T01:52:24.698 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:52:24.694+0000 7fd7feff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=2956545456}) 2024-09-15T01:52:24.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:52:25.706 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:52:25.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.813+0000 7fb23d89f640 10 monclient: get_monmap_and_config 2024-09-15T01:52:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.813+0000 7fb23d89f640 10 monclient: build_initial_monmap 2024-09-15T01:52:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.813+0000 7fb23d89f640 10 monclient: monmap: 2024-09-15T01:52:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:25.814293+0000 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:25.814293+0000 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:25.816 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 Processor -- start 2024-09-15T01:52:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 -- start start 2024-09-15T01:52:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient: init 2024-09-15T01:52:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:25.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb2380976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-b con 0x7fb238097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb23809a590 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb23809a960 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23812f4c0 con 0x7fb23809a590 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380985d0 con 0x7fb23809a960 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23809d200 con 0x7fb238097320 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb23d89f640 10 monclient(hunting): authenticate will time out at 805.810680s 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb2367fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb23809a590 0x7fb23812e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56214/0 (socket says 172.21.15.17:56214) 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb2380976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:25.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb2367fc640 1 -- 172.21.15.17:0/1468320929 learned_addr learned my addr 172.21.15.17:0/1468320929 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): get_auth_request con 0x7fb238097320 auth_method 0 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.814+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 10 monclient(hunting): handle_auth_done global_id 34115 payload 274 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 1 -- 172.21.15.17:0/1468320929 >> v1:172.21.15.67:6789/0 conn(0x7fb23809a960 legacy=0x7fb23812edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 1 -- 172.21.15.17:0/1468320929 >> v1:172.21.15.17:6789/0 conn(0x7fb23809a590 legacy=0x7fb23812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 1 monclient: found mon.noname-b 2024-09-15T01:52:25.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:25.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:25.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.815+0000 7fb236ffd640 1 -- 172.21.15.17:0/1468320929 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb238097c30 con 0x7fb238097320 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.817+0000 7fb236ffd640 1 -- 172.21.15.17:0/1468320929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 msgr2=0x7fb2380976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.817+0000 7fb236ffd640 1 --2- 172.21.15.17:0/1468320929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb2380976f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb2280090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994d11c94063a98a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 1 -- 172.21.15.17:0/1468320929 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228032440 con 0x7fb238097320 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: dump: 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: _finish_auth 0 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 5 monclient: authenticate success, global_id 34115 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 1 -- 172.21.15.17:0/1468320929 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2280325a0 con 0x7fb238097320 2024-09-15T01:52:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 4 set_mon_vals no callback set 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 1 -- 172.21.15.17:0/1468320929 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228032940 con 0x7fb238097320 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb235ffb640 10 monclient: dump: 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.826 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 10 monclient: shutdown 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 -- 172.21.15.17:0/1468320929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 msgr2=0x7fb2380976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 --2- 172.21.15.17:0/1468320929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb2380976f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb2280090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 -- 172.21.15.17:0/1468320929 shutdown_connections 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 --2- 172.21.15.17:0/1468320929 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb2380976f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 -- 172.21.15.17:0/1468320929 >> 172.21.15.17:0/1468320929 conn(0x7fb2380a1000 msgr2=0x7fb2380a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 -- 172.21.15.17:0/1468320929 shutdown_connections 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.818+0000 7fb23d89f640 1 -- 172.21.15.17:0/1468320929 wait complete. 2024-09-15T01:52:25.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.819+0000 7fb23d89f640 10 monclient: build_initial_monmap 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.819+0000 7fb23d89f640 10 monclient: monmap: 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.819+0000 7fb23d89f640 1 Processor -- start 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.819+0000 7fb23d89f640 1 -- start start 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.819+0000 7fb23d89f640 10 monclient: init 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-c con 0x7fb238097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23809a590 0x7fb238133330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-b con 0x7fb23809a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): picked mon.noname-a con 0x7fb23809a960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb238136c50 con 0x7fb23809a960 2024-09-15T01:52:25.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2381365d0 con 0x7fb23809a590 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23809ad50 con 0x7fb238097320 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb23d89f640 10 monclient(hunting): authenticate will time out at 805.816403s 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59490/0 (socket says 172.21.15.17:59490) 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47594/0 (socket says 172.21.15.17:47594) 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 -- 172.21.15.17:0/186191741 learned_addr learned my addr 172.21.15.17:0/186191741 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): get_auth_request con 0x7fb238097320 auth_method 0 2024-09-15T01:52:25.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): get_auth_request con 0x7fb23809a960 auth_method 0 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23809a590 0x7fb238133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): get_auth_request con 0x7fb23809a590 auth_method 0 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient(hunting): handle_auth_done global_id 34118 payload 824 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 -- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23809a590 msgr2=0x7fb238133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23809a590 0x7fb238133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 -- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 msgr2=0x7fb238133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:25.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:25.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 monclient: found mon.noname-c 2024-09-15T01:52:25.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:25.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:25.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb236ffd640 1 -- 172.21.15.17:0/186191741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228009390 con 0x7fb238097320 2024-09-15T01:52:25.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.820+0000 7fb2377fe640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb236ffd640 1 -- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 msgr2=0x7fb23812ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb236ffd640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb228032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228031990 con 0x7fb238097320 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: dump: 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: _finish_auth 0 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb228031af0 con 0x7fb238097320 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 5 monclient: authenticate success, global_id 34118 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 10 monclient: _renew_subs 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23806ed60 con 0x7fb238097320 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:25.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22803b400 con 0x7fb238097320 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb22effd640 10 monclient: dump: 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:25.837 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 10 monclient: _renew_subs 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb2357fa640 4 set_mon_vals no callback set 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.823+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238134d40 con 0x7fb238097320 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb228043030 con 0x7fb238097320 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb23d89f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb23d89f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb23d89f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:25.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2380724a0 con 0x7fb238097320 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb22effd640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 0x7fb21803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fb228041070 con 0x7fb238097320 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.824+0000 7fb2367fc640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 0x7fb21803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.825+0000 7fb2367fc640 10 monclient: get_auth_request con 0x7fb218038740 auth_method 0 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.827+0000 7fb2367fc640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 0x7fb21803abf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb23806a730 tx=0x7fb220007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.829+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb228030940 con 0x7fb238097320 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.829+0000 7fb22effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:25.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:25.829+0000 7fb22effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb23d89f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb23d89f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb23d89f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb2380724a0 con 0x7fb238097320 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb22effd640 1 -- 172.21.15.17:0/186191741 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fb228037020 con 0x7fb238097320 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb22effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.058+0000 7fb22effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:26.061 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 msgr2=0x7fb21803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 0x7fb21803abf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb23806a730 tx=0x7fb220007f00 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 10 monclient: shutdown 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 msgr2=0x7fb23812ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb228032000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 shutdown_connections 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb218038740 0x7fb21803abf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb23809a960 0x7fb238133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23809a590 0x7fb238133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 --2- 172.21.15.17:0/186191741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb238097320 0x7fb23812ea90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 >> 172.21.15.17:0/186191741 conn(0x7fb2380a1000 msgr2=0x7fb23806e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 shutdown_connections 2024-09-15T01:52:26.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.060+0000 7fb23d89f640 1 -- 172.21.15.17:0/186191741 wait complete. 2024-09-15T01:52:26.073 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:52:26.073 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:52:26.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:52:26.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 10 monclient: monmap: 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.170 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:26.169036+0000 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:26.169036+0000 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:26.171 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 1 Processor -- start 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 1 -- start start 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.168+0000 7fe92dd9d640 10 monclient: init 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe928131b20 0x7fe92809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe928131b20 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9280a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe92809f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:26.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9280a0040 con 0x7fe92809f1e0 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92809bff0 con 0x7fe9280a0280 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9280a14a0 con 0x7fe928131b20 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe92dd9d640 10 monclient(hunting): authenticate will time out at 806.165474s 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe928131b20 0x7fe92809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe928131b20 0x7fe92809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47598/0 (socket says 172.21.15.17:47598) 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 1 -- 172.21.15.17:0/4132342490 learned_addr learned my addr 172.21.15.17:0/4132342490 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): get_auth_request con 0x7fe928131b20 auth_method 0 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.169+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.171+0000 7fe926ffd640 1 -- 172.21.15.17:0/4132342490 >> v1:172.21.15.67:6789/0 conn(0x7fe9280a0280 legacy=0x7fe928131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe927fff640 1 -- 172.21.15.17:0/4132342490 >> v1:172.21.15.17:6789/0 conn(0x7fe92809f1e0 legacy=0x7fe928138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 10 monclient(hunting): handle_auth_done global_id 4440 payload 274 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 1 -- 172.21.15.17:0/4132342490 >> v1:172.21.15.67:6789/0 conn(0x7fe9280a0280 legacy=0x7fe928131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 1 -- 172.21.15.17:0/4132342490 >> v1:172.21.15.17:6789/0 conn(0x7fe92809f1e0 legacy=0x7fe928138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 1 monclient: found mon.noname-a 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.172+0000 7fe9277fe640 1 -- 172.21.15.17:0/4132342490 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928132600 con 0x7fe928131b20 2024-09-15T01:52:26.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.174+0000 7fe9277fe640 1 -- 172.21.15.17:0/4132342490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe928131b20 msgr2=0x7fe92809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.174+0000 7fe9277fe640 1 --2- 172.21.15.17:0/4132342490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe928131b20 0x7fe92809eca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b0d0 tx=0x7fe918009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30d655d9cc0ea61 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 1 -- 172.21.15.17:0/4132342490 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918007610 con 0x7fe928131b20 2024-09-15T01:52:26.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: dump: 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 1 -- 172.21.15.17:0/4132342490 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe918007770 con 0x7fe928131b20 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 5 monclient: authenticate success, global_id 4440 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 1 -- 172.21.15.17:0/4132342490 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918007b10 con 0x7fe928131b20 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 4 set_mon_vals no callback set 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe9267fc640 10 monclient: dump: 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 10 monclient: shutdown 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 -- 172.21.15.17:0/4132342490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe928131b20 msgr2=0x7fe92809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/4132342490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe928131b20 0x7fe92809eca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b0d0 tx=0x7fe918009840 comp rx=0 tx=0).stop 2024-09-15T01:52:26.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 -- 172.21.15.17:0/4132342490 shutdown_connections 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/4132342490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe928131b20 0x7fe92809eca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 -- 172.21.15.17:0/4132342490 >> 172.21.15.17:0/4132342490 conn(0x7fe928071270 msgr2=0x7fe9280a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 -- 172.21.15.17:0/4132342490 shutdown_connections 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.175+0000 7fe92dd9d640 1 -- 172.21.15.17:0/4132342490 wait complete. 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.176+0000 7fe92dd9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.176+0000 7fe92dd9d640 10 monclient: monmap: 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:26.182 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:26.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.176+0000 7fe92dd9d640 1 Processor -- start 2024-09-15T01:52:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.176+0000 7fe92dd9d640 1 -- start start 2024-09-15T01:52:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.176+0000 7fe92dd9d640 10 monclient: init 2024-09-15T01:52:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe92809f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9280a0280 0x7fe92809a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9280a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe928131b20 0x7fe92809abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe928131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280a09b0 con 0x7fe92809f1e0 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92812c830 con 0x7fe9280a0280 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92812cb10 con 0x7fe928131b20 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe92dd9d640 10 monclient(hunting): authenticate will time out at 806.173209s 2024-09-15T01:52:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47600/0 (socket says 172.21.15.17:47600) 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 -- 172.21.15.17:0/3388267282 learned_addr learned my addr 172.21.15.17:0/3388267282 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): get_auth_request con 0x7fe92809f1e0 auth_method 0 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe928131b20 0x7fe92809abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): get_auth_request con 0x7fe928131b20 auth_method 0 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9280a0280 0x7fe92809a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): get_auth_request con 0x7fe9280a0280 auth_method 0 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe928131b20 msgr2=0x7fe92809abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe928131b20 0x7fe92809abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9280a0280 msgr2=0x7fe92809a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9280a0280 0x7fe92809a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 monclient: found mon.noname-a 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe9277fe640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918009ae0 con 0x7fe92809f1e0 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.177+0000 7fe927fff640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe928131b20 0x7fe92809abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.179+0000 7fe9277fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 msgr2=0x7fe92809a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.179+0000 7fe9277fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe918009880 tx=0x7fe918037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918033ac0 con 0x7fe92809f1e0 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: dump: 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.193 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe918033c40 con 0x7fe92809f1e0 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 5 monclient: authenticate success, global_id 4443 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 monclient: _renew_subs 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92812cdf0 con 0x7fe92809f1e0 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91803c410 con 0x7fe92809f1e0 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 10 monclient: dump: 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 monclient: _renew_subs 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92812d9c0 con 0x7fe92809f1e0 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe925ffb640 4 set_mon_vals no callback set 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 0x7fe928096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928063a40 con 0x7fe92809b120 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.180+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe918003020 con 0x7fe92809f1e0 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 0x7fe928096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient: get_auth_request con 0x7fe92809b120 auth_method 0 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe90f7fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9140387c0 0x7fe91403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe91806c430 con 0x7fe92809f1e0 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe926ffd640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9280724a0 con 0x7fe92809b120 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe927fff640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9140387c0 0x7fe91403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.181+0000 7fe927fff640 10 monclient: get_auth_request con 0x7fe9140387c0 auth_method 0 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe926ffd640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 msgr2=0x7fe928096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe926ffd640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 0x7fe928096a10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe92809b5f0 tx=0x7fe91c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0074f0 con 0x7fe92809b120 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 10 monclient: dump: 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe91c005780 con 0x7fe92809b120 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 msgr2=0x7fe928096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 0x7fe928096a10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe92809b5f0 tx=0x7fe91c009040 comp rx=0 tx=0).stop 2024-09-15T01:52:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 msgr2=0x7fe928096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe90f7fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809b120 0x7fe928096a10 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe92809b5f0 tx=0x7fe91c009040 comp rx=0 tx=0).stop 2024-09-15T01:52:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.183+0000 7fe927fff640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9140387c0 0x7fe91403ac70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe920001eb0 tx=0x7fe920009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe92dd9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 0x7fe928096f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe92dd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92809d660 con 0x7fe92812b940 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 0x7fe928096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient: get_auth_request con 0x7fe92812b940 auth_method 0 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.189+0000 7fe926ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.190+0000 7fe926ffd640 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.190+0000 7fe926ffd640 1 -- 172.21.15.17:0/3388267282 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe92806efc0 con 0x7fe92812b940 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe926ffd640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 msgr2=0x7fe928096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe926ffd640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 0x7fe928096f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe91c017000 tx=0x7fe91c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c021620 con 0x7fe92812b940 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 10 monclient: dump: 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7fe91c021780 con 0x7fe92812b940 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 msgr2=0x7fe928096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 0x7fe928096f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe91c017000 tx=0x7fe91c018000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 msgr2=0x7fe928096f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe90f7fe640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92812b940 0x7fe928096f50 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe91c017000 tx=0x7fe91c018000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:26.204 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.205 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:26.207 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.208 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.210 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:26.212 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.214 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:26.216 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9140387c0 msgr2=0x7fe91403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9140387c0 0x7fe91403ac70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe920001eb0 tx=0x7fe920009040 comp rx=0 tx=0).stop 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 10 monclient: shutdown 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 msgr2=0x7fe92809a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe918009880 tx=0x7fe918037000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.192+0000 7fe927fff640 1 -- 172.21.15.17:0/3388267282 reap_dead start 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.193+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 shutdown_connections 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.193+0000 7fe92dd9d640 1 --2- 172.21.15.17:0/3388267282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe92809f1e0 0x7fe92809a160 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.193+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 >> 172.21.15.17:0/3388267282 conn(0x7fe928071270 msgr2=0x7fe9280a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.193+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 shutdown_connections 2024-09-15T01:52:26.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.193+0000 7fe92dd9d640 1 -- 172.21.15.17:0/3388267282 wait complete. 2024-09-15T01:52:26.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:26.219 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:52:26.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:52:26.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 10 monclient: get_monmap_and_config 2024-09-15T01:52:26.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 10 monclient: monmap: 2024-09-15T01:52:26.316 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:26.314936+0000 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:26.314936+0000 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 1 Processor -- start 2024-09-15T01:52:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 1 -- start start 2024-09-15T01:52:26.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.314+0000 7fce7c278640 10 monclient: init 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 0x7fce7409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-b con 0x7fce74131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fce740a0280 0x7fce74131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-a con 0x7fce740a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce7409f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce740a0040 con 0x7fce7409f1e0 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7409bff0 con 0x7fce740a0280 2024-09-15T01:52:26.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce740a14a0 con 0x7fce74131b20 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7c278640 10 monclient(hunting): authenticate will time out at 806.311405s 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 0x7fce7409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fce740a0280 0x7fce74131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fce740a0280 0x7fce74131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47622/0 (socket says 172.21.15.17:47622) 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 0x7fce7409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59514/0 (socket says 172.21.15.17:59514) 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 -- 172.21.15.17:0/3519084772 learned_addr learned my addr 172.21.15.17:0/3519084772 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): get_auth_request con 0x7fce740a0280 auth_method 0 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): get_auth_request con 0x7fce74131b20 auth_method 0 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient(hunting): handle_auth_done global_id 4452 payload 274 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 -- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 msgr2=0x7fce7409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 --2- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 0x7fce7409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 -- 172.21.15.17:0/3519084772 >> v1:172.21.15.67:6789/0 conn(0x7fce7409f1e0 legacy=0x7fce74068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 monclient: found mon.noname-a 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.315+0000 7fce79813640 1 -- 172.21.15.17:0/3519084772 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74132430 con 0x7fce740a0280 2024-09-15T01:52:26.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79813640 1 -- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce740a0280 msgr2=0x7fce74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79813640 1 --2- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce740a0280 0x7fce74131ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce64009910 tx=0x7fce640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f354d0debd6a4932 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 1 -- 172.21.15.17:0/3519084772 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6402aea0 con 0x7fce740a0280 2024-09-15T01:52:26.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: dump: 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 1 -- 172.21.15.17:0/3519084772 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce64032440 con 0x7fce740a0280 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce7c278640 5 monclient: authenticate success, global_id 4452 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce7c278640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce7c278640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 1 -- 172.21.15.17:0/3519084772 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce640327e0 con 0x7fce740a0280 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce7c278640 4 set_mon_vals no callback set 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.318+0000 7fce79012640 10 monclient: dump: 2024-09-15T01:52:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 10 monclient: shutdown 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 -- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce740a0280 msgr2=0x7fce74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 --2- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce740a0280 0x7fce74131ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce64009910 tx=0x7fce640091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 -- 172.21.15.17:0/3519084772 shutdown_connections 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 --2- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce740a0280 0x7fce74131ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 --2- 172.21.15.17:0/3519084772 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74131b20 0x7fce7409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 -- 172.21.15.17:0/3519084772 >> 172.21.15.17:0/3519084772 conn(0x7fce74071270 msgr2=0x7fce740a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 -- 172.21.15.17:0/3519084772 shutdown_connections 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 1 -- 172.21.15.17:0/3519084772 wait complete. 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.319+0000 7fce7c278640 10 monclient: monmap: 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:26.334 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:26.337 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 Processor -- start 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 -- start start 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient: init 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce7409f1e0 0x7fce74141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-a con 0x7fce7409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-c con 0x7fce740a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74131b20 0x7fce74098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): picked mon.noname-b con 0x7fce74131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce740a09b0 con 0x7fce7409f1e0 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74131910 con 0x7fce74131b20 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7411c310 con 0x7fce740a0280 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7c278640 10 monclient(hunting): authenticate will time out at 806.316795s 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a014640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce7409f1e0 0x7fce74141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59526/0 (socket says 172.21.15.17:59526) 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 1 -- 172.21.15.17:0/3531784286 learned_addr learned my addr 172.21.15.17:0/3531784286 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): get_auth_request con 0x7fce740a0280 auth_method 0 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a014640 10 monclient(hunting): get_auth_request con 0x7fce7409f1e0 auth_method 0 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a014640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a014640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce7a815640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74131b20 0x7fce74098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.320+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): get_auth_request con 0x7fce74131b20 auth_method 0 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a815640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 10 monclient(hunting): handle_auth_done global_id 34127 payload 824 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74131b20 msgr2=0x7fce74098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74131b20 0x7fce74098df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce7409f1e0 msgr2=0x7fce74141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce7409f1e0 0x7fce74141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 monclient: found mon.noname-c 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce79813640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce64009450 con 0x7fce740a0280 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.321+0000 7fce7a014640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce7409f1e0 0x7fce74141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce79813640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 msgr2=0x7fce7412ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce79813640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce64035040 tx=0x7fce64032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6402aea0 con 0x7fce740a0280 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: dump: 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce64000d10 con 0x7fce740a0280 2024-09-15T01:52:26.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 5 monclient: authenticate success, global_id 34127 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6403d440 con 0x7fce740a0280 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 10 monclient: _renew_subs 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7409a8f0 con 0x7fce740a0280 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce6a7fc640 10 monclient: dump: 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.344 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce78811640 4 set_mon_vals no callback set 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 10 monclient: _renew_subs 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.323+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7409b4c0 con 0x7fce740a0280 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7c278640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7c278640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 0x7fce7412f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7411c1d0 con 0x7fce741417b0 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fce64039040 con 0x7fce740a0280 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 0x7fce7412f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient: get_auth_request con 0x7fce741417b0 auth_method 0 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.324+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce6a7fc640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fce58038770 0x7fce5803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce7a014640 10 monclient(hunting): handle_auth_done global_id 34130 payload 824 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fce6406b440 con 0x7fce740a0280 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce7a014640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce7411c4a0 con 0x7fce741417b0 2024-09-15T01:52:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce7a815640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fce58038770 0x7fce5803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.325+0000 7fce7a815640 10 monclient: get_auth_request con 0x7fce58038770 auth_method 0 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce7a014640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 msgr2=0x7fce7412f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce7a014640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 0x7fce7412f450 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70014860 con 0x7fce741417b0 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 10 monclient: dump: 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fce700149c0 con 0x7fce741417b0 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 msgr2=0x7fce7412f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 0x7fce7412f450 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce700090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 msgr2=0x7fce7412f450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce6a7fc640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce741417b0 0x7fce7412f450 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce700090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.327+0000 7fce7a815640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fce58038770 0x7fce5803ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce6c007fd0 tx=0x7fce6c0025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7c278640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7c278640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 0x7fce74099730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7c278640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce74063a40 con 0x7fce74099330 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 0x7fce74099730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient: get_auth_request con 0x7fce74099330 auth_method 0 2024-09-15T01:52:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 10 monclient(hunting): handle_auth_done global_id 34133 payload 824 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.333+0000 7fce7a014640 1 -- 172.21.15.17:0/3531784286 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce74099c70 con 0x7fce74099330 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7a014640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 msgr2=0x7fce74099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7a014640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 0x7fce74099730 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce7001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70024040 con 0x7fce74099330 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 10 monclient: dump: 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7fce700133a0 con 0x7fce74099330 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 msgr2=0x7fce74099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 0x7fce74099730 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce7001c000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 msgr2=0x7fce74099730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce6a7fc640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce74099330 0x7fce74099730 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fce70009f30 tx=0x7fce7001c000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:52:26.353 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.354 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.355 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:26.356 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:26.357 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:26.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.360 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.361 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:26.362 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:26.365 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:26.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fce58038770 msgr2=0x7fce5803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fce58038770 0x7fce5803ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce6c007fd0 tx=0x7fce6c0025b0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 10 monclient: shutdown 2024-09-15T01:52:26.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 msgr2=0x7fce7412ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce64035040 tx=0x7fce64032000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7a815640 1 -- 172.21.15.17:0/3531784286 reap_dead start 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 shutdown_connections 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 --2- 172.21.15.17:0/3531784286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fce740a0280 0x7fce7412ef10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.336+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 >> 172.21.15.17:0/3531784286 conn(0x7fce74071270 msgr2=0x7fce740a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.337+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 shutdown_connections 2024-09-15T01:52:26.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.337+0000 7fce7c278640 1 -- 172.21.15.17:0/3531784286 wait complete. 2024-09-15T01:52:26.368 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:26.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:52:26.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 10 monclient: monmap: 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:26.464861+0000 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:26.464861+0000 2024-09-15T01:52:26.466 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:26.467 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 1 Processor -- start 2024-09-15T01:52:26.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 1 -- start start 2024-09-15T01:52:26.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.464+0000 7f4fe339d640 10 monclient: init 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fdc131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 0x7f4fdc131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-b con 0x7f4fdc0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 0x7f4fdc138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-a con 0x7f4fdc132430 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc0a0040 con 0x7f4fdc131b20 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc09bff0 con 0x7f4fdc132430 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0a14a0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe339d640 10 monclient(hunting): authenticate will time out at 806.461337s 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 0x7f4fdc138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 0x7f4fdc131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 0x7f4fdc138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47634/0 (socket says 172.21.15.17:47634) 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 1 -- 172.21.15.17:0/1904633477 learned_addr learned my addr 172.21.15.17:0/1904633477 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): get_auth_request con 0x7f4fdc132430 auth_method 0 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): get_auth_request con 0x7f4fdc0a0280 auth_method 0 2024-09-15T01:52:26.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient(hunting): handle_auth_done global_id 34136 payload 274 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 -- 172.21.15.17:0/1904633477 >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 msgr2=0x7f4fdc138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:26.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 --2- 172.21.15.17:0/1904633477 >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 0x7f4fdc138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 -- 172.21.15.17:0/1904633477 >> v1:172.21.15.17:6789/0 conn(0x7f4fdc131b20 legacy=0x7f4fdc09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 monclient: found mon.noname-b 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.465+0000 7f4fe0938640 1 -- 172.21.15.17:0/1904633477 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc09f3b0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fe0938640 1 -- 172.21.15.17:0/1904633477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 msgr2=0x7f4fdc131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fe0938640 1 --2- 172.21.15.17:0/1904633477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 0x7f4fdc131ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009850 tx=0x7f4fcc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c320e9923f1d7fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 1 -- 172.21.15.17:0/1904633477 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc02adf0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: dump: 2024-09-15T01:52:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 1 -- 172.21.15.17:0/1904633477 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc032440 con 0x7f4fdc0a0280 2024-09-15T01:52:26.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 1 -- 172.21.15.17:0/1904633477 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0327e0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fe339d640 5 monclient: authenticate success, global_id 34136 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fe339d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fe339d640 4 set_mon_vals no callback set 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.468+0000 7f4fd3fff640 10 monclient: dump: 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.476 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.477 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.477 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.477 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.477 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 10 monclient: shutdown 2024-09-15T01:52:26.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 -- 172.21.15.17:0/1904633477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 msgr2=0x7f4fdc131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 --2- 172.21.15.17:0/1904633477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 0x7f4fdc131ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009850 tx=0x7f4fcc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 -- 172.21.15.17:0/1904633477 shutdown_connections 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 --2- 172.21.15.17:0/1904633477 >> v2:172.21.15.17:3300/0 conn(0x7f4fdc132430 0x7f4fdc138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 --2- 172.21.15.17:0/1904633477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc0a0280 0x7f4fdc131ef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 -- 172.21.15.17:0/1904633477 >> 172.21.15.17:0/1904633477 conn(0x7f4fdc071270 msgr2=0x7f4fdc0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 -- 172.21.15.17:0/1904633477 shutdown_connections 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 1 -- 172.21.15.17:0/1904633477 wait complete. 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.469+0000 7f4fe339d640 10 monclient: monmap: 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 Processor -- start 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 -- start start 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient: init 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 0x7f4fdc132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-a con 0x7f4fdc0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:26.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc131b20 0x7f4fdc132e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-c con 0x7f4fdc131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc132430 0x7f4fdc133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): picked mon.noname-b con 0x7f4fdc132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc137f90 con 0x7f4fdc0a0280 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc134760 con 0x7f4fdc132430 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc138f60 con 0x7f4fdc131b20 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.470+0000 7f4fe339d640 10 monclient(hunting): authenticate will time out at 806.466995s 2024-09-15T01:52:26.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 0x7f4fdc132900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 0x7f4fdc132900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47640/0 (socket says 172.21.15.17:47640) 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 -- 172.21.15.17:0/3785571627 learned_addr learned my addr 172.21.15.17:0/3785571627 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): get_auth_request con 0x7f4fdc0a0280 auth_method 0 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc131b20 0x7f4fdc132e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): get_auth_request con 0x7f4fdc131b20 auth_method 0 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc132430 0x7f4fdc133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): get_auth_request con 0x7f4fdc132430 auth_method 0 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc131b20 msgr2=0x7f4fdc132e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fdc131b20 0x7f4fdc132e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc132430 msgr2=0x7f4fdc133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc132430 0x7f4fdc133380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 monclient: found mon.noname-a 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.471+0000 7f4fe1139640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc009390 con 0x7f4fdc0a0280 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe1139640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 msgr2=0x7f4fdc132900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe1139640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 0x7f4fdc132900 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800c610 tx=0x7f4fd8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8012840 con 0x7f4fdc0a0280 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: dump: 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd80129e0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 5 monclient: authenticate success, global_id 4461 2024-09-15T01:52:26.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 10 monclient: _renew_subs 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8012da0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc134270 con 0x7f4fdc0a0280 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd37fe640 4 set_mon_vals no callback set 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fd17fa640 10 monclient: dump: 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 10 monclient: _renew_subs 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.473+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc134930 con 0x7f4fdc0a0280 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4fd8011590 con 0x7f4fdc0a0280 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe339d640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 0x7f4fdc1338c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc063a40 con 0x7f4fdc02aa40 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fd17fa640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fc0038740 0x7f4fc003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4fd804bab0 con 0x7f4fdc0a0280 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe0938640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 0x7f4fdc1338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.474+0000 7f4fe0938640 10 monclient: get_auth_request con 0x7f4fdc02aa40 auth_method 0 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe193a640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fc0038740 0x7f4fc003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe193a640 10 monclient: get_auth_request con 0x7f4fc0038740 auth_method 0 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 10 monclient(hunting): handle_auth_done global_id 4456 payload 824 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.475+0000 7f4fe0938640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fdc02ae10 con 0x7f4fdc02aa40 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fe193a640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fc0038740 0x7f4fc003abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc136620 tx=0x7f4fd400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fe0938640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 msgr2=0x7f4fdc1338c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fe0938640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 0x7f4fdc1338c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc034000 tx=0x7f4fcc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc031a80 con 0x7f4fdc02aa40 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fd17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fd17fa640 10 monclient: dump: 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.477+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4fcc031be0 con 0x7f4fdc02aa40 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 msgr2=0x7f4fdc1338c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 0x7f4fdc1338c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc034000 tx=0x7f4fcc032000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 msgr2=0x7f4fdc1338c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.478+0000 7f4fd17fa640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc02aa40 0x7f4fdc1338c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc034000 tx=0x7f4fcc032000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.483+0000 7f4fe339d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:26.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.483+0000 7f4fe339d640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 0x7f4fdc05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.483+0000 7f4fe339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.483+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0724a0 con 0x7f4fdc05e310 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 0x7f4fdc05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient: get_auth_request con 0x7f4fdc05e310 auth_method 0 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.484+0000 7f4fe0938640 1 -- 172.21.15.17:0/3785571627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4fdc0980e0 con 0x7f4fdc05e310 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.486+0000 7f4fe0938640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 msgr2=0x7f4fdc05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.486+0000 7f4fe0938640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 0x7f4fdc05e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009130 tx=0x7f4fcc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc03cea0 con 0x7f4fdc05e310 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 10 monclient: dump: 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.499 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f4fcc046600 con 0x7f4fdc05e310 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 msgr2=0x7f4fdc05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 0x7f4fdc05e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009130 tx=0x7f4fcc03d000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 msgr2=0x7f4fdc05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fd17fa640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc05e310 0x7f4fdc05e710 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009130 tx=0x7f4fcc03d000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 14, 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fc0038740 msgr2=0x7f4fc003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fc0038740 0x7f4fc003abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc136620 tx=0x7f4fd400b040 comp rx=0 tx=0).stop 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 10 monclient: shutdown 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 msgr2=0x7f4fdc132900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe339d640 1 --2- 172.21.15.17:0/3785571627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fdc0a0280 0x7f4fdc132900 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800c610 tx=0x7f4fd8007ed0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe193a640 1 -- 172.21.15.17:0/3785571627 reap_dead start 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.487+0000 7f4fe193a640 1 -- 172.21.15.17:0/3785571627 reap_dead start 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.488+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 shutdown_connections 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.488+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 >> 172.21.15.17:0/3785571627 conn(0x7f4fdc071270 msgr2=0x7f4fdc131410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.488+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 shutdown_connections 2024-09-15T01:52:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.488+0000 7f4fe339d640 1 -- 172.21.15.17:0/3785571627 wait complete. 2024-09-15T01:52:26.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:52:26.521 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T01:52:26.521 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:52:26.521 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T01:52:26.521 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:52:26.521 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T01:52:26.521 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:52:26.532 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T01:52:26.532 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:52:26.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:26.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.666+0000 7f2cc76c0640 10 monclient: get_monmap_and_config 2024-09-15T01:52:26.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.666+0000 7f2cc76c0640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.666+0000 7f2cc76c0640 10 monclient: monmap: 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:26.667414+0000 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:26.667414+0000 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:26.669 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 1 Processor -- start 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 1 -- start start 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient: init 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cc00a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc0132040 0x7f2cc00987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc0132040 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 0x7f2cc01391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc0136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc545c640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f2cc00a58b0 legacy=0x7f2cc00980a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.667+0000 7f2cc76c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc00a01e0 con 0x7f2cc00a58b0 2024-09-15T01:52:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00a1580 con 0x7f2cc0136dd0 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00a5c80 con 0x7f2cc0132040 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc5c5d640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 msgr2=0x7f2cc01391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc76c0640 10 monclient(hunting): authenticate will time out at 806.664052s 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc5c5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 0x7f2cc01391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc0132040 0x7f2cc00987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc0132040 0x7f2cc00987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:35102/0 (socket says 172.21.15.17:35102) 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/955809944 learned_addr learned my addr 172.21.15.17:0/955809944 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): get_auth_request con 0x7f2cc0132040 auth_method 0 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_done global_id 4462 payload 274 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/955809944 >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 msgr2=0x7f2cc01391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/955809944 >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 0x7f2cc01391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/955809944 >> v1:172.21.15.17:6789/0 conn(0x7f2cc00a58b0 legacy=0x7f2cc00980a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 monclient: found mon.noname-c 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.668+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/955809944 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0132440 con 0x7f2cc0132040 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/955809944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0132040 msgr2=0x7f2cc00987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/955809944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0132040 0x7f2cc00987b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cc009f780 tx=0x7f2cbc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb9e1754a1dc271 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 1 -- 172.21.15.17:0/955809944 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc032440 con 0x7f2cc0132040 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: dump: 2024-09-15T01:52:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cc76c0640 5 monclient: authenticate success, global_id 4462 2024-09-15T01:52:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 1 -- 172.21.15.17:0/955809944 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc0325a0 con 0x7f2cc0132040 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cc76c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 1 -- 172.21.15.17:0/955809944 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc032940 con 0x7f2cc0132040 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cc76c0640 4 set_mon_vals no callback set 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.671+0000 7f2cb7fff640 10 monclient: dump: 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 10 monclient: shutdown 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 -- 172.21.15.17:0/955809944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0132040 msgr2=0x7f2cc00987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/955809944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0132040 0x7f2cc00987b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cc009f780 tx=0x7f2cbc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 -- 172.21.15.17:0/955809944 shutdown_connections 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/955809944 >> v2:172.21.15.17:3300/0 conn(0x7f2cc0136dd0 0x7f2cc01391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/955809944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0132040 0x7f2cc00987b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 -- 172.21.15.17:0/955809944 >> 172.21.15.17:0/955809944 conn(0x7f2cc0070ed0 msgr2=0x7f2cc00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 -- 172.21.15.17:0/955809944 shutdown_connections 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.672+0000 7f2cc76c0640 1 -- 172.21.15.17:0/955809944 wait complete. 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.673+0000 7f2cc76c0640 10 monclient: build_initial_monmap 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.673+0000 7f2cc76c0640 10 monclient: monmap: 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.673+0000 7f2cc76c0640 1 Processor -- start 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.673+0000 7f2cc76c0640 1 -- start start 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.673+0000 7f2cc76c0640 10 monclient: init 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2cc0141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc00a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc0132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2cc0132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc545c640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 msgr2=0x7f2cc0141a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc0136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0136660 con 0x7f2cc00a58b0 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc545c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2cc0141a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc011bde0 con 0x7f2cc0136dd0 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00a33f0 con 0x7f2cc0132040 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc76c0640 10 monclient(hunting): authenticate will time out at 806.670198s 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59592/0 (socket says 172.21.15.17:59592) 2024-09-15T01:52:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 learned_addr learned my addr 172.21.15.17:0/294813953 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): get_auth_request con 0x7f2cc0132040 auth_method 0 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2cc0132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): get_auth_request con 0x7f2cc0136dd0 auth_method 0 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc5c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient(hunting): handle_auth_done global_id 34142 payload 824 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 msgr2=0x7f2cc0132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2cc0132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 msgr2=0x7f2cc0141a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2cc0141a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 monclient: found mon.noname-c 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.674+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009450 con 0x7f2cc0132040 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.676+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 msgr2=0x7f2cc0141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.676+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009940 tx=0x7f2cbc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.676+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 msgr2=0x7f2cc0141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.676+0000 7f2cc4c5b640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 msgr2=0x7f2cc0141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.676+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009940 tx=0x7f2cbc032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009940 tx=0x7f2cbc032000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 msgr2=0x7f2cc0141f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2cc0132040 0x7f2cc0141f60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 0x7f2ca40058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca4003460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2ca40083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc0136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2ca400ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc00a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca400b830 con 0x7f2cc00a58b0 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca400b970 con 0x7f2cc0136dd0 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca400bb30 con 0x7f2ca4003460 2024-09-15T01:52:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cb57fa640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc5c5d640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 msgr2=0x7f2ca400ae20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc5c5d640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2ca400ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 0x7f2ca40058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): get_auth_request con 0x7f2ca4003460 auth_method 0 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient(hunting): handle_auth_done global_id 34142 payload 293 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 msgr2=0x7f2ca40083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2ca40083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 msgr2=0x7f2ca400ae20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2ca400ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 monclient: found mon.noname-c 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.677+0000 7f2cc545c640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca400bdb0 con 0x7f2ca4003460 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc545c640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 msgr2=0x7f2ca40058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc545c640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 0x7f2ca40058a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009990 tx=0x7f2cb0006f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003bd0 con 0x7f2ca4003460 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: dump: 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb0003d30 con 0x7f2ca4003460 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 5 monclient: authenticate success, global_id 34142 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 10 monclient: _renew_subs 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc01360a0 con 0x7f2ca4003460 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003ea0 con 0x7f2ca4003460 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb57fa640 10 monclient: dump: 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 10 monclient: _renew_subs 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cb6ffd640 4 set_mon_vals no callback set 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.679+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc01351b0 con 0x7f2ca4003460 2024-09-15T01:52:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2cb0005630 con 0x7f2ca4003460 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cb57fa640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 0x7f2ca4042ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f2cb0049e30 con 0x7f2ca4003460 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cc76c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cc76c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cc76c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.680+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0135400 con 0x7f2ca4003460 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.681+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 0x7f2ca4042ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.681+0000 7f2cc4c5b640 10 monclient: get_auth_request con 0x7f2cc0132040 auth_method 0 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.683+0000 7f2cc4c5b640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 0x7f2ca4042ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc000e50 tx=0x7f2cbc03c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.685+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2cb000b640 con 0x7f2ca4003460 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.685+0000 7f2cb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.685+0000 7f2cb57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:26.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cc76c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:26.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cc76c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:26.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cc76c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:26.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2cc01362b0 con 0x7f2ca4003460 2024-09-15T01:52:26.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cb57fa640 1 -- 172.21.15.17:0/294813953 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f2cb00192b0 con 0x7f2ca4003460 2024-09-15T01:52:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.911+0000 7f2cb57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:26.914 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 msgr2=0x7f2ca4042ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 0x7f2ca4042ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc000e50 tx=0x7f2cbc03c000 comp rx=0 tx=0).stop 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 10 monclient: shutdown 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 msgr2=0x7f2ca40058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:26.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 0x7f2ca40058a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009990 tx=0x7f2cb0006f70 comp rx=0 tx=0).stop 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 shutdown_connections 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2cc0132040 0x7f2ca4042ec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2cc00a58b0 0x7f2ca400ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc0136dd0 0x7f2ca40083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 --2- 172.21.15.17:0/294813953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4003460 0x7f2ca40058a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 >> 172.21.15.17:0/294813953 conn(0x7f2cc0070ed0 msgr2=0x7f2cc00a4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 shutdown_connections 2024-09-15T01:52:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:26.913+0000 7f2cc76c0640 1 -- 172.21.15.17:0/294813953 wait complete. 2024-09-15T01:52:28.692 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 97% (11828/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T01:52:29.684 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:29.685 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T01:52:29.689 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T01:52:29.755 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T01:52:29.758 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T01:52:29.758 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T01:52:29.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.032+0000 7f766a91d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.032+0000 7f766a91d640 10 monclient: build_initial_monmap 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.032+0000 7f766a91d640 10 monclient: monmap: 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:30.034011+0000 2024-09-15T01:52:30.035 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:30.034011+0000 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:30.036 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 Processor -- start 2024-09-15T01:52:30.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 -- start start 2024-09-15T01:52:30.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient: init 2024-09-15T01:52:30.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-b con 0x7f7664131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76640a0280 0x7f7664131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-c con 0x7f76640a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 0x7f7664132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-a con 0x7f766409f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76640a0040 con 0x7f766409f1e0 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766409bff0 con 0x7f76640a0280 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76640a14a0 con 0x7f7664131b20 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f766a91d640 10 monclient(hunting): authenticate will time out at 810.030828s 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f7668eba640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 msgr2=0x7f7664132430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f7668eba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 0x7f7664132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f7663fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.033+0000 7f7663fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59608/0 (socket says 172.21.15.17:59608) 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 -- 172.21.15.17:0/195217651 learned_addr learned my addr 172.21.15.17:0/195217651 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): get_auth_request con 0x7f7664131b20 auth_method 0 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 1 --2- 172.21.15.17:0/195217651 >> v2:172.21.15.67:3300/0 conn(0x7f76640a0280 0x7f7664131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): get_auth_request con 0x7f76640a0280 auth_method 0 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f765bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:30.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient(hunting): handle_auth_done global_id 34145 payload 274 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 -- 172.21.15.17:0/195217651 >> v2:172.21.15.67:3300/0 conn(0x7f76640a0280 msgr2=0x7f7664131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 --2- 172.21.15.17:0/195217651 >> v2:172.21.15.67:3300/0 conn(0x7f76640a0280 0x7f7664131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 -- 172.21.15.17:0/195217651 >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 msgr2=0x7f7664132430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 --2- 172.21.15.17:0/195217651 >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 0x7f7664132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 monclient: found mon.noname-b 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.034+0000 7f7663fff640 1 -- 172.21.15.17:0/195217651 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664068e80 con 0x7f7664131b20 2024-09-15T01:52:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f7663fff640 1 -- 172.21.15.17:0/195217651 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 msgr2=0x7f766409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f7663fff640 1 --2- 172.21.15.17:0/195217651 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f766406ab60 tx=0x7f76540091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb7096baebb0a439 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 1 -- 172.21.15.17:0/195217651 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654032440 con 0x7f7664131b20 2024-09-15T01:52:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: dump: 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.050 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 5 monclient: authenticate success, global_id 34145 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 1 -- 172.21.15.17:0/195217651 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76540325a0 con 0x7f7664131b20 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 4 set_mon_vals no callback set 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 1 -- 172.21.15.17:0/195217651 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654032940 con 0x7f7664131b20 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f76637fe640 10 monclient: dump: 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:30.051 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 10 monclient: shutdown 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 1 -- 172.21.15.17:0/195217651 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 msgr2=0x7f766409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.037+0000 7f766a91d640 1 --2- 172.21.15.17:0/195217651 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f766406ab60 tx=0x7f76540091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 -- 172.21.15.17:0/195217651 shutdown_connections 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 --2- 172.21.15.17:0/195217651 >> v2:172.21.15.67:3300/0 conn(0x7f76640a0280 0x7f7664131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 --2- 172.21.15.17:0/195217651 >> v2:172.21.15.17:3300/0 conn(0x7f766409f1e0 0x7f7664132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 --2- 172.21.15.17:0/195217651 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766409eca0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 -- 172.21.15.17:0/195217651 >> 172.21.15.17:0/195217651 conn(0x7f7664071270 msgr2=0x7f76640a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 -- 172.21.15.17:0/195217651 shutdown_connections 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 -- 172.21.15.17:0/195217651 wait complete. 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 10 monclient: build_initial_monmap 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 10 monclient: monmap: 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:30.053 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.038+0000 7f766a91d640 1 Processor -- start 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 -- start start 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient: init 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 0x7f7664141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-a con 0x7f766409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76640a0280 0x7f7664141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-b con 0x7f76640a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): picked mon.noname-c con 0x7f7664131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766411e2b0 con 0x7f766409f1e0 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76640a09b0 con 0x7f76640a0280 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766409d4e0 con 0x7f7664131b20 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7663fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 msgr2=0x7f7664141210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:30.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f766a91d640 10 monclient(hunting): authenticate will time out at 810.036550s 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7663fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 0x7f7664141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59626/0 (socket says 172.21.15.17:59626) 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 -- 172.21.15.17:0/2752139141 learned_addr learned my addr 172.21.15.17:0/2752139141 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): get_auth_request con 0x7f7664131b20 auth_method 0 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f765bfff640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76640a0280 0x7f7664141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f765bfff640 10 monclient(hunting): get_auth_request con 0x7f76640a0280 auth_method 0 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f765bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f765bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f765bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient(hunting): handle_auth_done global_id 34148 payload 824 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 -- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76640a0280 msgr2=0x7f7664141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76640a0280 0x7f7664141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 -- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 msgr2=0x7f7664141210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 0x7f7664141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient: _finish_hunting 0 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 monclient: found mon.noname-c 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.039+0000 7f7668eba640 1 -- 172.21.15.17:0/2752139141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009450 con 0x7f7664131b20 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7668eba640 1 -- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 msgr2=0x7f766411c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7668eba640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f765c00c040 tx=0x7f765c009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00ce20 con 0x7f7664131b20 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:30.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: dump: 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: _finish_auth 0 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:30.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c006360 con 0x7f7664131b20 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c006510 con 0x7f7664131b20 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 5 monclient: authenticate success, global_id 34148 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 10 monclient: _renew_subs 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766411fce0 con 0x7f7664131b20 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7662ffd640 4 set_mon_vals no callback set 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:30.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f7660ff9640 10 monclient: dump: 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 10 monclient: _renew_subs 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.042+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664120f80 con 0x7f7664131b20 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f765c012020 con 0x7f7664131b20 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f766a91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f766a91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f766a91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76640724a0 con 0x7f7664131b20 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f7660ff9640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 0x7f764403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f765c04bb50 con 0x7f7664131b20 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f7663fff640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 0x7f764403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.043+0000 7f7663fff640 10 monclient: get_auth_request con 0x7f76440386f0 auth_method 0 2024-09-15T01:52:30.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.046+0000 7f7663fff640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 0x7f764403aba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f76640a2f90 tx=0x7f7654032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.048+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f765c0132e0 con 0x7f7664131b20 2024-09-15T01:52:30.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.048+0000 7f7660ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.048+0000 7f7660ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.273+0000 7f766a91d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.273+0000 7f766a91d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.273+0000 7f766a91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.273+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f766406efc0 con 0x7f7664131b20 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.274+0000 7f7660ff9640 1 -- 172.21.15.17:0/2752139141 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f765c0072f0 con 0x7f7664131b20 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.274+0000 7f7660ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.274+0000 7f7660ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:30.277 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 msgr2=0x7f764403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 0x7f764403aba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f76640a2f90 tx=0x7f7654032000 comp rx=0 tx=0).stop 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 10 monclient: shutdown 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 msgr2=0x7f766411c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.275+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f765c00c040 tx=0x7f765c009730 comp rx=0 tx=0).stop 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 shutdown_connections 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76440386f0 0x7f764403aba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7664131b20 0x7f766411c640 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76640a0280 0x7f7664141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 --2- 172.21.15.17:0/2752139141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f766409f1e0 0x7f7664141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 >> 172.21.15.17:0/2752139141 conn(0x7f7664071270 msgr2=0x7f7664130920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 shutdown_connections 2024-09-15T01:52:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:30.276+0000 7f766a91d640 1 -- 172.21.15.17:0/2752139141 wait complete. 2024-09-15T01:52:30.397 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T01:52:30.465 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T01:52:30.507 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T01:52:30.508 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T01:52:30.508 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T01:52:30.548 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T01:52:30.551 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T01:52:30.551 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T01:52:30.607 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-15T01:52:30.608 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-15T01:52:30.608 DEBUG:teuthology.orchestra.run.smithi067:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-15T01:52:30.667 INFO:tasks.workunit.client.0.smithi067.stderr:+ TEST_POOL=testpool1234 2024-09-15T01:52:30.668 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-15T01:52:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.744+0000 7f05f279d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.744+0000 7f05f279d640 10 monclient: build_initial_monmap 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.744+0000 7f05f279d640 10 monclient: monmap: 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:30.745566+0000 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:30.745566+0000 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:30.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 Processor -- start 2024-09-15T01:52:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 -- start start 2024-09-15T01:52:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient: init 2024-09-15T01:52:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-b con 0x7f05ec0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05ec09eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05ec09f270 0x7f05ec132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-c con 0x7f05ec09f270 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec06a270 con 0x7f05ec09eea0 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ec06f290 con 0x7f05ec09f270 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0a0f20 con 0x7f05ec0a0b50 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f279d640 10 monclient(hunting): authenticate will time out at 783.214665s 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05eb7fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f05ec09eea0 legacy=0x7f05ec131b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05ec09f270 0x7f05ec132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05ec09f270 0x7f05ec132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60400/0 (socket says 172.21.15.67:60400) 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 1 -- 172.21.15.67:0/839548747 learned_addr learned my addr 172.21.15.67:0/839548747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): get_auth_request con 0x7f05ec09f270 auth_method 0 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.745+0000 7f05f0d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05ebfff640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05ebfff640 10 monclient(hunting): get_auth_request con 0x7f05ec0a0b50 auth_method 0 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 10 monclient(hunting): handle_auth_done global_id 4474 payload 274 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 1 -- 172.21.15.67:0/839548747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 msgr2=0x7f05ec069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 1 -- 172.21.15.67:0/839548747 >> v1:172.21.15.17:6789/0 conn(0x7f05ec09eea0 legacy=0x7f05ec131b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 1 monclient: found mon.noname-c 2024-09-15T01:52:30.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:30.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.746+0000 7f05f0d3a640 1 -- 172.21.15.67:0/839548747 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec0a6160 con 0x7f05ec09f270 2024-09-15T01:52:30.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.748+0000 7f05f0d3a640 1 -- 172.21.15.67:0/839548747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 msgr2=0x7f05ec132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:30.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.748+0000 7f05f0d3a640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec132220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009f20 tx=0x7f05e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d962234c8e71c915 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 1 -- 172.21.15.67:0/839548747 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4007610 con 0x7f05ec09f270 2024-09-15T01:52:30.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: dump: 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 1 -- 172.21.15.67:0/839548747 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e402d050 con 0x7f05ec09f270 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 5 monclient: authenticate success, global_id 4474 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 1 -- 172.21.15.67:0/839548747 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40079a0 con 0x7f05ec09f270 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 4 set_mon_vals no callback set 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05eaffd640 10 monclient: dump: 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.760 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 10 monclient: shutdown 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 -- 172.21.15.67:0/839548747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 msgr2=0x7f05ec132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec132220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009f20 tx=0x7f05e4009840 comp rx=0 tx=0).stop 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 -- 172.21.15.67:0/839548747 shutdown_connections 2024-09-15T01:52:30.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec132220 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 --2- 172.21.15.67:0/839548747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 -- 172.21.15.67:0/839548747 >> 172.21.15.67:0/839548747 conn(0x7f05ec071290 msgr2=0x7f05ec0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 -- 172.21.15.67:0/839548747 shutdown_connections 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.749+0000 7f05f279d640 1 -- 172.21.15.67:0/839548747 wait complete. 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.750+0000 7f05f279d640 10 monclient: build_initial_monmap 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.750+0000 7f05f279d640 10 monclient: monmap: 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:30.762 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.750+0000 7f05f279d640 1 Processor -- start 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.750+0000 7f05f279d640 1 -- start start 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.750+0000 7f05f279d640 10 monclient: init 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 0x7f05ec133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-a con 0x7f05ec09eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-b con 0x7f05ec09f270 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec11dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): picked mon.noname-c con 0x7f05ec0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:30.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0a1260 con 0x7f05ec09eea0 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0a49d0 con 0x7f05ec09f270 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec11fc50 con 0x7f05ec0a0b50 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f279d640 10 monclient(hunting): authenticate will time out at 783.220176s 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05ebfff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 msgr2=0x7f05ec133a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05ebfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 0x7f05ec133a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60412/0 (socket says 172.21.15.67:60412) 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 -- 172.21.15.67:0/1401237915 learned_addr learned my addr 172.21.15.67:0/1401237915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:30.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): get_auth_request con 0x7f05ec09f270 auth_method 0 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec11dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 10 monclient(hunting): get_auth_request con 0x7f05ec0a0b50 auth_method 0 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 -- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 msgr2=0x7f05ec11dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:30.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec11dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 -- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 msgr2=0x7f05ec133a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 0x7f05ec133a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 monclient: found mon.noname-b 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05eb7fe640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4009ae0 con 0x7f05ec09f270 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.751+0000 7f05f0d3a640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec11dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05eb7fe640 1 -- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 msgr2=0x7f05ec11d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05eb7fe640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05e000b5c0 tx=0x7f05e0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0003d40 con 0x7f05ec09f270 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: dump: 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:30.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0006150 con 0x7f05ec09f270 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05f279d640 5 monclient: authenticate success, global_id 4477 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0006510 con 0x7f05ec09f270 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05d3fff640 10 monclient: dump: 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:30.768 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05f279d640 10 monclient: _renew_subs 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05f279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec11f590 con 0x7f05ec09f270 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.753+0000 7f05ea7fc640 4 set_mon_vals no callback set 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 10 monclient: _renew_subs 2024-09-15T01:52:30.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ec11f080 con 0x7f05ec09f270 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f05e00105d0 con 0x7f05ec09f270 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.754+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.755+0000 7f05d3fff640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 0x7f05d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.755+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f05e004b100 con 0x7f05ec09f270 2024-09-15T01:52:30.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.755+0000 7f05ebfff640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 0x7f05d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:30.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.755+0000 7f05ebfff640 10 monclient: get_auth_request con 0x7f05d8038740 auth_method 0 2024-09-15T01:52:30.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.757+0000 7f05ebfff640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 0x7f05d803abf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05dc006f40 tx=0x7f05dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:30.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.759+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:30.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.759+0000 7f05d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:30.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.759+0000 7f05d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:30.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.927+0000 7f05f279d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:30.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.927+0000 7f05f279d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:30.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.927+0000 7f05f279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:30.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:30.927+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:33.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.396+0000 7f1eb182b640 10 monclient: get_monmap_and_config 2024-09-15T01:52:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.396+0000 7f1eb182b640 10 monclient: build_initial_monmap 2024-09-15T01:52:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.396+0000 7f1eb182b640 10 monclient: monmap: 2024-09-15T01:52:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:33.398346+0000 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:33.398346+0000 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:33.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 Processor -- start 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 -- start start 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient: init 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 0x7f1eac131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac1325e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:33.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eac0a0700 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac06a650 con 0x7f1eac0a0700 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac09f340 con 0x7f1eac1325e0 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1329b0 con 0x7f1eac136e50 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eb182b640 10 monclient(hunting): authenticate will time out at 813.394731s 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eaaffd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 msgr2=0x7f1eac131d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eaaffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 0x7f1eac131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45250/0 (socket says 172.21.15.17:45250) 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 1 -- 172.21.15.17:0/111815237 learned_addr learned my addr 172.21.15.17:0/111815237 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 10 monclient(hunting): get_auth_request con 0x7f1eac136e50 auth_method 0 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.397+0000 7f1eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_done global_id 34157 payload 274 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 1 -- 172.21.15.17:0/111815237 >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 msgr2=0x7f1eac131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 1 --2- 172.21.15.17:0/111815237 >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 0x7f1eac131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 1 -- 172.21.15.17:0/111815237 >> v1:172.21.15.67:6789/0 conn(0x7f1eac0a0700 legacy=0x7f1eac09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 1 monclient: found mon.noname-b 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.398+0000 7f1eab7fe640 1 -- 172.21.15.17:0/111815237 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0a19b0 con 0x7f1eac136e50 2024-09-15T01:52:33.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eab7fe640 1 -- 172.21.15.17:0/111815237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 msgr2=0x7f1eac139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:33.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eab7fe640 1 --2- 172.21.15.17:0/111815237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eac04b910 tx=0x7f1e9c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c12fcc784d75d956 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 1 -- 172.21.15.17:0/111815237 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c032440 con 0x7f1eac136e50 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: dump: 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 1 -- 172.21.15.17:0/111815237 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c0325a0 con 0x7f1eac136e50 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 1 -- 172.21.15.17:0/111815237 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c032940 con 0x7f1eac136e50 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 5 monclient: authenticate success, global_id 34157 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 4 set_mon_vals no callback set 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1ea9ffb640 10 monclient: dump: 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:33.414 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 10 monclient: shutdown 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 -- 172.21.15.17:0/111815237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 msgr2=0x7f1eac139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 --2- 172.21.15.17:0/111815237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eac04b910 tx=0x7f1e9c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 -- 172.21.15.17:0/111815237 shutdown_connections 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 --2- 172.21.15.17:0/111815237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac139230 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 --2- 172.21.15.17:0/111815237 >> v2:172.21.15.17:3300/0 conn(0x7f1eac1325e0 0x7f1eac131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 -- 172.21.15.17:0/111815237 >> 172.21.15.17:0/111815237 conn(0x7f1eac071320 msgr2=0x7f1eac0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 -- 172.21.15.17:0/111815237 shutdown_connections 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.401+0000 7f1eb182b640 1 -- 172.21.15.17:0/111815237 wait complete. 2024-09-15T01:52:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.402+0000 7f1eb182b640 10 monclient: build_initial_monmap 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.402+0000 7f1eb182b640 10 monclient: monmap: 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.402+0000 7f1eb182b640 1 Processor -- start 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 -- start start 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient: init 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 0x7f1eac098980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac0a0700 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 0x7f1eac098ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac1325e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac11c3a0 con 0x7f1eac0a0700 2024-09-15T01:52:33.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac09bef0 con 0x7f1eac1325e0 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eaaffd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 msgr2=0x7f1eac098980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac11b740 con 0x7f1eac136e50 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eb182b640 10 monclient(hunting): authenticate will time out at 813.400674s 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eaaffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 0x7f1eac098980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:33.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45252/0 (socket says 172.21.15.17:45252) 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 1 -- 172.21.15.17:0/2409845667 learned_addr learned my addr 172.21.15.17:0/2409845667 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): get_auth_request con 0x7f1eac136e50 auth_method 0 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.403+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eaa7fc640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 0x7f1eac098ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 10 monclient(hunting): handle_auth_done global_id 34160 payload 824 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 -- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 msgr2=0x7f1eac098ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:33.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 0x7f1eac098ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 -- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 msgr2=0x7f1eac098980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 0x7f1eac098980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 monclient: found mon.noname-c 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eab7fe640 1 -- 172.21.15.17:0/2409845667 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c009450 con 0x7f1eac136e50 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eaa7fc640 10 monclient: get_auth_request con 0x7f1eac1325e0 auth_method 0 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.404+0000 7f1eaa7fc640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 0x7f1eac098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eab7fe640 1 -- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 msgr2=0x7f1eac099400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eab7fe640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009420 tx=0x7f1e9c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c031b20 con 0x7f1eac136e50 2024-09-15T01:52:33.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:33.421 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c031c80 con 0x7f1eac136e50 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 5 monclient: authenticate success, global_id 34160 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 10 monclient: _renew_subs 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac09ae60 con 0x7f1eac136e50 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c03b400 con 0x7f1eac136e50 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-15T01:52:33.422 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1ea8ff9640 4 set_mon_vals no callback set 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 10 monclient: _renew_subs 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.406+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac11bd20 con 0x7f1eac136e50 2024-09-15T01:52:33.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1e9c039040 con 0x7f1eac136e50 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1eb182b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1eb182b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1eb182b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac0724a0 con 0x7f1eac136e50 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1ea2ffd640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.407+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f1e9c06bf10 con 0x7f1eac136e50 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.408+0000 7f1eaaffd640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:33.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.408+0000 7f1eaaffd640 10 monclient: get_auth_request con 0x7f1e8c038740 auth_method 0 2024-09-15T01:52:33.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.410+0000 7f1eaaffd640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4003cc0 tx=0x7f1ea4009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:33.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.412+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1e9c03b830 con 0x7f1eac136e50 2024-09-15T01:52:33.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.412+0000 7f1ea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:33.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.412+0000 7f1ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.639+0000 7f1eb182b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1eb182b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1eb182b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1eac135280 con 0x7f1eac136e50 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1ea2ffd640 1 -- 172.21.15.17:0/2409845667 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f1e9c049020 con 0x7f1eac136e50 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1ea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.640+0000 7f1ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:33.644 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 msgr2=0x7f1e8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:33.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4003cc0 tx=0x7f1ea4009310 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 10 monclient: shutdown 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 msgr2=0x7f1eac099400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009420 tx=0x7f1e9c032000 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 shutdown_connections 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1eac136e50 0x7f1eac099400 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eac1325e0 0x7f1eac098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 --2- 172.21.15.17:0/2409845667 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1eac0a0700 0x7f1eac098980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 >> 172.21.15.17:0/2409845667 conn(0x7f1eac071320 msgr2=0x7f1eac0a3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 shutdown_connections 2024-09-15T01:52:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:33.642+0000 7f1eb182b640 1 -- 172.21.15.17:0/2409845667 wait complete. 2024-09-15T01:52:33.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:33.751+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:33.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:33.751+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:33.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:33.751+0000 7f05d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:34.751+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:34.751+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:35.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:35.752+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:35.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:35.752+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:36.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:52:36.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:36.752+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:36.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:36.752+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:36.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.762+0000 7f2d6e82b640 10 monclient: get_monmap_and_config 2024-09-15T01:52:36.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.762+0000 7f2d6e82b640 10 monclient: build_initial_monmap 2024-09-15T01:52:36.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.762+0000 7f2d6e82b640 10 monclient: monmap: 2024-09-15T01:52:36.765 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:36.764474+0000 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:36.764474+0000 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:36.766 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 1 Processor -- start 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 1 -- start start 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 10 monclient: init 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-b con 0x7f2d68131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.763+0000 7f2d6e82b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-c con 0x7f2d680a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d6809f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680a0040 con 0x7f2d6809f1e0 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6809bff0 con 0x7f2d680a0280 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680a14a0 con 0x7f2d68131b20 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6e82b640 10 monclient(hunting): authenticate will time out at 816.761064s 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d6cdc8640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f2d6809f1e0 legacy=0x7f2d68068e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45266/0 (socket says 172.21.15.17:45266) 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 learned_addr learned my addr 172.21.15.17:0/3849012966 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d68131b20 auth_method 0 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:36.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 10 monclient(hunting): get_auth_request con 0x7f2d680a0280 auth_method 0 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:52:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.764+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:36.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d6cdc8640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.17:6789/0 conn(0x7f2d6809f1e0 legacy=0x7f2d68068e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d68131b20 auth_method 0 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 10 monclient(hunting): get_auth_request con 0x7f2d680a0280 auth_method 0 2024-09-15T01:52:36.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:36.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:36.965+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:52:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d6cdc8640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.17:6789/0 conn(0x7f2d6809f1e0 legacy=0x7f2d68068e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d68131b20 auth_method 0 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 10 monclient(hunting): get_auth_request con 0x7f2d680a0280 auth_method 0 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:52:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:37.366+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:52:37.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:37.752+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:37.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:37.752+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:38.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d6cdc8640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.17:6789/0 conn(0x7f2d6809f1e0 legacy=0x7f2d68068e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:38.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d68131b20 auth_method 0 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 10 monclient(hunting): get_auth_request con 0x7f2d680a0280 auth_method 0 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:38.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:38.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:38.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:52:38.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:52:38.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:52:38.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:38.167+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:52:38.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:38.751+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:38.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:38.751+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:39.752+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:39.752+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): tick 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 msgr2=0x7f2d6809eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d68131b20 0x7f2d6809eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 msgr2=0x7f2d68131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:39.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.67:3300/0 conn(0x7f2d680a0280 0x7f2d68131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.17:6789/0 conn(0x7f2d6809f1e0 legacy=0x7f2d68068e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient: _add_conns ranks=[3,1] 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f2d54003680 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d540061c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 -- 172.21.15.17:0/3849012966 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68132430 con 0x7f2d540061c0 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 1 -- 172.21.15.17:0/3849012966 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6809bff0 con 0x7f2d54003680 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d65ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 msgr2=0x7f2d54005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:39.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.764+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:39.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.766+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.67:6789/0 conn(0x7f2d540061c0 legacy=0x7f2d54008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.965+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 msgr2=0x7f2d54005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:39.965+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:52:40.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:40.366+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 msgr2=0x7f2d54005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:40.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:40.366+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:52:40.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:40.752+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:40.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:40.752+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.167+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 msgr2=0x7f2d54005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.167+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:52:41.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.431+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848590726 0 0) 0x7f2d68132430 con 0x7f2d540061c0 2024-09-15T01:52:41.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.431+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.431+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.431+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4480 2024-09-15T01:52:41.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.431+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d500034e0 con 0x7f2d540061c0 2024-09-15T01:52:41.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 760804293 0 0) 0x7f2d500034e0 con 0x7f2d540061c0 2024-09-15T01:52:41.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 msgr2=0x7f2d54005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:41.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 --2- 172.21.15.17:0/3849012966 >> v2:172.21.15.17:3300/0 conn(0x7f2d54003680 0x7f2d54005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680a0040 con 0x7f2d540061c0 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f2d5c00d480 con 0x7f2d540061c0 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6e82b640 5 monclient: authenticate success, global_id 4480 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6e82b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: dump: 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 >> v1:172.21.15.67:6789/0 conn(0x7f2d540061c0 legacy=0x7f2d54008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 1 -- 172.21.15.17:0/3849012966 reap_dead start 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 0x7f2d50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): picked mon.b con 0x7f2d500039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 0x7f2d50004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): picked mon.a con 0x7f2d50004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 0x7f2d5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): picked mon.c con 0x7f2d50005230 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6809bff0 con 0x7f2d50004780 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5000c1f0 con 0x7f2d500039d0 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5000c3b0 con 0x7f2d50005230 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6e82b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d677fe640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 msgr2=0x7f2d50004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d677fe640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 0x7f2d50004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:41.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 0x7f2d5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 0x7f2d50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d500039d0 auth_method 0 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): get_auth_request con 0x7f2d50005230 auth_method 0 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient(hunting): handle_auth_done global_id 4480 payload 293 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 msgr2=0x7f2d5000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 0x7f2d5000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 msgr2=0x7f2d50004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 0x7f2d50004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 monclient: found mon.b 2024-09-15T01:52:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:41.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500034e0 con 0x7f2d500039d0 2024-09-15T01:52:41.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:41.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.465+0000 7f2d6cdc8640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 0x7f2d5000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.468+0000 7f2d67fff640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 msgr2=0x7f2d50003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.468+0000 7f2d67fff640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 0x7f2d50003dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006f60 tx=0x7f2d58008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e5a6b890b2bd9a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5802b940 con 0x7f2d500039d0 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: dump: 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.482 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5802f070 con 0x7f2d500039d0 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 1 -- 172.21.15.17:0/3849012966 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5802bd10 con 0x7f2d500039d0 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:41.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 4 set_mon_vals no callback set 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d66ffd640 10 monclient: dump: 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.484 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 10 monclient: shutdown 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 1 -- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 msgr2=0x7f2d50003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.469+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 0x7f2d50003dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006f60 tx=0x7f2d58008040 comp rx=0 tx=0).stop 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 -- 172.21.15.17:0/3849012966 shutdown_connections 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50004780 0x7f2d50004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d50005230 0x7f2d5000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/3849012966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d500039d0 0x7f2d50003dc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 -- 172.21.15.17:0/3849012966 >> 172.21.15.17:0/3849012966 conn(0x7f2d68071270 msgr2=0x7f2d680a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 -- 172.21.15.17:0/3849012966 shutdown_connections 2024-09-15T01:52:41.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 1 -- 172.21.15.17:0/3849012966 wait complete. 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 10 monclient: build_initial_monmap 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.470+0000 7f2d6e82b640 10 monclient: monmap: 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.471+0000 7f2d6e82b640 1 Processor -- start 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.471+0000 7f2d6e82b640 1 -- start start 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.471+0000 7f2d6e82b640 10 monclient: init 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.471+0000 7f2d6e82b640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.471+0000 7f2d6e82b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-c con 0x7f2d500039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50004780 0x7f2d6811b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-b con 0x7f2d50004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 0x7f2d6811bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): picked mon.noname-a con 0x7f2d50005230 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680a09b0 con 0x7f2d50005230 2024-09-15T01:52:41.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6811ee60 con 0x7f2d50004780 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6811d050 con 0x7f2d500039d0 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6e82b640 10 monclient(hunting): authenticate will time out at 821.469114s 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6cdc8640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 msgr2=0x7f2d6811bc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d6cdc8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 0x7f2d6811bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45324/0 (socket says 172.21.15.17:45324) 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 -- 172.21.15.17:0/86918340 learned_addr learned my addr 172.21.15.17:0/86918340 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): get_auth_request con 0x7f2d500039d0 auth_method 0 2024-09-15T01:52:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50004780 0x7f2d6811b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): get_auth_request con 0x7f2d50004780 auth_method 0 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient(hunting): handle_auth_done global_id 34163 payload 824 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 -- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50004780 msgr2=0x7f2d6811b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50004780 0x7f2d6811b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 -- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 msgr2=0x7f2d6811bc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 0x7f2d6811bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 monclient: found mon.noname-c 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.472+0000 7f2d67fff640 1 -- 172.21.15.17:0/86918340 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6811dc20 con 0x7f2d500039d0 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d67fff640 1 -- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 msgr2=0x7f2d6812ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d67fff640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d58012cc0 tx=0x7f2d58036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58006a00 con 0x7f2d500039d0 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: dump: 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.491 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: _finish_auth 0 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5800c040 con 0x7f2d500039d0 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 5 monclient: authenticate success, global_id 34163 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 10 monclient: _renew_subs 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6811eb80 con 0x7f2d500039d0 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58035c90 con 0x7f2d500039d0 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d4ffff640 10 monclient: dump: 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 10 monclient: _renew_subs 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d65ffb640 4 set_mon_vals no callback set 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.475+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6811d750 con 0x7f2d500039d0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d58040620 con 0x7f2d500039d0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d4ffff640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 0x7f2d5403b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d6e82b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d6e82b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d6e82b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f2d5806c9a0 con 0x7f2d500039d0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6811f020 con 0x7f2d500039d0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.476+0000 7f2d677fe640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 0x7f2d5403b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.477+0000 7f2d677fe640 10 monclient: get_auth_request con 0x7f2d54038e40 auth_method 0 2024-09-15T01:52:41.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.479+0000 7f2d677fe640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 0x7f2d5403b2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c006e70 tx=0x7f2d5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.481+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2d58006c40 con 0x7f2d500039d0 2024-09-15T01:52:41.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.481+0000 7f2d4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:41.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.481+0000 7f2d4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d6e82b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:52:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d6e82b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d6e82b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2d680724a0 con 0x7f2d500039d0 2024-09-15T01:52:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d4ffff640 1 -- 172.21.15.17:0/86918340 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f2d5803db50 con 0x7f2d500039d0 2024-09-15T01:52:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:52:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.712+0000 7f2d4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:41.716 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:52:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 msgr2=0x7f2d5403b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 0x7f2d5403b2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c006e70 tx=0x7f2d5c008040 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 10 monclient: shutdown 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 msgr2=0x7f2d6812ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d58012cc0 tx=0x7f2d58036000 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 shutdown_connections 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2d54038e40 0x7f2d5403b2f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2d50005230 0x7f2d6811bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50004780 0x7f2d6811b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 --2- 172.21.15.17:0/86918340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2d500039d0 0x7f2d6812ea30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 >> 172.21.15.17:0/86918340 conn(0x7f2d68071270 msgr2=0x7f2d68130680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 shutdown_connections 2024-09-15T01:52:41.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.714+0000 7f2d6e82b640 1 -- 172.21.15.17:0/86918340 wait complete. 2024-09-15T01:52:41.728 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:52:41.728 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T01:52:41.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:52:41.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:41.753+0000 7f05d2ffd640 10 monclient: tick 2024-09-15T01:52:41.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:41.753+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.823+0000 7f821a44f640 10 monclient: get_monmap_and_config 2024-09-15T01:52:41.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.823+0000 7f821a44f640 10 monclient: build_initial_monmap 2024-09-15T01:52:41.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.823+0000 7f821a44f640 10 monclient: monmap: 2024-09-15T01:52:41.826 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:41.825307+0000 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:41.825307+0000 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:41.827 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 Processor -- start 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 -- start start 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient: init 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82140708a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 0x7f821409b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-a con 0x7f821406a510 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8214137130 0x7f8214139510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8214137130 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8214070c70 con 0x7f82140708a0 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821409da40 con 0x7f821406a510 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f8213fff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f82140708a0 legacy=0x7f821409ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8214070f30 con 0x7f8214137130 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f821a44f640 10 monclient(hunting): authenticate will time out at 821.821979s 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.824+0000 7f82137fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 msgr2=0x7f821409b530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82137fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 0x7f821409b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8214137130 0x7f8214139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8214137130 0x7f8214139510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:36026/0 (socket says 172.21.15.17:36026) 2024-09-15T01:52:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 -- 172.21.15.17:0/27593265 learned_addr learned my addr 172.21.15.17:0/27593265 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): get_auth_request con 0x7f8214137130 auth_method 0 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient(hunting): handle_auth_done global_id 4486 payload 274 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 -- 172.21.15.17:0/27593265 >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 msgr2=0x7f821409b530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 --2- 172.21.15.17:0/27593265 >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 0x7f821409b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 -- 172.21.15.17:0/27593265 >> v1:172.21.15.17:6789/0 conn(0x7f82140708a0 legacy=0x7f821409ae20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 monclient: found mon.noname-c 2024-09-15T01:52:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.825+0000 7f82189ec640 1 -- 172.21.15.17:0/27593265 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821409c4b0 con 0x7f8214137130 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.828+0000 7f82189ec640 1 -- 172.21.15.17:0/27593265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8214137130 msgr2=0x7f8214139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.828+0000 7f82189ec640 1 --2- 172.21.15.17:0/27593265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8214137130 0x7f8214139510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82140a2670 tx=0x7f820c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb8121be279bbea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 1 -- 172.21.15.17:0/27593265 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c02aea0 con 0x7f8214137130 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: dump: 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 5 monclient: authenticate success, global_id 4486 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 1 -- 172.21.15.17:0/27593265 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c02d050 con 0x7f8214137130 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 1 -- 172.21.15.17:0/27593265 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c007840 con 0x7f8214137130 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f8212ffd640 10 monclient: dump: 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 4 set_mon_vals no callback set 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 10 monclient: shutdown 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 -- 172.21.15.17:0/27593265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8214137130 msgr2=0x7f8214139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 --2- 172.21.15.17:0/27593265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8214137130 0x7f8214139510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82140a2670 tx=0x7f820c009840 comp rx=0 tx=0).stop 2024-09-15T01:52:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 -- 172.21.15.17:0/27593265 shutdown_connections 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 --2- 172.21.15.17:0/27593265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8214137130 0x7f8214139510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 --2- 172.21.15.17:0/27593265 >> v2:172.21.15.17:3300/0 conn(0x7f821406a510 0x7f821409b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 -- 172.21.15.17:0/27593265 >> 172.21.15.17:0/27593265 conn(0x7f8214067ae0 msgr2=0x7f82140a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 -- 172.21.15.17:0/27593265 shutdown_connections 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.829+0000 7f821a44f640 1 -- 172.21.15.17:0/27593265 wait complete. 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.830+0000 7f821a44f640 10 monclient: build_initial_monmap 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.830+0000 7f821a44f640 10 monclient: monmap: 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.830+0000 7f821a44f640 1 Processor -- start 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.830+0000 7f821a44f640 1 -- start start 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.830+0000 7f821a44f640 10 monclient: init 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f821406a510 0x7f821411e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-a con 0x7f821406a510 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82140708a0 0x7f821411ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-b con 0x7f82140708a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f8213fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f821406a510 msgr2=0x7f821411e680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 0x7f821411f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f8213fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f821406a510 0x7f821411e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8214137130 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821406deb0 con 0x7f821406a510 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821406d680 con 0x7f82140708a0 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8214121fd0 con 0x7f8214137130 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f821a44f640 10 monclient(hunting): authenticate will time out at 821.828358s 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 0x7f821411f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 0x7f821411f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45332/0 (socket says 172.21.15.17:45332) 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 learned_addr learned my addr 172.21.15.17:0/2866213606 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): get_auth_request con 0x7f8214137130 auth_method 0 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82140708a0 0x7f821411ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 10 monclient(hunting): get_auth_request con 0x7f82140708a0 auth_method 0 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient(hunting): handle_auth_done global_id 34166 payload 824 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82140708a0 msgr2=0x7f821411ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82140708a0 0x7f821411ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f821406a510 msgr2=0x7f821411e680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f821406a510 0x7f821411e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 monclient: found mon.noname-c 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c009ae0 con 0x7f8214137130 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.831+0000 7f82137fe640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82140708a0 0x7f821411ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 msgr2=0x7f821411f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f82189ec640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 0x7f821411f100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f820c009f50 tx=0x7f820c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c031cd0 con 0x7f8214137130 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: dump: 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: _finish_auth 0 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 5 monclient: authenticate success, global_id 34166 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 10 monclient: _renew_subs 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8214120df0 con 0x7f8214137130 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 10 monclient: _renew_subs 2024-09-15T01:52:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8214136d60 con 0x7f8214137130 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c031e30 con 0x7f8214137130 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c03c630 con 0x7f8214137130 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f82127fc640 4 set_mon_vals no callback set 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.834+0000 7f81fbfff640 10 monclient: dump: 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f821a44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f821a44f640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 0x7f81f4004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4007c30 con 0x7f81f4002850 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f820c03c9a0 con 0x7f8214137130 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 0x7f81f4004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f81f4002850 auth_method 0 2024-09-15T01:52:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f81fbfff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82000386f0 0x7f820003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f820c06c970 con 0x7f8214137130 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 10 monclient(hunting): handle_auth_done global_id 34169 payload 824 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f8213fff640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81f4007df0 con 0x7f81f4002850 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f82137fe640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82000386f0 0x7f820003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.835+0000 7f82137fe640 10 monclient: get_auth_request con 0x7f82000386f0 auth_method 0 2024-09-15T01:52:41.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f8213fff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 msgr2=0x7f81f4004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f8213fff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 0x7f81f4004ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f82040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040050f0 con 0x7f81f4002850 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f81fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f81fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.837+0000 7f81fbfff640 10 monclient: dump: 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.855 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8204005250 con 0x7f81f4002850 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 msgr2=0x7f81f4004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 0x7f81f4004ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f82040091a0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 msgr2=0x7f81f4004ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f81fbfff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4002850 0x7f81f4004ce0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f82040091a0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.838+0000 7f82137fe640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82000386f0 0x7f820003aba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8208004fd0 tx=0x7f8208004980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.843+0000 7f821a44f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.843+0000 7f821a44f640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 0x7f81f40070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.843+0000 7f821a44f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.843+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40076e0 con 0x7f81f4006c20 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.843+0000 7f8213fff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 0x7f81f40070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f81f4006c20 auth_method 0 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 10 monclient(hunting): handle_auth_done global_id 34172 payload 824 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.844+0000 7f8213fff640 1 -- 172.21.15.17:0/2866213606 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81f40078a0 con 0x7f81f4006c20 2024-09-15T01:52:41.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f8213fff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 msgr2=0x7f81f40070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f8213fff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 0x7f81f40070a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f8204015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820401b860 con 0x7f81f4006c20 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 10 monclient: dump: 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:41.858 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4554 (secure 0 0 0) 0x7f820401b9c0 con 0x7f81f4006c20 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 msgr2=0x7f81f40070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 0x7f81f40070a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f8204015000 comp rx=0 tx=0).stop 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 msgr2=0x7f81f40070a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.846+0000 7f81fbfff640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81f4006c20 0x7f81f40070a0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f820400b480 tx=0x7f8204015000 comp rx=0 tx=0).stop 2024-09-15T01:52:41.859 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 18, 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:52:41.860 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:41.861 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:41.862 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:41.864 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:41.865 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:41.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.868 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:41.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:41.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.871 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:41.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.873 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:52:41.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:41.875 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82000386f0 msgr2=0x7f820003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82000386f0 0x7f820003aba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8208004fd0 tx=0x7f8208004980 comp rx=0 tx=0).stop 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 10 monclient: shutdown 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 msgr2=0x7f821411f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 --2- 172.21.15.17:0/2866213606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8214137130 0x7f821411f100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f820c009f50 tx=0x7f820c036000 comp rx=0 tx=0).stop 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 reap_dead start 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f82189ec640 1 -- 172.21.15.17:0/2866213606 reap_dead start 2024-09-15T01:52:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 shutdown_connections 2024-09-15T01:52:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 >> 172.21.15.17:0/2866213606 conn(0x7f8214067ae0 msgr2=0x7f821406bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 shutdown_connections 2024-09-15T01:52:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.847+0000 7f821a44f640 1 -- 172.21.15.17:0/2866213606 wait complete. 2024-09-15T01:52:41.878 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:52:41.878 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:52:41.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.972+0000 7f6ffe91d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.972+0000 7f6ffe91d640 10 monclient: build_initial_monmap 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.972+0000 7f6ffe91d640 10 monclient: monmap: 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:52:41.974251+0000 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:52:41.974251+0000 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:41.976 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:41.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 Processor -- start 2024-09-15T01:52:41.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 -- start start 2024-09-15T01:52:41.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient: init 2024-09-15T01:52:41.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:41.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 0x7f6ff809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff80a5780 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 0x7f6ff8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff8069ac0 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff806a460 con 0x7f6ff8069ac0 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff80a18b0 con 0x7f6ff80a5780 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80a5b50 con 0x7f6ff809ccf0 2024-09-15T01:52:41.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffe91d640 10 monclient(hunting): authenticate will time out at 821.970700s 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffceba640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 msgr2=0x7f6ff8069f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ffceba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 0x7f6ff8069f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45352/0 (socket says 172.21.15.17:45352) 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 1 -- 172.21.15.17:0/2648012195 learned_addr learned my addr 172.21.15.17:0/2648012195 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): get_auth_request con 0x7f6ff809ccf0 auth_method 0 2024-09-15T01:52:41.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:41.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:41.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:41.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:41.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:41.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.973+0000 7f6ff77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 0x7f6ff809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 10 monclient(hunting): handle_auth_done global_id 34175 payload 274 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 -- 172.21.15.17:0/2648012195 >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 msgr2=0x7f6ff809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 0x7f6ff809bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 -- 172.21.15.17:0/2648012195 >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 msgr2=0x7f6ff8069f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 0x7f6ff8069f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 monclient: found mon.noname-b 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff77fe640 1 -- 172.21.15.17:0/2648012195 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff809c5f0 con 0x7f6ff809ccf0 2024-09-15T01:52:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff7fff640 10 monclient: get_auth_request con 0x7f6ff80a5780 auth_method 0 2024-09-15T01:52:41.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.974+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 0x7f6ff809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:52:42.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff77fe640 1 -- 172.21.15.17:0/2648012195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 msgr2=0x7f6ff8069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff77fe640 1 --2- 172.21.15.17:0/2648012195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ff806a5b0 tx=0x7f6fec0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53f9bbeb1192d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 1 -- 172.21.15.17:0/2648012195 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec032440 con 0x7f6ff809ccf0 2024-09-15T01:52:42.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: dump: 2024-09-15T01:52:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.003 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.003 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.003 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.003 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.003 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 5 monclient: authenticate success, global_id 34175 2024-09-15T01:52:42.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:42.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 1 -- 172.21.15.17:0/2648012195 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec0325a0 con 0x7f6ff809ccf0 2024-09-15T01:52:42.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:42.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:42.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 4 set_mon_vals no callback set 2024-09-15T01:52:42.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 1 -- 172.21.15.17:0/2648012195 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec032940 con 0x7f6ff809ccf0 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ff6ffd640 10 monclient: dump: 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 10 monclient: shutdown 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 -- 172.21.15.17:0/2648012195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 msgr2=0x7f6ff8069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/2648012195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ff806a5b0 tx=0x7f6fec0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 -- 172.21.15.17:0/2648012195 shutdown_connections 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.17:3300/0 conn(0x7f6ff8069ac0 0x7f6ff8069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/2648012195 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff809ccf0 0x7f6ff8069580 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/2648012195 >> v2:172.21.15.67:3300/0 conn(0x7f6ff80a5780 0x7f6ff809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 -- 172.21.15.17:0/2648012195 >> 172.21.15.17:0/2648012195 conn(0x7f6ff8070e10 msgr2=0x7f6ff80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 -- 172.21.15.17:0/2648012195 shutdown_connections 2024-09-15T01:52:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.977+0000 7f6ffe91d640 1 -- 172.21.15.17:0/2648012195 wait complete. 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.978+0000 7f6ffe91d640 10 monclient: build_initial_monmap 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.978+0000 7f6ffe91d640 10 monclient: monmap: 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.978+0000 7f6ffe91d640 1 Processor -- start 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.978+0000 7f6ffe91d640 1 -- start start 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.978+0000 7f6ffe91d640 10 monclient: init 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8069ac0 0x7f6ff813ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff8069ac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ff809ccf0 0x7f6ff813d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 0x7f6ff8141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8122da0 con 0x7f6ff809ccf0 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81227e0 con 0x7f6ff8069ac0 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8120350 con 0x7f6ff80a5780 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff77fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ff809ccf0 msgr2=0x7f6ff813d460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffe91d640 10 monclient(hunting): authenticate will time out at 821.976455s 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ff809ccf0 0x7f6ff813d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 0x7f6ff8141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 0x7f6ff8141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:45366/0 (socket says 172.21.15.17:45366) 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 learned_addr learned my addr 172.21.15.17:0/956215035 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:52:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): get_auth_request con 0x7f6ff80a5780 auth_method 0 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8069ac0 0x7f6ff813ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff7fff640 10 monclient(hunting): get_auth_request con 0x7f6ff8069ac0 auth_method 0 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient(hunting): handle_auth_done global_id 34178 payload 824 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8069ac0 msgr2=0x7f6ff813ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8069ac0 0x7f6ff813ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ff809ccf0 msgr2=0x7f6ff813d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6ff809ccf0 0x7f6ff813d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient: _finish_hunting 0 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 monclient: found mon.noname-c 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.979+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec009450 con 0x7f6ff80a5780 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 msgr2=0x7f6ff8141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffceba640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 0x7f6ff8141320 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009fc0 tx=0x7f6ff0009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0016430 con 0x7f6ff80a5780 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:42.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: dump: 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: _finish_auth 0 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00165d0 con 0x7f6ff80a5780 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 5 monclient: authenticate success, global_id 34178 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 10 monclient: _renew_subs 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8122500 con 0x7f6ff80a5780 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0016780 con 0x7f6ff80a5780 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6fdffff640 10 monclient: dump: 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 10 monclient: _renew_subs 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ff67fc640 4 set_mon_vals no callback set 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.982+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff813db40 con 0x7f6ff80a5780 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ffe91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 0x7f6ff8141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8063b20 con 0x7f6ff8121630 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6ff0012020 con 0x7f6ff80a5780 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6fdffff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6fe4038740 0x7f6fe403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f6ff004bed0 con 0x7f6ff80a5780 2024-09-15T01:52:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 0x7f6ff8141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient: get_auth_request con 0x7f6ff8121630 auth_method 0 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff77fe640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6fe4038740 0x7f6fe403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff77fe640 10 monclient: get_auth_request con 0x7f6fe4038740 auth_method 0 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.983+0000 7f6ff7fff640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ff8122f10 con 0x7f6ff8121630 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6ff7fff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 msgr2=0x7f6ff8141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 0x7f6ff8141860 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800f560 tx=0x7f6fe8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6ff77fe640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6fe4038740 0x7f6fe403abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80a0c50 tx=0x7f6fec032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8014980 con 0x7f6ff8121630 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 10 monclient: dump: 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6fe800c030 con 0x7f6ff8121630 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 msgr2=0x7f6ff8141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 0x7f6ff8141860 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800f560 tx=0x7f6fe8008040 comp rx=0 tx=0).stop 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 msgr2=0x7f6ff8141860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.986+0000 7f6fdffff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8121630 0x7f6ff8141860 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ffe91d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 0x7f6ff8141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ffe91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8122a20 con 0x7f6ff813e710 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 0x7f6ff8141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient: get_auth_request con 0x7f6ff813e710 auth_method 0 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.992+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.993+0000 7f6ff7fff640 10 monclient(hunting): handle_auth_done global_id 4498 payload 824 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.993+0000 7f6ff7fff640 1 -- 172.21.15.17:0/956215035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6ff8122be0 con 0x7f6ff813e710 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6ff7fff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 msgr2=0x7f6ff8141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6ff7fff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 0x7f6ff8141da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6fe801a000 tx=0x7f6fe801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8022070 con 0x7f6ff813e710 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 10 monclient: dump: 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.023 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4726 (secure 0 0 0) 0x7f6fe8014770 con 0x7f6ff813e710 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 msgr2=0x7f6ff8141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 0x7f6ff8141da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6fe801a000 tx=0x7f6fe801b000 comp rx=0 tx=0).stop 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 msgr2=0x7f6ff8141da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:52:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.995+0000 7f6fdffff640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff813e710 0x7f6ff8141da0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6fe801a000 tx=0x7f6fe801b000 comp rx=0 tx=0).stop 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 18, 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:52:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:52:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:52:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:52:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:52:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:52:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:52:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:52:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:52:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:52:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:52:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:52:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:52:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:52:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:52:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:52:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:52:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:52:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:52:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:52:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:52:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:52:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:52:42.047 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:52:42.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6fe4038740 msgr2=0x7f6fe403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6fe4038740 0x7f6fe403abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80a0c50 tx=0x7f6fec032000 comp rx=0 tx=0).stop 2024-09-15T01:52:42.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 10 monclient: shutdown 2024-09-15T01:52:42.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:42.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 msgr2=0x7f6ff8141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 --2- 172.21.15.17:0/956215035 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6ff80a5780 0x7f6ff8141320 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009fc0 tx=0x7f6ff0009730 comp rx=0 tx=0).stop 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 reap_dead start 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffceba640 1 -- 172.21.15.17:0/956215035 reap_dead start 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 shutdown_connections 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 >> 172.21.15.17:0/956215035 conn(0x7f6ff8070e10 msgr2=0x7f6ff80a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:42.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 shutdown_connections 2024-09-15T01:52:42.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:52:41.996+0000 7f6ffe91d640 1 -- 172.21.15.17:0/956215035 wait complete. 2024-09-15T01:52:42.052 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:52:42.052 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:52:42.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.125+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v18) v1 ==== 152+0+0 (secure 0 0 0) 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:42.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.125+0000 7f05d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:42.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.125+0000 7f05d3fff640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-15T01:52:42.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05f279d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:42.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05f279d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:42.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05f279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:42.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05d3fff640 1 -- 172.21.15.67:0/1401237915 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v18) v1 ==== 159+0+0 (secure 0 0 0) 0x7f05ec1363e0 con 0x7f05ec09f270 2024-09-15T01:52:42.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05d3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T01:52:42.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.228+0000 7f05d3fff640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-15T01:52:42.232 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool1234' already exists 2024-09-15T01:52:42.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 msgr2=0x7f05d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 0x7f05d803abf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05dc006f40 tx=0x7f05dc008040 comp rx=0 tx=0).stop 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 10 monclient: shutdown 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 msgr2=0x7f05ec11d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05e000b5c0 tx=0x7f05e0009120 comp rx=0 tx=0).stop 2024-09-15T01:52:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 shutdown_connections 2024-09-15T01:52:42.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05d8038740 0x7f05d803abf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05ec0a0b50 0x7f05ec11dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05ec09f270 0x7f05ec11d580 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 --2- 172.21.15.67:0/1401237915 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05ec09eea0 0x7f05ec133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 >> 172.21.15.67:0/1401237915 conn(0x7f05ec071290 msgr2=0x7f05ec134b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:42.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 shutdown_connections 2024-09-15T01:52:42.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.230+0000 7f05f279d640 1 -- 172.21.15.67:0/1401237915 wait complete. 2024-09-15T01:52:42.242 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-15T01:52:42.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:42.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 10 monclient: build_initial_monmap 2024-09-15T01:52:42.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 10 monclient: monmap: 2024-09-15T01:52:42.323 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:42.322298+0000 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:42.322298+0000 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:42.324 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 1 Processor -- start 2024-09-15T01:52:42.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 1 -- start start 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.321+0000 7f83ed19d640 10 monclient: init 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 0x7f83e8132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-b con 0x7f83e8131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83e8098420 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 0x7f83e8139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-a con 0x7f83e81371a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e80a13e0 con 0x7f83e8098420 2024-09-15T01:52:42.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e80a0440 con 0x7f83e81371a0 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80a01e0 con 0x7f83e8131e70 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83ed19d640 10 monclient(hunting): authenticate will time out at 794.791281s 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e759e640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 msgr2=0x7f83e8139580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 0x7f83e8139580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83e8098420 0x7f83e8098850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51998/0 (socket says 172.21.15.67:51998) 2024-09-15T01:52:42.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e659c640 1 -- 172.21.15.67:0/3849192707 learned_addr learned my addr 172.21.15.67:0/3849192707 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:42.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 --2- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 0x7f83e8132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): get_auth_request con 0x7f83e8131e70 auth_method 0 2024-09-15T01:52:42.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient(hunting): handle_auth_done global_id 34181 payload 274 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 -- 172.21.15.67:0/3849192707 >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 msgr2=0x7f83e8139580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 --2- 172.21.15.67:0/3849192707 >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 0x7f83e8139580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 -- 172.21.15.67:0/3849192707 >> v1:172.21.15.67:6789/0 conn(0x7f83e8098420 legacy=0x7f83e8098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 monclient: found mon.noname-b 2024-09-15T01:52:42.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.322+0000 7f83e6d9d640 1 -- 172.21.15.67:0/3849192707 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e80a19b0 con 0x7f83e8131e70 2024-09-15T01:52:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.325+0000 7f83e6d9d640 1 -- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 msgr2=0x7f83e8132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.325+0000 7f83e6d9d640 1 --2- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 0x7f83e8132240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009850 tx=0x7f83d80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce82ef41b24b11c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 1 -- 172.21.15.67:0/3849192707 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8032440 con 0x7f83e8131e70 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: dump: 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.341 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: _finish_auth 0 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 1 -- 172.21.15.67:0/3849192707 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d80325a0 con 0x7f83e8131e70 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 5 monclient: authenticate success, global_id 34181 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 4 set_mon_vals no callback set 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 1 -- 172.21.15.67:0/3849192707 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8032940 con 0x7f83e8131e70 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83e5d9b640 10 monclient: dump: 2024-09-15T01:52:42.342 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 10 monclient: shutdown 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 -- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 msgr2=0x7f83e8132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 --2- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 0x7f83e8132240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009850 tx=0x7f83d80090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 -- 172.21.15.67:0/3849192707 shutdown_connections 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 --2- 172.21.15.67:0/3849192707 >> v2:172.21.15.17:3300/0 conn(0x7f83e81371a0 0x7f83e8139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 --2- 172.21.15.67:0/3849192707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e8131e70 0x7f83e8132240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 -- 172.21.15.67:0/3849192707 >> 172.21.15.67:0/3849192707 conn(0x7f83e80a5c10 msgr2=0x7f83e80a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 -- 172.21.15.67:0/3849192707 shutdown_connections 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.326+0000 7f83ed19d640 1 -- 172.21.15.67:0/3849192707 wait complete. 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.327+0000 7f83ed19d640 10 monclient: build_initial_monmap 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.327+0000 7f83ed19d640 10 monclient: monmap: 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.327+0000 7f83ed19d640 1 Processor -- start 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.327+0000 7f83ed19d640 1 -- start start 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.327+0000 7f83ed19d640 10 monclient: init 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 0x7f83e81416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-a con 0x7f83e8098420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-b con 0x7f83e8131e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e81371a0 0x7f83e8133110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): picked mon.noname-c con 0x7f83e81371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80a0d80 con 0x7f83e8098420 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80a3300 con 0x7f83e8131e70 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81369d0 con 0x7f83e81371a0 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83ed19d640 10 monclient(hunting): authenticate will time out at 794.797253s 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e6d9d640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 msgr2=0x7f83e81416f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e6d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 0x7f83e81416f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47846/0 (socket says 172.21.15.67:47846) 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 -- 172.21.15.67:0/2736473729 learned_addr learned my addr 172.21.15.67:0/2736473729 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): get_auth_request con 0x7f83e8131e70 auth_method 0 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e81371a0 0x7f83e8133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): get_auth_request con 0x7f83e81371a0 auth_method 0 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient(hunting): handle_auth_done global_id 4501 payload 824 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 -- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e81371a0 msgr2=0x7f83e8133110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e81371a0 0x7f83e8133110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 -- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 msgr2=0x7f83e81416f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 0x7f83e81416f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 monclient: found mon.noname-b 2024-09-15T01:52:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.328+0000 7f83e659c640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8009390 con 0x7f83e8131e70 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.330+0000 7f83e659c640 1 -- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 msgr2=0x7f83e812eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.330+0000 7f83e659c640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00b540 tx=0x7f83dc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc011840 con 0x7f83e8131e70 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: dump: 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc0119a0 con 0x7f83e8131e70 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 5 monclient: authenticate success, global_id 4501 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 10 monclient: _renew_subs 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e8134b50 con 0x7f83e8131e70 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc011d60 con 0x7f83e8131e70 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ceffd640 10 monclient: dump: 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:42.351 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 10 monclient: _renew_subs 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83e559a640 4 set_mon_vals no callback set 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.331+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e8141c30 con 0x7f83e8131e70 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f83dc010590 con 0x7f83e8131e70 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ed19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ed19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ed19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ceffd640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 0x7f83d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f83dc04bb00 con 0x7f83e8131e70 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83e6d9d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 0x7f83d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.332+0000 7f83e6d9d640 10 monclient: get_auth_request con 0x7f83d40386f0 auth_method 0 2024-09-15T01:52:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.334+0000 7f83e6d9d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 0x7f83d403aba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83e8071b00 tx=0x7f83d8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.336+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.336+0000 7f83ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.336+0000 7f83ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:42.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.498+0000 7f83ed19d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:42.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.498+0000 7f83ed19d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:42.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.498+0000 7f83ed19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:42.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:42.498+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:43.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.123+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v19) v1 ==== 143+0+0 (secure 0 0 0) 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:43.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.123+0000 7f83ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:43.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.123+0000 7f83ceffd640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-15T01:52:43.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.226+0000 7f83ed19d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:43.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.226+0000 7f83ed19d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:43.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.226+0000 7f83ed19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:43.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:43.226+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:44.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.128+0000 7f83ceffd640 1 -- 172.21.15.67:0/2736473729 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f83e8141f10 con 0x7f83e8131e70 2024-09-15T01:52:44.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.128+0000 7f83ceffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T01:52:44.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.128+0000 7f83ceffd640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-15T01:52:44.131 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 3 pg_num_min to 2 2024-09-15T01:52:44.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 msgr2=0x7f83d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 0x7f83d403aba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83e8071b00 tx=0x7f83d8032000 comp rx=0 tx=0).stop 2024-09-15T01:52:44.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 10 monclient: shutdown 2024-09-15T01:52:44.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 msgr2=0x7f83e812eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.130+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00b540 tx=0x7f83dc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 shutdown_connections 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d40386f0 0x7f83d403aba0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83e81371a0 0x7f83e8133110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8131e70 0x7f83e812eef0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 --2- 172.21.15.67:0/2736473729 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83e8098420 0x7f83e81416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 >> 172.21.15.67:0/2736473729 conn(0x7f83e80a5c10 msgr2=0x7f83e8068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 shutdown_connections 2024-09-15T01:52:44.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.131+0000 7f83ed19d640 1 -- 172.21.15.67:0/2736473729 wait complete. 2024-09-15T01:52:44.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-15T01:52:44.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 2 2024-09-15T01:52:44.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.224+0000 7f6e34a4f640 10 monclient: get_monmap_and_config 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.224+0000 7f6e34a4f640 10 monclient: build_initial_monmap 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.224+0000 7f6e34a4f640 10 monclient: monmap: 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:44.225501+0000 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:44.225501+0000 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:44.227 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:44.228 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:44.228 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:44.228 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.224+0000 7f6e34a4f640 1 Processor -- start 2024-09-15T01:52:44.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 -- start start 2024-09-15T01:52:44.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient: init 2024-09-15T01:52:44.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:44.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 0x7f6e30132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f6e30068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 0x7f6e30069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f6e30069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e3012e960 0x7f6e3012edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f6e3012e960 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300a14d0 con 0x7f6e30069190 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e30096c10 con 0x7f6e3012e960 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3009ff20 con 0x7f6e30068dc0 2024-09-15T01:52:44.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e34a4f640 10 monclient(hunting): authenticate will time out at 796.694722s 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e3012e960 0x7f6e3012edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2dd9b640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 msgr2=0x7f6e30069560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2dd9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 0x7f6e30069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e3012e960 0x7f6e3012edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47856/0 (socket says 172.21.15.67:47856) 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 1 -- 172.21.15.67:0/202709793 learned_addr learned my addr 172.21.15.67:0/202709793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f6e3012e960 auth_method 0 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.225+0000 7f6e2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2e59c640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 0x7f6e30132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient(hunting): handle_auth_done global_id 4504 payload 274 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 -- 172.21.15.67:0/202709793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 msgr2=0x7f6e30132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 0x7f6e30132250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 -- 172.21.15.67:0/202709793 >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 msgr2=0x7f6e30069560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 --2- 172.21.15.67:0/202709793 >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 0x7f6e30069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 monclient: found mon.noname-c 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:44.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2ed9d640 1 -- 172.21.15.67:0/202709793 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300a0180 con 0x7f6e3012e960 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2e59c640 10 monclient: get_auth_request con 0x7f6e30068dc0 auth_method 0 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.226+0000 7f6e2e59c640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 0x7f6e30132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2ed9d640 1 -- 172.21.15.67:0/202709793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3012e960 msgr2=0x7f6e3012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2ed9d640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3012e960 0x7f6e3012edc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e3006a4e0 tx=0x7f6e28009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e28b2522b3f2f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 1 -- 172.21.15.67:0/202709793 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28007610 con 0x7f6e3012e960 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: dump: 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.244 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: _finish_auth 0 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 1 -- 172.21.15.67:0/202709793 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2802d050 con 0x7f6e3012e960 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 5 monclient: authenticate success, global_id 4504 2024-09-15T01:52:44.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 1 -- 172.21.15.67:0/202709793 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e280079a0 con 0x7f6e3012e960 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 4 set_mon_vals no callback set 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e2d59a640 10 monclient: dump: 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.246 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 10 monclient: shutdown 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 -- 172.21.15.67:0/202709793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3012e960 msgr2=0x7f6e3012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3012e960 0x7f6e3012edc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e3006a4e0 tx=0x7f6e28009840 comp rx=0 tx=0).stop 2024-09-15T01:52:44.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 -- 172.21.15.67:0/202709793 shutdown_connections 2024-09-15T01:52:44.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3012e960 0x7f6e3012edc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/202709793 >> v2:172.21.15.17:3300/0 conn(0x7f6e30069190 0x7f6e30069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/202709793 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30068dc0 0x7f6e30132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 -- 172.21.15.67:0/202709793 >> 172.21.15.67:0/202709793 conn(0x7f6e300711b0 msgr2=0x7f6e300a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:44.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.229+0000 7f6e34a4f640 1 -- 172.21.15.67:0/202709793 shutdown_connections 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 1 -- 172.21.15.67:0/202709793 wait complete. 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 10 monclient: build_initial_monmap 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 10 monclient: monmap: 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:44.249 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 1 Processor -- start 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 1 -- start start 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.230+0000 7f6e34a4f640 10 monclient: init 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:44.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f6e30068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30069190 0x7f6e3012dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f6e30069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 0x7f6e3011b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f6e3012e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30141f30 con 0x7f6e3012e960 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3011e740 con 0x7f6e30068dc0 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3011e180 con 0x7f6e30069190 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e34a4f640 10 monclient(hunting): authenticate will time out at 796.700405s 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47870/0 (socket says 172.21.15.67:47870) 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2ed9d640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 msgr2=0x7f6e3011b830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:44.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 -- 172.21.15.67:0/3594515869 learned_addr learned my addr 172.21.15.67:0/3594515869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): get_auth_request con 0x7f6e30068dc0 auth_method 0 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2ed9d640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 0x7f6e3011b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:44.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30069190 0x7f6e3012dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): get_auth_request con 0x7f6e30069190 auth_method 0 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:44.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient(hunting): handle_auth_done global_id 4507 payload 824 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 -- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30069190 msgr2=0x7f6e3012dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30069190 0x7f6e3012dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 -- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 msgr2=0x7f6e3011b830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 0x7f6e3011b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 monclient: found mon.noname-b 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.231+0000 7f6e2e59c640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28009ae0 con 0x7f6e30068dc0 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e2e59c640 1 -- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 msgr2=0x7f6e3009b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e2e59c640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c880 tx=0x7f6e20008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2000ce10 con 0x7f6e30068dc0 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:44.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: dump: 2024-09-15T01:52:44.260 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.260 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.260 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.260 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.260 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:44.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:44.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e20007aa0 con 0x7f6e30068dc0 2024-09-15T01:52:44.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 5 monclient: authenticate success, global_id 4507 2024-09-15T01:52:44.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 10 monclient: _renew_subs 2024-09-15T01:52:44.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3011e460 con 0x7f6e30068dc0 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e20007e60 con 0x7f6e30068dc0 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e267fc640 10 monclient: dump: 2024-09-15T01:52:44.263 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.264 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e2cd99640 4 set_mon_vals no callback set 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 10 monclient: _renew_subs 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.234+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3009bde0 con 0x7f6e30068dc0 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6e20011600 con 0x7f6e30068dc0 2024-09-15T01:52:44.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e34a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e34a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e34a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3004bf60 con 0x7f6e30068dc0 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e267fc640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 0x7f6e1003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f6e2004bb80 con 0x7f6e30068dc0 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e2dd9b640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 0x7f6e1003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.235+0000 7f6e2dd9b640 10 monclient: get_auth_request con 0x7f6e10038720 auth_method 0 2024-09-15T01:52:44.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.237+0000 7f6e2dd9b640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 0x7f6e1003abd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6e3011d5b0 tx=0x7f6e180090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.239+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6e20017ae0 con 0x7f6e30068dc0 2024-09-15T01:52:44.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.239+0000 7f6e267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.239+0000 7f6e267fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e34a4f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e34a4f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e34a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f6e30069aa0 con 0x7f6e30068dc0 2024-09-15T01:52:44.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7f6e2001e070 con 0x7f6e30068dc0 2024-09-15T01:52:44.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.402+0000 7f6e267fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e34a4f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e34a4f640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e34a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f6e30068510 con 0x7f6e30068dc0 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e267fc640 1 -- 172.21.15.67:0/3594515869 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7f6e20017300 con 0x7f6e30068dc0 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e267fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:44.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.498+0000 7f6e267fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:52:44.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 msgr2=0x7f6e1003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 0x7f6e1003abd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6e3011d5b0 tx=0x7f6e180090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 10 monclient: shutdown 2024-09-15T01:52:44.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:44.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 msgr2=0x7f6e3009b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c880 tx=0x7f6e20008040 comp rx=0 tx=0).stop 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 shutdown_connections 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e10038720 0x7f6e1003abd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3012e960 0x7f6e3011b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e30069190 0x7f6e3012dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 --2- 172.21.15.67:0/3594515869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e30068dc0 0x7f6e3009b320 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 >> 172.21.15.67:0/3594515869 conn(0x7f6e300711b0 msgr2=0x7f6e3009e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:44.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.500+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 shutdown_connections 2024-09-15T01:52:44.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.501+0000 7f6e34a4f640 1 -- 172.21.15.67:0/3594515869 wait complete. 2024-09-15T01:52:44.512 INFO:tasks.workunit.client.0.smithi067.stdout:pg_num_min: 2 2024-09-15T01:52:44.512 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.587+0000 7f25c035d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.587+0000 7f25c035d640 10 monclient: build_initial_monmap 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.587+0000 7f25c035d640 10 monclient: monmap: 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:44.588744+0000 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:44.588744+0000 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.590 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:44.591 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:44.591 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:44.591 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:44.591 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:44.591 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 Processor -- start 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 -- start start 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient: init 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T01:52:44.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25b8131aa0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25b809f490 0x7f25b8131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-c con 0x7f25b809f490 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25b80a0a60 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b8132930 con 0x7f25b8131aa0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80a1b00 con 0x7f25b80a0a60 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.588+0000 7f25c035d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b80a03d0 con 0x7f25b809f490 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25c035d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25c035d640 10 monclient(hunting): authenticate will time out at 797.058019s 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25be0f9640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f25b8131aa0 legacy=0x7f25b809ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25b809f490 0x7f25b8131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25b809f490 0x7f25b8131e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47872/0 (socket says 172.21.15.67:47872) 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 -- 172.21.15.67:0/3175379321 learned_addr learned my addr 172.21.15.67:0/3175379321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): get_auth_request con 0x7f25b809f490 auth_method 0 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25be8fa640 1 --1- 172.21.15.67:0/3175379321 >> v1:172.21.15.67:6789/0 conn(0x7f25b80a0a60 0x7f25b8138de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52004/0 (socket says 172.21.15.67:52004) 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient(hunting): handle_auth_done global_id 4510 payload 274 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 -- 172.21.15.67:0/3175379321 >> v1:172.21.15.67:6789/0 conn(0x7f25b80a0a60 legacy=0x7f25b8138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 -- 172.21.15.67:0/3175379321 >> v1:172.21.15.17:6789/0 conn(0x7f25b8131aa0 legacy=0x7f25b809ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient: _finish_hunting 0 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 monclient: found mon.noname-c 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.589+0000 7f25bd8f8640 1 -- 172.21.15.67:0/3175379321 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80a1540 con 0x7f25b809f490 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd8f8640 1 -- 172.21.15.67:0/3175379321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 msgr2=0x7f25b8131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd8f8640 1 --2- 172.21.15.67:0/3175379321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8131e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009910 tx=0x7f25a80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c230ca9799c96df9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 1 -- 172.21.15.67:0/3175379321 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a8032440 con 0x7f25b809f490 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: dump: 2024-09-15T01:52:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: _finish_auth 0 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 1 -- 172.21.15.67:0/3175379321 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a80325a0 con 0x7f25b809f490 2024-09-15T01:52:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 5 monclient: authenticate success, global_id 4510 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 4 set_mon_vals no callback set 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 1 -- 172.21.15.67:0/3175379321 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a8032940 con 0x7f25b809f490 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25bd0f7640 10 monclient: dump: 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 10 monclient: shutdown 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 -- 172.21.15.67:0/3175379321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 msgr2=0x7f25b8131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 --2- 172.21.15.67:0/3175379321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8131e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009910 tx=0x7f25a80091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 -- 172.21.15.67:0/3175379321 shutdown_connections 2024-09-15T01:52:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 --2- 172.21.15.67:0/3175379321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8131e70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 -- 172.21.15.67:0/3175379321 >> 172.21.15.67:0/3175379321 conn(0x7f25b8071390 msgr2=0x7f25b80a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 -- 172.21.15.67:0/3175379321 shutdown_connections 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.592+0000 7f25c035d640 1 -- 172.21.15.67:0/3175379321 wait complete. 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.593+0000 7f25c035d640 10 monclient: build_initial_monmap 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.593+0000 7f25c035d640 10 monclient: monmap: 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:44.602 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:44.603 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.593+0000 7f25c035d640 1 Processor -- start 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.593+0000 7f25c035d640 1 -- start start 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.593+0000 7f25c035d640 10 monclient: init 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-b con 0x7f25b809f490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 0x7f25b8096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-c con 0x7f25b80a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 0x7f25b80974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): picked mon.noname-a con 0x7f25b8131aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8099eb0 con 0x7f25b8131aa0 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b80a41e0 con 0x7f25b809f490 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8098f70 con 0x7f25b80a0a60 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25c035d640 10 monclient(hunting): authenticate will time out at 797.063337s 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47876/0 (socket says 172.21.15.67:47876) 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 -- 172.21.15.67:0/3487496905 learned_addr learned my addr 172.21.15.67:0/3487496905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): get_auth_request con 0x7f25b809f490 auth_method 0 2024-09-15T01:52:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be8fa640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 msgr2=0x7f25b80974f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be8fa640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 0x7f25b80974f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25bd8f8640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 0x7f25b8096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 msgr2=0x7f25b8096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 0x7f25b8096fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 msgr2=0x7f25b80974f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 0x7f25b80974f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient: _finish_hunting 0 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 monclient: found mon.noname-b 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25be0f9640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a8009450 con 0x7f25b809f490 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25bd8f8640 10 monclient: get_auth_request con 0x7f25b80a0a60 auth_method 0 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.594+0000 7f25bd8f8640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 0x7f25b8096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.596+0000 7f25be0f9640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 msgr2=0x7f25b8096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.596+0000 7f25be0f9640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f25b400b630 tx=0x7f25b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4015070 con 0x7f25b809f490 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: dump: 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4003d90 con 0x7f25b809f490 2024-09-15T01:52:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4006380 con 0x7f25b809f490 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 5 monclient: authenticate success, global_id 4513 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 10 monclient: _renew_subs 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b80989d0 con 0x7f25b809f490 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25bc8f6640 4 set_mon_vals no callback set 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 10 monclient: dump: 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 10 monclient: _renew_subs 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b81311f0 con 0x7f25b809f490 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.597+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f25b4010440 con 0x7f25b809f490 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25c035d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25c035d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25c035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b8099100 con 0x7f25b809f490 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25ae7fc640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 0x7f259c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f25b404b570 con 0x7f25b809f490 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25bd8f8640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 0x7f259c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.598+0000 7f25bd8f8640 10 monclient: get_auth_request con 0x7f259c038740 auth_method 0 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.600+0000 7f25bd8f8640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 0x7f259c03abf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25a8031000 tx=0x7f25a8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.602+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f25b4018a70 con 0x7f25b809f490 2024-09-15T01:52:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.602+0000 7f25ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.602+0000 7f25ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:44.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.763+0000 7f25c035d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:44.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.763+0000 7f25c035d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:44.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.763+0000 7f25c035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:44.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:44.763+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f25b80639e0 con 0x7f25b809f490 2024-09-15T01:52:45.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.129+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v21) v1 ==== 145+0+0 (secure 0 0 0) 0x7f25b4018a70 con 0x7f25b809f490 2024-09-15T01:52:45.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.129+0000 7f25ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:45.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.129+0000 7f25ae7fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-15T01:52:45.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.227+0000 7f25c035d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:45.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.227+0000 7f25c035d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:45.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.227+0000 7f25c035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:45.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:45.227+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f25b813a500 con 0x7f25b809f490 2024-09-15T01:52:46.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.145+0000 7f25ae7fc640 1 -- 172.21.15.67:0/3487496905 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (secure 0 0 0) 0x7f25b4018810 con 0x7f25b809f490 2024-09-15T01:52:46.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.145+0000 7f25ae7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T01:52:46.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.145+0000 7f25ae7fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-15T01:52:46.148 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 3 pg_num_max to 33 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 msgr2=0x7f259c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 0x7f259c03abf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25a8031000 tx=0x7f25a8032000 comp rx=0 tx=0).stop 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 10 monclient: shutdown 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 msgr2=0x7f25b8096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f25b400b630 tx=0x7f25b40091a0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 shutdown_connections 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f259c038740 0x7f259c03abf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25b8131aa0 0x7f25b80974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25b80a0a60 0x7f25b8096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 --2- 172.21.15.67:0/3487496905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25b809f490 0x7f25b8096a70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 >> 172.21.15.67:0/3487496905 conn(0x7f25b8071390 msgr2=0x7f25b80a30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:46.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 shutdown_connections 2024-09-15T01:52:46.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.148+0000 7f25c035d640 1 -- 172.21.15.67:0/3487496905 wait complete. 2024-09-15T01:52:46.162 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-15T01:52:46.162 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 33 2024-09-15T01:52:46.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 10 monclient: get_monmap_and_config 2024-09-15T01:52:46.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 10 monclient: build_initial_monmap 2024-09-15T01:52:46.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 10 monclient: monmap: 2024-09-15T01:52:46.244 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:46.243280+0000 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:46.243280+0000 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:46.245 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 1 Processor -- start 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 1 -- start start 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.242+0000 7f4e36c3f640 10 monclient: init 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e30098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:46.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 0x7f4e30098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e300a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 0x7f4e301390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e30098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e30098ef0 con 0x7f4e30098140 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e300a1730 con 0x7f4e300a0370 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300a0b80 con 0x7f4e30098a50 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:46.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e36c3f640 10 monclient(hunting): authenticate will time out at 798.712626s 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e349db640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e30098140 0x7f4e301321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52006/0 (socket says 172.21.15.67:52006) 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e2ffff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 msgr2=0x7f4e30098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e349db640 1 -- 172.21.15.67:0/3355902342 learned_addr learned my addr 172.21.15.67:0/3355902342 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e2ffff640 1 --2- 172.21.15.67:0/3355902342 >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 0x7f4e30098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e351dc640 1 --2- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 0x7f4e301390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e351dc640 10 monclient(hunting): get_auth_request con 0x7f4e30098a50 auth_method 0 2024-09-15T01:52:46.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e351dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e351dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.243+0000 7f4e351dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient(hunting): handle_auth_done global_id 34190 payload 274 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 1 -- 172.21.15.67:0/3355902342 >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 msgr2=0x7f4e30098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 1 --2- 172.21.15.67:0/3355902342 >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 0x7f4e30098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 1 -- 172.21.15.67:0/3355902342 >> v1:172.21.15.67:6789/0 conn(0x7f4e30098140 legacy=0x7f4e301321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient: _finish_hunting 0 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 1 monclient: found mon.noname-b 2024-09-15T01:52:46.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:46.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:46.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.244+0000 7f4e351dc640 1 -- 172.21.15.67:0/3355902342 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e301328f0 con 0x7f4e30098a50 2024-09-15T01:52:46.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e351dc640 1 -- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 msgr2=0x7f4e301390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e351dc640 1 --2- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 0x7f4e301390c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009e60 tx=0x7f4e28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f546bbdc1a051d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 1 -- 172.21.15.67:0/3355902342 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28007840 con 0x7f4e30098a50 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: dump: 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.258 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 1 -- 172.21.15.67:0/3355902342 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2802d050 con 0x7f4e30098a50 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e36c3f640 5 monclient: authenticate success, global_id 34190 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e36c3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e36c3f640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e36c3f640 4 set_mon_vals no callback set 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 1 -- 172.21.15.67:0/3355902342 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28007bd0 con 0x7f4e30098a50 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.247+0000 7f4e2f7fe640 10 monclient: dump: 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 10 monclient: shutdown 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 -- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 msgr2=0x7f4e301390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 0x7f4e301390c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009e60 tx=0x7f4e28009780 comp rx=0 tx=0).stop 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 -- 172.21.15.67:0/3355902342 shutdown_connections 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/3355902342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098a50 0x7f4e301390c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/3355902342 >> v2:172.21.15.17:3300/0 conn(0x7f4e300a0370 0x7f4e30098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 -- 172.21.15.67:0/3355902342 >> 172.21.15.67:0/3355902342 conn(0x7f4e300a5780 msgr2=0x7f4e300a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:46.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 -- 172.21.15.67:0/3355902342 shutdown_connections 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 1 -- 172.21.15.67:0/3355902342 wait complete. 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 10 monclient: build_initial_monmap 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.248+0000 7f4e36c3f640 10 monclient: monmap: 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 1 Processor -- start 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 1 -- start start 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 10 monclient: init 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098140 0x7f4e301388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e30098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.249+0000 7f4e36c3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e30098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 0x7f4e30131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e300a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30099090 con 0x7f4e300a0370 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3009f3f0 con 0x7f4e30098a50 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301355c0 con 0x7f4e30098140 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e36c3f640 10 monclient(hunting): authenticate will time out at 798.719058s 2024-09-15T01:52:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e351dc640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 msgr2=0x7f4e30131320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47888/0 (socket says 172.21.15.67:47888) 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 -- 172.21.15.67:0/723989780 learned_addr learned my addr 172.21.15.67:0/723989780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): get_auth_request con 0x7f4e30098a50 auth_method 0 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e351dc640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 0x7f4e30131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098140 0x7f4e301388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): get_auth_request con 0x7f4e30098140 auth_method 0 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e349db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient(hunting): handle_auth_done global_id 4516 payload 824 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 -- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098140 msgr2=0x7f4e301388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098140 0x7f4e301388d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 -- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 msgr2=0x7f4e30131320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 0x7f4e30131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 monclient: found mon.noname-b 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.250+0000 7f4e2ffff640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009a20 con 0x7f4e30098a50 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2ffff640 1 -- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 msgr2=0x7f4e30130de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2ffff640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b610 tx=0x7f4e24006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24015070 con 0x7f4e30098a50 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 10 monclient: dump: 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 10 monclient: _finish_auth 0 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24003d80 con 0x7f4e30098a50 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e36c3f640 5 monclient: authenticate success, global_id 4516 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e36c3f640 10 monclient: _renew_subs 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e36c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.252+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e30134900 con 0x7f4e30098a50 2024-09-15T01:52:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240063c0 con 0x7f4e30098a50 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 10 monclient: dump: 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.269 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 10 monclient: _renew_subs 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e30132e40 con 0x7f4e30098a50 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2effd640 4 set_mon_vals no callback set 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4e24010440 con 0x7f4e30098a50 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.253+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3004bf60 con 0x7f4e30098a50 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.254+0000 7f4e2cff9640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 0x7f4e1c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.254+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f4e2404b510 con 0x7f4e30098a50 2024-09-15T01:52:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.254+0000 7f4e349db640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 0x7f4e1c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.254+0000 7f4e349db640 10 monclient: get_auth_request con 0x7f4e1c038720 auth_method 0 2024-09-15T01:52:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.256+0000 7f4e349db640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 0x7f4e1c03abd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e20007fd0 tx=0x7f4e20007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.258+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4e240067f0 con 0x7f4e30098a50 2024-09-15T01:52:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.258+0000 7f4e2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.258+0000 7f4e2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.424+0000 7f4e36c3f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.424+0000 7f4e36c3f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.424+0000 7f4e36c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.424+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f4e3006f000 con 0x7f4e30098a50 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.425+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7f4e240196f0 con 0x7f4e30098a50 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.425+0000 7f4e2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.425+0000 7f4e2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:52:46.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e36c3f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e36c3f640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e36c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f4e30133a10 con 0x7f4e30098a50 2024-09-15T01:52:46.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e2cff9640 1 -- 172.21.15.67:0/723989780 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7f4e24019b10 con 0x7f4e30098a50 2024-09-15T01:52:46.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e2cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:46.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.522+0000 7f4e2cff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:52:46.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 msgr2=0x7f4e1c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 0x7f4e1c03abd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e20007fd0 tx=0x7f4e20007f00 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 10 monclient: shutdown 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 msgr2=0x7f4e30130de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b610 tx=0x7f4e24006f90 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 shutdown_connections 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e1c038720 0x7f4e1c03abd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e300a0370 0x7f4e30131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e30098a50 0x7f4e30130de0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 --2- 172.21.15.67:0/723989780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e30098140 0x7f4e301388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 >> 172.21.15.67:0/723989780 conn(0x7f4e300a5780 msgr2=0x7f4e300a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 shutdown_connections 2024-09-15T01:52:46.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.524+0000 7f4e36c3f640 1 -- 172.21.15.67:0/723989780 wait complete. 2024-09-15T01:52:46.535 INFO:tasks.workunit.client.0.smithi067.stdout:pg_num_max: 33 2024-09-15T01:52:46.535 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-15T01:52:46.536 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:46.536 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.610+0000 7fb77ec34640 10 monclient: get_monmap_and_config 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.610+0000 7fb77ec34640 10 monclient: build_initial_monmap 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.610+0000 7fb77ec34640 10 monclient: monmap: 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:46.612085+0000 2024-09-15T01:52:46.613 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:46.612085+0000 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:46.614 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.611+0000 7fb77ec34640 1 Processor -- start 2024-09-15T01:52:46.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.611+0000 7fb77ec34640 1 -- start start 2024-09-15T01:52:46.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.611+0000 7fb77ec34640 10 monclient: init 2024-09-15T01:52:46.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:46.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:52:46.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb778131aa0 0x7fb778131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-c con 0x7fb778131aa0 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7780a0a60 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 0x7fb778138de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-a con 0x7fb77809f490 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb778132930 con 0x7fb7780a0a60 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780a1b00 con 0x7fb77809f490 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780a03d0 con 0x7fb778131aa0 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77ec34640 10 monclient(hunting): authenticate will time out at 799.081228s 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb778131aa0 0x7fb778131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb777fff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fb7780a0a60 legacy=0x7fb77809ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb778131aa0 0x7fb778131e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47900/0 (socket says 172.21.15.67:47900) 2024-09-15T01:52:46.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77d1d1640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 msgr2=0x7fb778138de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:46.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 -- 172.21.15.67:0/796468224 learned_addr learned my addr 172.21.15.67:0/796468224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:46.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): get_auth_request con 0x7fb778131aa0 auth_method 0 2024-09-15T01:52:46.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77d1d1640 1 --2- 172.21.15.67:0/796468224 >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 0x7fb778138de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_done global_id 4519 payload 274 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 -- 172.21.15.67:0/796468224 >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 msgr2=0x7fb778138de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 --2- 172.21.15.67:0/796468224 >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 0x7fb778138de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 -- 172.21.15.67:0/796468224 >> v1:172.21.15.17:6789/0 conn(0x7fb7780a0a60 legacy=0x7fb77809ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient: _finish_hunting 0 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 monclient: found mon.noname-c 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.612+0000 7fb77c9d0640 1 -- 172.21.15.67:0/796468224 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780a1540 con 0x7fb778131aa0 2024-09-15T01:52:46.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb77c9d0640 1 -- 172.21.15.67:0/796468224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 msgr2=0x7fb778131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:46.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb77c9d0640 1 --2- 172.21.15.67:0/796468224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778131e70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb77806a5e0 tx=0x7fb76c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e77f9ad3372558f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 1 -- 172.21.15.67:0/796468224 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c032440 con 0x7fb778131aa0 2024-09-15T01:52:46.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: dump: 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb77ec34640 5 monclient: authenticate success, global_id 4519 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 1 -- 172.21.15.67:0/796468224 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c0325a0 con 0x7fb778131aa0 2024-09-15T01:52:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb77ec34640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 1 -- 172.21.15.67:0/796468224 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c032940 con 0x7fb778131aa0 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb77ec34640 4 set_mon_vals no callback set 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.615+0000 7fb7777fe640 10 monclient: dump: 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.631 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 10 monclient: shutdown 2024-09-15T01:52:46.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:46.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 -- 172.21.15.67:0/796468224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 msgr2=0x7fb778131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 --2- 172.21.15.67:0/796468224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778131e70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb77806a5e0 tx=0x7fb76c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 -- 172.21.15.67:0/796468224 shutdown_connections 2024-09-15T01:52:46.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 --2- 172.21.15.67:0/796468224 >> v2:172.21.15.17:3300/0 conn(0x7fb77809f490 0x7fb778138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 --2- 172.21.15.67:0/796468224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778131e70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 -- 172.21.15.67:0/796468224 >> 172.21.15.67:0/796468224 conn(0x7fb778071390 msgr2=0x7fb7780a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 -- 172.21.15.67:0/796468224 shutdown_connections 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 -- 172.21.15.67:0/796468224 wait complete. 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 10 monclient: build_initial_monmap 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 10 monclient: monmap: 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:46.634 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.616+0000 7fb77ec34640 1 Processor -- start 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 -- start start 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient: init 2024-09-15T01:52:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb77809f490 0x7fb778141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-c con 0x7fb77809f490 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7780a0a60 0x7fb7781348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-a con 0x7fb7780a0a60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): picked mon.noname-b con 0x7fb778131aa0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781387a0 con 0x7fb7780a0a60 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780a0f40 con 0x7fb778131aa0 2024-09-15T01:52:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780a44c0 con 0x7fb77809f490 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77ec34640 10 monclient(hunting): authenticate will time out at 799.086621s 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb777fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7780a0a60 msgr2=0x7fb7781348e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb777fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7780a0a60 0x7fb7781348e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778134e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47908/0 (socket says 172.21.15.67:47908) 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 1 -- 172.21.15.67:0/2653429270 learned_addr learned my addr 172.21.15.67:0/2653429270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): get_auth_request con 0x7fb778131aa0 auth_method 0 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb77809f490 0x7fb778141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77d1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): get_auth_request con 0x7fb77809f490 auth_method 0 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.617+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:46.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77c9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 10 monclient(hunting): handle_auth_done global_id 4522 payload 824 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 -- 172.21.15.67:0/2653429270 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb77809f490 msgr2=0x7fb778141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb77809f490 0x7fb778141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 -- 172.21.15.67:0/2653429270 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7780a0a60 msgr2=0x7fb7781348e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7780a0a60 0x7fb7781348e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 10 monclient: _finish_hunting 0 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 monclient: found mon.noname-b 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.618+0000 7fb77d1d1640 1 -- 172.21.15.67:0/2653429270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009450 con 0x7fb778131aa0 2024-09-15T01:52:46.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77d1d1640 1 -- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 msgr2=0x7fb778134e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:46.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77d1d1640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778134e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb77000bbb0 tx=0x7fb7700097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000ccb0 con 0x7fb778131aa0 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: dump: 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.642 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: _finish_auth 0 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77000ce30 con 0x7fb778131aa0 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 5 monclient: authenticate success, global_id 4522 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 10 monclient: _renew_subs 2024-09-15T01:52:46.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 1 -- 172.21.15.67:0/2653429270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb778139b90 con 0x7fb778131aa0 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770006150 con 0x7fb778131aa0 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb774ff9640 10 monclient: dump: 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:46.644 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb776ffd640 4 set_mon_vals no callback set 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 10 monclient: _renew_subs 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.620+0000 7fb77ec34640 1 -- 172.21.15.67:0/2653429270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb778137e70 con 0x7fb778131aa0 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb770012020 con 0x7fb778131aa0 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77ec34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77ec34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:46.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77ec34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77ec34640 1 -- 172.21.15.67:0/2653429270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb778138090 con 0x7fb778131aa0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb774ff9640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb75c0386f0 0x7fb75c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fb77004bb10 con 0x7fb778131aa0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77c9d0640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb75c0386f0 0x7fb75c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.621+0000 7fb77c9d0640 10 monclient: get_auth_request con 0x7fb75c0386f0 auth_method 0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.624+0000 7fb77c9d0640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb75c0386f0 0x7fb75c03aba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb7780a3120 tx=0x7fb76c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:46.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:46.625+0000 7fb77d1d1640 1 --2- 172.21.15.67:0/2653429270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778131aa0 0x7fb778134e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb77000bbb0 tx=0x7fb7700097b0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb77001d920 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:52:47.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.454+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb77001d920 con 0x7fb778131aa0 2024-09-15T01:52:47.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.454+0000 7fb774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.454+0000 7fb774ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb77ec34640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb77ec34640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb77ec34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb77ec34640 1 -- 172.21.15.67:0/2653429270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7fb7780639e0 con 0x7fb778131aa0 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb774ff9640 1 -- 172.21.15.67:0/2653429270 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7fb77001d340 con 0x7fb778131aa0 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb774ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.622+0000 7fb774ff9640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-15T01:52:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-15T01:52:47.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:47.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-15T01:52:47.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:47.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.703+0000 7fddd8d5d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.703+0000 7fddd8d5d640 10 monclient: build_initial_monmap 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.703+0000 7fddd8d5d640 10 monclient: monmap: 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:47.704868+0000 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:47.704868+0000 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:47.707 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 1 Processor -- start 2024-09-15T01:52:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 1 -- start start 2024-09-15T01:52:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient: init 2024-09-15T01:52:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4131f90 0x7fddd4132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fddd4131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd4098260 0x7fddd4098630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fddd4098260 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.704+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddd4098b70 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd4096a70 con 0x7fddd4098b70 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd4098fd0 con 0x7fddd4098260 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd2d9d640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fddd4098b70 legacy=0x7fddd4068e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd406a6b0 con 0x7fddd4131f90 2024-09-15T01:52:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd4098260 0x7fddd4098630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd8d5d640 10 monclient(hunting): authenticate will time out at 800.174150s 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd4098260 0x7fddd4098630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47918/0 (socket says 172.21.15.67:47918) 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 -- 172.21.15.67:0/1661424337 learned_addr learned my addr 172.21.15.67:0/1661424337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): get_auth_request con 0x7fddd4098260 auth_method 0 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd259c640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4131f90 0x7fddd4132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd259c640 10 monclient(hunting): get_auth_request con 0x7fddd4131f90 auth_method 0 2024-09-15T01:52:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient(hunting): handle_auth_done global_id 4525 payload 274 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 -- 172.21.15.67:0/1661424337 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4131f90 msgr2=0x7fddd4132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4131f90 0x7fddd4132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 -- 172.21.15.67:0/1661424337 >> v1:172.21.15.17:6789/0 conn(0x7fddd4098b70 legacy=0x7fddd4068e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 monclient: found mon.noname-c 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.705+0000 7fddd1d9b640 1 -- 172.21.15.67:0/1661424337 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40a16e0 con 0x7fddd4098260 2024-09-15T01:52:47.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd1d9b640 1 -- 172.21.15.67:0/1661424337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 msgr2=0x7fddd4098630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd4098630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddd4064e70 tx=0x7fddc80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962b85e9ae1dd939 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 1 -- 172.21.15.67:0/1661424337 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc802ae90 con 0x7fddd4098260 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: dump: 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.718 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: _finish_auth 0 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 5 monclient: authenticate success, global_id 4525 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 1 -- 172.21.15.67:0/1661424337 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc8032440 con 0x7fddd4098260 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 4 set_mon_vals no callback set 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 1 -- 172.21.15.67:0/1661424337 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc80327e0 con 0x7fddd4098260 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd159a640 10 monclient: dump: 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 10 monclient: shutdown 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:47.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 -- 172.21.15.67:0/1661424337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 msgr2=0x7fddd4098630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd4098630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddd4064e70 tx=0x7fddc80091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 -- 172.21.15.67:0/1661424337 shutdown_connections 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd4098630 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 --2- 172.21.15.67:0/1661424337 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4131f90 0x7fddd4132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.708+0000 7fddd8d5d640 1 -- 172.21.15.67:0/1661424337 >> 172.21.15.67:0/1661424337 conn(0x7fddd4071690 msgr2=0x7fddd40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 1 -- 172.21.15.67:0/1661424337 shutdown_connections 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 1 -- 172.21.15.67:0/1661424337 wait complete. 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 10 monclient: build_initial_monmap 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 10 monclient: monmap: 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:47.721 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 1 Processor -- start 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 1 -- start start 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.709+0000 7fddd8d5d640 10 monclient: init 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd41413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fddd4098260 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4098b70 0x7fddd4130100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fddd4098b70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddd4131f90 0x7fddd409da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fddd4131f90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4130ea0 con 0x7fddd4131f90 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4142210 con 0x7fddd4098260 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd411dac0 con 0x7fddd4098b70 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd8d5d640 10 monclient(hunting): authenticate will time out at 800.179413s 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd41413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd41413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47920/0 (socket says 172.21.15.67:47920) 2024-09-15T01:52:47.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 -- 172.21.15.67:0/995281752 learned_addr learned my addr 172.21.15.67:0/995281752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): get_auth_request con 0x7fddd4098260 auth_method 0 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd2d9d640 1 -- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddd4131f90 msgr2=0x7fddd409da90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd2d9d640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddd4131f90 0x7fddd409da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4098b70 0x7fddd4130100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 10 monclient(hunting): get_auth_request con 0x7fddd4098b70 auth_method 0 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient(hunting): handle_auth_done global_id 4528 payload 824 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 -- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4098b70 msgr2=0x7fddd4130100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4098b70 0x7fddd4130100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 -- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddd4131f90 msgr2=0x7fddd409da90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddd4131f90 0x7fddd409da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 monclient: found mon.noname-b 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd259c640 1 -- 172.21.15.67:0/995281752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8009450 con 0x7fddd4098260 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.710+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddd4098b70 0x7fddd4130100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:52:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd259c640 1 -- 172.21.15.67:0/995281752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 msgr2=0x7fddd41413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd259c640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd4098260 0x7fddd41413a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fddc400d890 tx=0x7fddc4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc400de20 con 0x7fddd4098260 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: dump: 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc400bae0 con 0x7fddd4098260 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 5 monclient: authenticate success, global_id 4528 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 10 monclient: _renew_subs 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 1 -- 172.21.15.67:0/995281752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd411d220 con 0x7fddd4098260 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc400be80 con 0x7fddd4098260 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc27fc640 10 monclient: dump: 2024-09-15T01:52:47.727 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddc3fff640 4 set_mon_vals no callback set 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 10 monclient: _renew_subs 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.713+0000 7fddd8d5d640 1 -- 172.21.15.67:0/995281752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd4141a80 con 0x7fddd4098260 2024-09-15T01:52:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fddc4014640 con 0x7fddd4098260 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd8d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd8d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd8d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd8d5d640 1 -- 172.21.15.67:0/995281752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd40639e0 con 0x7fddd4098260 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddc27fc640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fddb40386f0 0x7fddb403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fddc404c160 con 0x7fddd4098260 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fddb40386f0 0x7fddb403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.714+0000 7fddd1d9b640 10 monclient: get_auth_request con 0x7fddb40386f0 auth_method 0 2024-09-15T01:52:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.716+0000 7fddd1d9b640 1 --2- 172.21.15.67:0/995281752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fddb40386f0 0x7fddb403aba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fddc8031000 tx=0x7fddc8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.718+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fddc4011570 con 0x7fddd4098260 2024-09-15T01:52:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.718+0000 7fddc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.718+0000 7fddc27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.879+0000 7fddd8d5d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T01:52:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.879+0000 7fddd8d5d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T01:52:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.879+0000 7fddd8d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.879+0000 7fddd8d5d640 1 -- 172.21.15.67:0/995281752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7fddd411d410 con 0x7fddd4098260 2024-09-15T01:52:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.880+0000 7fddc27fc640 1 -- 172.21.15.67:0/995281752 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7fddc401dbb0 con 0x7fddd4098260 2024-09-15T01:52:47.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.880+0000 7fddc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T01:52:47.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.880+0000 7fddc27fc640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-15T01:52:47.883 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-15T01:52:47.886 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:47.886 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-15T01:52:47.886 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:47.886 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-15T01:52:47.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.961+0000 7fb15c35d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.961+0000 7fb15c35d640 10 monclient: build_initial_monmap 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.961+0000 7fb15c35d640 10 monclient: monmap: 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:47.962584+0000 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:47.962584+0000 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:47.965 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:47.965 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 Processor -- start 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 -- start start 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient: init 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb154068dc0 0x7fb154132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb154068dc0 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb154069190 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 0x7fb154139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb154137150 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb154096c10 con 0x7fb154069190 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15406a2a0 con 0x7fb154137150 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1540a14d0 con 0x7fb154068dc0 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15c35d640 10 monclient(hunting): authenticate will time out at 800.431802s 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb154068dc0 0x7fb154132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb1598f8640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fb154069190 legacy=0x7fb154069590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a8fa640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 msgr2=0x7fb154139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb154068dc0 0x7fb154132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47924/0 (socket says 172.21.15.67:47924) 2024-09-15T01:52:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 1 -- 172.21.15.67:0/4258773326 learned_addr learned my addr 172.21.15.67:0/4258773326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 10 monclient(hunting): get_auth_request con 0x7fb154068dc0 auth_method 0 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.962+0000 7fb15a8fa640 1 --2- 172.21.15.67:0/4258773326 >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 0x7fb154139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_done global_id 4531 payload 274 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 1 -- 172.21.15.67:0/4258773326 >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 msgr2=0x7fb154139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 1 --2- 172.21.15.67:0/4258773326 >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 0x7fb154139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 1 -- 172.21.15.67:0/4258773326 >> v1:172.21.15.17:6789/0 conn(0x7fb154069190 legacy=0x7fb154069590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient: _finish_hunting 0 2024-09-15T01:52:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 1 monclient: found mon.noname-c 2024-09-15T01:52:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.963+0000 7fb15a0f9640 1 -- 172.21.15.67:0/4258773326 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15409ff20 con 0x7fb154068dc0 2024-09-15T01:52:47.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.965+0000 7fb15a0f9640 1 -- 172.21.15.67:0/4258773326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 msgr2=0x7fb154132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.965+0000 7fb15a0f9640 1 --2- 172.21.15.67:0/4258773326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb154132250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb15406a3f0 tx=0x7fb1500091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a11371d582e636b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 1 -- 172.21.15.67:0/4258773326 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150032440 con 0x7fb154068dc0 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: dump: 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.976 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: _finish_auth 0 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 1 -- 172.21.15.67:0/4258773326 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1500325a0 con 0x7fb154068dc0 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 5 monclient: authenticate success, global_id 4531 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 1 -- 172.21.15.67:0/4258773326 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150032940 con 0x7fb154068dc0 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 4 set_mon_vals no callback set 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb1590f7640 10 monclient: dump: 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 10 monclient: shutdown 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 -- 172.21.15.67:0/4258773326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 msgr2=0x7fb154132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 --2- 172.21.15.67:0/4258773326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb154132250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb15406a3f0 tx=0x7fb1500091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 -- 172.21.15.67:0/4258773326 shutdown_connections 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 --2- 172.21.15.67:0/4258773326 >> v2:172.21.15.17:3300/0 conn(0x7fb154137150 0x7fb154139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 --2- 172.21.15.67:0/4258773326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb154132250 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 -- 172.21.15.67:0/4258773326 >> 172.21.15.67:0/4258773326 conn(0x7fb1540711b0 msgr2=0x7fb1540a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 -- 172.21.15.67:0/4258773326 shutdown_connections 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.966+0000 7fb15c35d640 1 -- 172.21.15.67:0/4258773326 wait complete. 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient: build_initial_monmap 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient: monmap: 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:47.980 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 Processor -- start 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 -- start start 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient: init 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb15412e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb154068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb154069190 0x7fb15412e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb154069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb154137150 0x7fb154097fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb154137150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15412f2b0 con 0x7fb154069190 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15409b610 con 0x7fb154068dc0 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15409a4e0 con 0x7fb154137150 2024-09-15T01:52:47.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.967+0000 7fb15c35d640 10 monclient(hunting): authenticate will time out at 800.436982s 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb15412e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb15412e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47930/0 (socket says 172.21.15.67:47930) 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 -- 172.21.15.67:0/2416825331 learned_addr learned my addr 172.21.15.67:0/2416825331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): get_auth_request con 0x7fb154068dc0 auth_method 0 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb1598f8640 1 -- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb154069190 msgr2=0x7fb15412e970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb1598f8640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb154069190 0x7fb15412e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a8fa640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb154137150 0x7fb154097fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 -- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb154137150 msgr2=0x7fb154097fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb154137150 0x7fb154097fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 -- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb154069190 msgr2=0x7fb15412e970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:47.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb154069190 0x7fb15412e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient: _finish_hunting 0 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 monclient: found mon.noname-b 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.968+0000 7fb15a0f9640 1 -- 172.21.15.67:0/2416825331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150009450 con 0x7fb154068dc0 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.970+0000 7fb15a0f9640 1 -- 172.21.15.67:0/2416825331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 msgr2=0x7fb15412e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.970+0000 7fb15a0f9640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb154068dc0 0x7fb15412e430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb150009940 tx=0x7fb150032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.970+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15003ca40 con 0x7fb154068dc0 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: dump: 2024-09-15T01:52:47.985 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 5 monclient: authenticate success, global_id 4534 2024-09-15T01:52:47.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 10 monclient: _renew_subs 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15003cba0 con 0x7fb154068dc0 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 1 -- 172.21.15.67:0/2416825331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1540a0890 con 0x7fb154068dc0 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150031420 con 0x7fb154068dc0 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb14a7fc640 10 monclient: dump: 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.987 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 10 monclient: _renew_subs 2024-09-15T01:52:47.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb15c35d640 1 -- 172.21.15.67:0/2416825331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb154099b60 con 0x7fb154068dc0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.971+0000 7fb1588f6640 4 set_mon_vals no callback set 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb15c35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb15c35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb15c35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb150039040 con 0x7fb154068dc0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb15c35d640 1 -- 172.21.15.67:0/2416825331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15409a9b0 con 0x7fb154068dc0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb14a7fc640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb138038740 0x7fb13803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fb15006c2f0 con 0x7fb154068dc0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb1598f8640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb138038740 0x7fb13803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.972+0000 7fb1598f8640 10 monclient: get_auth_request con 0x7fb138038740 auth_method 0 2024-09-15T01:52:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.974+0000 7fb1598f8640 1 --2- 172.21.15.67:0/2416825331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb138038740 0x7fb13803abf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1540a26e0 tx=0x7fb144006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.976+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb150043370 con 0x7fb154068dc0 2024-09-15T01:52:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.976+0000 7fb14a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:47.976+0000 7fb14a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:48.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.139+0000 7fb15c35d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T01:52:48.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.139+0000 7fb15c35d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T01:52:48.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.139+0000 7fb15c35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.139+0000 7fb15c35d640 1 -- 172.21.15.67:0/2416825331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7fb15404bf60 con 0x7fb154068dc0 2024-09-15T01:52:48.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.140+0000 7fb14a7fc640 1 -- 172.21.15.67:0/2416825331 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v22) v1 ==== 146+0+0 (secure 0 0 0) 0x7fb150037020 con 0x7fb154068dc0 2024-09-15T01:52:48.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.140+0000 7fb14a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T01:52:48.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.140+0000 7fb14a7fc640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-15T01:52:48.143 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-15T01:52:48.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:48.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-15T01:52:48.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:48.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-15T01:52:48.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.220+0000 7fb8378e8640 10 monclient: get_monmap_and_config 2024-09-15T01:52:48.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.220+0000 7fb8378e8640 10 monclient: build_initial_monmap 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.220+0000 7fb8378e8640 10 monclient: monmap: 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:48.222078+0000 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:48.222078+0000 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:48.224 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.221+0000 7fb8378e8640 1 Processor -- start 2024-09-15T01:52:48.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.221+0000 7fb8378e8640 1 -- start start 2024-09-15T01:52:48.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.221+0000 7fb8378e8640 10 monclient: init 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb830098140 0x7fb8301321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb830098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8300a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 0x7fb8301390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb830098c20 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300a0b80 con 0x7fb8300a0370 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8300a00b0 con 0x7fb830098c20 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb830071390 con 0x7fb830098140 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb8378e8640 10 monclient(hunting): authenticate will time out at 800.691178s 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb830098140 0x7fb8301321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835e85640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 msgr2=0x7fb8301390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835e85640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 0x7fb8301390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb830098140 0x7fb8301321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47942/0 (socket says 172.21.15.67:47942) 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb834e83640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb8300a0370 0x7fb830098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52008/0 (socket says 172.21.15.67:52008) 2024-09-15T01:52:48.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 -- 172.21.15.67:0/2991360981 learned_addr learned my addr 172.21.15.67:0/2991360981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): get_auth_request con 0x7fb830098140 auth_method 0 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb827fff640 1 -- 172.21.15.67:0/2991360981 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient(hunting): handle_auth_done global_id 4537 payload 274 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 -- 172.21.15.67:0/2991360981 >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 msgr2=0x7fb8301390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 --2- 172.21.15.67:0/2991360981 >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 0x7fb8301390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 -- 172.21.15.67:0/2991360981 >> v1:172.21.15.67:6789/0 conn(0x7fb8300a0370 legacy=0x7fb830098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:48.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient: _finish_hunting 0 2024-09-15T01:52:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 monclient: found mon.noname-c 2024-09-15T01:52:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.222+0000 7fb835684640 1 -- 172.21.15.67:0/2991360981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830071500 con 0x7fb830098140 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb835684640 1 -- 172.21.15.67:0/2991360981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098140 msgr2=0x7fb8301321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb835684640 1 --2- 172.21.15.67:0/2991360981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098140 0x7fb8301321e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8300a5340 tx=0x7fb8200091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eccfb3aabd1ec6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 1 -- 172.21.15.67:0/2991360981 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820032440 con 0x7fb830098140 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: dump: 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.237 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: _finish_auth 0 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb8378e8640 5 monclient: authenticate success, global_id 4537 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 1 -- 172.21.15.67:0/2991360981 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8200325a0 con 0x7fb830098140 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb8378e8640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 1 -- 172.21.15.67:0/2991360981 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820032940 con 0x7fb830098140 2024-09-15T01:52:48.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb8378e8640 4 set_mon_vals no callback set 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.225+0000 7fb827fff640 10 monclient: dump: 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:48.239 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 10 monclient: shutdown 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 -- 172.21.15.67:0/2991360981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098140 msgr2=0x7fb8301321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 --2- 172.21.15.67:0/2991360981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098140 0x7fb8301321e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8300a5340 tx=0x7fb8200091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 -- 172.21.15.67:0/2991360981 shutdown_connections 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 --2- 172.21.15.67:0/2991360981 >> v2:172.21.15.17:3300/0 conn(0x7fb830098c20 0x7fb8301390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 --2- 172.21.15.67:0/2991360981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098140 0x7fb8301321e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 -- 172.21.15.67:0/2991360981 >> 172.21.15.67:0/2991360981 conn(0x7fb8300a5780 msgr2=0x7fb8300a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 -- 172.21.15.67:0/2991360981 shutdown_connections 2024-09-15T01:52:48.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 1 -- 172.21.15.67:0/2991360981 wait complete. 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 10 monclient: build_initial_monmap 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.226+0000 7fb8378e8640 10 monclient: monmap: 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.241 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 1 Processor -- start 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 1 -- start start 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 10 monclient: init 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb830098140 0x7fb830130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb830098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 0x7fb830132e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.227+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb830098c20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8300a0370 0x7fb830133340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb8300a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830132080 con 0x7fb8300a0370 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830133fd0 con 0x7fb830098c20 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb830099090 con 0x7fb830098140 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb8378e8640 10 monclient(hunting): authenticate will time out at 800.697037s 2024-09-15T01:52:48.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 0x7fb830132e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 0x7fb830132e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47944/0 (socket says 172.21.15.67:47944) 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 -- 172.21.15.67:0/1108264430 learned_addr learned my addr 172.21.15.67:0/1108264430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): get_auth_request con 0x7fb830098c20 auth_method 0 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835e85640 1 -- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8300a0370 msgr2=0x7fb830133340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835e85640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8300a0370 0x7fb830133340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:48.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835684640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb830098140 0x7fb830130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835684640 10 monclient(hunting): get_auth_request con 0x7fb830098140 auth_method 0 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835684640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835684640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb835684640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 -- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb830098140 msgr2=0x7fb830130dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb830098140 0x7fb830130dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 -- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8300a0370 msgr2=0x7fb830133340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8300a0370 0x7fb830133340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient: _finish_hunting 0 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 monclient: found mon.noname-b 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:48.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.228+0000 7fb834e83640 1 -- 172.21.15.67:0/1108264430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb820009450 con 0x7fb830098c20 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb834e83640 1 -- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 msgr2=0x7fb830132e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb834e83640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 0x7fb830132e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c530 tx=0x7fb82c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb834e83640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb830098c20 0x7fb830132e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c530 tx=0x7fb82c007ed0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb82c0074f0 mon_map magic: 0 v1 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0074f0 con 0x7fb830098c20 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 10 monclient: dump: 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.246 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 10 monclient: _finish_auth 0 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c007670 con 0x7fb830098c20 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 5 monclient: authenticate success, global_id 4540 2024-09-15T01:52:48.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 10 monclient: _renew_subs 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 1 -- 172.21.15.67:0/1108264430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8300a3a10 con 0x7fb830098c20 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 10 monclient: _renew_subs 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8378e8640 1 -- 172.21.15.67:0/1108264430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb830131930 con 0x7fb830098c20 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.230+0000 7fb8277fe640 4 set_mon_vals no callback set 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8378e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c005a20 con 0x7fb830098c20 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8378e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8378e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8378e8640 1 -- 172.21.15.67:0/1108264430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83004bf60 con 0x7fb830098c20 2024-09-15T01:52:48.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 10 monclient: dump: 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:48.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb82c01b3b0 con 0x7fb830098c20 2024-09-15T01:52:48.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.231+0000 7fb8257fa640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb814038740 0x7fb81403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:48.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.232+0000 7fb835684640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb814038740 0x7fb81403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:48.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.232+0000 7fb835684640 10 monclient: get_auth_request con 0x7fb814038740 auth_method 0 2024-09-15T01:52:48.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.234+0000 7fb835684640 1 --2- 172.21.15.67:0/1108264430 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb814038740 0x7fb81403abf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8300717a0 tx=0x7fb820032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:48.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.236+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fb82c04b0c0 con 0x7fb830098c20 2024-09-15T01:52:48.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.853+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb82c018650 con 0x7fb830098c20 2024-09-15T01:52:48.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.853+0000 7fb8257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:48.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:48.853+0000 7fb8257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.018+0000 7fb8378e8640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T01:52:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.018+0000 7fb8378e8640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T01:52:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.018+0000 7fb8378e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.018+0000 7fb8378e8640 1 -- 172.21.15.67:0/1108264430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7fb830132720 con 0x7fb830098c20 2024-09-15T01:52:49.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.019+0000 7fb8257fa640 1 -- 172.21.15.67:0/1108264430 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v22) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb82c00e290 con 0x7fb830098c20 2024-09-15T01:52:49.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.019+0000 7fb8257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T01:52:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.019+0000 7fb8257fa640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-15T01:52:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-15T01:52:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 10 monclient: get_monmap_and_config 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 10 monclient: build_initial_monmap 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 10 monclient: monmap: 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:49.103 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:49.102246+0000 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:49.102246+0000 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:49.104 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 1 Processor -- start 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 1 -- start start 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.101+0000 7fee49b09640 10 monclient: init 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee44098300 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 0x7fee44136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-b con 0x7fee440986d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 0x7fee44139390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-a con 0x7fee44098b00 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee44132830 con 0x7fee44098300 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee44096ba0 con 0x7fee44098b00 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4409d030 con 0x7fee440986d0 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee49b09640 10 monclient(hunting): authenticate will time out at 801.571377s 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee43fff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 msgr2=0x7fee44139390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee437fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee44098300 0x7fee44132120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52014/0 (socket says 172.21.15.67:52014) 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee437fe640 1 -- 172.21.15.67:0/3478015383 learned_addr learned my addr 172.21.15.67:0/3478015383 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee43fff640 1 --2- 172.21.15.67:0/3478015383 >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 0x7fee44139390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 1 --2- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 0x7fee44136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): get_auth_request con 0x7fee440986d0 auth_method 0 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.102+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 10 monclient(hunting): handle_auth_done global_id 34208 payload 274 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 1 -- 172.21.15.67:0/3478015383 >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 msgr2=0x7fee44139390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 1 --2- 172.21.15.67:0/3478015383 >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 0x7fee44139390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 1 -- 172.21.15.67:0/3478015383 >> v1:172.21.15.67:6789/0 conn(0x7fee44098300 legacy=0x7fee44132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 1 monclient: found mon.noname-b 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:49.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:49.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.103+0000 7fee3affd640 1 -- 172.21.15.67:0/3478015383 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4409e360 con 0x7fee440986d0 2024-09-15T01:52:49.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee3affd640 1 -- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 msgr2=0x7fee44136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:49.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee3affd640 1 --2- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 0x7fee44136e40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009850 tx=0x7fee2c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b489f574cbdab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:49.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 1 -- 172.21.15.67:0/3478015383 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c032440 con 0x7fee440986d0 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: dump: 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:49.116 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.117 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:49.117 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:49.117 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:49.117 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 1 -- 172.21.15.67:0/3478015383 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c0325a0 con 0x7fee440986d0 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 5 monclient: authenticate success, global_id 34208 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:49.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 4 set_mon_vals no callback set 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 1 -- 172.21.15.67:0/3478015383 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c032940 con 0x7fee440986d0 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee42ffd640 10 monclient: dump: 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.119 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:49.120 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 10 monclient: shutdown 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 1 -- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 msgr2=0x7fee44136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.106+0000 7fee49b09640 1 --2- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 0x7fee44136e40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009850 tx=0x7fee2c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 -- 172.21.15.67:0/3478015383 shutdown_connections 2024-09-15T01:52:49.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 --2- 172.21.15.67:0/3478015383 >> v2:172.21.15.17:3300/0 conn(0x7fee44098b00 0x7fee44139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 --2- 172.21.15.67:0/3478015383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee440986d0 0x7fee44136e40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 -- 172.21.15.67:0/3478015383 >> 172.21.15.67:0/3478015383 conn(0x7fee44070fb0 msgr2=0x7fee440a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 -- 172.21.15.67:0/3478015383 shutdown_connections 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 -- 172.21.15.67:0/3478015383 wait complete. 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 10 monclient: build_initial_monmap 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 10 monclient: monmap: 2024-09-15T01:52:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.123 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.107+0000 7fee49b09640 1 Processor -- start 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 -- start start 2024-09-15T01:52:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient: init 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee44098300 0x7fee4411c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-c con 0x7fee44098300 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-b con 0x7fee440986d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 0x7fee4411cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): picked mon.noname-a con 0x7fee44098b00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4412edf0 con 0x7fee44098b00 2024-09-15T01:52:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4411e720 con 0x7fee440986d0 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4409d720 con 0x7fee44098300 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee49b09640 10 monclient(hunting): authenticate will time out at 801.577492s 2024-09-15T01:52:49.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:49.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee43fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 msgr2=0x7fee4411cc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:49.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47960/0 (socket says 172.21.15.67:47960) 2024-09-15T01:52:49.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 -- 172.21.15.67:0/1726965392 learned_addr learned my addr 172.21.15.67:0/1726965392 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:49.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): get_auth_request con 0x7fee440986d0 auth_method 0 2024-09-15T01:52:49.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee43fff640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 0x7fee4411cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:49.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee437fe640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee44098300 0x7fee4411c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee437fe640 10 monclient(hunting): get_auth_request con 0x7fee44098300 auth_method 0 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient(hunting): handle_auth_done global_id 4543 payload 824 2024-09-15T01:52:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 -- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee44098300 msgr2=0x7fee4411c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee44098300 0x7fee4411c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 -- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 msgr2=0x7fee4411cc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 0x7fee4411cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 monclient: found mon.noname-b 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.108+0000 7fee3affd640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c009390 con 0x7fee440986d0 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3affd640 1 -- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 msgr2=0x7fee4411c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3affd640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009850 tx=0x7fee2c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c031990 con 0x7fee440986d0 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: dump: 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:49.133 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:49.135 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: _finish_auth 0 2024-09-15T01:52:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c031af0 con 0x7fee440986d0 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 5 monclient: authenticate success, global_id 4543 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 10 monclient: _renew_subs 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee440a1970 con 0x7fee440986d0 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:49.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c031e70 con 0x7fee440986d0 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee3bfff640 10 monclient: dump: 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:49.137 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:49.138 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee427fc640 4 set_mon_vals no callback set 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 10 monclient: _renew_subs 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.111+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4411dd20 con 0x7fee440986d0 2024-09-15T01:52:49.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fee2c039040 con 0x7fee440986d0 2024-09-15T01:52:49.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee49b09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:49.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee49b09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:49.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee49b09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:49.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee3bfff640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 0x7fee2403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:49.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fee2c06bbd0 con 0x7fee440986d0 2024-09-15T01:52:49.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee437fe640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 0x7fee2403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:49.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.112+0000 7fee437fe640 10 monclient: get_auth_request con 0x7fee240386f0 auth_method 0 2024-09-15T01:52:49.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.115+0000 7fee437fe640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 0x7fee2403aba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fee34007fd0 tx=0x7fee34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:49.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.116+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:49.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.116+0000 7fee3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:49.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.116+0000 7fee3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:49.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.279+0000 7fee49b09640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:49.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.279+0000 7fee49b09640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:49.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.279+0000 7fee49b09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:49.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:49.279+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:50.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.159+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v23) v1 ==== 143+0+0 (secure 0 0 0) 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:50.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.159+0000 7fee3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:50.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.159+0000 7fee3bfff640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-15T01:52:50.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.258+0000 7fee49b09640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:50.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.258+0000 7fee49b09640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:50.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.258+0000 7fee49b09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:50.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:50.258+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:51.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.159+0000 7fee3bfff640 1 -- 172.21.15.67:0/1726965392 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (secure 0 0 0) 0x7fee4411fb20 con 0x7fee440986d0 2024-09-15T01:52:51.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.159+0000 7fee3bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T01:52:51.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.159+0000 7fee3bfff640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-15T01:52:51.162 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 3 pg_num_min to 0 2024-09-15T01:52:51.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 msgr2=0x7fee2403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 0x7fee2403aba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fee34007fd0 tx=0x7fee34009040 comp rx=0 tx=0).stop 2024-09-15T01:52:51.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 10 monclient: shutdown 2024-09-15T01:52:51.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 msgr2=0x7fee4411c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.161+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009850 tx=0x7fee2c032000 comp rx=0 tx=0).stop 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 shutdown_connections 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee240386f0 0x7fee2403aba0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fee44098b00 0x7fee4411cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee440986d0 0x7fee4411c710 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 --2- 172.21.15.67:0/1726965392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fee44098300 0x7fee4411c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 >> 172.21.15.67:0/1726965392 conn(0x7fee44070fb0 msgr2=0x7fee44131110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 shutdown_connections 2024-09-15T01:52:51.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.162+0000 7fee49b09640 1 -- 172.21.15.67:0/1726965392 wait complete. 2024-09-15T01:52:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-15T01:52:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.255+0000 7f2e82f1d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.255+0000 7f2e82f1d640 10 monclient: build_initial_monmap 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.255+0000 7f2e82f1d640 10 monclient: monmap: 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.258 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:51.257066+0000 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:51.257066+0000 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:51.259 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.256+0000 7f2e82f1d640 1 Processor -- start 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.256+0000 7f2e82f1d640 1 -- start start 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.256+0000 7f2e82f1d640 10 monclient: init 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e7c068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2e7c069190 0x7f2e7c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e7c137150 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c096c10 con 0x7f2e7c068dc0 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c132960 con 0x7f2e7c137150 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a14d0 con 0x7f2e7c069190 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e82f1d640 10 monclient(hunting): authenticate will time out at 803.726250s 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e7bfff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f2e7c069190 msgr2=0x7f2e7c069590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e80cb9640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f2e7c068dc0 legacy=0x7f2e7c132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e814ba640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2e7c137150 0x7f2e7c139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47378/0 (socket says 172.21.15.67:47378) 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e814ba640 1 -- 172.21.15.67:0/1092422731 learned_addr learned my addr 172.21.15.67:0/1092422731 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:51.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.257+0000 7f2e7bfff640 1 --2- 172.21.15.67:0/1092422731 >> v2:172.21.15.17:3300/0 conn(0x7f2e7c069190 0x7f2e7c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:51.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e814ba640 1 -- 172.21.15.67:0/1092422731 >> v1:172.21.15.67:6789/0 conn(0x7f2e7c137150 legacy=0x7f2e7c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:51.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253625279 0 0) 0x7f2e7c132960 con 0x7f2e7c137150 2024-09-15T01:52:51.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 10 monclient(hunting): my global_id is 4546 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e680034e0 con 0x7f2e7c137150 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.259+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003629863 0 0) 0x7f2e680034e0 con 0x7f2e7c137150 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 >> v2:172.21.15.17:3300/0 conn(0x7f2e7c069190 msgr2=0x7f2e7c069590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 --2- 172.21.15.67:0/1092422731 >> v2:172.21.15.17:3300/0 conn(0x7f2e7c069190 0x7f2e7c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 >> v1:172.21.15.17:6789/0 conn(0x7f2e7c068dc0 legacy=0x7f2e7c132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:52:51.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c09ff20 con 0x7f2e7c137150 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f2e74001aa0 con 0x7f2e7c137150 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: dump: 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 >> v1:172.21.15.67:6789/0 conn(0x7f2e7c137150 legacy=0x7f2e7c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 0x7f2e68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): picked mon.b con 0x7f2e680039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e68004780 0x7f2e68004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): picked mon.c con 0x7f2e68004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e68005330 0x7f2e6800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): picked mon.a con 0x7f2e68005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6800c260 con 0x7f2e68005330 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6800c3d0 con 0x7f2e680039d0 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6800c590 con 0x7f2e68004780 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e7b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:52:51.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 0x7f2e68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): get_auth_request con 0x7f2e680039d0 auth_method 0 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e814ba640 1 -- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e68005330 msgr2=0x7f2e6800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e814ba640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e68005330 0x7f2e6800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient(hunting): handle_auth_done global_id 4546 payload 293 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 -- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e68004780 msgr2=0x7f2e68004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e68004780 0x7f2e68004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 -- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e68005330 msgr2=0x7f2e6800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e68005330 0x7f2e6800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient: _finish_hunting 0 2024-09-15T01:52:51.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 monclient: found mon.b 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 1 -- 172.21.15.67:0/1092422731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680034e0 con 0x7f2e680039d0 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e814ba640 1 -- 172.21.15.67:0/1092422731 reap_dead start 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 10 monclient: _finish_auth 0 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e80cb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e82f1d640 5 monclient: authenticate success, global_id 4546 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.260+0000 7f2e82f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e80cb9640 1 -- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 msgr2=0x7f2e68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 0x7f2e68003dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0099c0 tx=0x7f2e6c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5588e03dc451ea42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c003b80 con 0x7f2e680039d0 2024-09-15T01:52:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: dump: 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.279 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c003d20 con 0x7f2e680039d0 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 1 -- 172.21.15.67:0/1092422731 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c02ab80 con 0x7f2e680039d0 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 4 set_mon_vals no callback set 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e7b7fe640 10 monclient: dump: 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.280 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 10 monclient: shutdown 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 1 -- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 msgr2=0x7f2e68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.263+0000 7f2e82f1d640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 0x7f2e68003dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0099c0 tx=0x7f2e6c009060 comp rx=0 tx=0).stop 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 -- 172.21.15.67:0/1092422731 shutdown_connections 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 --2- 172.21.15.67:0/1092422731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e680039d0 0x7f2e68003dc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 -- 172.21.15.67:0/1092422731 >> 172.21.15.67:0/1092422731 conn(0x7f2e7c0711b0 msgr2=0x7f2e7c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 -- 172.21.15.67:0/1092422731 shutdown_connections 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 -- 172.21.15.67:0/1092422731 wait complete. 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 10 monclient: build_initial_monmap 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 10 monclient: monmap: 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:51.282 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.264+0000 7f2e82f1d640 1 Processor -- start 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 -- start start 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient: init 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e680039d0 0x7f2e7c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e680039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e7c1348d0 0x7f2e7c132c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c1348d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e7c1331d0 0x7f2e7c133630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c1331d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a4fe0 con 0x7f2e680039d0 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c09bfa0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a0900 con 0x7f2e7c1348d0 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e82f1d640 10 monclient(hunting): authenticate will time out at 803.734585s 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e80cb9640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e680039d0 msgr2=0x7f2e7c09ecc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e80cb9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e680039d0 0x7f2e7c09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e7c1331d0 0x7f2e7c133630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e7c1331d0 0x7f2e7c133630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48366/0 (socket says 172.21.15.67:48366) 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 -- 172.21.15.67:0/780861882 learned_addr learned my addr 172.21.15.67:0/780861882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): get_auth_request con 0x7f2e7c1331d0 auth_method 0 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e7bfff640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e7c1348d0 0x7f2e7c132c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request con 0x7f2e7c1348d0 auth_method 0 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient(hunting): handle_auth_done global_id 4549 payload 824 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 -- 172.21.15.67:0/780861882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e7c1348d0 msgr2=0x7f2e7c132c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e7c1348d0 0x7f2e7c132c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 -- 172.21.15.67:0/780861882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e680039d0 msgr2=0x7f2e7c09ecc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e680039d0 0x7f2e7c09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient: _finish_hunting 0 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 monclient: found mon.noname-b 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.265+0000 7f2e814ba640 1 -- 172.21.15.67:0/780861882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c136b90 con 0x7f2e7c1331d0 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e814ba640 1 -- 172.21.15.67:0/780861882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e7c1331d0 msgr2=0x7f2e7c133630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e814ba640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e7c1331d0 0x7f2e7c133630 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e74001c80 tx=0x7f2e7c0a00b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400cea0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: dump: 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: _finish_auth 0 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74004d30 con 0x7f2e7c1331d0 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 5 monclient: authenticate success, global_id 4549 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 10 monclient: _renew_subs 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 1 -- 172.21.15.67:0/780861882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c135830 con 0x7f2e7c1331d0 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74004ea0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e78ff9640 10 monclient: dump: 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.291 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e7affd640 4 set_mon_vals no callback set 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 10 monclient: _renew_subs 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.268+0000 7f2e82f1d640 1 -- 172.21.15.67:0/780861882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c135df0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2e740075f0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e82f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e82f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e82f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e82f1d640 1 -- 172.21.15.67:0/780861882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c133dd0 con 0x7f2e7c1331d0 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e78ff9640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e5c038740 0x7f2e5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f2e7404ad40 con 0x7f2e7c1331d0 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e5c038740 0x7f2e5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.269+0000 7f2e80cb9640 10 monclient: get_auth_request con 0x7f2e5c038740 auth_method 0 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.272+0000 7f2e80cb9640 1 --2- 172.21.15.67:0/780861882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e5c038740 0x7f2e5c03abf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c02f000 tx=0x7f2e6c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.274+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2e74010300 con 0x7f2e7c1331d0 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.274+0000 7f2e78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:51.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.274+0000 7f2e78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e82f1d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e82f1d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e82f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e82f1d640 1 -- 172.21.15.67:0/780861882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f2e7c04bf60 con 0x7f2e7c1331d0 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e78ff9640 1 -- 172.21.15.67:0/780861882 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v24) v1 ==== 158+0+0 (secure 0 0 0) 0x7f2e74019030 con 0x7f2e7c1331d0 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.435+0000 7f2e78ff9640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-15T01:52:51.438 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-15T01:52:51.441 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:51.441 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-15T01:52:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 10 monclient: build_initial_monmap 2024-09-15T01:52:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 10 monclient: monmap: 2024-09-15T01:52:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:51.518373+0000 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:51.518373+0000 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:51.520 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:51.521 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:51.521 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:51.521 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:51.521 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 1 Processor -- start 2024-09-15T01:52:51.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 1 -- start start 2024-09-15T01:52:51.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.517+0000 7efc3c39d640 10 monclient: init 2024-09-15T01:52:51.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:51.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc340a3800 0x7efc3406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-c con 0x7efc340a3800 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 0x7efc3409da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-a con 0x7efc3406be80 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc3406a670 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc340a3bd0 con 0x7efc3406a670 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3409a030 con 0x7efc3406be80 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3409f720 con 0x7efc340a3800 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3c39d640 10 monclient(hunting): authenticate will time out at 803.987530s 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc39938640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 msgr2=0x7efc3409da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc340a3800 0x7efc3406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc39938640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 0x7efc3409da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a93a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc3406a670 0x7efc3409df70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47390/0 (socket says 172.21.15.67:47390) 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a93a640 1 -- 172.21.15.67:0/959879959 learned_addr learned my addr 172.21.15.67:0/959879959 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:51.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): get_auth_request con 0x7efc340a3800 auth_method 0 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient(hunting): handle_auth_done global_id 4552 payload 274 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 1 -- 172.21.15.67:0/959879959 >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 msgr2=0x7efc3409da30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 1 --2- 172.21.15.67:0/959879959 >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 0x7efc3409da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 1 -- 172.21.15.67:0/959879959 >> v1:172.21.15.67:6789/0 conn(0x7efc3406a670 legacy=0x7efc3409df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:51.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 10 monclient: _finish_hunting 0 2024-09-15T01:52:51.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.518+0000 7efc3a139640 1 monclient: found mon.noname-c 2024-09-15T01:52:51.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.519+0000 7efc3a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:51.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.519+0000 7efc3a139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.519+0000 7efc3a139640 1 -- 172.21.15.67:0/959879959 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc340699b0 con 0x7efc340a3800 2024-09-15T01:52:51.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.521+0000 7efc3a139640 1 -- 172.21.15.67:0/959879959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 msgr2=0x7efc3406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:51.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.521+0000 7efc3a139640 1 --2- 172.21.15.67:0/959879959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3406e310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc3409eb50 tx=0x7efc300090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6048860af1c7b0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 1 -- 172.21.15.67:0/959879959 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30032440 con 0x7efc340a3800 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: dump: 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: _finish_auth 0 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 1 -- 172.21.15.67:0/959879959 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc300325a0 con 0x7efc340a3800 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 5 monclient: authenticate success, global_id 4552 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 4 set_mon_vals no callback set 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 1 -- 172.21.15.67:0/959879959 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30032940 con 0x7efc340a3800 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc39137640 10 monclient: dump: 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 10 monclient: shutdown 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:51.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 -- 172.21.15.67:0/959879959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 msgr2=0x7efc3406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 --2- 172.21.15.67:0/959879959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3406e310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc3409eb50 tx=0x7efc300090f0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 -- 172.21.15.67:0/959879959 shutdown_connections 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 --2- 172.21.15.67:0/959879959 >> v2:172.21.15.17:3300/0 conn(0x7efc3406be80 0x7efc3409da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 --2- 172.21.15.67:0/959879959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3406e310 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 -- 172.21.15.67:0/959879959 >> 172.21.15.67:0/959879959 conn(0x7efc34071090 msgr2=0x7efc340a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 -- 172.21.15.67:0/959879959 shutdown_connections 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.522+0000 7efc3c39d640 1 -- 172.21.15.67:0/959879959 wait complete. 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.523+0000 7efc3c39d640 10 monclient: build_initial_monmap 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.523+0000 7efc3c39d640 10 monclient: monmap: 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.523+0000 7efc3c39d640 1 Processor -- start 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.523+0000 7efc3c39d640 1 -- start start 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.523+0000 7efc3c39d640 10 monclient: init 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 0x7efc34141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-a con 0x7efc3406a670 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc3406be80 0x7efc34135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-c con 0x7efc3406be80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): picked mon.noname-b con 0x7efc340a3800 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc34122220 con 0x7efc3406a670 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc34142260 con 0x7efc340a3800 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3406eb00 con 0x7efc3406be80 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3c39d640 10 monclient(hunting): authenticate will time out at 803.993091s 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a139640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 msgr2=0x7efc34141360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48374/0 (socket says 172.21.15.67:48374) 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 0x7efc34141360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 -- 172.21.15.67:0/2384299079 learned_addr learned my addr 172.21.15.67:0/2384299079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): get_auth_request con 0x7efc340a3800 auth_method 0 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient(hunting): handle_auth_done global_id 4555 payload 824 2024-09-15T01:52:51.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 -- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc3406be80 msgr2=0x7efc34135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc39938640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc3406be80 0x7efc34135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc3406be80 0x7efc34135100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 -- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 msgr2=0x7efc34141360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 0x7efc34141360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 monclient: found mon.noname-b 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.524+0000 7efc3a93a640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009390 con 0x7efc340a3800 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3a93a640 1 -- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 msgr2=0x7efc3411ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3a93a640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00bbb0 tx=0x7efc2c0097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c00cc80 con 0x7efc340a3800 2024-09-15T01:52:51.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: dump: 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 5 monclient: authenticate success, global_id 4555 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c00cde0 con 0x7efc340a3800 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 10 monclient: _renew_subs 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc341207d0 con 0x7efc340a3800 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c014690 con 0x7efc340a3800 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc2a7fc640 10 monclient: dump: 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:51.542 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc38936640 4 set_mon_vals no callback set 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 10 monclient: _renew_subs 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.526+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc34120c20 con 0x7efc340a3800 2024-09-15T01:52:51.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efc2c01d3b0 con 0x7efc340a3800 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc2a7fc640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 0x7efc1803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc3c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc3c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc3c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.527+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc340639e0 con 0x7efc340a3800 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.528+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7efc2c04bab0 con 0x7efc340a3800 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.528+0000 7efc3a139640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 0x7efc1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.528+0000 7efc3a139640 10 monclient: get_auth_request con 0x7efc180386f0 auth_method 0 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.530+0000 7efc3a139640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 0x7efc1803aba0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efc3406a3c0 tx=0x7efc30032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:51.532+0000 7efc3a93a640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00bbb0 tx=0x7efc2c0097e0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7efc2c019020 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:52:52.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.361+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efc2c019020 con 0x7efc340a3800 2024-09-15T01:52:52.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.361+0000 7efc2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:52.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.361+0000 7efc2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:52.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.528+0000 7efc3c39d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:52.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.528+0000 7efc3c39d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:52.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.528+0000 7efc3c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:52.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:52.528+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7efc3409f480 con 0x7efc340a3800 2024-09-15T01:52:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.074+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v25) v1 ==== 143+0+0 (secure 0 0 0) 0x7efc2c011cf0 con 0x7efc340a3800 2024-09-15T01:52:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.074+0000 7efc2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.074+0000 7efc2a7fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-15T01:52:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.175+0000 7efc3c39d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.175+0000 7efc3c39d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.175+0000 7efc3c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.175+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7efc340a2e00 con 0x7efc340a3800 2024-09-15T01:52:54.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.525+0000 7efc297fa640 10 monclient: tick 2024-09-15T01:52:54.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.525+0000 7efc297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:54.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:54.525+0000 7efc297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.085+0000 7efc2a7fc640 1 -- 172.21.15.67:0/2384299079 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (secure 0 0 0) 0x7efc2c011710 con 0x7efc340a3800 2024-09-15T01:52:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.085+0000 7efc2a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T01:52:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.085+0000 7efc2a7fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-15T01:52:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 3 pg_num_max to 0 2024-09-15T01:52:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 msgr2=0x7efc1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 0x7efc1803aba0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efc3406a3c0 tx=0x7efc30032000 comp rx=0 tx=0).stop 2024-09-15T01:52:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 10 monclient: shutdown 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 msgr2=0x7efc3411ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00bbb0 tx=0x7efc2c0097e0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 shutdown_connections 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc180386f0 0x7efc1803aba0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc340a3800 0x7efc3411ec40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc3406be80 0x7efc34135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 --2- 172.21.15.67:0/2384299079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc3406a670 0x7efc34141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 >> 172.21.15.67:0/2384299079 conn(0x7efc34071090 msgr2=0x7efc340a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 shutdown_connections 2024-09-15T01:52:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.088+0000 7efc3c39d640 1 -- 172.21.15.67:0/2384299079 wait complete. 2024-09-15T01:52:55.100 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-15T01:52:55.100 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:55.101 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-15T01:52:55.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 10 monclient: get_monmap_and_config 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 10 monclient: build_initial_monmap 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 10 monclient: monmap: 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:55.182437+0000 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:55.182437+0000 2024-09-15T01:52:55.184 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 1 Processor -- start 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 1 -- start start 2024-09-15T01:52:55.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.182+0000 7f1100300640 10 monclient: init 2024-09-15T01:52:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T01:52:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10f8098140 0x7f10f81321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): picked mon.noname-c con 0x7f10f8098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10f80a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10f8098c20 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80a0b80 con 0x7f10f80a0370 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80a00b0 con 0x7f10f8098c20 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f8071390 con 0x7f10f8098140 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f1100300640 10 monclient(hunting): authenticate will time out at 807.651664s 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10f8098140 0x7f10f81321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fd89b640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f10f80a0370 legacy=0x7f10f8098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10f8098140 0x7f10f81321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48390/0 (socket says 172.21.15.67:48390) 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 1 -- 172.21.15.67:0/4257229438 learned_addr learned my addr 172.21.15.67:0/4257229438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): get_auth_request con 0x7f10f8098140 auth_method 0 2024-09-15T01:52:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe89d640 1 --1- 172.21.15.67:0/4257229438 >> v1:172.21.15.67:6789/0 conn(0x7f10f8098c20 0x7f10f81390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47394/0 (socket says 172.21.15.67:47394) 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.183+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 10 monclient(hunting): handle_auth_done global_id 4558 payload 274 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 1 -- 172.21.15.67:0/4257229438 >> v1:172.21.15.67:6789/0 conn(0x7f10f8098c20 legacy=0x7f10f81390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 1 -- 172.21.15.67:0/4257229438 >> v1:172.21.15.17:6789/0 conn(0x7f10f80a0370 legacy=0x7f10f8098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 1 monclient: found mon.noname-c 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.184+0000 7f10fe09c640 1 -- 172.21.15.67:0/4257229438 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80714d0 con 0x7f10f8098140 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.186+0000 7f10fe09c640 1 -- 172.21.15.67:0/4257229438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 msgr2=0x7f10f81321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.186+0000 7f10fe09c640 1 --2- 172.21.15.67:0/4257229438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81321e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009910 tx=0x7f10f40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=696042a037ecd695 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 1 -- 172.21.15.67:0/4257229438 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4032440 con 0x7f10f8098140 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: dump: 2024-09-15T01:52:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: _finish_auth 0 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 5 monclient: authenticate success, global_id 4558 2024-09-15T01:52:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 1 -- 172.21.15.67:0/4257229438 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f40325a0 con 0x7f10f8098140 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 4 set_mon_vals no callback set 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 1 -- 172.21.15.67:0/4257229438 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4032940 con 0x7f10f8098140 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f10fd09a640 10 monclient: dump: 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 10 monclient: shutdown 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 -- 172.21.15.67:0/4257229438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 msgr2=0x7f10f81321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 --2- 172.21.15.67:0/4257229438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81321e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009910 tx=0x7f10f40091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 -- 172.21.15.67:0/4257229438 shutdown_connections 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 --2- 172.21.15.67:0/4257229438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81321e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 -- 172.21.15.67:0/4257229438 >> 172.21.15.67:0/4257229438 conn(0x7f10f80a5780 msgr2=0x7f10f80a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 -- 172.21.15.67:0/4257229438 shutdown_connections 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.187+0000 7f1100300640 1 -- 172.21.15.67:0/4257229438 wait complete. 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.188+0000 7f1100300640 10 monclient: build_initial_monmap 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.188+0000 7f1100300640 10 monclient: monmap: 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.188+0000 7f1100300640 1 Processor -- start 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.188+0000 7f1100300640 1 -- start start 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.188+0000 7f1100300640 10 monclient: init 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): picked mon.noname-b con 0x7f10f8098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f10f8098c20 0x7f10f8135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): picked mon.noname-a con 0x7f10f8098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f10f80a0370 0x7f10f81356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): picked mon.noname-c con 0x7f10f80a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:55.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8139030 con 0x7f10f8098c20 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8138790 con 0x7f10f8098140 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8138a70 con 0x7f10f80a0370 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f1100300640 10 monclient(hunting): authenticate will time out at 807.657153s 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48400/0 (socket says 172.21.15.67:48400) 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 -- 172.21.15.67:0/4013503017 learned_addr learned my addr 172.21.15.67:0/4013503017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): get_auth_request con 0x7f10f8098140 auth_method 0 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:55.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fd89b640 1 -- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f10f8098c20 msgr2=0x7f10f8135180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fd89b640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f10f8098c20 0x7f10f8135180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient(hunting): handle_auth_done global_id 4561 payload 824 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 -- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f10f80a0370 msgr2=0x7f10f81356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f10f80a0370 0x7f10f81356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 -- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f10f8098c20 msgr2=0x7f10f8135180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f10f8098c20 0x7f10f8135180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient: _finish_hunting 0 2024-09-15T01:52:55.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 monclient: found mon.noname-b 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.189+0000 7f10fe09c640 1 -- 172.21.15.67:0/4013503017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009450 con 0x7f10f8098140 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10fe09c640 1 -- 172.21.15.67:0/4013503017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 msgr2=0x7f10f81396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10fe09c640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10f8098140 0x7f10f81396d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009910 tx=0x7f10f4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f403ca40 con 0x7f10f8098140 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: dump: 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.203 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f403cba0 con 0x7f10f8098140 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f1100300640 5 monclient: authenticate success, global_id 4561 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4031420 con 0x7f10f8098140 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f1100300640 10 monclient: _renew_subs 2024-09-15T01:52:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f1100300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f1100300640 1 -- 172.21.15.67:0/4013503017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f8138d50 con 0x7f10f8098140 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10fc899640 4 set_mon_vals no callback set 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.191+0000 7f10ee7fc640 10 monclient: dump: 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.192+0000 7f1100300640 10 monclient: _renew_subs 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.192+0000 7f1100300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.192+0000 7f1100300640 1 -- 172.21.15.67:0/4013503017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f8137830 con 0x7f10f8098140 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f10f4039040 con 0x7f10f8098140 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f1100300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f1100300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f1100300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f1100300640 1 -- 172.21.15.67:0/4013503017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f8136b80 con 0x7f10f8098140 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f10ee7fc640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10dc038720 0x7f10dc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f10f406be50 con 0x7f10f8098140 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f10fd89b640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10dc038720 0x7f10dc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.193+0000 7f10fd89b640 10 monclient: get_auth_request con 0x7f10dc038720 auth_method 0 2024-09-15T01:52:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.195+0000 7f10fd89b640 1 --2- 172.21.15.67:0/4013503017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10dc038720 0x7f10dc03abd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006fd0 tx=0x7f10e8006d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.197+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f10f4031850 con 0x7f10f8098140 2024-09-15T01:52:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.197+0000 7f10ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.197+0000 7f10ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:55.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f1100300640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f1100300640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f1100300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f1100300640 1 -- 172.21.15.67:0/4013503017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f10f8136b80 con 0x7f10f8098140 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f10ee7fc640 1 -- 172.21.15.67:0/4013503017 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v26) v1 ==== 158+0+0 (secure 0 0 0) 0x7f10f4040020 con 0x7f10f8098140 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f10ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.362+0000 7f10ee7fc640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-15T01:52:55.365 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-15T01:52:55.368 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:55.368 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-15T01:52:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 10 monclient: build_initial_monmap 2024-09-15T01:52:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 10 monclient: monmap: 2024-09-15T01:52:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:55.444291+0000 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:55.444291+0000 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:55.446 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 1 Processor -- start 2024-09-15T01:52:55.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 1 -- start start 2024-09-15T01:52:55.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.444+0000 7f8af1b1d640 10 monclient: init 2024-09-15T01:52:55.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 0x7f8aec132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8aec131e70 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec098420 0x7f8aec098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8aec098420 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8aec068e70 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0a13e0 con 0x7f8aec068e70 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0a0440 con 0x7f8aec131e70 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0a01e0 con 0x7f8aec098420 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:55.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8af1b1d640 10 monclient(hunting): authenticate will time out at 807.913422s 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec098420 0x7f8aec098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeb7fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 msgr2=0x7f8aec132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aebfff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f8aec068e70 legacy=0x7f8aec0692f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec098420 0x7f8aec098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48410/0 (socket says 172.21.15.67:48410) 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 -- 172.21.15.67:0/3713821218 learned_addr learned my addr 172.21.15.67:0/3713821218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): get_auth_request con 0x7f8aec098420 auth_method 0 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeb7fe640 1 --2- 172.21.15.67:0/3713821218 >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 0x7f8aec132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient(hunting): handle_auth_done global_id 4564 payload 274 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 -- 172.21.15.67:0/3713821218 >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 msgr2=0x7f8aec132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 --2- 172.21.15.67:0/3713821218 >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 0x7f8aec132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 -- 172.21.15.67:0/3713821218 >> v1:172.21.15.17:6789/0 conn(0x7f8aec068e70 legacy=0x7f8aec0692f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 monclient: found mon.noname-c 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.445+0000 7f8aeaffd640 1 -- 172.21.15.67:0/3713821218 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0a19b0 con 0x7f8aec098420 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aeaffd640 1 -- 172.21.15.67:0/3713821218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec098420 msgr2=0x7f8aec098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aeaffd640 1 --2- 172.21.15.67:0/3713821218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec098420 0x7f8aec098850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0669f0 tx=0x7f8ae00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d985de7ee190bd53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 1 -- 172.21.15.67:0/3713821218 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0032440 con 0x7f8aec098420 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: dump: 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.455 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8af1b1d640 5 monclient: authenticate success, global_id 4564 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 1 -- 172.21.15.67:0/3713821218 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae00325a0 con 0x7f8aec098420 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8af1b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:55.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8af1b1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8af1b1d640 4 set_mon_vals no callback set 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 1 -- 172.21.15.67:0/3713821218 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0032940 con 0x7f8aec098420 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.448+0000 7f8aea7fc640 10 monclient: dump: 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.457 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 10 monclient: shutdown 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 -- 172.21.15.67:0/3713821218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec098420 msgr2=0x7f8aec098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/3713821218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec098420 0x7f8aec098850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0669f0 tx=0x7f8ae00091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 -- 172.21.15.67:0/3713821218 shutdown_connections 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/3713821218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec098420 0x7f8aec098850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/3713821218 >> v2:172.21.15.17:3300/0 conn(0x7f8aec131e70 0x7f8aec132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 -- 172.21.15.67:0/3713821218 >> 172.21.15.67:0/3713821218 conn(0x7f8aec0a5c10 msgr2=0x7f8aec0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 -- 172.21.15.67:0/3713821218 shutdown_connections 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 1 -- 172.21.15.67:0/3713821218 wait complete. 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 10 monclient: build_initial_monmap 2024-09-15T01:52:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.449+0000 7f8af1b1d640 10 monclient: monmap: 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 Processor -- start 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 -- start start 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient: init 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:55.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f8aec068e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aec098420 0x7f8aec12ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8aec098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 0x7f8aec09c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8aec131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0a0d80 con 0x7f8aec131e70 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec142420 con 0x7f8aec068e70 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec09e400 con 0x7f8aec098420 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8af1b1d640 10 monclient(hunting): authenticate will time out at 807.918812s 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48414/0 (socket says 172.21.15.67:48414) 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 1 -- 172.21.15.67:0/1481725483 learned_addr learned my addr 172.21.15.67:0/1481725483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f8aec068e70 auth_method 0 2024-09-15T01:52:55.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aebfff640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 msgr2=0x7f8aec09c900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aebfff640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 0x7f8aec09c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.450+0000 7f8aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 10 monclient(hunting): handle_auth_done global_id 4567 payload 824 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aec098420 msgr2=0x7f8aec12ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aec098420 0x7f8aec12ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 msgr2=0x7f8aec09c900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 0x7f8aec09c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 monclient: found mon.noname-b 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.451+0000 7f8aeb7fe640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0009450 con 0x7f8aec068e70 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8aeb7fe640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 msgr2=0x7f8aec141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8aeb7fe640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00b600 tx=0x7f8adc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc015070 con 0x7f8aec068e70 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: dump: 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.463 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc003d90 con 0x7f8aec068e70 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 5 monclient: authenticate success, global_id 4567 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 10 monclient: _renew_subs 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec09e8f0 con 0x7f8aec068e70 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc006380 con 0x7f8aec068e70 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ad37fe640 10 monclient: dump: 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:55.465 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:55.466 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:55.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 10 monclient: _renew_subs 2024-09-15T01:52:55.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec09efb0 con 0x7f8aec068e70 2024-09-15T01:52:55.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.453+0000 7f8ae9ffb640 4 set_mon_vals no callback set 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8adc010440 con 0x7f8aec068e70 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8ad37fe640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 0x7f8ad803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8af1b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8af1b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8af1b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f8adc04b5d0 con 0x7f8aec068e70 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec0639e0 con 0x7f8aec068e70 2024-09-15T01:52:55.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8aeaffd640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 0x7f8ad803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:55.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.454+0000 7f8aeaffd640 10 monclient: get_auth_request con 0x7f8ad8038720 auth_method 0 2024-09-15T01:52:55.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.457+0000 7f8aeaffd640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 0x7f8ad803abd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0a2c50 tx=0x7f8ae0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:55.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.459+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8adc018af0 con 0x7f8aec068e70 2024-09-15T01:52:55.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.459+0000 7f8ad37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:55.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.459+0000 7f8ad37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:55.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.621+0000 7f8af1b1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:55.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.621+0000 7f8af1b1d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:55.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.621+0000 7f8af1b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:55.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:55.621+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8aec067a50 con 0x7f8aec068e70 2024-09-15T01:52:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.095+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v27) v1 ==== 176+0+0 (secure 0 0 0) 0x7f8adc018310 con 0x7f8aec068e70 2024-09-15T01:52:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.095+0000 7f8ad37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.095+0000 7f8ad37fe640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-15T01:52:56.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.193+0000 7f8af1b1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:56.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.193+0000 7f8af1b1d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:56.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.193+0000 7f8af1b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:56.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.193+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8aec06f120 con 0x7f8aec068e70 2024-09-15T01:52:56.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:56.194+0000 7f8aeb7fe640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00b600 tx=0x7f8adc0091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8adc0184f0 mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v27) v1 2024-09-15T01:52:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.022+0000 7f8ad37fe640 1 -- 172.21.15.67:0/1481725483 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v27) v1 ==== 183+0+0 (secure 0 0 0) 0x7f8adc0184f0 con 0x7f8aec068e70 2024-09-15T01:52:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.022+0000 7f8ad37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T01:52:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.022+0000 7f8ad37fe640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-15T01:52:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool1234' does not exist 2024-09-15T01:52:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 msgr2=0x7f8ad803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 0x7f8ad803abd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0a2c50 tx=0x7f8ae0032000 comp rx=0 tx=0).stop 2024-09-15T01:52:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 10 monclient: shutdown 2024-09-15T01:52:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 msgr2=0x7f8aec141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00b600 tx=0x7f8adc0091a0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 shutdown_connections 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ad8038720 0x7f8ad803abd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aec131e70 0x7f8aec09c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aec098420 0x7f8aec12ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 --2- 172.21.15.67:0/1481725483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec068e70 0x7f8aec141560 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.024+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 >> 172.21.15.67:0/1481725483 conn(0x7f8aec0a5c10 msgr2=0x7f8aec1313d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.025+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 shutdown_connections 2024-09-15T01:52:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.025+0000 7f8af1b1d640 1 -- 172.21.15.67:0/1481725483 wait complete. 2024-09-15T01:52:57.038 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-15T01:52:57.038 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:52:57.038 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.122+0000 7fb55dd5d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.122+0000 7fb55dd5d640 10 monclient: build_initial_monmap 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.122+0000 7fb55dd5d640 10 monclient: monmap: 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:57.122894+0000 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:57.122894+0000 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.124 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:57.125 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:57.125 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:57.125 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:57.125 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:57.125 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.123+0000 7fb55dd5d640 1 Processor -- start 2024-09-15T01:52:57.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.123+0000 7fb55dd5d640 1 -- start start 2024-09-15T01:52:57.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.123+0000 7fb55dd5d640 10 monclient: init 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb558068dc0 0x7fb558132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb558068dc0 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 0x7fb558069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb558069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb55812e960 0x7fb55812ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb55812e960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580a14d0 con 0x7fb558069190 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb558096c10 con 0x7fb558068dc0 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55809ff20 con 0x7fb55812e960 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb55dd5d640 10 monclient(hunting): authenticate will time out at 809.592221s 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb558068dc0 0x7fb558132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb558068dc0 0x7fb558132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48430/0 (socket says 172.21.15.67:48430) 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 -- 172.21.15.67:0/839888682 learned_addr learned my addr 172.21.15.67:0/839888682 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): get_auth_request con 0x7fb558068dc0 auth_method 0 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb54effd640 1 -- 172.21.15.67:0/839888682 >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 msgr2=0x7fb558069590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb54effd640 1 --2- 172.21.15.67:0/839888682 >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 0x7fb558069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:57.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb557fff640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb55812e960 0x7fb55812ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb557fff640 10 monclient(hunting): get_auth_request con 0x7fb55812e960 auth_method 0 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb557fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb557fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient(hunting): handle_auth_done global_id 4570 payload 274 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 -- 172.21.15.67:0/839888682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb55812e960 msgr2=0x7fb55812ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb55812e960 0x7fb55812ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 -- 172.21.15.67:0/839888682 >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 msgr2=0x7fb558069590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 --2- 172.21.15.67:0/839888682 >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 0x7fb558069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient: _finish_hunting 0 2024-09-15T01:52:57.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 monclient: found mon.noname-c 2024-09-15T01:52:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.124+0000 7fb5577fe640 1 -- 172.21.15.67:0/839888682 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580a0180 con 0x7fb558068dc0 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb5577fe640 1 -- 172.21.15.67:0/839888682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558068dc0 msgr2=0x7fb558132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb5577fe640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558068dc0 0x7fb558132250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb55806a3f0 tx=0x7fb5480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3c482bbc5b7680e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 1 -- 172.21.15.67:0/839888682 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548032440 con 0x7fb558068dc0 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: dump: 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.136 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: _finish_auth 0 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb55dd5d640 5 monclient: authenticate success, global_id 4570 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 1 -- 172.21.15.67:0/839888682 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480325a0 con 0x7fb558068dc0 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb55dd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:57.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb55dd5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb55dd5d640 4 set_mon_vals no callback set 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 1 -- 172.21.15.67:0/839888682 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548032980 con 0x7fb558068dc0 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.127+0000 7fb556ffd640 10 monclient: dump: 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.138 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 10 monclient: shutdown 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 -- 172.21.15.67:0/839888682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558068dc0 msgr2=0x7fb558132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558068dc0 0x7fb558132250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb55806a3f0 tx=0x7fb5480091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 -- 172.21.15.67:0/839888682 shutdown_connections 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb55812e960 0x7fb55812ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 --2- 172.21.15.67:0/839888682 >> v2:172.21.15.17:3300/0 conn(0x7fb558069190 0x7fb558069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 --2- 172.21.15.67:0/839888682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558068dc0 0x7fb558132250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 -- 172.21.15.67:0/839888682 >> 172.21.15.67:0/839888682 conn(0x7fb5580711b0 msgr2=0x7fb5580a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 -- 172.21.15.67:0/839888682 shutdown_connections 2024-09-15T01:52:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 1 -- 172.21.15.67:0/839888682 wait complete. 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 10 monclient: build_initial_monmap 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.128+0000 7fb55dd5d640 10 monclient: monmap: 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 Processor -- start 2024-09-15T01:52:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 -- start start 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient: init 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb558068dc0 0x7fb558141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb558068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558069190 0x7fb55812dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb558069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb55812e960 0x7fb55812e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb55812e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55809ad30 con 0x7fb55812e960 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55811c0c0 con 0x7fb558069190 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb558141d00 con 0x7fb558068dc0 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb55dd5d640 10 monclient(hunting): authenticate will time out at 809.597770s 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb557fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb55812e960 msgr2=0x7fb55812e1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558069190 0x7fb55812dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb557fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb55812e960 0x7fb55812e1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558069190 0x7fb55812dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48440/0 (socket says 172.21.15.67:48440) 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 1 -- 172.21.15.67:0/3563450958 learned_addr learned my addr 172.21.15.67:0/3563450958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 10 monclient(hunting): get_auth_request con 0x7fb558069190 auth_method 0 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb5577fe640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb558068dc0 0x7fb558141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb5577fe640 10 monclient(hunting): get_auth_request con 0x7fb558068dc0 auth_method 0 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb5577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb5577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb5577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.129+0000 7fb54effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb5577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 10 monclient(hunting): handle_auth_done global_id 4573 payload 824 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 -- 172.21.15.67:0/3563450958 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb558068dc0 msgr2=0x7fb558141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb558068dc0 0x7fb558141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 -- 172.21.15.67:0/3563450958 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb55812e960 msgr2=0x7fb55812e1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb55812e960 0x7fb55812e1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 monclient: found mon.noname-b 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.130+0000 7fb54effd640 1 -- 172.21.15.67:0/3563450958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009450 con 0x7fb558069190 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54effd640 1 -- 172.21.15.67:0/3563450958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558069190 msgr2=0x7fb55812dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54effd640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb558069190 0x7fb55812dcb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb54000b630 tx=0x7fb5400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540015070 con 0x7fb558069190 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: dump: 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: _finish_auth 0 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:57.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540003d90 con 0x7fb558069190 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540006380 con 0x7fb558069190 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb54ffff640 10 monclient: dump: 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 5 monclient: authenticate success, global_id 4573 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 10 monclient: _renew_subs 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 1 -- 172.21.15.67:0/3563450958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55811bde0 con 0x7fb558069190 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb5567fc640 4 set_mon_vals no callback set 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 10 monclient: _renew_subs 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.132+0000 7fb55dd5d640 1 -- 172.21.15.67:0/3563450958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558141960 con 0x7fb558069190 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb540010440 con 0x7fb558069190 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb55dd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb55dd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb55dd5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb55dd5d640 1 -- 172.21.15.67:0/3563450958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5580a0900 con 0x7fb558069190 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb54ffff640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb538038720 0x7fb53803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb54004b320 con 0x7fb558069190 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.133+0000 7fb5577fe640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb538038720 0x7fb53803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.134+0000 7fb5577fe640 10 monclient: get_auth_request con 0x7fb538038720 auth_method 0 2024-09-15T01:52:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.136+0000 7fb5577fe640 1 --2- 172.21.15.67:0/3563450958 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb538038720 0x7fb53803abd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb55806a680 tx=0x7fb548032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.138+0000 7fb54ffff640 1 -- 172.21.15.67:0/3563450958 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb5580a0900 con 0x7fb558069190 2024-09-15T01:52:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.138+0000 7fb54ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.138+0000 7fb54ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:57.304 INFO:tasks.workunit.client.0.smithi067.stderr:foo-rule not valid: foo-rule doesn't represent a float 2024-09-15T01:52:57.304 INFO:tasks.workunit.client.0.smithi067.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-15T01:52:57.304 INFO:tasks.workunit.client.0.smithi067.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] : create pool 2024-09-15T01:52:57.304 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: invalid command 2024-09-15T01:52:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:52:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.383+0000 7f488db9d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.383+0000 7f488db9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.384+0000 7f488db9d640 10 monclient: monmap: 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:57.384148+0000 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:57.384148+0000 2024-09-15T01:52:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:57.386 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.384+0000 7f488db9d640 1 Processor -- start 2024-09-15T01:52:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.384+0000 7f488db9d640 1 -- start start 2024-09-15T01:52:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.384+0000 7f488db9d640 10 monclient: init 2024-09-15T01:52:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4888068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4888069190 0x7f4888069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4888069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 0x7f4888139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4888137150 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4888096c10 con 0x7f4888068dc0 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48880a14d0 con 0x7f4888137150 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4888132960 con 0x7f4888069190 2024-09-15T01:52:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f488db9d640 10 monclient(hunting): authenticate will time out at 809.853248s 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4887fff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 msgr2=0x7f4888139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4888069190 0x7f4888069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4887fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 0x7f4888139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f48877fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f4888068dc0 legacy=0x7f4888132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4888069190 0x7f4888069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48446/0 (socket says 172.21.15.67:48446) 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 -- 172.21.15.67:0/2681367227 learned_addr learned my addr 172.21.15.67:0/2681367227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): get_auth_request con 0x7f4888069190 auth_method 0 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient(hunting): handle_auth_done global_id 4576 payload 274 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 -- 172.21.15.67:0/2681367227 >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 msgr2=0x7f4888139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 --2- 172.21.15.67:0/2681367227 >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 0x7f4888139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 -- 172.21.15.67:0/2681367227 >> v1:172.21.15.17:6789/0 conn(0x7f4888068dc0 legacy=0x7f4888132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 monclient: found mon.noname-c 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.385+0000 7f4886ffd640 1 -- 172.21.15.67:0/2681367227 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488809ff20 con 0x7f4888069190 2024-09-15T01:52:57.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f4886ffd640 1 -- 172.21.15.67:0/2681367227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888069190 msgr2=0x7f4888069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f4886ffd640 1 --2- 172.21.15.67:0/2681367227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888069190 0x7f4888069590 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f488806a680 tx=0x7f487c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7498f4e9f9b6163 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 1 -- 172.21.15.67:0/2681367227 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c032440 con 0x7f4888069190 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: dump: 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.397 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: _finish_auth 0 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 1 -- 172.21.15.67:0/2681367227 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c0325a0 con 0x7f4888069190 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f488db9d640 5 monclient: authenticate success, global_id 4576 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f488db9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f488db9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 1 -- 172.21.15.67:0/2681367227 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c032940 con 0x7f4888069190 2024-09-15T01:52:57.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f488db9d640 4 set_mon_vals no callback set 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.388+0000 7f48867fc640 10 monclient: dump: 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 10 monclient: shutdown 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 -- 172.21.15.67:0/2681367227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888069190 msgr2=0x7f4888069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 --2- 172.21.15.67:0/2681367227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888069190 0x7f4888069590 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f488806a680 tx=0x7f487c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 -- 172.21.15.67:0/2681367227 shutdown_connections 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 --2- 172.21.15.67:0/2681367227 >> v2:172.21.15.17:3300/0 conn(0x7f4888137150 0x7f4888139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 --2- 172.21.15.67:0/2681367227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888069190 0x7f4888069590 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 -- 172.21.15.67:0/2681367227 >> 172.21.15.67:0/2681367227 conn(0x7f48880711b0 msgr2=0x7f48880a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 -- 172.21.15.67:0/2681367227 shutdown_connections 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 1 -- 172.21.15.67:0/2681367227 wait complete. 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 10 monclient: build_initial_monmap 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.389+0000 7f488db9d640 10 monclient: monmap: 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:57.400 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 Processor -- start 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 -- start start 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient: init 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:57.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4888068dc0 0x7f488812ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4888068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 0x7f4888098030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4888069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4888137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488809c040 con 0x7f4888069190 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488809b980 con 0x7f4888137150 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488812fb60 con 0x7f4888068dc0 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:57.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.390+0000 7f488db9d640 10 monclient(hunting): authenticate will time out at 809.858944s 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4886ffd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 msgr2=0x7f4888098030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 0x7f4888098030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48458/0 (socket says 172.21.15.67:48458) 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 -- 172.21.15.67:0/1937166160 learned_addr learned my addr 172.21.15.67:0/1937166160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): get_auth_request con 0x7f4888137150 auth_method 0 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:57.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f48877fe640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4888068dc0 0x7f488812ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f48877fe640 10 monclient(hunting): get_auth_request con 0x7f4888068dc0 auth_method 0 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f48877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f48877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f48877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 -- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4888068dc0 msgr2=0x7f488812ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4888068dc0 0x7f488812ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 -- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 msgr2=0x7f4888098030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 0x7f4888098030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient: _finish_hunting 0 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 monclient: found mon.noname-b 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:57.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.391+0000 7f4887fff640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c009450 con 0x7f4888137150 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f4887fff640 1 -- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 msgr2=0x7f4888098570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f4887fff640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f488000bb30 tx=0x7f48800097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f4887fff640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f488000bb30 tx=0x7f48800097a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f4880006150 mon_map magic: 0 v1 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000cbc0 con 0x7f4888137150 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 10 monclient: dump: 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488000cd20 con 0x7f4888137150 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 5 monclient: authenticate success, global_id 4579 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f486f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 10 monclient: _renew_subs 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4888099b70 con 0x7f4888137150 2024-09-15T01:52:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f4885ffb640 4 set_mon_vals no callback set 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 10 monclient: _renew_subs 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.393+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48881379e0 con 0x7f4888137150 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880006150 con 0x7f4888137150 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 10 monclient: dump: 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:57.407 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4880012020 con 0x7f4888137150 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f488db9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f488db9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f488db9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488804bf60 con 0x7f4888137150 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.394+0000 7f486f7fe640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 0x7f487403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.395+0000 7f48877fe640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 0x7f487403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:57.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.395+0000 7f48877fe640 10 monclient: get_auth_request con 0x7f4874038740 auth_method 0 2024-09-15T01:52:57.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.397+0000 7f48877fe640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 0x7f487403abf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f48780038f0 tx=0x7f4878006ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:57.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:57.399+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f488004b900 con 0x7f4888137150 2024-09-15T01:52:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.016+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f488001dd50 con 0x7f4888137150 2024-09-15T01:52:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.016+0000 7f486f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.016+0000 7f486f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.185+0000 7f488db9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.185+0000 7f488db9d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.185+0000 7f488db9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:58.185+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f4888130080 con 0x7f4888137150 2024-09-15T01:52:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.096+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v28) v1 ==== 154+0+0 (secure 0 0 0) 0x7f4880007e60 con 0x7f4888137150 2024-09-15T01:52:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.096+0000 7f486f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.096+0000 7f486f7fe640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-15T01:52:59.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f488db9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:59.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f488db9d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:59.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f488db9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f488806f000 con 0x7f4888137150 2024-09-15T01:52:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f486f7fe640 1 -- 172.21.15.67:0/1937166160 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f488001d770 con 0x7f4888137150 2024-09-15T01:52:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f486f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T01:52:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.216+0000 7f486f7fe640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-15T01:52:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' already exists 2024-09-15T01:52:59.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 msgr2=0x7f487403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:59.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 0x7f487403abf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f48780038f0 tx=0x7f4878006ef0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 10 monclient: shutdown 2024-09-15T01:52:59.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:59.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 msgr2=0x7f4888098570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f488000bb30 tx=0x7f48800097a0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 shutdown_connections 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4874038740 0x7f487403abf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.218+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4888069190 0x7f4888098030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.219+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4888137150 0x7f4888098570 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.219+0000 7f488db9d640 1 --2- 172.21.15.67:0/1937166160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4888068dc0 0x7f488812ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.219+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 >> 172.21.15.67:0/1937166160 conn(0x7f48880711b0 msgr2=0x7f48880a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.219+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 shutdown_connections 2024-09-15T01:52:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.219+0000 7f488db9d640 1 -- 172.21.15.67:0/1937166160 wait complete. 2024-09-15T01:52:59.231 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-15T01:52:59.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.308+0000 7f1e5719d640 10 monclient: get_monmap_and_config 2024-09-15T01:52:59.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.308+0000 7f1e5719d640 10 monclient: build_initial_monmap 2024-09-15T01:52:59.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.308+0000 7f1e5719d640 10 monclient: monmap: 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:52:59.309462+0000 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:52:59.309462+0000 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:52:59.311 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 Processor -- start 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 -- start start 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient: init 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 0x7f1e50069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e500a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:52:59.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e5009eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e50132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e5009f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e500a61a0 con 0x7f1e5009eea0 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e500a1610 con 0x7f1e500a0b50 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e500a04c0 con 0x7f1e5009f2c0 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5719d640 10 monclient(hunting): authenticate will time out at 811.778633s 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e4ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e5009eea0 0x7f1e50131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47410/0 (socket says 172.21.15.67:47410) 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e4ffff640 1 -- 172.21.15.67:0/94723331 learned_addr learned my addr 172.21.15.67:0/94723331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e54f39640 1 -- 172.21.15.67:0/94723331 >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 msgr2=0x7f1e50069380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:59.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e54f39640 1 --2- 172.21.15.67:0/94723331 >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 0x7f1e50069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5573a640 1 --2- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e50132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5573a640 10 monclient(hunting): get_auth_request con 0x7f1e5009f2c0 auth_method 0 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5573a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.309+0000 7f1e5573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient(hunting): handle_auth_done global_id 34226 payload 274 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 1 -- 172.21.15.67:0/94723331 >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 msgr2=0x7f1e50069380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 1 --2- 172.21.15.67:0/94723331 >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 0x7f1e50069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 1 -- 172.21.15.67:0/94723331 >> v1:172.21.15.67:6789/0 conn(0x7f1e5009eea0 legacy=0x7f1e50131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:52:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient: _finish_hunting 0 2024-09-15T01:52:59.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 1 monclient: found mon.noname-b 2024-09-15T01:52:59.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:59.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:52:59.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.310+0000 7f1e5573a640 1 -- 172.21.15.67:0/94723331 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e500a0f20 con 0x7f1e5009f2c0 2024-09-15T01:52:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5573a640 1 -- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 msgr2=0x7f1e50132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5573a640 1 --2- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e50132220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009e60 tx=0x7f1e48009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9a2d9113e51e2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 1 -- 172.21.15.67:0/94723331 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e48007840 con 0x7f1e5009f2c0 2024-09-15T01:52:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: dump: 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:52:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 1 -- 172.21.15.67:0/94723331 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4802d050 con 0x7f1e5009f2c0 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 1 -- 172.21.15.67:0/94723331 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e48007c10 con 0x7f1e5009f2c0 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 5 monclient: authenticate success, global_id 34226 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 10 monclient: get_monmap_and_config success 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 4 set_mon_vals no callback set 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e4f7fe640 10 monclient: dump: 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.324 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 10 monclient: shutdown 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 -- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 msgr2=0x7f1e50132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 --2- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e50132220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009e60 tx=0x7f1e48009780 comp rx=0 tx=0).stop 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 -- 172.21.15.67:0/94723331 shutdown_connections 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 --2- 172.21.15.67:0/94723331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e50132220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 --2- 172.21.15.67:0/94723331 >> v2:172.21.15.17:3300/0 conn(0x7f1e500a0b50 0x7f1e50069380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.313+0000 7f1e5719d640 1 -- 172.21.15.67:0/94723331 >> 172.21.15.67:0/94723331 conn(0x7f1e50071330 msgr2=0x7f1e500a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.314+0000 7f1e5719d640 1 -- 172.21.15.67:0/94723331 shutdown_connections 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.314+0000 7f1e5719d640 1 -- 172.21.15.67:0/94723331 wait complete. 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.314+0000 7f1e5719d640 10 monclient: build_initial_monmap 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.314+0000 7f1e5719d640 10 monclient: monmap: 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:52:59.326 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.314+0000 7f1e5719d640 1 Processor -- start 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 -- start start 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient: init 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient: _reopen_session rank -1 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e5009eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e5011c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5009f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 0x7f1e5011ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e500a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e50141d40 con 0x7f1e500a0b50 2024-09-15T01:52:59.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e50120030 con 0x7f1e5009eea0 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5011fd50 con 0x7f1e5009f2c0 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): _renew_subs 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5719d640 10 monclient(hunting): authenticate will time out at 811.784537s 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5573a640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 msgr2=0x7f1e5011ce50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e5573a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 0x7f1e5011ce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48492/0 (socket says 172.21.15.67:48492) 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 -- 172.21.15.67:0/3137832818 learned_addr learned my addr 172.21.15.67:0/3137832818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): get_auth_request con 0x7f1e5009eea0 auth_method 0 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:59.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e5011c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): get_auth_request con 0x7f1e5009f2c0 auth_method 0 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient(hunting): handle_auth_done global_id 4582 payload 824 2024-09-15T01:52:59.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 -- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 msgr2=0x7f1e5011c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e5011c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 -- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 msgr2=0x7f1e5011ce50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 0x7f1e5011ce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient: _finish_hunting 0 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 monclient: found mon.noname-b 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.315+0000 7f1e54f39640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48009a20 con 0x7f1e5009eea0 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e54f39640 1 -- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 msgr2=0x7f1e50131c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e54f39640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000b600 tx=0x7f1e400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40015070 con 0x7f1e5009eea0 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:52:59.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: dump: 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: _finish_auth 0 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e40003d90 con 0x7f1e5009eea0 2024-09-15T01:52:59.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 5 monclient: authenticate success, global_id 4582 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 10 monclient: _renew_subs 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e50120310 con 0x7f1e5009eea0 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40006380 con 0x7f1e5009eea0 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4cff9640 10 monclient: dump: 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:52:59.332 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 10 monclient: _renew_subs 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e4effd640 4 set_mon_vals no callback set 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.318+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5011efe0 con 0x7f1e5009eea0 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1e40010440 con 0x7f1e5009eea0 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e5719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e5719d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e5719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5004bf60 con 0x7f1e5009eea0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e4cff9640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 0x7f1e3c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4684+0+0 (secure 0 0 0) 0x7f1e4004b4c0 con 0x7f1e5009eea0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e4ffff640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 0x7f1e3c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.319+0000 7f1e4ffff640 10 monclient: get_auth_request con 0x7f1e3c0386f0 auth_method 0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.321+0000 7f1e4ffff640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 0x7f1e3c03aba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1e50134450 tx=0x7f1e44007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.321+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.324+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1e40018910 con 0x7f1e5009eea0 2024-09-15T01:52:59.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.324+0000 7f1e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:52:59.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.324+0000 7f1e4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:52:59.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.488+0000 7f1e5719d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:52:59.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.488+0000 7f1e5719d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:52:59.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.488+0000 7f1e5719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:52:59.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:52:59.488+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f1e500a0220 con 0x7f1e5009eea0 2024-09-15T01:53:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.107+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v30) v1 ==== 188+0+0 (secure 0 0 0) 0x7f1e40018310 con 0x7f1e5009eea0 2024-09-15T01:53:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.107+0000 7f1e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:53:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.107+0000 7f1e4cff9640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-15T01:53:01.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.223+0000 7f1e5719d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:53:01.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.223+0000 7f1e5719d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:53:01.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.223+0000 7f1e5719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.223+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f1e5006f000 con 0x7f1e5009eea0 2024-09-15T01:53:01.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.224+0000 7f1e4cff9640 1 -- 172.21.15.67:0/3137832818 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v30) v1 ==== 195+0+0 (secure 0 0 0) 0x7f1e400270f0 con 0x7f1e5009eea0 2024-09-15T01:53:01.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.224+0000 7f1e4cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T01:53:01.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.224+0000 7f1e4cff9640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-15T01:53:01.227 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fooo' already exists 2024-09-15T01:53:01.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 msgr2=0x7f1e3c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:01.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 0x7f1e3c03aba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1e50134450 tx=0x7f1e44007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:01.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 10 monclient: shutdown 2024-09-15T01:53:01.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:01.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 msgr2=0x7f1e50131c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000b600 tx=0x7f1e400091a0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 shutdown_connections 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e3c0386f0 0x7f1e3c03aba0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1e500a0b50 0x7f1e5011ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1e5009f2c0 0x7f1e5011c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 --2- 172.21.15.67:0/3137832818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5009eea0 0x7f1e50131c20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 >> 172.21.15.67:0/3137832818 conn(0x7f1e50071330 msgr2=0x7f1e500a3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 shutdown_connections 2024-09-15T01:53:01.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.226+0000 7f1e5719d640 1 -- 172.21.15.67:0/3137832818 wait complete. 2024-09-15T01:53:01.238 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create foooo 123 2024-09-15T01:53:01.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.314+0000 7fe280c3f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:01.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.314+0000 7fe280c3f640 10 monclient: build_initial_monmap 2024-09-15T01:53:01.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.314+0000 7fe280c3f640 10 monclient: monmap: 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:01.315984+0000 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:01.315984+0000 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.315+0000 7fe280c3f640 1 Processor -- start 2024-09-15T01:53:01.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.315+0000 7fe280c3f640 1 -- start start 2024-09-15T01:53:01.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.315+0000 7fe280c3f640 10 monclient: init 2024-09-15T01:53:01.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe27c098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0a0370 0x7fe27c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-c con 0x7fe27c0a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098a50 0x7fe27c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fe27c098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c098ef0 con 0x7fe27c098140 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe27c0a1730 con 0x7fe27c0a0370 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c0a0b80 con 0x7fe27c098a50 2024-09-15T01:53:01.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe280c3f640 10 monclient(hunting): authenticate will time out at 813.785159s 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0a0370 0x7fe27c098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0a0370 0x7fe27c098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40952/0 (socket says 172.21.15.67:40952) 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe27a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe27c098140 0x7fe27c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57052/0 (socket says 172.21.15.67:57052) 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 -- 172.21.15.67:0/1296001479 learned_addr learned my addr 172.21.15.67:0/1296001479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): get_auth_request con 0x7fe27c0a0370 auth_method 0 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient(hunting): handle_auth_done global_id 4585 payload 274 2024-09-15T01:53:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 -- 172.21.15.67:0/1296001479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098a50 msgr2=0x7fe27c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe27ad9d640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098a50 0x7fe27c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098a50 0x7fe27c1390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 -- 172.21.15.67:0/1296001479 >> v1:172.21.15.67:6789/0 conn(0x7fe27c098140 legacy=0x7fe27c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 monclient: found mon.noname-c 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.316+0000 7fe279d9b640 1 -- 172.21.15.67:0/1296001479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c1328f0 con 0x7fe27c0a0370 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe279d9b640 1 -- 172.21.15.67:0/1296001479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 msgr2=0x7fe27c098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe279d9b640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c098510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe264009910 tx=0x7fe2640091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c8428e49f21c292 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 1 -- 172.21.15.67:0/1296001479 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264032440 con 0x7fe27c0a0370 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: dump: 2024-09-15T01:53:01.327 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: _finish_auth 0 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:01.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 1 -- 172.21.15.67:0/1296001479 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2640325a0 con 0x7fe27c0a0370 2024-09-15T01:53:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe280c3f640 5 monclient: authenticate success, global_id 4585 2024-09-15T01:53:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe280c3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe280c3f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 1 -- 172.21.15.67:0/1296001479 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264032940 con 0x7fe27c0a0370 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe280c3f640 4 set_mon_vals no callback set 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.319+0000 7fe27959a640 10 monclient: dump: 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 10 monclient: shutdown 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 -- 172.21.15.67:0/1296001479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 msgr2=0x7fe27c098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c098510 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe264009910 tx=0x7fe2640091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 -- 172.21.15.67:0/1296001479 shutdown_connections 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098a50 0x7fe27c1390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 --2- 172.21.15.67:0/1296001479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c098510 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 -- 172.21.15.67:0/1296001479 >> 172.21.15.67:0/1296001479 conn(0x7fe27c0a5780 msgr2=0x7fe27c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 -- 172.21.15.67:0/1296001479 shutdown_connections 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 1 -- 172.21.15.67:0/1296001479 wait complete. 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 10 monclient: build_initial_monmap 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.320+0000 7fe280c3f640 10 monclient: monmap: 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:01.338 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 Processor -- start 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 -- start start 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient: init 2024-09-15T01:53:01.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098140 0x7fe27c09b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-c con 0x7fe27c098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 0x7fe27c12f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-a con 0x7fe27c098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fe27c0a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c09bee0 con 0x7fe27c098a50 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c131780 con 0x7fe27c0a0370 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:01.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c099090 con 0x7fe27c098140 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe280c3f640 10 monclient(hunting): authenticate will time out at 813.790762s 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe279d9b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 msgr2=0x7fe27c12f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe279d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 0x7fe27c12f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40966/0 (socket says 172.21.15.67:40966) 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 1 -- 172.21.15.67:0/3326412161 learned_addr learned my addr 172.21.15.67:0/3326412161 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): get_auth_request con 0x7fe27c0a0370 auth_method 0 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27a59c640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098140 0x7fe27c09b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:01.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27a59c640 10 monclient(hunting): get_auth_request con 0x7fe27c098140 auth_method 0 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.321+0000 7fe27a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 10 monclient(hunting): handle_auth_done global_id 4588 payload 824 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 -- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098140 msgr2=0x7fe27c09b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098140 0x7fe27c09b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 -- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 msgr2=0x7fe27c12f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:01.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 0x7fe27c12f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 monclient: found mon.noname-b 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.322+0000 7fe27ad9d640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264009450 con 0x7fe27c0a0370 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.324+0000 7fe27ad9d640 1 -- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 msgr2=0x7fe27c12f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:01.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.324+0000 7fe27ad9d640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe27400cee0 tx=0x7fe274007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:01.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.324+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27400e610 con 0x7fe27c0a0370 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: dump: 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:01.347 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:01.347 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:01.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27400e770 con 0x7fe27c0a0370 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 5 monclient: authenticate success, global_id 4588 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 10 monclient: _renew_subs 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27400eb30 con 0x7fe27c0a0370 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27c1314a0 con 0x7fe27c0a0370 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe2727fc640 10 monclient: dump: 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe278d99640 4 set_mon_vals no callback set 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 10 monclient: _renew_subs 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.326+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27c1305b0 con 0x7fe27c0a0370 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe274009030 con 0x7fe27c0a0370 2024-09-15T01:53:01.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe280c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe280c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe280c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27c130830 con 0x7fe27c0a0370 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe2727fc640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 0x7fe25c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.327+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5126+0+0 (secure 0 0 0) 0x7fe274012070 con 0x7fe27c0a0370 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.328+0000 7fe27a59c640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 0x7fe25c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.328+0000 7fe27a59c640 10 monclient: get_auth_request con 0x7fe25c038740 auth_method 0 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.330+0000 7fe27a59c640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 0x7fe25c03abf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe26c007310 tx=0x7fe26c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.332+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe27401d910 con 0x7fe27c0a0370 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.332+0000 7fe2727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:01.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.332+0000 7fe2727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:01.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.499+0000 7fe280c3f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:01.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.499+0000 7fe280c3f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:01.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.499+0000 7fe280c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:01.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:01.499+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7fe27c04bf60 con 0x7fe27c0a0370 2024-09-15T01:53:02.053 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:53:02.053 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T01:53:02.053 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T01:53:02.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T01:53:02.056 INFO:tasks.ceph.mon.a:Started 2024-09-15T01:53:02.056 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:53:02.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.114+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe274014330 con 0x7fe27c0a0370 2024-09-15T01:53:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.114+0000 7fe2727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.114+0000 7fe2727fc640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-15T01:53:02.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.201+0000 7efea3f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:02.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.201+0000 7efea3f9d640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.201+0000 7efea3f9d640 10 monclient: monmap: 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:02.202053+0000 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:02.202053+0000 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:02.204 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:02.205 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.201+0000 7efea3f9d640 1 Processor -- start 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.202+0000 7efea3f9d640 1 -- start start 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.202+0000 7efea3f9d640 10 monclient: init 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.202+0000 7efea3f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.202+0000 7efea3f9d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe9c0a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe9c132040 0x7efe9c0987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c132040 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe9c136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0a01e0 con 0x7efe9c0a58b0 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0a1580 con 0x7efe9c136dd0 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9c0a5c80 con 0x7efe9c132040 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea3f9d640 10 monclient(hunting): authenticate will time out at 842.199204s 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1d39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7efe9c0a58b0 0x7efe9c0980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38894/0 (socket says 172.21.15.17:38894) 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1d39640 1 -- 172.21.15.17:0/3857542545 learned_addr learned my addr 172.21.15.17:0/3857542545 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:02.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 1 --2- 172.21.15.17:0/3857542545 >> v2:172.21.15.67:3300/0 conn(0x7efe9c132040 0x7efe9c0987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): get_auth_request con 0x7efe9c132040 auth_method 0 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient(hunting): handle_auth_done global_id 4591 payload 274 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 1 -- 172.21.15.17:0/3857542545 >> v1:172.21.15.67:6789/0 conn(0x7efe9c136dd0 legacy=0x7efe9c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 1 -- 172.21.15.17:0/3857542545 >> v1:172.21.15.17:6789/0 conn(0x7efe9c0a58b0 legacy=0x7efe9c0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 1 monclient: found mon.noname-c 2024-09-15T01:53:02.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.203+0000 7efea1538640 1 -- 172.21.15.17:0/3857542545 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c132440 con 0x7efe9c132040 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea1538640 1 -- 172.21.15.17:0/3857542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 msgr2=0x7efe9c0987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea1538640 1 --2- 172.21.15.17:0/3857542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c0987b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0a1710 tx=0x7efe980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31d2b6653a3f603f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 1 -- 172.21.15.17:0/3857542545 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98032440 con 0x7efe9c132040 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:02.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 10 monclient: dump: 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea3f9d640 5 monclient: authenticate success, global_id 4591 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea3f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.206+0000 7efea0d37640 1 -- 172.21.15.17:0/3857542545 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe980325a0 con 0x7efe9c132040 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea0d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 4 set_mon_vals no callback set 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea0d37640 1 -- 172.21.15.17:0/3857542545 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98032940 con 0x7efe9c132040 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea0d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea0d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea0d37640 10 monclient: dump: 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:02.218 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 10 monclient: shutdown 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 -- 172.21.15.17:0/3857542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 msgr2=0x7efe9c0987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 --2- 172.21.15.17:0/3857542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c0987b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0a1710 tx=0x7efe980090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 -- 172.21.15.17:0/3857542545 shutdown_connections 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 --2- 172.21.15.17:0/3857542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c0987b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 -- 172.21.15.17:0/3857542545 >> 172.21.15.17:0/3857542545 conn(0x7efe9c070ed0 msgr2=0x7efe9c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 -- 172.21.15.17:0/3857542545 shutdown_connections 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.207+0000 7efea3f9d640 1 -- 172.21.15.17:0/3857542545 wait complete. 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.208+0000 7efea3f9d640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.208+0000 7efea3f9d640 10 monclient: monmap: 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:02.220 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.208+0000 7efea3f9d640 1 Processor -- start 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.208+0000 7efea3f9d640 1 -- start start 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.208+0000 7efea3f9d640 10 monclient: init 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c132040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c09b7d0 con 0x7efe9c136dd0 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c135060 con 0x7efe9c132040 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0a3210 con 0x7efe9c0a58b0 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea3f9d640 10 monclient(hunting): authenticate will time out at 842.205176s 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:56612/0 (socket says 172.21.15.17:56612) 2024-09-15T01:53:02.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 -- 172.21.15.17:0/84007113 learned_addr learned my addr 172.21.15.17:0/84007113 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60708/0 (socket says 172.21.15.17:60708) 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 10 monclient(hunting): get_auth_request con 0x7efe9c136dd0 auth_method 0 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): get_auth_request con 0x7efe9c132040 auth_method 0 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): get_auth_request con 0x7efe9c0a58b0 auth_method 0 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 msgr2=0x7efe9c133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 msgr2=0x7efe9c133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea253a640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 msgr2=0x7efe9c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 msgr2=0x7efe9c133060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 monclient: found mon.noname-b 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1538640 1 -- 172.21.15.17:0/84007113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98009390 con 0x7efe9c132040 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.209+0000 7efea1d39640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.211+0000 7efea1538640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 msgr2=0x7efe9c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea1538640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c132b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0a18a0 tx=0x7efe98032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98031940 con 0x7efe9c132040 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: dump: 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe98031ac0 con 0x7efe9c132040 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 5 monclient: authenticate success, global_id 4594 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 10 monclient: _renew_subs 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c134d80 con 0x7efe9c132040 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98031e80 con 0x7efe9c132040 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe91ffb640 10 monclient: dump: 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efe93fff640 4 set_mon_vals no callback set 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 10 monclient: _renew_subs 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.212+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c133e20 con 0x7efe9c132040 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efe98039070 con 0x7efe9c132040 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea3f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea3f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea3f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c134af0 con 0x7efe9c132040 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efe91ffb640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 0x7efe8003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5470+0+0 (secure 0 0 0) 0x7efe9806c490 con 0x7efe9c132040 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea1d39640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 0x7efe8003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.213+0000 7efea1d39640 10 monclient: get_auth_request con 0x7efe80038790 auth_method 0 2024-09-15T01:53:02.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.216+0000 7efea1d39640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 0x7efe8003ac40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009e40 tx=0x7efe8c006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.218+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efe9803caf0 con 0x7efe9c132040 2024-09-15T01:53:02.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.218+0000 7efe91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.218+0000 7efe91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.263+0000 7fe280c3f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.263+0000 7fe280c3f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.263+0000 7fe280c3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.263+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7fe27c06f000 con 0x7fe27c0a0370 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.263+0000 7fe2727fc640 1 -- 172.21.15.67:0/3326412161 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe27401d310 con 0x7fe27c0a0370 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.264+0000 7fe2727fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.264+0000 7fe2727fc640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-15T01:53:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foooo' already exists 2024-09-15T01:53:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 msgr2=0x7fe25c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 0x7fe25c03abf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe26c007310 tx=0x7fe26c007e40 comp rx=0 tx=0).stop 2024-09-15T01:53:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 10 monclient: shutdown 2024-09-15T01:53:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 msgr2=0x7fe27c12f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.265+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe27400cee0 tx=0x7fe274007870 comp rx=0 tx=0).stop 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 shutdown_connections 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe25c038740 0x7fe25c03abf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe27c098a50 0x7fe27c12f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0a0370 0x7fe27c12f5f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 --2- 172.21.15.67:0/3326412161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe27c098140 0x7fe27c09b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 >> 172.21.15.67:0/3326412161 conn(0x7fe27c0a5780 msgr2=0x7fe27c12dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 shutdown_connections 2024-09-15T01:53:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.266+0000 7fe280c3f640 1 -- 172.21.15.67:0/3326412161 wait complete. 2024-09-15T01:53:02.277 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create foo 123 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.352+0000 7f82fbe20640 10 monclient: get_monmap_and_config 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.352+0000 7f82fbe20640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.352+0000 7f82fbe20640 10 monclient: monmap: 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:02.353951+0000 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:02.353951+0000 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:02.355 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:02.356 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:02.356 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:02.356 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:02.356 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.353+0000 7f82fbe20640 1 Processor -- start 2024-09-15T01:53:02.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.353+0000 7f82fbe20640 1 -- start start 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.353+0000 7f82fbe20640 10 monclient: init 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.353+0000 7f82fbe20640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 0x7f82f406b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-b con 0x7f82f40a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82f406e2b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-a con 0x7f82f409e220 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f4069c60 con 0x7f82f406e2b0 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40a3c00 con 0x7f82f409e220 2024-09-15T01:53:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4069650 con 0x7f82f40a3830 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fbe20640 10 monclient(hunting): authenticate will time out at 814.823087s 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f93bb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82f406e2b0 0x7f82f409db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57068/0 (socket says 172.21.15.67:57068) 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f93bb640 1 -- 172.21.15.67:0/3633807282 learned_addr learned my addr 172.21.15.67:0/3633807282 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 0x7f82f406b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 10 monclient(hunting): get_auth_request con 0x7f82f409e220 auth_method 0 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): get_auth_request con 0x7f82f40a3830 auth_method 0 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 1 -- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 msgr2=0x7f82f4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 1 -- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 msgr2=0x7f82f4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_done global_id 34238 payload 274 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 -- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 msgr2=0x7f82f4139d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 -- 172.21.15.67:0/3633807282 >> v1:172.21.15.67:6789/0 conn(0x7f82f406e2b0 legacy=0x7f82f409db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 monclient: found mon.noname-b 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.354+0000 7f82f9bbc640 1 -- 172.21.15.67:0/3633807282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f409f500 con 0x7f82f40a3830 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.357+0000 7f82f9bbc640 1 -- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 msgr2=0x7f82f406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.357+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 0x7f82f406b9b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009850 tx=0x7f82ec0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b30c47d10a689bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 1 -- 172.21.15.67:0/3633807282 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec02ae80 con 0x7f82f40a3830 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: dump: 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 1 -- 172.21.15.67:0/3633807282 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec032440 con 0x7f82f40a3830 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 1 -- 172.21.15.67:0/3633807282 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0327c0 con 0x7f82f40a3830 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 5 monclient: authenticate success, global_id 34238 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 4 set_mon_vals no callback set 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82f8bba640 10 monclient: dump: 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 10 monclient: shutdown 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 -- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 msgr2=0x7f82f406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 --2- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 0x7f82f406b9b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009850 tx=0x7f82ec0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 -- 172.21.15.67:0/3633807282 shutdown_connections 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 --2- 172.21.15.67:0/3633807282 >> v2:172.21.15.17:3300/0 conn(0x7f82f409e220 0x7f82f4139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 --2- 172.21.15.67:0/3633807282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f40a3830 0x7f82f406b9b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 -- 172.21.15.67:0/3633807282 >> 172.21.15.67:0/3633807282 conn(0x7f82f4070ff0 msgr2=0x7f82f40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 -- 172.21.15.67:0/3633807282 shutdown_connections 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.358+0000 7f82fbe20640 1 -- 172.21.15.67:0/3633807282 wait complete. 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient: monmap: 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 1 Processor -- start 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 1 -- start start 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient: init 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-b con 0x7f82f406e2b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f409e220 0x7f82f413d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-c con 0x7f82f409e220 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.359+0000 7f82fbe20640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): picked mon.noname-a con 0x7f82f40a3830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4122ff0 con 0x7f82f40a3830 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4122a30 con 0x7f82f406e2b0 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4122750 con 0x7f82f409e220 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fbe20640 10 monclient(hunting): authenticate will time out at 814.829050s 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40980/0 (socket says 172.21.15.67:40980) 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 -- 172.21.15.67:0/1666851331 learned_addr learned my addr 172.21.15.67:0/1666851331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): get_auth_request con 0x7f82f406e2b0 auth_method 0 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f409e220 0x7f82f413d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): get_auth_request con 0x7f82f409e220 auth_method 0 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 10 monclient(hunting): get_auth_request con 0x7f82f40a3830 auth_method 0 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f93bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 msgr2=0x7f82f4141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 msgr2=0x7f82f4141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient(hunting): handle_auth_done global_id 4597 payload 824 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82fa3bd640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f409e220 msgr2=0x7f82f413d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f409e220 0x7f82f413d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 msgr2=0x7f82f4141570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 monclient: found mon.noname-b 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.360+0000 7f82f9bbc640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009390 con 0x7f82f406e2b0 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82f9bbc640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 msgr2=0x7f82f413e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82f9bbc640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009850 tx=0x7f82ec032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0318c0 con 0x7f82f406e2b0 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: dump: 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82fbe20640 5 monclient: authenticate success, global_id 4597 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec031a20 con 0x7f82f406e2b0 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec031da0 con 0x7f82f406e2b0 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82fbe20640 10 monclient: _renew_subs 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82fbe20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f4122d10 con 0x7f82f406e2b0 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82ebfff640 4 set_mon_vals no callback set 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.362+0000 7f82e9ffb640 10 monclient: dump: 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.363+0000 7f82fbe20640 10 monclient: _renew_subs 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.363+0000 7f82fbe20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.363+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f413dbf0 con 0x7f82f406e2b0 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.363+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f82ec039070 con 0x7f82f406e2b0 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82fbe20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82fbe20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82fbe20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82e9ffb640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 0x7f82d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f404bf60 con 0x7f82f406e2b0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f82ec06bd10 con 0x7f82f406e2b0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82f93bb640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 0x7f82d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.364+0000 7f82f93bb640 10 monclient: get_auth_request con 0x7f82d8038740 auth_method 0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.366+0000 7f82f93bb640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 0x7f82d803abf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f82f40675e0 tx=0x7f82e4006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.368+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f82ec037070 con 0x7f82f406e2b0 2024-09-15T01:53:02.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.368+0000 7f82e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.368+0000 7f82e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:02.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.447+0000 7efea3f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.447+0000 7efea3f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.447+0000 7efea3f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.447+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe9c1351a0 con 0x7efe9c132040 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.448+0000 7efe91ffb640 1 -- 172.21.15.17:0/84007113 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7efe9c1351a0 con 0x7efe9c132040 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.448+0000 7efe91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:02.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.448+0000 7efe91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:02.451 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:02.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 msgr2=0x7efe8003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 0x7efe8003ac40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009e40 tx=0x7efe8c006f50 comp rx=0 tx=0).stop 2024-09-15T01:53:02.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 10 monclient: shutdown 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 msgr2=0x7efe9c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c132b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0a18a0 tx=0x7efe98032000 comp rx=0 tx=0).stop 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 shutdown_connections 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe80038790 0x7efe8003ac40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe9c136dd0 0x7efe9c133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe9c132040 0x7efe9c132b20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 --2- 172.21.15.17:0/84007113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe9c0a58b0 0x7efe9c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 >> 172.21.15.17:0/84007113 conn(0x7efe9c070ed0 msgr2=0x7efe9c096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 shutdown_connections 2024-09-15T01:53:02.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:02.450+0000 7efea3f9d640 1 -- 172.21.15.17:0/84007113 wait complete. 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82fbe20640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82fbe20640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82fbe20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7f82f406c070 con 0x7f82f406e2b0 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82ec040030 con 0x7f82f406e2b0 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.533+0000 7f82e9ffb640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-15T01:53:02.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.629+0000 7f82fbe20640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.629+0000 7f82fbe20640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.629+0000 7f82fbe20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.629+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7f82f4122890 con 0x7f82f406e2b0 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.630+0000 7f82e9ffb640 1 -- 172.21.15.67:0/1666851331 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82f4122890 con 0x7f82f406e2b0 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.630+0000 7f82e9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.630+0000 7f82e9ffb640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-15T01:53:02.633 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' already exists 2024-09-15T01:53:02.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 msgr2=0x7f82d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 0x7f82d803abf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f82f40675e0 tx=0x7f82e4006f50 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 10 monclient: shutdown 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 msgr2=0x7f82f413e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009850 tx=0x7f82ec032000 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 shutdown_connections 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82d8038740 0x7f82d803abf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82f40a3830 0x7f82f4141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82f409e220 0x7f82f413d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 --2- 172.21.15.67:0/1666851331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406e2b0 0x7f82f413e000 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 >> 172.21.15.67:0/1666851331 conn(0x7f82f4070ff0 msgr2=0x7f82f4099400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 shutdown_connections 2024-09-15T01:53:02.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.632+0000 7f82fbe20640 1 -- 172.21.15.67:0/1666851331 wait complete. 2024-09-15T01:53:02.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.719+0000 7fc8b9500640 10 monclient: get_monmap_and_config 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.719+0000 7fc8b9500640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.719+0000 7fc8b9500640 10 monclient: monmap: 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:02.721013+0000 2024-09-15T01:53:02.722 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:02.721013+0000 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:02.723 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 1 Processor -- start 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 1 -- start start 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient: init 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T01:53:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8b4098140 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b40a0370 0x7fc8b4098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b40a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b4098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.720+0000 7fc8b9500640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4098ef0 con 0x7fc8b4098140 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b40a1730 con 0x7fc8b40a0370 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40a0b80 con 0x7fc8b4098a50 2024-09-15T01:53:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b9500640 10 monclient(hunting): authenticate will time out at 815.190018s 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b40a0370 0x7fc8b4098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b40a0370 0x7fc8b4098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40988/0 (socket says 172.21.15.67:40988) 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 -- 172.21.15.67:0/3548857237 learned_addr learned my addr 172.21.15.67:0/3548857237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): get_auth_request con 0x7fc8b40a0370 auth_method 0 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b37fe640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b37fe640 10 monclient(hunting): get_auth_request con 0x7fc8b4098a50 auth_method 0 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient(hunting): handle_auth_done global_id 4600 payload 274 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 -- 172.21.15.67:0/3548857237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 msgr2=0x7fc8b41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 -- 172.21.15.67:0/3548857237 >> v1:172.21.15.17:6789/0 conn(0x7fc8b4098140 legacy=0x7fc8b41321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 monclient: found mon.noname-c 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.721+0000 7fc8b27fc640 1 -- 172.21.15.67:0/3548857237 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b41328f0 con 0x7fc8b40a0370 2024-09-15T01:53:02.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b27fc640 1 -- 172.21.15.67:0/3548857237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b40a0370 msgr2=0x7fc8b4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b27fc640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b40a0370 0x7fc8b4098510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009910 tx=0x7fc89c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e81cc27bd33337 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 1 -- 172.21.15.67:0/3548857237 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c02aea0 con 0x7fc8b40a0370 2024-09-15T01:53:02.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: dump: 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.737 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 1 -- 172.21.15.67:0/3548857237 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c032440 con 0x7fc8b40a0370 2024-09-15T01:53:02.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 5 monclient: authenticate success, global_id 4600 2024-09-15T01:53:02.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 1 -- 172.21.15.67:0/3548857237 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c032820 con 0x7fc8b40a0370 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 4 set_mon_vals no callback set 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b1ffb640 10 monclient: dump: 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.739 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 10 monclient: shutdown 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 1 -- 172.21.15.67:0/3548857237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b40a0370 msgr2=0x7fc8b4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.724+0000 7fc8b9500640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b40a0370 0x7fc8b4098510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009910 tx=0x7fc89c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 -- 172.21.15.67:0/3548857237 shutdown_connections 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b41390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 --2- 172.21.15.67:0/3548857237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b40a0370 0x7fc8b4098510 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 -- 172.21.15.67:0/3548857237 >> 172.21.15.67:0/3548857237 conn(0x7fc8b40a5780 msgr2=0x7fc8b40a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 -- 172.21.15.67:0/3548857237 shutdown_connections 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 -- 172.21.15.67:0/3548857237 wait complete. 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 10 monclient: build_initial_monmap 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 10 monclient: monmap: 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:02.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.725+0000 7fc8b9500640 1 Processor -- start 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 -- start start 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient: init 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:02.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b4098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b4130800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b4098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 0x7fc8b4130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b40a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4138d20 con 0x7fc8b40a0370 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4133f90 con 0x7fc8b4098140 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4134200 con 0x7fc8b4098a50 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:02.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b9500640 10 monclient(hunting): authenticate will time out at 815.195518s 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40994/0 (socket says 172.21.15.67:40994) 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 -- 172.21.15.67:0/1547786885 learned_addr learned my addr 172.21.15.67:0/1547786885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): get_auth_request con 0x7fc8b4098140 auth_method 0 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b27fc640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b4130800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b37fe640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 0x7fc8b4130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient(hunting): handle_auth_done global_id 4603 payload 824 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 -- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 msgr2=0x7fc8b4130800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b4130800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 -- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 msgr2=0x7fc8b4130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 0x7fc8b4130d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 monclient: found mon.noname-b 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b2ffd640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c009450 con 0x7fc8b4098140 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b37fe640 10 monclient: get_auth_request con 0x7fc8b40a0370 auth_method 0 2024-09-15T01:53:02.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.726+0000 7fc8b37fe640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 0x7fc8b4130d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b2ffd640 1 -- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 msgr2=0x7fc8b41302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b2ffd640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400c8e0 tx=0x7fc8a40026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a400aa60 con 0x7fc8b4098140 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: dump: 2024-09-15T01:53:02.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a400abe0 con 0x7fc8b4098140 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 5 monclient: authenticate success, global_id 4603 2024-09-15T01:53:02.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4012420 con 0x7fc8b4098140 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 10 monclient: _renew_subs 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b4139230 con 0x7fc8b4098140 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b17fa640 4 set_mon_vals no callback set 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8aaffd640 10 monclient: dump: 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:02.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 10 monclient: _renew_subs 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.729+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b41322d0 con 0x7fc8b4098140 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc8a4012850 con 0x7fc8b4098140 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8aaffd640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 0x7fc89403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:02.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b9500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b9500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc8a404c2f0 con 0x7fc8b4098140 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b27fc640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 0x7fc89403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.730+0000 7fc8b27fc640 10 monclient: get_auth_request con 0x7fc894038740 auth_method 0 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.732+0000 7fc8b27fc640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 0x7fc89403abf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40a5340 tx=0x7fc89c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.734+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.734+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:02.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.734+0000 7fc8aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:02.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.896+0000 7fc8b9500640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:02.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.896+0000 7fc8b9500640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:02.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.896+0000 7fc8b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:02.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:02.896+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.115+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v32) v1 ==== 152+0+0 (secure 0 0 0) 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.115+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.115+0000 7fc8aaffd640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-15T01:53:03.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.250+0000 7fc8b9500640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:03.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.250+0000 7fc8b9500640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:03.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.250+0000 7fc8b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:03.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:03.250+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:04.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.117+0000 7fc8aaffd640 1 -- 172.21.15.67:0/1547786885 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v33) v1 ==== 152+0+0 (secure 0 0 0) 0x7fc8b4070b10 con 0x7fc8b4098140 2024-09-15T01:53:04.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.118+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T01:53:04.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.118+0000 7fc8aaffd640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-15T01:53:04.120 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 4 size to 1 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.123+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 msgr2=0x7fc89403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.123+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 0x7fc89403abf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40a5340 tx=0x7fc89c032000 comp rx=0 tx=0).stop 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.124+0000 7fc8b9500640 10 monclient: shutdown 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.124+0000 7fc8b9500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.124+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 msgr2=0x7fc8b41302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.124+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400c8e0 tx=0x7fc8a40026c0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 shutdown_connections 2024-09-15T01:53:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc894038740 0x7fc89403abf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8b40a0370 0x7fc8b4130d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8b4098a50 0x7fc8b4130800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 --2- 172.21.15.67:0/1547786885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b4098140 0x7fc8b41302c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 >> 172.21.15.67:0/1547786885 conn(0x7fc8b40a5780 msgr2=0x7fc8b40a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.129+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 shutdown_connections 2024-09-15T01:53:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.130+0000 7fc8b9500640 1 -- 172.21.15.67:0/1547786885 wait complete. 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_config_value foo min_size 1 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+ local pool_name config_opt expected_val val 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+ pool_name=foo 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+ config_opt=min_size 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_val=1 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ get_config_value_or_die foo min_size 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ local pool_name config_opt raw val 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ pool_name=foo 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ config_opt=min_size 2024-09-15T01:53:04.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ ceph osd pool get foo min_size 2024-09-15T01:53:04.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw='min_size: 1' 2024-09-15T01:53:04.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T01:53:04.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size: 1 2024-09-15T01:53:04.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T01:53:04.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw=min_size:1 2024-09-15T01:53:04.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size:1 2024-09-15T01:53:04.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ cut -f2 -d: 2024-09-15T01:53:04.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ val=1 2024-09-15T01:53:04.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo 1 2024-09-15T01:53:04.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ return 0 2024-09-15T01:53:04.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ val=1 2024-09-15T01:53:04.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 1 != \1 ]] 2024-09-15T01:53:04.524 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set foo size 4 2024-09-15T01:53:04.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.598+0000 7f271331d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.598+0000 7f271331d640 10 monclient: build_initial_monmap 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.598+0000 7f271331d640 10 monclient: monmap: 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:04.599490+0000 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:04.599490+0000 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:04.601 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:04.602 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:04.602 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:04.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.598+0000 7f271331d640 1 Processor -- start 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 1 -- start start 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient: init 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 0x7f270c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): picked mon.noname-a con 0x7f270c098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f270c0a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f270c098c20 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0a0b80 con 0x7f270c098c20 2024-09-15T01:53:04.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0a00b0 con 0x7f270c0a0370 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270c071390 con 0x7f270c098140 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f271331d640 10 monclient(hunting): authenticate will time out at 817.068641s 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f27108b8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f270c0a0370 0x7f270c098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57088/0 (socket says 172.21.15.67:57088) 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.599+0000 7f27108b8640 1 -- 172.21.15.67:0/2397516063 learned_addr learned my addr 172.21.15.67:0/2397516063 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 1 --2- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 0x7f270c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 10 monclient(hunting): get_auth_request con 0x7f270c098140 auth_method 0 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 1 -- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 msgr2=0x7f270c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 1 -- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 msgr2=0x7f270c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 1 --2- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 0x7f270c1321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.600+0000 7f27110b9640 1 --2- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 0x7f270c1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.601+0000 7f27108b8640 1 -- 172.21.15.67:0/2397516063 >> v1:172.21.15.67:6789/0 conn(0x7f270c0a0370 legacy=0x7f270c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825064393 0 0) 0x7f270c0a00b0 con 0x7f270c0a0370 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): my global_id is 4612 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f80034e0 con 0x7f270c0a0370 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 >> v1:172.21.15.17:6789/0 conn(0x7f270c098c20 legacy=0x7f270c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507931233 0 0) 0x7f26f80034e0 con 0x7f270c0a0370 2024-09-15T01:53:04.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 msgr2=0x7f270c1321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 --2- 172.21.15.67:0/2397516063 >> v2:172.21.15.17:3300/0 conn(0x7f270c098140 0x7f270c1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 >> v1:172.21.15.17:6789/0 conn(0x7f270c098c20 legacy=0x7f270c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c0714d0 con 0x7f270c0a0370 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: _finish_auth 0 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f27000017e0 con 0x7f270c0a0370 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f271331d640 5 monclient: authenticate success, global_id 4612 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f271331d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: dump: 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.612 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 >> v1:172.21.15.67:6789/0 conn(0x7f270c0a0370 legacy=0x7f270c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f8003a50 0x7f26f8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): picked mon.a con 0x7f26f8003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f8004780 0x7f26f8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): picked mon.c con 0x7f26f8004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 0x7f26f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): picked mon.b con 0x7f26f80052e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c071390 con 0x7f26f8003a50 2024-09-15T01:53:04.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f800c2b0 con 0x7f26f80052e0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f800c470 con 0x7f26f8004780 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f26fffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f271331d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 0x7f26f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f8004780 0x7f26f8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): get_auth_request con 0x7f26f80052e0 auth_method 0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f8003a50 0x7f26f8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:04.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): get_auth_request con 0x7f26f8004780 auth_method 0 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): get_auth_request con 0x7f26f8003a50 auth_method 0 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient(hunting): handle_auth_done global_id 4612 payload 293 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f8004780 msgr2=0x7f26f8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26f8004780 0x7f26f8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f8003a50 msgr2=0x7f26f8003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f8003a50 0x7f26f8003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient: _finish_hunting 0 2024-09-15T01:53:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 monclient: found mon.b 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f80034e0 con 0x7f26f80052e0 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 10 monclient: _finish_auth 0 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.602+0000 7f27118ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:04.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.603+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 reap_dead start 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f27118ba640 1 -- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 msgr2=0x7f26f800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f27118ba640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 0x7f26f800b9d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f270c068d00 tx=0x7f2704001890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c1e79f9d93ae0d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704005b00 con 0x7f26f80052e0 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: dump: 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f271331d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270402d070 con 0x7f26f80052e0 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 1 -- 172.21.15.67:0/2397516063 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704005ca0 con 0x7f26f80052e0 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f271331d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f271331d640 4 set_mon_vals no callback set 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.605+0000 7f26fffff640 10 monclient: dump: 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 10 monclient: shutdown 2024-09-15T01:53:04.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 -- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 msgr2=0x7f26f800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 0x7f26f800b9d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f270c068d00 tx=0x7f2704001890 comp rx=0 tx=0).stop 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 -- 172.21.15.67:0/2397516063 shutdown_connections 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 --2- 172.21.15.67:0/2397516063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f80052e0 0x7f26f800b9d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 -- 172.21.15.67:0/2397516063 >> 172.21.15.67:0/2397516063 conn(0x7f270c0a5780 msgr2=0x7f270c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 -- 172.21.15.67:0/2397516063 shutdown_connections 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.606+0000 7f271331d640 1 -- 172.21.15.67:0/2397516063 wait complete. 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient: build_initial_monmap 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient: monmap: 2024-09-15T01:53:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 1 Processor -- start 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 1 -- start start 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient: init 2024-09-15T01:53:04.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f270c140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.607+0000 7f271331d640 10 monclient(hunting): picked mon.noname-a con 0x7f26f80052e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): picked mon.noname-b con 0x7f270c120f80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f270c11e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): picked mon.noname-c con 0x7f270c1322d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c099090 con 0x7f26f80052e0 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c0a4220 con 0x7f270c120f80 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c1221c0 con 0x7f270c1322d0 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f271331d640 10 monclient(hunting): authenticate will time out at 817.077047s 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41022/0 (socket says 172.21.15.67:41022) 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 learned_addr learned my addr 172.21.15.67:0/1443985342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): get_auth_request con 0x7f270c120f80 auth_method 0 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:04.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f270c140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27118ba640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f270c11e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient(hunting): handle_auth_done global_id 4615 payload 824 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 msgr2=0x7f270c11e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f270c11e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 msgr2=0x7f270c140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f270c140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient: _finish_hunting 0 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 monclient: found mon.noname-b 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c1203b0 con 0x7f270c120f80 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27110b9640 10 monclient: get_auth_request con 0x7f26f80052e0 auth_method 0 2024-09-15T01:53:04.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.608+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f270c140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 msgr2=0x7f270c131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2700001ed0 tx=0x7f2700007af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700018030 con 0x7f270c120f80 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: dump: 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.631 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270000fbd0 con 0x7f270c120f80 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 5 monclient: authenticate success, global_id 4615 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 10 monclient: _renew_subs 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c121ee0 con 0x7f270c120f80 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27000133f0 con 0x7f270c120f80 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26fd7fa640 10 monclient: dump: 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f26ff7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 10 monclient: _renew_subs 2024-09-15T01:53:04.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.610+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c141490 con 0x7f270c120f80 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.611+0000 7f271331d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.611+0000 7f271331d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.611+0000 7f271331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.611+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c04bf60 con 0x7f270c120f80 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.612+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f270001b070 con 0x7f270c120f80 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.612+0000 7f26fd7fa640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 0x7f26f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.612+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f2700019070 con 0x7f270c120f80 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.612+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 0x7f26f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.612+0000 7f27110b9640 10 monclient: get_auth_request con 0x7f26f0038740 auth_method 0 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.614+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 0x7f26f003abf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2708006e70 tx=0x7f2708008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.616+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f270001bd90 con 0x7f270c120f80 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.616+0000 7f26fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:04.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.616+0000 7f26fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:04.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f271331d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:04.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f271331d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f271331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f270c098510 con 0x7f270c120f80 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 msgr2=0x7f270c131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27108b8640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 msgr2=0x7f270c131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2700001ed0 tx=0x7f2700007af0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2700001ed0 tx=0x7f2700007af0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 msgr2=0x7f270c131d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f270c120f80 0x7f270c131d90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:04.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f26f0047820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): picked mon.c con 0x7f270c1322d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f26f004a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): picked mon.a con 0x7f26f80052e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 0x7f26f004ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): picked mon.b con 0x7f26f004a980 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f004d820 con 0x7f26f80052e0 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f004d960 con 0x7f26f004a980 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f004daf0 con 0x7f270c1322d0 2024-09-15T01:53:04.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f26fd7fa640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 0x7f26f004ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): get_auth_request con 0x7f26f004a980 auth_method 0 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.780+0000 7f27110b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27118ba640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f26f004a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27108b8640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f26f0047820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient(hunting): handle_auth_done global_id 4615 payload 293 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 msgr2=0x7f26f0047820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f26f0047820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 msgr2=0x7f26f004a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f26f004a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 monclient: found mon.b 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f26f004dcb0 con 0x7f26f004a980 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f270800e040 con 0x7f26f004a980 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 10 monclient: _finish_auth 0 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27110b9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:04.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27118ba640 10 monclient: get_auth_request con 0x7f26f80052e0 auth_method 0 2024-09-15T01:53:04.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.781+0000 7f27118ba640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f26f004a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:04.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f27110b9640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 msgr2=0x7f26f004ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f27110b9640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 0x7f26f004ce00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2708006ea0 tx=0x7f2708016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708015680 con 0x7f26f004a980 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 10 monclient: dump: 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:04.790 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708015cd0 con 0x7f26f004a980 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.783+0000 7f26ff7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:04.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:04.784+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f27080147b0 con 0x7f26f004a980 2024-09-15T01:53:05.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.145+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f27080124a0 con 0x7f26f004a980 2024-09-15T01:53:05.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.145+0000 7f26fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:05.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.145+0000 7f26fd7fa640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-15T01:53:05.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.263+0000 7f271331d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:05.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.263+0000 7f271331d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:05.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.263+0000 7f271331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:05.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:05.263+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f270c122350 con 0x7f26f004a980 2024-09-15T01:53:05.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:05.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.568+0000 7fe7b6cef640 10 monclient: get_monmap_and_config 2024-09-15T01:53:05.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.568+0000 7fe7b6cef640 10 monclient: build_initial_monmap 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.568+0000 7fe7b6cef640 10 monclient: monmap: 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:05.569545+0000 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:05.569545+0000 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:05.571 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:05.572 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.569+0000 7fe7b6cef640 1 Processor -- start 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.569+0000 7fe7b6cef640 1 -- start start 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.569+0000 7fe7b6cef640 10 monclient: init 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7b009c0d0 0x7fe7b009a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b009c0d0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b009c4a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b009c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b006a5e0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b006a290 con 0x7fe7b009c0d0 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00699c0 con 0x7fe7b009c8a0 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:05.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b6cef640 10 monclient(hunting): authenticate will time out at 845.566136s 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40430/0 (socket says 172.21.15.17:40430) 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 -- 172.21.15.17:0/3539077815 learned_addr learned my addr 172.21.15.17:0/3539077815 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): get_auth_request con 0x7fe7b009c8a0 auth_method 0 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 10 monclient(hunting): get_auth_request con 0x7fe7b009c4a0 auth_method 0 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 1 -- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 msgr2=0x7fe7b006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 1 -- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 msgr2=0x7fe7b006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.67:3300/0 conn(0x7fe7b009c0d0 0x7fe7b009a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7affff640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): get_auth_request con 0x7fe7b009c0d0 auth_method 0 2024-09-15T01:53:05.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient(hunting): handle_auth_done global_id 34247 payload 274 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 -- 172.21.15.17:0/3539077815 >> v2:172.21.15.67:3300/0 conn(0x7fe7b009c0d0 msgr2=0x7fe7b009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.67:3300/0 conn(0x7fe7b009c0d0 0x7fe7b009a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 -- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 msgr2=0x7fe7b006eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 monclient: found mon.noname-b 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:05.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.570+0000 7fe7b528c640 1 -- 172.21.15.17:0/3539077815 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b009f6e0 con 0x7fe7b009c8a0 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b528c640 1 -- 172.21.15.17:0/3539077815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 msgr2=0x7fe7b006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b528c640 1 --2- 172.21.15.17:0/3539077815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7b00a3e50 tx=0x7fe7a8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eb70ba62be109cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 1 -- 172.21.15.17:0/3539077815 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8007610 con 0x7fe7b009c8a0 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: dump: 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.584 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 1 -- 172.21.15.17:0/3539077815 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a802d050 con 0x7fe7b009c8a0 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b6cef640 5 monclient: authenticate success, global_id 34247 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b6cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b6cef640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7b6cef640 4 set_mon_vals no callback set 2024-09-15T01:53:05.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 1 -- 172.21.15.17:0/3539077815 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80079a0 con 0x7fe7b009c8a0 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.573+0000 7fe7af7fe640 10 monclient: dump: 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 10 monclient: shutdown 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 -- 172.21.15.17:0/3539077815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 msgr2=0x7fe7b006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/3539077815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7b00a3e50 tx=0x7fe7a8009840 comp rx=0 tx=0).stop 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 -- 172.21.15.17:0/3539077815 shutdown_connections 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/3539077815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c8a0 0x7fe7b006f070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.17:3300/0 conn(0x7fe7b009c4a0 0x7fe7b006eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/3539077815 >> v2:172.21.15.67:3300/0 conn(0x7fe7b009c0d0 0x7fe7b009a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 -- 172.21.15.17:0/3539077815 >> 172.21.15.17:0/3539077815 conn(0x7fe7b00712f0 msgr2=0x7fe7b00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 -- 172.21.15.17:0/3539077815 shutdown_connections 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 1 -- 172.21.15.17:0/3539077815 wait complete. 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 10 monclient: build_initial_monmap 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.574+0000 7fe7b6cef640 10 monclient: monmap: 2024-09-15T01:53:05.587 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 Processor -- start 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 -- start start 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient: init 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:05.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7b009c0d0 0x7fe7b013d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b009c0d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b009c4a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b009c8a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0122690 con 0x7fe7b009c8a0 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b01220d0 con 0x7fe7b009c0d0 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b011fb60 con 0x7fe7b009c4a0 2024-09-15T01:53:05.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.575+0000 7fe7b6cef640 10 monclient(hunting): authenticate will time out at 845.571932s 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40438/0 (socket says 172.21.15.17:40438) 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 -- 172.21.15.17:0/2502158953 learned_addr learned my addr 172.21.15.17:0/2502158953 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): get_auth_request con 0x7fe7b009c4a0 auth_method 0 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 10 monclient(hunting): get_auth_request con 0x7fe7b009c8a0 auth_method 0 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 msgr2=0x7fe7b0140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 msgr2=0x7fe7b0140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b528c640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7b009c0d0 0x7fe7b013d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): get_auth_request con 0x7fe7b009c0d0 auth_method 0 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7b4a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient(hunting): handle_auth_done global_id 34250 payload 824 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7b009c0d0 msgr2=0x7fe7b013d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7b009c0d0 0x7fe7b013d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 msgr2=0x7fe7b0140c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 monclient: found mon.noname-c 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.576+0000 7fe7affff640 1 -- 172.21.15.17:0/2502158953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a8009ae0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7affff640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 msgr2=0x7fe7b013cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7affff640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00d8c0 tx=0x7fe79c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c00de50 con 0x7fe7b009c4a0 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: dump: 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:05.593 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c00ba60 con 0x7fe7b009c4a0 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 5 monclient: authenticate success, global_id 34250 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 10 monclient: _renew_subs 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b0121df0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c00be20 con 0x7fe7b009c4a0 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7acff9640 10 monclient: dump: 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7aeffd640 4 set_mon_vals no callback set 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 10 monclient: _renew_subs 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.578+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b013d430 con 0x7fe7b009c4a0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.579+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe79c014640 con 0x7fe7b009c4a0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.579+0000 7fe7b6cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.579+0000 7fe7b6cef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.579+0000 7fe7b6cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.579+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b01224d0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.580+0000 7fe7acff9640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 0x7fe79403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.580+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fe79c04c2b0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.580+0000 7fe7b4a8b640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 0x7fe79403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:05.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.580+0000 7fe7b4a8b640 10 monclient: get_auth_request con 0x7fe794038790 auth_method 0 2024-09-15T01:53:05.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.582+0000 7fe7b4a8b640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 0x7fe79403ac40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d20 tx=0x7fe7a40090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:05.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.584+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe79c0148f0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.584+0000 7fe7acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:05.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.584+0000 7fe7acff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:05.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.809+0000 7fe7b6cef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:05.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.809+0000 7fe7b6cef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:05.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.809+0000 7fe7b6cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:05.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.809+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe7b00724a0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.810+0000 7fe7acff9640 1 -- 172.21.15.17:0/2502158953 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7fe79c01dbb0 con 0x7fe7b009c4a0 2024-09-15T01:53:05.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.810+0000 7fe7acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:05.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.810+0000 7fe7acff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:05.814 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:05.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 msgr2=0x7fe79403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:05.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 0x7fe79403ac40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d20 tx=0x7fe7a40090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 10 monclient: shutdown 2024-09-15T01:53:05.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:05.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 msgr2=0x7fe7b013cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:05.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00d8c0 tx=0x7fe79c008040 comp rx=0 tx=0).stop 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 shutdown_connections 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe794038790 0x7fe79403ac40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7b009c8a0 0x7fe7b0140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe7b009c4a0 0x7fe7b013cd50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 --2- 172.21.15.17:0/2502158953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7b009c0d0 0x7fe7b013d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 >> 172.21.15.17:0/2502158953 conn(0x7fe7b00712f0 msgr2=0x7fe7b00a1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:05.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 shutdown_connections 2024-09-15T01:53:05.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:05.812+0000 7fe7b6cef640 1 -- 172.21.15.17:0/2502158953 wait complete. 2024-09-15T01:53:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.147+0000 7f26fd7fa640 1 -- 172.21.15.67:0/1443985342 <== mon.1 v2:172.21.15.67:3300/0 5 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2708012bf0 con 0x7f26f004a980 2024-09-15T01:53:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.147+0000 7f26fd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T01:53:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.147+0000 7f26fd7fa640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-15T01:53:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 4 size to 4 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 msgr2=0x7f26f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 0x7f26f003abf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2708006e70 tx=0x7f2708008040 comp rx=0 tx=0).stop 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 10 monclient: shutdown 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 msgr2=0x7f26f004ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.154+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 0x7f26f004ce00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2708006ea0 tx=0x7f2708016000 comp rx=0 tx=0).stop 2024-09-15T01:53:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 shutdown_connections 2024-09-15T01:53:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26f80052e0 0x7f26f004a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26f0038740 0x7f26f003abf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f004a980 0x7f26f004ce00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 --2- 172.21.15.67:0/1443985342 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f270c1322d0 0x7f26f0047820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.157+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 >> 172.21.15.67:0/1443985342 conn(0x7f270c0a5780 msgr2=0x7f270c0a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:06.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.159+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 shutdown_connections 2024-09-15T01:53:06.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.159+0000 7f271331d640 1 -- 172.21.15.67:0/1443985342 wait complete. 2024-09-15T01:53:06.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_config_value foo min_size 2 2024-09-15T01:53:06.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ local pool_name config_opt expected_val val 2024-09-15T01:53:06.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ pool_name=foo 2024-09-15T01:53:06.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ config_opt=min_size 2024-09-15T01:53:06.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_val=2 2024-09-15T01:53:06.184 INFO:tasks.workunit.client.0.smithi067.stderr:++ get_config_value_or_die foo min_size 2024-09-15T01:53:06.184 INFO:tasks.workunit.client.0.smithi067.stderr:++ local pool_name config_opt raw val 2024-09-15T01:53:06.184 INFO:tasks.workunit.client.0.smithi067.stderr:++ pool_name=foo 2024-09-15T01:53:06.184 INFO:tasks.workunit.client.0.smithi067.stderr:++ config_opt=min_size 2024-09-15T01:53:06.185 INFO:tasks.workunit.client.0.smithi067.stderr:+++ ceph osd pool get foo min_size 2024-09-15T01:53:06.591 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw='min_size: 2' 2024-09-15T01:53:06.591 INFO:tasks.workunit.client.0.smithi067.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T01:53:06.591 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size: 2 2024-09-15T01:53:06.591 INFO:tasks.workunit.client.0.smithi067.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T01:53:06.592 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw=min_size:2 2024-09-15T01:53:06.593 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size:2 2024-09-15T01:53:06.593 INFO:tasks.workunit.client.0.smithi067.stderr:+++ cut -f2 -d: 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:++ val=2 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo 2 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:++ return 0 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:+ val=2 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 2 != \2 ]] 2024-09-15T01:53:06.594 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set foo size 10 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 10 monclient: get_monmap_and_config 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 10 monclient: build_initial_monmap 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 10 monclient: monmap: 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:06.670362+0000 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:06.670362+0000 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.672 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 1 Processor -- start 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 1 -- start start 2024-09-15T01:53:06.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.669+0000 7f1bedf14640 10 monclient: init 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 0x7f1be8097f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-b con 0x7f1be8097ba0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1be80672c0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be8137000 0x7f1be81393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-c con 0x7f1be8137000 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be809e430 con 0x7f1be80672c0 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be8067690 con 0x7f1be8137000 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80977a0 con 0x7f1be8097ba0 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1bedf14640 10 monclient(hunting): authenticate will time out at 819.139479s 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be8137000 0x7f1be81393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be8137000 0x7f1be81393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41054/0 (socket says 172.21.15.67:41054) 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 -- 172.21.15.67:0/1676336368 learned_addr learned my addr 172.21.15.67:0/1676336368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): get_auth_request con 0x7f1be8137000 auth_method 0 2024-09-15T01:53:06.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 0x7f1be8097f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 10 monclient(hunting): get_auth_request con 0x7f1be8097ba0 auth_method 0 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient(hunting): handle_auth_done global_id 4630 payload 274 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 -- 172.21.15.67:0/1676336368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 msgr2=0x7f1be8097f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 0x7f1be8097f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 -- 172.21.15.67:0/1676336368 >> v1:172.21.15.17:6789/0 conn(0x7f1be80672c0 legacy=0x7f1be8097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 monclient: found mon.noname-c 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be7fff640 1 -- 172.21.15.67:0/1676336368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be809e150 con 0x7f1be8137000 2024-09-15T01:53:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.670+0000 7f1be77fe640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 0x7f1be8097f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be7fff640 1 -- 172.21.15.67:0/1676336368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8137000 msgr2=0x7f1be81393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be7fff640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8137000 0x7f1be81393e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009910 tx=0x7f1be00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34967380b0991f88 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 1 -- 172.21.15.67:0/1676336368 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be002aea0 con 0x7f1be8137000 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: dump: 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:06.683 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 1 -- 172.21.15.67:0/1676336368 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be0032440 con 0x7f1be8137000 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1bedf14640 5 monclient: authenticate success, global_id 4630 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1bedf14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1bedf14640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 1 -- 172.21.15.67:0/1676336368 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00327e0 con 0x7f1be8137000 2024-09-15T01:53:06.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1bedf14640 4 set_mon_vals no callback set 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.673+0000 7f1be67fc640 10 monclient: dump: 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 10 monclient: shutdown 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 -- 172.21.15.67:0/1676336368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8137000 msgr2=0x7f1be81393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8137000 0x7f1be81393e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009910 tx=0x7f1be00091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 -- 172.21.15.67:0/1676336368 shutdown_connections 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8137000 0x7f1be81393e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 --2- 172.21.15.67:0/1676336368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8097ba0 0x7f1be8097f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 -- 172.21.15.67:0/1676336368 >> 172.21.15.67:0/1676336368 conn(0x7f1be80a2110 msgr2=0x7f1be80a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 -- 172.21.15.67:0/1676336368 shutdown_connections 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 1 -- 172.21.15.67:0/1676336368 wait complete. 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 10 monclient: build_initial_monmap 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.674+0000 7f1bedf14640 10 monclient: monmap: 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:06.686 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 Processor -- start 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 -- start start 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient: init 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:06.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 0x7f1be8141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-a con 0x7f1be80672c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-b con 0x7f1be8097ba0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 0x7f1be8141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): picked mon.noname-c con 0x7f1be8137000 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8067ad0 con 0x7f1be80672c0 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81207f0 con 0x7f1be8097ba0 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81230e0 con 0x7f1be8137000 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.675+0000 7f1bedf14640 10 monclient(hunting): authenticate will time out at 819.144981s 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41062/0 (socket says 172.21.15.67:41062) 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3494004431 learned_addr learned my addr 172.21.15.67:0/3494004431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): get_auth_request con 0x7f1be8097ba0 auth_method 0 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be7fff640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 0x7f1be8141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 0x7f1be8141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient(hunting): handle_auth_done global_id 4633 payload 824 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 msgr2=0x7f1be8141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 0x7f1be8141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 msgr2=0x7f1be8141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 0x7f1be8141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 monclient: found mon.noname-b 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0009450 con 0x7f1be8097ba0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be7fff640 10 monclient: get_auth_request con 0x7f1be8137000 auth_method 0 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be7fff640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 0x7f1be8141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be77fe640 10 monclient: get_auth_request con 0x7f1be80672c0 auth_method 0 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.676+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 0x7f1be8141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 msgr2=0x7f1be81419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1be6ffd640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c5e0 tx=0x7f1bdc007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc012920 con 0x7f1be8097ba0 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: dump: 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bdc012aa0 con 0x7f1be8097ba0 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 5 monclient: authenticate success, global_id 4633 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 10 monclient: _renew_subs 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be80a0cf0 con 0x7f1be8097ba0 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:06.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc012e60 con 0x7f1be8097ba0 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bcf7fe640 10 monclient: dump: 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:06.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 10 monclient: _renew_subs 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be8120eb0 con 0x7f1be8097ba0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.678+0000 7f1be5ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bedf14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bedf14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1bdc011590 con 0x7f1be8097ba0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bedf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be804bf60 con 0x7f1be8097ba0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bcf7fe640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 0x7f1bd403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.679+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f1bdc04bfc0 con 0x7f1be8097ba0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.680+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 0x7f1bd403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:06.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.680+0000 7f1be77fe640 10 monclient: get_auth_request con 0x7f1bd4038740 auth_method 0 2024-09-15T01:53:06.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.682+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 0x7f1bd403abf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80094d0 tx=0x7f1bd800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:06.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.684+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1bdc010920 con 0x7f1be8097ba0 2024-09-15T01:53:06.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.684+0000 7f1bcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:06.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.684+0000 7f1bcf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:06.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.847+0000 7f1bedf14640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:06.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.847+0000 7f1bedf14640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:06.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.847+0000 7f1bedf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:06.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:06.848+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f1be809e710 con 0x7f1be8097ba0 2024-09-15T01:53:07.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.173+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v36) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1bdc016030 con 0x7f1be8097ba0 2024-09-15T01:53:07.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.173+0000 7f1bcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:07.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.173+0000 7f1bcf7fe640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-15T01:53:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.310+0000 7f1bedf14640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.310+0000 7f1bedf14640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.310+0000 7f1bedf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:07.310+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f1be8067840 con 0x7f1be8097ba0 2024-09-15T01:53:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.177+0000 7f1bcf7fe640 1 -- 172.21.15.67:0/3494004431 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v37) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1bdc016020 con 0x7f1be8097ba0 2024-09-15T01:53:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.177+0000 7f1bcf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T01:53:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.177+0000 7f1bcf7fe640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-15T01:53:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 4 size to 10 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 msgr2=0x7f1bd403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 0x7f1bd403abf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80094d0 tx=0x7f1bd800b040 comp rx=0 tx=0).stop 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 10 monclient: shutdown 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 msgr2=0x7f1be81419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.184+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c5e0 tx=0x7f1bdc007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 shutdown_connections 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd4038740 0x7f1bd403abf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1be8137000 0x7f1be8141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be8097ba0 0x7f1be81419b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 --2- 172.21.15.67:0/3494004431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1be80672c0 0x7f1be8141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.185+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 >> 172.21.15.67:0/3494004431 conn(0x7f1be80a2110 msgr2=0x7f1be809ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.186+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 shutdown_connections 2024-09-15T01:53:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.186+0000 7f1bedf14640 1 -- 172.21.15.67:0/3494004431 wait complete. 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_config_value foo min_size 5 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:+ local pool_name config_opt expected_val val 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:+ pool_name=foo 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:+ config_opt=min_size 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_val=5 2024-09-15T01:53:08.208 INFO:tasks.workunit.client.0.smithi067.stderr:++ get_config_value_or_die foo min_size 2024-09-15T01:53:08.209 INFO:tasks.workunit.client.0.smithi067.stderr:++ local pool_name config_opt raw val 2024-09-15T01:53:08.209 INFO:tasks.workunit.client.0.smithi067.stderr:++ pool_name=foo 2024-09-15T01:53:08.209 INFO:tasks.workunit.client.0.smithi067.stderr:++ config_opt=min_size 2024-09-15T01:53:08.209 INFO:tasks.workunit.client.0.smithi067.stderr:+++ ceph osd pool get foo min_size 2024-09-15T01:53:08.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw='min_size: 5' 2024-09-15T01:53:08.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T01:53:08.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size: 5 2024-09-15T01:53:08.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T01:53:08.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ raw=min_size:5 2024-09-15T01:53:08.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ echo min_size:5 2024-09-15T01:53:08.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ cut -f2 -d: 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ val=5 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo 5 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ return 0 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:+ val=5 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 5 != \5 ]] 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:08.619 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set foo size 0 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 10 monclient: monmap: 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:08.695268+0000 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:08.695268+0000 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.697 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 1 Processor -- start 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 1 -- start start 2024-09-15T01:53:08.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.694+0000 7fb75e0e8640 10 monclient: init 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb758098140 0x7fb7581321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb758098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7580a0370 0x7fb758098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7580a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb758098a50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb758098ef0 con 0x7fb758098a50 2024-09-15T01:53:08.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580a0b80 con 0x7fb758098140 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580a00b0 con 0x7fb7580a0370 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb75e0e8640 10 monclient(hunting): authenticate will time out at 821.164457s 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7580a0370 0x7fb758098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb757fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb758098a50 0x7fb7580695c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57118/0 (socket says 172.21.15.67:57118) 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb757fff640 1 -- 172.21.15.67:0/3461599364 learned_addr learned my addr 172.21.15.67:0/3461599364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): get_auth_request con 0x7fb7580a0370 auth_method 0 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb7577fe640 1 --2- 172.21.15.67:0/3461599364 >> v2:172.21.15.17:3300/0 conn(0x7fb758098140 0x7fb7581321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb7577fe640 10 monclient(hunting): get_auth_request con 0x7fb758098140 auth_method 0 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb7577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb7577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient(hunting): handle_auth_done global_id 4639 payload 274 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 -- 172.21.15.67:0/3461599364 >> v2:172.21.15.17:3300/0 conn(0x7fb758098140 msgr2=0x7fb7581321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 --2- 172.21.15.67:0/3461599364 >> v2:172.21.15.17:3300/0 conn(0x7fb758098140 0x7fb7581321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 -- 172.21.15.67:0/3461599364 >> v1:172.21.15.67:6789/0 conn(0x7fb758098a50 legacy=0x7fb7580695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:08.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 monclient: found mon.noname-c 2024-09-15T01:53:08.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:08.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.695+0000 7fb74effd640 1 -- 172.21.15.67:0/3461599364 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758071390 con 0x7fb7580a0370 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb74effd640 1 -- 172.21.15.67:0/3461599364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7580a0370 msgr2=0x7fb758098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb74effd640 1 --2- 172.21.15.67:0/3461599364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7580a0370 0x7fb758098510 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb7580717a0 tx=0x7fb7400090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243e41e9d5ba3278 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 1 -- 172.21.15.67:0/3461599364 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74002ae00 con 0x7fb7580a0370 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: dump: 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.709 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb75e0e8640 5 monclient: authenticate success, global_id 4639 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb75e0e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 1 -- 172.21.15.67:0/3461599364 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740032440 con 0x7fb7580a0370 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb75e0e8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 1 -- 172.21.15.67:0/3461599364 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400327e0 con 0x7fb7580a0370 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb75e0e8640 4 set_mon_vals no callback set 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.698+0000 7fb756ffd640 10 monclient: dump: 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 10 monclient: shutdown 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 -- 172.21.15.67:0/3461599364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7580a0370 msgr2=0x7fb758098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 --2- 172.21.15.67:0/3461599364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7580a0370 0x7fb758098510 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb7580717a0 tx=0x7fb7400090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 -- 172.21.15.67:0/3461599364 shutdown_connections 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 --2- 172.21.15.67:0/3461599364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7580a0370 0x7fb758098510 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 --2- 172.21.15.67:0/3461599364 >> v2:172.21.15.17:3300/0 conn(0x7fb758098140 0x7fb7581321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 -- 172.21.15.67:0/3461599364 >> 172.21.15.67:0/3461599364 conn(0x7fb7580a5780 msgr2=0x7fb7580a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 -- 172.21.15.67:0/3461599364 shutdown_connections 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.699+0000 7fb75e0e8640 1 -- 172.21.15.67:0/3461599364 wait complete. 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.700+0000 7fb75e0e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.700+0000 7fb75e0e8640 10 monclient: monmap: 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:08.713 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.700+0000 7fb75e0e8640 1 Processor -- start 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.700+0000 7fb75e0e8640 1 -- start start 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.700+0000 7fb75e0e8640 10 monclient: init 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb758098140 0x7fb75809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb758098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb758098a50 0x7fb758130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb758098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7580a0370 0x7fb75811ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7580a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758131c60 con 0x7fb758098140 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75811e1c0 con 0x7fb758098a50 2024-09-15T01:53:08.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb758141f30 con 0x7fb7580a0370 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb75e0e8640 10 monclient(hunting): authenticate will time out at 821.170145s 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb758098a50 0x7fb758130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb758098a50 0x7fb758130dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41098/0 (socket says 172.21.15.67:41098) 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 -- 172.21.15.67:0/1456161927 learned_addr learned my addr 172.21.15.67:0/1456161927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): get_auth_request con 0x7fb758098a50 auth_method 0 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb758098140 0x7fb75809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7580a0370 0x7fb75811ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): get_auth_request con 0x7fb758098140 auth_method 0 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): get_auth_request con 0x7fb7580a0370 auth_method 0 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb7577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb757fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient(hunting): handle_auth_done global_id 4642 payload 824 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 -- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7580a0370 msgr2=0x7fb75811ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7580a0370 0x7fb75811ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 -- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb758098140 msgr2=0x7fb75809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb758098140 0x7fb75809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 monclient: found mon.noname-b 2024-09-15T01:53:08.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.701+0000 7fb74effd640 1 -- 172.21.15.67:0/1456161927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740009390 con 0x7fb758098a50 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.703+0000 7fb74effd640 1 -- 172.21.15.67:0/1456161927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb758098a50 msgr2=0x7fb758130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.703+0000 7fb74effd640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb758098a50 0x7fb758130dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb740009360 tx=0x7fb740032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.703+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740031d40 con 0x7fb758098a50 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.703+0000 7fb74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: dump: 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: _finish_auth 0 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74003d440 con 0x7fb758098a50 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740031ea0 con 0x7fb758098a50 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 5 monclient: authenticate success, global_id 4642 2024-09-15T01:53:08.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 10 monclient: _renew_subs 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 1 -- 172.21.15.67:0/1456161927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75811e4a0 con 0x7fb758098a50 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb7567fc640 4 set_mon_vals no callback set 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 10 monclient: dump: 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 10 monclient: _renew_subs 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb75e0e8640 1 -- 172.21.15.67:0/1456161927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75811cf80 con 0x7fb758098a50 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.704+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb740039040 con 0x7fb758098a50 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb75e0e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb75e0e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb75e0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb75e0e8640 1 -- 172.21.15.67:0/1456161927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75811d170 con 0x7fb758098a50 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb74ffff640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb738038790 0x7fb73803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fb74006bd00 con 0x7fb758098a50 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb7577fe640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb738038790 0x7fb73803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.705+0000 7fb7577fe640 10 monclient: get_auth_request con 0x7fb738038790 auth_method 0 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.707+0000 7fb7577fe640 1 --2- 172.21.15.67:0/1456161927 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb738038790 0x7fb73803ac40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb748007ba0 tx=0x7fb748008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.709+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb74002f4d0 con 0x7fb758098a50 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.709+0000 7fb74ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.709+0000 7fb74ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:08.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:08.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.871+0000 7fb75e0e8640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T01:53:08.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.871+0000 7fb75e0e8640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T01:53:08.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.871+0000 7fb75e0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.871+0000 7fb75e0e8640 1 -- 172.21.15.67:0/1456161927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7fb758098ef0 con 0x7fb758098a50 2024-09-15T01:53:08.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.872+0000 7fb74ffff640 1 -- 172.21.15.67:0/1456161927 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb758098ef0 con 0x7fb758098a50 2024-09-15T01:53:08.875 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.872+0000 7fb74ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T01:53:08.875 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.872+0000 7fb74ffff640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-15T01:53:08.875 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-15T01:53:08.878 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:08.878 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-15T01:53:08.878 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:08.878 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set foo size 20 2024-09-15T01:53:08.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 10 monclient: get_monmap_and_config 2024-09-15T01:53:08.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 10 monclient: monmap: 2024-09-15T01:53:08.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:08.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:08.923867+0000 2024-09-15T01:53:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:08.923867+0000 2024-09-15T01:53:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 1 Processor -- start 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 1 -- start start 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.923+0000 7f54c4c78640 10 monclient: init 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54c0131b20 0x7f54c009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-c con 0x7f54c0131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:08.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54c00a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:08.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54c009f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:08.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c00a0040 con 0x7f54c00a0280 2024-09-15T01:53:08.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c009bff0 con 0x7f54c009f1e0 2024-09-15T01:53:08.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00a14a0 con 0x7f54c0131b20 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54c4c78640 10 monclient(hunting): authenticate will time out at 848.920346s 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54bdd9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f54c00a0280 0x7f54c0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58698/0 (socket says 172.21.15.17:58698) 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54bdd9b640 1 -- 172.21.15.17:0/135547739 learned_addr learned my addr 172.21.15.17:0/135547739 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:08.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 1 --2- 172.21.15.17:0/135547739 >> v2:172.21.15.67:3300/0 conn(0x7f54c0131b20 0x7f54c009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): get_auth_request con 0x7f54c0131b20 auth_method 0 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient(hunting): handle_auth_done global_id 4645 payload 274 2024-09-15T01:53:08.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 1 -- 172.21.15.17:0/135547739 >> v1:172.21.15.67:6789/0 conn(0x7f54c009f1e0 legacy=0x7f54c0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 1 -- 172.21.15.17:0/135547739 >> v1:172.21.15.17:6789/0 conn(0x7f54c00a0280 legacy=0x7f54c0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:08.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 1 monclient: found mon.noname-c 2024-09-15T01:53:08.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:08.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.924+0000 7f54be59c640 1 -- 172.21.15.17:0/135547739 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c0132600 con 0x7f54c0131b20 2024-09-15T01:53:08.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.927+0000 7f54be59c640 1 -- 172.21.15.17:0/135547739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c0131b20 msgr2=0x7f54c009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:08.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.927+0000 7f54be59c640 1 --2- 172.21.15.17:0/135547739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c0131b20 0x7f54c009eca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009850 tx=0x7f54b00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9188dc7e9a66d8e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 1 -- 172.21.15.17:0/135547739 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0032440 con 0x7f54c0131b20 2024-09-15T01:53:08.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: dump: 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: _finish_auth 0 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 1 -- 172.21.15.17:0/135547739 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b00325a0 con 0x7f54c0131b20 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 5 monclient: authenticate success, global_id 4645 2024-09-15T01:53:08.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 1 -- 172.21.15.17:0/135547739 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0032940 con 0x7f54c0131b20 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 4 set_mon_vals no callback set 2024-09-15T01:53:08.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54bd59a640 10 monclient: dump: 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 10 monclient: shutdown 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 -- 172.21.15.17:0/135547739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c0131b20 msgr2=0x7f54c009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 --2- 172.21.15.17:0/135547739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c0131b20 0x7f54c009eca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009850 tx=0x7f54b00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 -- 172.21.15.17:0/135547739 shutdown_connections 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 --2- 172.21.15.17:0/135547739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c0131b20 0x7f54c009eca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 -- 172.21.15.17:0/135547739 >> 172.21.15.17:0/135547739 conn(0x7f54c0071270 msgr2=0x7f54c00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:08.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 -- 172.21.15.17:0/135547739 shutdown_connections 2024-09-15T01:53:08.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.928+0000 7f54c4c78640 1 -- 172.21.15.17:0/135547739 wait complete. 2024-09-15T01:53:08.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.929+0000 7f54c4c78640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.929+0000 7f54c4c78640 10 monclient: monmap: 2024-09-15T01:53:08.956 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.957 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.929+0000 7f54c4c78640 1 Processor -- start 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.929+0000 7f54c4c78640 1 -- start start 2024-09-15T01:53:08.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.929+0000 7f54c4c78640 10 monclient: init 2024-09-15T01:53:08.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:08.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-a con 0x7f54c009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:08.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 0x7f54c009b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-b con 0x7f54c00a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 0x7f54c0096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): picked mon.noname-c con 0x7f54c0131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c009b960 con 0x7f54c009f1e0 2024-09-15T01:53:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00982b0 con 0x7f54c00a0280 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c009a5d0 con 0x7f54c0131b20 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54c4c78640 10 monclient(hunting): authenticate will time out at 848.926202s 2024-09-15T01:53:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 0x7f54c0096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54524/0 (socket says 172.21.15.17:54524) 2024-09-15T01:53:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 -- 172.21.15.17:0/1049459429 learned_addr learned my addr 172.21.15.17:0/1049459429 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): get_auth_request con 0x7f54c009f1e0 auth_method 0 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): get_auth_request con 0x7f54c0131b20 auth_method 0 2024-09-15T01:53:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 0x7f54c009b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): get_auth_request con 0x7f54c00a0280 auth_method 0 2024-09-15T01:53:08.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient(hunting): handle_auth_done global_id 44103 payload 824 2024-09-15T01:53:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 -- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 msgr2=0x7f54c0096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 0x7f54c0096dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 -- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 msgr2=0x7f54c009b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 0x7f54c009b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 monclient: found mon.noname-a 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54be59c640 1 -- 172.21.15.17:0/1049459429 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0009390 con 0x7f54c009f1e0 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bed9d640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 0x7f54c0096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.930+0000 7f54bdd9b640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 0x7f54c009b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:08.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54be59c640 1 -- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 msgr2=0x7f54c009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:08.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54be59c640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009850 tx=0x7f54b0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0031940 con 0x7f54c009f1e0 2024-09-15T01:53:08.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: dump: 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 5 monclient: authenticate success, global_id 44103 2024-09-15T01:53:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 10 monclient: _renew_subs 2024-09-15T01:53:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b0031aa0 con 0x7f54c009f1e0 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0099400 con 0x7f54c009f1e0 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0031e40 con 0x7f54c009f1e0 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:08.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54b67fc640 10 monclient: dump: 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.974 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54bcd99640 4 set_mon_vals no callback set 2024-09-15T01:53:08.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 10 monclient: _renew_subs 2024-09-15T01:53:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.933+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c009a8b0 con 0x7f54c009f1e0 2024-09-15T01:53:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f54b0039070 con 0x7f54c009f1e0 2024-09-15T01:53:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54c4c78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54c4c78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54c4c78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c00724a0 con 0x7f54c009f1e0 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54b67fc640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 0x7f54a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f54b006bce0 con 0x7f54c009f1e0 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54bdd9b640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 0x7f54a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.934+0000 7f54bdd9b640 10 monclient: get_auth_request con 0x7f54a0038740 auth_method 0 2024-09-15T01:53:08.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.936+0000 7f54bdd9b640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 0x7f54a003abf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007fd0 tx=0x7f54a8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.938+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f54b0071020 con 0x7f54c009f1e0 2024-09-15T01:53:08.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.938+0000 7f54b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:08.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:08.938+0000 7f54b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:08.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.953+0000 7ff475d00640 10 monclient: get_monmap_and_config 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.953+0000 7ff475d00640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.953+0000 7ff475d00640 10 monclient: monmap: 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:08.954696+0000 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:08.954696+0000 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.989 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 Processor -- start 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 -- start start 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient: init 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff470068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:08.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470069190 0x7ff470069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-c con 0x7ff470069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff470137150 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff470096c10 con 0x7ff470068dc0 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff470132960 con 0x7ff470137150 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700a14d0 con 0x7ff470069190 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff475d00640 10 monclient(hunting): authenticate will time out at 821.423835s 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470069190 0x7ff470069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470069190 0x7ff470069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41102/0 (socket says 172.21.15.67:41102) 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff470137150 0x7ff470139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57130/0 (socket says 172.21.15.67:57130) 2024-09-15T01:53:08.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 1 -- 172.21.15.67:0/3513409961 learned_addr learned my addr 172.21.15.67:0/3513409961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 10 monclient(hunting): get_auth_request con 0x7ff470069190 auth_method 0 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.954+0000 7ff46f7fe640 1 --1- 172.21.15.67:0/3513409961 >> v1:172.21.15.17:6789/0 conn(0x7ff470068dc0 0x7ff470132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:53758/0 (socket says 172.21.15.67:53758) 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient(hunting): handle_auth_done global_id 4651 payload 274 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 1 -- 172.21.15.67:0/3513409961 >> v1:172.21.15.67:6789/0 conn(0x7ff470137150 legacy=0x7ff470139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 1 -- 172.21.15.67:0/3513409961 >> v1:172.21.15.17:6789/0 conn(0x7ff470068dc0 legacy=0x7ff470132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:08.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 1 monclient: found mon.noname-c 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.955+0000 7ff46effd640 1 -- 172.21.15.67:0/3513409961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47009ff20 con 0x7ff470069190 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.957+0000 7ff46effd640 1 -- 172.21.15.67:0/3513409961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 msgr2=0x7ff470069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46effd640 1 --2- 172.21.15.67:0/3513409961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470069590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff464009910 tx=0x7ff4640091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a21194a7d82cb7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 1 -- 172.21.15.67:0/3513409961 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464032440 con 0x7ff470069190 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: dump: 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 1 -- 172.21.15.67:0/3513409961 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4640325a0 con 0x7ff470069190 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 5 monclient: authenticate success, global_id 4651 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 1 -- 172.21.15.67:0/3513409961 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464032940 con 0x7ff470069190 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 4 set_mon_vals no callback set 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff46e7fc640 10 monclient: dump: 2024-09-15T01:53:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 10 monclient: shutdown 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 -- 172.21.15.67:0/3513409961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 msgr2=0x7ff470069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 --2- 172.21.15.67:0/3513409961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470069590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff464009910 tx=0x7ff4640091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 -- 172.21.15.67:0/3513409961 shutdown_connections 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 --2- 172.21.15.67:0/3513409961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470069590 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 -- 172.21.15.67:0/3513409961 >> 172.21.15.67:0/3513409961 conn(0x7ff4700711b0 msgr2=0x7ff4700a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 -- 172.21.15.67:0/3513409961 shutdown_connections 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.958+0000 7ff475d00640 1 -- 172.21.15.67:0/3513409961 wait complete. 2024-09-15T01:53:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.959+0000 7ff475d00640 10 monclient: build_initial_monmap 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.959+0000 7ff475d00640 10 monclient: monmap: 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.959+0000 7ff475d00640 1 Processor -- start 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.959+0000 7ff475d00640 1 -- start start 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.959+0000 7ff475d00640 10 monclient: init 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff470068dc0 0x7ff4701417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-c con 0x7ff470068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-b con 0x7ff470069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff470137150 0x7ff470097d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): picked mon.noname-a con 0x7ff470137150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:08.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700a0890 con 0x7ff470137150 2024-09-15T01:53:09.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47012fb80 con 0x7ff470069190 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff470099e40 con 0x7ff470068dc0 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff475d00640 10 monclient(hunting): authenticate will time out at 821.429283s 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470141d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41116/0 (socket says 172.21.15.67:41116) 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 -- 172.21.15.67:0/2830014431 learned_addr learned my addr 172.21.15.67:0/2830014431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): get_auth_request con 0x7ff470069190 auth_method 0 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff470068dc0 0x7ff4701417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): get_auth_request con 0x7ff470068dc0 auth_method 0 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff470137150 0x7ff470097d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): get_auth_request con 0x7ff470137150 auth_method 0 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient(hunting): handle_auth_done global_id 4654 payload 824 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 -- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff470068dc0 msgr2=0x7ff4701417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff470068dc0 0x7ff4701417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 -- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff470137150 msgr2=0x7ff470097d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff470137150 0x7ff470097d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.960+0000 7ff46effd640 1 -- 172.21.15.67:0/2830014431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff464009450 con 0x7ff470069190 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.962+0000 7ff46effd640 1 -- 172.21.15.67:0/2830014431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 msgr2=0x7ff470141d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.962+0000 7ff46effd640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470069190 0x7ff470141d30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff46402a720 tx=0x7ff464032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464031940 con 0x7ff470069190 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: dump: 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.005 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff464031aa0 con 0x7ff470069190 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 5 monclient: authenticate success, global_id 4654 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 10 monclient: _renew_subs 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 1 -- 172.21.15.67:0/2830014431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff470099930 con 0x7ff470069190 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464031e60 con 0x7ff470069190 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 10 monclient: dump: 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 10 monclient: _renew_subs 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff46dffb640 4 set_mon_vals no callback set 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff475d00640 1 -- 172.21.15.67:0/2830014431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47009a500 con 0x7ff470069190 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.963+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff464039070 con 0x7ff470069190 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff475d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff475d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff475d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff475d00640 1 -- 172.21.15.67:0/2830014431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47004bf60 con 0x7ff470069190 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff4577fe640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff45c038790 0x7ff45c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7ff46406c3a0 con 0x7ff470069190 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff46f7fe640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff45c038790 0x7ff45c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.964+0000 7ff46f7fe640 10 monclient: get_auth_request con 0x7ff45c038790 auth_method 0 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.966+0000 7ff46f7fe640 1 --2- 172.21.15.67:0/2830014431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff45c038790 0x7ff45c03ac40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff460007290 tx=0x7ff460007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.968+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff46403c410 con 0x7ff470069190 2024-09-15T01:53:09.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.968+0000 7ff4577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:08.968+0000 7ff4577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.130+0000 7ff475d00640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.130+0000 7ff475d00640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.130+0000 7ff475d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.130+0000 7ff475d00640 1 -- 172.21.15.67:0/2830014431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7ff47004bf60 con 0x7ff470069190 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.131+0000 7ff4577fe640 1 -- 172.21.15.67:0/2830014431 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff4640370a0 con 0x7ff470069190 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.131+0000 7ff4577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T01:53:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.131+0000 7ff4577fe640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-15T01:53:09.134 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-15T01:53:09.136 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool delete foo 2024-09-15T01:53:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo 2024-09-15T01:53:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54c4c78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54c4c78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54c4c78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f54c00724a0 con 0x7f54c009f1e0 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54b67fc640 1 -- 172.21.15.17:0/1049459429 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f54b0040020 con 0x7f54c009f1e0 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.170+0000 7f54b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:09.173 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 msgr2=0x7f54a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 0x7f54a003abf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007fd0 tx=0x7f54a8007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 10 monclient: shutdown 2024-09-15T01:53:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 msgr2=0x7f54c009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009850 tx=0x7f54b0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 shutdown_connections 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54a0038740 0x7f54a003abf0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f54c0131b20 0x7f54c0096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c00a0280 0x7f54c009b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 --2- 172.21.15.17:0/1049459429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f54c009f1e0 0x7f54c009aee0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 >> 172.21.15.17:0/1049459429 conn(0x7f54c0071270 msgr2=0x7f54c0129c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 shutdown_connections 2024-09-15T01:53:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.173+0000 7f54c4c78640 1 -- 172.21.15.17:0/1049459429 wait complete. 2024-09-15T01:53:09.185 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:53:09.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:53:09.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:53:09.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.210+0000 7fb9fa35d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.210+0000 7fb9fa35d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.210+0000 7fb9fa35d640 10 monclient: monmap: 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:09.211474+0000 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:09.211474+0000 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.213 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.214 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.214 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.214 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.214 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 Processor -- start 2024-09-15T01:53:09.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 -- start start 2024-09-15T01:53:09.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient: init 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 0x7fb9f4132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f4068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9f4069190 0x7fb9f4069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f4069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9f412e960 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f4096c10 con 0x7fb9f412e960 2024-09-15T01:53:09.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f40a14d0 con 0x7fb9f4068dc0 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f409ff20 con 0x7fb9f4069190 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9fa35d640 10 monclient(hunting): authenticate will time out at 821.680584s 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9f4069190 0x7fb9f4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f88fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9f412e960 0x7fb9f412ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57144/0 (socket says 172.21.15.67:57144) 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f88fa640 1 -- 172.21.15.67:0/2237910637 learned_addr learned my addr 172.21.15.67:0/2237910637 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): get_auth_request con 0x7fb9f4069190 auth_method 0 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f3fff640 1 --2- 172.21.15.67:0/2237910637 >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 0x7fb9f4132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f3fff640 10 monclient(hunting): get_auth_request con 0x7fb9f4068dc0 auth_method 0 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.211+0000 7fb9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_done global_id 4657 payload 274 2024-09-15T01:53:09.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2237910637 >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 msgr2=0x7fb9f4132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 1 --2- 172.21.15.67:0/2237910637 >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 0x7fb9f4132250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2237910637 >> v1:172.21.15.67:6789/0 conn(0x7fb9f412e960 legacy=0x7fb9f412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 1 monclient: found mon.noname-c 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2237910637 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40a0180 con 0x7fb9f4069190 2024-09-15T01:53:09.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.212+0000 7fb9f3fff640 1 --2- 172.21.15.67:0/2237910637 >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 0x7fb9f4132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:09.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.214+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2237910637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 msgr2=0x7fb9f4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.214+0000 7fb9f37fe640 1 --2- 172.21.15.67:0/2237910637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f4069590 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb9f406a680 tx=0x7fb9e80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728dd327b00dfdaa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.214+0000 7fb9f2ffd640 1 -- 172.21.15.67:0/2237910637 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e802ae00 con 0x7fb9f4069190 2024-09-15T01:53:09.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.214+0000 7fb9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: dump: 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.228 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 1 -- 172.21.15.67:0/2237910637 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e8032440 con 0x7fb9f4069190 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 5 monclient: authenticate success, global_id 4657 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 1 -- 172.21.15.67:0/2237910637 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e80327e0 con 0x7fb9f4069190 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 4 set_mon_vals no callback set 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9f2ffd640 10 monclient: dump: 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.230 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 10 monclient: shutdown 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2237910637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 msgr2=0x7fb9f4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 --2- 172.21.15.67:0/2237910637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f4069590 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb9f406a680 tx=0x7fb9e80090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2237910637 shutdown_connections 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 --2- 172.21.15.67:0/2237910637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f4069590 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 --2- 172.21.15.67:0/2237910637 >> v2:172.21.15.17:3300/0 conn(0x7fb9f4068dc0 0x7fb9f4132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2237910637 >> 172.21.15.67:0/2237910637 conn(0x7fb9f40711b0 msgr2=0x7fb9f40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2237910637 shutdown_connections 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.215+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2237910637 wait complete. 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.216+0000 7fb9fa35d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.216+0000 7fb9fa35d640 10 monclient: monmap: 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.216+0000 7fb9fa35d640 1 Processor -- start 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.216+0000 7fb9fa35d640 1 -- start start 2024-09-15T01:53:09.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.216+0000 7fb9fa35d640 10 monclient: init 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9f4068dc0 0x7fb9f4140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f4068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f412db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f4069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9f412e960 0x7fb9f4097e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f412e960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4099ff0 con 0x7fb9f4068dc0 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4141530 con 0x7fb9f4069190 2024-09-15T01:53:09.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40a0890 con 0x7fb9f412e960 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9fa35d640 10 monclient(hunting): authenticate will time out at 821.686134s 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f412db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f412db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41124/0 (socket says 172.21.15.67:41124) 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2916333962 learned_addr learned my addr 172.21.15.67:0/2916333962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): get_auth_request con 0x7fb9f4069190 auth_method 0 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9f4068dc0 0x7fb9f4140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9f412e960 0x7fb9f4097e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): get_auth_request con 0x7fb9f4068dc0 auth_method 0 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): get_auth_request con 0x7fb9f412e960 auth_method 0 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f88fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient(hunting): handle_auth_done global_id 4660 payload 824 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9f412e960 msgr2=0x7fb9f4097e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9f412e960 0x7fb9f4097e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9f4068dc0 msgr2=0x7fb9f4140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9f4068dc0 0x7fb9f4140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.217+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2916333962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8009390 con 0x7fb9f4069190 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9f37fe640 1 -- 172.21.15.67:0/2916333962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 msgr2=0x7fb9f412db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9f37fe640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f4069190 0x7fb9f412db00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8035040 tx=0x7fb9e8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8031d40 con 0x7fb9f4069190 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: dump: 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.237 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9fa35d640 5 monclient: authenticate success, global_id 4660 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e803d440 con 0x7fb9f4069190 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8031ea0 con 0x7fb9f4069190 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9fa35d640 10 monclient: _renew_subs 2024-09-15T01:53:09.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9fa35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2916333962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f40a4780 con 0x7fb9f4069190 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9dbfff640 10 monclient: dump: 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.239 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9f27fc640 4 set_mon_vals no callback set 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.219+0000 7fb9fa35d640 10 monclient: _renew_subs 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2916333962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f409b5e0 con 0x7fb9f4069190 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb9e8039040 con 0x7fb9f4069190 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9dbfff640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb9e0038740 0x7fb9e003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.220+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2916333962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f4141e80 con 0x7fb9f4069190 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.221+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fb9e806bb00 con 0x7fb9f4069190 2024-09-15T01:53:09.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.221+0000 7fb9f3fff640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb9e0038740 0x7fb9e003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.221+0000 7fb9f3fff640 10 monclient: get_auth_request con 0x7fb9e0038740 auth_method 0 2024-09-15T01:53:09.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.223+0000 7fb9f3fff640 1 --2- 172.21.15.67:0/2916333962 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb9e0038740 0x7fb9e003abf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4006e70 tx=0x7fb9e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.225+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb9e8042020 con 0x7fb9f4069190 2024-09-15T01:53:09.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.225+0000 7fb9dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.225+0000 7fb9dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.281+0000 7f274b309640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.281+0000 7f274b309640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.281+0000 7f274b309640 10 monclient: monmap: 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.284 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:09.282470+0000 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:09.282470+0000 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 1 Processor -- start 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 1 -- start start 2024-09-15T01:53:09.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient: init 2024-09-15T01:53:09.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:53:09.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 0x7f274406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient(hunting): picked mon.noname-b con 0x7f274409c500 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f274409c8d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f274406f280 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.282+0000 7f274b309640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274409ba20 con 0x7f274409c8d0 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f274b309640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274406f670 con 0x7f274406f280 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f274b309640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274409e780 con 0x7f274409c500 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f274b309640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f274b309640 10 monclient(hunting): authenticate will time out at 849.279071s 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 0x7f274406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27488a4640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f274409c8d0 0x7f274409d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58710/0 (socket says 172.21.15.17:58710) 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27488a4640 1 -- 172.21.15.17:0/898429783 learned_addr learned my addr 172.21.15.17:0/898429783 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:09.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): get_auth_request con 0x7f274409c500 auth_method 0 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient(hunting): handle_auth_done global_id 34277 payload 274 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 1 -- 172.21.15.17:0/898429783 >> v1:172.21.15.67:6789/0 conn(0x7f274406f280 legacy=0x7f274409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 1 -- 172.21.15.17:0/898429783 >> v1:172.21.15.17:6789/0 conn(0x7f274409c8d0 legacy=0x7f274409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.283+0000 7f27490a5640 1 -- 172.21.15.17:0/898429783 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274409f6c0 con 0x7f274409c500 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f27490a5640 1 -- 172.21.15.17:0/898429783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 msgr2=0x7f274406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f27490a5640 1 --2- 172.21.15.17:0/898429783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 0x7f274406ed40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2740009850 tx=0x7f27400090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1a79043796c2d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 1 -- 172.21.15.17:0/898429783 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740032440 con 0x7f274409c500 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: dump: 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.298 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 1 -- 172.21.15.17:0/898429783 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27400325a0 con 0x7f274409c500 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 1 -- 172.21.15.17:0/898429783 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740032940 con 0x7f274409c500 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 5 monclient: authenticate success, global_id 34277 2024-09-15T01:53:09.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 4 set_mon_vals no callback set 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f273bfff640 10 monclient: dump: 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 10 monclient: shutdown 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 1 -- 172.21.15.17:0/898429783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 msgr2=0x7f274406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.286+0000 7f274b309640 1 --2- 172.21.15.17:0/898429783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 0x7f274406ed40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2740009850 tx=0x7f27400090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 -- 172.21.15.17:0/898429783 shutdown_connections 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 --2- 172.21.15.17:0/898429783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c500 0x7f274406ed40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 -- 172.21.15.17:0/898429783 >> 172.21.15.17:0/898429783 conn(0x7f27440712f0 msgr2=0x7f27440a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 -- 172.21.15.17:0/898429783 shutdown_connections 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 -- 172.21.15.17:0/898429783 wait complete. 2024-09-15T01:53:09.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 10 monclient: monmap: 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 Processor -- start 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 1 -- start start 2024-09-15T01:53:09.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.287+0000 7f274b309640 10 monclient: init 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274406f280 0x7f274413dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): picked mon.noname-b con 0x7f274406f280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 0x7f274413d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): picked mon.noname-a con 0x7f274409c500 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 0x7f2744141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): picked mon.noname-c con 0x7f274409c8d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274409cf00 con 0x7f274409c500 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2744098d60 con 0x7f274406f280 2024-09-15T01:53:09.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274413e810 con 0x7f274409c8d0 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f274b309640 10 monclient(hunting): authenticate will time out at 849.284593s 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 0x7f2744141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 0x7f274413d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 0x7f2744141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40490/0 (socket says 172.21.15.17:40490) 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 1 -- 172.21.15.17:0/2404530695 learned_addr learned my addr 172.21.15.17:0/2404530695 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): get_auth_request con 0x7f274409c500 auth_method 0 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): get_auth_request con 0x7f274409c8d0 auth_method 0 2024-09-15T01:53:09.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274406f280 0x7f274413dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): get_auth_request con 0x7f274406f280 auth_method 0 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27488a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27498a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.288+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 10 monclient(hunting): handle_auth_done global_id 44115 payload 824 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 msgr2=0x7f2744141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 0x7f2744141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274406f280 msgr2=0x7f274413dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274406f280 0x7f274413dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 monclient: found mon.noname-a 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27488a4640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2740009390 con 0x7f274409c500 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.289+0000 7f27498a6640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f274409c8d0 0x7f2744141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27488a4640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 msgr2=0x7f274413d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27488a4640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 0x7f274413d2f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f273400c490 tx=0x7f2734007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734005cb0 con 0x7f274409c500 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: dump: 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.307 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734005e30 con 0x7f274409c500 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 5 monclient: authenticate success, global_id 44115 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734007320 con 0x7f274409c500 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f27397fa640 10 monclient: dump: 2024-09-15T01:53:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 10 monclient: _renew_subs 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f273b7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2744122390 con 0x7f274409c500 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 10 monclient: _renew_subs 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.291+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274413d830 con 0x7f274409c500 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f274b309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f273401b3b0 con 0x7f274409c500 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f274b309640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 0x7f27441416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2744122810 con 0x7f27441214c0 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 0x7f27441416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient: get_auth_request con 0x7f27441214c0 auth_method 0 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27397fa640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2728038740 0x7f272803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f273404b280 con 0x7f274409c500 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 10 monclient(hunting): handle_auth_done global_id 44118 payload 824 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27490a5640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f274413da50 con 0x7f27441214c0 2024-09-15T01:53:09.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27498a6640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2728038740 0x7f272803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.292+0000 7f27498a6640 10 monclient: get_auth_request con 0x7f2728038740 auth_method 0 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27490a5640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 msgr2=0x7f27441416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27490a5640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 0x7f27441416d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740034000 tx=0x7f2740032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740031a70 con 0x7f27441214c0 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 10 monclient: dump: 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.312 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27498a6640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2728038740 0x7f272803abf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f274413e1c0 tx=0x7f273c009400 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2740031bd0 con 0x7f27441214c0 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 msgr2=0x7f27441416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 0x7f27441416d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740034000 tx=0x7f2740032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 msgr2=0x7f27441416d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.295+0000 7f27397fa640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27441214c0 0x7f27441416d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740034000 tx=0x7f2740032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f274b309640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f274b309640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 0x7f2744141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f274b309640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2744063b90 con 0x7f2744069c90 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 0x7f2744141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient: get_auth_request con 0x7f2744069c90 auth_method 0 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 10 monclient(hunting): handle_auth_done global_id 44121 payload 824 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.301+0000 7f27490a5640 1 -- 172.21.15.17:0/2404530695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f27440724a0 con 0x7f2744069c90 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27490a5640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 msgr2=0x7f2744141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27490a5640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 0x7f2744141c10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2740009850 tx=0x7f274003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274004e070 con 0x7f2744069c90 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.303+0000 7f27397fa640 10 monclient: dump: 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.317 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7f274003c9b0 con 0x7f2744069c90 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 msgr2=0x7f2744141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 0x7f2744141c10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2740009850 tx=0x7f274003d000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 msgr2=0x7f2744141c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27397fa640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2744069c90 0x7f2744141c10 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2740009850 tx=0x7f274003d000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.320 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.322 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.323 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:09.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:09.326 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.328 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.330 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:09.331 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2728038740 msgr2=0x7f272803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2728038740 0x7f272803abf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f274413e1c0 tx=0x7f273c009400 comp rx=0 tx=0).stop 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 10 monclient: shutdown 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 msgr2=0x7f274413d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 0x7f274413d2f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f273400c490 tx=0x7f2734007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f27498a6640 1 -- 172.21.15.17:0/2404530695 reap_dead start 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 shutdown_connections 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 --2- 172.21.15.17:0/2404530695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f274409c500 0x7f274413d2f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 >> 172.21.15.17:0/2404530695 conn(0x7f27440712f0 msgr2=0x7f27440984b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 shutdown_connections 2024-09-15T01:53:09.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.304+0000 7f274b309640 1 -- 172.21.15.17:0/2404530695 wait complete. 2024-09-15T01:53:09.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:09.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:53:09.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:53:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.388+0000 7fb9fa35d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T01:53:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.388+0000 7fb9fa35d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T01:53:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.388+0000 7fb9fa35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.388+0000 7fb9fa35d640 1 -- 172.21.15.67:0/2916333962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7fb9f404bf60 con 0x7fb9f4069190 2024-09-15T01:53:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.389+0000 7fb9dbfff640 1 -- 172.21.15.67:0/2916333962 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 274+0+0 (secure 0 0 0) 0x7fb9e802f2a0 con 0x7fb9f4069190 2024-09-15T01:53:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.389+0000 7fb9dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T01:53:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.389+0000 7fb9dbfff640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T01:53:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T01:53:09.395 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:09.395 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-15T01:53:09.395 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:09.395 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo foo 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.429+0000 7ff4b2f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.429+0000 7ff4b2f5d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.429+0000 7ff4b2f5d640 10 monclient: monmap: 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:09.430757+0000 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:09.430757+0000 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.432 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.433 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.433 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.433 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.433 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.430+0000 7ff4b2f5d640 1 Processor -- start 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.430+0000 7ff4b2f5d640 1 -- start start 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.430+0000 7ff4b2f5d640 10 monclient: init 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 0x7ff4ac1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4ac068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4ac0692f0 0x7ff4ac0696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4ac0692f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 0x7ff4ac12ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4ac12e900 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ac0a1950 con 0x7ff4ac0692f0 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ac0a1670 con 0x7ff4ac12e900 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0a0020 con 0x7ff4ac068f20 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b2f5d640 10 monclient(hunting): authenticate will time out at 849.427237s 2024-09-15T01:53:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4ac0692f0 0x7ff4ac0696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 0x7ff4ac1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4ac0692f0 0x7ff4ac0696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54548/0 (socket says 172.21.15.17:54548) 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 -- 172.21.15.17:0/3848431161 learned_addr learned my addr 172.21.15.17:0/3848431161 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): get_auth_request con 0x7ff4ac068f20 auth_method 0 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): get_auth_request con 0x7ff4ac0692f0 auth_method 0 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 1 --2- 172.21.15.17:0/3848431161 >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 0x7ff4ac12ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 10 monclient(hunting): get_auth_request con 0x7ff4ac12e900 auth_method 0 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient(hunting): handle_auth_done global_id 44124 payload 274 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 -- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 msgr2=0x7ff4ac1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 0x7ff4ac1321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 -- 172.21.15.17:0/3848431161 >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 msgr2=0x7ff4ac12ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 --2- 172.21.15.17:0/3848431161 >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 0x7ff4ac12ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 monclient: found mon.noname-a 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4abfff640 1 -- 172.21.15.17:0/3848431161 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0a1390 con 0x7ff4ac0692f0 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b0cf9640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 0x7ff4ac1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:09.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.431+0000 7ff4b14fa640 1 --2- 172.21.15.17:0/3848431161 >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 0x7ff4ac12ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4abfff640 1 -- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac0692f0 msgr2=0x7ff4ac0696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4abfff640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac0692f0 0x7ff4ac0696c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0a2f70 tx=0x7ff4a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ea4205d382f29a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 1 -- 172.21.15.17:0/3848431161 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0032440 con 0x7ff4ac0692f0 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: dump: 2024-09-15T01:53:09.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4b2f5d640 5 monclient: authenticate success, global_id 44124 2024-09-15T01:53:09.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 1 -- 172.21.15.17:0/3848431161 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a00325a0 con 0x7ff4ac0692f0 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4b2f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4b2f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 1 -- 172.21.15.17:0/3848431161 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0032940 con 0x7ff4ac0692f0 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4b2f5d640 4 set_mon_vals no callback set 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.434+0000 7ff4ab7fe640 10 monclient: dump: 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.448 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 10 monclient: shutdown 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac0692f0 msgr2=0x7ff4ac0696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac0692f0 0x7ff4ac0696c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0a2f70 tx=0x7ff4a00091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/3848431161 shutdown_connections 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/3848431161 >> v2:172.21.15.67:3300/0 conn(0x7ff4ac12e900 0x7ff4ac12ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac0692f0 0x7ff4ac0696c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/3848431161 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac068f20 0x7ff4ac1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/3848431161 >> 172.21.15.17:0/3848431161 conn(0x7ff4ac070fb0 msgr2=0x7ff4ac0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/3848431161 shutdown_connections 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/3848431161 wait complete. 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 10 monclient: monmap: 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.435+0000 7ff4b2f5d640 1 Processor -- start 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 -- start start 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient: init 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4ac068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0692f0 0x7ff4ac12dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4ac0692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ac12e900 0x7ff4ac12e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4ac12e900 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac09a0f0 con 0x7ff4ac068f20 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac142030 con 0x7ff4ac12e900 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac130c00 con 0x7ff4ac0692f0 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b2f5d640 10 monclient(hunting): authenticate will time out at 849.432669s 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0692f0 0x7ff4ac12dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54564/0 (socket says 172.21.15.17:54564) 2024-09-15T01:53:09.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 1 -- 172.21.15.17:0/709506067 learned_addr learned my addr 172.21.15.17:0/709506067 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): get_auth_request con 0x7ff4ac0692f0 auth_method 0 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): get_auth_request con 0x7ff4ac068f20 auth_method 0 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ac12e900 0x7ff4ac12e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): get_auth_request con 0x7ff4ac12e900 auth_method 0 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.436+0000 7ff4b14fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 10 monclient(hunting): handle_auth_done global_id 44127 payload 824 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0692f0 msgr2=0x7ff4ac12dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0692f0 0x7ff4ac12dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ac12e900 msgr2=0x7ff4ac12e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ac12e900 0x7ff4ac12e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 monclient: found mon.noname-a 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4b0cf9640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a4009a20 con 0x7ff4ac068f20 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.437+0000 7ff4abfff640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0692f0 0x7ff4ac12dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:09.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b0cf9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 msgr2=0x7ff4ac141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b0cf9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c170 tx=0x7ff4a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4014690 con 0x7ff4ac068f20 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: dump: 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 5 monclient: authenticate success, global_id 44127 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a400f040 con 0x7ff4ac068f20 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 10 monclient: _renew_subs 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a0009450 con 0x7ff4ac068f20 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4014a20 con 0x7ff4ac068f20 2024-09-15T01:53:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4a8ff9640 10 monclient: dump: 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 10 monclient: _renew_subs 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac09b6a0 con 0x7ff4ac068f20 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.439+0000 7ff4aaffd640 4 set_mon_vals no callback set 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4b0cf9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c170 tx=0x7ff4a400b040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7ff4a401d400 mgrmap(e 5) v1 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff4a401d400 con 0x7ff4ac068f20 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4b2f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 0x7ff4ac098380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac063b90 con 0x7ff4ac097fb0 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4a8ff9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff498038740 0x7ff49803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 0x7ff4ac098380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient: get_auth_request con 0x7ff4ac097fb0 auth_method 0 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.440+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.441+0000 7ff4b14fa640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff498038740 0x7ff49803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.441+0000 7ff4b14fa640 10 monclient: get_auth_request con 0x7ff498038740 auth_method 0 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.441+0000 7ff4abfff640 10 monclient(hunting): handle_auth_done global_id 34289 payload 824 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.441+0000 7ff4abfff640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4ac09bb10 con 0x7ff4ac097fb0 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4abfff640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 msgr2=0x7ff4ac098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4abfff640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 0x7ff4ac098380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0034000 tx=0x7ff4a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4b14fa640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff498038740 0x7ff49803abf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff49c007d60 tx=0x7ff49c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0032440 con 0x7ff4ac097fb0 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 10 monclient: dump: 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff4a003d690 con 0x7ff4ac097fb0 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 msgr2=0x7ff4ac098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 0x7ff4ac098380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0034000 tx=0x7ff4a0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 msgr2=0x7ff4ac098380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.443+0000 7ff4a8ff9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac097fb0 0x7ff4ac098380 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0034000 tx=0x7ff4a0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4b2f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:09.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 0x7ff4ac09c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4b2f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0724a0 con 0x7ff4ac0988c0 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 0x7ff4ac09c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient: get_auth_request con 0x7ff4ac0988c0 auth_method 0 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 10 monclient(hunting): handle_auth_done global_id 34292 payload 824 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.449+0000 7ff4abfff640 1 -- 172.21.15.17:0/709506067 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff4ac12fe00 con 0x7ff4ac0988c0 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4abfff640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 msgr2=0x7ff4ac09c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4abfff640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 0x7ff4ac09c550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac063960 tx=0x7ff4a003d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0046c20 con 0x7ff4ac0988c0 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 10 monclient: dump: 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.466 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7ff4a0046d80 con 0x7ff4ac0988c0 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 msgr2=0x7ff4ac09c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 0x7ff4ac09c550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac063960 tx=0x7ff4a003d000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 msgr2=0x7ff4ac09c550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4ac0988c0 0x7ff4ac09c550 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac063960 tx=0x7ff4a003d000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:09.468 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.469 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.471 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:09.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.478 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.479 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.480 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:09.481 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:09.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff498038740 msgr2=0x7ff49803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff498038740 0x7ff49803abf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff49c007d60 tx=0x7ff49c008040 comp rx=0 tx=0).stop 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 10 monclient: shutdown 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 msgr2=0x7ff4ac141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 --2- 172.21.15.17:0/709506067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4ac068f20 0x7ff4ac141170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c170 tx=0x7ff4a400b040 comp rx=0 tx=0).stop 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b14fa640 1 -- 172.21.15.17:0/709506067 reap_dead start 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b14fa640 1 -- 172.21.15.17:0/709506067 reap_dead start 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4a8ff9640 1 -- 172.21.15.17:0/709506067 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (unknown 0 0 0) 0x7ff4a404cfb0 con 0x7ff4ac068f20 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 shutdown_connections 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 >> 172.21.15.17:0/709506067 conn(0x7ff4ac070fb0 msgr2=0x7ff4ac130e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.452+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 shutdown_connections 2024-09-15T01:53:09.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.453+0000 7ff4b2f5d640 1 -- 172.21.15.17:0/709506067 wait complete. 2024-09-15T01:53:09.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:09.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:53:09.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.470+0000 7f969151d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.470+0000 7f969151d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.470+0000 7f969151d640 10 monclient: monmap: 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:09.471521+0000 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:09.471521+0000 2024-09-15T01:53:09.495 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 Processor -- start 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 -- start start 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient: init 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T01:53:09.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f968c098140 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f968c0a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f968c098c20 0x7f968c13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): picked mon.noname-a con 0x7f968c098c20 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c0a0b80 con 0x7f968c098140 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c1328f0 con 0x7f968c0a0370 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c0a00b0 con 0x7f968c098c20 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f969151d640 10 monclient(hunting): authenticate will time out at 821.940683s 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f9683fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f968c0a0370 0x7f968c098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57150/0 (socket says 172.21.15.67:57150) 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 learned_addr learned my addr 172.21.15.67:0/2472591176 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.471+0000 7f968b7fe640 1 --2- 172.21.15.67:0/2472591176 >> v2:172.21.15.17:3300/0 conn(0x7f968c098c20 0x7f968c13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.472+0000 7f968b7fe640 10 monclient(hunting): get_auth_request con 0x7f968c098c20 auth_method 0 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.472+0000 7f968b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.472+0000 7f968b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.472+0000 7f968b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 >> v1:172.21.15.67:6789/0 conn(0x7f968c0a0370 legacy=0x7f968c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746936477 0 0) 0x7f968c1328f0 con 0x7f968c0a0370 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): my global_id is 4672 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96780034e0 con 0x7f968c0a0370 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 1 -- 172.21.15.67:0/2472591176 >> v1:172.21.15.17:6789/0 conn(0x7f968c098140 legacy=0x7f968c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219852772 0 0) 0x7f96780034e0 con 0x7f968c0a0370 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 >> v2:172.21.15.17:3300/0 conn(0x7f968c098c20 msgr2=0x7f968c13a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 --2- 172.21.15.67:0/2472591176 >> v2:172.21.15.17:3300/0 conn(0x7f968c098c20 0x7f968c13a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 >> v1:172.21.15.17:6789/0 conn(0x7f968c098140 legacy=0x7f968c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c071390 con 0x7f968c0a0370 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f969151d640 5 monclient: authenticate success, global_id 4672 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f9674001720 con 0x7f968c0a0370 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f969151d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 1 --2- 172.21.15.67:0/2472591176 >> v2:172.21.15.17:3300/0 conn(0x7f968c098c20 0x7f968c13a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: dump: 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.500 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 >> v1:172.21.15.67:6789/0 conn(0x7f968c0a0370 legacy=0x7f968c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9678003b20 0x7f9678003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): picked mon.c con 0x7f9678003b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 0x7f9678004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): picked mon.b con 0x7f9678004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678005330 0x7f967800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): picked mon.a con 0x7f9678005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0a00b0 con 0x7f9678005330 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967800c2d0 con 0x7f9678004780 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967800c490 con 0x7f9678003b20 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f969151d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 0x7f9678004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): get_auth_request con 0x7f9678004780 auth_method 0 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678005330 0x7f967800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9678003b20 0x7f9678003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): get_auth_request con 0x7f9678005330 auth_method 0 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): get_auth_request con 0x7f9678003b20 auth_method 0 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:09.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 10 monclient(hunting): handle_auth_done global_id 4672 payload 293 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9678003b20 msgr2=0x7f9678003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9678003b20 0x7f9678003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678005330 msgr2=0x7f967800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.474+0000 7f9683fff640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678005330 0x7f967800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 1 monclient: found mon.b 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f968b7fe640 1 -- 172.21.15.67:0/2472591176 reap_dead start 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780034e0 con 0x7f9678004780 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.475+0000 7f9683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f9683fff640 1 -- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 msgr2=0x7f9678004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f9683fff640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 0x7f9678004bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9674001900 tx=0x7f9674007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962295489865a600 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674005890 con 0x7f9678004780 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: dump: 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f969151d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674005a10 con 0x7f9678004780 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f969151d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f969151d640 4 set_mon_vals no callback set 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 1 -- 172.21.15.67:0/2472591176 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674005dd0 con 0x7f9678004780 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.477+0000 7f968a7fc640 10 monclient: dump: 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.511 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 10 monclient: shutdown 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 -- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 msgr2=0x7f9678004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 0x7f9678004bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9674001900 tx=0x7f9674007a50 comp rx=0 tx=0).stop 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 -- 172.21.15.67:0/2472591176 shutdown_connections 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 --2- 172.21.15.67:0/2472591176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678004780 0x7f9678004bf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 -- 172.21.15.67:0/2472591176 >> 172.21.15.67:0/2472591176 conn(0x7f968c0a5780 msgr2=0x7f968c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 -- 172.21.15.67:0/2472591176 shutdown_connections 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.478+0000 7f969151d640 1 -- 172.21.15.67:0/2472591176 wait complete. 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient: monmap: 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.513 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 Processor -- start 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 -- start start 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient: init 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678004780 0x7f968c1413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): picked mon.noname-a con 0x7f9678004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f968c1312c0 0x7f968c1316d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): picked mon.noname-c con 0x7f968c1312c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f968c11de30 0x7f968c11e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): picked mon.noname-b con 0x7f968c11de30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c121d40 con 0x7f9678004780 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c1216f0 con 0x7f968c11de30 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c099090 con 0x7f968c1312c0 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f969151d640 10 monclient(hunting): authenticate will time out at 821.948811s 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f968c11de30 0x7f968c11e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f968c11de30 0x7f968c11e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41146/0 (socket says 172.21.15.67:41146) 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 1 -- 172.21.15.67:0/1951716577 learned_addr learned my addr 172.21.15.67:0/1951716577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): get_auth_request con 0x7f968c11de30 auth_method 0 2024-09-15T01:53:09.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968affd640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678004780 0x7f968c1413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f9683fff640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f968c1312c0 0x7f968c1316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.479+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): get_auth_request con 0x7f9678004780 auth_method 0 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): get_auth_request con 0x7f968c1312c0 auth_method 0 2024-09-15T01:53:09.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 10 monclient(hunting): handle_auth_done global_id 4675 payload 824 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 -- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f968c1312c0 msgr2=0x7f968c1316d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f968c1312c0 0x7f968c1316d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 -- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678004780 msgr2=0x7f968c1413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9678004780 0x7f968c1413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.480+0000 7f968b7fe640 1 -- 172.21.15.67:0/1951716577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c121410 con 0x7f968c11de30 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f968b7fe640 1 -- 172.21.15.67:0/1951716577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f968c11de30 msgr2=0x7f968c11e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f968b7fe640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f968c11de30 0x7f968c11e2a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f968c099060 tx=0x7f968400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400e040 con 0x7f968c11de30 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: dump: 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968400f040 con 0x7f968c11de30 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684015b70 con 0x7f968c11de30 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 5 monclient: authenticate success, global_id 4675 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 10 monclient: _renew_subs 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 1 -- 172.21.15.67:0/1951716577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c1219d0 con 0x7f968c11de30 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9689ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f9682ffd640 10 monclient: dump: 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.520 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 10 monclient: _renew_subs 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.482+0000 7f969151d640 1 -- 172.21.15.67:0/1951716577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c141920 con 0x7f968c11de30 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f96840263f0 con 0x7f968c11de30 2024-09-15T01:53:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f969151d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f969151d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f969151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f969151d640 1 -- 172.21.15.67:0/1951716577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c11f910 con 0x7f968c11de30 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f9682ffd640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f966c038790 0x7f966c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f968affd640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f966c038790 0x7f966c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.483+0000 7f968affd640 10 monclient: get_auth_request con 0x7f966c038790 auth_method 0 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.484+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f96840133e0 con 0x7f968c11de30 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.486+0000 7f968affd640 1 --2- 172.21.15.67:0/1951716577 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f966c038790 0x7f966c03ac40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f967c007ad0 tx=0x7f967c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.489+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f968401bd50 con 0x7f968c11de30 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.489+0000 7f9682ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.489+0000 7f9682ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.583+0000 7f531579d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.583+0000 7f531579d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.583+0000 7f531579d640 10 monclient: monmap: 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:09.584464+0000 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:09.584464+0000 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.587 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 Processor -- start 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 -- start start 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient: init 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 0x7f531009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): picked mon.noname-b con 0x7f53100a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 0x7f5310069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): picked mon.noname-c con 0x7f531009ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5310069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:09.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53100a5b50 con 0x7f5310069ac0 2024-09-15T01:53:09.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53100a18b0 con 0x7f531009ccf0 2024-09-15T01:53:09.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531009c5f0 con 0x7f53100a5780 2024-09-15T01:53:09.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f531579d640 10 monclient(hunting): authenticate will time out at 849.580781s 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 0x7f531009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5310069ac0 0x7f5310138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58726/0 (socket says 172.21.15.17:58726) 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530f7fe640 1 -- 172.21.15.17:0/1580448068 learned_addr learned my addr 172.21.15.17:0/1580448068 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530effd640 10 monclient(hunting): get_auth_request con 0x7f53100a5780 auth_method 0 2024-09-15T01:53:09.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.584+0000 7f530effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 1 --2- 172.21.15.17:0/1580448068 >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 0x7f5310069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 10 monclient(hunting): get_auth_request con 0x7f531009ccf0 auth_method 0 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient(hunting): handle_auth_done global_id 34298 payload 274 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 1 -- 172.21.15.17:0/1580448068 >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 msgr2=0x7f5310069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 1 --2- 172.21.15.17:0/1580448068 >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 0x7f5310069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 1 -- 172.21.15.17:0/1580448068 >> v1:172.21.15.17:6789/0 conn(0x7f5310069ac0 legacy=0x7f5310138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530effd640 1 -- 172.21.15.17:0/1580448068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531009d210 con 0x7f53100a5780 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.585+0000 7f530e7fc640 1 --2- 172.21.15.17:0/1580448068 >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 0x7f5310069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.587+0000 7f530effd640 1 -- 172.21.15.17:0/1580448068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 msgr2=0x7f531009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.587+0000 7f530effd640 1 --2- 172.21.15.17:0/1580448068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 0x7f531009bc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5300009850 tx=0x7f53000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bdae88c5248e567 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 1 -- 172.21.15.17:0/1580448068 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530002adf0 con 0x7f53100a5780 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: dump: 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 1 -- 172.21.15.17:0/1580448068 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5300032440 con 0x7f53100a5780 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 5 monclient: authenticate success, global_id 34298 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 1 -- 172.21.15.17:0/1580448068 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000327e0 con 0x7f53100a5780 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f530dffb640 10 monclient: dump: 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 4 set_mon_vals no callback set 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 10 monclient: shutdown 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 -- 172.21.15.17:0/1580448068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 msgr2=0x7f531009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 --2- 172.21.15.17:0/1580448068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 0x7f531009bc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5300009850 tx=0x7f53000090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 -- 172.21.15.17:0/1580448068 shutdown_connections 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 --2- 172.21.15.17:0/1580448068 >> v2:172.21.15.67:3300/0 conn(0x7f531009ccf0 0x7f5310069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 --2- 172.21.15.17:0/1580448068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53100a5780 0x7f531009bc90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 -- 172.21.15.17:0/1580448068 >> 172.21.15.17:0/1580448068 conn(0x7f5310070e10 msgr2=0x7f53100a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 -- 172.21.15.17:0/1580448068 shutdown_connections 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.588+0000 7f531579d640 1 -- 172.21.15.17:0/1580448068 wait complete. 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.589+0000 7f531579d640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.589+0000 7f531579d640 10 monclient: monmap: 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.600 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.589+0000 7f531579d640 1 Processor -- start 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.589+0000 7f531579d640 1 -- start start 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.589+0000 7f531579d640 10 monclient: init 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 0x7f531013de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): picked mon.noname-c con 0x7f5310069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531009ccf0 0x7f531013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): picked mon.noname-a con 0x7f531009ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100a5780 0x7f5310141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): picked mon.noname-b con 0x7f53100a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310122c10 con 0x7f531009ccf0 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310122830 con 0x7f53100a5780 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5310120b20 con 0x7f5310069ac0 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f531579d640 10 monclient(hunting): authenticate will time out at 849.586175s 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 0x7f531013de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531009ccf0 0x7f531013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 0x7f531013de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40536/0 (socket says 172.21.15.17:40536) 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 -- 172.21.15.17:0/643338758 learned_addr learned my addr 172.21.15.17:0/643338758 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): get_auth_request con 0x7f5310069ac0 auth_method 0 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): get_auth_request con 0x7f531009ccf0 auth_method 0 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530f7fe640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100a5780 0x7f5310141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient(hunting): handle_auth_done global_id 34301 payload 824 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100a5780 msgr2=0x7f5310141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100a5780 0x7f5310141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531009ccf0 msgr2=0x7f531013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531009ccf0 0x7f531013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 monclient: found mon.noname-c 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530effd640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300009390 con 0x7f5310069ac0 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530f7fe640 10 monclient: get_auth_request con 0x7f53100a5780 auth_method 0 2024-09-15T01:53:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.590+0000 7f530f7fe640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100a5780 0x7f5310141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:09.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f530effd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 msgr2=0x7f531013de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f530effd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 0x7f531013de00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5300009360 tx=0x7f5300032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5300031a40 con 0x7f5310069ac0 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 10 monclient: dump: 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.609 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f531579d640 5 monclient: authenticate success, global_id 34301 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f531579d640 10 monclient: _renew_subs 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5300031ba0 con 0x7f5310069ac0 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f531579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.592+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53101202c0 con 0x7f5310069ac0 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530003c400 con 0x7f5310069ac0 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 10 monclient: dump: 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.611 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 10 monclient: _renew_subs 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531013e810 con 0x7f5310069ac0 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f530d7fa640 4 set_mon_vals no callback set 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 0x7f53101418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53101226f0 con 0x7f5310121680 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.593+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f530003c830 con 0x7f5310069ac0 2024-09-15T01:53:09.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 0x7f53101418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient: get_auth_request con 0x7f5310121680 auth_method 0 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f52f6ffd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52fc038740 0x7f52fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f530006c6b0 con 0x7f5310069ac0 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 10 monclient(hunting): handle_auth_done global_id 4681 payload 824 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530e7fc640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5310122ef0 con 0x7f5310121680 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530f7fe640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52fc038740 0x7f52fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.594+0000 7f530f7fe640 10 monclient: get_auth_request con 0x7f52fc038740 auth_method 0 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f530e7fc640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 msgr2=0x7f53101418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f530e7fc640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 0x7f53101418b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5304006e70 tx=0x7f5304008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530400ed50 con 0x7f5310121680 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f52f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f52f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f52f6ffd640 10 monclient: dump: 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.614 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.596+0000 7f530f7fe640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52fc038740 0x7f52fc03abf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5310122bc0 tx=0x7f5308007490 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f530400eeb0 con 0x7f5310121680 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 msgr2=0x7f53101418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 0x7f53101418b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5304006e70 tx=0x7f5304008040 comp rx=0 tx=0).stop 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 msgr2=0x7f53101418b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.597+0000 7f52f6ffd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310121680 0x7f53101418b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.602+0000 7f531579d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.602+0000 7f531579d640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 0x7f531011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.602+0000 7f531579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.602+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310063b20 con 0x7f5310141df0 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 0x7f531011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient: get_auth_request con 0x7f5310141df0 auth_method 0 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 10 monclient(hunting): handle_auth_done global_id 4684 payload 824 2024-09-15T01:53:09.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.603+0000 7f530e7fc640 1 -- 172.21.15.17:0/643338758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f53101421f0 con 0x7f5310141df0 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f530e7fc640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 msgr2=0x7f531011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f530e7fc640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 0x7f531011b6d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f530401a000 tx=0x7f530401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530402b070 con 0x7f5310141df0 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 10 monclient: dump: 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f5304023040 con 0x7f5310141df0 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:09.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 msgr2=0x7f531011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 0x7f531011b6d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f530401a000 tx=0x7f530401d000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.605+0000 7f52f6ffd640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 msgr2=0x7f531011b6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f52f6ffd640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310141df0 0x7f531011b6d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:09.619 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:09.620 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.621 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:09.623 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:09.624 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:09.625 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.626 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.628 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:09.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:09.630 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:09.631 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:09.632 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:09.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52fc038740 msgr2=0x7f52fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52fc038740 0x7f52fc03abf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5310122bc0 tx=0x7f5308007490 comp rx=0 tx=0).stop 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 10 monclient: shutdown 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 msgr2=0x7f531013de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 --2- 172.21.15.17:0/643338758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5310069ac0 0x7f531013de00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5300009360 tx=0x7f5300032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f530f7fe640 1 -- 172.21.15.17:0/643338758 reap_dead start 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f530f7fe640 1 -- 172.21.15.17:0/643338758 reap_dead start 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 shutdown_connections 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 >> 172.21.15.17:0/643338758 conn(0x7f5310070e10 msgr2=0x7f531009af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 shutdown_connections 2024-09-15T01:53:09.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.606+0000 7f531579d640 1 -- 172.21.15.17:0/643338758 wait complete. 2024-09-15T01:53:09.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:09.635 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:53:09.635 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:53:09.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.653+0000 7f969151d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T01:53:09.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.653+0000 7f969151d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T01:53:09.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.653+0000 7f969151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.653+0000 7f969151d640 1 -- 172.21.15.67:0/1951716577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7f968c121b60 con 0x7f968c11de30 2024-09-15T01:53:09.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.654+0000 7f9682ffd640 1 -- 172.21.15.67:0/1951716577 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 290+0+0 (secure 0 0 0) 0x7f968401bd50 con 0x7f968c11de30 2024-09-15T01:53:09.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.654+0000 7f9682ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T01:53:09.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.654+0000 7f9682ffd640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T01:53:09.657 INFO:tasks.workunit.client.0.smithi067.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T01:53:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-15T01:53:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo foo --force 2024-09-15T01:53:09.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.729+0000 7f92444a8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.729+0000 7f92444a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.729+0000 7f92444a8640 10 monclient: monmap: 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:09.730313+0000 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:09.730313+0000 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 Processor -- start 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 -- start start 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient: init 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-b con 0x7f923c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f923c09ccf0 0x7f923c069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-c con 0x7f923c09ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f923c069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923c0a5b50 con 0x7f923c069ac0 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f923c0a18b0 con 0x7f923c09ccf0 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c09c5f0 con 0x7f923c0a5780 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f92444a8640 10 monclient(hunting): authenticate will time out at 849.726754s 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242a45640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f923c069ac0 0x7f923c138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58742/0 (socket says 172.21.15.17:58742) 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242a45640 1 -- 172.21.15.17:0/1358810640 learned_addr learned my addr 172.21.15.17:0/1358810640 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:09.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242244640 1 --2- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242244640 10 monclient(hunting): get_auth_request con 0x7f923c0a5780 auth_method 0 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242244640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9242244640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.730+0000 7f9241a43640 1 --2- 172.21.15.17:0/1358810640 >> v2:172.21.15.67:3300/0 conn(0x7f923c09ccf0 0x7f923c069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): get_auth_request con 0x7f923c09ccf0 auth_method 0 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient(hunting): handle_auth_done global_id 34304 payload 274 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 1 -- 172.21.15.17:0/1358810640 >> v2:172.21.15.67:3300/0 conn(0x7f923c09ccf0 msgr2=0x7f923c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 1 --2- 172.21.15.17:0/1358810640 >> v2:172.21.15.67:3300/0 conn(0x7f923c09ccf0 0x7f923c069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 1 -- 172.21.15.17:0/1358810640 >> v1:172.21.15.17:6789/0 conn(0x7f923c069ac0 legacy=0x7f923c138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.731+0000 7f9242244640 1 -- 172.21.15.17:0/1358810640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c09d210 con 0x7f923c0a5780 2024-09-15T01:53:09.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9242244640 1 -- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 msgr2=0x7f923c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9242244640 1 --2- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c09bc90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9238009850 tx=0x7f92380090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cdac29568cf7106 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 1 -- 172.21.15.17:0/1358810640 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923802adf0 con 0x7f923c0a5780 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: dump: 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.751 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 5 monclient: authenticate success, global_id 34304 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 1 -- 172.21.15.17:0/1358810640 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238032440 con 0x7f923c0a5780 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 4 set_mon_vals no callback set 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 1 -- 172.21.15.17:0/1358810640 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92380327e0 con 0x7f923c0a5780 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f9241242640 10 monclient: dump: 2024-09-15T01:53:09.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 10 monclient: shutdown 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 -- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 msgr2=0x7f923c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 --2- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c09bc90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9238009850 tx=0x7f92380090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 -- 172.21.15.17:0/1358810640 shutdown_connections 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 --2- 172.21.15.17:0/1358810640 >> v2:172.21.15.67:3300/0 conn(0x7f923c09ccf0 0x7f923c069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 --2- 172.21.15.17:0/1358810640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c09bc90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 -- 172.21.15.17:0/1358810640 >> 172.21.15.17:0/1358810640 conn(0x7f923c070e10 msgr2=0x7f923c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 -- 172.21.15.17:0/1358810640 shutdown_connections 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.734+0000 7f92444a8640 1 -- 172.21.15.17:0/1358810640 wait complete. 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.735+0000 7f92444a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.735+0000 7f92444a8640 10 monclient: monmap: 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:09.754 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.735+0000 7f92444a8640 1 Processor -- start 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.735+0000 7f92444a8640 1 -- start start 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.735+0000 7f92444a8640 10 monclient: init 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f923c069ac0 0x7f923c13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-b con 0x7f923c069ac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-a con 0x7f923c09ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): picked mon.noname-c con 0x7f923c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c121620 con 0x7f923c09ccf0 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c1225d0 con 0x7f923c069ac0 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c1227d0 con 0x7f923c0a5780 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f92444a8640 10 monclient(hunting): authenticate will time out at 849.732097s 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54586/0 (socket says 172.21.15.17:54586) 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 -- 172.21.15.17:0/4044111323 learned_addr learned my addr 172.21.15.17:0/4044111323 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): get_auth_request con 0x7f923c09ccf0 auth_method 0 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f923c069ac0 0x7f923c13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): get_auth_request con 0x7f923c0a5780 auth_method 0 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): get_auth_request con 0x7f923c069ac0 auth_method 0 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient(hunting): handle_auth_done global_id 44142 payload 824 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 -- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 msgr2=0x7f923c141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 -- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f923c069ac0 msgr2=0x7f923c13dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f923c069ac0 0x7f923c13dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 monclient: found mon.noname-a 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9241a43640 1 -- 172.21.15.17:0/4044111323 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9238009390 con 0x7f923c09ccf0 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.736+0000 7f9242a45640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f9241a43640 1 -- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 msgr2=0x7f923c13d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f9241a43640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f922c00ea20 tx=0x7f922c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c009600 con 0x7f923c09ccf0 2024-09-15T01:53:09.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 10 monclient: dump: 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c00cc20 con 0x7f923c09ccf0 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92444a8640 5 monclient: authenticate success, global_id 44142 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92444a8640 10 monclient: _renew_subs 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92444a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.738+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c122d90 con 0x7f923c09ccf0 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c013470 con 0x7f923c09ccf0 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 10 monclient: dump: 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 10 monclient: _renew_subs 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c13e8a0 con 0x7f923c09ccf0 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f9240a41640 4 set_mon_vals no callback set 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f922c013860 con 0x7f923c09ccf0 2024-09-15T01:53:09.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.739+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c0724a0 con 0x7f923c09ccf0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.740+0000 7f92327fc640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 0x7f922003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.740+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f922c04c0d0 con 0x7f923c09ccf0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.740+0000 7f9242244640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 0x7f922003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.740+0000 7f9242244640 10 monclient: get_auth_request con 0x7f9220038790 auth_method 0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.742+0000 7f9242244640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 0x7f922003ac40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f923c0a0c50 tx=0x7f9238032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.744+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f922c0236b0 con 0x7f923c09ccf0 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.744+0000 7f92327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.744+0000 7f92327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 10 monclient: monmap: 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:09.736251+0000 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:09.736251+0000 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 1 Processor -- start 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 1 -- start start 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.735+0000 7f9954509640 10 monclient: init 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f994c098140 0x7f994c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): picked mon.noname-c con 0x7f994c098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f994c0a0370 0x7f994c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): picked mon.noname-a con 0x7f994c0a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f994c098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c098ef0 con 0x7f994c098a50 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0a0b80 con 0x7f994c0a0370 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0a00b0 con 0x7f994c098140 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f9954509640 10 monclient(hunting): authenticate will time out at 822.205368s 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f994c098140 0x7f994c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f994c098140 0x7f994c1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41152/0 (socket says 172.21.15.67:41152) 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 -- 172.21.15.67:0/2031022761 learned_addr learned my addr 172.21.15.67:0/2031022761 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): get_auth_request con 0x7f994c098140 auth_method 0 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient(hunting): handle_auth_done global_id 4693 payload 274 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 -- 172.21.15.67:0/2031022761 >> v2:172.21.15.17:3300/0 conn(0x7f994c0a0370 msgr2=0x7f994c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 --2- 172.21.15.67:0/2031022761 >> v2:172.21.15.17:3300/0 conn(0x7f994c0a0370 0x7f994c098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 -- 172.21.15.67:0/2031022761 >> v1:172.21.15.17:6789/0 conn(0x7f994c098a50 legacy=0x7f994c0695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 monclient: found mon.noname-c 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.736+0000 7f99522a5640 1 -- 172.21.15.67:0/2031022761 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c071390 con 0x7f994c098140 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99522a5640 1 -- 172.21.15.67:0/2031022761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098140 msgr2=0x7f994c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99522a5640 1 --2- 172.21.15.67:0/2031022761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098140 0x7f994c1321e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f994c0a5340 tx=0x7f993c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c1cdc0404ee55b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 1 -- 172.21.15.67:0/2031022761 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c007610 con 0x7f994c098140 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: dump: 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.769 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 1 -- 172.21.15.67:0/2031022761 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c02d050 con 0x7f994c098140 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f9954509640 5 monclient: authenticate success, global_id 4693 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f9954509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f9954509640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f9954509640 4 set_mon_vals no callback set 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 1 -- 172.21.15.67:0/2031022761 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0079a0 con 0x7f994c098140 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.739+0000 7f99512a3640 10 monclient: dump: 2024-09-15T01:53:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 10 monclient: shutdown 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 -- 172.21.15.67:0/2031022761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098140 msgr2=0x7f994c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 --2- 172.21.15.67:0/2031022761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098140 0x7f994c1321e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f994c0a5340 tx=0x7f993c009840 comp rx=0 tx=0).stop 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 -- 172.21.15.67:0/2031022761 shutdown_connections 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 --2- 172.21.15.67:0/2031022761 >> v2:172.21.15.17:3300/0 conn(0x7f994c0a0370 0x7f994c098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 --2- 172.21.15.67:0/2031022761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098140 0x7f994c1321e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 -- 172.21.15.67:0/2031022761 >> 172.21.15.67:0/2031022761 conn(0x7f994c0a5780 msgr2=0x7f994c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 -- 172.21.15.67:0/2031022761 shutdown_connections 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 1 -- 172.21.15.67:0/2031022761 wait complete. 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.740+0000 7f9954509640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient: monmap: 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:09.772 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 Processor -- start 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 -- start start 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient: init 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f994c098140 0x7f994c09ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): picked mon.noname-a con 0x7f994c098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098a50 0x7f994c131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): picked mon.noname-b con 0x7f994c098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f994c0a0370 0x7f994c131640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): picked mon.noname-c con 0x7f994c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c131c80 con 0x7f994c098140 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c11e220 con 0x7f994c098a50 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c11e500 con 0x7f994c0a0370 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9954509640 10 monclient(hunting): authenticate will time out at 822.210797s 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098a50 0x7f994c131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098a50 0x7f994c131100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41154/0 (socket says 172.21.15.67:41154) 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f99522a5640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f994c098140 0x7f994c09ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 1 -- 172.21.15.67:0/332425637 learned_addr learned my addr 172.21.15.67:0/332425637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 10 monclient(hunting): get_auth_request con 0x7f994c098a50 auth_method 0 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9951aa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f99522a5640 10 monclient(hunting): get_auth_request con 0x7f994c098140 auth_method 0 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f99522a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f99522a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f99522a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.741+0000 7f9952aa6640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f994c0a0370 0x7f994c131640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): get_auth_request con 0x7f994c0a0370 auth_method 0 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f99522a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9952aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2024-09-15T01:53:09.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 -- 172.21.15.67:0/332425637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f994c0a0370 msgr2=0x7f994c131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f994c0a0370 0x7f994c131640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 -- 172.21.15.67:0/332425637 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f994c098140 msgr2=0x7f994c09ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f994c098140 0x7f994c09ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient: _finish_hunting 0 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 monclient: found mon.noname-b 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.742+0000 7f9951aa4640 1 -- 172.21.15.67:0/332425637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c009ae0 con 0x7f994c098a50 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9951aa4640 1 -- 172.21.15.67:0/332425637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098a50 msgr2=0x7f994c131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9951aa4640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994c098a50 0x7f994c131100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f994000b600 tx=0x7f99400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000f800 con 0x7f994c098a50 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: dump: 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994000f960 con 0x7f994c098a50 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 5 monclient: authenticate success, global_id 4696 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000fd20 con 0x7f994c098a50 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 10 monclient: _renew_subs 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 1 -- 172.21.15.67:0/332425637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c09f640 con 0x7f994c098a50 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9950aa2640 4 set_mon_vals no callback set 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f994a7fc640 10 monclient: dump: 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 10 monclient: _renew_subs 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.744+0000 7f9954509640 1 -- 172.21.15.67:0/332425637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c11c8f0 con 0x7f994c098a50 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9940010690 con 0x7f994c098a50 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f9954509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f9954509640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f9954509640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f994a7fc640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9938038740 0x7f993803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f9954509640 1 -- 172.21.15.67:0/332425637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c04bf60 con 0x7f994c098a50 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f994004bc90 con 0x7f994c098a50 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f99522a5640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9938038740 0x7f993803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.745+0000 7f99522a5640 10 monclient: get_auth_request con 0x7f9938038740 auth_method 0 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.748+0000 7f99522a5640 1 --2- 172.21.15.67:0/332425637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9938038740 0x7f993803abf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f993c030000 tx=0x7f993c036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.750+0000 7f994a7fc640 1 -- 172.21.15.67:0/332425637 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f994001ab40 con 0x7f994c098a50 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.750+0000 7f994a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:09.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.750+0000 7f994a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:09.914 INFO:tasks.workunit.client.0.smithi067.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-15T01:53:09.914 INFO:tasks.workunit.client.0.smithi067.stderr:Invalid command: unused arguments: ['--force'] 2024-09-15T01:53:09.914 INFO:tasks.workunit.client.0.smithi067.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T01:53:09.915 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: invalid command 2024-09-15T01:53:09.915 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:09.916 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-15T01:53:09.916 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:09.916 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-15T01:53:09.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.973+0000 7f92444a8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:53:09.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.973+0000 7f92444a8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:53:09.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.973+0000 7f92444a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:09.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.973+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f923c06f4f0 con 0x7f923c09ccf0 2024-09-15T01:53:09.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f92327fc640 1 -- 172.21.15.17:0/4044111323 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f922c0232f0 con 0x7f923c09ccf0 2024-09-15T01:53:09.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f92327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:53:09.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f92327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:09.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=8,logm=69} crc {auth=1515775660,config=3140514009,health=3533490341,kv=1529054991,logm=484033262}) 2024-09-15T01:53:09.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=10,kv=8,logm=71} crc {auth=448131596,config=3140514009,health=3945781735,kv=1529054990,logm=2479686573}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=8,logm=69} crc {auth=1515775660,config=3140514009,health=3533490341,kv=1529054991,logm=484033262}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.974+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=8,logm=68} crc {auth=263810266,config=3140514009,health=1564601805,kv=1529054990,logm=1355618252}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2944797943}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406685891}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2944797943}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3432120320}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368131845}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945633849}) 2024-09-15T01:53:09.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368131845}) 2024-09-15T01:53:09.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.975+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903014723}) 2024-09-15T01:53:09.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006182764}) 2024-09-15T01:53:09.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 msgr2=0x7f922003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 0x7f922003ac40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f923c0a0c50 tx=0x7f9238032000 comp rx=0 tx=0).stop 2024-09-15T01:53:09.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259379978}) 2024-09-15T01:53:09.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006182764}) 2024-09-15T01:53:09.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2211020692}) 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 10 monclient: shutdown 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 msgr2=0x7f923c13d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f922c00ea20 tx=0x7f922c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 shutdown_connections 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9220038790 0x7f922003ac40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f923c09ccf0 0x7f923c13d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f923c0a5780 0x7f923c141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 --2- 172.21.15.17:0/4044111323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f923c069ac0 0x7f923c13dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 >> 172.21.15.17:0/4044111323 conn(0x7f923c070e10 msgr2=0x7f923c09af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 shutdown_connections 2024-09-15T01:53:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:09.976+0000 7f92444a8640 1 -- 172.21.15.17:0/4044111323 wait complete. 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981483959}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=846319644}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981483959}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.977+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834641487}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=852309287,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=545380415,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=4137261901,osd_snap=1063953654,osdmap=3088477936}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=12} crc {logm=2143799565,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgrstat=1228447407,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=4137261901,osd_snap=2673976383,osdmap=3589706695}) 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:09.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=852309287,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=545380415,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=4137261901,osd_snap=1063953654,osdmap=3088477936}) 2024-09-15T01:53:09.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:09.978+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=2,osdmap=11} crc {logm=2251003335,mdsmap=3140514009,mgr=1655507991,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3776821395,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_snap=1063953654,osdmap=2290711523}) 2024-09-15T01:53:09.989 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:53:09.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.991+0000 7fbc75e34640 10 monclient: get_monmap_and_config 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.991+0000 7fbc75e34640 10 monclient: build_initial_monmap 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.991+0000 7fbc75e34640 10 monclient: monmap: 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:09.992858+0000 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:09.992858+0000 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:09.994 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:09.995 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 Processor -- start 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 -- start start 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient: init 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbc700a0b50 0x7fbc70069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-a con 0x7fbc700a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc7009eea0 0x7fbc7009f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-c con 0x7fbc7009eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc70131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-b con 0x7fbc70131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7006f290 con 0x7fbc700a0b50 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7009f7e0 con 0x7fbc7009eea0 2024-09-15T01:53:10.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7006a270 con 0x7fbc70131b10 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.992+0000 7fbc75e34640 10 monclient(hunting): authenticate will time out at 822.461977s 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc7009eea0 0x7fbc7009f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc7009eea0 0x7fbc7009f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41160/0 (socket says 172.21.15.67:41160) 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 -- 172.21.15.67:0/3372465691 learned_addr learned my addr 172.21.15.67:0/3372465691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): get_auth_request con 0x7fbc7009eea0 auth_method 0 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc70131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 1 --2- 172.21.15.67:0/3372465691 >> v2:172.21.15.17:3300/0 conn(0x7fbc700a0b50 0x7fbc70069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request con 0x7fbc70131b10 auth_method 0 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): get_auth_request con 0x7fbc700a0b50 auth_method 0 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient(hunting): handle_auth_done global_id 4699 payload 274 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 -- 172.21.15.67:0/3372465691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 msgr2=0x7fbc70131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc70131f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 -- 172.21.15.67:0/3372465691 >> v2:172.21.15.17:3300/0 conn(0x7fbc700a0b50 msgr2=0x7fbc70069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 --2- 172.21.15.67:0/3372465691 >> v2:172.21.15.17:3300/0 conn(0x7fbc700a0b50 0x7fbc70069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 monclient: found mon.noname-c 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.993+0000 7fbc6effd640 1 -- 172.21.15.67:0/3372465691 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700a0f20 con 0x7fbc7009eea0 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6effd640 1 -- 172.21.15.67:0/3372465691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc7009eea0 msgr2=0x7fbc7009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6effd640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc7009eea0 0x7fbc7009f2a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbc700a08b0 tx=0x7fbc640091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f7081cfa9286d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 1 -- 172.21.15.67:0/3372465691 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64032440 con 0x7fbc7009eea0 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: dump: 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 1 -- 172.21.15.67:0/3372465691 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc640325a0 con 0x7fbc7009eea0 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 5 monclient: authenticate success, global_id 4699 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:10.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 1 -- 172.21.15.67:0/3372465691 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64032980 con 0x7fbc7009eea0 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 4 set_mon_vals no callback set 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc6e7fc640 10 monclient: dump: 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.007 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 10 monclient: shutdown 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 -- 172.21.15.67:0/3372465691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc7009eea0 msgr2=0x7fbc7009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc7009eea0 0x7fbc7009f2a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbc700a08b0 tx=0x7fbc640091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 -- 172.21.15.67:0/3372465691 shutdown_connections 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc7009eea0 0x7fbc7009f2a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 --2- 172.21.15.67:0/3372465691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc70131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 --2- 172.21.15.67:0/3372465691 >> v2:172.21.15.17:3300/0 conn(0x7fbc700a0b50 0x7fbc70069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.996+0000 7fbc75e34640 1 -- 172.21.15.67:0/3372465691 >> 172.21.15.67:0/3372465691 conn(0x7fbc70071290 msgr2=0x7fbc700a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 1 -- 172.21.15.67:0/3372465691 shutdown_connections 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 1 -- 172.21.15.67:0/3372465691 wait complete. 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 10 monclient: build_initial_monmap 2024-09-15T01:53:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 10 monclient: monmap: 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 1 Processor -- start 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 1 -- start start 2024-09-15T01:53:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.997+0000 7fbc75e34640 10 monclient: init 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc7009eea0 0x7fbc7009a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-a con 0x7fbc7009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700a0b50 0x7fbc70130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-b con 0x7fbc700a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:10.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc7011b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): picked mon.noname-c con 0x7fbc70131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc70131990 con 0x7fbc7009eea0 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc70134bb0 con 0x7fbc700a0b50 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7011def0 con 0x7fbc70131b10 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc75e34640 10 monclient(hunting): authenticate will time out at 822.467445s 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700a0b50 0x7fbc70130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc7009eea0 0x7fbc7009a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700a0b50 0x7fbc70130af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41164/0 (socket says 172.21.15.67:41164) 2024-09-15T01:53:10.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 -- 172.21.15.67:0/1563565290 learned_addr learned my addr 172.21.15.67:0/1563565290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): get_auth_request con 0x7fbc700a0b50 auth_method 0 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc7011b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): get_auth_request con 0x7fbc7009eea0 auth_method 0 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request con 0x7fbc70131b10 auth_method 0 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient(hunting): handle_auth_done global_id 4702 payload 824 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 -- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 msgr2=0x7fbc7011b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc7011b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 -- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc7009eea0 msgr2=0x7fbc7009a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc7009eea0 0x7fbc7009a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 monclient: found mon.noname-b 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6effd640 1 -- 172.21.15.67:0/1563565290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64009450 con 0x7fbc700a0b50 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6f7fe640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc7009eea0 0x7fbc7009a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:09.998+0000 7fbc6ffff640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc70131b10 0x7fbc7011b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.001+0000 7fbc6effd640 1 -- 172.21.15.67:0/1563565290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700a0b50 msgr2=0x7fbc70130af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.001+0000 7fbc6effd640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700a0b50 0x7fbc70130af0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc6402a6b0 tx=0x7fbc64032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640319c0 con 0x7fbc700a0b50 2024-09-15T01:53:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: dump: 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64031b40 con 0x7fbc700a0b50 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 5 monclient: authenticate success, global_id 4702 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 10 monclient: _renew_subs 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 1 -- 172.21.15.67:0/1563565290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7009b240 con 0x7fbc700a0b50 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6403c400 con 0x7fbc700a0b50 2024-09-15T01:53:10.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc577fe640 10 monclient: dump: 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 10 monclient: _renew_subs 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc75e34640 1 -- 172.21.15.67:0/1563565290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc70141e20 con 0x7fbc700a0b50 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.003+0000 7fbc6dffb640 4 set_mon_vals no callback set 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbc6403c870 con 0x7fbc700a0b50 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc75e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc75e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc75e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc75e34640 1 -- 172.21.15.67:0/1563565290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7004bf60 con 0x7fbc700a0b50 2024-09-15T01:53:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc577fe640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc5c038790 0x7fbc5c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fbc6406c520 con 0x7fbc700a0b50 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc6f7fe640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc5c038790 0x7fbc5c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.004+0000 7fbc6f7fe640 10 monclient: get_auth_request con 0x7fbc5c038790 auth_method 0 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.007+0000 7fbc6f7fe640 1 --2- 172.21.15.67:0/1563565290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc5c038790 0x7fbc5c03ac40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbc600113e0 tx=0x7fbc600090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.009+0000 7fbc577fe640 1 -- 172.21.15.67:0/1563565290 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbc640424e0 con 0x7fbc700a0b50 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.009+0000 7fbc577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.009+0000 7fbc577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=3130912521}) 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=2040776133}) 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=3130912521}) 2024-09-15T01:53:10.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:10.028+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=472339002}) 2024-09-15T01:53:10.177 INFO:tasks.workunit.client.0.smithi067.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-15T01:53:10.178 INFO:tasks.workunit.client.0.smithi067.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-15T01:53:10.178 INFO:tasks.workunit.client.0.smithi067.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T01:53:10.178 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: invalid command 2024-09-15T01:53:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-15T01:53:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-15T01:53:10.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.257+0000 7fb78bcc9640 10 monclient: get_monmap_and_config 2024-09-15T01:53:10.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.257+0000 7fb78bcc9640 10 monclient: build_initial_monmap 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.257+0000 7fb78bcc9640 10 monclient: monmap: 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:10.258500+0000 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:10.258500+0000 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:10.260 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.257+0000 7fb78bcc9640 1 Processor -- start 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 -- start start 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient: init 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7840a3850 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 0x7fb784136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-a con 0x7fb78406a640 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 0x7fb784139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-b con 0x7fb784137240 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb784069c60 con 0x7fb7840a3850 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb784068b90 con 0x7fb78406a640 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb784069670 con 0x7fb784137240 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78bcc9640 10 monclient(hunting): authenticate will time out at 822.727662s 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb789a65640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7840a3850 0x7fb78406e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57154/0 (socket says 172.21.15.67:57154) 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb789a65640 1 -- 172.21.15.67:0/2315464383 learned_addr learned my addr 172.21.15.67:0/2315464383 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb789264640 1 --2- 172.21.15.67:0/2315464383 >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 0x7fb784136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.258+0000 7fb78a266640 1 --2- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 0x7fb784139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): get_auth_request con 0x7fb784137240 auth_method 0 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): get_auth_request con 0x7fb78406a640 auth_method 0 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient(hunting): handle_auth_done global_id 34319 payload 274 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 1 -- 172.21.15.67:0/2315464383 >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 msgr2=0x7fb784136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 1 --2- 172.21.15.67:0/2315464383 >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 0x7fb784136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 1 -- 172.21.15.67:0/2315464383 >> v1:172.21.15.67:6789/0 conn(0x7fb7840a3850 legacy=0x7fb78406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 1 monclient: found mon.noname-b 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb78a266640 1 -- 172.21.15.67:0/2315464383 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840a3c20 con 0x7fb784137240 2024-09-15T01:53:10.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.259+0000 7fb789264640 1 --2- 172.21.15.67:0/2315464383 >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 0x7fb784136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78a266640 1 -- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 msgr2=0x7fb784139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78a266640 1 --2- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 0x7fb784139620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009e60 tx=0x7fb77c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0fc00b91d39a335 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 1 -- 172.21.15.67:0/2315464383 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c007760 con 0x7fb784137240 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:10.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: dump: 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 1 -- 172.21.15.67:0/2315464383 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c02d050 con 0x7fb784137240 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 5 monclient: authenticate success, global_id 34319 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 1 -- 172.21.15.67:0/2315464383 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c007af0 con 0x7fb784137240 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 4 set_mon_vals no callback set 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb788a63640 10 monclient: dump: 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.275 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 10 monclient: shutdown 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 -- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 msgr2=0x7fb784139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 --2- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 0x7fb784139620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009e60 tx=0x7fb77c009780 comp rx=0 tx=0).stop 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 -- 172.21.15.67:0/2315464383 shutdown_connections 2024-09-15T01:53:10.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 --2- 172.21.15.67:0/2315464383 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb784137240 0x7fb784139620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 --2- 172.21.15.67:0/2315464383 >> v2:172.21.15.17:3300/0 conn(0x7fb78406a640 0x7fb784136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.262+0000 7fb78bcc9640 1 -- 172.21.15.67:0/2315464383 >> 172.21.15.67:0/2315464383 conn(0x7fb784070fb0 msgr2=0x7fb7840a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 1 -- 172.21.15.67:0/2315464383 shutdown_connections 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 1 -- 172.21.15.67:0/2315464383 wait complete. 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 10 monclient: build_initial_monmap 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 10 monclient: monmap: 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:10.277 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 1 Processor -- start 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 1 -- start start 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.263+0000 7fb78bcc9640 10 monclient: init 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb78406a640 0x7fb78411e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-b con 0x7fb78406a640 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7840a3850 0x7fb78411eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-c con 0x7fb7840a3850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb784137240 0x7fb78411f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): picked mon.noname-a con 0x7fb784137240 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7841206e0 con 0x7fb784137240 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840a24f0 con 0x7fb78406a640 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78406ea50 con 0x7fb7840a3850 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78bcc9640 10 monclient(hunting): authenticate will time out at 822.733370s 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb78406a640 0x7fb78411e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb78406a640 0x7fb78411e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41166/0 (socket says 172.21.15.67:41166) 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 -- 172.21.15.67:0/1270555376 learned_addr learned my addr 172.21.15.67:0/1270555376 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): get_auth_request con 0x7fb78406a640 auth_method 0 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb784137240 0x7fb78411f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7840a3850 0x7fb78411eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): get_auth_request con 0x7fb7840a3850 auth_method 0 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): get_auth_request con 0x7fb784137240 auth_method 0 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789264640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb78a266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient(hunting): handle_auth_done global_id 4705 payload 824 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 -- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7840a3850 msgr2=0x7fb78411eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7840a3850 0x7fb78411eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 -- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb784137240 msgr2=0x7fb78411f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb784137240 0x7fb78411f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 monclient: found mon.noname-b 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.264+0000 7fb789a65640 1 -- 172.21.15.67:0/1270555376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb778009390 con 0x7fb78406a640 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.266+0000 7fb789a65640 1 -- 172.21.15.67:0/1270555376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb78406a640 msgr2=0x7fb78411e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.266+0000 7fb789a65640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb78406a640 0x7fb78411e640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb78000c5e0 tx=0x7fb780004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780005cb0 con 0x7fb78406a640 2024-09-15T01:53:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: dump: 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb780005e30 con 0x7fb78406a640 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780007340 con 0x7fb78406a640 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 5 monclient: authenticate success, global_id 4705 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb775ffb640 10 monclient: dump: 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.284 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 10 monclient: _renew_subs 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb777fff640 4 set_mon_vals no callback set 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 1 -- 172.21.15.67:0/1270555376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c009a20 con 0x7fb78406a640 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 10 monclient: _renew_subs 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.267+0000 7fb78bcc9640 1 -- 172.21.15.67:0/1270555376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb784121c50 con 0x7fb78406a640 2024-09-15T01:53:10.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb78001b400 con 0x7fb78406a640 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb775ffb640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb768038740 0x7fb76803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb78bcc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb78bcc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb78bcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fb78004b990 con 0x7fb78406a640 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb78bcc9640 1 -- 172.21.15.67:0/1270555376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb784120250 con 0x7fb78406a640 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb789264640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb768038740 0x7fb76803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.268+0000 7fb789264640 10 monclient: get_auth_request con 0x7fb768038740 auth_method 0 2024-09-15T01:53:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.270+0000 7fb789264640 1 --2- 172.21.15.67:0/1270555376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb768038740 0x7fb76803abf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb78411fe20 tx=0x7fb7780090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.272+0000 7fb775ffb640 1 -- 172.21.15.67:0/1270555376 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb78001b6b0 con 0x7fb78406a640 2024-09-15T01:53:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.272+0000 7fb775ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.272+0000 7fb775ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:10.437 INFO:tasks.workunit.client.0.smithi067.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-15T01:53:10.438 INFO:tasks.workunit.client.0.smithi067.stderr:Invalid command: unused arguments: ['foo'] 2024-09-15T01:53:10.438 INFO:tasks.workunit.client.0.smithi067.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T01:53:10.438 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: invalid command 2024-09-15T01:53:10.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:10.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.515+0000 7f4b62cdd640 10 monclient: get_monmap_and_config 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.515+0000 7f4b62cdd640 10 monclient: build_initial_monmap 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.515+0000 7f4b62cdd640 10 monclient: monmap: 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:10.516871+0000 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:10.516871+0000 2024-09-15T01:53:10.518 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:10.519 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 Processor -- start 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 -- start start 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient: init 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:53:10.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b5c068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4b5c069190 0x7f4b5c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f4b5c069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b5c137150 0x7f4b5c139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c137150 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c096c10 con 0x7f4b5c068dc0 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c0a14d0 con 0x7f4b5c069190 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c132960 con 0x7f4b5c137150 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.516+0000 7f4b62cdd640 10 monclient(hunting): authenticate will time out at 822.985945s 2024-09-15T01:53:10.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b5c137150 0x7f4b5c139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b5c137150 0x7f4b5c139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55624/0 (socket says 172.21.15.67:55624) 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 -- 172.21.15.67:0/1280324704 learned_addr learned my addr 172.21.15.67:0/1280324704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): get_auth_request con 0x7f4b5c137150 auth_method 0 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b5bfff640 1 --2- 172.21.15.67:0/1280324704 >> v2:172.21.15.17:3300/0 conn(0x7f4b5c069190 0x7f4b5c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b5bfff640 10 monclient(hunting): get_auth_request con 0x7f4b5c069190 auth_method 0 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient(hunting): handle_auth_done global_id 4708 payload 274 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 -- 172.21.15.67:0/1280324704 >> v2:172.21.15.17:3300/0 conn(0x7f4b5c069190 msgr2=0x7f4b5c069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 --2- 172.21.15.67:0/1280324704 >> v2:172.21.15.17:3300/0 conn(0x7f4b5c069190 0x7f4b5c069590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 -- 172.21.15.67:0/1280324704 >> v1:172.21.15.17:6789/0 conn(0x7f4b5c068dc0 legacy=0x7f4b5c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 monclient: found mon.noname-c 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.517+0000 7f4b6127a640 1 -- 172.21.15.67:0/1280324704 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c09ff20 con 0x7f4b5c137150 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b6127a640 1 -- 172.21.15.67:0/1280324704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 msgr2=0x7f4b5c139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b6127a640 1 --2- 172.21.15.67:0/1280324704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c139530 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c096f10 tx=0x7f4b54009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4583abc879d5de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/1280324704 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54007610 con 0x7f4b5c137150 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: dump: 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.535 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/1280324704 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5402d050 con 0x7f4b5c137150 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/1280324704 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b540079a0 con 0x7f4b5c137150 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 5 monclient: authenticate success, global_id 4708 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 4 set_mon_vals no callback set 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b5b7fe640 10 monclient: dump: 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.536 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 10 monclient: shutdown 2024-09-15T01:53:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 -- 172.21.15.67:0/1280324704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 msgr2=0x7f4b5c139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/1280324704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c139530 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c096f10 tx=0x7f4b54009840 comp rx=0 tx=0).stop 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 -- 172.21.15.67:0/1280324704 shutdown_connections 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/1280324704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c139530 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/1280324704 >> v2:172.21.15.17:3300/0 conn(0x7f4b5c069190 0x7f4b5c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 -- 172.21.15.67:0/1280324704 >> 172.21.15.67:0/1280324704 conn(0x7f4b5c0711b0 msgr2=0x7f4b5c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 -- 172.21.15.67:0/1280324704 shutdown_connections 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.520+0000 7f4b62cdd640 1 -- 172.21.15.67:0/1280324704 wait complete. 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.521+0000 7f4b62cdd640 10 monclient: build_initial_monmap 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.521+0000 7f4b62cdd640 10 monclient: monmap: 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.521+0000 7f4b62cdd640 1 Processor -- start 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.521+0000 7f4b62cdd640 1 -- start start 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.521+0000 7f4b62cdd640 10 monclient: init 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:10.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b5c068dc0 0x7f4b5c098090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b5c069190 0x7f4b5c0985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f4b5c069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5c137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0a0890 con 0x7f4b5c069190 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c1317a0 con 0x7f4b5c137150 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c09b120 con 0x7f4b5c068dc0 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b62cdd640 10 monclient(hunting): authenticate will time out at 822.991377s 2024-09-15T01:53:10.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55630/0 (socket says 172.21.15.67:55630) 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 -- 172.21.15.67:0/738798506 learned_addr learned my addr 172.21.15.67:0/738798506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): get_auth_request con 0x7f4b5c137150 auth_method 0 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b5c068dc0 0x7f4b5c098090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b5c069190 0x7f4b5c0985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): get_auth_request con 0x7f4b5c069190 auth_method 0 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): get_auth_request con 0x7f4b5c068dc0 auth_method 0 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:10.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b60a79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient(hunting): handle_auth_done global_id 4711 payload 824 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 -- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b5c068dc0 msgr2=0x7f4b5c098090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b5c068dc0 0x7f4b5c098090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 -- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b5c069190 msgr2=0x7f4b5c0985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b5c069190 0x7f4b5c0985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 monclient: found mon.noname-b 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.522+0000 7f4b6127a640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b54009ae0 con 0x7f4b5c137150 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b6127a640 1 -- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 msgr2=0x7f4b5c098b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b6127a640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009ab0 tx=0x7f4b54036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54032c60 con 0x7f4b5c137150 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: dump: 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.544 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b54032dc0 con 0x7f4b5c137150 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 5 monclient: authenticate success, global_id 4711 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 10 monclient: _renew_subs 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c12f7b0 con 0x7f4b5c137150 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5403b640 con 0x7f4b5c137150 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b58ff9640 10 monclient: dump: 2024-09-15T01:53:10.545 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b5affd640 4 set_mon_vals no callback set 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 10 monclient: _renew_subs 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.525+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c09c030 con 0x7f4b5c137150 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4b54038050 con 0x7f4b5c137150 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b58ff9640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 0x7f4b4003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b62cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b62cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b62cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4b5406c9c0 con 0x7f4b5c137150 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c09b5a0 con 0x7f4b5c137150 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b60a79640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 0x7f4b4003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.526+0000 7f4b60a79640 10 monclient: get_auth_request con 0x7f4b40038790 auth_method 0 2024-09-15T01:53:10.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.528+0000 7f4b60a79640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 0x7f4b4003ac40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b50007fd0 tx=0x7f4b50007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:10.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.530+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4b5403f070 con 0x7f4b5c137150 2024-09-15T01:53:10.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.531+0000 7f4b58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:10.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.531+0000 7f4b58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:10.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.692+0000 7f4b62cdd640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:10.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.692+0000 7f4b62cdd640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:10.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.692+0000 7f4b62cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:10.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:10.693+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4b5c04bf60 con 0x7f4b5c137150 2024-09-15T01:53:10.989 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:53:10.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:11.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.091+0000 7fd63b35d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.091+0000 7fd63b35d640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.091+0000 7fd63b35d640 10 monclient: monmap: 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:11.092461+0000 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:11.092461+0000 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.094 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.092+0000 7fd63b35d640 1 Processor -- start 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.092+0000 7fd63b35d640 1 -- start start 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.092+0000 7fd63b35d640 10 monclient: init 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.092+0000 7fd63b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.092+0000 7fd63b35d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6340a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd634132040 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 0x7fd63413a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6340987b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340a1580 con 0x7fd634132040 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340a5c80 con 0x7fd6340a58b0 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd634132440 con 0x7fd6340987b0 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd63b35d640 10 monclient(hunting): authenticate will time out at 851.089140s 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6388f8640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd634132040 0x7fd634136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58752/0 (socket says 172.21.15.17:58752) 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 0x7fd63413a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6388f8640 1 -- 172.21.15.17:0/1386108930 learned_addr learned my addr 172.21.15.17:0/1386108930 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): get_auth_request con 0x7fd6340987b0 auth_method 0 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient(hunting): handle_auth_done global_id 34328 payload 274 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 1 -- 172.21.15.17:0/1386108930 >> v1:172.21.15.67:6789/0 conn(0x7fd6340a58b0 legacy=0x7fd6340980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 1 -- 172.21.15.17:0/1386108930 >> v1:172.21.15.17:6789/0 conn(0x7fd634132040 legacy=0x7fd634136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 1 monclient: found mon.noname-b 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.093+0000 7fd6398fa640 1 -- 172.21.15.17:0/1386108930 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634098bd0 con 0x7fd6340987b0 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd6398fa640 1 -- 172.21.15.17:0/1386108930 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 msgr2=0x7fd63413a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd6398fa640 1 --2- 172.21.15.17:0/1386108930 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 0x7fd63413a500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009e90 tx=0x7fd62c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d6a2e2f9680861 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 1 -- 172.21.15.17:0/1386108930 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c007840 con 0x7fd6340987b0 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: dump: 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.106 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 1 -- 172.21.15.17:0/1386108930 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c02d050 con 0x7fd6340987b0 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd63b35d640 5 monclient: authenticate success, global_id 34328 2024-09-15T01:53:11.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd63b35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd63b35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd63b35d640 4 set_mon_vals no callback set 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 1 -- 172.21.15.17:0/1386108930 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c007bd0 con 0x7fd6340987b0 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.096+0000 7fd62bfff640 10 monclient: dump: 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.108 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 10 monclient: shutdown 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 -- 172.21.15.17:0/1386108930 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 msgr2=0x7fd63413a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1386108930 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 0x7fd63413a500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009e90 tx=0x7fd62c009780 comp rx=0 tx=0).stop 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 -- 172.21.15.17:0/1386108930 shutdown_connections 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1386108930 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340987b0 0x7fd63413a500 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 -- 172.21.15.17:0/1386108930 >> 172.21.15.17:0/1386108930 conn(0x7fd634070ed0 msgr2=0x7fd6340a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 -- 172.21.15.17:0/1386108930 shutdown_connections 2024-09-15T01:53:11.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 1 -- 172.21.15.17:0/1386108930 wait complete. 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.097+0000 7fd63b35d640 10 monclient: monmap: 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.110 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 Processor -- start 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 -- start start 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient: init 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6340987b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6340a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd634132040 0x7fd634141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): picked mon.noname-b con 0x7fd634132040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63409e8f0 con 0x7fd6340987b0 2024-09-15T01:53:11.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63411e230 con 0x7fd634132040 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63411df50 con 0x7fd6340a58b0 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd63b35d640 10 monclient(hunting): authenticate will time out at 851.094807s 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40568/0 (socket says 172.21.15.17:40568) 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 1 -- 172.21.15.17:0/1274601803 learned_addr learned my addr 172.21.15.17:0/1274601803 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54596/0 (socket says 172.21.15.17:54596) 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 10 monclient(hunting): get_auth_request con 0x7fd6340a58b0 auth_method 0 2024-09-15T01:53:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6388f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 10 monclient(hunting): get_auth_request con 0x7fd6340987b0 auth_method 0 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.098+0000 7fd6390f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6390f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6390f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6390f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6398fa640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd634132040 0x7fd634141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6398fa640 10 monclient(hunting): get_auth_request con 0x7fd634132040 auth_method 0 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6398fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6398fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6398fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient(hunting): handle_auth_done global_id 34331 payload 824 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 -- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd634132040 msgr2=0x7fd634141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd634132040 0x7fd634141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 -- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 msgr2=0x7fd634134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 monclient: found mon.noname-c 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6388f8640 1 -- 172.21.15.17:0/1274601803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c009a20 con 0x7fd6340a58b0 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.099+0000 7fd6390f9640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6388f8640 1 -- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 msgr2=0x7fd634130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6388f8640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd62400c610 tx=0x7fd624004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624005cb0 con 0x7fd6340a58b0 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: dump: 2024-09-15T01:53:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 5 monclient: authenticate success, global_id 34331 2024-09-15T01:53:11.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624005e30 con 0x7fd6340a58b0 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 10 monclient: _renew_subs 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63411c960 con 0x7fd6340a58b0 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624007340 con 0x7fd6340a58b0 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd6297fa640 10 monclient: dump: 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.117 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd62b7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 10 monclient: _renew_subs 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.101+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63411cf20 con 0x7fd6340a58b0 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd62401b400 con 0x7fd6340a58b0 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd63b35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd63b35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd63b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6340724a0 con 0x7fd6340a58b0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd6297fa640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 0x7fd61803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.102+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fd62404b8f0 con 0x7fd6340a58b0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.103+0000 7fd6390f9640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 0x7fd61803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.103+0000 7fd6390f9640 10 monclient: get_auth_request con 0x7fd618038790 auth_method 0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.105+0000 7fd6390f9640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 0x7fd61803ac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd630007fd0 tx=0x7fd630007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.107+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd624012760 con 0x7fd6340a58b0 2024-09-15T01:53:11.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.107+0000 7fd6297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.107+0000 7fd6297fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd63b35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd63b35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd63b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd63409fc90 con 0x7fd6340a58b0 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd6297fa640 1 -- 172.21.15.17:0/1274601803 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fd62401a360 con 0x7fd6340a58b0 2024-09-15T01:53:11.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd6297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:11.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.337+0000 7fd6297fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:11.340 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 msgr2=0x7fd61803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 0x7fd61803ac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd630007fd0 tx=0x7fd630007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 10 monclient: shutdown 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 msgr2=0x7fd634130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd62400c610 tx=0x7fd624004960 comp rx=0 tx=0).stop 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 shutdown_connections 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd618038790 0x7fd61803ac40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd634132040 0x7fd634141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6340a58b0 0x7fd634130b50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 --2- 172.21.15.17:0/1274601803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6340987b0 0x7fd634134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.339+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 >> 172.21.15.17:0/1274601803 conn(0x7fd634070ed0 msgr2=0x7fd634096e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.340+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 shutdown_connections 2024-09-15T01:53:11.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.340+0000 7fd63b35d640 1 -- 172.21.15.17:0/1274601803 wait complete. 2024-09-15T01:53:11.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:53:11.352 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:53:11.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:53:11.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:53:11.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.447+0000 7f97f2c34640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.447+0000 7f97f2c34640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.447+0000 7f97f2c34640 10 monclient: monmap: 2024-09-15T01:53:11.452 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.452 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.452 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:11.448488+0000 2024-09-15T01:53:11.452 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:11.448488+0000 2024-09-15T01:53:11.452 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.453 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 Processor -- start 2024-09-15T01:53:11.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 -- start start 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient: init 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97ec068f20 0x7f97ec1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-c con 0x7f97ec068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97ec0692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec1393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-b con 0x7f97ec136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0a1950 con 0x7f97ec0692f0 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0a1670 con 0x7f97ec068f20 2024-09-15T01:53:11.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0a0020 con 0x7f97ec136fe0 2024-09-15T01:53:11.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.448+0000 7f97f2c34640 10 monclient(hunting): authenticate will time out at 851.444957s 2024-09-15T01:53:11.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec1393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97ebfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f97ec0692f0 0x7f97ec0696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58758/0 (socket says 172.21.15.17:58758) 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97ebfff640 1 -- 172.21.15.17:0/2727288534 learned_addr learned my addr 172.21.15.17:0/2727288534 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): get_auth_request con 0x7f97ec136fe0 auth_method 0 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2727288534 >> v2:172.21.15.67:3300/0 conn(0x7f97ec068f20 0x7f97ec1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient(hunting): handle_auth_done global_id 34334 payload 274 2024-09-15T01:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 -- 172.21.15.17:0/2727288534 >> v2:172.21.15.67:3300/0 conn(0x7f97ec068f20 msgr2=0x7f97ec1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 --2- 172.21.15.17:0/2727288534 >> v2:172.21.15.67:3300/0 conn(0x7f97ec068f20 0x7f97ec1321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 -- 172.21.15.17:0/2727288534 >> v1:172.21.15.17:6789/0 conn(0x7f97ec0692f0 legacy=0x7f97ec0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 monclient: found mon.noname-b 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.449+0000 7f97f11d1640 1 -- 172.21.15.17:0/2727288534 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0a1390 con 0x7f97ec136fe0 2024-09-15T01:53:11.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f11d1640 1 -- 172.21.15.17:0/2727288534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 msgr2=0x7f97ec1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f11d1640 1 --2- 172.21.15.17:0/2727288534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec1393c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009e90 tx=0x7f97e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c59b154b7ac30b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 1 -- 172.21.15.17:0/2727288534 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4007840 con 0x7f97ec136fe0 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: dump: 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 1 -- 172.21.15.17:0/2727288534 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e402d050 con 0x7f97ec136fe0 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 5 monclient: authenticate success, global_id 34334 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 4 set_mon_vals no callback set 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 1 -- 172.21.15.17:0/2727288534 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4007bd0 con 0x7f97ec136fe0 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97eb7fe640 10 monclient: dump: 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 10 monclient: shutdown 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 -- 172.21.15.17:0/2727288534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 msgr2=0x7f97ec1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2727288534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec1393c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009e90 tx=0x7f97e4009780 comp rx=0 tx=0).stop 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 -- 172.21.15.17:0/2727288534 shutdown_connections 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2727288534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec1393c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2727288534 >> v2:172.21.15.67:3300/0 conn(0x7f97ec068f20 0x7f97ec1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 -- 172.21.15.17:0/2727288534 >> 172.21.15.17:0/2727288534 conn(0x7f97ec070fb0 msgr2=0x7f97ec0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 -- 172.21.15.17:0/2727288534 shutdown_connections 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.452+0000 7f97f2c34640 1 -- 172.21.15.17:0/2727288534 wait complete. 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.453+0000 7f97f2c34640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.453+0000 7f97f2c34640 10 monclient: monmap: 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.474 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.453+0000 7f97f2c34640 1 Processor -- start 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.453+0000 7f97f2c34640 1 -- start start 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.453+0000 7f97f2c34640 10 monclient: init 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec068f20 0x7f97ec141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-b con 0x7f97ec068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 0x7f97ec141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-a con 0x7f97ec0692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): picked mon.noname-c con 0x7f97ec136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec11cf50 con 0x7f97ec0692f0 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec09b690 con 0x7f97ec068f20 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0a0750 con 0x7f97ec136fe0 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f2c34640 10 monclient(hunting): authenticate will time out at 851.450352s 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 0x7f97ec141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 0x7f97ec141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54628/0 (socket says 172.21.15.17:54628) 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 -- 172.21.15.17:0/2925312271 learned_addr learned my addr 172.21.15.17:0/2925312271 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): get_auth_request con 0x7f97ec0692f0 auth_method 0 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): get_auth_request con 0x7f97ec136fe0 auth_method 0 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec068f20 0x7f97ec141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): get_auth_request con 0x7f97ec068f20 auth_method 0 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient(hunting): handle_auth_done global_id 44169 payload 824 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 msgr2=0x7f97ec141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec068f20 msgr2=0x7f97ec141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec068f20 0x7f97ec141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 monclient: found mon.noname-a 2024-09-15T01:53:11.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97ebfff640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009a20 con 0x7f97ec0692f0 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.454+0000 7f97f11d1640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97ec136fe0 0x7f97ec141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97ebfff640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 msgr2=0x7f97ec141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97ebfff640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 0x7f97ec141990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f97d800c5e0 tx=0x7f97d8004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8005cb0 con 0x7f97ec0692f0 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: dump: 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d8005e30 con 0x7f97ec0692f0 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 5 monclient: authenticate success, global_id 44169 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 10 monclient: _renew_subs 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec11dc50 con 0x7f97ec0692f0 2024-09-15T01:53:11.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8007340 con 0x7f97ec0692f0 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97e8ff9640 10 monclient: dump: 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 10 monclient: _renew_subs 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97ea7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.457+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec11c900 con 0x7f97ec0692f0 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f2c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 0x7f97ec05e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec063b90 con 0x7f97ec09ac80 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97d801b400 con 0x7f97ec0692f0 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 0x7f97ec05e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient: get_auth_request con 0x7f97ec09ac80 auth_method 0 2024-09-15T01:53:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97e8ff9640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97d0038740 0x7f97d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f97d804b900 con 0x7f97ec0692f0 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 10 monclient(hunting): handle_auth_done global_id 44172 payload 824 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f09d0640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97ec11bad0 con 0x7f97ec09ac80 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f11d1640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97d0038740 0x7f97d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.458+0000 7f97f11d1640 10 monclient: get_auth_request con 0x7f97d0038740 auth_method 0 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.460+0000 7f97f09d0640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 msgr2=0x7f97ec05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.460+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 0x7f97ec05e310 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97e000f7f0 tx=0x7f97e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000db30 con 0x7f97ec09ac80 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 10 monclient: dump: 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.484 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f97e000c040 con 0x7f97ec09ac80 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 msgr2=0x7f97ec05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 0x7f97ec05e310 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97e000f7f0 tx=0x7f97e0008040 comp rx=0 tx=0).stop 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 msgr2=0x7f97ec05e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97e8ff9640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec09ac80 0x7f97ec05e310 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97e000f7f0 tx=0x7f97e0008040 comp rx=0 tx=0).stop 2024-09-15T01:53:11.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.461+0000 7f97f11d1640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97d0038740 0x7f97d003abf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97e4034000 tx=0x7f97e4037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.466+0000 7f97f2c34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 0x7f97ec0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f2c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec09c550 con 0x7f97ec05e850 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 0x7f97ec0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient: get_auth_request con 0x7f97ec05e850 auth_method 0 2024-09-15T01:53:11.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 10 monclient(hunting): handle_auth_done global_id 44175 payload 824 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.467+0000 7f97f09d0640 1 -- 172.21.15.17:0/2925312271 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97ec0724a0 con 0x7f97ec05e850 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97f09d0640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 msgr2=0x7f97ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97f09d0640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 0x7f97ec0298c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97e0019000 tx=0x7f97e001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e001b900 con 0x7f97ec05e850 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 10 monclient: dump: 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7f97e001ba60 con 0x7f97ec05e850 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:11.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 msgr2=0x7f97ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 0x7f97ec0298c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97e0019000 tx=0x7f97e001c000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 msgr2=0x7f97ec0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.469+0000 7f97e8ff9640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec05e850 0x7f97ec0298c0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97e0019000 tx=0x7f97e001c000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:11.490 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.491 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.492 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:11.493 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.494 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.496 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.497 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.498 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.501 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:11.502 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:11.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.498+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v38) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4b5403b9f0 con 0x7f4b5c137150 2024-09-15T01:53:11.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.498+0000 7f4b58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.498+0000 7f4b58ff9640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-15T01:53:11.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97d0038740 msgr2=0x7f97d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97d0038740 0x7f97d003abf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97e4034000 tx=0x7f97e4037000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 10 monclient: shutdown 2024-09-15T01:53:11.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 msgr2=0x7f97ec141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 --2- 172.21.15.17:0/2925312271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97ec0692f0 0x7f97ec141990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f97d800c5e0 tx=0x7f97d8004960 comp rx=0 tx=0).stop 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f11d1640 1 -- 172.21.15.17:0/2925312271 reap_dead start 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f11d1640 1 -- 172.21.15.17:0/2925312271 reap_dead start 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 shutdown_connections 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 >> 172.21.15.17:0/2925312271 conn(0x7f97ec070fb0 msgr2=0x7f97ec09de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 shutdown_connections 2024-09-15T01:53:11.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.470+0000 7f97f2c34640 1 -- 172.21.15.17:0/2925312271 wait complete. 2024-09-15T01:53:11.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:11.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:53:11.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:53:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.602+0000 7fd6bb0a8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.602+0000 7fd6bb0a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.602+0000 7fd6bb0a8640 10 monclient: monmap: 2024-09-15T01:53:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:11.603095+0000 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:11.603095+0000 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.605 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.602+0000 7fd6bb0a8640 1 Processor -- start 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 -- start start 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient: init 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b409a9f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6b412e420 0x7fd6b409b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b412e420 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6b412e810 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b4098fb0 con 0x7fd6b412e810 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b412e130 con 0x7fd6b412e420 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b409b300 con 0x7fd6b409a9f0 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6bb0a8640 10 monclient(hunting): authenticate will time out at 851.599604s 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b9645640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd6b412e810 0x7fd6b409bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58760/0 (socket says 172.21.15.17:58760) 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40594/0 (socket says 172.21.15.17:40594) 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b9645640 1 -- 172.21.15.17:0/1544929456 learned_addr learned my addr 172.21.15.17:0/1544929456 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): get_auth_request con 0x7fd6b409a9f0 auth_method 0 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1544929456 >> v2:172.21.15.67:3300/0 conn(0x7fd6b412e420 0x7fd6b409b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): get_auth_request con 0x7fd6b412e420 auth_method 0 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.603+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_done global_id 34340 payload 274 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1544929456 >> v2:172.21.15.67:3300/0 conn(0x7fd6b412e420 msgr2=0x7fd6b409b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1544929456 >> v2:172.21.15.67:3300/0 conn(0x7fd6b412e420 0x7fd6b409b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1544929456 >> v1:172.21.15.17:6789/0 conn(0x7fd6b412e810 legacy=0x7fd6b409bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 1 monclient: found mon.noname-b 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.604+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1544929456 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4098cd0 con 0x7fd6b409a9f0 2024-09-15T01:53:11.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.605+0000 7f4b62cdd640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.605+0000 7f4b62cdd640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.605+0000 7f4b62cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.605+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4b5c09b770 con 0x7f4b5c137150 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.606+0000 7f4b58ff9640 1 -- 172.21.15.67:0/738798506 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v38) v1 ==== 162+0+0 (secure 0 0 0) 0x7f4b540433b0 con 0x7f4b5c137150 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.606+0000 7f4b58ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.606+0000 7f4b58ff9640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foooo' does not exist 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 msgr2=0x7f4b4003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 0x7f4b4003ac40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b50007fd0 tx=0x7f4b50007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 10 monclient: shutdown 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 msgr2=0x7f4b5c098b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009ab0 tx=0x7f4b54036000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 shutdown_connections 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b40038790 0x7f4b4003ac40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b5c069190 0x7f4b5c0985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c137150 0x7f4b5c098b10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 --2- 172.21.15.67:0/738798506 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b5c068dc0 0x7f4b5c098090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 >> 172.21.15.67:0/738798506 conn(0x7f4b5c0711b0 msgr2=0x7f4b5c09eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 shutdown_connections 2024-09-15T01:53:11.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.608+0000 7f4b62cdd640 1 -- 172.21.15.67:0/738798506 wait complete. 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1544929456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 msgr2=0x7fd6b409adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1544929456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009850 tx=0x7fd6b00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39507ec742257a42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 1 -- 172.21.15.17:0/1544929456 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b002be80 con 0x7fd6b409a9f0 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 10 monclient: dump: 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.606+0000 7fd6ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 5 monclient: authenticate success, global_id 34340 2024-09-15T01:53:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 1 -- 172.21.15.17:0/1544929456 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b00304b0 con 0x7fd6b409a9f0 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 4 set_mon_vals no callback set 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 1 -- 172.21.15.17:0/1544929456 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0030830 con 0x7fd6b409a9f0 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6ab7fe640 10 monclient: dump: 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.622 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 10 monclient: shutdown 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1544929456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 msgr2=0x7fd6b409adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1544929456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009850 tx=0x7fd6b00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1544929456 shutdown_connections 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1544929456 >> v2:172.21.15.67:3300/0 conn(0x7fd6b412e420 0x7fd6b409b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1544929456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b409adc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1544929456 >> 172.21.15.17:0/1544929456 conn(0x7fd6b40a1000 msgr2=0x7fd6b40a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1544929456 shutdown_connections 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.607+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1544929456 wait complete. 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 10 monclient: monmap: 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 1 Processor -- start 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 1 -- start start 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 10 monclient: init 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.608+0000 7fd6bb0a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b409a9f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6b412e420 0x7fd6b406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b412e420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6b412e810 0x7fd6b4133110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b412e810 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40983b0 con 0x7fd6b412e420 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b409ef50 con 0x7fd6b412e810 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4134bd0 con 0x7fd6b409a9f0 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6bb0a8640 10 monclient(hunting): authenticate will time out at 851.605066s 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6b412e420 0x7fd6b406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40606/0 (socket says 172.21.15.17:40606) 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1442043281 learned_addr learned my addr 172.21.15.17:0/1442043281 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:11.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): get_auth_request con 0x7fd6b409a9f0 auth_method 0 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): get_auth_request con 0x7fd6b412e420 auth_method 0 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6b412e810 0x7fd6b4133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): get_auth_request con 0x7fd6b412e810 auth_method 0 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b9645640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient(hunting): handle_auth_done global_id 34343 payload 824 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6b412e810 msgr2=0x7fd6b4133110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6b412e810 0x7fd6b4133110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6b412e420 msgr2=0x7fd6b406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6b412e420 0x7fd6b406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 monclient: found mon.noname-c 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b0009390 con 0x7fd6b409a9f0 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.609+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6b412e420 0x7fd6b406f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6b8e44640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 msgr2=0x7fd6b41416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036040 tx=0x7fd6b0033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00339b0 con 0x7fd6b409a9f0 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: dump: 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0033b30 con 0x7fd6b409a9f0 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 5 monclient: authenticate success, global_id 34343 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 10 monclient: _renew_subs 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b406d240 con 0x7fd6b409a9f0 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b003e470 con 0x7fd6b409a9f0 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6a8ff9640 10 monclient: dump: 2024-09-15T01:53:11.631 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6aa7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 10 monclient: _renew_subs 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.611+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b4135730 con 0x7fd6b409a9f0 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6bb0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6b8e44640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036040 tx=0x7fd6b0033000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fd6b006c740 osd_map(38..38 src has 1..38) v4 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 0x7fd6b4133650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4063c20 con 0x7fd6b406f550 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd6b003a070 con 0x7fd6b409a9f0 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 0x7fd6b4133650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient: get_auth_request con 0x7fd6b406f550 auth_method 0 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6a8ff9640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a0038740 0x7fd6a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.612+0000 7fd6abfff640 10 monclient(hunting): handle_auth_done global_id 34346 payload 824 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.613+0000 7fd6abfff640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6b4141d80 con 0x7fd6b406f550 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.613+0000 7fd6b9645640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a0038740 0x7fd6a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.613+0000 7fd6b9645640 10 monclient: get_auth_request con 0x7fd6a0038740 auth_method 0 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6abfff640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 msgr2=0x7fd6b4133650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 0x7fd6b4133650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4014860 con 0x7fd6b406f550 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 10 monclient: dump: 2024-09-15T01:53:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6b9645640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a0038740 0x7fd6a003abf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac006d70 tx=0x7fd6ac0052b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd6a40149c0 con 0x7fd6b406f550 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 msgr2=0x7fd6b4133650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 0x7fd6b4133650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a40090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 msgr2=0x7fd6b4133650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.615+0000 7fd6a8ff9640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b406f550 0x7fd6b4133650 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a40090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.636 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6bb0a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 0x7fd6b4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6bb0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4135920 con 0x7fd6b4142360 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 0x7fd6b4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient: get_auth_request con 0x7fd6b4142360 auth_method 0 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 10 monclient(hunting): handle_auth_done global_id 34349 payload 824 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.621+0000 7fd6abfff640 1 -- 172.21.15.17:0/1442043281 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6b40724a0 con 0x7fd6b4142360 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6abfff640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 msgr2=0x7fd6b4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6abfff640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 0x7fd6b4133b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4020050 con 0x7fd6b4142360 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 10 monclient: dump: 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7fd6a40133a0 con 0x7fd6b4142360 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 msgr2=0x7fd6b4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 0x7fd6b4133b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a401c000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 msgr2=0x7fd6b4133b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.624+0000 7fd6a8ff9640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b4142360 0x7fd6b4133b90 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4134810 tx=0x7fd6a401c000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:11.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.642 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:11.644 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.645 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.649 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:11.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.653 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:11.654 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a0038740 msgr2=0x7fd6a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a0038740 0x7fd6a003abf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac006d70 tx=0x7fd6ac0052b0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 10 monclient: shutdown 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 msgr2=0x7fd6b41416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 --2- 172.21.15.17:0/1442043281 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6b409a9f0 0x7fd6b41416a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036040 tx=0x7fd6b0033000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6b9645640 1 -- 172.21.15.17:0/1442043281 reap_dead start 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6a8ff9640 1 -- 172.21.15.17:0/1442043281 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (unknown 0 0 0) 0x7fd6b006c740 con 0x7fd6b409a9f0 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6b9645640 1 -- 172.21.15.17:0/1442043281 reap_dead start 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 shutdown_connections 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 >> 172.21.15.17:0/1442043281 conn(0x7fd6b40a1000 msgr2=0x7fd6b40997b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 shutdown_connections 2024-09-15T01:53:11.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.625+0000 7fd6bb0a8640 1 -- 172.21.15.17:0/1442043281 wait complete. 2024-09-15T01:53:11.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:11.658 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:53:11.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:53:11.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 10 monclient: monmap: 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:11.696346+0000 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:11.696346+0000 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.699 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.699 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.699 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.699 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 1 Processor -- start 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 1 -- start start 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.695+0000 7f9d97899640 10 monclient: init 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d90068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:11.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d90069190 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d90138160 0x7f9d9013a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-c con 0x7f9d90138160 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d90096c10 con 0x7f9d90069190 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d90132960 con 0x7f9d90068dc0 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9006a2a0 con 0x7f9d90138160 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d97899640 10 monclient(hunting): authenticate will time out at 824.165478s 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d90138160 0x7f9d9013a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95635640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d90068dc0 0x7f9d90132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55034/0 (socket says 172.21.15.67:55034) 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95635640 1 -- 172.21.15.67:0/3537938256 learned_addr learned my addr 172.21.15.67:0/3537938256 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): get_auth_request con 0x7f9d90138160 auth_method 0 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient(hunting): handle_auth_done global_id 4726 payload 274 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 1 -- 172.21.15.67:0/3537938256 >> v1:172.21.15.67:6789/0 conn(0x7f9d90068dc0 legacy=0x7f9d90132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 1 -- 172.21.15.67:0/3537938256 >> v1:172.21.15.17:6789/0 conn(0x7f9d90069190 legacy=0x7f9d90069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 1 monclient: found mon.noname-c 2024-09-15T01:53:11.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.696+0000 7f9d95e36640 1 -- 172.21.15.67:0/3537938256 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9009ff20 con 0x7f9d90138160 2024-09-15T01:53:11.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.699+0000 7f9d95e36640 1 -- 172.21.15.67:0/3537938256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 msgr2=0x7f9d9013a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.699+0000 7f9d95e36640 1 --2- 172.21.15.67:0/3537938256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d9013a540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e60 tx=0x7f9d88009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad2c86a9d7e13b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 1 -- 172.21.15.67:0/3537938256 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88007630 con 0x7f9d90138160 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: dump: 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.710 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 1 -- 172.21.15.67:0/3537938256 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8802d050 con 0x7f9d90138160 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 1 -- 172.21.15.67:0/3537938256 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d880079c0 con 0x7f9d90138160 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 5 monclient: authenticate success, global_id 4726 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 4 set_mon_vals no callback set 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d87fff640 10 monclient: dump: 2024-09-15T01:53:11.711 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 10 monclient: shutdown 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 -- 172.21.15.67:0/3537938256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 msgr2=0x7f9d9013a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 --2- 172.21.15.67:0/3537938256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d9013a540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e60 tx=0x7f9d88009780 comp rx=0 tx=0).stop 2024-09-15T01:53:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 -- 172.21.15.67:0/3537938256 shutdown_connections 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 --2- 172.21.15.67:0/3537938256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d9013a540 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 -- 172.21.15.67:0/3537938256 >> 172.21.15.67:0/3537938256 conn(0x7f9d900711b0 msgr2=0x7f9d900a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 -- 172.21.15.67:0/3537938256 shutdown_connections 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.701+0000 7f9d97899640 1 -- 172.21.15.67:0/3537938256 wait complete. 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.702+0000 7f9d97899640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.702+0000 7f9d97899640 10 monclient: monmap: 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.713 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.702+0000 7f9d97899640 1 Processor -- start 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.702+0000 7f9d97899640 1 -- start start 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.702+0000 7f9d97899640 10 monclient: init 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d90068dc0 0x7f9d900970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-c con 0x7f9d90068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d90069190 0x7f9d90097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-a con 0x7f9d90069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): picked mon.noname-b con 0x7f9d90138160 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d90131960 con 0x7f9d90069190 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900a0890 con 0x7f9d90138160 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9009a3f0 con 0x7f9d90068dc0 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d97899640 10 monclient(hunting): authenticate will time out at 824.172134s 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d90069190 0x7f9d90097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55648/0 (socket says 172.21.15.67:55648) 2024-09-15T01:53:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 -- 172.21.15.67:0/1115307241 learned_addr learned my addr 172.21.15.67:0/1115307241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): get_auth_request con 0x7f9d90138160 auth_method 0 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): get_auth_request con 0x7f9d90069190 auth_method 0 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d90068dc0 0x7f9d900970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): get_auth_request con 0x7f9d90068dc0 auth_method 0 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d94e34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95635640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient(hunting): handle_auth_done global_id 4729 payload 824 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 -- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d90068dc0 msgr2=0x7f9d900970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d90068dc0 0x7f9d900970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 -- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d90069190 msgr2=0x7f9d90097600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d90069190 0x7f9d90097600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 monclient: found mon.noname-b 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.703+0000 7f9d95e36640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88009a20 con 0x7f9d90138160 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.705+0000 7f9d95e36640 1 -- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 msgr2=0x7f9d90097b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.705+0000 7f9d95e36640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e60 tx=0x7f9d88037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88007ec0 con 0x7f9d90138160 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: dump: 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88040070 con 0x7f9d90138160 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 5 monclient: authenticate success, global_id 4729 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 10 monclient: _renew_subs 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d900985e0 con 0x7f9d90138160 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88031d10 con 0x7f9d90138160 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 10 monclient: dump: 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.720 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d877fe640 4 set_mon_vals no callback set 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 10 monclient: _renew_subs 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d90137920 con 0x7f9d90138160 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.706+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d88042030 con 0x7f9d90138160 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d97899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d97899640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d97899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9009a220 con 0x7f9d90138160 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d857fa640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 0x7f9d7403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f9d8806d4e0 con 0x7f9d90138160 2024-09-15T01:53:11.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d95635640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 0x7f9d7403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.707+0000 7f9d95635640 10 monclient: get_auth_request con 0x7f9d74038740 auth_method 0 2024-09-15T01:53:11.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.709+0000 7f9d95635640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 0x7f9d7403abf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007fd0 tx=0x7f9d8c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.711+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d8803b420 con 0x7f9d90138160 2024-09-15T01:53:11.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.711+0000 7f9d857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.711+0000 7f9d857fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.753+0000 7faa6759d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.753+0000 7faa6759d640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.753+0000 7faa6759d640 10 monclient: monmap: 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:11.754301+0000 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:11.754301+0000 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.757 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 Processor -- start 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 -- start start 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient: init 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 0x7faa601321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-c con 0x7faa60068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa600692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 0x7faa601393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-b con 0x7faa60136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa600a1950 con 0x7faa600692f0 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa600a1670 con 0x7faa60068f20 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa600a0020 con 0x7faa60136fe0 2024-09-15T01:53:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa6759d640 10 monclient(hunting): authenticate will time out at 851.750893s 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.754+0000 7faa65b3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 0x7faa601393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa64b38640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7faa600692f0 0x7faa600696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58762/0 (socket says 172.21.15.17:58762) 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa64b38640 1 -- 172.21.15.17:0/806632830 learned_addr learned my addr 172.21.15.17:0/806632830 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): get_auth_request con 0x7faa60136fe0 auth_method 0 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 1 --2- 172.21.15.17:0/806632830 >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 0x7faa601321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 10 monclient(hunting): get_auth_request con 0x7faa60068f20 auth_method 0 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient(hunting): handle_auth_done global_id 34355 payload 274 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 1 -- 172.21.15.17:0/806632830 >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 msgr2=0x7faa601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 1 --2- 172.21.15.17:0/806632830 >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 0x7faa601321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 1 -- 172.21.15.17:0/806632830 >> v1:172.21.15.17:6789/0 conn(0x7faa600692f0 legacy=0x7faa600696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 1 monclient: found mon.noname-b 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65b3a640 1 -- 172.21.15.17:0/806632830 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600a1390 con 0x7faa60136fe0 2024-09-15T01:53:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.755+0000 7faa65339640 1 --2- 172.21.15.17:0/806632830 >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 0x7faa601321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa65b3a640 1 -- 172.21.15.17:0/806632830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 msgr2=0x7faa601393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa65b3a640 1 --2- 172.21.15.17:0/806632830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 0x7faa601393c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa58009e90 tx=0x7faa58009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e81e24948586d49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 1 -- 172.21.15.17:0/806632830 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58007760 con 0x7faa60136fe0 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: dump: 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 5 monclient: authenticate success, global_id 34355 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 1 -- 172.21.15.17:0/806632830 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5802d050 con 0x7faa60136fe0 2024-09-15T01:53:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 4 set_mon_vals no callback set 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 1 -- 172.21.15.17:0/806632830 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58007af0 con 0x7faa60136fe0 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa57fff640 10 monclient: dump: 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 10 monclient: shutdown 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 -- 172.21.15.17:0/806632830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 msgr2=0x7faa601393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 --2- 172.21.15.17:0/806632830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 0x7faa601393c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa58009e90 tx=0x7faa58009780 comp rx=0 tx=0).stop 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 -- 172.21.15.17:0/806632830 shutdown_connections 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 --2- 172.21.15.17:0/806632830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa60136fe0 0x7faa601393c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 --2- 172.21.15.17:0/806632830 >> v2:172.21.15.67:3300/0 conn(0x7faa60068f20 0x7faa601321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 -- 172.21.15.17:0/806632830 >> 172.21.15.17:0/806632830 conn(0x7faa60070fb0 msgr2=0x7faa600a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 -- 172.21.15.17:0/806632830 shutdown_connections 2024-09-15T01:53:11.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.758+0000 7faa6759d640 1 -- 172.21.15.17:0/806632830 wait complete. 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.759+0000 7faa6759d640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.759+0000 7faa6759d640 10 monclient: monmap: 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.759+0000 7faa6759d640 1 Processor -- start 2024-09-15T01:53:11.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.759+0000 7faa6759d640 1 -- start start 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.759+0000 7faa6759d640 10 monclient: init 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60068f20 0x7faa60130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-b con 0x7faa60068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-c con 0x7faa600692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa60136fe0 0x7faa6009a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): picked mon.noname-a con 0x7faa60136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6011cf40 con 0x7faa60136fe0 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6009c550 con 0x7faa60068f20 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa60142270 con 0x7faa600692f0 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa6759d640 10 monclient(hunting): authenticate will time out at 851.756330s 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa60136fe0 0x7faa6009a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40638/0 (socket says 172.21.15.17:40638) 2024-09-15T01:53:11.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 -- 172.21.15.17:0/2681039947 learned_addr learned my addr 172.21.15.17:0/2681039947 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): get_auth_request con 0x7faa600692f0 auth_method 0 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60068f20 0x7faa60130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): get_auth_request con 0x7faa60136fe0 auth_method 0 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): get_auth_request con 0x7faa60068f20 auth_method 0 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient(hunting): handle_auth_done global_id 34358 payload 824 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60068f20 msgr2=0x7faa60130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60068f20 0x7faa60130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa60136fe0 msgr2=0x7faa6009a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa60136fe0 0x7faa6009a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 monclient: found mon.noname-c 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa64b38640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58009a20 con 0x7faa600692f0 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.760+0000 7faa65b3a640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa60136fe0 0x7faa6009a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa64b38640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 msgr2=0x7faa60099e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa64b38640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c5e0 tx=0x7faa50004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50005cb0 con 0x7faa600692f0 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: dump: 2024-09-15T01:53:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa50005e30 con 0x7faa600692f0 2024-09-15T01:53:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 5 monclient: authenticate success, global_id 34358 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 10 monclient: _renew_subs 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50007320 con 0x7faa600692f0 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa601318e0 con 0x7faa600692f0 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa557fa640 10 monclient: dump: 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.780 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa577fe640 4 set_mon_vals no callback set 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 10 monclient: _renew_subs 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.763+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6011bd70 con 0x7faa600692f0 2024-09-15T01:53:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa6759d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa6759d640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 0x7faa600298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6011ce00 con 0x7faa60137810 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faa5001b3b0 con 0x7faa600692f0 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 0x7faa600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient: get_auth_request con 0x7faa60137810 auth_method 0 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa557fa640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa44038740 0x7faa4403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7faa5004b220 con 0x7faa600692f0 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 10 monclient(hunting): handle_auth_done global_id 4738 payload 824 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65b3a640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa44038740 0x7faa4403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65339640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa6011d180 con 0x7faa60137810 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.764+0000 7faa65b3a640 10 monclient: get_auth_request con 0x7faa44038740 auth_method 0 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa65339640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 msgr2=0x7faa600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa65339640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 0x7faa600298c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa5c010900 tx=0x7faa5c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00ed50 con 0x7faa60137810 2024-09-15T01:53:11.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa557fa640 10 monclient: dump: 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.784 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.766+0000 7faa65b3a640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa44038740 0x7faa4403abf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa58031000 tx=0x7faa58037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faa5c00eeb0 con 0x7faa60137810 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 msgr2=0x7faa600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 0x7faa600298c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa5c010900 tx=0x7faa5c008040 comp rx=0 tx=0).stop 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 msgr2=0x7faa600298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.767+0000 7faa557fa640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60137810 0x7faa600298c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.772+0000 7faa6759d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:11.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa6759d640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 0x7faa60029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa6759d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa60063b90 con 0x7faa60131090 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 0x7faa60029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient: get_auth_request con 0x7faa60131090 auth_method 0 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 10 monclient(hunting): handle_auth_done global_id 4741 payload 824 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.773+0000 7faa65339640 1 -- 172.21.15.17:0/2681039947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa600724a0 con 0x7faa60131090 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa65339640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 msgr2=0x7faa60029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa65339640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 0x7faa60029e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa5c01a000 tx=0x7faa5c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c022070 con 0x7faa60131090 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.775+0000 7faa557fa640 10 monclient: dump: 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.789 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7faa5c014e80 con 0x7faa60131090 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 msgr2=0x7faa60029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 0x7faa60029e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa5c01a000 tx=0x7faa5c01d000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 msgr2=0x7faa60029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa557fa640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa60131090 0x7faa60029e00 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa5c01a000 tx=0x7faa5c01d000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 22, 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:11.807 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa44038740 msgr2=0x7faa4403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa44038740 0x7faa4403abf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa58031000 tx=0x7faa58037000 comp rx=0 tx=0).stop 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 10 monclient: shutdown 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 msgr2=0x7faa60099e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c5e0 tx=0x7faa50004960 comp rx=0 tx=0).stop 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa65b3a640 1 -- 172.21.15.17:0/2681039947 reap_dead start 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 shutdown_connections 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 --2- 172.21.15.17:0/2681039947 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa600692f0 0x7faa60099e90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 >> 172.21.15.17:0/2681039947 conn(0x7faa60070fb0 msgr2=0x7faa6009e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 shutdown_connections 2024-09-15T01:53:11.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.776+0000 7faa6759d640 1 -- 172.21.15.17:0/2681039947 wait complete. 2024-09-15T01:53:11.810 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:11.810 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T01:53:11.810 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:53:11.811 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T01:53:11.811 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:53:11.811 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T01:53:11.811 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:53:11.822 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T01:53:11.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:53:11.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.875+0000 7f9d97899640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.875+0000 7f9d97899640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.876+0000 7f9d97899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:11.876+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d9009a220 con 0x7f9d90138160 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 10 monclient: monmap: 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:11.957875+0000 2024-09-15T01:53:11.963 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:11.957875+0000 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 1 Processor -- start 2024-09-15T01:53:11.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 1 -- start start 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.957+0000 7f6e4389f640 10 monclient: init 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e3c131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 0x7f6e3c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-b con 0x7f6e3c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e3c132430 0x7f6e3c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-a con 0x7f6e3c132430 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e3c0a0040 con 0x7f6e3c131b20 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3c09bff0 con 0x7f6e3c132430 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0a14a0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4389f640 10 monclient(hunting): authenticate will time out at 851.954358s 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e40e3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 msgr2=0x7f6e3c131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e40e3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 0x7f6e3c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4163b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6e3c131b20 0x7f6e3c09eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58774/0 (socket says 172.21.15.17:58774) 2024-09-15T01:53:11.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e4163b640 1 -- 172.21.15.17:0/1134743846 learned_addr learned my addr 172.21.15.17:0/1134743846 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e41e3c640 1 --2- 172.21.15.17:0/1134743846 >> v2:172.21.15.17:3300/0 conn(0x7f6e3c132430 0x7f6e3c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e41e3c640 10 monclient(hunting): get_auth_request con 0x7f6e3c132430 auth_method 0 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e41e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e41e3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.958+0000 7f6e41e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e4163b640 1 -- 172.21.15.17:0/1134743846 >> v1:172.21.15.17:6789/0 conn(0x7f6e3c131b20 legacy=0x7f6e3c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_done global_id 44187 payload 274 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 1 -- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 msgr2=0x7f6e3c131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 1 --2- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 0x7f6e3c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 1 -- 172.21.15.17:0/1134743846 >> v1:172.21.15.17:6789/0 conn(0x7f6e3c131b20 legacy=0x7f6e3c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 1 monclient: found mon.noname-a 2024-09-15T01:53:11.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.960+0000 7f6e41e3c640 1 -- 172.21.15.17:0/1134743846 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c09f3b0 con 0x7f6e3c132430 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e41e3c640 1 -- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c132430 msgr2=0x7f6e3c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e41e3c640 1 --2- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c132430 0x7f6e3c138e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ee0 tx=0x7f6e34009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8891fd7830a253 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 1 -- 172.21.15.17:0/1134743846 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e34007840 con 0x7f6e3c132430 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: dump: 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.974 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e4389f640 5 monclient: authenticate success, global_id 44187 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e4389f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:11.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 1 -- 172.21.15.17:0/1134743846 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3402d050 con 0x7f6e3c132430 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e4389f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e4389f640 4 set_mon_vals no callback set 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 1 -- 172.21.15.17:0/1134743846 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e34007bd0 con 0x7f6e3c132430 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.963+0000 7f6e33fff640 10 monclient: dump: 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 10 monclient: shutdown 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 -- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c132430 msgr2=0x7f6e3c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 --2- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c132430 0x7f6e3c138e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ee0 tx=0x7f6e34009780 comp rx=0 tx=0).stop 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 -- 172.21.15.17:0/1134743846 shutdown_connections 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 --2- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c132430 0x7f6e3c138e60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 --2- 172.21.15.17:0/1134743846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c0a0280 0x7f6e3c131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 -- 172.21.15.17:0/1134743846 >> 172.21.15.17:0/1134743846 conn(0x7f6e3c071270 msgr2=0x7f6e3c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 -- 172.21.15.17:0/1134743846 shutdown_connections 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 1 -- 172.21.15.17:0/1134743846 wait complete. 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 10 monclient: build_initial_monmap 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.964+0000 7f6e4389f640 10 monclient: monmap: 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.965+0000 7f6e4389f640 1 Processor -- start 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.965+0000 7f6e4389f640 1 -- start start 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.965+0000 7f6e4389f640 10 monclient: init 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-a con 0x7f6e3c0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:11.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 0x7f6e3c12d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-c con 0x7f6e3c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3c132430 0x7f6e3c12d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): picked mon.noname-b con 0x7f6e3c132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0a09b0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c138f60 con 0x7f6e3c132430 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c097850 con 0x7f6e3c131b20 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4389f640 10 monclient(hunting): authenticate will time out at 851.962327s 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e40e3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 msgr2=0x7f6e3c12d0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:11.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e40e3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 0x7f6e3c12d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54696/0 (socket says 172.21.15.17:54696) 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 -- 172.21.15.17:0/212268452 learned_addr learned my addr 172.21.15.17:0/212268452 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): get_auth_request con 0x7f6e3c0a0280 auth_method 0 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3c132430 0x7f6e3c12d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): get_auth_request con 0x7f6e3c132430 auth_method 0 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e41e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient(hunting): handle_auth_done global_id 44193 payload 824 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 -- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 msgr2=0x7f6e3c12d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:11.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 0x7f6e3c12d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 -- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3c132430 msgr2=0x7f6e3c12d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3c132430 0x7f6e3c12d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 monclient: found mon.noname-a 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.966+0000 7f6e4163b640 1 -- 172.21.15.17:0/212268452 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34009a20 con 0x7f6e3c0a0280 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.969+0000 7f6e4163b640 1 -- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 msgr2=0x7f6e3c09b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.969+0000 7f6e4163b640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c590 tx=0x7f6e38007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38005cb0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:11.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: dump: 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38005e50 con 0x7f6e3c0a0280 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 5 monclient: authenticate success, global_id 44193 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 10 monclient: _renew_subs 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c097de0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38007340 con 0x7f6e3c0a0280 2024-09-15T01:53:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e317fa640 10 monclient: dump: 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 10 monclient: _renew_subs 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e337fe640 4 set_mon_vals no callback set 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.970+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3c09a850 con 0x7f6e3c0a0280 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.971+0000 7f6e4389f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.971+0000 7f6e4389f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.971+0000 7f6e4389f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.971+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3c0724a0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.971+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6e3801b400 con 0x7f6e3c0a0280 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.972+0000 7f6e317fa640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 0x7f6e2003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.972+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f6e3804b630 con 0x7f6e3c0a0280 2024-09-15T01:53:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.972+0000 7f6e40e3a640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 0x7f6e2003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.972+0000 7f6e40e3a640 10 monclient: get_auth_request con 0x7f6e200386f0 auth_method 0 2024-09-15T01:53:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.975+0000 7f6e40e3a640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 0x7f6e2003aba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c097410 tx=0x7f6e2c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.979+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6e380076f0 con 0x7f6e3c0a0280 2024-09-15T01:53:11.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.979+0000 7f6e317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:11.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:11.979+0000 7f6e317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:12.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e4389f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:12.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e4389f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:12.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e4389f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:12.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e3c06efc0 con 0x7f6e3c0a0280 2024-09-15T01:53:12.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e317fa640 1 -- 172.21.15.17:0/212268452 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f6e38019300 con 0x7f6e3c0a0280 2024-09-15T01:53:12.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:12.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.221+0000 7f6e317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:12.224 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:12.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 msgr2=0x7f6e2003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 0x7f6e2003aba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c097410 tx=0x7f6e2c006d80 comp rx=0 tx=0).stop 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 10 monclient: shutdown 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 msgr2=0x7f6e3c09b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c590 tx=0x7f6e38007d80 comp rx=0 tx=0).stop 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 shutdown_connections 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e200386f0 0x7f6e2003aba0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e3c132430 0x7f6e3c12d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e3c131b20 0x7f6e3c12d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:12.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 --2- 172.21.15.17:0/212268452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e3c0a0280 0x7f6e3c09b4b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:12.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 >> 172.21.15.17:0/212268452 conn(0x7f6e3c071270 msgr2=0x7f6e3c130a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:12.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 shutdown_connections 2024-09-15T01:53:12.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:12.223+0000 7f6e4389f640 1 -- 172.21.15.17:0/212268452 wait complete. 2024-09-15T01:53:14.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:14.703+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:14.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:14.703+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:14.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:14.703+0000 7f9d7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:15.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 10 monclient: get_monmap_and_config 2024-09-15T01:53:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 10 monclient: build_initial_monmap 2024-09-15T01:53:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 10 monclient: monmap: 2024-09-15T01:53:15.345 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:15.343955+0000 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:15.343955+0000 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 1 Processor -- start 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 1 -- start start 2024-09-15T01:53:15.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.343+0000 7fc62962b640 10 monclient: init 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 0x7fc62409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-b con 0x7fc624131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc6240a0280 0x7fc624131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc62409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6240a0040 con 0x7fc62409f1e0 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62409bff0 con 0x7fc6240a0280 2024-09-15T01:53:15.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240a14a0 con 0x7fc624131b20 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc62962b640 10 monclient(hunting): authenticate will time out at 855.340335s 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc622ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 msgr2=0x7fc62409eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc622ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 0x7fc62409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc6240a0280 0x7fc624131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc6240a0280 0x7fc624131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46476/0 (socket says 172.21.15.17:46476) 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 1 -- 172.21.15.17:0/433809857 learned_addr learned my addr 172.21.15.17:0/433809857 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 10 monclient(hunting): get_auth_request con 0x7fc6240a0280 auth_method 0 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.344+0000 7fc6227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:15.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6237fe640 1 -- 172.21.15.17:0/433809857 >> v1:172.21.15.17:6789/0 conn(0x7fc62409f1e0 legacy=0x7fc624068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 10 monclient(hunting): handle_auth_done global_id 44196 payload 274 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 1 -- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 msgr2=0x7fc62409eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.346+0000 7fc6227fc640 1 --2- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 0x7fc62409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 1 -- 172.21.15.17:0/433809857 >> v1:172.21.15.17:6789/0 conn(0x7fc62409f1e0 legacy=0x7fc624068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.347+0000 7fc6227fc640 1 -- 172.21.15.17:0/433809857 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624132430 con 0x7fc6240a0280 2024-09-15T01:53:15.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.349+0000 7fc6227fc640 1 -- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 msgr2=0x7fc624131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:15.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.349+0000 7fc6227fc640 1 --2- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc624131ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009990 tx=0x7fc60c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea09cd2caae1188 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:15.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 1 -- 172.21.15.17:0/433809857 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02f4b0 con 0x7fc6240a0280 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: dump: 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 5 monclient: authenticate success, global_id 44196 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 1 -- 172.21.15.17:0/433809857 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c02f610 con 0x7fc6240a0280 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 4 set_mon_vals no callback set 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 1 -- 172.21.15.17:0/433809857 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02f9f0 con 0x7fc6240a0280 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc621ffb640 10 monclient: dump: 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:15.355 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 10 monclient: shutdown 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 -- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 msgr2=0x7fc624131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 --2- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc624131ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009990 tx=0x7fc60c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 -- 172.21.15.17:0/433809857 shutdown_connections 2024-09-15T01:53:15.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 --2- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc624131ef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 --2- 172.21.15.17:0/433809857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc624131b20 0x7fc62409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 -- 172.21.15.17:0/433809857 >> 172.21.15.17:0/433809857 conn(0x7fc624071270 msgr2=0x7fc6240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 -- 172.21.15.17:0/433809857 shutdown_connections 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.350+0000 7fc62962b640 1 -- 172.21.15.17:0/433809857 wait complete. 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.351+0000 7fc62962b640 10 monclient: build_initial_monmap 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.351+0000 7fc62962b640 10 monclient: monmap: 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.351+0000 7fc62962b640 1 Processor -- start 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.351+0000 7fc62962b640 1 -- start start 2024-09-15T01:53:15.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.351+0000 7fc62962b640 10 monclient: init 2024-09-15T01:53:15.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:15.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:15.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 0x7fc6241410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-c con 0x7fc62409f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 0x7fc624098b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): picked mon.noname-b con 0x7fc624131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62411be30 con 0x7fc6240a0280 2024-09-15T01:53:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240a09b0 con 0x7fc624131b20 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6241318f0 con 0x7fc62409f1e0 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc622ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 msgr2=0x7fc6241410d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:15.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc62962b640 10 monclient(hunting): authenticate will time out at 855.348313s 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc622ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 0x7fc6241410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46490/0 (socket says 172.21.15.17:46490) 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 -- 172.21.15.17:0/4229293803 learned_addr learned my addr 172.21.15.17:0/4229293803 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): get_auth_request con 0x7fc6240a0280 auth_method 0 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6237fe640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 0x7fc624098b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:15.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient(hunting): handle_auth_done global_id 44202 payload 824 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 -- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 msgr2=0x7fc6241410d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 0x7fc6241410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 -- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 msgr2=0x7fc624098b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 0x7fc624098b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6227fc640 1 -- 172.21.15.17:0/4229293803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c009450 con 0x7fc6240a0280 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6237fe640 10 monclient: get_auth_request con 0x7fc624131b20 auth_method 0 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.352+0000 7fc6237fe640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 0x7fc624098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.355+0000 7fc6227fc640 1 -- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 msgr2=0x7fc62412ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:15.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.355+0000 7fc6227fc640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009210 tx=0x7fc60c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.355+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c032a40 con 0x7fc6240a0280 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: dump: 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: _finish_auth 0 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 5 monclient: authenticate success, global_id 44202 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c032bc0 con 0x7fc6240a0280 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 10 monclient: _renew_subs 2024-09-15T01:53:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c03c410 con 0x7fc6240a0280 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc624141fb0 con 0x7fc6240a0280 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc61affd640 10 monclient: dump: 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:15.368 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc6217fa640 4 set_mon_vals no callback set 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 10 monclient: _renew_subs 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.357+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62409b250 con 0x7fc6240a0280 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc60c039070 con 0x7fc6240a0280 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc61affd640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 0x7fc60403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:15.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7fc60c06bfb0 con 0x7fc6240a0280 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc62962b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc62962b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc62962b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624141fb0 con 0x7fc6240a0280 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc622ffd640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 0x7fc60403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.358+0000 7fc622ffd640 10 monclient: get_auth_request con 0x7fc604038740 auth_method 0 2024-09-15T01:53:15.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.361+0000 7fc622ffd640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 0x7fc60403abf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc614003cc0 tx=0x7fc614009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:15.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.363+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc624141fb0 con 0x7fc6240a0280 2024-09-15T01:53:15.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.363+0000 7fc61affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:15.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.363+0000 7fc61affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:15.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc62962b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:15.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc62962b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc62962b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc624141fb0 con 0x7fc6240a0280 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc61affd640 1 -- 172.21.15.17:0/4229293803 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fc624141fb0 con 0x7fc6240a0280 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc61affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.587+0000 7fc61affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:15.590 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 msgr2=0x7fc60403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 0x7fc60403abf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc614003cc0 tx=0x7fc614009310 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 10 monclient: shutdown 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 msgr2=0x7fc62412ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009210 tx=0x7fc60c032000 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 shutdown_connections 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc604038740 0x7fc60403abf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6240a0280 0x7fc62412ed10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc624131b20 0x7fc624098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 --2- 172.21.15.17:0/4229293803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc62409f1e0 0x7fc6241410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:15.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 >> 172.21.15.17:0/4229293803 conn(0x7fc624071270 msgr2=0x7fc6240a4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:15.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 shutdown_connections 2024-09-15T01:53:15.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:15.589+0000 7fc62962b640 1 -- 172.21.15.17:0/4229293803 wait complete. 2024-09-15T01:53:15.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:15.703+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:15.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:15.703+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:16.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:16.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:16.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:16.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:17.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:17.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:18.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:18.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:18.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:18.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:18.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:18.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.709+0000 7fac4519d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:18.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.709+0000 7fac4519d640 10 monclient: build_initial_monmap 2024-09-15T01:53:18.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.709+0000 7fac4519d640 10 monclient: monmap: 2024-09-15T01:53:18.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:18.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:18.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:18.710390+0000 2024-09-15T01:53:18.712 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:18.710390+0000 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:18.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 Processor -- start 2024-09-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 -- start start 2024-09-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient: init 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac4006e920 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac4006ecf0 0x7fac4006f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-c con 0x7fac4006ecf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac40134b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4009aef0 con 0x7fac40134b50 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4009d720 con 0x7fac4006e920 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4009b0c0 con 0x7fac4006ecf0 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.710+0000 7fac4519d640 10 monclient(hunting): authenticate will time out at 858.706938s 2024-09-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac3f59e640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fac40134b50 0x7fac40136f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51438/0 (socket says 172.21.15.17:51438) 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac3f59e640 1 -- 172.21.15.17:0/1717251913 learned_addr learned my addr 172.21.15.17:0/1717251913 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 1 --2- 172.21.15.17:0/1717251913 >> v2:172.21.15.67:3300/0 conn(0x7fac4006ecf0 0x7fac4006f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): get_auth_request con 0x7fac4006ecf0 auth_method 0 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient(hunting): handle_auth_done global_id 4747 payload 274 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 1 -- 172.21.15.17:0/1717251913 >> v1:172.21.15.67:6789/0 conn(0x7fac4006e920 legacy=0x7fac4012fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 1 -- 172.21.15.17:0/1717251913 >> v1:172.21.15.17:6789/0 conn(0x7fac40134b50 legacy=0x7fac40136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:18.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 1 monclient: found mon.noname-c 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.711+0000 7fac37fff640 1 -- 172.21.15.17:0/1717251913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4009c860 con 0x7fac4006ecf0 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac37fff640 1 -- 172.21.15.17:0/1717251913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4006ecf0 msgr2=0x7fac4006f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac37fff640 1 --2- 172.21.15.17:0/1717251913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4006ecf0 0x7fac4006f0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac28009850 tx=0x7fac280090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16211ef9586cf8f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 1 -- 172.21.15.17:0/1717251913 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28032440 con 0x7fac4006ecf0 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: dump: 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.735 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:18.735 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.735 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: _finish_auth 0 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 1 -- 172.21.15.17:0/1717251913 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac280325a0 con 0x7fac4006ecf0 2024-09-15T01:53:18.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac4519d640 5 monclient: authenticate success, global_id 4747 2024-09-15T01:53:18.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac4519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:18.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:18.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac4519d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 1 -- 172.21.15.17:0/1717251913 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28032940 con 0x7fac4006ecf0 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac4519d640 4 set_mon_vals no callback set 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.714+0000 7fac3e59c640 10 monclient: dump: 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:18.738 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 10 monclient: shutdown 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 -- 172.21.15.17:0/1717251913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4006ecf0 msgr2=0x7fac4006f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 --2- 172.21.15.17:0/1717251913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4006ecf0 0x7fac4006f0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac28009850 tx=0x7fac280090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 -- 172.21.15.17:0/1717251913 shutdown_connections 2024-09-15T01:53:18.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 --2- 172.21.15.17:0/1717251913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4006ecf0 0x7fac4006f0f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 -- 172.21.15.17:0/1717251913 >> 172.21.15.17:0/1717251913 conn(0x7fac400a1000 msgr2=0x7fac400a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 -- 172.21.15.17:0/1717251913 shutdown_connections 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 1 -- 172.21.15.17:0/1717251913 wait complete. 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 10 monclient: build_initial_monmap 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.715+0000 7fac4519d640 10 monclient: monmap: 2024-09-15T01:53:18.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 Processor -- start 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 -- start start 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient: init 2024-09-15T01:53:18.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-a con 0x7fac4006e920 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 0x7fac4006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-c con 0x7fac4006ecf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 0x7fac4006c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): picked mon.noname-b con 0x7fac40134b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4009bbb0 con 0x7fac4006e920 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4009fb30 con 0x7fac40134b50 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:18.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac401175f0 con 0x7fac4006ecf0 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac37fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 msgr2=0x7fac4006bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac4519d640 10 monclient(hunting): authenticate will time out at 858.712817s 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac37fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 0x7fac4006bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46500/0 (socket says 172.21.15.17:46500) 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 1 -- 172.21.15.17:0/980001906 learned_addr learned my addr 172.21.15.17:0/980001906 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 10 monclient(hunting): get_auth_request con 0x7fac4006e920 auth_method 0 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.716+0000 7fac3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:18.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 0x7fac4006c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 10 monclient(hunting): get_auth_request con 0x7fac40134b50 auth_method 0 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient(hunting): handle_auth_done global_id 44205 payload 824 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 -- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 msgr2=0x7fac4006bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 0x7fac4006bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 -- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 msgr2=0x7fac4006c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 0x7fac4006c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 monclient: found mon.noname-a 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3ed9d640 1 -- 172.21.15.17:0/980001906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28009390 con 0x7fac4006e920 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.717+0000 7fac3f59e640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 0x7fac4006c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac3ed9d640 1 -- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 msgr2=0x7fac4006b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac3ed9d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac3000c5e0 tx=0x7fac30007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30012840 con 0x7fac4006e920 2024-09-15T01:53:18.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: dump: 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac300129c0 con 0x7fac4006e920 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30012d80 con 0x7fac4006e920 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 5 monclient: authenticate success, global_id 44205 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 10 monclient: _renew_subs 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4006d4f0 con 0x7fac4006e920 2024-09-15T01:53:18.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac3d59a640 4 set_mon_vals no callback set 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac36ffd640 10 monclient: dump: 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:18.748 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 10 monclient: _renew_subs 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.719+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4006db50 con 0x7fac4006e920 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac4519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fac30011590 con 0x7fac4006e920 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac4519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac4519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac40117810 con 0x7fac4006e920 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac36ffd640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 0x7fac2003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.720+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7fac3004bd70 con 0x7fac4006e920 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.721+0000 7fac37fff640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 0x7fac2003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:18.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.721+0000 7fac37fff640 10 monclient: get_auth_request con 0x7fac200386f0 auth_method 0 2024-09-15T01:53:18.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.723+0000 7fac37fff640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 0x7fac2003aba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac40069720 tx=0x7fac28032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:18.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.725+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fac40117810 con 0x7fac4006e920 2024-09-15T01:53:18.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.725+0000 7fac36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:18.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.725+0000 7fac36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:18.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac4519d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:18.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac4519d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:18.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac4519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:18.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fac40117810 con 0x7fac4006e920 2024-09-15T01:53:18.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac36ffd640 1 -- 172.21.15.17:0/980001906 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fac40117810 con 0x7fac4006e920 2024-09-15T01:53:18.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:18.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.951+0000 7fac36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:18.954 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 msgr2=0x7fac2003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 0x7fac2003aba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac40069720 tx=0x7fac28032000 comp rx=0 tx=0).stop 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 10 monclient: shutdown 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 msgr2=0x7fac4006b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac3000c5e0 tx=0x7fac30007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 shutdown_connections 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac200386f0 0x7fac2003aba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac40134b50 0x7fac4006c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac4006ecf0 0x7fac4006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 --2- 172.21.15.17:0/980001906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac4006e920 0x7fac4006b6a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 >> 172.21.15.17:0/980001906 conn(0x7fac400a1000 msgr2=0x7fac40096be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 shutdown_connections 2024-09-15T01:53:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:18.954+0000 7fac4519d640 1 -- 172.21.15.17:0/980001906 wait complete. 2024-09-15T01:53:19.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:19.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:19.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:19.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:20.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:20.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:21.324 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T01:53:21.322+0000 7f0afdf15640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-15T01:53:21.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:21.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:21.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:21.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:21.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.073+0000 7faafe59d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.073+0000 7faafe59d640 10 monclient: build_initial_monmap 2024-09-15T01:53:22.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.073+0000 7faafe59d640 10 monclient: monmap: 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:22.074316+0000 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:22.074316+0000 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:22.077 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 Processor -- start 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 -- start start 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient: init 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:53:22.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-c con 0x7faaf806e920 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaf806ecf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-a con 0x7faaf806f110 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf809c860 con 0x7faaf806ecf0 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf809d090 con 0x7faaf806f110 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf809d4b0 con 0x7faaf806e920 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.074+0000 7faafe59d640 10 monclient(hunting): authenticate will time out at 862.070861s 2024-09-15T01:53:22.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46502/0 (socket says 172.21.15.17:46502) 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 learned_addr learned my addr 172.21.15.17:0/754603990 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 10 monclient(hunting): get_auth_request con 0x7faaf806f110 auth_method 0 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faaf806e920 auth_method 0 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.075+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:22.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.077+0000 7faaf77fe640 1 -- 172.21.15.17:0/754603990 >> v1:172.21.15.17:6789/0 conn(0x7faaf806ecf0 legacy=0x7faaf8096c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 10 monclient(hunting): get_auth_request con 0x7faaf806f110 auth_method 0 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faaf806e920 auth_method 0 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.276+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 10 monclient(hunting): get_auth_request con 0x7faaf806f110 auth_method 0 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:22.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faaf806e920 auth_method 0 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:22.677+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:53:22.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:22.705+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:22.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:22.705+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 10 monclient(hunting): get_auth_request con 0x7faaf806f110 auth_method 0 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:23.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.478+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faaf806e920 auth_method 0 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:23.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:23.479+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:53:23.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:23.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:23.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:23.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:24.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:24.705+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:24.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:24.705+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:24.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:24.705+0000 7f9d7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:25.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.074+0000 7faaf5ffb640 10 monclient(hunting): tick 2024-09-15T01:53:25.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.074+0000 7faaf5ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:53:25.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.074+0000 7faaf5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:53:25.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.074+0000 7faaf5ffb640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 msgr2=0x7faaf812fc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:25.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.67:3300/0 conn(0x7faaf806e920 0x7faaf812fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 -- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 msgr2=0x7faaf8097320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 --2- 172.21.15.17:0/754603990 >> v2:172.21.15.17:3300/0 conn(0x7faaf806f110 0x7faaf8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 -- 172.21.15.17:0/754603990 >> v1:172.21.15.17:6789/0 conn(0x7faaf806ecf0 legacy=0x7faaf8096c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7faae40036f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faae4006180 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 -- 172.21.15.17:0/754603990 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf809cad0 con 0x7faae4006180 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 1 -- 172.21.15.17:0/754603990 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf809d090 con 0x7faae40036f0 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf5ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 msgr2=0x7faae4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.075+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.077+0000 7faaf77fe640 1 -- 172.21.15.17:0/754603990 >> v1:172.21.15.67:6789/0 conn(0x7faae4006180 legacy=0x7faae4008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:25.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.276+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 msgr2=0x7faae4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:25.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.276+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:53:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.677+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 msgr2=0x7faae4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:25.677+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:53:25.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:25.704+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:25.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:25.704+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.478+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 msgr2=0x7faae4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:26.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.478+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:53:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.510+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379524070 0 0) 0x7faaf809cad0 con 0x7faae4006180 2024-09-15T01:53:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.510+0000 7faaf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.510+0000 7faaf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.510+0000 7faaf6ffd640 10 monclient(hunting): my global_id is 4753 2024-09-15T01:53:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.510+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae00034e0 con 0x7faae4006180 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3272986606 0 0) 0x7faae00034e0 con 0x7faae4006180 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 msgr2=0x7faae4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae40036f0 0x7faae4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:26.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae400af70 con 0x7faae4006180 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7faaec001730 con 0x7faae4006180 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafe59d640 5 monclient: authenticate success, global_id 4753 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafe59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:26.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: dump: 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.522 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 >> v1:172.21.15.67:6789/0 conn(0x7faae4006180 legacy=0x7faae4008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 reap_dead start 2024-09-15T01:53:26.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 0x7faae0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): picked mon.a con 0x7faae00039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faae0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): picked mon.b con 0x7faae0004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 0x7faae000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): picked mon.c con 0x7faae0005230 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:26.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf809d090 con 0x7faae00039d0 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae000c1f0 con 0x7faae0004780 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae000c3b0 con 0x7faae0005230 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:26.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafcb3a640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 msgr2=0x7faae000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafe59d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faafcb3a640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 0x7faae000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 0x7faae0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faae0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faae00039d0 auth_method 0 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): get_auth_request con 0x7faae0004780 auth_method 0 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:26.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.515+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 10 monclient(hunting): handle_auth_done global_id 4753 payload 293 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 msgr2=0x7faae000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 0x7faae000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 msgr2=0x7faae0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faae0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 monclient: found mon.a 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00034e0 con 0x7faae00039d0 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.516+0000 7faaf77fe640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faae0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:26.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf7fff640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 msgr2=0x7faae0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf7fff640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 0x7faae0003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faae8006ea0 tx=0x7faae8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0f1d4cae5d4d14 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae802bab0 con 0x7faae00039d0 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: dump: 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.537 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae802f070 con 0x7faae00039d0 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faafe59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faafe59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faafe59d640 4 set_mon_vals no callback set 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 1 -- 172.21.15.17:0/754603990 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae802be80 con 0x7faae00039d0 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.519+0000 7faaf6ffd640 10 monclient: dump: 2024-09-15T01:53:26.538 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 10 monclient: shutdown 2024-09-15T01:53:26.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 -- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 msgr2=0x7faae0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 0x7faae0003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faae8006ea0 tx=0x7faae8008040 comp rx=0 tx=0).stop 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 -- 172.21.15.17:0/754603990 shutdown_connections 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae0005230 0x7faae000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faae0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 --2- 172.21.15.17:0/754603990 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae00039d0 0x7faae0003dc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 -- 172.21.15.17:0/754603990 >> 172.21.15.17:0/754603990 conn(0x7faaf80a1000 msgr2=0x7faaf80a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 -- 172.21.15.17:0/754603990 shutdown_connections 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 1 -- 172.21.15.17:0/754603990 wait complete. 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 10 monclient: build_initial_monmap 2024-09-15T01:53:26.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.520+0000 7faafe59d640 10 monclient: monmap: 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.521+0000 7faafe59d640 1 Processor -- start 2024-09-15T01:53:26.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.521+0000 7faafe59d640 1 -- start start 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.521+0000 7faafe59d640 10 monclient: init 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.521+0000 7faafe59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.521+0000 7faafe59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 0x7faaf8140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-c con 0x7faae00039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faaf812ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-b con 0x7faae0004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): picked mon.noname-a con 0x7faae0005230 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf811c170 con 0x7faae0005230 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806f640 con 0x7faae0004780 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806e0b0 con 0x7faae00039d0 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafe59d640 10 monclient(hunting): authenticate will time out at 866.518080s 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf7fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 msgr2=0x7faaf8140c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 0x7faaf8140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40540/0 (socket says 172.21.15.17:40540) 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 -- 172.21.15.17:0/3183423291 learned_addr learned my addr 172.21.15.17:0/3183423291 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): get_auth_request con 0x7faae0005230 auth_method 0 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faaf812ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 10 monclient(hunting): get_auth_request con 0x7faae0004780 auth_method 0 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient(hunting): handle_auth_done global_id 44208 payload 824 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 -- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 msgr2=0x7faaf8140c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 0x7faaf8140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 -- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 msgr2=0x7faaf812ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faaf812ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:26.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 monclient: found mon.noname-a 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faafcb3a640 1 -- 172.21.15.17:0/3183423291 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf811a570 con 0x7faae0005230 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.522+0000 7faaf77fe640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faaf812ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.524+0000 7faafcb3a640 1 -- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 msgr2=0x7faaf8118480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.524+0000 7faafcb3a640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faae4009e70 tx=0x7faaf0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000b2e0 con 0x7faae0005230 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: dump: 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.547 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: _finish_auth 0 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf000f040 con 0x7faae0005230 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 5 monclient: authenticate success, global_id 44208 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 10 monclient: _renew_subs 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81414e0 con 0x7faae0005230 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:26.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000b690 con 0x7faae0005230 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faadffff640 10 monclient: dump: 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 10 monclient: _renew_subs 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8141160 con 0x7faae0005230 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.525+0000 7faaf67fc640 4 set_mon_vals no callback set 2024-09-15T01:53:26.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faaf0009930 con 0x7faae0005230 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faafe59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faafe59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faafe59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf80724a0 con 0x7faae0005230 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faadffff640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 0x7faae403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7faaf004c260 con 0x7faae0005230 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faaf7fff640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 0x7faae403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.526+0000 7faaf7fff640 10 monclient: get_auth_request con 0x7faae4038fa0 auth_method 0 2024-09-15T01:53:26.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.529+0000 7faaf7fff640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 0x7faae403b450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faae8033000 tx=0x7faae8037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.531+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faaf001d960 con 0x7faae0005230 2024-09-15T01:53:26.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.531+0000 7faadffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:26.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.531+0000 7faadffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:26.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:26.705+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:26.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:26.705+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faafe59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faafe59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faafe59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faaf811b790 con 0x7faae0005230 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faadffff640 1 -- 172.21.15.17:0/3183423291 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7faaf0027350 con 0x7faae0005230 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faadffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.763+0000 7faadffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:26.766 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:26.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 msgr2=0x7faae403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 0x7faae403b450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faae8033000 tx=0x7faae8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 10 monclient: shutdown 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 msgr2=0x7faaf8118480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faae4009e70 tx=0x7faaf0009040 comp rx=0 tx=0).stop 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 shutdown_connections 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faae4038fa0 0x7faae403b450 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faae0005230 0x7faaf8118480 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faae0004780 0x7faaf812ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 --2- 172.21.15.17:0/3183423291 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faae00039d0 0x7faaf8140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 >> 172.21.15.17:0/3183423291 conn(0x7faaf80a1000 msgr2=0x7faaf809ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:26.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 shutdown_connections 2024-09-15T01:53:26.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.765+0000 7faafe59d640 1 -- 172.21.15.17:0/3183423291 wait complete. 2024-09-15T01:53:26.778 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:53:26.778 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:53:26.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.873+0000 7f09bd634640 10 monclient: get_monmap_and_config 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.873+0000 7f09bd634640 10 monclient: build_initial_monmap 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.874+0000 7f09bd634640 10 monclient: monmap: 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:26.874836+0000 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:26.874836+0000 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.874+0000 7f09bd634640 1 Processor -- start 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.874+0000 7f09bd634640 1 -- start start 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.874+0000 7f09bd634640 10 monclient: init 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b80a5780 0x7f09b809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-c con 0x7f09b80a5780 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 0x7f09b8069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-b con 0x7f09b809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09b8069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b80a5b50 con 0x7f09b8069ac0 2024-09-15T01:53:26.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b80a18b0 con 0x7f09b80a5780 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b809c5f0 con 0x7f09b809ccf0 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09bd634640 10 monclient(hunting): authenticate will time out at 866.871138s 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b67fc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 msgr2=0x7f09b8069580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b67fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 0x7f09b8069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b77fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f09b8069ac0 0x7f09b8138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47118/0 (socket says 172.21.15.17:47118) 2024-09-15T01:53:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b77fe640 1 -- 172.21.15.17:0/3177452338 learned_addr learned my addr 172.21.15.17:0/3177452338 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/3177452338 >> v2:172.21.15.67:3300/0 conn(0x7f09b80a5780 0x7f09b809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request con 0x7f09b80a5780 auth_method 0 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_done global_id 4759 payload 274 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 -- 172.21.15.17:0/3177452338 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 msgr2=0x7f09b8069580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/3177452338 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 0x7f09b8069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 -- 172.21.15.17:0/3177452338 >> v1:172.21.15.17:6789/0 conn(0x7f09b8069ac0 legacy=0x7f09b8138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 monclient: found mon.noname-c 2024-09-15T01:53:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.875+0000 7f09b6ffd640 1 -- 172.21.15.17:0/3177452338 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b809d210 con 0x7f09b80a5780 2024-09-15T01:53:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b6ffd640 1 -- 172.21.15.17:0/3177452338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 msgr2=0x7f09b809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/3177452338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b809bc90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e60 tx=0x7f09a8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61796ad4cec54b27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b5ffb640 1 -- 172.21.15.17:0/3177452338 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007840 con 0x7f09b80a5780 2024-09-15T01:53:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.878+0000 7f09b5ffb640 10 monclient: dump: 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.889 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 1 -- 172.21.15.17:0/3177452338 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a802d050 con 0x7f09b80a5780 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 5 monclient: authenticate success, global_id 4759 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:26.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 4 set_mon_vals no callback set 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 1 -- 172.21.15.17:0/3177452338 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007c10 con 0x7f09b80a5780 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09b5ffb640 10 monclient: dump: 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.891 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 10 monclient: shutdown 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 -- 172.21.15.17:0/3177452338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 msgr2=0x7f09b809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 --2- 172.21.15.17:0/3177452338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b809bc90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e60 tx=0x7f09a8009780 comp rx=0 tx=0).stop 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 -- 172.21.15.17:0/3177452338 shutdown_connections 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 --2- 172.21.15.17:0/3177452338 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b809ccf0 0x7f09b8069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 --2- 172.21.15.17:0/3177452338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b809bc90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 -- 172.21.15.17:0/3177452338 >> 172.21.15.17:0/3177452338 conn(0x7f09b8070e10 msgr2=0x7f09b80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 -- 172.21.15.17:0/3177452338 shutdown_connections 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.879+0000 7f09bd634640 1 -- 172.21.15.17:0/3177452338 wait complete. 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient: build_initial_monmap 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient: monmap: 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:26.893 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 Processor -- start 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 -- start start 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient: init 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b8069ac0 0x7f09b813e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-c con 0x7f09b8069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 0x7f09b813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-a con 0x7f09b809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b81415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): picked mon.noname-b con 0x7f09b80a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81218d0 con 0x7f09b809ccf0 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8122880 con 0x7f09b80a5780 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8122a80 con 0x7f09b8069ac0 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09b6ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b8069ac0 msgr2=0x7f09b813e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.880+0000 7f09bd634640 10 monclient(hunting): authenticate will time out at 866.876997s 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b8069ac0 0x7f09b813e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 0x7f09b813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 0x7f09b813d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40548/0 (socket says 172.21.15.17:40548) 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 -- 172.21.15.17:0/648954698 learned_addr learned my addr 172.21.15.17:0/648954698 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): get_auth_request con 0x7f09b809ccf0 auth_method 0 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b77fe640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b81415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient(hunting): handle_auth_done global_id 44211 payload 824 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b8069ac0 msgr2=0x7f09b813e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09b8069ac0 0x7f09b813e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 msgr2=0x7f09b81415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80a5780 0x7f09b81415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.881+0000 7f09b67fc640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009a20 con 0x7f09b809ccf0 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09b67fc640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 msgr2=0x7f09b813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09b67fc640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 0x7f09b813d700 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00b5c0 tx=0x7f09ac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac003d90 con 0x7f09b809ccf0 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: dump: 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: _finish_auth 0 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 5 monclient: authenticate success, global_id 44211 2024-09-15T01:53:26.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac006150 con 0x7f09b809ccf0 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 10 monclient: _renew_subs 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b8123040 con 0x7f09b809ccf0 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac006510 con 0x7f09b809ccf0 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f099effd640 10 monclient: dump: 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.900 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 10 monclient: _renew_subs 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b813eb50 con 0x7f09b809ccf0 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.883+0000 7f09b57fa640 4 set_mon_vals no callback set 2024-09-15T01:53:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09bd634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09bd634640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 0x7f09b8141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8063b20 con 0x7f09b81202e0 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f09ac0103c0 con 0x7f09b809ccf0 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 0x7f09b8141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient: get_auth_request con 0x7f09b81202e0 auth_method 0 2024-09-15T01:53:26.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.884+0000 7f099effd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a40386f0 0x7f09a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.885+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f09ac04b4a0 con 0x7f09b809ccf0 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.885+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_done global_id 44214 payload 824 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.885+0000 7f09b6ffd640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09b80724a0 con 0x7f09b81202e0 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.885+0000 7f09b77fe640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a40386f0 0x7f09a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.885+0000 7f09b77fe640 10 monclient: get_auth_request con 0x7f09a40386f0 auth_method 0 2024-09-15T01:53:26.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f09b6ffd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 msgr2=0x7f09b8141b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 0x7f09b8141b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09b80a0c50 tx=0x7f09a8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a80079e0 con 0x7f09b81202e0 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 10 monclient: dump: 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.906 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f09a8040070 con 0x7f09b81202e0 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 msgr2=0x7f09b8141b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f09b77fe640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a40386f0 0x7f09a403aba0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f09b81216f0 tx=0x7f09b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 0x7f09b8141b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09b80a0c50 tx=0x7f09a8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 msgr2=0x7f09b8141b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.887+0000 7f099effd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b81202e0 0x7f09b8141b00 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09b80a0c50 tx=0x7f09a8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09bd634640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09bd634640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 0x7f09b811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09bd634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8142440 con 0x7f09b8142040 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 0x7f09b811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient: get_auth_request con 0x7f09b8142040 auth_method 0 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_done global_id 44217 payload 824 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.893+0000 7f09b6ffd640 1 -- 172.21.15.17:0/648954698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09b806f4f0 con 0x7f09b8142040 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09b6ffd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 msgr2=0x7f09b811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09b6ffd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 0x7f09b811b6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09b81229c0 tx=0x7f09a803e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8056070 con 0x7f09b8142040 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 10 monclient: dump: 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:26.909 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:26.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4556 (secure 0 0 0) 0x7f09a803c920 con 0x7f09b8142040 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 msgr2=0x7f09b811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 0x7f09b811b6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09b81229c0 tx=0x7f09a803e070 comp rx=0 tx=0).stop 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 msgr2=0x7f09b811b6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f099effd640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b8142040 0x7f09b811b6d0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09b81229c0 tx=0x7f09a803e070 comp rx=0 tx=0).stop 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:26.911 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 26, 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:26.912 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:26.913 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.914 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:26.915 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:26.916 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:26.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:26.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:26.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:26.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:53:26.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:26.925 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:26.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:26.925 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:26.925 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:26.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a40386f0 msgr2=0x7f09a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a40386f0 0x7f09a403aba0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f09b81216f0 tx=0x7f09b0008040 comp rx=0 tx=0).stop 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 10 monclient: shutdown 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 msgr2=0x7f09b813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 --2- 172.21.15.17:0/648954698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09b809ccf0 0x7f09b813d700 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00b5c0 tx=0x7f09ac009120 comp rx=0 tx=0).stop 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09b77fe640 1 -- 172.21.15.17:0/648954698 reap_dead start 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09b77fe640 1 -- 172.21.15.17:0/648954698 reap_dead start 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 shutdown_connections 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 >> 172.21.15.17:0/648954698 conn(0x7f09b8070e10 msgr2=0x7f09b80a3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 shutdown_connections 2024-09-15T01:53:26.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:26.896+0000 7f09bd634640 1 -- 172.21.15.17:0/648954698 wait complete. 2024-09-15T01:53:26.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:53:26.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:53:26.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.023+0000 7ff8a615d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.023+0000 7ff8a615d640 10 monclient: build_initial_monmap 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.023+0000 7ff8a615d640 10 monclient: monmap: 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:27.024290+0000 2024-09-15T01:53:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:27.024290+0000 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 Processor -- start 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 -- start start 2024-09-15T01:53:27.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient: init 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8a0097320 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 0x7ff8a01339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a009a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff8a009a960 0x7ff8a0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a009a960 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00985d0 con 0x7ff8a0097320 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a009d200 con 0x7ff8a009a960 2024-09-15T01:53:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00992f0 con 0x7ff8a009a590 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff8a615d640 10 monclient(hunting): authenticate will time out at 867.020870s 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff89effd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 msgr2=0x7ff8a01339d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff89effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 0x7ff8a01339d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff89f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff8a0097320 0x7ff8a00976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47120/0 (socket says 172.21.15.17:47120) 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff89f7fe640 1 -- 172.21.15.17:0/2223605568 learned_addr learned my addr 172.21.15.17:0/2223605568 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.024+0000 7ff89ffff640 1 --2- 172.21.15.17:0/2223605568 >> v2:172.21.15.17:3300/0 conn(0x7ff8a009a960 0x7ff8a0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.025+0000 7ff89ffff640 10 monclient(hunting): get_auth_request con 0x7ff8a009a960 auth_method 0 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.025+0000 7ff89ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.025+0000 7ff89ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.025+0000 7ff89ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:27.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89f7fe640 1 -- 172.21.15.17:0/2223605568 >> v1:172.21.15.17:6789/0 conn(0x7ff8a0097320 legacy=0x7ff8a00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient(hunting): handle_auth_done global_id 44220 payload 274 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 1 -- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 msgr2=0x7ff8a01339d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 1 --2- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 0x7ff8a01339d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 1 -- 172.21.15.17:0/2223605568 >> v1:172.21.15.17:6789/0 conn(0x7ff8a0097320 legacy=0x7ff8a00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 1 monclient: found mon.noname-a 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:27.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.027+0000 7ff89ffff640 1 -- 172.21.15.17:0/2223605568 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a009cb90 con 0x7ff8a009a960 2024-09-15T01:53:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89ffff640 1 -- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a009a960 msgr2=0x7ff8a0135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89ffff640 1 --2- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a009a960 0x7ff8a0135f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff898009ee0 tx=0x7ff898009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf20ef6d1a20ce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 1 -- 172.21.15.17:0/2223605568 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898007840 con 0x7ff8a009a960 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: dump: 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 5 monclient: authenticate success, global_id 44220 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 1 -- 172.21.15.17:0/2223605568 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89802d050 con 0x7ff8a009a960 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 4 set_mon_vals no callback set 2024-09-15T01:53:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 1 -- 172.21.15.17:0/2223605568 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898007bd0 con 0x7ff8a009a960 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff89e7fc640 10 monclient: dump: 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.042 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 10 monclient: shutdown 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 -- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a009a960 msgr2=0x7ff8a0135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 --2- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a009a960 0x7ff8a0135f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff898009ee0 tx=0x7ff898009780 comp rx=0 tx=0).stop 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 -- 172.21.15.17:0/2223605568 shutdown_connections 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 --2- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a009a960 0x7ff8a0135f20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 --2- 172.21.15.17:0/2223605568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a590 0x7ff8a01339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 -- 172.21.15.17:0/2223605568 >> 172.21.15.17:0/2223605568 conn(0x7ff8a00a1000 msgr2=0x7ff8a00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 -- 172.21.15.17:0/2223605568 shutdown_connections 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.030+0000 7ff8a615d640 1 -- 172.21.15.17:0/2223605568 wait complete. 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.031+0000 7ff8a615d640 10 monclient: build_initial_monmap 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.031+0000 7ff8a615d640 10 monclient: monmap: 2024-09-15T01:53:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.031+0000 7ff8a615d640 1 Processor -- start 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.031+0000 7ff8a615d640 1 -- start start 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.031+0000 7ff8a615d640 10 monclient: init 2024-09-15T01:53:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 0x7ff8a012e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a0097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a009a590 0x7ff8a01183c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a009a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a960 0x7ff8a0118900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a009a960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a009ad50 con 0x7ff8a0097320 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0135c70 con 0x7ff8a009a590 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a01426f0 con 0x7ff8a009a960 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff8a615d640 10 monclient(hunting): authenticate will time out at 867.028341s 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a960 msgr2=0x7ff8a0118900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a960 0x7ff8a0118900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 0x7ff8a012e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 0x7ff8a012e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40590/0 (socket says 172.21.15.17:40590) 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 -- 172.21.15.17:0/4207727238 learned_addr learned my addr 172.21.15.17:0/4207727238 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8a0097320 auth_method 0 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient(hunting): handle_auth_done global_id 44226 payload 824 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a960 msgr2=0x7ff8a0118900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8a009a960 0x7ff8a0118900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89effd640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a009a590 0x7ff8a01183c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a009a590 msgr2=0x7ff8a01183c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a009a590 0x7ff8a01183c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 monclient: found mon.noname-a 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.032+0000 7ff89f7fe640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff898009a20 con 0x7ff8a0097320 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.034+0000 7ff89f7fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 msgr2=0x7ff8a012e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff89f7fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 0x7ff8a012e740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff89000c5c0 tx=0x7ff890007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890005d90 con 0x7ff8a0097320 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: dump: 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 5 monclient: authenticate success, global_id 44226 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8900070f0 con 0x7ff8a0097320 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 10 monclient: _renew_subs 2024-09-15T01:53:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8900074b0 con 0x7ff8a0097320 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a011b860 con 0x7ff8a0097320 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8977fe640 10 monclient: dump: 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff89dffb640 4 set_mon_vals no callback set 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 10 monclient: _renew_subs 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.035+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a0136d10 con 0x7ff8a0097320 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8a615d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8a615d640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 0x7ff8a00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0063c20 con 0x7ff8a012ec80 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff89001b400 con 0x7ff8a0097320 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8977fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff880038720 0x7ff88003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7ff89004b7c0 con 0x7ff8a0097320 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 0x7ff8a00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient: get_auth_request con 0x7ff8a012ec80 auth_method 0 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89ffff640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff880038720 0x7ff88003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89ffff640 10 monclient: get_auth_request con 0x7ff880038720 auth_method 0 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.036+0000 7ff89effd640 10 monclient(hunting): handle_auth_done global_id 4762 payload 824 2024-09-15T01:53:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.037+0000 7ff89effd640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8a011bce0 con 0x7ff8a012ec80 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff89effd640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 msgr2=0x7ff8a00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff89effd640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 0x7ff8a00298c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff888003940 tx=0x7ff888008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff89ffff640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff880038720 0x7ff88003abd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0098070 tx=0x7ff898037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888003d90 con 0x7ff8a012ec80 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 10 monclient: dump: 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.055 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff888006150 con 0x7ff8a012ec80 2024-09-15T01:53:27.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 msgr2=0x7ff8a00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 0x7ff8a00298c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff888003940 tx=0x7ff888008040 comp rx=0 tx=0).stop 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 msgr2=0x7ff8a00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.039+0000 7ff8977fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a012ec80 0x7ff8a00298c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff8a615d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff8a615d640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 0x7ff8a002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff8a615d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a012f050 con 0x7ff8a0029e00 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff89effd640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 0x7ff8a002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff89effd640 10 monclient: get_auth_request con 0x7ff8a0029e00 auth_method 0 2024-09-15T01:53:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff89effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.045+0000 7ff89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.046+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.046+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.046+0000 7ff89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.046+0000 7ff89effd640 10 monclient(hunting): handle_auth_done global_id 4765 payload 824 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.046+0000 7ff89effd640 1 -- 172.21.15.17:0/4207727238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff8a00724a0 con 0x7ff8a0029e00 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.048+0000 7ff89effd640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 msgr2=0x7ff8a002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.048+0000 7ff89effd640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 0x7ff8a002a200 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff888003940 tx=0x7ff888017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8880117f0 con 0x7ff8a0029e00 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 10 monclient: dump: 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4724 (secure 0 0 0) 0x7ff888003b80 con 0x7ff8a0029e00 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 msgr2=0x7ff8a002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 0x7ff8a002a200 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff888003940 tx=0x7ff888017000 comp rx=0 tx=0).stop 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 msgr2=0x7ff8a002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8977fe640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a0029e00 0x7ff8a002a200 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff888003940 tx=0x7ff888017000 comp rx=0 tx=0).stop 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 26, 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:53:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:27.074 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff880038720 msgr2=0x7ff88003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff880038720 0x7ff88003abd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0098070 tx=0x7ff898037000 comp rx=0 tx=0).stop 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 10 monclient: shutdown 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 msgr2=0x7ff8a012e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:27.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 --2- 172.21.15.17:0/4207727238 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8a0097320 0x7ff8a012e740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff89000c5c0 tx=0x7ff890007d80 comp rx=0 tx=0).stop 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff89ffff640 1 -- 172.21.15.17:0/4207727238 reap_dead start 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff89ffff640 1 -- 172.21.15.17:0/4207727238 reap_dead start 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 shutdown_connections 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.049+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 >> 172.21.15.17:0/4207727238 conn(0x7ff8a00a1000 msgr2=0x7ff8a00a05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.050+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 shutdown_connections 2024-09-15T01:53:27.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:27.050+0000 7ff8a615d640 1 -- 172.21.15.17:0/4207727238 wait complete. 2024-09-15T01:53:27.076 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:53:27.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:53:27.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.509+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v39) v1 ==== 152+0+0 (secure 0 0 0) 0x7f9d8803a3a0 con 0x7f9d90138160 2024-09-15T01:53:27.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.509+0000 7f9d857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:27.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.509+0000 7f9d857fa640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-15T01:53:27.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.628+0000 7f9d97899640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:27.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.628+0000 7f9d97899640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:27.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.628+0000 7f9d97899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:27.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.628+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d9009a220 con 0x7f9d90138160 2024-09-15T01:53:27.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.705+0000 7f9d7bfff640 10 monclient: tick 2024-09-15T01:53:27.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:27.705+0000 7f9d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.416+0000 7f9d857fa640 1 -- 172.21.15.67:0/1115307241 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v39) v1 ==== 159+0+0 (secure 0 0 0) 0x7f9d8803a580 con 0x7f9d90138160 2024-09-15T01:53:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.416+0000 7f9d857fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.416+0000 7f9d857fa640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T01:53:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fooo' does not exist 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 msgr2=0x7f9d7403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 0x7f9d7403abf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007fd0 tx=0x7f9d8c007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 10 monclient: shutdown 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 msgr2=0x7f9d90097b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e60 tx=0x7f9d88037000 comp rx=0 tx=0).stop 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 shutdown_connections 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d74038740 0x7f9d7403abf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d90069190 0x7f9d90097600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d90138160 0x7f9d90097b40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 --2- 172.21.15.67:0/1115307241 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d90068dc0 0x7f9d900970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 >> 172.21.15.67:0/1115307241 conn(0x7f9d900711b0 msgr2=0x7f9d9009eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 shutdown_connections 2024-09-15T01:53:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.418+0000 7f9d97899640 1 -- 172.21.15.67:0/1115307241 wait complete. 2024-09-15T01:53:28.432 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T01:53:28.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:28.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 10 monclient: build_initial_monmap 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 10 monclient: monmap: 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:28.514442+0000 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:28.514442+0000 2024-09-15T01:53:28.516 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:28.517 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 1 Processor -- start 2024-09-15T01:53:28.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 1 -- start start 2024-09-15T01:53:28.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.513+0000 7ff96979d640 10 monclient: init 2024-09-15T01:53:28.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff9641321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-b con 0x7ff964098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9640a0370 0x7ff964098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9640a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff964098a50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff964098ef0 con 0x7ff964098a50 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9640a1730 con 0x7ff9640a0370 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:28.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9640a0b80 con 0x7ff964098140 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff96979d640 10 monclient(hunting): authenticate will time out at 840.983591s 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff962ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 msgr2=0x7ff9641321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff962ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff9641321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9637fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff964098a50 0x7ff9640695c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47464/0 (socket says 172.21.15.67:47464) 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9637fe640 1 -- 172.21.15.67:0/449801426 learned_addr learned my addr 172.21.15.67:0/449801426 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 1 --2- 172.21.15.67:0/449801426 >> v2:172.21.15.17:3300/0 conn(0x7ff9640a0370 0x7ff964098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): get_auth_request con 0x7ff9640a0370 auth_method 0 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:28.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.514+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 10 monclient(hunting): handle_auth_done global_id 44229 payload 274 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 1 -- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 msgr2=0x7ff9641321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 1 --2- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff9641321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 1 -- 172.21.15.67:0/449801426 >> v1:172.21.15.67:6789/0 conn(0x7ff964098a50 legacy=0x7ff9640695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:28.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.515+0000 7ff9627fc640 1 -- 172.21.15.67:0/449801426 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640a00b0 con 0x7ff9640a0370 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff9627fc640 1 -- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9640a0370 msgr2=0x7ff964098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:28.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff9627fc640 1 --2- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9640a0370 0x7ff964098510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9640a52f0 tx=0x7ff9580090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e55648b7722d397 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 1 -- 172.21.15.67:0/449801426 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958032440 con 0x7ff9640a0370 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: dump: 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:28.530 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:28.531 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 1 -- 172.21.15.67:0/449801426 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9580325a0 con 0x7ff9640a0370 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 1 -- 172.21.15.67:0/449801426 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958032940 con 0x7ff9640a0370 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff96979d640 5 monclient: authenticate success, global_id 44229 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff96979d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff96979d640 4 set_mon_vals no callback set 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.518+0000 7ff961ffb640 10 monclient: dump: 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 10 monclient: shutdown 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 -- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9640a0370 msgr2=0x7ff964098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 --2- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9640a0370 0x7ff964098510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9640a52f0 tx=0x7ff9580090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 -- 172.21.15.67:0/449801426 shutdown_connections 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 --2- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9640a0370 0x7ff964098510 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 --2- 172.21.15.67:0/449801426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff9641321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 -- 172.21.15.67:0/449801426 >> 172.21.15.67:0/449801426 conn(0x7ff9640a5780 msgr2=0x7ff9640a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 -- 172.21.15.67:0/449801426 shutdown_connections 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.519+0000 7ff96979d640 1 -- 172.21.15.67:0/449801426 wait complete. 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.520+0000 7ff96979d640 10 monclient: build_initial_monmap 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.520+0000 7ff96979d640 10 monclient: monmap: 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:28.534 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.520+0000 7ff96979d640 1 Processor -- start 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.520+0000 7ff96979d640 1 -- start start 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.520+0000 7ff96979d640 10 monclient: init 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:28.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff96409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-c con 0x7ff964098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff964098a50 0x7ff964130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-a con 0x7ff964098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9640a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96411e510 con 0x7ff964098a50 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff964131c40 con 0x7ff9640a0370 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96411e230 con 0x7ff964098140 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff96979d640 10 monclient(hunting): authenticate will time out at 840.990145s 2024-09-15T01:53:28.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff962ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 msgr2=0x7ff96409e8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff962ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff96409e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35738/0 (socket says 172.21.15.67:35738) 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 -- 172.21.15.67:0/1799679110 learned_addr learned my addr 172.21.15.67:0/1799679110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): get_auth_request con 0x7ff9640a0370 auth_method 0 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff964098a50 0x7ff964130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): get_auth_request con 0x7ff964098a50 auth_method 0 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:28.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:28.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient(hunting): handle_auth_done global_id 4768 payload 824 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 -- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 msgr2=0x7ff96409e8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff96409e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 -- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff964098a50 msgr2=0x7ff964130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff964098a50 0x7ff964130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:28.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.521+0000 7ff9637fe640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958009390 con 0x7ff9640a0370 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff9637fe640 1 -- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 msgr2=0x7ff964141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff9637fe640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00ccc0 tx=0x7ff95c005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00ce80 con 0x7ff9640a0370 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff94affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff94affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.523+0000 7ff94affd640 10 monclient: dump: 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:28.540 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:28.541 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 10 monclient: _finish_auth 0 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c00aa60 con 0x7ff9640a0370 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 5 monclient: authenticate success, global_id 4768 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 10 monclient: _renew_subs 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:28.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96411cbb0 con 0x7ff9640a0370 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00ae00 con 0x7ff9640a0370 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 10 monclient: dump: 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:28.543 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff9617fa640 4 set_mon_vals no callback set 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 10 monclient: _renew_subs 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96411c0f0 con 0x7ff9640a0370 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.524+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff95c007300 con 0x7ff9640a0370 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff96979d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff96979d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff96979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96404bf60 con 0x7ff9640a0370 2024-09-15T01:53:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff94affd640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 0x7ff95003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4776+0+0 (secure 0 0 0) 0x7ff95c04bce0 con 0x7ff9640a0370 2024-09-15T01:53:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff962ffd640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 0x7ff95003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.525+0000 7ff962ffd640 10 monclient: get_auth_request con 0x7ff9500386f0 auth_method 0 2024-09-15T01:53:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.527+0000 7ff962ffd640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 0x7ff95003aba0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff9540038f0 tx=0x7ff954006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.529+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff95c051020 con 0x7ff9640a0370 2024-09-15T01:53:28.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.529+0000 7ff94affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:28.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.529+0000 7ff94affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:28.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.693+0000 7ff96979d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:28.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.693+0000 7ff96979d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:28.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.693+0000 7ff96979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:28.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:28.693+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff96406f000 con 0x7ff9640a0370 2024-09-15T01:53:29.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.519+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v40) v1 ==== 149+0+0 (secure 0 0 0) 0x7ff95c0075b0 con 0x7ff9640a0370 2024-09-15T01:53:29.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.519+0000 7ff94affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:29.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.519+0000 7ff94affd640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-15T01:53:29.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.644+0000 7ff96979d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:29.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.644+0000 7ff96979d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:29.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.644+0000 7ff96979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.644+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff96411cff0 con 0x7ff9640a0370 2024-09-15T01:53:29.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.646+0000 7ff94affd640 1 -- 172.21.15.67:0/1799679110 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7ff95c0290c0 con 0x7ff9640a0370 2024-09-15T01:53:29.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.646+0000 7ff94affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:29.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.646+0000 7ff94affd640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-15T01:53:29.648 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' does not exist 2024-09-15T01:53:29.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 msgr2=0x7ff95003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:29.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 0x7ff95003aba0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff9540038f0 tx=0x7ff954006d80 comp rx=0 tx=0).stop 2024-09-15T01:53:29.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 10 monclient: shutdown 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 msgr2=0x7ff964141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.647+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00ccc0 tx=0x7ff95c005370 comp rx=0 tx=0).stop 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 shutdown_connections 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9500386f0 0x7ff95003aba0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff964098a50 0x7ff964130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9640a0370 0x7ff964141160 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 --2- 172.21.15.67:0/1799679110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff964098140 0x7ff96409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 >> 172.21.15.67:0/1799679110 conn(0x7ff9640a5780 msgr2=0x7ff964134e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 shutdown_connections 2024-09-15T01:53:29.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.648+0000 7ff96979d640 1 -- 172.21.15.67:0/1799679110 wait complete. 2024-09-15T01:53:29.659 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.734+0000 7fe5ee8a8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.734+0000 7fe5ee8a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.734+0000 7fe5ee8a8640 10 monclient: monmap: 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:29.735804+0000 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:29.735804+0000 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:29.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:29.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:29.738 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:29.738 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:29.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 Processor -- start 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 -- start start 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient: init 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5e80a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5e809eea0 0x7fe5e809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:29.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e809eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5e8136e40 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e806a270 con 0x7fe5e8136e40 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80a0f20 con 0x7fe5e80a0b50 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e806f290 con 0x7fe5e809eea0 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.735+0000 7fe5ee8a8640 10 monclient(hunting): authenticate will time out at 842.204986s 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe5e80a0b50 0x7fe5e8069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47478/0 (socket says 172.21.15.67:47478) 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e7fff640 1 -- 172.21.15.67:0/1055517333 learned_addr learned my addr 172.21.15.67:0/1055517333 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 1 --2- 172.21.15.67:0/1055517333 >> v2:172.21.15.67:3300/0 conn(0x7fe5e809eea0 0x7fe5e809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request con 0x7fe5e809eea0 auth_method 0 2024-09-15T01:53:29.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_done global_id 4771 payload 274 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 1 -- 172.21.15.67:0/1055517333 >> v1:172.21.15.67:6789/0 conn(0x7fe5e80a0b50 legacy=0x7fe5e8069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 1 -- 172.21.15.67:0/1055517333 >> v1:172.21.15.17:6789/0 conn(0x7fe5e8136e40 legacy=0x7fe5e8139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 1 monclient: found mon.noname-c 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.736+0000 7fe5e77fe640 1 -- 172.21.15.67:0/1055517333 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80a6160 con 0x7fe5e809eea0 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e77fe640 1 -- 172.21.15.67:0/1055517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 msgr2=0x7fe5e809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e77fe640 1 --2- 172.21.15.67:0/1055517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809f2a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009850 tx=0x7fe5dc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2acdf00a415d3018 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 1 -- 172.21.15.67:0/1055517333 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc032440 con 0x7fe5e809eea0 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: dump: 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 1 -- 172.21.15.67:0/1055517333 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc0325a0 con 0x7fe5e809eea0 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 5 monclient: authenticate success, global_id 4771 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 1 -- 172.21.15.67:0/1055517333 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc032940 con 0x7fe5e809eea0 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 4 set_mon_vals no callback set 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5e6ffd640 10 monclient: dump: 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:29.752 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 10 monclient: shutdown 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/1055517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 msgr2=0x7fe5e809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/1055517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809f2a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009850 tx=0x7fe5dc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/1055517333 shutdown_connections 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/1055517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809f2a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/1055517333 >> 172.21.15.67:0/1055517333 conn(0x7fe5e8071290 msgr2=0x7fe5e80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/1055517333 shutdown_connections 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.739+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/1055517333 wait complete. 2024-09-15T01:53:29.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.740+0000 7fe5ee8a8640 10 monclient: build_initial_monmap 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.740+0000 7fe5ee8a8640 10 monclient: monmap: 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.740+0000 7fe5ee8a8640 1 Processor -- start 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.740+0000 7fe5ee8a8640 1 -- start start 2024-09-15T01:53:29.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.740+0000 7fe5ee8a8640 10 monclient: init 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e809eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 0x7fe5e811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5e8136e40 0x7fe5e811be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e8136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e811f440 con 0x7fe5e8136e40 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e811ed40 con 0x7fe5e809eea0 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:29.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80a1260 con 0x7fe5e80a0b50 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ee8a8640 10 monclient(hunting): authenticate will time out at 842.210339s 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35756/0 (socket says 172.21.15.67:35756) 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 -- 172.21.15.67:0/3613422708 learned_addr learned my addr 172.21.15.67:0/3613422708 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): get_auth_request con 0x7fe5e809eea0 auth_method 0 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e77fe640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 msgr2=0x7fe5e811b940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e77fe640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 0x7fe5e811b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient(hunting): handle_auth_done global_id 4774 payload 824 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5ece45640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5e8136e40 0x7fe5e811be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 msgr2=0x7fe5e811b940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 0x7fe5e811b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5e8136e40 msgr2=0x7fe5e811be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5e8136e40 0x7fe5e811be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 monclient: found mon.noname-b 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:29.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.741+0000 7fe5e7fff640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc009390 con 0x7fe5e809eea0 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.743+0000 7fe5e7fff640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 msgr2=0x7fe5e809b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5e7fff640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c610 tx=0x7fe5d8004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5e7fff640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c610 tx=0x7fe5d8004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fe5d8005cb0 mon_map magic: 0 v1 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8005cb0 con 0x7fe5e809eea0 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 10 monclient: dump: 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.758 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 10 monclient: _finish_auth 0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8005e30 con 0x7fe5e809eea0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 5 monclient: authenticate success, global_id 4774 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 10 monclient: _renew_subs 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e813a4e0 con 0x7fe5e809eea0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 10 monclient: _renew_subs 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e811d140 con 0x7fe5e809eea0 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.744+0000 7fe5e67fc640 4 set_mon_vals no callback set 2024-09-15T01:53:29.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8007340 con 0x7fe5e809eea0 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 10 monclient: dump: 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:29.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe5d801b400 con 0x7fe5e809eea0 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5ee8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5ee8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5ee8a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5cffff640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 0x7fe5d403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e804bf60 con 0x7fe5e809eea0 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5e77fe640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 0x7fe5d403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.745+0000 7fe5e77fe640 10 monclient: get_auth_request con 0x7fe5d4038770 auth_method 0 2024-09-15T01:53:29.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.747+0000 7fe5e77fe640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 0x7fe5d403ac20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe5e809cdb0 tx=0x7fe5dc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:29.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:29.749+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fe5d804b6b0 con 0x7fe5e809eea0 2024-09-15T01:53:30.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.367+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe5d80076f0 con 0x7fe5e809eea0 2024-09-15T01:53:30.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.367+0000 7fe5cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:30.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.367+0000 7fe5cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.533+0000 7fe5ee8a8640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.533+0000 7fe5ee8a8640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.533+0000 7fe5ee8a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.533+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5e806f000 con 0x7fe5e809eea0 2024-09-15T01:53:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.535+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe5d801a360 con 0x7fe5e809eea0 2024-09-15T01:53:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.535+0000 7fe5cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.535+0000 7fe5cffff640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-15T01:53:30.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.635+0000 7fe5ee8a8640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.635+0000 7fe5ee8a8640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.635+0000 7fe5ee8a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.635+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5e811d800 con 0x7fe5e809eea0 2024-09-15T01:53:30.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.636+0000 7fe5cffff640 1 -- 172.21.15.67:0/3613422708 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe5d801a540 con 0x7fe5e809eea0 2024-09-15T01:53:30.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.636+0000 7fe5cffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.636+0000 7fe5cffff640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-15T01:53:30.639 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' does not exist 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 msgr2=0x7fe5d403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 0x7fe5d403ac20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe5e809cdb0 tx=0x7fe5dc032000 comp rx=0 tx=0).stop 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 10 monclient: shutdown 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 msgr2=0x7fe5e809b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c610 tx=0x7fe5d8004960 comp rx=0 tx=0).stop 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 shutdown_connections 2024-09-15T01:53:30.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5d4038770 0x7fe5d403ac20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5e8136e40 0x7fe5e811be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5e80a0b50 0x7fe5e811b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 --2- 172.21.15.67:0/3613422708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e809eea0 0x7fe5e809b1d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 >> 172.21.15.67:0/3613422708 conn(0x7fe5e8071290 msgr2=0x7fe5e8139c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 shutdown_connections 2024-09-15T01:53:30.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.638+0000 7fe5ee8a8640 1 -- 172.21.15.67:0/3613422708 wait complete. 2024-09-15T01:53:30.649 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T01:53:30.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.724+0000 7f6a96657640 10 monclient: get_monmap_and_config 2024-09-15T01:53:30.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.724+0000 7f6a96657640 10 monclient: build_initial_monmap 2024-09-15T01:53:30.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.724+0000 7f6a96657640 10 monclient: monmap: 2024-09-15T01:53:30.727 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:30.727 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:30.725992+0000 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:30.725992+0000 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:30.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.725+0000 7f6a96657640 1 Processor -- start 2024-09-15T01:53:30.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.725+0000 7f6a96657640 1 -- start start 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.725+0000 7f6a96657640 10 monclient: init 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a900a0b50 0x7f6a90069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-c con 0x7f6a900a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a9009eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 0x7f6a90132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-a con 0x7f6a9009f2a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9006a270 con 0x7f6a9009eea0 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9006f290 con 0x7f6a9009f2a0 2024-09-15T01:53:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a900a0f20 con 0x7f6a900a0b50 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a96657640 10 monclient(hunting): authenticate will time out at 843.195154s 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a900a0b50 0x7f6a90069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a900a0b50 0x7f6a90069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43192/0 (socket says 172.21.15.67:43192) 2024-09-15T01:53:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6a9009eea0 0x7f6a90131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60374/0 (socket says 172.21.15.67:60374) 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 -- 172.21.15.67:0/3841731306 learned_addr learned my addr 172.21.15.67:0/3841731306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): get_auth_request con 0x7f6a900a0b50 auth_method 0 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient(hunting): handle_auth_done global_id 4777 payload 274 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a94bf4640 1 --2- 172.21.15.67:0/3841731306 >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 0x7f6a90132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 -- 172.21.15.67:0/3841731306 >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 msgr2=0x7f6a90132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 --2- 172.21.15.67:0/3841731306 >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 0x7f6a90132220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 -- 172.21.15.67:0/3841731306 >> v1:172.21.15.67:6789/0 conn(0x7f6a9009eea0 legacy=0x7f6a90131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 monclient: found mon.noname-c 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a8ffff640 1 -- 172.21.15.67:0/3841731306 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900a6160 con 0x7f6a900a0b50 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a94bf4640 10 monclient: get_auth_request con 0x7f6a9009f2a0 auth_method 0 2024-09-15T01:53:30.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.726+0000 7f6a94bf4640 1 --2- 172.21.15.67:0/3841731306 >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 0x7f6a90132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8ffff640 1 -- 172.21.15.67:0/3841731306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a900a0b50 msgr2=0x7f6a90069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8ffff640 1 --2- 172.21.15.67:0/3841731306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a900a0b50 0x7f6a90069340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a9006a450 tx=0x7f6a800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8f1044aeae0fd49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 1 -- 172.21.15.67:0/3841731306 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80032440 con 0x7f6a900a0b50 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:30.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: dump: 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: _finish_auth 0 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:30.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 1 -- 172.21.15.67:0/3841731306 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a800325a0 con 0x7f6a900a0b50 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a96657640 5 monclient: authenticate success, global_id 4777 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a96657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 1 -- 172.21.15.67:0/3841731306 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80032940 con 0x7f6a900a0b50 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a96657640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a96657640 4 set_mon_vals no callback set 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.729+0000 7f6a8effd640 10 monclient: dump: 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.741 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 10 monclient: shutdown 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 -- 172.21.15.67:0/3841731306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a900a0b50 msgr2=0x7f6a90069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 --2- 172.21.15.67:0/3841731306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a900a0b50 0x7f6a90069340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a9006a450 tx=0x7f6a800091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 -- 172.21.15.67:0/3841731306 shutdown_connections 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 --2- 172.21.15.67:0/3841731306 >> v2:172.21.15.17:3300/0 conn(0x7f6a9009f2a0 0x7f6a90132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 --2- 172.21.15.67:0/3841731306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a900a0b50 0x7f6a90069340 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 -- 172.21.15.67:0/3841731306 >> 172.21.15.67:0/3841731306 conn(0x7f6a90071290 msgr2=0x7f6a900a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 -- 172.21.15.67:0/3841731306 shutdown_connections 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 -- 172.21.15.67:0/3841731306 wait complete. 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 10 monclient: build_initial_monmap 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 10 monclient: monmap: 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:30.743 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.730+0000 7f6a96657640 1 Processor -- start 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 -- start start 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient: init 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6a9009eea0 0x7f6a90141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-a con 0x7f6a9009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-b con 0x7f6a9009f2a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 0x7f6a9011e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): picked mon.noname-c con 0x7f6a900a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90122170 con 0x7f6a9009eea0 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90120870 con 0x7f6a9009f2a0 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90121e90 con 0x7f6a900a0b50 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a96657640 10 monclient(hunting): authenticate will time out at 843.200734s 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43202/0 (socket says 172.21.15.67:43202) 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 1 -- 172.21.15.67:0/2506522805 learned_addr learned my addr 172.21.15.67:0/2506522805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:30.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): get_auth_request con 0x7f6a9009f2a0 auth_method 0 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a94bf4640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 msgr2=0x7f6a9011e8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a94bf4640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 0x7f6a9011e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.731+0000 7f6a8ffff640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6a9009eea0 0x7f6a90141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8ffff640 10 monclient(hunting): get_auth_request con 0x7f6a9009eea0 auth_method 0 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:30.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 10 monclient(hunting): handle_auth_done global_id 4780 payload 824 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 msgr2=0x7f6a9011e8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 0x7f6a9011e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6a9009eea0 msgr2=0x7f6a90141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6a9009eea0 0x7f6a90141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.732+0000 7f6a8f7fe640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a80009450 con 0x7f6a9009f2a0 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a8f7fe640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 msgr2=0x7f6a90133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:30.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a8f7fe640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800b510 tx=0x7f6a780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78011840 con 0x7f6a9009f2a0 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: dump: 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:30.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a780119c0 con 0x7f6a9009f2a0 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 5 monclient: authenticate success, global_id 4780 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 10 monclient: _renew_subs 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a90120360 con 0x7f6a9009f2a0 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78011d80 con 0x7f6a9009f2a0 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a87fff640 10 monclient: dump: 2024-09-15T01:53:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 10 monclient: _renew_subs 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a8e7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.734+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a90141e00 con 0x7f6a9009f2a0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6a78010590 con 0x7f6a9009f2a0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a96657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a96657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a96657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a87fff640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 0x7f6a7003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9004bf60 con 0x7f6a9009f2a0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f6a7804ba10 con 0x7f6a9009f2a0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a8ffff640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 0x7f6a7003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:30.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.735+0000 7f6a8ffff640 10 monclient: get_auth_request con 0x7f6a700386f0 auth_method 0 2024-09-15T01:53:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.738+0000 7f6a8ffff640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 0x7f6a7003aba0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a900a08b0 tx=0x7f6a80032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.739+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6a7800f6a0 con 0x7f6a9009f2a0 2024-09-15T01:53:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.740+0000 7f6a87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:30.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.740+0000 7f6a87fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:30.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.905+0000 7f6a96657640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.905+0000 7f6a96657640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.905+0000 7f6a96657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:30.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.905+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6a9006f000 con 0x7f6a9009f2a0 2024-09-15T01:53:30.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.916+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7f6a78015030 con 0x7f6a9009f2a0 2024-09-15T01:53:30.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.916+0000 7f6a87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:30.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:30.916+0000 7f6a87fff640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-15T01:53:31.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.017+0000 7f6a96657640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.017+0000 7f6a96657640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.017+0000 7f6a96657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.017+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6a901222e0 con 0x7f6a9009f2a0 2024-09-15T01:53:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.018+0000 7f6a87fff640 1 -- 172.21.15.67:0/2506522805 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7f6a78015020 con 0x7f6a9009f2a0 2024-09-15T01:53:31.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.018+0000 7f6a87fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.018+0000 7f6a87fff640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T01:53:31.021 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fooo' does not exist 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 msgr2=0x7f6a7003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 0x7f6a7003aba0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a900a08b0 tx=0x7f6a80032000 comp rx=0 tx=0).stop 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 10 monclient: shutdown 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 msgr2=0x7f6a90133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800b510 tx=0x7f6a780090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 shutdown_connections 2024-09-15T01:53:31.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6a700386f0 0x7f6a7003aba0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6a900a0b50 0x7f6a9011e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a9009f2a0 0x7f6a90133d60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 --2- 172.21.15.67:0/2506522805 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6a9009eea0 0x7f6a90141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 >> 172.21.15.67:0/2506522805 conn(0x7f6a90071290 msgr2=0x7f6a9009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 shutdown_connections 2024-09-15T01:53:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.020+0000 7f6a96657640 1 -- 172.21.15.67:0/2506522805 wait complete. 2024-09-15T01:53:31.031 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.106+0000 7f6731f1d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.106+0000 7f6731f1d640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.106+0000 7f6731f1d640 10 monclient: monmap: 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.109 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:31.108017+0000 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:31.108017+0000 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:31.110 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.107+0000 7f6731f1d640 1 Processor -- start 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.107+0000 7f6731f1d640 1 -- start start 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.107+0000 7f6731f1d640 10 monclient: init 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f672c068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f672c069190 0x7f672c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f672c069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f672c137150 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:31.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c096c10 con 0x7f672c137150 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c132960 con 0x7f672c068dc0 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0a14d0 con 0x7f672c069190 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f6731f1d640 10 monclient(hunting): authenticate will time out at 843.577138s 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f672c068dc0 0x7f672c132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60382/0 (socket says 172.21.15.67:60382) 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 learned_addr learned my addr 172.21.15.67:0/3221895569 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 1 --2- 172.21.15.67:0/3221895569 >> v2:172.21.15.17:3300/0 conn(0x7f672c069190 0x7f672c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): get_auth_request con 0x7f672c069190 auth_method 0 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.108+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 >> v1:172.21.15.67:6789/0 conn(0x7f672c068dc0 legacy=0x7f672c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279376567 0 0) 0x7f672c132960 con 0x7f672c068dc0 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): my global_id is 4783 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67140034e0 con 0x7f672c068dc0 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196322125 0 0) 0x7f67140034e0 con 0x7f672c068dc0 2024-09-15T01:53:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672bfff640 1 -- 172.21.15.67:0/3221895569 >> v1:172.21.15.17:6789/0 conn(0x7f672c137150 legacy=0x7f672c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 >> v2:172.21.15.17:3300/0 conn(0x7f672c069190 msgr2=0x7f672c069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 --2- 172.21.15.67:0/3221895569 >> v2:172.21.15.17:3300/0 conn(0x7f672c069190 0x7f672c069590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 >> v1:172.21.15.17:6789/0 conn(0x7f672c137150 legacy=0x7f672c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c09ff20 con 0x7f672c068dc0 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f6731f1d640 5 monclient: authenticate success, global_id 4783 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f671c001720 con 0x7f672c068dc0 2024-09-15T01:53:31.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f6731f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: dump: 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.122 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 >> v1:172.21.15.67:6789/0 conn(0x7f672c068dc0 legacy=0x7f672c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f6714003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): picked mon.b con 0x7f6714003b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6714004780 0x7f6714004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): picked mon.a con 0x7f6714004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6714005330 0x7f671400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): picked mon.c con 0x7f6714005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671400c2a0 con 0x7f6714004780 2024-09-15T01:53:31.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671400c410 con 0x7f6714003b20 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671400c5d0 con 0x7f6714005330 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f672a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.110+0000 7f6731f1d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f6714003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672bfff640 1 -- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6714005330 msgr2=0x7f671400b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672bfff640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6714005330 0x7f671400b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): get_auth_request con 0x7f6714003b20 auth_method 0 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6714004780 0x7f6714004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): get_auth_request con 0x7f6714004780 auth_method 0 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:31.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient(hunting): handle_auth_done global_id 4783 payload 293 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6714005330 msgr2=0x7f671400b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6714005330 0x7f671400b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6714004780 msgr2=0x7f6714004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6714004780 0x7f6714004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 monclient: found mon.b 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672bfff640 1 -- 172.21.15.67:0/3221895569 reap_dead start 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67140034e0 con 0x7f6714003b20 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.111+0000 7f672b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672b7fe640 1 -- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 msgr2=0x7f6714003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672b7fe640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f6714003f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f671c001490 tx=0x7f672c0a00b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2920d72f319bee12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c005890 con 0x7f6714003b20 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: dump: 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.127 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c005a10 con 0x7f6714003b20 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 4 set_mon_vals no callback set 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 1 -- 172.21.15.67:0/3221895569 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c005dd0 con 0x7f6714003b20 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f672a7fc640 10 monclient: dump: 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.128 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 10 monclient: shutdown 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 -- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 msgr2=0x7f6714003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f6714003f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f671c001490 tx=0x7f672c0a00b0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 -- 172.21.15.67:0/3221895569 shutdown_connections 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 --2- 172.21.15.67:0/3221895569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f6714003f30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 -- 172.21.15.67:0/3221895569 >> 172.21.15.67:0/3221895569 conn(0x7f672c0711b0 msgr2=0x7f672c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 -- 172.21.15.67:0/3221895569 shutdown_connections 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.114+0000 7f6731f1d640 1 -- 172.21.15.67:0/3221895569 wait complete. 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.115+0000 7f6731f1d640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.115+0000 7f6731f1d640 10 monclient: monmap: 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.130 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.115+0000 7f6731f1d640 1 Processor -- start 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.115+0000 7f6731f1d640 1 -- start start 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.115+0000 7f6731f1d640 10 monclient: init 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6714003b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c0980c0 0x7f672c098490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f672c0980c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 0x7f672c1475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f672c0989d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c12f820 con 0x7f672c0980c0 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0a0890 con 0x7f6714003b20 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1318b0 con 0x7f672c0989d0 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f6731f1d640 10 monclient(hunting): authenticate will time out at 843.585228s 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672bfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 msgr2=0x7f672c1475f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43228/0 (socket says 172.21.15.67:43228) 2024-09-15T01:53:31.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 -- 172.21.15.67:0/2488751187 learned_addr learned my addr 172.21.15.67:0/2488751187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): get_auth_request con 0x7f6714003b20 auth_method 0 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672bfff640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 0x7f672c1475f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672affd640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c0980c0 0x7f672c098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672affd640 10 monclient(hunting): get_auth_request con 0x7f672c0980c0 auth_method 0 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient(hunting): handle_auth_done global_id 4786 payload 824 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 -- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 msgr2=0x7f672c1475f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 0x7f672c1475f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 -- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c0980c0 msgr2=0x7f672c098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c0980c0 0x7f672c098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.116+0000 7f672b7fe640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c09b430 con 0x7f6714003b20 2024-09-15T01:53:31.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.118+0000 7f672b7fe640 1 -- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 msgr2=0x7f672c097b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.118+0000 7f672b7fe640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f671c02af70 tx=0x7f671c034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c007510 con 0x7f6714003b20 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: dump: 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.137 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c007670 con 0x7f6714003b20 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 5 monclient: authenticate success, global_id 4786 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 10 monclient: _renew_subs 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c099620 con 0x7f6714003b20 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c03a640 con 0x7f6714003b20 2024-09-15T01:53:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f67137fe640 10 monclient: dump: 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 10 monclient: _renew_subs 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c098fa0 con 0x7f6714003b20 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.119+0000 7f6729ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f671c03a9f0 con 0x7f6714003b20 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f6731f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f6731f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f6731f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c04bf60 con 0x7f6714003b20 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f67137fe640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 0x7f671803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f671c06c0a0 con 0x7f6714003b20 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f672affd640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 0x7f671803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.120+0000 7f672affd640 10 monclient: get_auth_request con 0x7f6718038720 auth_method 0 2024-09-15T01:53:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.123+0000 7f672affd640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 0x7f671803abd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6720006d30 tx=0x7f6720008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.124+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f671c042550 con 0x7f6714003b20 2024-09-15T01:53:31.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.124+0000 7f67137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.124+0000 7f67137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:31.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.288+0000 7f6731f1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.288+0000 7f6731f1d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.288+0000 7f6731f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.288+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f672c098fa0 con 0x7f6714003b20 2024-09-15T01:53:31.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.289+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7f672c098fa0 con 0x7f6714003b20 2024-09-15T01:53:31.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.289+0000 7f67137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.289+0000 7f67137fe640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-15T01:53:31.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.387+0000 7f6731f1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.387+0000 7f6731f1d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.387+0000 7f6731f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.387+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f672c098fa0 con 0x7f6714003b20 2024-09-15T01:53:31.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.388+0000 7f67137fe640 1 -- 172.21.15.67:0/2488751187 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7f672c098fa0 con 0x7f6714003b20 2024-09-15T01:53:31.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.388+0000 7f67137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.388+0000 7f67137fe640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T01:53:31.391 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fooo' does not exist 2024-09-15T01:53:31.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 msgr2=0x7f671803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 0x7f671803abd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6720006d30 tx=0x7f6720008040 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 10 monclient: shutdown 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 msgr2=0x7f672c097b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f671c02af70 tx=0x7f671c034000 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 shutdown_connections 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6718038720 0x7f671803abd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f672c0980c0 0x7f672c098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f672c0989d0 0x7f672c1475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 --2- 172.21.15.67:0/2488751187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6714003b20 0x7f672c097b80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 >> 172.21.15.67:0/2488751187 conn(0x7f672c0711b0 msgr2=0x7f672c0a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 shutdown_connections 2024-09-15T01:53:31.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.390+0000 7f6731f1d640 1 -- 172.21.15.67:0/2488751187 wait complete. 2024-09-15T01:53:31.401 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.476+0000 7faa7fa6f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.476+0000 7faa7fa6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.476+0000 7faa7fa6f640 10 monclient: monmap: 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:31.477656+0000 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:31.477656+0000 2024-09-15T01:53:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 Processor -- start 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 -- start start 2024-09-15T01:53:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient: init 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa78131c80 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faa78132050 0x7faa78132450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-a con 0x7faa78132050 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 0x7faa78139390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-b con 0x7faa78136fb0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa780a1c10 con 0x7faa78131c80 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa780a0030 con 0x7faa78132050 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa780a0470 con 0x7faa78136fb0 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7fa6f640 10 monclient(hunting): authenticate will time out at 843.946806s 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7d80b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faa78131c80 0x7faa7809eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60388/0 (socket says 172.21.15.67:60388) 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7d80b640 1 -- 172.21.15.67:0/1559052715 learned_addr learned my addr 172.21.15.67:0/1559052715 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7e00c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 msgr2=0x7faa78139390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7e00c640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 0x7faa78139390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.477+0000 7faa7d00a640 1 --2- 172.21.15.67:0/1559052715 >> v2:172.21.15.17:3300/0 conn(0x7faa78132050 0x7faa78132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): get_auth_request con 0x7faa78132050 auth_method 0 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient(hunting): handle_auth_done global_id 44244 payload 274 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 1 -- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 msgr2=0x7faa78139390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 0x7faa78139390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 1 -- 172.21.15.67:0/1559052715 >> v1:172.21.15.67:6789/0 conn(0x7faa78131c80 legacy=0x7faa7809eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 1 monclient: found mon.noname-a 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:31.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.478+0000 7faa7d00a640 1 -- 172.21.15.67:0/1559052715 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780a0290 con 0x7faa78132050 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7d00a640 1 -- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78132050 msgr2=0x7faa78132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7d00a640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78132050 0x7faa78132450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faa68009850 tx=0x7faa680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2970c06d2d101450 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 1 -- 172.21.15.67:0/1559052715 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68032440 con 0x7faa78132050 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: dump: 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 1 -- 172.21.15.67:0/1559052715 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa680325a0 con 0x7faa78132050 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 1 -- 172.21.15.67:0/1559052715 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68032940 con 0x7faa78132050 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7c809640 10 monclient: dump: 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.493 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7fa6f640 5 monclient: authenticate success, global_id 44244 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7fa6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.481+0000 7faa7fa6f640 4 set_mon_vals no callback set 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 10 monclient: shutdown 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78132050 msgr2=0x7faa78132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78132050 0x7faa78132450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faa68009850 tx=0x7faa680090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1559052715 shutdown_connections 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78136fb0 0x7faa78139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1559052715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78132050 0x7faa78132450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1559052715 >> 172.21.15.67:0/1559052715 conn(0x7faa780a52e0 msgr2=0x7faa780a56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1559052715 shutdown_connections 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1559052715 wait complete. 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.482+0000 7faa7fa6f640 10 monclient: monmap: 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 Processor -- start 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 -- start start 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient: init 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78131c80 0x7faa78130bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-a con 0x7faa78131c80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 0x7faa78131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-c con 0x7faa78132050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): picked mon.noname-b con 0x7faa78136fb0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7811d330 con 0x7faa78131c80 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa78137850 con 0x7faa78136fb0 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7811c9f0 con 0x7faa78132050 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7fa6f640 10 monclient(hunting): authenticate will time out at 843.952578s 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d00a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 msgr2=0x7faa78131110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d00a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 0x7faa78131110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43238/0 (socket says 172.21.15.67:43238) 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 1 -- 172.21.15.67:0/1128300082 learned_addr learned my addr 172.21.15.67:0/1128300082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): get_auth_request con 0x7faa78136fb0 auth_method 0 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78131c80 0x7faa78130bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): get_auth_request con 0x7faa78131c80 auth_method 0 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7e00c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.483+0000 7faa7d80b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 10 monclient(hunting): handle_auth_done global_id 4789 payload 824 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 -- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 msgr2=0x7faa78131110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 0x7faa78131110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 -- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78131c80 msgr2=0x7faa78130bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78131c80 0x7faa78130bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 monclient: found mon.noname-b 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.484+0000 7faa7e00c640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa68009390 con 0x7faa78136fb0 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7e00c640 1 -- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 msgr2=0x7faa78099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7e00c640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa7000bf50 tx=0x7faa70009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000cb80 con 0x7faa78136fb0 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: dump: 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.500 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 5 monclient: authenticate success, global_id 4789 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7000cd00 con 0x7faa78136fb0 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 10 monclient: _renew_subs 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7811bd80 con 0x7faa78136fb0 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70006150 con 0x7faa78136fb0 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6dffb640 10 monclient: dump: 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 10 monclient: _renew_subs 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa6ffff640 4 set_mon_vals no callback set 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.486+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7811cd90 con 0x7faa78136fb0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faa70012020 con 0x7faa78136fb0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa6dffb640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 0x7faa5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7faa7004b720 con 0x7faa78136fb0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7fa6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7fa6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7d80b640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 0x7faa5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.487+0000 7faa7d80b640 10 monclient: get_auth_request con 0x7faa5c038740 auth_method 0 2024-09-15T01:53:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.490+0000 7faa7d80b640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 0x7faa5c03abf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa74007fd0 tx=0x7faa74007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.492+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.492+0000 7faa6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.492+0000 7faa6dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.654+0000 7faa7fa6f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.654+0000 7faa7fa6f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.654+0000 7faa7fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.654+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.655+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (secure 0 0 0) 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.655+0000 7faa6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.655+0000 7faa6dffb640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-15T01:53:31.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.752+0000 7faa7fa6f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.752+0000 7faa7fa6f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.752+0000 7faa7fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.752+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.753+0000 7faa6dffb640 1 -- 172.21.15.67:0/1128300082 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (secure 0 0 0) 0x7faa7809b5c0 con 0x7faa78136fb0 2024-09-15T01:53:31.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.753+0000 7faa6dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:31.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.753+0000 7faa6dffb640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-15T01:53:31.756 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fuggg' does not exist 2024-09-15T01:53:31.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 msgr2=0x7faa5c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 0x7faa5c03abf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa74007fd0 tx=0x7faa74007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:31.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 10 monclient: shutdown 2024-09-15T01:53:31.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 msgr2=0x7faa78099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa7000bf50 tx=0x7faa70009700 comp rx=0 tx=0).stop 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 shutdown_connections 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa5c038740 0x7faa5c03abf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa78136fb0 0x7faa78099a20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa78132050 0x7faa78131110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 --2- 172.21.15.67:0/1128300082 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa78131c80 0x7faa78130bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 >> 172.21.15.67:0/1128300082 conn(0x7faa780a52e0 msgr2=0x7faa780a3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 shutdown_connections 2024-09-15T01:53:31.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.755+0000 7faa7fa6f640 1 -- 172.21.15.67:0/1128300082 wait complete. 2024-09-15T01:53:31.766 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo OK 2024-09-15T01:53:31.767 INFO:tasks.workunit.client.0.smithi067.stdout:OK 2024-09-15T01:53:31.768 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T01:53:31.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T01:53:31.794 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-15T01:53:31.795 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-15T01:53:31.795 DEBUG:teuthology.orchestra.run.smithi067:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-15T01:53:31.855 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush dump 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.927+0000 7f65c395d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.927+0000 7f65c395d640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.927+0000 7f65c395d640 10 monclient: monmap: 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:31.928788+0000 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:31.928788+0000 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:31.930 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:31.931 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:31.931 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:31.931 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:31.931 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 Processor -- start 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 -- start start 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient: init 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f65bc130e00 0x7f65bc132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-a con 0x7f65bc130e00 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65bc1311d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65bc1315f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc09ff20 con 0x7f65bc1315f0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc069be0 con 0x7f65bc1311d0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc1320b0 con 0x7f65bc130e00 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.928+0000 7f65c395d640 10 monclient(hunting): authenticate will time out at 844.397923s 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c0ef8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f65bc1311d0 0x7f65bc068dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60390/0 (socket says 172.21.15.67:60390) 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 learned_addr learned my addr 172.21.15.67:0/119649993 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 1 --2- 172.21.15.67:0/119649993 >> v2:172.21.15.17:3300/0 conn(0x7f65bc130e00 0x7f65bc132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): get_auth_request con 0x7f65bc130e00 auth_method 0 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.929+0000 7f65c16f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 >> v1:172.21.15.67:6789/0 conn(0x7f65bc1311d0 legacy=0x7f65bc068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209678122 0 0) 0x7f65bc069be0 con 0x7f65bc1311d0 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): my global_id is 4792 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a80034e0 con 0x7f65bc1311d0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c1efa640 1 -- 172.21.15.67:0/119649993 >> v1:172.21.15.17:6789/0 conn(0x7f65bc1315f0 legacy=0x7f65bc0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237316242 0 0) 0x7f65a80034e0 con 0x7f65bc1311d0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 >> v2:172.21.15.17:3300/0 conn(0x7f65bc130e00 msgr2=0x7f65bc132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 --2- 172.21.15.67:0/119649993 >> v2:172.21.15.17:3300/0 conn(0x7f65bc130e00 0x7f65bc132430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 >> v1:172.21.15.17:6789/0 conn(0x7f65bc1315f0 legacy=0x7f65bc0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0a00f0 con 0x7f65bc1311d0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c395d640 5 monclient: authenticate success, global_id 4792 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c395d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f65b8001720 con 0x7f65bc1311d0 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: dump: 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.940 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 >> v1:172.21.15.67:6789/0 conn(0x7f65bc1311d0 legacy=0x7f65bc068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8003b20 0x7f65a8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): picked mon.c con 0x7f65a8003b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 0x7f65a8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): picked mon.b con 0x7f65a8004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65a8005330 0x7f65a800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): picked mon.a con 0x7f65a8005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:31.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc1320b0 con 0x7f65a8005330 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800c2d0 con 0x7f65a8004780 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800c490 con 0x7f65a8003b20 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65affff640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c395d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c16f9640 1 -- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8003b20 msgr2=0x7f65a8003f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c16f9640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8003b20 0x7f65a8003f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 0x7f65a8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): get_auth_request con 0x7f65a8004780 auth_method 0 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c1efa640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65a8005330 0x7f65a800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.931+0000 7f65c0ef8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): get_auth_request con 0x7f65a8005330 auth_method 0 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 10 monclient(hunting): handle_auth_done global_id 4792 payload 293 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8003b20 msgr2=0x7f65a8003f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8003b20 0x7f65a8003f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65a8005330 msgr2=0x7f65a800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65a8005330 0x7f65a800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 monclient: found mon.b 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c1efa640 1 -- 172.21.15.67:0/119649993 reap_dead start 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a80034e0 con 0x7f65a8004780 2024-09-15T01:53:31.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.932+0000 7f65c0ef8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c0ef8640 1 -- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 msgr2=0x7f65a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c0ef8640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 0x7f65a8004bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f65b8001490 tx=0x7f65b8007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b630586d6746e3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8005890 con 0x7f65a8004780 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: dump: 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.951 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8005a10 con 0x7f65a8004780 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 1 -- 172.21.15.67:0/119649993 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8005dd0 con 0x7f65a8004780 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 4 set_mon_vals no callback set 2024-09-15T01:53:31.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65affff640 10 monclient: dump: 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 10 monclient: shutdown 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 1 -- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 msgr2=0x7f65a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 0x7f65a8004bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f65b8001490 tx=0x7f65b8007a50 comp rx=0 tx=0).stop 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 1 -- 172.21.15.67:0/119649993 shutdown_connections 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 1 --2- 172.21.15.67:0/119649993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a8004780 0x7f65a8004bf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.935+0000 7f65c395d640 1 -- 172.21.15.67:0/119649993 >> 172.21.15.67:0/119649993 conn(0x7f65bc0710f0 msgr2=0x7f65bc0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 1 -- 172.21.15.67:0/119649993 shutdown_connections 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 1 -- 172.21.15.67:0/119649993 wait complete. 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 10 monclient: build_initial_monmap 2024-09-15T01:53:31.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 10 monclient: monmap: 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:31.955 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 1 Processor -- start 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 1 -- start start 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.936+0000 7f65c395d640 10 monclient: init 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 0x7f65bc09c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-c con 0x7f65a8004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:31.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65bc11b8a0 0x7f65bc09cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-a con 0x7f65bc11b8a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): picked mon.noname-b con 0x7f65bc09d140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc09d5b0 con 0x7f65bc11b8a0 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc11c390 con 0x7f65bc09d140 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0981a0 con 0x7f65a8004780 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c395d640 10 monclient(hunting): authenticate will time out at 844.406524s 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c16f9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 msgr2=0x7f65bc09c6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c16f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 0x7f65bc09c6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43256/0 (socket says 172.21.15.67:43256) 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 -- 172.21.15.67:0/2051588305 learned_addr learned my addr 172.21.15.67:0/2051588305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): get_auth_request con 0x7f65bc09d140 auth_method 0 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c0ef8640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65bc11b8a0 0x7f65bc09cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c0ef8640 10 monclient(hunting): get_auth_request con 0x7f65bc11b8a0 auth_method 0 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c0ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c0ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c0ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient(hunting): handle_auth_done global_id 4795 payload 824 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 -- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 msgr2=0x7f65bc09c6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 0x7f65bc09c6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 -- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65bc11b8a0 msgr2=0x7f65bc09cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65bc11b8a0 0x7f65bc09cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient: _finish_hunting 0 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 monclient: found mon.noname-b 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.937+0000 7f65c1efa640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc11c0b0 con 0x7f65bc09d140 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c1efa640 1 -- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 msgr2=0x7f65bc140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c1efa640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0a6250 tx=0x7f65b4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b400fe90 con 0x7f65bc09d140 2024-09-15T01:53:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: dump: 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.961 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4005d10 con 0x7f65bc09d140 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 5 monclient: authenticate success, global_id 4795 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 10 monclient: _renew_subs 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc11bdd0 con 0x7f65bc09d140 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4007300 con 0x7f65bc09d140 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65ad7fa640 10 monclient: dump: 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:31.962 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 10 monclient: _renew_subs 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc12f2b0 con 0x7f65bc09d140 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.940+0000 7f65af7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:31.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f65b4007770 con 0x7f65bc09d140 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c395d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c395d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc11c590 con 0x7f65bc09d140 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65ad7fa640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 0x7f65a003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f65b404b2f0 con 0x7f65bc09d140 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c16f9640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 0x7f65a003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:31.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.941+0000 7f65c16f9640 10 monclient: get_auth_request con 0x7f65a00386f0 auth_method 0 2024-09-15T01:53:31.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.944+0000 7f65c16f9640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 0x7f65a003aba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009430 tx=0x7f65b00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:31.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.945+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f65b401a910 con 0x7f65bc09d140 2024-09-15T01:53:31.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.945+0000 7f65ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:31.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:31.945+0000 7f65ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65c395d640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65c395d640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65c395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f65bc04bf60 con 0x7f65bc09d140 2024-09-15T01:53:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f65b401b360 con 0x7f65bc09d140 2024-09-15T01:53:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.111+0000 7f65ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.211+0000 7f65c395d640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65c395d640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65c395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f65bc0008d0 con 0x7f65bc09d140 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65ad7fa640 1 -- 172.21.15.67:0/2051588305 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f65b401a310 con 0x7f65bc09d140 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65ad7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.212+0000 7f65ad7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stdout: "devices": [ 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.215 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 0, 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.0", 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 1, 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.1", 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.2", 2024-09-15T01:53:32.216 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.3", 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 4, 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.4", 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.217 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd.5", 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "class": "ssd" 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "types": [ 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 0, 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "osd" 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 1, 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "host" 2024-09-15T01:53:32.218 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 2, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "chassis" 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 3, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "rack" 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 4, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "row" 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.219 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 5, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "pdu" 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 6, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "pod" 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 7, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "room" 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 8, 2024-09-15T01:53:32.220 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "datacenter" 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 9, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "zone" 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 10, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "region" 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 11, 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "root" 2024-09-15T01:53:32.221 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "buckets": [ 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -1, 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "default", 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 11, 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "root", 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 34560, 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.222 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -3, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -5, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -2, 2024-09-15T01:53:32.223 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "default~ssd", 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 11, 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "root", 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 34560, 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -4, 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.224 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -6, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -3, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "smithi017", 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 1, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "host", 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.225 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 0, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 1, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.226 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 2 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -4, 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "smithi017~ssd", 2024-09-15T01:53:32.227 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 1, 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "host", 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 0, 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.228 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 1, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 2 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -5, 2024-09-15T01:53:32.229 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "smithi067", 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 1, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "host", 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 4, 2024-09-15T01:53:32.230 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 2 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "id": -6, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "smithi067~ssd", 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "type_id": 1, 2024-09-15T01:53:32.231 INFO:tasks.workunit.client.0.smithi067.stdout: "type_name": "host", 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 17280, 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "alg": "straw2", 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "hash": "rjenkins1", 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "items": [ 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 0 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 4, 2024-09-15T01:53:32.232 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 1 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: "weight": 5760, 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: "pos": 2 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: "rules": [ 2024-09-15T01:53:32.233 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_id": 0, 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_name": "replicated_rule", 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "type": 1, 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "steps": [ 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "take", 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "item": -1, 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "item_name": "default" 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "choose_firstn", 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 0, 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: "type": "osd" 2024-09-15T01:53:32.234 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "emit" 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_id": 1, 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_name": "erasure-code", 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "type": 3, 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "steps": [ 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "set_chooseleaf_tries", 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 5 2024-09-15T01:53:32.235 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "set_choose_tries", 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 100 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "take", 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "item": -1, 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "item_name": "default" 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "choose_indep", 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 0, 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: "type": "osd" 2024-09-15T01:53:32.236 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "emit" 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "tunables": { 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "choose_local_tries": 0, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "choose_local_fallback_tries": 0, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "choose_total_tries": 50, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "chooseleaf_descend_once": 1, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "chooseleaf_vary_r": 1, 2024-09-15T01:53:32.237 INFO:tasks.workunit.client.0.smithi067.stdout: "chooseleaf_stable": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "straw_calc_version": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "allowed_bucket_algs": 54, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "profile": "jewel", 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "optimal_tunables": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "legacy_tunables": 0, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "minimum_required_version": "jewel", 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "require_feature_tunables": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "require_feature_tunables2": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "has_v2_rules": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "require_feature_tunables3": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "has_v3_rules": 0, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "has_v4_buckets": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "require_feature_tunables5": 1, 2024-09-15T01:53:32.238 INFO:tasks.workunit.client.0.smithi067.stdout: "has_v5_rules": 0 2024-09-15T01:53:32.239 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.239 INFO:tasks.workunit.client.0.smithi067.stdout: "choose_args": {} 2024-09-15T01:53:32.239 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-09-15T01:53:32.239 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 msgr2=0x7f65a003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 0x7f65a003aba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009430 tx=0x7f65b00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 10 monclient: shutdown 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 msgr2=0x7f65bc140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0a6250 tx=0x7f65b4009040 comp rx=0 tx=0).stop 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 shutdown_connections 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65a00386f0 0x7f65a003aba0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65bc11b8a0 0x7f65bc09cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65bc09d140 0x7f65bc140fe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 --2- 172.21.15.67:0/2051588305 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65a8004780 0x7f65bc09c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 >> 172.21.15.67:0/2051588305 conn(0x7f65bc0710f0 msgr2=0x7f65bc132c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 shutdown_connections 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.214+0000 7f65c395d640 1 -- 172.21.15.67:0/2051588305 wait complete. 2024-09-15T01:53:32.241 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule dump 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.299+0000 7f86c831d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.299+0000 7f86c831d640 10 monclient: build_initial_monmap 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.299+0000 7f86c831d640 10 monclient: monmap: 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:32.300895+0000 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:32.300895+0000 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:32.303 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 1 Processor -- start 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 1 -- start start 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 10 monclient: init 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.300+0000 7f86c831d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 0x7f86c0132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c0068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c0069190 0x7f86c0069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-c con 0x7f86c0069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:32.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86c012e960 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0096c10 con 0x7f86c012e960 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c00a14d0 con 0x7f86c0069190 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c009ff20 con 0x7f86c0068dc0 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c831d640 10 monclient(hunting): authenticate will time out at 844.770077s 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c60b9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 msgr2=0x7f86c0132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c0069190 0x7f86c0069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c60b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 0x7f86c0132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c0069190 0x7f86c0069560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43264/0 (socket says 172.21.15.67:43264) 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 1 -- 172.21.15.67:0/1577328498 learned_addr learned my addr 172.21.15.67:0/1577328498 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:32.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 10 monclient(hunting): get_auth_request con 0x7f86c0069190 auth_method 0 2024-09-15T01:53:32.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.301+0000 7f86c58b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c68ba640 1 -- 172.21.15.67:0/1577328498 >> v1:172.21.15.67:6789/0 conn(0x7f86c012e960 legacy=0x7f86c012edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient(hunting): handle_auth_done global_id 4798 payload 274 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 1 -- 172.21.15.67:0/1577328498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 msgr2=0x7f86c0132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 1 --2- 172.21.15.67:0/1577328498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 0x7f86c0132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 1 -- 172.21.15.67:0/1577328498 >> v1:172.21.15.67:6789/0 conn(0x7f86c012e960 legacy=0x7f86c012edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient: _finish_hunting 0 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 1 monclient: found mon.noname-c 2024-09-15T01:53:32.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:32.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.303+0000 7f86c58b8640 1 -- 172.21.15.67:0/1577328498 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00a0180 con 0x7f86c0069190 2024-09-15T01:53:32.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c58b8640 1 -- 172.21.15.67:0/1577328498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c0069190 msgr2=0x7f86c0069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c58b8640 1 --2- 172.21.15.67:0/1577328498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c0069190 0x7f86c0069560 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009990 tx=0x7f86b00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ba3acd37ec1f6b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 1 -- 172.21.15.67:0/1577328498 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b002f4b0 con 0x7f86c0069190 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: dump: 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: _finish_auth 0 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 1 -- 172.21.15.67:0/1577328498 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b002f610 con 0x7f86c0069190 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c831d640 5 monclient: authenticate success, global_id 4798 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c831d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c831d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 1 -- 172.21.15.67:0/1577328498 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b002f9f0 con 0x7f86c0069190 2024-09-15T01:53:32.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c831d640 4 set_mon_vals no callback set 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.306+0000 7f86c50b7640 10 monclient: dump: 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.315 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 10 monclient: shutdown 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- 172.21.15.67:0/1577328498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c0069190 msgr2=0x7f86c0069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 --2- 172.21.15.67:0/1577328498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c0069190 0x7f86c0069560 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009990 tx=0x7f86b00091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- 172.21.15.67:0/1577328498 shutdown_connections 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 --2- 172.21.15.67:0/1577328498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c0069190 0x7f86c0069560 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 --2- 172.21.15.67:0/1577328498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0068dc0 0x7f86c0132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- 172.21.15.67:0/1577328498 >> 172.21.15.67:0/1577328498 conn(0x7f86c00711b0 msgr2=0x7f86c00a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- 172.21.15.67:0/1577328498 shutdown_connections 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- 172.21.15.67:0/1577328498 wait complete. 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 10 monclient: build_initial_monmap 2024-09-15T01:53:32.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 10 monclient: monmap: 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 Processor -- start 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.307+0000 7f86c831d640 1 -- start start 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient: init 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f86c0068dc0 0x7f86c012da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-a con 0x7f86c0068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 0x7f86c012dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-c con 0x7f86c0069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c012e960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c009b8a0 con 0x7f86c0068dc0 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c01420c0 con 0x7f86c012e960 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0099a90 con 0x7f86c0069190 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c831d640 10 monclient(hunting): authenticate will time out at 844.777441s 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c58b8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 msgr2=0x7f86c012dfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c58b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 0x7f86c012dfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43280/0 (socket says 172.21.15.67:43280) 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 -- 172.21.15.67:0/869872386 learned_addr learned my addr 172.21.15.67:0/869872386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): get_auth_request con 0x7f86c012e960 auth_method 0 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f86c0068dc0 0x7f86c012da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): get_auth_request con 0x7f86c0068dc0 auth_method 0 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c60b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient(hunting): handle_auth_done global_id 4804 payload 824 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 -- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 msgr2=0x7f86c012dfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 0x7f86c012dfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 -- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f86c0068dc0 msgr2=0x7f86c012da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f86c0068dc0 0x7f86c012da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient: _finish_hunting 0 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 monclient: found mon.noname-b 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.308+0000 7f86c68ba640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0009450 con 0x7f86c012e960 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c68ba640 1 -- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 msgr2=0x7f86c0098080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c68ba640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c840 tx=0x7f86b80018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8001950 con 0x7f86c012e960 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: dump: 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:32.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 5 monclient: authenticate success, global_id 4804 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b800a640 con 0x7f86c012e960 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 10 monclient: _renew_subs 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c009b5c0 con 0x7f86c012e960 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b800aa00 con 0x7f86c012e960 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86b67fc640 10 monclient: dump: 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.325 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 10 monclient: _renew_subs 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c48b6640 4 set_mon_vals no callback set 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.311+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0136810 con 0x7f86c012e960 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c831d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c831d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c831d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f86b80070f0 con 0x7f86c012e960 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c009bcd0 con 0x7f86c012e960 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86b67fc640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 0x7f86a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f86b804b640 con 0x7f86c012e960 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c60b9640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 0x7f86a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.312+0000 7f86c60b9640 10 monclient: get_auth_request con 0x7f86a40386f0 auth_method 0 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.314+0000 7f86c60b9640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 0x7f86a403aba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0038f0 tx=0x7f86bc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.316+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f86b801b400 con 0x7f86c012e960 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.316+0000 7f86b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.316+0000 7f86b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:32.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86c831d640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86c831d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86c831d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f86c009ba10 con 0x7f86c012e960 2024-09-15T01:53:32.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (secure 0 0 0) 0x7f86b8010300 con 0x7f86c012e960 2024-09-15T01:53:32.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.481+0000 7f86b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:32.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86c831d640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86c831d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86c831d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f86c004bf60 con 0x7f86c012e960 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86b67fc640 1 -- 172.21.15.67:0/869872386 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (secure 0 0 0) 0x7f86b8028440 con 0x7f86c012e960 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86b67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.580+0000 7f86b67fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stdout:[ 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.583 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_id": 0, 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_name": "replicated_rule", 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "type": 1, 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "steps": [ 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "take", 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "item": -1, 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "item_name": "default" 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.584 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "choose_firstn", 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 0, 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "type": "osd" 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "emit" 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_id": 1, 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "rule_name": "erasure-code", 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "type": 3, 2024-09-15T01:53:32.585 INFO:tasks.workunit.client.0.smithi067.stdout: "steps": [ 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "set_chooseleaf_tries", 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 5 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "set_choose_tries", 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 100 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.586 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "take", 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "item": -1, 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "item_name": "default" 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "choose_indep", 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "num": 0, 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "type": "osd" 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: "op": "emit" 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:53:32.587 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:53:32.588 INFO:tasks.workunit.client.0.smithi067.stdout:] 2024-09-15T01:53:32.588 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 msgr2=0x7f86a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 0x7f86a403aba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0038f0 tx=0x7f86bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 10 monclient: shutdown 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 msgr2=0x7f86c0098080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c840 tx=0x7f86b80018c0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 shutdown_connections 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f86a40386f0 0x7f86a403aba0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c012e960 0x7f86c0098080 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f86c0069190 0x7f86c012dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 --2- 172.21.15.67:0/869872386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f86c0068dc0 0x7f86c012da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 >> 172.21.15.67:0/869872386 conn(0x7f86c00711b0 msgr2=0x7f86c0134f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 shutdown_connections 2024-09-15T01:53:32.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.582+0000 7f86c831d640 1 -- 172.21.15.67:0/869872386 wait complete. 2024-09-15T01:53:32.593 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.666+0000 7f75aa83f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.666+0000 7f75aa83f640 10 monclient: build_initial_monmap 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.666+0000 7f75aa83f640 10 monclient: monmap: 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:32.667882+0000 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:32.667882+0000 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:32.669 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:32.670 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 Processor -- start 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 -- start start 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient: init 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f75a4068dc0 0x7f75a4132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75a4069190 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75a4137150 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a4096c10 con 0x7f75a4069190 2024-09-15T01:53:32.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a409ff20 con 0x7f75a4137150 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a406a2a0 con 0x7f75a4068dc0 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.667+0000 7f75aa83f640 10 monclient(hunting): authenticate will time out at 845.136979s 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a8ddc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75a4137150 0x7f75a4139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60402/0 (socket says 172.21.15.67:60402) 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a8ddc640 1 -- 172.21.15.67:0/249250412 learned_addr learned my addr 172.21.15.67:0/249250412 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 1 --2- 172.21.15.67:0/249250412 >> v2:172.21.15.17:3300/0 conn(0x7f75a4068dc0 0x7f75a4132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75a4068dc0 auth_method 0 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.668+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a8ddc640 1 -- 172.21.15.67:0/249250412 >> v1:172.21.15.67:6789/0 conn(0x7f75a4137150 legacy=0x7f75a4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a2ffd640 1 -- 172.21.15.67:0/249250412 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707685786 0 0) 0x7f75a409ff20 con 0x7f75a4137150 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a2ffd640 10 monclient(hunting): my global_id is 4807 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a2ffd640 1 -- 172.21.15.67:0/249250412 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f758c0034e0 con 0x7f75a4137150 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a37fe640 1 -- 172.21.15.67:0/249250412 >> v1:172.21.15.17:6789/0 conn(0x7f75a4069190 legacy=0x7f75a4069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 10 monclient(hunting): handle_auth_done global_id 44259 payload 274 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 1 -- 172.21.15.67:0/249250412 >> v1:172.21.15.67:6789/0 conn(0x7f75a4137150 legacy=0x7f75a4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 1 -- 172.21.15.67:0/249250412 >> v1:172.21.15.17:6789/0 conn(0x7f75a4069190 legacy=0x7f75a4069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 1 monclient: found mon.noname-a 2024-09-15T01:53:32.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:32.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:32.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.670+0000 7f75a3fff640 1 -- 172.21.15.67:0/249250412 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40a0180 con 0x7f75a4068dc0 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.673+0000 7f75a3fff640 1 -- 172.21.15.67:0/249250412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 msgr2=0x7f75a4132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.673+0000 7f75a3fff640 1 --2- 172.21.15.67:0/249250412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4132250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7594009950 tx=0x7f75940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8548ecf2d8ba6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 1 -- 172.21.15.67:0/249250412 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759402ae10 con 0x7f75a4068dc0 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: dump: 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 5 monclient: authenticate success, global_id 44259 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 1 -- 172.21.15.67:0/249250412 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759402f4b0 con 0x7f75a4068dc0 2024-09-15T01:53:32.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 4 set_mon_vals no callback set 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 1 -- 172.21.15.67:0/249250412 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759402f830 con 0x7f75a4068dc0 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75a2ffd640 10 monclient: dump: 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.682 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 10 monclient: shutdown 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 -- 172.21.15.67:0/249250412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 msgr2=0x7f75a4132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 --2- 172.21.15.67:0/249250412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4132250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7594009950 tx=0x7f75940090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 -- 172.21.15.67:0/249250412 shutdown_connections 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 --2- 172.21.15.67:0/249250412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4132250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 -- 172.21.15.67:0/249250412 >> 172.21.15.67:0/249250412 conn(0x7f75a40711b0 msgr2=0x7f75a40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 -- 172.21.15.67:0/249250412 shutdown_connections 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.674+0000 7f75aa83f640 1 -- 172.21.15.67:0/249250412 wait complete. 2024-09-15T01:53:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.675+0000 7f75aa83f640 10 monclient: build_initial_monmap 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.675+0000 7f75aa83f640 10 monclient: monmap: 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.675+0000 7f75aa83f640 1 Processor -- start 2024-09-15T01:53:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.675+0000 7f75aa83f640 1 -- start start 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.675+0000 7f75aa83f640 10 monclient: init 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4097b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75a40980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-c con 0x7f75a4069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): picked mon.noname-b con 0x7f75a4137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4099c80 con 0x7f75a4068dc0 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40a4c90 con 0x7f75a4137150 2024-09-15T01:53:32.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40a0890 con 0x7f75a4069190 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75aa83f640 10 monclient(hunting): authenticate will time out at 845.145151s 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a37fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 msgr2=0x7f75a40980d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75a40980d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43296/0 (socket says 172.21.15.67:43296) 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 learned_addr learned my addr 172.21.15.67:0/762988351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:32.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): get_auth_request con 0x7f75a4137150 auth_method 0 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4097b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75a4068dc0 auth_method 0 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient(hunting): handle_auth_done global_id 4810 payload 824 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 msgr2=0x7f75a40980d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75a40980d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 msgr2=0x7f75a4097b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4097b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 monclient: found mon.noname-b 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594009390 con 0x7f75a4137150 2024-09-15T01:53:32.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.676+0000 7f75a3fff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f75a4097b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:32.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.678+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 msgr2=0x7f75a4098610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.678+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f759c0017e0 tx=0x7f759c001c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0020c0 con 0x7f75a4137150 2024-09-15T01:53:32.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: dump: 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c00aa00 con 0x7f75a4137150 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 5 monclient: authenticate success, global_id 4810 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 10 monclient: _renew_subs 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a409b290 con 0x7f75a4137150 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00ad80 con 0x7f75a4137150 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f758bfff640 10 monclient: dump: 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 10 monclient: _renew_subs 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a409bef0 con 0x7f75a4137150 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.679+0000 7f75a27fc640 4 set_mon_vals no callback set 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f759c007300 con 0x7f75a4137150 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75aa83f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75aa83f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75aa83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a4099790 con 0x7f75a4137150 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f758bfff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 0x7f759003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f759c04baa0 con 0x7f75a4137150 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75a3fff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 0x7f759003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.680+0000 7f75a3fff640 10 monclient: get_auth_request con 0x7f7590038740 auth_method 0 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.682+0000 7f75a3fff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 0x7f759003abf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7594034000 tx=0x7f7594032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.684+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f759c01bae0 con 0x7f75a4137150 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.684+0000 7f758bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:32.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.684+0000 7f758bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:32.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.848+0000 7f75aa83f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.848+0000 7f75aa83f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75aa83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f75a404bf60 con 0x7f75a4137150 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 msgr2=0x7f75a4098610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 msgr2=0x7f75a4098610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f759c0017e0 tx=0x7f759c001c50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f759c0017e0 tx=0x7f759c001c50 comp rx=0 tx=0).stop 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 msgr2=0x7f75a4098610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75a4137150 0x7f75a4098610 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:32.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 0x7f75900408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): picked mon.b con 0x7f759003e4c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75900431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): picked mon.c con 0x7f75a4069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f7590045c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): picked mon.a con 0x7f75a4068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900465d0 con 0x7f75a4068dc0 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590046710 con 0x7f759003e4c0 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75900468a0 con 0x7f75a4069190 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f758bfff640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:32.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 0x7f75900408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 msgr2=0x7f75900431c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): get_auth_request con 0x7f759003e4c0 auth_method 0 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a8ddc640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75900431c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a3fff640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f7590045c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75a4068dc0 auth_method 0 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:32.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient(hunting): handle_auth_done global_id 4810 payload 293 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 msgr2=0x7f75900431c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75900431c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 msgr2=0x7f7590045c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f7590045c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 monclient: found mon.b 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f7590046a30 con 0x7f759003e4c0 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f75a4096c10 con 0x7f759003e4c0 2024-09-15T01:53:32.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:32.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.849+0000 7f75a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:32.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.851+0000 7f75a37fe640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 msgr2=0x7f75900408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f75a37fe640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 0x7f75900408a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7598001a80 tx=0x7f7598001da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759800ca70 con 0x7f759003e4c0 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 10 monclient: dump: 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759800cc10 con 0x7f759003e4c0 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f75a27fc640 4 set_mon_vals no callback set 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.852+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f759800cd80 con 0x7f759003e4c0 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.853+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (secure 0 0 0) 0x7f75980480f0 con 0x7f759003e4c0 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.853+0000 7f758bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.853+0000 7f758bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:32.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f75aa83f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f75aa83f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f75aa83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f75a409b6a0 con 0x7f759003e4c0 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f758bfff640 1 -- 172.21.15.67:0/762988351 <== mon.1 v2:172.21.15.67:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (secure 0 0 0) 0x7f7598018020 con 0x7f759003e4c0 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f758bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.952+0000 7f758bfff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stdout:replicated_rule 2024-09-15T01:53:32.955 INFO:tasks.workunit.client.0.smithi067.stdout:erasure-code 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 msgr2=0x7f759003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 0x7f759003abf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7594034000 tx=0x7f7594032000 comp rx=0 tx=0).stop 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 10 monclient: shutdown 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 msgr2=0x7f75900408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:32.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 0x7f75900408a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7598001a80 tx=0x7f7598001da0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 shutdown_connections 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f75a4068dc0 0x7f7590045c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7590038740 0x7f759003abf0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f75a4069190 0x7f75900431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 --2- 172.21.15.67:0/762988351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759003e4c0 0x7f75900408a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 >> 172.21.15.67:0/762988351 conn(0x7f75a40711b0 msgr2=0x7f75a40a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 shutdown_connections 2024-09-15T01:53:32.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:32.954+0000 7f75aa83f640 1 -- 172.21.15.67:0/762988351 wait complete. 2024-09-15T01:53:32.966 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule list 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.039+0000 7f69bd6b8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.039+0000 7f69bd6b8640 10 monclient: build_initial_monmap 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.039+0000 7f69bd6b8640 10 monclient: monmap: 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:33.041050+0000 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:33.041050+0000 2024-09-15T01:53:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:33.043 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 1 Processor -- start 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 1 -- start start 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 10 monclient: init 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.040+0000 7f69bd6b8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f69b8068dc0 0x7f69b8132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-a con 0x7f69b8068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69b8069190 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 0x7f69b8139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f69b8137150 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b8096c10 con 0x7f69b8069190 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b806a2a0 con 0x7f69b8068dc0 2024-09-15T01:53:33.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80a14d0 con 0x7f69b8137150 2024-09-15T01:53:33.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69bd6b8640 10 monclient(hunting): authenticate will time out at 845.510065s 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 0x7f69b8139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 0x7f69b8139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43310/0 (socket says 172.21.15.67:43310) 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 1 -- 172.21.15.67:0/3821830356 learned_addr learned my addr 172.21.15.67:0/3821830356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 10 monclient(hunting): get_auth_request con 0x7f69b8137150 auth_method 0 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b67fc640 1 --1- 172.21.15.67:0/3821830356 >> v1:172.21.15.67:6789/0 conn(0x7f69b8069190 0x7f69b8069590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60418/0 (socket says 172.21.15.67:60418) 2024-09-15T01:53:33.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 --2- 172.21.15.67:0/3821830356 >> v2:172.21.15.17:3300/0 conn(0x7f69b8068dc0 0x7f69b8132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): get_auth_request con 0x7f69b8068dc0 auth_method 0 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient(hunting): handle_auth_done global_id 44268 payload 274 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 -- 172.21.15.67:0/3821830356 >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 msgr2=0x7f69b8139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 --2- 172.21.15.67:0/3821830356 >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 0x7f69b8139530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 -- 172.21.15.67:0/3821830356 >> v1:172.21.15.67:6789/0 conn(0x7f69b8069190 legacy=0x7f69b8069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 monclient: found mon.noname-a 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:33.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.041+0000 7f69b6ffd640 1 -- 172.21.15.67:0/3821830356 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b809ff20 con 0x7f69b8068dc0 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.044+0000 7f69b6ffd640 1 -- 172.21.15.67:0/3821830356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8068dc0 msgr2=0x7f69b8132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.044+0000 7f69b6ffd640 1 --2- 172.21.15.67:0/3821830356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8068dc0 0x7f69b8132250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69b8096f10 tx=0x7f69a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b441e3d852a66c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 1 -- 172.21.15.67:0/3821830356 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a802ae50 con 0x7f69b8068dc0 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: dump: 2024-09-15T01:53:33.057 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 1 -- 172.21.15.67:0/3821830356 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8032440 con 0x7f69b8068dc0 2024-09-15T01:53:33.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 5 monclient: authenticate success, global_id 44268 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 1 -- 172.21.15.67:0/3821830356 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a80327e0 con 0x7f69b8068dc0 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 4 set_mon_vals no callback set 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69b5ffb640 10 monclient: dump: 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.059 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 10 monclient: shutdown 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 -- 172.21.15.67:0/3821830356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8068dc0 msgr2=0x7f69b8132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/3821830356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8068dc0 0x7f69b8132250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69b8096f10 tx=0x7f69a80090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 -- 172.21.15.67:0/3821830356 shutdown_connections 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/3821830356 >> v2:172.21.15.67:3300/0 conn(0x7f69b8137150 0x7f69b8139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/3821830356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8068dc0 0x7f69b8132250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 -- 172.21.15.67:0/3821830356 >> 172.21.15.67:0/3821830356 conn(0x7f69b80711b0 msgr2=0x7f69b80a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 -- 172.21.15.67:0/3821830356 shutdown_connections 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.045+0000 7f69bd6b8640 1 -- 172.21.15.67:0/3821830356 wait complete. 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.046+0000 7f69bd6b8640 10 monclient: build_initial_monmap 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.046+0000 7f69bd6b8640 10 monclient: monmap: 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:33.061 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.046+0000 7f69bd6b8640 1 Processor -- start 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.046+0000 7f69bd6b8640 1 -- start start 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.046+0000 7f69bd6b8640 10 monclient: init 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 0x7f69b80980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f69b8068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8069190 0x7f69b8098600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-a con 0x7f69b8069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): picked mon.noname-b con 0x7f69b8137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b811bcc0 con 0x7f69b8069190 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80a08c0 con 0x7f69b8137150 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b809b4b0 con 0x7f69b8068dc0 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69bd6b8640 10 monclient(hunting): authenticate will time out at 845.516404s 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b6ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 msgr2=0x7f69b80980c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 0x7f69b80980c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43320/0 (socket says 172.21.15.67:43320) 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 -- 172.21.15.67:0/55373950 learned_addr learned my addr 172.21.15.67:0/55373950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): get_auth_request con 0x7f69b8137150 auth_method 0 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8069190 0x7f69b8098600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): get_auth_request con 0x7f69b8069190 auth_method 0 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient(hunting): handle_auth_done global_id 4816 payload 824 2024-09-15T01:53:33.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 -- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 msgr2=0x7f69b80980c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 0x7f69b80980c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 -- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8069190 msgr2=0x7f69b8098600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8069190 0x7f69b8098600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.047+0000 7f69b77fe640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009390 con 0x7f69b8137150 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69b77fe640 1 -- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 msgr2=0x7f69b8098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69b77fe640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f69b0007c50 tx=0x7f69b00092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0017070 con 0x7f69b8137150 2024-09-15T01:53:33.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: dump: 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: _finish_auth 0 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000f040 con 0x7f69b8137150 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 5 monclient: authenticate success, global_id 4816 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00133f0 con 0x7f69b8137150 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f699effd640 10 monclient: dump: 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.069 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 10 monclient: _renew_subs 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b809b790 con 0x7f69b8137150 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69b57fa640 4 set_mon_vals no callback set 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 10 monclient: _renew_subs 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.050+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b809a550 con 0x7f69b8137150 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f69b0003020 con 0x7f69b8137150 2024-09-15T01:53:33.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69bd6b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69bd6b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69bd6b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b812fec0 con 0x7f69b8137150 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f699effd640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 0x7f69a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f69b004b900 con 0x7f69b8137150 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69b6ffd640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 0x7f69a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.051+0000 7f69b6ffd640 10 monclient: get_auth_request con 0x7f69a40386f0 auth_method 0 2024-09-15T01:53:33.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.053+0000 7f69b6ffd640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 0x7f69a403aba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f69a8031000 tx=0x7f69a8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.055+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f69b00137a0 con 0x7f69b8137150 2024-09-15T01:53:33.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.055+0000 7f699effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.055+0000 7f699effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f69bd6b8640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f69bd6b8640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f69bd6b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f69b804bf60 con 0x7f69b8137150 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (secure 0 0 0) 0x7f69b0011a10 con 0x7f69b8137150 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f699effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.221+0000 7f699effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:33.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f69bd6b8640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f69bd6b8640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f69bd6b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f69b809b980 con 0x7f69b8137150 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f699effd640 1 -- 172.21.15.67:0/55373950 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (secure 0 0 0) 0x7f69b00137a0 con 0x7f69b8137150 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f699effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.322+0000 7f699effd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stdout:replicated_rule 2024-09-15T01:53:33.325 INFO:tasks.workunit.client.0.smithi067.stdout:erasure-code 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 msgr2=0x7f69a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 0x7f69a403aba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f69a8031000 tx=0x7f69a8032000 comp rx=0 tx=0).stop 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 10 monclient: shutdown 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 msgr2=0x7f69b8098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f69b0007c50 tx=0x7f69b00092d0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 shutdown_connections 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f69a40386f0 0x7f69a403aba0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f69b8069190 0x7f69b8098600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b8137150 0x7f69b8098b40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 --2- 172.21.15.67:0/55373950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f69b8068dc0 0x7f69b80980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 >> 172.21.15.67:0/55373950 conn(0x7f69b80711b0 msgr2=0x7f69b80a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:33.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 shutdown_connections 2024-09-15T01:53:33.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.324+0000 7f69bd6b8640 1 -- 172.21.15.67:0/55373950 wait complete. 2024-09-15T01:53:33.335 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.411+0000 7f3f7315d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.411+0000 7f3f7315d640 10 monclient: build_initial_monmap 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.411+0000 7f3f7315d640 10 monclient: monmap: 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:33.412978+0000 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:33.412978+0000 2024-09-15T01:53:33.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:33.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.412+0000 7f3f7315d640 1 Processor -- start 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.412+0000 7f3f7315d640 1 -- start start 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.412+0000 7f3f7315d640 10 monclient: init 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f6c068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f6c069190 0x7f3f6c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f6c137150 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c096c10 con 0x7f3f6c068dc0 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c132960 con 0x7f3f6c137150 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0a14d0 con 0x7f3f6c069190 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f7315d640 10 monclient(hunting): authenticate will time out at 845.882239s 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f716fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3f6c137150 0x7f3f6c139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60424/0 (socket says 172.21.15.67:60424) 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f716fa640 1 -- 172.21.15.67:0/2093055466 learned_addr learned my addr 172.21.15.67:0/2093055466 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/2093055466 >> v2:172.21.15.17:3300/0 conn(0x7f3f6c069190 0x7f3f6c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request con 0x7f3f6c069190 auth_method 0 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.413+0000 7f3f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f716fa640 1 -- 172.21.15.67:0/2093055466 >> v1:172.21.15.67:6789/0 conn(0x7f3f6c137150 legacy=0x7f3f6c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173431009 0 0) 0x7f3f6c132960 con 0x7f3f6c137150 2024-09-15T01:53:33.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient(hunting): my global_id is 4819 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f540034e0 con 0x7f3f6c137150 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f70ef9640 1 -- 172.21.15.67:0/2093055466 >> v1:172.21.15.17:6789/0 conn(0x7f3f6c068dc0 legacy=0x7f3f6c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212439765 0 0) 0x7f3f540034e0 con 0x7f3f6c137150 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 >> v2:172.21.15.17:3300/0 conn(0x7f3f6c069190 msgr2=0x7f3f6c069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 --2- 172.21.15.67:0/2093055466 >> v2:172.21.15.17:3300/0 conn(0x7f3f6c069190 0x7f3f6c069590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 >> v1:172.21.15.17:6789/0 conn(0x7f3f6c068dc0 legacy=0x7f3f6c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:33.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c09ff20 con 0x7f3f6c137150 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f7315d640 5 monclient: authenticate success, global_id 4819 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f3f64001aa0 con 0x7f3f6c137150 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.415+0000 7f3f7315d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient: dump: 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 >> v1:172.21.15.67:6789/0 conn(0x7f3f6c137150 legacy=0x7f3f6c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f54003b20 0x7f3f54003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): picked mon.a con 0x7f3f54003b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f54004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): picked mon.b con 0x7f3f54004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f54005330 0x7f3f5400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): picked mon.c con 0x7f3f54005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5400c2a0 con 0x7f3f54003b20 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5400c410 con 0x7f3f54004780 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5400c5d0 con 0x7f3f54005330 2024-09-15T01:53:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f7315d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f54004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f716fa640 1 -- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f54005330 msgr2=0x7f3f5400b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request con 0x7f3f54004780 auth_method 0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f54003b20 0x7f3f54003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f716fa640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f54005330 0x7f3f5400b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): get_auth_request con 0x7f3f54003b20 auth_method 0 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_done global_id 4819 payload 293 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 -- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f54005330 msgr2=0x7f3f5400b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f54005330 0x7f3f5400b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 -- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f54003b20 msgr2=0x7f3f54003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f54003b20 0x7f3f54003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 monclient: found mon.b 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f716fa640 1 -- 172.21.15.67:0/2093055466 reap_dead start 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 1 -- 172.21.15.67:0/2093055466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540034e0 con 0x7f3f54004780 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.416+0000 7f3f6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6bfff640 1 -- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 msgr2=0x7f3f54004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f54004bf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009b70 tx=0x7f3f600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80917163974ae81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6002f510 con 0x7f3f54004780 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: dump: 2024-09-15T01:53:33.432 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6002d050 con 0x7f3f54004780 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f7315d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:33.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f7315d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 1 -- 172.21.15.67:0/2093055466 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6002f8a0 con 0x7f3f54004780 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f7315d640 4 set_mon_vals no callback set 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.419+0000 7f3f6b7fe640 10 monclient: dump: 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 10 monclient: shutdown 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 -- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 msgr2=0x7f3f54004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f54004bf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009b70 tx=0x7f3f600090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 -- 172.21.15.67:0/2093055466 shutdown_connections 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 --2- 172.21.15.67:0/2093055466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f54004bf0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 -- 172.21.15.67:0/2093055466 >> 172.21.15.67:0/2093055466 conn(0x7f3f6c0711b0 msgr2=0x7f3f6c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 -- 172.21.15.67:0/2093055466 shutdown_connections 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 -- 172.21.15.67:0/2093055466 wait complete. 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 10 monclient: build_initial_monmap 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 10 monclient: monmap: 2024-09-15T01:53:33.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.420+0000 7f3f7315d640 1 Processor -- start 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 -- start start 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient: init 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f54004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f6c097f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c09be20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f6c098960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c0984c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c098ea0 con 0x7f3f6c0984c0 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c099730 con 0x7f3f54004780 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0a0890 con 0x7f3f6c09be20 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f7315d640 10 monclient(hunting): authenticate will time out at 845.890665s 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43330/0 (socket says 172.21.15.67:43330) 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 learned_addr learned my addr 172.21.15.67:0/3060004387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): get_auth_request con 0x7f3f54004780 auth_method 0 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f6bfff640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 msgr2=0x7f3f6c097f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f6c097f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient(hunting): handle_auth_done global_id 4822 payload 824 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f716fa640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f6c098960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 msgr2=0x7f3f6c097f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f6c097f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 msgr2=0x7f3f6c098960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f6c098960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 monclient: found mon.noname-b 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.421+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c131110 con 0x7f3f54004780 2024-09-15T01:53:33.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.422+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 msgr2=0x7f3f6c097a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-15T01:53:33.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.422+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00f8b0 tx=0x7f3f5c009040 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-15T01:53:33.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.422+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:33.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 10 monclient: get_auth_request con 0x7f3f54004780 auth_method 0 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 0 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 msgr2=0x7f3f6c097a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f54004780 0x7f3f6c097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f580054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c09be20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:33.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 0x7f3f58008340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f3f58005ec0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f5800af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c0984c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5800b880 con 0x7f3f6c0984c0 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5800b9c0 con 0x7f3f58005ec0 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:33.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5800bb80 con 0x7f3f6c09be20 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f68ff9640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 msgr2=0x7f3f580054b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 0x7f3f58008340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f580054b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request con 0x7f3f58005ec0 auth_method 0 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f716fa640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f5800af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_done global_id 4822 payload 293 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 msgr2=0x7f3f580054b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f580054b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 msgr2=0x7f3f5800af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f5800af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 monclient: found mon.noname-b 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.623+0000 7f3f6bfff640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5800be00 con 0x7f3f58005ec0 2024-09-15T01:53:33.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f6bfff640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 msgr2=0x7f3f58008340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:33.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f6bfff640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 0x7f3f58008340 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3f60034000 tx=0x7f3f60035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6002fcd0 con 0x7f3f58005ec0 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: dump: 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.633 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60040070 con 0x7f3f58005ec0 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 5 monclient: authenticate success, global_id 4822 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6002f510 con 0x7f3f58005ec0 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 10 monclient: _renew_subs 2024-09-15T01:53:33.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c09b160 con 0x7f3f58005ec0 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f68ff9640 10 monclient: dump: 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:33.635 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f6affd640 4 set_mon_vals no callback set 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 10 monclient: _renew_subs 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.625+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c098960 con 0x7f3f58005ec0 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f60042030 con 0x7f3f58005ec0 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f68ff9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 0x7f3f58042b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f7315d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f7315d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f7315d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f3f6006b820 con 0x7f3f58005ec0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 0x7f3f58042b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.626+0000 7f3f70ef9640 10 monclient: get_auth_request con 0x7f3f54004780 auth_method 0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.629+0000 7f3f70ef9640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 0x7f3f58042b90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00f4a0 tx=0x7f3f5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.631+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.631+0000 7f3f68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:33.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.631+0000 7f3f68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:33.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.795+0000 7f3f7315d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:33.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.795+0000 7f3f7315d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:33.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.795+0000 7f3f7315d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:33.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:33.795+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:34.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.530+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v41) v1 ==== 126+0+0 (secure 0 0 0) 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:34.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.530+0000 7f3f68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.530+0000 7f3f68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.634+0000 7f3f7315d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.634+0000 7f3f7315d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.634+0000 7f3f7315d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.634+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:34.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.635+0000 7f3f68ff9640 1 -- 172.21.15.67:0/3060004387 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7f3f6c09b8d0 con 0x7f3f58005ec0 2024-09-15T01:53:34.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.635+0000 7f3f68ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.635+0000 7f3f68ff9640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-15T01:53:34.637 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo already exists 2024-09-15T01:53:34.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 msgr2=0x7f3f58042b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 0x7f3f58042b90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00f4a0 tx=0x7f3f5c009040 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 10 monclient: shutdown 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 msgr2=0x7f3f58008340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 0x7f3f58008340 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3f60034000 tx=0x7f3f60035000 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 shutdown_connections 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f54004780 0x7f3f58042b90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f6c0984c0 0x7f3f5800af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f58005ec0 0x7f3f58008340 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 --2- 172.21.15.67:0/3060004387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f6c09be20 0x7f3f580054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 >> 172.21.15.67:0/3060004387 conn(0x7f3f6c0711b0 msgr2=0x7f3f6c09ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:34.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 shutdown_connections 2024-09-15T01:53:34.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.637+0000 7f3f7315d640 1 -- 172.21.15.67:0/3060004387 wait complete. 2024-09-15T01:53:34.649 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.726+0000 7fb0b362b640 10 monclient: get_monmap_and_config 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.726+0000 7fb0b362b640 10 monclient: build_initial_monmap 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.726+0000 7fb0b362b640 10 monclient: monmap: 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:34.727938+0000 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:34.727938+0000 2024-09-15T01:53:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 1 Processor -- start 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 1 -- start start 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 10 monclient: init 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:53:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb0ac0a0b50 0x7fb0ac069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac0a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.727+0000 7fb0b362b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0ac131b10 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac06a270 con 0x7fb0ac131b10 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac06f290 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac09f7e0 con 0x7fb0ac09eea0 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b362b640 10 monclient(hunting): authenticate will time out at 847.197045s 2024-09-15T01:53:34.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b0bc6640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 msgr2=0x7fb0ac09f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b0bc6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb0ac0a0b50 0x7fb0ac069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb0ac0a0b50 0x7fb0ac069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:45848/0 (socket says 172.21.15.67:45848) 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 1 -- 172.21.15.67:0/3367449052 learned_addr learned my addr 172.21.15.67:0/3367449052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): get_auth_request con 0x7fb0ac0a0b50 auth_method 0 2024-09-15T01:53:34.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:34.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:34.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:34.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b1bc8640 1 --1- 172.21.15.67:0/3367449052 >> v1:172.21.15.17:6789/0 conn(0x7fb0ac131b10 0x7fb0ac131f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:48074/0 (socket says 172.21.15.67:48074) 2024-09-15T01:53:34.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:34.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:34.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.728+0000 7fb0b13c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b1bc8640 1 -- 172.21.15.67:0/3367449052 >> v1:172.21.15.17:6789/0 conn(0x7fb0ac131b10 legacy=0x7fb0ac131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 10 monclient(hunting): handle_auth_done global_id 44280 payload 274 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 1 -- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 msgr2=0x7fb0ac09f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 1 --2- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 1 -- 172.21.15.67:0/3367449052 >> v1:172.21.15.17:6789/0 conn(0x7fb0ac131b10 legacy=0x7fb0ac131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 10 monclient: _finish_hunting 0 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 1 monclient: found mon.noname-a 2024-09-15T01:53:34.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.730+0000 7fb0b13c7640 1 -- 172.21.15.67:0/3367449052 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0a0f20 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.733+0000 7fb0b13c7640 1 -- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac0a0b50 msgr2=0x7fb0ac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.733+0000 7fb0b13c7640 1 --2- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac069340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009fa0 tx=0x7fb0a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df495cdf15beaca2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 1 -- 172.21.15.67:0/3367449052 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8007610 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: dump: 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.741 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 1 -- 172.21.15.67:0/3367449052 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a802d050 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 1 -- 172.21.15.67:0/3367449052 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80079a0 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 5 monclient: authenticate success, global_id 44280 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 4 set_mon_vals no callback set 2024-09-15T01:53:34.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0a3fff640 10 monclient: dump: 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 10 monclient: shutdown 2024-09-15T01:53:34.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 -- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac0a0b50 msgr2=0x7fb0ac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 --2- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac069340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009fa0 tx=0x7fb0a8009840 comp rx=0 tx=0).stop 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 -- 172.21.15.67:0/3367449052 shutdown_connections 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 --2- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 --2- 172.21.15.67:0/3367449052 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac069340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 -- 172.21.15.67:0/3367449052 >> 172.21.15.67:0/3367449052 conn(0x7fb0ac071290 msgr2=0x7fb0ac0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 -- 172.21.15.67:0/3367449052 shutdown_connections 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.734+0000 7fb0b362b640 1 -- 172.21.15.67:0/3367449052 wait complete. 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.735+0000 7fb0b362b640 10 monclient: build_initial_monmap 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.735+0000 7fb0b362b640 10 monclient: monmap: 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:34.744 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.735+0000 7fb0b362b640 1 Processor -- start 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.735+0000 7fb0b362b640 1 -- start start 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.735+0000 7fb0b362b640 10 monclient: init 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ac09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:34.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac0a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 0x7fb0ac11b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac131b10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac11e030 con 0x7fb0ac131b10 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac11dd50 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac11e310 con 0x7fb0ac09eea0 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b362b640 10 monclient(hunting): authenticate will time out at 847.205251s 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b13c7640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 msgr2=0x7fb0ac09a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:34.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b13c7640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43350/0 (socket says 172.21.15.67:43350) 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 -- 172.21.15.67:0/2863335931 learned_addr learned my addr 172.21.15.67:0/2863335931 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): get_auth_request con 0x7fb0ac0a0b50 auth_method 0 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 0x7fb0ac11b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 10 monclient(hunting): get_auth_request con 0x7fb0ac131b10 auth_method 0 2024-09-15T01:53:34.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient(hunting): handle_auth_done global_id 4825 payload 824 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 -- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 msgr2=0x7fb0ac09a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 -- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 msgr2=0x7fb0ac11b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 0x7fb0ac11b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient: _finish_hunting 0 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 monclient: found mon.noname-b 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b0bc6640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a8009ae0 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.736+0000 7fb0b1bc8640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 0x7fb0ac11b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b0bc6640 1 -- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 msgr2=0x7fb0ac130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b0bc6640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00b600 tx=0x7fb09c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c015070 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:34.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: dump: 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:34.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c003d90 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 5 monclient: authenticate success, global_id 4825 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 10 monclient: _renew_subs 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac141f70 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c006380 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a17fa640 10 monclient: dump: 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:34.752 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0a37fe640 4 set_mon_vals no callback set 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 10 monclient: _renew_subs 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.739+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac141650 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b362b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b362b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb09c010440 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac04bf60 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0a17fa640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 0x7fb09003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7fb09c04b4e0 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b13c7640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 0x7fb09003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.740+0000 7fb0b13c7640 10 monclient: get_auth_request con 0x7fb0900386f0 auth_method 0 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.743+0000 7fb0b13c7640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 0x7fb09003aba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac09c460 tx=0x7fb0a8036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.744+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb09c018d60 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.744+0000 7fb0a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.744+0000 7fb0a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:34.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.910+0000 7fb0b362b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.910+0000 7fb0b362b640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.910+0000 7fb0b362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:34.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.910+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fb0ac06f000 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.912+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb09c018780 con 0x7fb0ac0a0b50 2024-09-15T01:53:34.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.912+0000 7fb0a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:34.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:34.912+0000 7fb0a17fa640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-15T01:53:35.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.012+0000 7fb0b362b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:35.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.012+0000 7fb0b362b640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:35.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.012+0000 7fb0b362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.012+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fb0ac11cdf0 con 0x7fb0ac0a0b50 2024-09-15T01:53:35.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.013+0000 7fb0a17fa640 1 -- 172.21.15.67:0/2863335931 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb09c018780 con 0x7fb0ac0a0b50 2024-09-15T01:53:35.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.013+0000 7fb0a17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T01:53:35.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.013+0000 7fb0a17fa640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-15T01:53:35.016 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo already exists 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 msgr2=0x7fb09003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 0x7fb09003aba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac09c460 tx=0x7fb0a8036000 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 10 monclient: shutdown 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 msgr2=0x7fb0ac130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00b600 tx=0x7fb09c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 shutdown_connections 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0900386f0 0x7fb09003aba0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac131b10 0x7fb0ac11b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac0a0b50 0x7fb0ac130c30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 --2- 172.21.15.67:0/2863335931 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac09eea0 0x7fb0ac09a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 >> 172.21.15.67:0/2863335931 conn(0x7fb0ac071290 msgr2=0x7fb0ac134e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:35.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 shutdown_connections 2024-09-15T01:53:35.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.015+0000 7fb0b362b640 1 -- 172.21.15.67:0/2863335931 wait complete. 2024-09-15T01:53:35.026 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.102+0000 7f92cb0e8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.102+0000 7f92cb0e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.102+0000 7f92cb0e8640 10 monclient: monmap: 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:35.103698+0000 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:35.103698+0000 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:35.105 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:35.106 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:35.106 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:35.106 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:35.106 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 Processor -- start 2024-09-15T01:53:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 -- start start 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient: init 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92c40a0b50 0x7f92c4069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-c con 0x7f92c40a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92c409eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92c409f2c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40a61a0 con 0x7f92c409eea0 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40a04c0 con 0x7f92c409f2c0 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c40a1610 con 0x7f92c40a0b50 2024-09-15T01:53:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.103+0000 7f92cb0e8640 10 monclient(hunting): authenticate will time out at 847.572877s 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92c40a0b50 0x7f92c4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92c40a0b50 0x7f92c4069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43354/0 (socket says 172.21.15.67:43354) 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 1 -- 172.21.15.67:0/3586341390 learned_addr learned my addr 172.21.15.67:0/3586341390 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): get_auth_request con 0x7f92c40a0b50 auth_method 0 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.104+0000 7f92c8e84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c9685640 1 -- 172.21.15.67:0/3586341390 >> v1:172.21.15.67:6789/0 conn(0x7f92c409f2c0 legacy=0x7f92c4132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c3fff640 1 -- 172.21.15.67:0/3586341390 >> v1:172.21.15.17:6789/0 conn(0x7f92c409eea0 legacy=0x7f92c4131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 10 monclient(hunting): handle_auth_done global_id 4828 payload 274 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 1 -- 172.21.15.67:0/3586341390 >> v1:172.21.15.67:6789/0 conn(0x7f92c409f2c0 legacy=0x7f92c4132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 1 -- 172.21.15.67:0/3586341390 >> v1:172.21.15.17:6789/0 conn(0x7f92c409eea0 legacy=0x7f92c4131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 10 monclient: _finish_hunting 0 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 1 monclient: found mon.noname-c 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.106+0000 7f92c8e84640 1 -- 172.21.15.67:0/3586341390 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40a0f20 con 0x7f92c40a0b50 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c8e84640 1 -- 172.21.15.67:0/3586341390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 msgr2=0x7f92c4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c8e84640 1 --2- 172.21.15.67:0/3586341390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4069380 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009a10 tx=0x7f92b80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c43192de149261f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 1 -- 172.21.15.67:0/3586341390 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b802f4b0 con 0x7f92c40a0b50 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: dump: 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 1 -- 172.21.15.67:0/3586341390 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b802f610 con 0x7f92c40a0b50 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 5 monclient: authenticate success, global_id 4828 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:35.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 1 -- 172.21.15.67:0/3586341390 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b802f9f0 con 0x7f92c40a0b50 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 4 set_mon_vals no callback set 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92c37fe640 10 monclient: dump: 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.120 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 10 monclient: shutdown 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3586341390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 msgr2=0x7f92c4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3586341390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4069380 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009a10 tx=0x7f92b80091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3586341390 shutdown_connections 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3586341390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4069380 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3586341390 >> 172.21.15.67:0/3586341390 conn(0x7f92c4071330 msgr2=0x7f92c40a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3586341390 shutdown_connections 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3586341390 wait complete. 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 10 monclient: monmap: 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.110+0000 7f92cb0e8640 1 Processor -- start 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 -- start start 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient: init 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92c409eea0 0x7f92c409a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-a con 0x7f92c409eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 0x7f92c4130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-c con 0x7f92c409f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): picked mon.noname-b con 0x7f92c40a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c411e130 con 0x7f92c409eea0 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c411de50 con 0x7f92c40a0b50 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c411e410 con 0x7f92c409f2c0 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92cb0e8640 10 monclient(hunting): authenticate will time out at 847.580473s 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43368/0 (socket says 172.21.15.67:43368) 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 -- 172.21.15.67:0/3654339630 learned_addr learned my addr 172.21.15.67:0/3654339630 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): get_auth_request con 0x7f92c40a0b50 auth_method 0 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c3fff640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 msgr2=0x7f92c4130d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:35.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c3fff640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 0x7f92c4130d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient(hunting): handle_auth_done global_id 4831 payload 824 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c8e84640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92c409eea0 0x7f92c409a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 msgr2=0x7f92c4130d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 0x7f92c4130d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92c409eea0 msgr2=0x7f92c409a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92c409eea0 0x7f92c409a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient: _finish_hunting 0 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 monclient: found mon.noname-b 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.111+0000 7f92c9685640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8009450 con 0x7f92c40a0b50 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c9685640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 msgr2=0x7f92c4141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c9685640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c840 tx=0x7f92bc0018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc001950 con 0x7f92c40a0b50 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: dump: 2024-09-15T01:53:35.127 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00a640 con 0x7f92c40a0b50 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 5 monclient: authenticate success, global_id 4831 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 10 monclient: _renew_subs 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c411ca80 con 0x7f92c40a0b50 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00aa00 con 0x7f92c40a0b50 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:35.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c0ff9640 10 monclient: dump: 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 10 monclient: _renew_subs 2024-09-15T01:53:35.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92c2ffd640 4 set_mon_vals no callback set 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.114+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c411d1c0 con 0x7f92c40a0b50 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f92bc0070f0 con 0x7f92c40a0b50 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92cb0e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92cb0e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92cb0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c409b590 con 0x7f92c40a0b50 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92c0ff9640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 0x7f92a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f92bc04b800 con 0x7f92c40a0b50 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92c8e84640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 0x7f92a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.115+0000 7f92c8e84640 10 monclient: get_auth_request con 0x7f92a80386f0 auth_method 0 2024-09-15T01:53:35.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.117+0000 7f92c8e84640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 0x7f92a803aba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.119+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f92bc01b400 con 0x7f92c40a0b50 2024-09-15T01:53:35.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.119+0000 7f92c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.119+0000 7f92c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:35.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.284+0000 7f92cb0e8640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.284+0000 7f92cb0e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.284+0000 7f92cb0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.284+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f92c404bf60 con 0x7f92c40a0b50 2024-09-15T01:53:35.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.538+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v42) v1 ==== 126+0+0 (secure 0 0 0) 0x7f92bc010300 con 0x7f92c40a0b50 2024-09-15T01:53:35.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.538+0000 7f92c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.538+0000 7f92c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:35.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.638+0000 7f92cb0e8640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.638+0000 7f92cb0e8640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.638+0000 7f92cb0e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.638+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f92c406f000 con 0x7f92c40a0b50 2024-09-15T01:53:35.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.639+0000 7f92c0ff9640 1 -- 172.21.15.67:0/3654339630 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v42) v1 ==== 149+0+0 (secure 0 0 0) 0x7f92bc010950 con 0x7f92c40a0b50 2024-09-15T01:53:35.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.639+0000 7f92c0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T01:53:35.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.639+0000 7f92c0ff9640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-15T01:53:35.642 INFO:tasks.workunit.client.0.smithi067.stderr:rule bar already exists 2024-09-15T01:53:35.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 msgr2=0x7f92a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 0x7f92a803aba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b8032000 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 10 monclient: shutdown 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 msgr2=0x7f92c4141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c840 tx=0x7f92bc0018c0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 shutdown_connections 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92a80386f0 0x7f92a803aba0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c40a0b50 0x7f92c4141130 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92c409f2c0 0x7f92c4130d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 --2- 172.21.15.67:0/3654339630 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92c409eea0 0x7f92c409a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 >> 172.21.15.67:0/3654339630 conn(0x7f92c4071330 msgr2=0x7f92c4069380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 shutdown_connections 2024-09-15T01:53:35.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.642+0000 7f92cb0e8640 1 -- 172.21.15.67:0/3654339630 wait complete. 2024-09-15T01:53:35.652 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm-device-class all 2024-09-15T01:53:35.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.725+0000 7f5ad015d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:35.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.725+0000 7f5ad015d640 10 monclient: build_initial_monmap 2024-09-15T01:53:35.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.725+0000 7f5ad015d640 10 monclient: monmap: 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:35.726508+0000 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:35.726508+0000 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:35.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.725+0000 7f5ad015d640 1 Processor -- start 2024-09-15T01:53:35.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 -- start start 2024-09-15T01:53:35.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient: init 2024-09-15T01:53:35.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:35.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 0x7f5ac8069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac80a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 0x7f5ac809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ac8131b10 0x7f5ac8131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac8131b10 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac806f290 con 0x7f5ac80a0b50 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac809f7e0 con 0x7f5ac8131b10 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac806a270 con 0x7f5ac809eea0 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ad015d640 10 monclient(hunting): authenticate will time out at 848.195659s 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ac8131b10 0x7f5ac8131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5acd6f8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 msgr2=0x7f5ac809f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ac8131b10 0x7f5ac8131f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43384/0 (socket says 172.21.15.67:43384) 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 1 -- 172.21.15.67:0/3071048141 learned_addr learned my addr 172.21.15.67:0/3071048141 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): get_auth_request con 0x7f5ac8131b10 auth_method 0 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5acd6f8640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 0x7f5ac809f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:35.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.726+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5acdef9640 1 --2- 172.21.15.67:0/3071048141 >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 0x7f5ac8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_done global_id 4834 payload 274 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 -- 172.21.15.67:0/3071048141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 msgr2=0x7f5ac809f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 0x7f5ac809f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 -- 172.21.15.67:0/3071048141 >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 msgr2=0x7f5ac8069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/3071048141 >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 0x7f5ac8069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 10 monclient: _finish_hunting 0 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 monclient: found mon.noname-c 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5ace6fa640 1 -- 172.21.15.67:0/3071048141 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80a0f20 con 0x7f5ac8131b10 2024-09-15T01:53:35.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5acdef9640 10 monclient: get_auth_request con 0x7f5ac80a0b50 auth_method 0 2024-09-15T01:53:35.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.727+0000 7f5acdef9640 1 --2- 172.21.15.67:0/3071048141 >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 0x7f5ac8069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.729+0000 7f5ace6fa640 1 -- 172.21.15.67:0/3071048141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 msgr2=0x7f5ac8131f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.729+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac8131f50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5ac809c460 tx=0x7f5ac0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2c8e7eadcc3c40b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 1 -- 172.21.15.67:0/3071048141 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0007610 con 0x7f5ac8131b10 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: dump: 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.740 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: _finish_auth 0 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 1 -- 172.21.15.67:0/3071048141 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac002d050 con 0x7f5ac8131b10 2024-09-15T01:53:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 5 monclient: authenticate success, global_id 4834 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 1 -- 172.21.15.67:0/3071048141 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00079a0 con 0x7f5ac8131b10 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5accef7640 10 monclient: dump: 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.742 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 4 set_mon_vals no callback set 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 10 monclient: shutdown 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 -- 172.21.15.67:0/3071048141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 msgr2=0x7f5ac8131f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac8131f50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5ac809c460 tx=0x7f5ac0009840 comp rx=0 tx=0).stop 2024-09-15T01:53:35.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 -- 172.21.15.67:0/3071048141 shutdown_connections 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac8131f50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 --2- 172.21.15.67:0/3071048141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac809eea0 0x7f5ac809f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 --2- 172.21.15.67:0/3071048141 >> v2:172.21.15.17:3300/0 conn(0x7f5ac80a0b50 0x7f5ac8069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.730+0000 7f5ad015d640 1 -- 172.21.15.67:0/3071048141 >> 172.21.15.67:0/3071048141 conn(0x7f5ac8071290 msgr2=0x7f5ac80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 1 -- 172.21.15.67:0/3071048141 shutdown_connections 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 1 -- 172.21.15.67:0/3071048141 wait complete. 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 10 monclient: build_initial_monmap 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 10 monclient: monmap: 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:35.744 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 1 Processor -- start 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 1 -- start start 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.731+0000 7f5ad015d640 10 monclient: init 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 0x7f5ac809a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:35.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 0x7f5ac8130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac8131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac809b400 con 0x7f5ac809eea0 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80a1260 con 0x7f5ac8131b10 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81327f0 con 0x7f5ac80a0b50 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ad015d640 10 monclient(hunting): authenticate will time out at 848.201455s 2024-09-15T01:53:35.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5acd6f8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 msgr2=0x7f5ac8130c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5acd6f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 0x7f5ac8130c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43400/0 (socket says 172.21.15.67:43400) 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 -- 172.21.15.67:0/1389724904 learned_addr learned my addr 172.21.15.67:0/1389724904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): get_auth_request con 0x7f5ac8131b10 auth_method 0 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:35.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5acdef9640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 0x7f5ac809a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient(hunting): handle_auth_done global_id 4837 payload 824 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 -- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 msgr2=0x7f5ac8130c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 0x7f5ac8130c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 -- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 msgr2=0x7f5ac809a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 0x7f5ac809a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient: _finish_hunting 0 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 monclient: found mon.noname-b 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5ace6fa640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac0009ae0 con 0x7f5ac8131b10 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5acdef9640 10 monclient: get_auth_request con 0x7f5ac809eea0 auth_method 0 2024-09-15T01:53:35.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.732+0000 7f5acdef9640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 0x7f5ac809a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ace6fa640 1 -- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 msgr2=0x7f5ac811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ace6fa640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009ab0 tx=0x7f5ac0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0032ce0 con 0x7f5ac8131b10 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: dump: 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.749 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 5 monclient: authenticate success, global_id 4837 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0032e40 con 0x7f5ac8131b10 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 10 monclient: _renew_subs 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac811dd30 con 0x7f5ac8131b10 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac003b620 con 0x7f5ac8131b10 2024-09-15T01:53:35.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abdffb640 10 monclient: dump: 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5abffff640 4 set_mon_vals no callback set 2024-09-15T01:53:35.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 10 monclient: _renew_subs 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.735+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac811cf70 con 0x7f5ac8131b10 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ac0003020 con 0x7f5ac8131b10 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5ad015d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5ad015d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5ad015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac804bf60 con 0x7f5ac8131b10 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5abdffb640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 0x7f5aac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5ac006c5a0 con 0x7f5ac8131b10 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5acdef9640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 0x7f5aac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:35.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.736+0000 7f5acdef9640 10 monclient: get_auth_request con 0x7f5aac038720 auth_method 0 2024-09-15T01:53:35.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.739+0000 7f5acdef9640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 0x7f5aac03abd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4008a80 tx=0x7f5ac4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:35.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.740+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ac003ba10 con 0x7f5ac8131b10 2024-09-15T01:53:35.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.740+0000 7f5abdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:35.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.740+0000 7f5abdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:35.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.910+0000 7f5ad015d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:35.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.910+0000 7f5ad015d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:35.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.910+0000 7f5ad015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:35.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:35.910+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f5ac811cf70 con 0x7f5ac8131b10 2024-09-15T01:53:36.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.540+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7f5ac811cf70 con 0x7f5ac8131b10 2024-09-15T01:53:36.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.540+0000 7f5abdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:36.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.540+0000 7f5abdffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-15T01:53:36.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.647+0000 7f5ad015d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:36.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.647+0000 7f5ad015d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:36.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.647+0000 7f5ad015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:36.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:36.647+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f5ac811cf70 con 0x7f5ac8131b10 2024-09-15T01:53:37.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.190+0000 7f5abdffb640 1 -- 172.21.15.67:0/1389724904 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v44) v1 ==== 282+0+0 (secure 0 0 0) 0x7f5ac811cf70 con 0x7f5ac8131b10 2024-09-15T01:53:37.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.190+0000 7f5abdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T01:53:37.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.190+0000 7f5abdffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-15T01:53:37.192 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-15T01:53:37.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 msgr2=0x7f5aac03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 0x7f5aac03abd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4008a80 tx=0x7f5ac4008040 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 10 monclient: shutdown 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 msgr2=0x7f5ac811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009ab0 tx=0x7f5ac0036000 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 shutdown_connections 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5aac038720 0x7f5aac03abd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac8131b10 0x7f5ac811b940 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac80a0b50 0x7f5ac8130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 --2- 172.21.15.67:0/1389724904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac809eea0 0x7f5ac809a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 >> 172.21.15.67:0/1389724904 conn(0x7f5ac8071290 msgr2=0x7f5ac8134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 shutdown_connections 2024-09-15T01:53:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.193+0000 7f5ad015d640 1 -- 172.21.15.67:0/1389724904 wait complete. 2024-09-15T01:53:37.207 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.288+0000 7f80ead5d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.288+0000 7f80ead5d640 10 monclient: build_initial_monmap 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.288+0000 7f80ead5d640 10 monclient: monmap: 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:37.290 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:37.288924+0000 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:37.288924+0000 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:37.291 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.289+0000 7f80ead5d640 1 Processor -- start 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.289+0000 7f80ead5d640 1 -- start start 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.289+0000 7f80ead5d640 10 monclient: init 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.289+0000 7f80ead5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.289+0000 7f80ead5d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T01:53:37.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80e40a3830 0x7f80e406b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-c con 0x7f80e40a3830 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 0x7f80e409db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-b con 0x7f80e406e2b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80e409e050 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e4069c60 con 0x7f80e409e050 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40a3c00 con 0x7f80e40a3830 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:37.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e4069650 con 0x7f80e406e2b0 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80ead5d640 10 monclient(hunting): authenticate will time out at 849.758079s 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80dbfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 msgr2=0x7f80e409db10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 0x7f80e409db10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80e40a3830 0x7f80e406b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80e40a3830 0x7f80e406b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43408/0 (socket says 172.21.15.67:43408) 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 1 -- 172.21.15.67:0/1469678627 learned_addr learned my addr 172.21.15.67:0/1469678627 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 10 monclient(hunting): get_auth_request con 0x7f80e40a3830 auth_method 0 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e92fa640 1 --1- 172.21.15.67:0/1469678627 >> v1:172.21.15.67:6789/0 conn(0x7f80e409e050 0x7f80e4138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60446/0 (socket says 172.21.15.67:60446) 2024-09-15T01:53:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.290+0000 7f80e8af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e92fa640 1 -- 172.21.15.67:0/1469678627 >> v1:172.21.15.67:6789/0 conn(0x7f80e409e050 legacy=0x7f80e4138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient(hunting): handle_auth_done global_id 4840 payload 274 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 1 -- 172.21.15.67:0/1469678627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 msgr2=0x7f80e409db10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 1 --2- 172.21.15.67:0/1469678627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 0x7f80e409db10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 1 -- 172.21.15.67:0/1469678627 >> v1:172.21.15.67:6789/0 conn(0x7f80e409e050 legacy=0x7f80e4138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 1 monclient: found mon.noname-c 2024-09-15T01:53:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:37.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.292+0000 7f80e8af9640 1 -- 172.21.15.67:0/1469678627 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e409f500 con 0x7f80e40a3830 2024-09-15T01:53:37.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80e8af9640 1 -- 172.21.15.67:0/1469678627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 msgr2=0x7f80e406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80e8af9640 1 --2- 172.21.15.67:0/1469678627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e406b9b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009990 tx=0x7f80e00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c09a560a26a752f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 1 -- 172.21.15.67:0/1469678627 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e002f4b0 con 0x7f80e40a3830 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: dump: 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.304 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80ead5d640 5 monclient: authenticate success, global_id 4840 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 1 -- 172.21.15.67:0/1469678627 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e002f610 con 0x7f80e40a3830 2024-09-15T01:53:37.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80ead5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80ead5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80ead5d640 4 set_mon_vals no callback set 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 1 -- 172.21.15.67:0/1469678627 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e002f9b0 con 0x7f80e40a3830 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.295+0000 7f80db7fe640 10 monclient: dump: 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.306 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 10 monclient: shutdown 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 -- 172.21.15.67:0/1469678627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 msgr2=0x7f80e406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 --2- 172.21.15.67:0/1469678627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e406b9b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009990 tx=0x7f80e00091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 -- 172.21.15.67:0/1469678627 shutdown_connections 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 --2- 172.21.15.67:0/1469678627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e406e2b0 0x7f80e409db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 --2- 172.21.15.67:0/1469678627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e406b9b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 -- 172.21.15.67:0/1469678627 >> 172.21.15.67:0/1469678627 conn(0x7f80e4070ff0 msgr2=0x7f80e40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 -- 172.21.15.67:0/1469678627 shutdown_connections 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 1 -- 172.21.15.67:0/1469678627 wait complete. 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 10 monclient: build_initial_monmap 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.296+0000 7f80ead5d640 10 monclient: monmap: 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:37.308 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 Processor -- start 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 -- start start 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient: init 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 0x7f80e413e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-a con 0x7f80e406e2b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 0x7f80e413d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-c con 0x7f80e409e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): picked mon.noname-b con 0x7f80e40a3830 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e4123160 con 0x7f80e406e2b0 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e4122ba0 con 0x7f80e40a3830 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e41228c0 con 0x7f80e409e050 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80ead5d640 10 monclient(hunting): authenticate will time out at 849.765756s 2024-09-15T01:53:37.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80dbfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 msgr2=0x7f80e413d820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 0x7f80e413d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43414/0 (socket says 172.21.15.67:43414) 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 1 -- 172.21.15.67:0/4174401403 learned_addr learned my addr 172.21.15.67:0/4174401403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): get_auth_request con 0x7f80e40a3830 auth_method 0 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:37.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.297+0000 7f80e92fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 0x7f80e413e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 10 monclient(hunting): get_auth_request con 0x7f80e406e2b0 auth_method 0 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 10 monclient(hunting): handle_auth_done global_id 4846 payload 824 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 -- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 msgr2=0x7f80e413d820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 0x7f80e413d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 -- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 msgr2=0x7f80e413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 0x7f80e413e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 10 monclient: _finish_hunting 0 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 monclient: found mon.noname-b 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e92fa640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0009450 con 0x7f80e40a3830 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.298+0000 7f80e8af9640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 0x7f80e413e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80e92fa640 1 -- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 msgr2=0x7f80e41416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80e92fa640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c860 tx=0x7f80dc001870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00ce50 con 0x7f80e40a3830 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: dump: 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc00a640 con 0x7f80e40a3830 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 5 monclient: authenticate success, global_id 4846 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00aa00 con 0x7f80e40a3830 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 10 monclient: _renew_subs 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e4122e80 con 0x7f80e40a3830 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80d8ff9640 10 monclient: dump: 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:37.316 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80daffd640 4 set_mon_vals no callback set 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 10 monclient: _renew_subs 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.300+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e413dd60 con 0x7f80e40a3830 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f80dc0070f0 con 0x7f80e40a3830 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80ead5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80ead5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80ead5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:37.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80d8ff9640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 0x7f80d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.301+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4248+0+0 (secure 0 0 0) 0x7f80dc04b5d0 con 0x7f80e40a3830 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.302+0000 7f80e8af9640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 0x7f80d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.302+0000 7f80e8af9640 10 monclient: get_auth_request con 0x7f80d0038740 auth_method 0 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.304+0000 7f80e8af9640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 0x7f80d003abf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e0031000 tx=0x7f80e0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.306+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:37.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.306+0000 7f80d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:37.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.306+0000 7f80d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:37.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.474+0000 7f80ead5d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:37.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.474+0000 7f80ead5d640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:37.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.474+0000 7f80ead5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:37.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:37.474+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:38.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.201+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v45) v1 ==== 135+0+0 (secure 0 0 0) 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:38.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.201+0000 7f80d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:38.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.201+0000 7f80d8ff9640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-15T01:53:38.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.307+0000 7f80ead5d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:38.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.307+0000 7f80ead5d640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:38.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.307+0000 7f80ead5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:38.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:38.307+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:39.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.199+0000 7f80d8ff9640 1 -- 172.21.15.67:0/4174401403 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v46) v1 ==== 237+0+0 (secure 0 0 0) 0x7f80e409ef90 con 0x7f80e40a3830 2024-09-15T01:53:39.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.199+0000 7f80d8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T01:53:39.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.199+0000 7f80d8ff9640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-15T01:53:39.201 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 msgr2=0x7f80d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 0x7f80d003abf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e0031000 tx=0x7f80e0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 10 monclient: shutdown 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 msgr2=0x7f80e41416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c860 tx=0x7f80dc001870 comp rx=0 tx=0).stop 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 shutdown_connections 2024-09-15T01:53:39.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80d0038740 0x7f80d003abf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e40a3830 0x7f80e41416e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80e409e050 0x7f80e413d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 --2- 172.21.15.67:0/4174401403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80e406e2b0 0x7f80e413e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 >> 172.21.15.67:0/4174401403 conn(0x7f80e4070ff0 msgr2=0x7f80e40a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.202+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 shutdown_connections 2024-09-15T01:53:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.203+0000 7f80ead5d640 1 -- 172.21.15.67:0/4174401403 wait complete. 2024-09-15T01:53:39.215 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.296+0000 7f993cf00640 10 monclient: get_monmap_and_config 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.296+0000 7f993cf00640 10 monclient: build_initial_monmap 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.296+0000 7f993cf00640 10 monclient: monmap: 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:39.296909+0000 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:39.296909+0000 2024-09-15T01:53:39.298 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:39.299 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.297+0000 7f993cf00640 1 Processor -- start 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.297+0000 7f993cf00640 1 -- start start 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.297+0000 7f993cf00640 10 monclient: init 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99380a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f993809eea0 0x7f993809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-c con 0x7f993809eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9938136e40 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993806a270 con 0x7f9938136e40 2024-09-15T01:53:39.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99380a0f20 con 0x7f99380a0b50 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f993806f290 con 0x7f993809eea0 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993cf00640 10 monclient(hunting): authenticate will time out at 851.766245s 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f993809eea0 0x7f993809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f993809eea0 0x7f993809f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43428/0 (socket says 172.21.15.67:43428) 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 1 -- 172.21.15.67:0/4189354836 learned_addr learned my addr 172.21.15.67:0/4189354836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 10 monclient(hunting): get_auth_request con 0x7f993809eea0 auth_method 0 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f993659c640 1 --1- 172.21.15.67:0/4189354836 >> v1:172.21.15.67:6789/0 conn(0x7f99380a0b50 0x7f9938069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60452/0 (socket says 172.21.15.67:60452) 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:39.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:39.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.298+0000 7f9935d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:39.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f993659c640 1 -- 172.21.15.67:0/4189354836 >> v1:172.21.15.67:6789/0 conn(0x7f99380a0b50 legacy=0x7f9938069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient(hunting): handle_auth_done global_id 4849 payload 274 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9936d9d640 1 -- 172.21.15.67:0/4189354836 >> v1:172.21.15.17:6789/0 conn(0x7f9938136e40 legacy=0x7f9938139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 1 -- 172.21.15.67:0/4189354836 >> v1:172.21.15.67:6789/0 conn(0x7f99380a0b50 legacy=0x7f9938069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 1 -- 172.21.15.67:0/4189354836 >> v1:172.21.15.17:6789/0 conn(0x7f9938136e40 legacy=0x7f9938139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 1 monclient: found mon.noname-c 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:39.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.300+0000 7f9935d9b640 1 -- 172.21.15.67:0/4189354836 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99380a6160 con 0x7f993809eea0 2024-09-15T01:53:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f9935d9b640 1 -- 172.21.15.67:0/4189354836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f993809eea0 msgr2=0x7f993809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f9935d9b640 1 --2- 172.21.15.67:0/4189354836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f993809eea0 0x7f993809f2a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9920009a10 tx=0x7f99200091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c1ab9f98cfed47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 1 -- 172.21.15.67:0/4189354836 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992002f4b0 con 0x7f993809eea0 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: dump: 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: _finish_auth 0 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 1 -- 172.21.15.67:0/4189354836 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992002f610 con 0x7f993809eea0 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993cf00640 5 monclient: authenticate success, global_id 4849 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993cf00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993cf00640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993cf00640 4 set_mon_vals no callback set 2024-09-15T01:53:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 1 -- 172.21.15.67:0/4189354836 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992002f9b0 con 0x7f993809eea0 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.303+0000 7f993559a640 10 monclient: dump: 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 10 monclient: shutdown 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 -- 172.21.15.67:0/4189354836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f993809eea0 msgr2=0x7f993809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 --2- 172.21.15.67:0/4189354836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f993809eea0 0x7f993809f2a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9920009a10 tx=0x7f99200091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 -- 172.21.15.67:0/4189354836 shutdown_connections 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 --2- 172.21.15.67:0/4189354836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f993809eea0 0x7f993809f2a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 -- 172.21.15.67:0/4189354836 >> 172.21.15.67:0/4189354836 conn(0x7f9938071290 msgr2=0x7f99380a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 -- 172.21.15.67:0/4189354836 shutdown_connections 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 1 -- 172.21.15.67:0/4189354836 wait complete. 2024-09-15T01:53:39.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 10 monclient: build_initial_monmap 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.304+0000 7f993cf00640 10 monclient: monmap: 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 Processor -- start 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 -- start start 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient: init 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 0x7f9938136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-a con 0x7f993809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-b con 0x7f99380a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 0x7f993811be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): picked mon.noname-c con 0x7f9938136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:39.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993811d0f0 con 0x7f993809eea0 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993811f360 con 0x7f99380a0b50 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9938135d30 con 0x7f9938136e40 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f993cf00640 10 monclient(hunting): authenticate will time out at 851.773743s 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9936d9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 msgr2=0x7f993811be80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9936d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 0x7f993811be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43440/0 (socket says 172.21.15.67:43440) 2024-09-15T01:53:39.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 1 -- 172.21.15.67:0/3484677262 learned_addr learned my addr 172.21.15.67:0/3484677262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): get_auth_request con 0x7f99380a0b50 auth_method 0 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.305+0000 7f9935d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 0x7f9938136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 10 monclient(hunting): get_auth_request con 0x7f993809eea0 auth_method 0 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 10 monclient(hunting): handle_auth_done global_id 4855 payload 824 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 -- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 msgr2=0x7f993811be80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 0x7f993811be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 -- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 msgr2=0x7f9938136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 0x7f9938136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 monclient: found mon.noname-b 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f9935d9b640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9920009450 con 0x7f99380a0b50 2024-09-15T01:53:39.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.306+0000 7f993659c640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 0x7f9938136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:39.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f9935d9b640 1 -- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 msgr2=0x7f993811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:39.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f9935d9b640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9920009a40 tx=0x7f9920032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99200329e0 con 0x7f99380a0b50 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: dump: 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9920032b40 con 0x7f99380a0b50 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 5 monclient: authenticate success, global_id 4855 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 10 monclient: _renew_subs 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99380a1260 con 0x7f99380a0b50 2024-09-15T01:53:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992003c440 con 0x7f99380a0b50 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f992e7fc640 10 monclient: dump: 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f9934d99640 4 set_mon_vals no callback set 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 10 monclient: _renew_subs 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.308+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993811e720 con 0x7f99380a0b50 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f992003c8b0 con 0x7f99380a0b50 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993cf00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993cf00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993cf00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993811cc50 con 0x7f99380a0b50 2024-09-15T01:53:39.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f992e7fc640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 0x7f991803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4472+0+0 (secure 0 0 0) 0x7f992006be40 con 0x7f99380a0b50 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993659c640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 0x7f991803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.309+0000 7f993659c640 10 monclient: get_auth_request con 0x7f99180386f0 auth_method 0 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.312+0000 7f993659c640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 0x7f991803aba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9928007660 tx=0x7f9928007b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.314+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9920044de0 con 0x7f99380a0b50 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.314+0000 7f992e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.314+0000 7f992e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:39.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.483+0000 7f993cf00640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:39.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.483+0000 7f993cf00640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:39.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.483+0000 7f993cf00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:39.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:39.483+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f993804bf60 con 0x7f99380a0b50 2024-09-15T01:53:40.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.214+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f9920037070 con 0x7f99380a0b50 2024-09-15T01:53:40.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.214+0000 7f992e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:40.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.214+0000 7f992e7fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-15T01:53:40.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.318+0000 7f993cf00640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:40.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.318+0000 7f993cf00640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:40.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.318+0000 7f993cf00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:40.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:40.318+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f99380008d0 con 0x7f99380a0b50 2024-09-15T01:53:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.212+0000 7f992e7fc640 1 -- 172.21.15.67:0/3484677262 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v48) v1 ==== 237+0+0 (secure 0 0 0) 0x7f99200447e0 con 0x7f99380a0b50 2024-09-15T01:53:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.212+0000 7f992e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T01:53:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.212+0000 7f992e7fc640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-15T01:53:41.214 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-15T01:53:41.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.214+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 msgr2=0x7f991803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.214+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 0x7f991803aba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9928007660 tx=0x7f9928007b50 comp rx=0 tx=0).stop 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 10 monclient: shutdown 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 msgr2=0x7f993811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9920009a40 tx=0x7f9920032000 comp rx=0 tx=0).stop 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 shutdown_connections 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99180386f0 0x7f991803aba0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9938136e40 0x7f993811be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99380a0b50 0x7f993811b940 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 --2- 172.21.15.67:0/3484677262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f993809eea0 0x7f9938136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 >> 172.21.15.67:0/3484677262 conn(0x7f9938071290 msgr2=0x7f9938139220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:41.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 shutdown_connections 2024-09-15T01:53:41.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.215+0000 7f993cf00640 1 -- 172.21.15.67:0/3484677262 wait complete. 2024-09-15T01:53:41.229 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-15T01:53:41.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.315+0000 7fd4da8ef640 10 monclient: get_monmap_and_config 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.315+0000 7fd4da8ef640 10 monclient: build_initial_monmap 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.315+0000 7fd4da8ef640 10 monclient: monmap: 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:41.315559+0000 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:41.315559+0000 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:41.317 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:41.318 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:41.318 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.316+0000 7fd4da8ef640 1 Processor -- start 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.316+0000 7fd4da8ef640 1 -- start start 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.316+0000 7fd4da8ef640 10 monclient: init 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4d4131cf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4098a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d40a0470 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:41.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd4d4132310 0x7fd4d4139030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d4132310 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d40a1890 con 0x7fd4d4131cf0 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d41320c0 con 0x7fd4d4132310 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d41329d0 con 0x7fd4d40a0470 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4da8ef640 10 monclient(hunting): authenticate will time out at 853.785238s 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d37fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 msgr2=0x7fd4d4098a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4098a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd4d4132310 0x7fd4d4139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:41.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd4d4132310 0x7fd4d4139030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:35808/0 (socket says 172.21.15.67:35808) 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3041921395 learned_addr learned my addr 172.21.15.67:0/3041921395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): get_auth_request con 0x7fd4d4132310 auth_method 0 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.317+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d3fff640 1 -- 172.21.15.67:0/3041921395 >> v1:172.21.15.17:6789/0 conn(0x7fd4d4131cf0 legacy=0x7fd4d4098370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_done global_id 44298 payload 274 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 msgr2=0x7fd4d4098a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 1 --2- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4098a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3041921395 >> v1:172.21.15.17:6789/0 conn(0x7fd4d4131cf0 legacy=0x7fd4d4098370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 1 monclient: found mon.noname-a 2024-09-15T01:53:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:41.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:41.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.319+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3041921395 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4132710 con 0x7fd4d4132310 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4132310 msgr2=0x7fd4d4139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d8e8c640 1 --2- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4132310 0x7fd4d4139030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4132900 tx=0x7fd4cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4585ed16210a1845 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 1 -- 172.21.15.67:0/3041921395 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc007610 con 0x7fd4d4132310 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: dump: 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:41.326 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:41.327 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.327 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 5 monclient: authenticate success, global_id 44298 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 1 -- 172.21.15.67:0/3041921395 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc02d050 con 0x7fd4d4132310 2024-09-15T01:53:41.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 4 set_mon_vals no callback set 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 1 -- 172.21.15.67:0/3041921395 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc0079a0 con 0x7fd4d4132310 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4d2ffd640 10 monclient: dump: 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.329 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 10 monclient: shutdown 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4132310 msgr2=0x7fd4d4139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4132310 0x7fd4d4139030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4132900 tx=0x7fd4cc009840 comp rx=0 tx=0).stop 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3041921395 shutdown_connections 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4132310 0x7fd4d4139030 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3041921395 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4098a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.323+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3041921395 >> 172.21.15.67:0/3041921395 conn(0x7fd4d4071280 msgr2=0x7fd4d40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.324+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3041921395 shutdown_connections 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.324+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3041921395 wait complete. 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.324+0000 7fd4da8ef640 10 monclient: build_initial_monmap 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.324+0000 7fd4da8ef640 10 monclient: monmap: 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:41.331 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:41.332 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:41.332 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.324+0000 7fd4da8ef640 1 Processor -- start 2024-09-15T01:53:41.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 -- start start 2024-09-15T01:53:41.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient: init 2024-09-15T01:53:41.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:41.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d40a0470 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 0x7fd4d4133190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d4131cf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d4132310 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:41.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4136800 con 0x7fd4d4131cf0 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d40a0b30 con 0x7fd4d4132310 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4136520 con 0x7fd4d40a0470 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4da8ef640 10 monclient(hunting): authenticate will time out at 853.793576s 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d3fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 msgr2=0x7fd4d4132c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60402/0 (socket says 172.21.15.67:60402) 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3112308101 learned_addr learned my addr 172.21.15.67:0/3112308101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): get_auth_request con 0x7fd4d4132310 auth_method 0 2024-09-15T01:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d37fe640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 0x7fd4d4133190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d37fe640 10 monclient(hunting): get_auth_request con 0x7fd4d4131cf0 auth_method 0 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.325+0000 7fd4d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 10 monclient(hunting): handle_auth_done global_id 4858 payload 824 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 msgr2=0x7fd4d4132c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:41.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 msgr2=0x7fd4d4133190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 0x7fd4d4133190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 monclient: found mon.noname-b 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc009ae0 con 0x7fd4d4132310 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.326+0000 7fd4d37fe640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 0x7fd4d4133190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.328+0000 7fd4d8e8c640 1 -- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 msgr2=0x7fd4d41336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.328+0000 7fd4d8e8c640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009880 tx=0x7fd4cc036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.328+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc031cb0 con 0x7fd4d4132310 2024-09-15T01:53:41.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: dump: 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:41.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc031e10 con 0x7fd4d4132310 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc03c650 con 0x7fd4d4132310 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4bbfff640 10 monclient: dump: 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.342 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 5 monclient: authenticate success, global_id 4858 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 10 monclient: _renew_subs 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d4136240 con 0x7fd4d4132310 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4d27fc640 4 set_mon_vals no callback set 2024-09-15T01:53:41.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 10 monclient: _renew_subs 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.330+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d4138760 con 0x7fd4d4132310 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd4cc03ca40 con 0x7fd4d4132310 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4da8ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4da8ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4da8ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d404bf60 con 0x7fd4d4132310 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4bbfff640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 0x7fd4c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4704+0+0 (secure 0 0 0) 0x7fd4cc06ce00 con 0x7fd4d4132310 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4d3fff640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 0x7fd4c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.331+0000 7fd4d3fff640 10 monclient: get_auth_request con 0x7fd4c0038740 auth_method 0 2024-09-15T01:53:41.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.334+0000 7fd4d3fff640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 0x7fd4c003abf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40014b0 tx=0x7fd4c4007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:41.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.336+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd4cc03ccf0 con 0x7fd4d4132310 2024-09-15T01:53:41.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.336+0000 7fd4bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:41.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.336+0000 7fd4bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:41.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.502+0000 7fd4da8ef640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:41.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.502+0000 7fd4da8ef640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:41.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.502+0000 7fd4da8ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:41.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:41.502+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7fd4d40a15b0 con 0x7fd4d4132310 2024-09-15T01:53:42.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.192+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v49) v1 ==== 150+0+0 (secure 0 0 0) 0x7fd4cc040030 con 0x7fd4d4132310 2024-09-15T01:53:42.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.192+0000 7fd4bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:42.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.192+0000 7fd4bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.297+0000 7fd4da8ef640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.297+0000 7fd4da8ef640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.297+0000 7fd4da8ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.297+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7fd4d40679b0 con 0x7fd4d4132310 2024-09-15T01:53:42.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.298+0000 7fd4bbfff640 1 -- 172.21.15.67:0/3112308101 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v49) v1 ==== 177+0+0 (secure 0 0 0) 0x7fd4cc040020 con 0x7fd4d4132310 2024-09-15T01:53:42.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.298+0000 7fd4bbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T01:53:42.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.298+0000 7fd4bbfff640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-15T01:53:42.301 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo-ssd already exists 2024-09-15T01:53:42.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 msgr2=0x7fd4c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:42.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 0x7fd4c003abf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40014b0 tx=0x7fd4c4007a10 comp rx=0 tx=0).stop 2024-09-15T01:53:42.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 10 monclient: shutdown 2024-09-15T01:53:42.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:42.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 msgr2=0x7fd4d41336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009880 tx=0x7fd4cc036000 comp rx=0 tx=0).stop 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 shutdown_connections 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4c0038740 0x7fd4c003abf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4d4131cf0 0x7fd4d4133190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4132310 0x7fd4d41336d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 --2- 172.21.15.67:0/3112308101 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4d40a0470 0x7fd4d4132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.300+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 >> 172.21.15.67:0/3112308101 conn(0x7fd4d4071280 msgr2=0x7fd4d40a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.301+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 shutdown_connections 2024-09-15T01:53:42.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.301+0000 7fd4da8ef640 1 -- 172.21.15.67:0/3112308101 wait complete. 2024-09-15T01:53:42.312 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.389+0000 7fc5f2b00640 10 monclient: get_monmap_and_config 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.389+0000 7fc5f2b00640 10 monclient: build_initial_monmap 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.389+0000 7fc5f2b00640 10 monclient: monmap: 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:42.391085+0000 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:42.391085+0000 2024-09-15T01:53:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:42.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.390+0000 7fc5f2b00640 1 Processor -- start 2024-09-15T01:53:42.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.390+0000 7fc5f2b00640 1 -- start start 2024-09-15T01:53:42.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.390+0000 7fc5f2b00640 10 monclient: init 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc5ec0a0b50 0x7fc5ec069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec0a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5ec09eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5ec09f2a0 0x7fc5ec132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ec09f2a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec06a270 con 0x7fc5ec09eea0 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec06f290 con 0x7fc5ec0a0b50 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec0a0f20 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f2b00640 10 monclient(hunting): authenticate will time out at 854.860180s 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5ec09f2a0 0x7fc5ec132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5ec09f2a0 0x7fc5ec132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60406/0 (socket says 172.21.15.67:60406) 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 -- 172.21.15.67:0/871629778 learned_addr learned my addr 172.21.15.67:0/871629778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): get_auth_request con 0x7fc5ec09f2a0 auth_method 0 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f089c640 1 --2- 172.21.15.67:0/871629778 >> v2:172.21.15.17:3300/0 conn(0x7fc5ec0a0b50 0x7fc5ec069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:42.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f089c640 10 monclient(hunting): get_auth_request con 0x7fc5ec0a0b50 auth_method 0 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f089c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f089c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f089c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient(hunting): handle_auth_done global_id 4861 payload 274 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 -- 172.21.15.67:0/871629778 >> v2:172.21.15.17:3300/0 conn(0x7fc5ec0a0b50 msgr2=0x7fc5ec069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 --2- 172.21.15.67:0/871629778 >> v2:172.21.15.17:3300/0 conn(0x7fc5ec0a0b50 0x7fc5ec069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 -- 172.21.15.67:0/871629778 >> v1:172.21.15.17:6789/0 conn(0x7fc5ec09eea0 legacy=0x7fc5ec131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 monclient: found mon.noname-c 2024-09-15T01:53:42.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:42.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.391+0000 7fc5f109d640 1 -- 172.21.15.67:0/871629778 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0a6160 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f109d640 1 -- 172.21.15.67:0/871629778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 msgr2=0x7fc5ec132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f109d640 1 --2- 172.21.15.67:0/871629778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec132220 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec09cdb0 tx=0x7fc5e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f64d0ceaa13f39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 1 -- 172.21.15.67:0/871629778 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4007610 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: dump: 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:42.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 1 -- 172.21.15.67:0/871629778 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e402d050 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f2b00640 5 monclient: authenticate success, global_id 4861 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f2b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:42.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f2b00640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5f2b00640 4 set_mon_vals no callback set 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 1 -- 172.21.15.67:0/871629778 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40079a0 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.394+0000 7fc5eb7fe640 10 monclient: dump: 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:42.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 10 monclient: shutdown 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 -- 172.21.15.67:0/871629778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 msgr2=0x7fc5ec132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/871629778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec132220 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec09cdb0 tx=0x7fc5e4009840 comp rx=0 tx=0).stop 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 -- 172.21.15.67:0/871629778 shutdown_connections 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/871629778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec132220 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/871629778 >> v2:172.21.15.17:3300/0 conn(0x7fc5ec0a0b50 0x7fc5ec069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 -- 172.21.15.67:0/871629778 >> 172.21.15.67:0/871629778 conn(0x7fc5ec071290 msgr2=0x7fc5ec0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 -- 172.21.15.67:0/871629778 shutdown_connections 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 1 -- 172.21.15.67:0/871629778 wait complete. 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 10 monclient: build_initial_monmap 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.395+0000 7fc5f2b00640 10 monclient: monmap: 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:42.409 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 Processor -- start 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 -- start start 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient: init 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5ec09eea0 0x7fc5ec141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec09eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ec09f2a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 0x7fc5ec11d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ec0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:42.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0a31b0 con 0x7fc5ec09eea0 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec121500 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec120eb0 con 0x7fc5ec0a0b50 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f2b00640 10 monclient(hunting): authenticate will time out at 854.865730s 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f109d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 msgr2=0x7fc5ec11d8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f109d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 0x7fc5ec11d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60410/0 (socket says 172.21.15.67:60410) 2024-09-15T01:53:42.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 1 -- 172.21.15.67:0/1517934540 learned_addr learned my addr 172.21.15.67:0/1517934540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request con 0x7fc5ec09f2a0 auth_method 0 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f089c640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5ec09eea0 0x7fc5ec141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f089c640 10 monclient(hunting): get_auth_request con 0x7fc5ec09eea0 auth_method 0 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f089c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f089c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5f089c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.396+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:42.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5f089c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5f089c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5f089c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_done global_id 4864 payload 824 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 -- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 msgr2=0x7fc5ec11d8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 0x7fc5ec11d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 -- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5ec09eea0 msgr2=0x7fc5ec141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5ec09eea0 0x7fc5ec141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 monclient: found mon.noname-b 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.397+0000 7fc5ebfff640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4009ae0 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5ebfff640 1 -- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 msgr2=0x7fc5ec133d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5ebfff640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000b4f0 tx=0x7fc5e00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000f720 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: dump: 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:42.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e000f8a0 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 5 monclient: authenticate success, global_id 4864 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 10 monclient: _renew_subs 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec121190 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000fc60 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:42.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5e8ff9640 10 monclient: dump: 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5eaffd640 4 set_mon_vals no callback set 2024-09-15T01:53:42.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 10 monclient: _renew_subs 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.399+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec141790 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc5e0011610 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f2b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f2b00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f2b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec11f340 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5e8ff9640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 0x7fc5d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4767+0+0 (secure 0 0 0) 0x7fc5e004b610 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f089c640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 0x7fc5d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:42.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.400+0000 7fc5f089c640 10 monclient: get_auth_request con 0x7fc5d80386f0 auth_method 0 2024-09-15T01:53:42.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.403+0000 7fc5f089c640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 0x7fc5d803aba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc007ba0 tx=0x7fc5dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:42.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.405+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc5e00118c0 con 0x7fc5ec09f2a0 2024-09-15T01:53:42.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.405+0000 7fc5e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:42.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.405+0000 7fc5e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:42.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.571+0000 7fc5f2b00640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:42.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.571+0000 7fc5f2b00640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:42.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.571+0000 7fc5f2b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:42.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:42.571+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7fc5ec04bf60 con 0x7fc5ec09f2a0 2024-09-15T01:53:43.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.231+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v50) v1 ==== 150+0+0 (secure 0 0 0) 0x7fc5e0017720 con 0x7fc5ec09f2a0 2024-09-15T01:53:43.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.231+0000 7fc5e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:43.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.231+0000 7fc5e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:43.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.335+0000 7fc5f2b00640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:43.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.335+0000 7fc5f2b00640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:43.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.335+0000 7fc5f2b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.335+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7fc5ec0a35f0 con 0x7fc5ec09f2a0 2024-09-15T01:53:43.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.336+0000 7fc5e8ff9640 1 -- 172.21.15.67:0/1517934540 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v50) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc5e0017900 con 0x7fc5ec09f2a0 2024-09-15T01:53:43.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.336+0000 7fc5e8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T01:53:43.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.336+0000 7fc5e8ff9640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-15T01:53:43.339 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo-hdd already exists 2024-09-15T01:53:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 msgr2=0x7fc5d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 0x7fc5d803aba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc007ba0 tx=0x7fc5dc008040 comp rx=0 tx=0).stop 2024-09-15T01:53:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 10 monclient: shutdown 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 msgr2=0x7fc5ec133d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.338+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000b4f0 tx=0x7fc5e00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 shutdown_connections 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc5d80386f0 0x7fc5d803aba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc5ec0a0b50 0x7fc5ec11d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec09f2a0 0x7fc5ec133d90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 --2- 172.21.15.67:0/1517934540 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc5ec09eea0 0x7fc5ec141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 >> 172.21.15.67:0/1517934540 conn(0x7fc5ec071290 msgr2=0x7fc5ec134bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 shutdown_connections 2024-09-15T01:53:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.339+0000 7fc5f2b00640 1 -- 172.21.15.67:0/1517934540 wait complete. 2024-09-15T01:53:43.351 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-15T01:53:43.351 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-ssd 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.425+0000 7fcfae694640 10 monclient: get_monmap_and_config 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.425+0000 7fcfae694640 10 monclient: build_initial_monmap 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.425+0000 7fcfae694640 10 monclient: monmap: 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:43.426821+0000 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:43.426821+0000 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:43.428 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:43.429 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:43.429 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:43.429 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:43.429 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 1 Processor -- start 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 1 -- start start 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 10 monclient: init 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T01:53:43.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.426+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfa806e860 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcfa80980d0 0x7fcfa8131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa80980d0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa8136ea0 0x7fcfa8139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa8136ea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa809dce0 con 0x7fcfa806e860 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa806ec30 con 0x7fcfa80980d0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa81326a0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfae694640 10 monclient(hunting): authenticate will time out at 855.896073s 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa8136ea0 0x7fcfa8139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa8136ea0 0x7fcfa8139280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60420/0 (socket says 172.21.15.67:60420) 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 -- 172.21.15.67:0/3607410229 learned_addr learned my addr 172.21.15.67:0/3607410229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): get_auth_request con 0x7fcfa8136ea0 auth_method 0 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcf9ffff640 1 --2- 172.21.15.67:0/3607410229 >> v2:172.21.15.17:3300/0 conn(0x7fcfa80980d0 0x7fcfa8131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcf9ffff640 10 monclient(hunting): get_auth_request con 0x7fcfa80980d0 auth_method 0 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcf9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcf9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcf9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient(hunting): handle_auth_done global_id 4867 payload 274 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 -- 172.21.15.67:0/3607410229 >> v2:172.21.15.17:3300/0 conn(0x7fcfa80980d0 msgr2=0x7fcfa8131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 --2- 172.21.15.67:0/3607410229 >> v2:172.21.15.17:3300/0 conn(0x7fcfa80980d0 0x7fcfa8131fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 -- 172.21.15.67:0/3607410229 >> v1:172.21.15.17:6789/0 conn(0x7fcfa806e860 legacy=0x7fcfa80979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient: _finish_hunting 0 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 monclient: found mon.noname-c 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.427+0000 7fcfacc31640 1 -- 172.21.15.67:0/3607410229 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa809d620 con 0x7fcfa8136ea0 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfacc31640 1 -- 172.21.15.67:0/3607410229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 msgr2=0x7fcfa8139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfacc31640 1 --2- 172.21.15.67:0/3607410229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8139280 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f20 tx=0x7fcfa0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aeedcaa83f97340 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 1 -- 172.21.15.67:0/3607410229 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0007610 con 0x7fcfa8136ea0 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: dump: 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.440 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 1 -- 172.21.15.67:0/3607410229 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa002d050 con 0x7fcfa8136ea0 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfae694640 5 monclient: authenticate success, global_id 4867 2024-09-15T01:53:43.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfae694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 1 -- 172.21.15.67:0/3607410229 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00079a0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfa77fe640 10 monclient: dump: 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfae694640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.430+0000 7fcfae694640 4 set_mon_vals no callback set 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 10 monclient: shutdown 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 -- 172.21.15.67:0/3607410229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 msgr2=0x7fcfa8139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 --2- 172.21.15.67:0/3607410229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8139280 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f20 tx=0x7fcfa0009840 comp rx=0 tx=0).stop 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 -- 172.21.15.67:0/3607410229 shutdown_connections 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 --2- 172.21.15.67:0/3607410229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8139280 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 --2- 172.21.15.67:0/3607410229 >> v2:172.21.15.17:3300/0 conn(0x7fcfa80980d0 0x7fcfa8131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 -- 172.21.15.67:0/3607410229 >> 172.21.15.67:0/3607410229 conn(0x7fcfa80a13a0 msgr2=0x7fcfa80a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:43.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 -- 172.21.15.67:0/3607410229 shutdown_connections 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 -- 172.21.15.67:0/3607410229 wait complete. 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 10 monclient: build_initial_monmap 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 10 monclient: monmap: 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:43.444 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.431+0000 7fcfae694640 1 Processor -- start 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 -- start start 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient: init 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfa806e860 0x7fcfa80989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa806e860 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 0x7fcfa8098ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa80980d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa8136ea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa809a920 con 0x7fcfa806e860 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8132d00 con 0x7fcfa8136ea0 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa809a5a0 con 0x7fcfa80980d0 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfae694640 10 monclient(hunting): authenticate will time out at 855.901555s 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcf9ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 msgr2=0x7fcfa8098ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 0x7fcfa8098ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60426/0 (socket says 172.21.15.67:60426) 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 -- 172.21.15.67:0/401538891 learned_addr learned my addr 172.21.15.67:0/401538891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): get_auth_request con 0x7fcfa8136ea0 auth_method 0 2024-09-15T01:53:43.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfa806e860 0x7fcfa80989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): get_auth_request con 0x7fcfa806e860 auth_method 0 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:43.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient(hunting): handle_auth_done global_id 4870 payload 824 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 -- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 msgr2=0x7fcfa8098ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 0x7fcfa8098ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 -- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfa806e860 msgr2=0x7fcfa80989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfa806e860 0x7fcfa80989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient: _finish_hunting 0 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 monclient: found mon.noname-b 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.432+0000 7fcfacc31640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0009ae0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfacc31640 1 -- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 msgr2=0x7fcfa8099430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfacc31640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f20 tx=0x7fcfa0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0009520 con 0x7fcfa8136ea0 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: dump: 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.449 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 5 monclient: authenticate success, global_id 4870 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa0032ce0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa003b660 con 0x7fcfa8136ea0 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa4ff9640 10 monclient: dump: 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.450 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfa6ffd640 4 set_mon_vals no callback set 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 10 monclient: _renew_subs 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa809b450 con 0x7fcfa8136ea0 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 10 monclient: _renew_subs 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.435+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa8133d70 con 0x7fcfa8136ea0 2024-09-15T01:53:43.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcfa0003030 con 0x7fcfa8136ea0 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfae694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfae694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfae694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa804bf60 con 0x7fcfa8136ea0 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfa4ff9640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 0x7fcf8803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fcfa006c0c0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfa7fff640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 0x7fcf8803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.436+0000 7fcfa7fff640 10 monclient: get_auth_request con 0x7fcf88038740 auth_method 0 2024-09-15T01:53:43.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.439+0000 7fcfa7fff640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 0x7fcf8803abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009a60 tx=0x7fcf98006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.440+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcfa003b9d0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.440+0000 7fcfa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.440+0000 7fcfa4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.606+0000 7fcfae694640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.606+0000 7fcfae694640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.606+0000 7fcfae694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.606+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fcfa8063e50 con 0x7fcfa8136ea0 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.607+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fcfa003f020 con 0x7fcfa8136ea0 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.607+0000 7fcfa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.607+0000 7fcfa4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:43.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfae694640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfae694640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfae694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fcfa806eec0 con 0x7fcfa8136ea0 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfa4ff9640 1 -- 172.21.15.67:0/401538891 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fcfa003f020 con 0x7fcfa8136ea0 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfa4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.707+0000 7fcfa4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 msgr2=0x7fcf8803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 0x7fcf8803abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009a60 tx=0x7fcf98006f50 comp rx=0 tx=0).stop 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 10 monclient: shutdown 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 msgr2=0x7fcfa8099430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f20 tx=0x7fcfa0036000 comp rx=0 tx=0).stop 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 shutdown_connections 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf88038740 0x7fcf8803abf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa8136ea0 0x7fcfa8099430 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfa80980d0 0x7fcfa8098ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 --2- 172.21.15.67:0/401538891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfa806e860 0x7fcfa80989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 >> 172.21.15.67:0/401538891 conn(0x7fcfa80a13a0 msgr2=0x7fcfa809ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 shutdown_connections 2024-09-15T01:53:43.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.709+0000 7fcfae694640 1 -- 172.21.15.67:0/401538891 wait complete. 2024-09-15T01:53:43.720 INFO:tasks.workunit.client.0.smithi067.stdout:foo-ssd 2024-09-15T01:53:43.722 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-15T01:53:43.722 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false grep foo-hdd 2024-09-15T01:53:43.722 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:43.722 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-hdd 2024-09-15T01:53:43.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.795+0000 7f3c0be20640 10 monclient: get_monmap_and_config 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.795+0000 7f3c0be20640 10 monclient: build_initial_monmap 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.795+0000 7f3c0be20640 10 monclient: monmap: 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:43.796691+0000 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:43.796691+0000 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:43.798 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:43.799 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:43.799 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:43.799 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 Processor -- start 2024-09-15T01:53:43.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 -- start start 2024-09-15T01:53:43.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient: init 2024-09-15T01:53:43.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:43.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c0406e210 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c0406e5e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c04138420 0x7f3c0413a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-c con 0x7f3c04138420 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0406aa60 con 0x7f3c0406e5e0 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c04069bd0 con 0x7f3c0406e210 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0406a190 con 0x7f3c04138420 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0be20640 10 monclient(hunting): authenticate will time out at 856.265812s 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c04138420 0x7f3c0413a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c04138420 0x7f3c0413a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60430/0 (socket says 172.21.15.67:60430) 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 1 -- 172.21.15.67:0/924909198 learned_addr learned my addr 172.21.15.67:0/924909198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 10 monclient(hunting): get_auth_request con 0x7f3c04138420 auth_method 0 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.796+0000 7f3c0a3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.797+0000 7f3c0a3bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.797+0000 7f3c0a3bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.797+0000 7f3c0a3bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c09bbc640 1 -- 172.21.15.67:0/924909198 >> v1:172.21.15.67:6789/0 conn(0x7f3c0406e210 legacy=0x7f3c0406b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c093bb640 1 -- 172.21.15.67:0/924909198 >> v1:172.21.15.17:6789/0 conn(0x7f3c0406e5e0 legacy=0x7f3c04136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 10 monclient(hunting): handle_auth_done global_id 4873 payload 274 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 1 -- 172.21.15.67:0/924909198 >> v1:172.21.15.67:6789/0 conn(0x7f3c0406e210 legacy=0x7f3c0406b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 1 -- 172.21.15.67:0/924909198 >> v1:172.21.15.17:6789/0 conn(0x7f3c0406e5e0 legacy=0x7f3c04136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 1 monclient: found mon.noname-c 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.799+0000 7f3c0a3bd640 1 -- 172.21.15.67:0/924909198 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040698f0 con 0x7f3c04138420 2024-09-15T01:53:43.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0a3bd640 1 -- 172.21.15.67:0/924909198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c04138420 msgr2=0x7f3c0413a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0a3bd640 1 --2- 172.21.15.67:0/924909198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c04138420 0x7f3c0413a800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00b0d0 tx=0x7f3bfc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b35bc4c1d1d41a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 1 -- 172.21.15.67:0/924909198 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc007610 con 0x7f3c04138420 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: dump: 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.806 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: _finish_auth 0 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 1 -- 172.21.15.67:0/924909198 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc007770 con 0x7f3c04138420 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 5 monclient: authenticate success, global_id 4873 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 1 -- 172.21.15.67:0/924909198 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc007b10 con 0x7f3c04138420 2024-09-15T01:53:43.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 4 set_mon_vals no callback set 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c08bba640 10 monclient: dump: 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.808 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 10 monclient: shutdown 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 -- 172.21.15.67:0/924909198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c04138420 msgr2=0x7f3c0413a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 --2- 172.21.15.67:0/924909198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c04138420 0x7f3c0413a800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00b0d0 tx=0x7f3bfc009840 comp rx=0 tx=0).stop 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 -- 172.21.15.67:0/924909198 shutdown_connections 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 --2- 172.21.15.67:0/924909198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c04138420 0x7f3c0413a800 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 -- 172.21.15.67:0/924909198 >> 172.21.15.67:0/924909198 conn(0x7f3c04070ff0 msgr2=0x7f3c040a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 -- 172.21.15.67:0/924909198 shutdown_connections 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.802+0000 7f3c0be20640 1 -- 172.21.15.67:0/924909198 wait complete. 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.803+0000 7f3c0be20640 10 monclient: build_initial_monmap 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.803+0000 7f3c0be20640 10 monclient: monmap: 2024-09-15T01:53:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.803+0000 7f3c0be20640 1 Processor -- start 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.803+0000 7f3c0be20640 1 -- start start 2024-09-15T01:53:43.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.803+0000 7f3c0be20640 10 monclient: init 2024-09-15T01:53:43.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:43.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 0x7f3c0413de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0406e210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0406e5e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c04138420 0x7f3c041412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): picked mon.noname-a con 0x7f3c04138420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0406d860 con 0x7f3c04138420 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0411e0d0 con 0x7f3c0406e5e0 2024-09-15T01:53:43.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0411fb90 con 0x7f3c0406e210 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0be20640 10 monclient(hunting): authenticate will time out at 856.273276s 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c09bbc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 msgr2=0x7f3c0413de60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c09bbc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 0x7f3c0413de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60436/0 (socket says 172.21.15.67:60436) 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 -- 172.21.15.67:0/1622406468 learned_addr learned my addr 172.21.15.67:0/1622406468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): get_auth_request con 0x7f3c0406e5e0 auth_method 0 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0a3bd640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c04138420 0x7f3c041412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0a3bd640 10 monclient(hunting): get_auth_request con 0x7f3c04138420 auth_method 0 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0a3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0a3bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c0a3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient(hunting): handle_auth_done global_id 4876 payload 824 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 -- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 msgr2=0x7f3c0413de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 0x7f3c0413de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 -- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c04138420 msgr2=0x7f3c041412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:43.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c04138420 0x7f3c041412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient: _finish_hunting 0 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 monclient: found mon.noname-b 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.804+0000 7f3c093bb640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009ae0 con 0x7f3c0406e5e0 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.806+0000 7f3c093bb640 1 -- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 msgr2=0x7f3c04137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.806+0000 7f3c093bb640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c570 tx=0x7f3bf4007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4007510 con 0x7f3c0406e5e0 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: dump: 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.818 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4005780 con 0x7f3c0406e5e0 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 5 monclient: authenticate success, global_id 4876 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 10 monclient: _renew_subs 2024-09-15T01:53:43.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0411fe70 con 0x7f3c0406e5e0 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4005b40 con 0x7f3c0406e5e0 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bf9ffb640 10 monclient: dump: 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:43.820 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 10 monclient: _renew_subs 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3bfbfff640 4 set_mon_vals no callback set 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.807+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c04138000 con 0x7f3c0406e5e0 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3bf401b400 con 0x7f3c0406e5e0 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3bf9ffb640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 0x7f3be803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c0be20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c0be20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c0be20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0404bf60 con 0x7f3c0406e5e0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f3bf404b540 con 0x7f3c0406e5e0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c09bbc640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 0x7f3be803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.808+0000 7f3c09bbc640 10 monclient: get_auth_request con 0x7f3be8038740 auth_method 0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.811+0000 7f3c09bbc640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 0x7f3be803abf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c04069d60 tx=0x7f3c00007a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.813+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3bf401b6b0 con 0x7f3c0406e5e0 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.813+0000 7f3bf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:43.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.813+0000 7f3bf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:43.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3c0be20640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3c0be20640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3c0be20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:43.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3c04068a20 con 0x7f3c0406e5e0 2024-09-15T01:53:43.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f3bf401a360 con 0x7f3c0406e5e0 2024-09-15T01:53:43.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3bf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:43.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:43.978+0000 7f3bf9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.075+0000 7f3c0be20640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.075+0000 7f3c0be20640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.075+0000 7f3c0be20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.075+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3c0409f940 con 0x7f3c0406e5e0 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.076+0000 7f3bf9ffb640 1 -- 172.21.15.67:0/1622406468 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f3bf401a540 con 0x7f3c0406e5e0 2024-09-15T01:53:44.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.076+0000 7f3bf9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T01:53:44.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.076+0000 7f3bf9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:44.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 msgr2=0x7f3be803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 0x7f3be803abf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c04069d60 tx=0x7f3c00007a30 comp rx=0 tx=0).stop 2024-09-15T01:53:44.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 10 monclient: shutdown 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 msgr2=0x7f3c04137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.077+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c570 tx=0x7f3bf4007d30 comp rx=0 tx=0).stop 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 shutdown_connections 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3be8038740 0x7f3be803abf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c04138420 0x7f3c041412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0406e5e0 0x7f3c04137ac0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 --2- 172.21.15.67:0/1622406468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c0406e210 0x7f3c0413de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 >> 172.21.15.67:0/1622406468 conn(0x7f3c04070ff0 msgr2=0x7f3c041394c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 shutdown_connections 2024-09-15T01:53:44.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.078+0000 7f3c0be20640 1 -- 172.21.15.67:0/1622406468 wait complete. 2024-09-15T01:53:44.089 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:44.089 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-15T01:53:44.090 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-hdd 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 10 monclient: get_monmap_and_config 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 10 monclient: monmap: 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.166 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:44.165246+0000 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:44.165246+0000 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:44.167 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 1 Processor -- start 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 1 -- start start 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.164+0000 7f36dbadd640 10 monclient: init 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36d4068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f36d4069190 0x7f36d4069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-a con 0x7f36d4069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 0x7f36d4139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-b con 0x7f36d4137150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d4096c10 con 0x7f36d4068dc0 2024-09-15T01:53:44.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d40a14d0 con 0x7f36d4069190 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4132960 con 0x7f36d4137150 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36dbadd640 10 monclient(hunting): authenticate will time out at 856.634516s 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9879640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f36d4068dc0 0x7f36d4132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52796/0 (socket says 172.21.15.67:52796) 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9879640 1 -- 172.21.15.67:0/2339418494 learned_addr learned my addr 172.21.15.67:0/2339418494 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36da07a640 1 -- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 msgr2=0x7f36d4139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36da07a640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 0x7f36d4139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 1 --2- 172.21.15.67:0/2339418494 >> v2:172.21.15.17:3300/0 conn(0x7f36d4069190 0x7f36d4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): get_auth_request con 0x7f36d4069190 auth_method 0 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.165+0000 7f36d9078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 10 monclient(hunting): handle_auth_done global_id 44319 payload 274 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 1 -- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 msgr2=0x7f36d4139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 0x7f36d4139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 1 -- 172.21.15.67:0/2339418494 >> v1:172.21.15.67:6789/0 conn(0x7f36d4068dc0 legacy=0x7f36d4132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 1 monclient: found mon.noname-a 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:44.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.166+0000 7f36d9078640 1 -- 172.21.15.67:0/2339418494 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d409ff20 con 0x7f36d4069190 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d9078640 1 -- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 msgr2=0x7f36d4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d9078640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4069590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009850 tx=0x7f36d00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b76f5c0303a631 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 1 -- 172.21.15.67:0/2339418494 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0032440 con 0x7f36d4069190 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: dump: 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 1 -- 172.21.15.67:0/2339418494 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d00325a0 con 0x7f36d4069190 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 1 -- 172.21.15.67:0/2339418494 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0032940 con 0x7f36d4069190 2024-09-15T01:53:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 5 monclient: authenticate success, global_id 44319 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 4 set_mon_vals no callback set 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36d8877640 10 monclient: dump: 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.180 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 10 monclient: shutdown 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 -- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 msgr2=0x7f36d4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4069590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009850 tx=0x7f36d00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 -- 172.21.15.67:0/2339418494 shutdown_connections 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4137150 0x7f36d4139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 --2- 172.21.15.67:0/2339418494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4069590 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.169+0000 7f36dbadd640 1 -- 172.21.15.67:0/2339418494 >> 172.21.15.67:0/2339418494 conn(0x7f36d40711b0 msgr2=0x7f36d40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 1 -- 172.21.15.67:0/2339418494 shutdown_connections 2024-09-15T01:53:44.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 1 -- 172.21.15.67:0/2339418494 wait complete. 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 10 monclient: monmap: 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:44.182 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 1 Processor -- start 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 1 -- start start 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.170+0000 7f36dbadd640 10 monclient: init 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 0x7f36d41415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-c con 0x7f36d4068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-a con 0x7f36d4069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): picked mon.noname-b con 0x7f36d4137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4099ee0 con 0x7f36d4069190 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40a0890 con 0x7f36d4137150 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d412fb60 con 0x7f36d4068dc0 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36dbadd640 10 monclient(hunting): authenticate will time out at 856.640433s 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9879640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 msgr2=0x7f36d41415c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60448/0 (socket says 172.21.15.67:60448) 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 -- 172.21.15.67:0/1376694743 learned_addr learned my addr 172.21.15.67:0/1376694743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): get_auth_request con 0x7f36d4137150 auth_method 0 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9879640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 0x7f36d41415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9078640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9078640 10 monclient(hunting): get_auth_request con 0x7f36d4069190 auth_method 0 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9078640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36d9078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient(hunting): handle_auth_done global_id 4879 payload 824 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 -- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 msgr2=0x7f36d41415c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 0x7f36d41415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 -- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 msgr2=0x7f36d4141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 monclient: found mon.noname-b 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.171+0000 7f36da07a640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d0009390 con 0x7f36d4137150 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36da07a640 1 -- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 msgr2=0x7f36d4142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36da07a640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00bb30 tx=0x7f36cc0097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00cbc0 con 0x7f36d4137150 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: dump: 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.187 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00cd20 con 0x7f36d4137150 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 5 monclient: authenticate success, global_id 4879 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 10 monclient: _renew_subs 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d409a530 con 0x7f36d4137150 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc006150 con 0x7f36d4137150 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36c9ffb640 10 monclient: dump: 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.189 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 10 monclient: _renew_subs 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36cbfff640 4 set_mon_vals no callback set 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.174+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d4099960 con 0x7f36d4137150 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f36cc012020 con 0x7f36d4137150 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36dbadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36dbadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36dbadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d404bf60 con 0x7f36d4137150 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36c9ffb640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 0x7f36b803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f36cc04b9b0 con 0x7f36d4137150 2024-09-15T01:53:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36d9879640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 0x7f36b803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.175+0000 7f36d9879640 10 monclient: get_auth_request con 0x7f36b80386f0 auth_method 0 2024-09-15T01:53:44.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.178+0000 7f36d9879640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 0x7f36b803aba0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f36c40047d0 tx=0x7f36c4007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.180+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36cc0068f0 con 0x7f36d4137150 2024-09-15T01:53:44.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.180+0000 7f36c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.180+0000 7f36c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.346+0000 7f36dbadd640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.346+0000 7f36dbadd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.346+0000 7f36dbadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.346+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f36d406f000 con 0x7f36d4137150 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.347+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f36cc01d300 con 0x7f36d4137150 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.347+0000 7f36c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.347+0000 7f36c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.446+0000 7f36dbadd640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.446+0000 7f36dbadd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.446+0000 7f36dbadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.446+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f36d4097d50 con 0x7f36d4137150 2024-09-15T01:53:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.446+0000 7f36c9ffb640 1 -- 172.21.15.67:0/1376694743 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f36cc01d4e0 con 0x7f36d4137150 2024-09-15T01:53:44.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.447+0000 7f36c9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.447+0000 7f36c9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 msgr2=0x7f36b803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 0x7f36b803aba0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f36c40047d0 tx=0x7f36c4007b80 comp rx=0 tx=0).stop 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 10 monclient: shutdown 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 msgr2=0x7f36d4142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00bb30 tx=0x7f36cc0097a0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 shutdown_connections 2024-09-15T01:53:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36b80386f0 0x7f36b803aba0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36d4069190 0x7f36d4141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36d4137150 0x7f36d4142040 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 --2- 172.21.15.67:0/1376694743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36d4068dc0 0x7f36d41415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 >> 172.21.15.67:0/1376694743 conn(0x7f36d40711b0 msgr2=0x7f36d40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 shutdown_connections 2024-09-15T01:53:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.448+0000 7f36dbadd640 1 -- 172.21.15.67:0/1376694743 wait complete. 2024-09-15T01:53:44.460 INFO:tasks.workunit.client.0.smithi067.stdout:foo-hdd 2024-09-15T01:53:44.460 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-15T01:53:44.460 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false grep foo-ssd 2024-09-15T01:53:44.460 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:44.461 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-ssd 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.534+0000 7fc43331d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.534+0000 7fc43331d640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.534+0000 7fc43331d640 10 monclient: monmap: 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:44.535888+0000 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:44.535888+0000 2024-09-15T01:53:44.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:44.538 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 1 Processor -- start 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 1 -- start start 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 10 monclient: init 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:53:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc42c0a2270 0x7fc42c09c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-c con 0x7fc42c0a2270 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.535+0000 7fc43331d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc42c06d990 0x7fc42c09c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-a con 0x7fc42c06d990 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 0x7fc42c136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-b con 0x7fc42c09cdd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c06cfb0 con 0x7fc42c06d990 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c0a2640 con 0x7fc42c0a2270 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c06c9f0 con 0x7fc42c09cdd0 2024-09-15T01:53:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc43331d640 10 monclient(hunting): authenticate will time out at 857.005053s 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4318ba640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 msgr2=0x7fc42c136170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc42c0a2270 0x7fc42c09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4318ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 0x7fc42c136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc42c0a2270 0x7fc42c09c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60464/0 (socket says 172.21.15.67:60464) 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 -- 172.21.15.67:0/723916367 learned_addr learned my addr 172.21.15.67:0/723916367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): get_auth_request con 0x7fc42c0a2270 auth_method 0 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 1 --2- 172.21.15.67:0/723916367 >> v2:172.21.15.17:3300/0 conn(0x7fc42c06d990 0x7fc42c09c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): get_auth_request con 0x7fc42c06d990 auth_method 0 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4308b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient(hunting): handle_auth_done global_id 4882 payload 274 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 -- 172.21.15.67:0/723916367 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 msgr2=0x7fc42c136170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 --2- 172.21.15.67:0/723916367 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 0x7fc42c136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 -- 172.21.15.67:0/723916367 >> v2:172.21.15.17:3300/0 conn(0x7fc42c06d990 msgr2=0x7fc42c09c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 --2- 172.21.15.67:0/723916367 >> v2:172.21.15.17:3300/0 conn(0x7fc42c06d990 0x7fc42c09c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 monclient: found mon.noname-c 2024-09-15T01:53:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.536+0000 7fc4310b9640 1 -- 172.21.15.67:0/723916367 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c06ccd0 con 0x7fc42c0a2270 2024-09-15T01:53:44.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc4310b9640 1 -- 172.21.15.67:0/723916367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 msgr2=0x7fc42c09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc4310b9640 1 --2- 172.21.15.67:0/723916367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c09c350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc42c06f1b0 tx=0x7fc4280091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c94593e848cfd801 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 1 -- 172.21.15.67:0/723916367 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428032440 con 0x7fc42c0a2270 2024-09-15T01:53:44.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: dump: 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 1 -- 172.21.15.67:0/723916367 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4280325a0 con 0x7fc42c0a2270 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 1 -- 172.21.15.67:0/723916367 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428032940 con 0x7fc42c0a2270 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc43331d640 5 monclient: authenticate success, global_id 4882 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc43331d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc43331d640 4 set_mon_vals no callback set 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.539+0000 7fc423fff640 10 monclient: dump: 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.552 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 10 monclient: shutdown 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 -- 172.21.15.67:0/723916367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 msgr2=0x7fc42c09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 --2- 172.21.15.67:0/723916367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c09c350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc42c06f1b0 tx=0x7fc4280091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 -- 172.21.15.67:0/723916367 shutdown_connections 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 --2- 172.21.15.67:0/723916367 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c09cdd0 0x7fc42c136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 --2- 172.21.15.67:0/723916367 >> v2:172.21.15.17:3300/0 conn(0x7fc42c06d990 0x7fc42c09c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 --2- 172.21.15.67:0/723916367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c09c350 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 -- 172.21.15.67:0/723916367 >> 172.21.15.67:0/723916367 conn(0x7fc42c067ae0 msgr2=0x7fc42c0a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 -- 172.21.15.67:0/723916367 shutdown_connections 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 1 -- 172.21.15.67:0/723916367 wait complete. 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.540+0000 7fc43331d640 10 monclient: monmap: 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 Processor -- start 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 -- start start 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient: init 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:44.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 0x7fc42c13ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-c con 0x7fc42c06d990 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 0x7fc42c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-a con 0x7fc42c09cdd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): picked mon.noname-b con 0x7fc42c0a2270 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c120420 con 0x7fc42c09cdd0 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c120140 con 0x7fc42c0a2270 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c120700 con 0x7fc42c06d990 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc43331d640 10 monclient(hunting): authenticate will time out at 857.010886s 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc4310b9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 msgr2=0x7fc42c13ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc4310b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 0x7fc42c13ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.541+0000 7fc4318ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60466/0 (socket says 172.21.15.67:60466) 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 -- 172.21.15.67:0/2042828848 learned_addr learned my addr 172.21.15.67:0/2042828848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): get_auth_request con 0x7fc42c0a2270 auth_method 0 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4308b8640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 0x7fc42c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient(hunting): handle_auth_done global_id 4885 payload 824 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 -- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 msgr2=0x7fc42c13ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 0x7fc42c13ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 -- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 msgr2=0x7fc42c13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 0x7fc42c13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 monclient: found mon.noname-b 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4318ba640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428009450 con 0x7fc42c0a2270 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4308b8640 10 monclient: get_auth_request con 0x7fc42c09cdd0 auth_method 0 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.542+0000 7fc4308b8640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 0x7fc42c13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4318ba640 1 -- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 msgr2=0x7fc42c13b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4318ba640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc42400bb10 tx=0x7fc4240097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42400cbc0 con 0x7fc42c0a2270 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: dump: 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.559 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42400cd40 con 0x7fc42c0a2270 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424006150 con 0x7fc42c0a2270 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 5 monclient: authenticate success, global_id 4885 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 10 monclient: _renew_subs 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c11df00 con 0x7fc42c0a2270 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4237fe640 4 set_mon_vals no callback set 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc4217fa640 10 monclient: dump: 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.561 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 10 monclient: _renew_subs 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.544+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c13b760 con 0x7fc42c0a2270 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc424012020 con 0x7fc42c0a2270 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc4217fa640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 0x7fc41003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fc42404bae0 con 0x7fc42c0a2270 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc43331d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc43331d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc43331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42c04bf60 con 0x7fc42c0a2270 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc4310b9640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 0x7fc41003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.545+0000 7fc4310b9640 10 monclient: get_auth_request con 0x7fc4100386f0 auth_method 0 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.548+0000 7fc4310b9640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 0x7fc41003aba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42c06f760 tx=0x7fc428032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.550+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc42401d4e0 con 0x7fc42c0a2270 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.550+0000 7fc4217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.550+0000 7fc4217fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:44.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.716+0000 7fc43331d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.716+0000 7fc43331d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.716+0000 7fc43331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.716+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fc42c071330 con 0x7fc42c0a2270 2024-09-15T01:53:44.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.717+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fc424024030 con 0x7fc42c0a2270 2024-09-15T01:53:44.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.717+0000 7fc4217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.717+0000 7fc4217fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.818+0000 7fc43331d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.818+0000 7fc43331d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.818+0000 7fc43331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.818+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fc42c13c6f0 con 0x7fc42c0a2270 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.819+0000 7fc4217fa640 1 -- 172.21.15.67:0/2042828848 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fc424024030 con 0x7fc42c0a2270 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.819+0000 7fc4217fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T01:53:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.819+0000 7fc4217fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 msgr2=0x7fc41003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 0x7fc41003aba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42c06f760 tx=0x7fc428032000 comp rx=0 tx=0).stop 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 10 monclient: shutdown 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 msgr2=0x7fc42c13b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc42400bb10 tx=0x7fc4240097d0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 shutdown_connections 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4100386f0 0x7fc41003aba0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc42c09cdd0 0x7fc42c13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc42c0a2270 0x7fc42c13b9f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 --2- 172.21.15.67:0/2042828848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc42c06d990 0x7fc42c13ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.820+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 >> 172.21.15.67:0/2042828848 conn(0x7fc42c067ae0 msgr2=0x7fc42c099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.821+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 shutdown_connections 2024-09-15T01:53:44.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.821+0000 7fc43331d640 1 -- 172.21.15.67:0/2042828848 wait complete. 2024-09-15T01:53:44.832 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:44.832 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-15T01:53:44.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 10 monclient: get_monmap_and_config 2024-09-15T01:53:44.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 10 monclient: monmap: 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:44.911259+0000 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:44.911259+0000 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:44.913 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:44.916 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 1 Processor -- start 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 1 -- start start 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.910+0000 7f0811c60640 10 monclient: init 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f080c098140 0x7f080c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-c con 0x7f080c098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 0x7f080c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-b con 0x7f080c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f080c098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c098ef0 con 0x7f080c098a50 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c0a1730 con 0x7f080c098140 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0a0b80 con 0x7f080c0a0370 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f0811c60640 10 monclient(hunting): authenticate will time out at 857.380514s 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f080c098140 0x7f080c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080affd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 msgr2=0x7f080c098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f080c098140 0x7f080c1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60480/0 (socket says 172.21.15.67:60480) 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 -- 172.21.15.67:0/3208663354 learned_addr learned my addr 172.21.15.67:0/3208663354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:44.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): get_auth_request con 0x7f080c098140 auth_method 0 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080affd640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 0x7f080c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient(hunting): handle_auth_done global_id 4888 payload 274 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 -- 172.21.15.67:0/3208663354 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 msgr2=0x7f080c098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 0x7f080c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 -- 172.21.15.67:0/3208663354 >> v1:172.21.15.17:6789/0 conn(0x7f080c098a50 legacy=0x7f080c0695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.911+0000 7f080b7fe640 1 -- 172.21.15.67:0/3208663354 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c0a00b0 con 0x7f080c098140 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080b7fe640 1 -- 172.21.15.67:0/3208663354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098140 msgr2=0x7f080c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080b7fe640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098140 0x7f080c1321e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f080c0a5340 tx=0x7f07fc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51ea2d9e030a644b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 1 -- 172.21.15.67:0/3208663354 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc032440 con 0x7f080c098140 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: dump: 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.924 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 1 -- 172.21.15.67:0/3208663354 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc0325a0 con 0x7f080c098140 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 1 -- 172.21.15.67:0/3208663354 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc032940 con 0x7f080c098140 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f0811c60640 5 monclient: authenticate success, global_id 4888 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f0811c60640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f0811c60640 4 set_mon_vals no callback set 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.914+0000 7f080a7fc640 10 monclient: dump: 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.926 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 10 monclient: shutdown 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 -- 172.21.15.67:0/3208663354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098140 msgr2=0x7f080c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098140 0x7f080c1321e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f080c0a5340 tx=0x7f07fc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 -- 172.21.15.67:0/3208663354 shutdown_connections 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c0a0370 0x7f080c098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 --2- 172.21.15.67:0/3208663354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098140 0x7f080c1321e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 -- 172.21.15.67:0/3208663354 >> 172.21.15.67:0/3208663354 conn(0x7f080c0a5780 msgr2=0x7f080c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 -- 172.21.15.67:0/3208663354 shutdown_connections 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 1 -- 172.21.15.67:0/3208663354 wait complete. 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.915+0000 7f0811c60640 10 monclient: build_initial_monmap 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 10 monclient: monmap: 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:44.927 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 1 Processor -- start 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 1 -- start start 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 10 monclient: init 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.916+0000 7f0811c60640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 0x7f080c09e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-c con 0x7f080c098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-b con 0x7f080c098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f080c0a0370 0x7f080c141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): picked mon.noname-a con 0x7f080c0a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c131c40 con 0x7f080c0a0370 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c11cbd0 con 0x7f080c098a50 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:44.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c11dee0 con 0x7f080c098140 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f0811c60640 10 monclient(hunting): authenticate will time out at 857.386089s 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080b7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 msgr2=0x7f080c09e670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 0x7f080c09e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60492/0 (socket says 172.21.15.67:60492) 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 -- 172.21.15.67:0/2946281803 learned_addr learned my addr 172.21.15.67:0/2946281803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): get_auth_request con 0x7f080c098a50 auth_method 0 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:44.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080bfff640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f080c0a0370 0x7f080c141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080bfff640 10 monclient(hunting): get_auth_request con 0x7f080c0a0370 auth_method 0 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient(hunting): handle_auth_done global_id 4891 payload 824 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 -- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 msgr2=0x7f080c09e670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 0x7f080c09e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 -- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f080c0a0370 msgr2=0x7f080c141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f080c0a0370 0x7f080c141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 monclient: found mon.noname-b 2024-09-15T01:53:44.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:44.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.917+0000 7f080affd640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc009450 con 0x7f080c098a50 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.919+0000 7f080affd640 1 -- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 msgr2=0x7f080c130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.919+0000 7f080affd640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f080000b5e0 tx=0x7f0800006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.919+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800003d80 con 0x7f080c098a50 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: dump: 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800006150 con 0x7f080c098a50 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 5 monclient: authenticate success, global_id 4891 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 10 monclient: _renew_subs 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c11e1c0 con 0x7f080c098a50 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:44.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800006510 con 0x7f080c098a50 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 10 monclient: dump: 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:44.936 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0809ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 10 monclient: _renew_subs 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c11c0b0 con 0x7f080c098a50 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.920+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f08000103e0 con 0x7f080c098a50 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f0811c60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f0811c60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f0811c60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f07f37fe640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 0x7f07f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c04bf60 con 0x7f080c098a50 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f080004b350 con 0x7f080c098a50 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f080b7fe640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 0x7f07f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.921+0000 7f080b7fe640 10 monclient: get_auth_request con 0x7f07f80386f0 auth_method 0 2024-09-15T01:53:44.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.923+0000 7f080b7fe640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 0x7f07f803aba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f07fc031000 tx=0x7f07fc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:44.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.925+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f080000ed00 con 0x7f080c098a50 2024-09-15T01:53:44.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.925+0000 7f07f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:44.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:44.925+0000 7f07f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.099+0000 7f0811c60640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.099+0000 7f0811c60640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.099+0000 7f0811c60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.099+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f080c06f000 con 0x7f080c098a50 2024-09-15T01:53:45.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.235+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (secure 0 0 0) 0x7f08000170d0 con 0x7f080c098a50 2024-09-15T01:53:45.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.235+0000 7f07f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.235+0000 7f07f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:45.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.344+0000 7f0811c60640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.344+0000 7f0811c60640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.344+0000 7f0811c60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.344+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f080c11cff0 con 0x7f080c098a50 2024-09-15T01:53:45.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.346+0000 7f07f37fe640 1 -- 172.21.15.67:0/2946281803 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (secure 0 0 0) 0x7f080000e700 con 0x7f080c098a50 2024-09-15T01:53:45.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.346+0000 7f07f37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T01:53:45.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.346+0000 7f07f37fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:45.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 msgr2=0x7f07f803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:45.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 0x7f07f803aba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f07fc031000 tx=0x7f07fc032000 comp rx=0 tx=0).stop 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 10 monclient: shutdown 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 msgr2=0x7f080c130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f080000b5e0 tx=0x7f0800006f90 comp rx=0 tx=0).stop 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 shutdown_connections 2024-09-15T01:53:45.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07f80386f0 0x7f07f803aba0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f080c0a0370 0x7f080c141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f080c098a50 0x7f080c130dc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 --2- 172.21.15.67:0/2946281803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f080c098140 0x7f080c09e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 >> 172.21.15.67:0/2946281803 conn(0x7f080c0a5780 msgr2=0x7f080c09b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:45.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 shutdown_connections 2024-09-15T01:53:45.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.348+0000 7f0811c60640 1 -- 172.21.15.67:0/2946281803 wait complete. 2024-09-15T01:53:45.359 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-15T01:53:45.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.436+0000 7f9ed2ac9640 10 monclient: get_monmap_and_config 2024-09-15T01:53:45.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.436+0000 7f9ed2ac9640 10 monclient: build_initial_monmap 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.436+0000 7f9ed2ac9640 10 monclient: monmap: 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:45.437608+0000 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:45.437608+0000 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:45.439 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 Processor -- start 2024-09-15T01:53:45.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 -- start start 2024-09-15T01:53:45.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient: init 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ecc0a0b50 0x7f9ecc069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ecc0a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 0x7f9ecc09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ecc09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ecc131b10 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc06a270 con 0x7f9ecc131b10 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc06f290 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc09f7e0 con 0x7f9ecc09eea0 2024-09-15T01:53:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed2ac9640 10 monclient(hunting): authenticate will time out at 857.906861s 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ecbfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 msgr2=0x7f9ecc09f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ecc0a0b50 0x7f9ecc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ecbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 0x7f9ecc09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ecc0a0b50 0x7f9ecc069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60502/0 (socket says 172.21.15.67:60502) 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 1 -- 172.21.15.67:0/2047067413 learned_addr learned my addr 172.21.15.67:0/2047067413 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 10 monclient(hunting): get_auth_request con 0x7f9ecc0a0b50 auth_method 0 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.437+0000 7f9ed0865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.438+0000 7f9ed0865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.438+0000 7f9ed0865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:45.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.438+0000 7f9ed0865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:45.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed1066640 1 -- 172.21.15.67:0/2047067413 >> v1:172.21.15.67:6789/0 conn(0x7f9ecc131b10 legacy=0x7f9ecc131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:45.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 10 monclient(hunting): handle_auth_done global_id 4894 payload 274 2024-09-15T01:53:45.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 1 -- 172.21.15.67:0/2047067413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 msgr2=0x7f9ecc09f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:45.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 1 --2- 172.21.15.67:0/2047067413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 0x7f9ecc09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 1 -- 172.21.15.67:0/2047067413 >> v1:172.21.15.67:6789/0 conn(0x7f9ecc131b10 legacy=0x7f9ecc131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 10 monclient: _finish_hunting 0 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 1 monclient: found mon.noname-c 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.440+0000 7f9ed0865640 1 -- 172.21.15.67:0/2047067413 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0a0f20 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed0865640 1 -- 172.21.15.67:0/2047067413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 msgr2=0x7f9ecc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed0865640 1 --2- 172.21.15.67:0/2047067413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc069340 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009990 tx=0x7f9ec00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47a25f2acbddc81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 1 -- 172.21.15.67:0/2047067413 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec002f4b0 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: dump: 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 1 -- 172.21.15.67:0/2047067413 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec002f610 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 5 monclient: authenticate success, global_id 4894 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 1 -- 172.21.15.67:0/2047067413 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec002f9b0 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 4 set_mon_vals no callback set 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ecb7fe640 10 monclient: dump: 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 10 monclient: shutdown 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2047067413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 msgr2=0x7f9ecc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2047067413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc069340 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009990 tx=0x7f9ec00091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2047067413 shutdown_connections 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2047067413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc09eea0 0x7f9ecc09f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2047067413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc069340 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2047067413 >> 172.21.15.67:0/2047067413 conn(0x7f9ecc071290 msgr2=0x7f9ecc0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2047067413 shutdown_connections 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.443+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2047067413 wait complete. 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.444+0000 7f9ed2ac9640 10 monclient: build_initial_monmap 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.444+0000 7f9ed2ac9640 10 monclient: monmap: 2024-09-15T01:53:45.452 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.444+0000 7f9ed2ac9640 1 Processor -- start 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 -- start start 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient: init 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ecc09eea0 0x7f9ecc141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f9ecc09eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ecc0a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 0x7f9ecc130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ecc131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc0a1260 con 0x7f9ecc09eea0 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc11db40 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc11d5f0 con 0x7f9ecc131b10 2024-09-15T01:53:45.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed2ac9640 10 monclient(hunting): authenticate will time out at 857.914538s 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60506/0 (socket says 172.21.15.67:60506) 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 -- 172.21.15.67:0/2876167393 learned_addr learned my addr 172.21.15.67:0/2876167393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): get_auth_request con 0x7f9ecc0a0b50 auth_method 0 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed1066640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 msgr2=0x7f9ecc130f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ed1066640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 0x7f9ecc130f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:45.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient(hunting): handle_auth_done global_id 4897 payload 824 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 msgr2=0x7f9ecc130f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 0x7f9ecc130f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ecc09eea0 msgr2=0x7f9ecc141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ecc09eea0 0x7f9ecc141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 monclient: found mon.noname-b 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.445+0000 7f9ecbfff640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec0009450 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ecbfff640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 msgr2=0x7f9ecc1417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:45.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ecbfff640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b600 tx=0x7f9eb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:45.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8015070 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: dump: 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:45.459 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb8003d90 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 5 monclient: authenticate success, global_id 4897 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8006380 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 10 monclient: _renew_subs 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc11f0b0 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ec8ff9640 10 monclient: dump: 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:45.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9eca7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 10 monclient: _renew_subs 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.448+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc1314c0 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9eb8010440 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed2ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed2ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed2ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc04bf60 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ec8ff9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 0x7f9eb003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f9eb804b750 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed0865640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 0x7f9eb003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.449+0000 7f9ed0865640 10 monclient: get_auth_request con 0x7f9eb00386f0 auth_method 0 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.452+0000 7f9ed0865640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 0x7f9eb003aba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0031000 tx=0x7f9ec0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:45.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.453+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9eb8018910 con 0x7f9ecc0a0b50 2024-09-15T01:53:45.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.453+0000 7f9ec8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:45.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.453+0000 7f9ec8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.616+0000 7f9ed2ac9640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.616+0000 7f9ed2ac9640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.616+0000 7f9ed2ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:45.616+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f9ecc11f700 con 0x7f9ecc0a0b50 2024-09-15T01:53:47.077 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:53:47.078 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T01:53:47.078 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:53:47.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:53:47.081 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:53:47.081 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:53:47.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:47.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.188+0000 7f3d07817640 10 monclient: get_monmap_and_config 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.188+0000 7f3d07817640 10 monclient: build_initial_monmap 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.188+0000 7f3d07817640 10 monclient: monmap: 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:47.189594+0000 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:47.189594+0000 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:47.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.189+0000 7f3d07817640 1 Processor -- start 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.189+0000 7f3d07817640 1 -- start start 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.189+0000 7f3d07817640 10 monclient: init 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3d000a58b0 0x7f3d00132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-a con 0x7f3d000a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d000980a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:47.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-b con 0x7f3d00136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000a01e0 con 0x7f3d000980a0 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d000a1580 con 0x7f3d000a58b0 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000a5c80 con 0x7f3d00136dd0 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d07817640 10 monclient(hunting): authenticate will time out at 887.186564s 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3d000a58b0 0x7f3d00132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d04db2640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3d000980a0 0x7f3d000984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56256/0 (socket says 172.21.15.17:56256) 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d04db2640 1 -- 172.21.15.17:0/2782658041 learned_addr learned my addr 172.21.15.17:0/2782658041 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:47.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): get_auth_request con 0x7f3d000a58b0 auth_method 0 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d05db4640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d05db4640 10 monclient(hunting): get_auth_request con 0x7f3d00136dd0 auth_method 0 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d05db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d05db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d05db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.190+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d05db4640 1 -- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d05db4640 1 -- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d05db4640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d05db4640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 10 monclient(hunting): handle_auth_done global_id 44331 payload 274 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 1 -- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 1 -- 172.21.15.17:0/2782658041 >> v1:172.21.15.17:6789/0 conn(0x7f3d000980a0 legacy=0x7f3d000984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 10 monclient: _finish_hunting 0 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 1 monclient: found mon.noname-a 2024-09-15T01:53:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:47.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.191+0000 7f3d055b3640 1 -- 172.21.15.17:0/2782658041 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00098bb0 con 0x7f3d000a58b0 2024-09-15T01:53:47.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.190+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v53) v1 ==== 177+0+0 (secure 0 0 0) 0x7f9eb8018310 con 0x7f9ecc0a0b50 2024-09-15T01:53:47.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.190+0000 7f9ec8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:47.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.190+0000 7f9ec8ff9640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-15T01:53:47.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.194+0000 7f3d055b3640 1 -- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 msgr2=0x7f3d00132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:47.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.194+0000 7f3d055b3640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00132040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009850 tx=0x7f3cfc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62cbaab2c76d232 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 1 -- 172.21.15.17:0/2782658041 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc02ae80 con 0x7f3d000a58b0 2024-09-15T01:53:47.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: dump: 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.204 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 1 -- 172.21.15.17:0/2782658041 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc032440 con 0x7f3d000a58b0 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 5 monclient: authenticate success, global_id 44331 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 1 -- 172.21.15.17:0/2782658041 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0327c0 con 0x7f3d000a58b0 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 4 set_mon_vals no callback set 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3cf7fff640 10 monclient: dump: 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 10 monclient: shutdown 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:47.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 1 -- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 msgr2=0x7f3d00132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.195+0000 7f3d07817640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00132040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009850 tx=0x7f3cfc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 -- 172.21.15.17:0/2782658041 shutdown_connections 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 --2- 172.21.15.17:0/2782658041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00132040 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 -- 172.21.15.17:0/2782658041 >> 172.21.15.17:0/2782658041 conn(0x7f3d00070ed0 msgr2=0x7f3d000a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 -- 172.21.15.17:0/2782658041 shutdown_connections 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 1 -- 172.21.15.17:0/2782658041 wait complete. 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 10 monclient: build_initial_monmap 2024-09-15T01:53:47.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.196+0000 7f3d07817640 10 monclient: monmap: 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:47.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 Processor -- start 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 -- start start 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient: init 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d000980a0 0x7f3d00132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-b con 0x7f3d000980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-a con 0x7f3d000a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): picked mon.noname-c con 0x7f3d00136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001369c0 con 0x7f3d000a58b0 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00134170 con 0x7f3d000980a0 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00098dc0 con 0x7f3d00136dd0 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.197+0000 7f3d07817640 10 monclient(hunting): authenticate will time out at 887.193940s 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51926/0 (socket says 172.21.15.17:51926) 2024-09-15T01:53:47.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 -- 172.21.15.17:0/2254708520 learned_addr learned my addr 172.21.15.17:0/2254708520 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): get_auth_request con 0x7f3d000a58b0 auth_method 0 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 10 monclient(hunting): get_auth_request con 0x7f3d00136dd0 auth_method 0 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d000980a0 0x7f3d00132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): get_auth_request con 0x7f3d000980a0 auth_method 0 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d05db4640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d055b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient(hunting): handle_auth_done global_id 44334 payload 824 2024-09-15T01:53:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 msgr2=0x7f3d001337d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d000980a0 msgr2=0x7f3d00132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d000980a0 0x7f3d00132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient: _finish_hunting 0 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 monclient: found mon.noname-a 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.198+0000 7f3d04db2640 1 -- 172.21.15.17:0/2254708520 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc009390 con 0x7f3d000a58b0 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3d04db2640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 msgr2=0x7f3d00133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3d04db2640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b540 tx=0x7f3cf00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011840 con 0x7f3d000a58b0 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: dump: 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf00119a0 con 0x7f3d000a58b0 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011d60 con 0x7f3d000a58b0 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf57fa640 10 monclient: dump: 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3d07817640 5 monclient: authenticate success, global_id 44334 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 10 monclient: _renew_subs 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d00136150 con 0x7f3d000a58b0 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.200+0000 7f3cf77fe640 4 set_mon_vals no callback set 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 10 monclient: _renew_subs 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.201+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d00135a30 con 0x7f3d000a58b0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3cf0010590 con 0x7f3d000a58b0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d07817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d07817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d07817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d000724a0 con 0x7f3d000a58b0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3cf57fa640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 0x7f3ce403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5496+0+0 (secure 0 0 0) 0x7f3cf004b5a0 con 0x7f3d000a58b0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d055b3640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 0x7f3ce403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.202+0000 7f3d055b3640 10 monclient: get_auth_request con 0x7f3ce4038790 auth_method 0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.205+0000 7f3d055b3640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 0x7f3ce403ac40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc031000 tx=0x7f3cfc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.207+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3cf000d850 con 0x7f3d000a58b0 2024-09-15T01:53:47.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.207+0000 7f3cf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.207+0000 7f3cf57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:47.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.294+0000 7f9ed2ac9640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:47.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.294+0000 7f9ed2ac9640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:47.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.294+0000 7f9ed2ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.294+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f9ecc0a5070 con 0x7f9ecc0a0b50 2024-09-15T01:53:47.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.295+0000 7f9ec8ff9640 1 -- 172.21.15.67:0/2876167393 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v53) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9eb80184f0 con 0x7f9ecc0a0b50 2024-09-15T01:53:47.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.295+0000 7f9ec8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T01:53:47.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.295+0000 7f9ec8ff9640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-15T01:53:47.298 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ec-foo' already exists 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 msgr2=0x7f9eb003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 0x7f9eb003aba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0031000 tx=0x7f9ec0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 10 monclient: shutdown 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 msgr2=0x7f9ecc1417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b600 tx=0x7f9eb80091a0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 shutdown_connections 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9eb00386f0 0x7f9eb003aba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ecc131b10 0x7f9ecc130f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0a0b50 0x7f9ecc1417c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 --2- 172.21.15.67:0/2876167393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ecc09eea0 0x7f9ecc141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 >> 172.21.15.67:0/2876167393 conn(0x7f9ecc071290 msgr2=0x7f9ecc0a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 shutdown_connections 2024-09-15T01:53:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.297+0000 7f9ed2ac9640 1 -- 172.21.15.67:0/2876167393 wait complete. 2024-09-15T01:53:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.386+0000 7f0a4b99d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.386+0000 7f0a4b99d640 10 monclient: build_initial_monmap 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.386+0000 7f0a4b99d640 10 monclient: monmap: 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:47.387829+0000 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:47.387829+0000 2024-09-15T01:53:47.389 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 Processor -- start 2024-09-15T01:53:47.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 -- start start 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient: init 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a44098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0a440a0370 0x7f0a44098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a440a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a44098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a44098ef0 con 0x7f0a44098a50 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a440a0b80 con 0x7f0a44098140 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a441328f0 con 0x7f0a440a0370 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.387+0000 7f0a4b99d640 10 monclient(hunting): authenticate will time out at 859.856976s 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a49739640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a44098140 0x7f0a441321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52808/0 (socket says 172.21.15.67:52808) 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a49739640 1 -- 172.21.15.67:0/2729379658 learned_addr learned my addr 172.21.15.67:0/2729379658 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 1 --2- 172.21.15.67:0/2729379658 >> v2:172.21.15.17:3300/0 conn(0x7f0a440a0370 0x7f0a44098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): get_auth_request con 0x7f0a440a0370 auth_method 0 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient(hunting): handle_auth_done global_id 44337 payload 274 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 1 -- 172.21.15.67:0/2729379658 >> v1:172.21.15.67:6789/0 conn(0x7f0a44098140 legacy=0x7f0a441321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 1 -- 172.21.15.67:0/2729379658 >> v1:172.21.15.17:6789/0 conn(0x7f0a44098a50 legacy=0x7f0a441390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient: _finish_hunting 0 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 1 monclient: found mon.noname-a 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.388+0000 7f0a48f38640 1 -- 172.21.15.67:0/2729379658 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440a00b0 con 0x7f0a440a0370 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a48f38640 1 -- 172.21.15.67:0/2729379658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a440a0370 msgr2=0x7f0a44098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a48f38640 1 --2- 172.21.15.67:0/2729379658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a440a0370 0x7f0a44098510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009850 tx=0x7f0a340090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5dcf31b226f9e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 1 -- 172.21.15.67:0/2729379658 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34032440 con 0x7f0a440a0370 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:47.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: dump: 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a4b99d640 5 monclient: authenticate success, global_id 44337 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a4b99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 1 -- 172.21.15.67:0/2729379658 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a340325a0 con 0x7f0a440a0370 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a4b99d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a4b99d640 4 set_mon_vals no callback set 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 1 -- 172.21.15.67:0/2729379658 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34032940 con 0x7f0a440a0370 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.391+0000 7f0a3bfff640 10 monclient: dump: 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.405 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 10 monclient: shutdown 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 -- 172.21.15.67:0/2729379658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a440a0370 msgr2=0x7f0a44098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/2729379658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a440a0370 0x7f0a44098510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009850 tx=0x7f0a340090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 -- 172.21.15.67:0/2729379658 shutdown_connections 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/2729379658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a440a0370 0x7f0a44098510 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 -- 172.21.15.67:0/2729379658 >> 172.21.15.67:0/2729379658 conn(0x7f0a440a5780 msgr2=0x7f0a440a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 -- 172.21.15.67:0/2729379658 shutdown_connections 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 -- 172.21.15.67:0/2729379658 wait complete. 2024-09-15T01:53:47.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 10 monclient: build_initial_monmap 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 10 monclient: monmap: 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.392+0000 7f0a4b99d640 1 Processor -- start 2024-09-15T01:53:47.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 -- start start 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient: init 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a44098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a44098a50 0x7f0a4412e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a44098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a440a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44131ba0 con 0x7f0a44098a50 2024-09-15T01:53:47.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4413a2d0 con 0x7f0a44098140 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44132520 con 0x7f0a440a0370 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a4b99d640 10 monclient(hunting): authenticate will time out at 859.862637s 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60518/0 (socket says 172.21.15.67:60518) 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 1 -- 172.21.15.67:0/782648171 learned_addr learned my addr 172.21.15.67:0/782648171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): get_auth_request con 0x7f0a44098140 auth_method 0 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a44098a50 0x7f0a4412e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): get_auth_request con 0x7f0a44098a50 auth_method 0 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 10 monclient(hunting): get_auth_request con 0x7f0a440a0370 auth_method 0 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 msgr2=0x7f0a4412e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 msgr2=0x7f0a4412e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a48f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.393+0000 7f0a49f3a640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 msgr2=0x7f0a4412e570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a44098a50 msgr2=0x7f0a4412e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a44098a50 0x7f0a4412e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 10 monclient: _finish_hunting 0 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 monclient: found mon.noname-b 2024-09-15T01:53:47.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.394+0000 7f0a49739640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a34009390 con 0x7f0a44098140 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a49739640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 msgr2=0x7f0a4409b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a49739640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c5e0 tx=0x7f0a40004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40005cb0 con 0x7f0a44098140 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: dump: 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a40005e30 con 0x7f0a44098140 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 5 monclient: authenticate success, global_id 4903 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 10 monclient: _renew_subs 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a441305b0 con 0x7f0a44098140 2024-09-15T01:53:47.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40007340 con 0x7f0a44098140 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a397fa640 10 monclient: dump: 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:47.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 10 monclient: _renew_subs 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a3b7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.396+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a44130c20 con 0x7f0a44098140 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.397+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a4001b400 con 0x7f0a44098140 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.397+0000 7f0a4b99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.397+0000 7f0a4b99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.397+0000 7f0a4b99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.397+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404bf60 con 0x7f0a44098140 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.398+0000 7f0a397fa640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 0x7f0a2803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.398+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5496+0+0 (secure 0 0 0) 0x7f0a4001a700 con 0x7f0a44098140 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.398+0000 7f0a48f38640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 0x7f0a2803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:47.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.398+0000 7f0a48f38640 10 monclient: get_auth_request con 0x7f0a28038740 auth_method 0 2024-09-15T01:53:47.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.400+0000 7f0a48f38640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 0x7f0a2803abf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a440a1b30 tx=0x7f0a34032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:47.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.402+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0a4001b6b0 con 0x7f0a44098140 2024-09-15T01:53:47.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.402+0000 7f0a397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:47.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.402+0000 7f0a397fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.436+0000 7f3d07817640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.436+0000 7f3d07817640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.436+0000 7f3d07817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:47.436+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3d0009fc90 con 0x7f3d000a58b0 2024-09-15T01:53:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.565+0000 7f0a4b99d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.565+0000 7f0a4b99d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.565+0000 7f0a4b99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:47.565+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a440713d0 con 0x7f0a44098140 2024-09-15T01:53:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:50.197+0000 7f3cebfff640 10 monclient: tick 2024-09-15T01:53:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:50.197+0000 7f3cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:50.198+0000 7f3cebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:50.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:50.393+0000 7f0a2ffff640 10 monclient: tick 2024-09-15T01:53:50.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:50.393+0000 7f0a2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:50.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:50.393+0000 7f0a2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:51.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:51.198+0000 7f3cebfff640 10 monclient: tick 2024-09-15T01:53:51.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:51.198+0000 7f3cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:51.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:51.394+0000 7f0a2ffff640 10 monclient: tick 2024-09-15T01:53:51.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:51.394+0000 7f0a2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:52.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.198+0000 7f3cebfff640 10 monclient: tick 2024-09-15T01:53:52.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.198+0000 7f3cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:52.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.394+0000 7f0a2ffff640 10 monclient: tick 2024-09-15T01:53:52.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.394+0000 7f0a2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:52.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.400+0000 7f3cf57fa640 1 -- 172.21.15.17:0/2254708520 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f3cf0015050 con 0x7f3d000a58b0 2024-09-15T01:53:52.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.400+0000 7f3cf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:52.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.400+0000 7f3cf57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:52.402 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 msgr2=0x7f3ce403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 0x7f3ce403ac40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc031000 tx=0x7f3cfc032000 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 10 monclient: shutdown 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 msgr2=0x7f3d00133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b540 tx=0x7f3cf00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 shutdown_connections 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3ce4038790 0x7f3ce403ac40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d000a58b0 0x7f3d00133290 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d00136dd0 0x7f3d001337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 --2- 172.21.15.17:0/2254708520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d000980a0 0x7f3d00132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.403+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 >> 172.21.15.17:0/2254708520 conn(0x7f3d00070ed0 msgr2=0x7f3d00068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:52.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.404+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 shutdown_connections 2024-09-15T01:53:52.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:52.404+0000 7f3d07817640 1 -- 172.21.15.17:0/2254708520 wait complete. 2024-09-15T01:53:52.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.457+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v54) v1 ==== 154+0+0 (secure 0 0 0) 0x7f0a4001a360 con 0x7f0a44098140 2024-09-15T01:53:52.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.457+0000 7f0a397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:52.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.457+0000 7f0a397fa640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-15T01:53:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.560+0000 7f0a4b99d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.560+0000 7f0a4b99d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:52.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.560+0000 7f0a4b99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.560+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a44130980 con 0x7f0a44098140 2024-09-15T01:53:52.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.561+0000 7f0a397fa640 1 -- 172.21.15.67:0/782648171 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v54) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0a4003e0c0 con 0x7f0a44098140 2024-09-15T01:53:52.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.561+0000 7f0a397fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T01:53:52.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.561+0000 7f0a397fa640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-15T01:53:52.564 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ec-foo' does not exist 2024-09-15T01:53:52.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 msgr2=0x7f0a2803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 0x7f0a2803abf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a440a1b30 tx=0x7f0a34032000 comp rx=0 tx=0).stop 2024-09-15T01:53:52.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 10 monclient: shutdown 2024-09-15T01:53:52.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:52.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 msgr2=0x7f0a4409b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c5e0 tx=0x7f0a40004960 comp rx=0 tx=0).stop 2024-09-15T01:53:52.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 shutdown_connections 2024-09-15T01:53:52.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a28038740 0x7f0a2803abf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a44098a50 0x7f0a4412e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a440a0370 0x7f0a4412e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 --2- 172.21.15.67:0/782648171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a44098140 0x7f0a4409b520 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.563+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 >> 172.21.15.67:0/782648171 conn(0x7f0a440a5780 msgr2=0x7f0a4412ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.564+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 shutdown_connections 2024-09-15T01:53:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.564+0000 7f0a4b99d640 1 -- 172.21.15.67:0/782648171 wait complete. 2024-09-15T01:53:52.576 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:52.576 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo 2024-09-15T01:53:52.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:52.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 10 monclient: build_initial_monmap 2024-09-15T01:53:52.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 10 monclient: monmap: 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:52.652253+0000 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:52.652253+0000 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 1 Processor -- start 2024-09-15T01:53:52.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 1 -- start start 2024-09-15T01:53:52.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.651+0000 7fdbe751d640 10 monclient: init 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdbe0098140 0x7fdbe01321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe0098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbe00a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbe0098c20 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00a0b80 con 0x7fdbe0098c20 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00a00b0 con 0x7fdbe00a0370 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe0071390 con 0x7fdbe0098140 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe751d640 10 monclient(hunting): authenticate will time out at 865.121404s 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe4ab8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdbe00a0370 0x7fdbe0098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57916/0 (socket says 172.21.15.67:57916) 2024-09-15T01:53:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe4ab8640 1 -- 172.21.15.67:0/3270675155 learned_addr learned my addr 172.21.15.67:0/3270675155 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 1 --2- 172.21.15.67:0/3270675155 >> v2:172.21.15.17:3300/0 conn(0x7fdbe0098140 0x7fdbe01321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): get_auth_request con 0x7fdbe0098140 auth_method 0 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.652+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_done global_id 44343 payload 274 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 1 -- 172.21.15.67:0/3270675155 >> v1:172.21.15.67:6789/0 conn(0x7fdbe00a0370 legacy=0x7fdbe0098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 1 -- 172.21.15.67:0/3270675155 >> v1:172.21.15.17:6789/0 conn(0x7fdbe0098c20 legacy=0x7fdbe01390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 1 monclient: found mon.noname-a 2024-09-15T01:53:52.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:52.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:52.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.653+0000 7fdbe52b9640 1 -- 172.21.15.67:0/3270675155 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00714d0 con 0x7fdbe0098140 2024-09-15T01:53:52.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.655+0000 7fdbe52b9640 1 -- 172.21.15.67:0/3270675155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098140 msgr2=0x7fdbe01321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:52.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.655+0000 7fdbe52b9640 1 --2- 172.21.15.67:0/3270675155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098140 0x7fdbe01321e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009e60 tx=0x7fdbd0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cda1b1692f4938 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:52.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 1 -- 172.21.15.67:0/3270675155 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0007630 con 0x7fdbe0098140 2024-09-15T01:53:52.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:52.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: dump: 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: _finish_auth 0 2024-09-15T01:53:52.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 5 monclient: authenticate success, global_id 44343 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 1 -- 172.21.15.67:0/3270675155 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd002d050 con 0x7fdbe0098140 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 1 -- 172.21.15.67:0/3270675155 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00079c0 con 0x7fdbe0098140 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 4 set_mon_vals no callback set 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbdffff640 10 monclient: dump: 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.664 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 10 monclient: shutdown 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 -- 172.21.15.67:0/3270675155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098140 msgr2=0x7fdbe01321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 --2- 172.21.15.67:0/3270675155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098140 0x7fdbe01321e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009e60 tx=0x7fdbd0009780 comp rx=0 tx=0).stop 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 -- 172.21.15.67:0/3270675155 shutdown_connections 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 --2- 172.21.15.67:0/3270675155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098140 0x7fdbe01321e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 -- 172.21.15.67:0/3270675155 >> 172.21.15.67:0/3270675155 conn(0x7fdbe00a5780 msgr2=0x7fdbe00a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 -- 172.21.15.67:0/3270675155 shutdown_connections 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.656+0000 7fdbe751d640 1 -- 172.21.15.67:0/3270675155 wait complete. 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.657+0000 7fdbe751d640 10 monclient: build_initial_monmap 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.657+0000 7fdbe751d640 10 monclient: monmap: 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:52.666 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.657+0000 7fdbe751d640 1 Processor -- start 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.657+0000 7fdbe751d640 1 -- start start 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.657+0000 7fdbe751d640 10 monclient: init 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe0098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098c20 0x7fdbe0135400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe0098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 0x7fdbe0135940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe0138760 con 0x7fdbe0098c20 2024-09-15T01:53:52.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00a37c0 con 0x7fdbe0098140 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe0138480 con 0x7fdbe00a0370 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe751d640 10 monclient(hunting): authenticate will time out at 865.127332s 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56094/0 (socket says 172.21.15.67:56094) 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 -- 172.21.15.67:0/2575649135 learned_addr learned my addr 172.21.15.67:0/2575649135 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): get_auth_request con 0x7fdbe0098140 auth_method 0 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:52.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe4ab8640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098c20 0x7fdbe0135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe5aba640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 0x7fdbe0135940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient(hunting): handle_auth_done global_id 4906 payload 824 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 -- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 msgr2=0x7fdbe0135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 0x7fdbe0135940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 -- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098c20 msgr2=0x7fdbe0135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098c20 0x7fdbe0135400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 monclient: found mon.noname-b 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe52b9640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0009a20 con 0x7fdbe0098140 2024-09-15T01:53:52.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe5aba640 10 monclient: get_auth_request con 0x7fdbe00a0370 auth_method 0 2024-09-15T01:53:52.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.658+0000 7fdbe5aba640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 0x7fdbe0135940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:53:52.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.660+0000 7fdbe52b9640 1 -- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 msgr2=0x7fdbe0134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:52.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.660+0000 7fdbe52b9640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009e60 tx=0x7fdbd0037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:52.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.660+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00318f0 con 0x7fdbe0098140 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: dump: 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:52.672 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0040070 con 0x7fdbe0098140 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0031c60 con 0x7fdbe0098140 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 5 monclient: authenticate success, global_id 4906 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 10 monclient: _renew_subs 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe0137590 con 0x7fdbe0098140 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:52.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdd7fa640 10 monclient: dump: 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbdf7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 10 monclient: _renew_subs 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.663+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe01381e0 con 0x7fdbe0098140 2024-09-15T01:53:52.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdbd0042030 con 0x7fdbe0098140 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbdd7fa640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 0x7fdbcc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fdbd006d340 con 0x7fdbe0098140 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe751d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe751d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe4ab8640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 0x7fdbcc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.664+0000 7fdbe4ab8640 10 monclient: get_auth_request con 0x7fdbcc038790 auth_method 0 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.666+0000 7fdbe4ab8640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 0x7fdbcc03ac40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbe01323c0 tx=0x7fdbd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.668+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.668+0000 7fdbdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:52.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.668+0000 7fdbdd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:52.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.835+0000 7fdbe751d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.835+0000 7fdbe751d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.835+0000 7fdbe751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.835+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.836+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v54) v1 ==== 65+0+60 (secure 0 0 0) 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.836+0000 7fdbdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.836+0000 7fdbdd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:52.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.935+0000 7fdbe751d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.935+0000 7fdbe751d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.936+0000 7fdbe751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:52.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.936+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.936+0000 7fdbdd7fa640 1 -- 172.21.15.67:0/2575649135 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v54) v1 ==== 65+0+60 (secure 0 0 0) 0x7fdbe0099100 con 0x7fdbe0098140 2024-09-15T01:53:52.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.936+0000 7fdbdd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:52.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.936+0000 7fdbdd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 msgr2=0x7fdbcc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 0x7fdbcc03ac40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbe01323c0 tx=0x7fdbd4009040 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 10 monclient: shutdown 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 msgr2=0x7fdbe0134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009e60 tx=0x7fdbd0037000 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 shutdown_connections 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdbcc038790 0x7fdbcc03ac40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdbe0098c20 0x7fdbe0135400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdbe00a0370 0x7fdbe0135940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 --2- 172.21.15.67:0/2575649135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe0098140 0x7fdbe0134ec0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:52.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 >> 172.21.15.67:0/2575649135 conn(0x7fdbe00a5780 msgr2=0x7fdbe009b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:52.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 shutdown_connections 2024-09-15T01:53:52.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:52.938+0000 7fdbe751d640 1 -- 172.21.15.67:0/2575649135 wait complete. 2024-09-15T01:53:52.949 INFO:tasks.workunit.client.0.smithi067.stdout:foo 2024-09-15T01:53:52.949 INFO:tasks.workunit.client.0.smithi067.stdout:foo-ssd 2024-09-15T01:53:52.949 INFO:tasks.workunit.client.0.smithi067.stdout:foo-hdd 2024-09-15T01:53:52.949 INFO:tasks.workunit.client.0.smithi067.stdout:ec-foo 2024-09-15T01:53:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-15T01:53:53.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.024+0000 7fde82a4f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:53.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.025+0000 7fde82a4f640 10 monclient: build_initial_monmap 2024-09-15T01:53:53.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.025+0000 7fde82a4f640 10 monclient: monmap: 2024-09-15T01:53:53.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:53.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:53.026180+0000 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:53.026180+0000 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:53.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.025+0000 7fde82a4f640 1 Processor -- start 2024-09-15T01:53:53.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.025+0000 7fde82a4f640 1 -- start start 2024-09-15T01:53:53.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.025+0000 7fde82a4f640 10 monclient: init 2024-09-15T01:53:53.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde7c068dc0 0x7fde7c132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-c con 0x7fde7c068dc0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde7c069190 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fde7c137150 0x7fde7c139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fde7c137150 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c096c10 con 0x7fde7c069190 2024-09-15T01:53:53.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c06a2a0 con 0x7fde7c137150 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c0a14d0 con 0x7fde7c068dc0 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde82a4f640 10 monclient(hunting): authenticate will time out at 865.495413s 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde7c068dc0 0x7fde7c132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde7c068dc0 0x7fde7c132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56104/0 (socket says 172.21.15.67:56104) 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 -- 172.21.15.67:0/1609054574 learned_addr learned my addr 172.21.15.67:0/1609054574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): get_auth_request con 0x7fde7c068dc0 auth_method 0 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:53.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde80fec640 1 --2- 172.21.15.67:0/1609054574 >> v2:172.21.15.17:3300/0 conn(0x7fde7c137150 0x7fde7c139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient(hunting): handle_auth_done global_id 4909 payload 274 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 -- 172.21.15.67:0/1609054574 >> v2:172.21.15.17:3300/0 conn(0x7fde7c137150 msgr2=0x7fde7c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 --2- 172.21.15.67:0/1609054574 >> v2:172.21.15.17:3300/0 conn(0x7fde7c137150 0x7fde7c139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 -- 172.21.15.67:0/1609054574 >> v1:172.21.15.17:6789/0 conn(0x7fde7c069190 legacy=0x7fde7c069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:53.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 monclient: found mon.noname-c 2024-09-15T01:53:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.026+0000 7fde7bfff640 1 -- 172.21.15.67:0/1609054574 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c09ff20 con 0x7fde7c068dc0 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7bfff640 1 -- 172.21.15.67:0/1609054574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c068dc0 msgr2=0x7fde7c132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7bfff640 1 --2- 172.21.15.67:0/1609054574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c068dc0 0x7fde7c132250 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06a3f0 tx=0x7fde6c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997c3906ef1c4c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 1 -- 172.21.15.67:0/1609054574 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c032440 con 0x7fde7c068dc0 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: dump: 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: _finish_auth 0 2024-09-15T01:53:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 1 -- 172.21.15.67:0/1609054574 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c0325a0 con 0x7fde7c068dc0 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 1 -- 172.21.15.67:0/1609054574 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c032940 con 0x7fde7c068dc0 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde7affd640 10 monclient: dump: 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde82a4f640 5 monclient: authenticate success, global_id 4909 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde82a4f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.029+0000 7fde82a4f640 4 set_mon_vals no callback set 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 10 monclient: shutdown 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 -- 172.21.15.67:0/1609054574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c068dc0 msgr2=0x7fde7c132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 --2- 172.21.15.67:0/1609054574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c068dc0 0x7fde7c132250 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06a3f0 tx=0x7fde6c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 -- 172.21.15.67:0/1609054574 shutdown_connections 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 --2- 172.21.15.67:0/1609054574 >> v2:172.21.15.17:3300/0 conn(0x7fde7c137150 0x7fde7c139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 --2- 172.21.15.67:0/1609054574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c068dc0 0x7fde7c132250 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 -- 172.21.15.67:0/1609054574 >> 172.21.15.67:0/1609054574 conn(0x7fde7c0711b0 msgr2=0x7fde7c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 -- 172.21.15.67:0/1609054574 shutdown_connections 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 1 -- 172.21.15.67:0/1609054574 wait complete. 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 10 monclient: build_initial_monmap 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.030+0000 7fde82a4f640 10 monclient: monmap: 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:53.043 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 Processor -- start 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 -- start start 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient: init 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:53.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-c con 0x7fde7c068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7c069190 0x7fde7c098520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fde7c069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): picked mon.noname-b con 0x7fde7c137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0a0890 con 0x7fde7c069190 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c131890 con 0x7fde7c137150 2024-09-15T01:53:53.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c09b0e0 con 0x7fde7c068dc0 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.031+0000 7fde82a4f640 10 monclient(hunting): authenticate will time out at 865.500955s 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56112/0 (socket says 172.21.15.67:56112) 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 -- 172.21.15.67:0/3231970179 learned_addr learned my addr 172.21.15.67:0/3231970179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): get_auth_request con 0x7fde7c137150 auth_method 0 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:53.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7c069190 0x7fde7c098520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 10 monclient(hunting): get_auth_request con 0x7fde7c068dc0 auth_method 0 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request con 0x7fde7c069190 auth_method 0 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:53.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 msgr2=0x7fde7c097fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 msgr2=0x7fde7c097fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient(hunting): handle_auth_done global_id 4912 payload 824 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde7bfff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 msgr2=0x7fde7c097fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7c069190 msgr2=0x7fde7c098520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7c069190 0x7fde7c098520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:53.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient: _finish_hunting 0 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 monclient: found mon.noname-b 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.032+0000 7fde80fec640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c009450 con 0x7fde7c137150 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde80fec640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 msgr2=0x7fde7c098a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde80fec640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde7400ce80 tx=0x7fde74007840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7400e610 con 0x7fde7c137150 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: dump: 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:53.049 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7400e770 con 0x7fde7c137150 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 5 monclient: authenticate success, global_id 4912 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 10 monclient: _renew_subs 2024-09-15T01:53:53.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c09b940 con 0x7fde7c137150 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7400eb30 con 0x7fde7c137150 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde63fff640 10 monclient: dump: 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:53.051 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 10 monclient: _renew_subs 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c09a4e0 con 0x7fde7c137150 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.034+0000 7fde7a7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde74003020 con 0x7fde7c137150 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde82a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde82a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde82a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c099db0 con 0x7fde7c137150 2024-09-15T01:53:53.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde63fff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 0x7fde6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fde74012070 con 0x7fde7c137150 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.035+0000 7fde7bfff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 0x7fde6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.036+0000 7fde7bfff640 10 monclient: get_auth_request con 0x7fde68038740 auth_method 0 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.038+0000 7fde7bfff640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 0x7fde6803abf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06a3f0 tx=0x7fde6c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.040+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fde74015730 con 0x7fde7c137150 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.040+0000 7fde63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.040+0000 7fde63fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:53.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.206+0000 7fde82a4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:53.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.206+0000 7fde82a4f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:53.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.206+0000 7fde82a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:53.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.206+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fde7c04bf60 con 0x7fde7c137150 2024-09-15T01:53:53.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.967+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v55) v1 ==== 110+0+0 (secure 0 0 0) 0x7fde740147a0 con 0x7fde7c137150 2024-09-15T01:53:53.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.967+0000 7fde63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:53.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:53.967+0000 7fde63fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:54.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.071+0000 7fde82a4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.071+0000 7fde82a4f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.071+0000 7fde82a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.071+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fde7c134360 con 0x7fde7c137150 2024-09-15T01:53:54.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.072+0000 7fde63fff640 1 -- 172.21.15.67:0/3231970179 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v55) v1 ==== 139+0+0 (secure 0 0 0) 0x7fde74014d50 con 0x7fde7c137150 2024-09-15T01:53:54.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.072+0000 7fde63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.072+0000 7fde63fff640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-15T01:53:54.075 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'foo-asdf' 2024-09-15T01:53:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 msgr2=0x7fde6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 0x7fde6803abf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06a3f0 tx=0x7fde6c032000 comp rx=0 tx=0).stop 2024-09-15T01:53:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 10 monclient: shutdown 2024-09-15T01:53:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 msgr2=0x7fde7c098a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde7400ce80 tx=0x7fde74007840 comp rx=0 tx=0).stop 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 shutdown_connections 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde68038740 0x7fde6803abf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7c069190 0x7fde7c098520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c137150 0x7fde7c098a60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 --2- 172.21.15.67:0/3231970179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7c068dc0 0x7fde7c097fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.074+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 >> 172.21.15.67:0/3231970179 conn(0x7fde7c0711b0 msgr2=0x7fde7c09e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.075+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 shutdown_connections 2024-09-15T01:53:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.075+0000 7fde82a4f640 1 -- 172.21.15.67:0/3231970179 wait complete. 2024-09-15T01:53:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 10 monclient: get_monmap_and_config 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 10 monclient: build_initial_monmap 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 10 monclient: monmap: 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.165 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:54.164203+0000 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:54.164203+0000 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:54.166 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 1 Processor -- start 2024-09-15T01:53:54.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 1 -- start start 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.163+0000 7fbcd8834640 10 monclient: init 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcd4098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbcd40a0370 0x7fbcd4098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd40a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd4098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd4098ef0 con 0x7fbcd4098140 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd40a1730 con 0x7fbcd40a0370 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40a0b80 con 0x7fbcd4098a50 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd8834640 10 monclient(hunting): authenticate will time out at 866.633347s 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcd4098140 0x7fbcd41321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57928/0 (socket says 172.21.15.67:57928) 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd259c640 1 -- 172.21.15.67:0/1978061142 learned_addr learned my addr 172.21.15.67:0/1978061142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1978061142 >> v2:172.21.15.17:3300/0 conn(0x7fbcd40a0370 0x7fbcd4098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 10 monclient(hunting): get_auth_request con 0x7fbcd4098a50 auth_method 0 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): get_auth_request con 0x7fbcd40a0370 auth_method 0 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 1 -- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 msgr2=0x7fbcd41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 1 -- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 msgr2=0x7fbcd41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.164+0000 7fbcd2d9d640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_done global_id 44349 payload 274 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 msgr2=0x7fbcd41390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1978061142 >> v1:172.21.15.67:6789/0 conn(0x7fbcd4098140 legacy=0x7fbcd41321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:54.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 1 monclient: found mon.noname-a 2024-09-15T01:53:54.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:54.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:54.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.165+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1978061142 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd41328f0 con 0x7fbcd40a0370 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 msgr2=0x7fbcd4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd4098510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009850 tx=0x7fbcc80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b28e67c47c5b998 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 1 -- 172.21.15.67:0/1978061142 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802adf0 con 0x7fbcd40a0370 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: dump: 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: _finish_auth 0 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 5 monclient: authenticate success, global_id 44349 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 1 -- 172.21.15.67:0/1978061142 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc8032440 con 0x7fbcd40a0370 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 4 set_mon_vals no callback set 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 1 -- 172.21.15.67:0/1978061142 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8032820 con 0x7fbcd40a0370 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd159a640 10 monclient: dump: 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 10 monclient: shutdown 2024-09-15T01:53:54.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 1 -- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 msgr2=0x7fbcd4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd4098510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009850 tx=0x7fbcc80090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 1 -- 172.21.15.67:0/1978061142 shutdown_connections 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.168+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098a50 0x7fbcd41390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1978061142 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd4098510 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 -- 172.21.15.67:0/1978061142 >> 172.21.15.67:0/1978061142 conn(0x7fbcd40a5780 msgr2=0x7fbcd40a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 -- 172.21.15.67:0/1978061142 shutdown_connections 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 -- 172.21.15.67:0/1978061142 wait complete. 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 10 monclient: build_initial_monmap 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 10 monclient: monmap: 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:54.181 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 Processor -- start 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 1 -- start start 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.169+0000 7fbcd8834640 10 monclient: init 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd4098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd4098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd41360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd40a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd411bc90 con 0x7fbcd40a0370 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4099090 con 0x7fbcd4098a50 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd41392c0 con 0x7fbcd4098140 2024-09-15T01:53:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd8834640 10 monclient(hunting): authenticate will time out at 866.639430s 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56124/0 (socket says 172.21.15.67:56124) 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1563428666 learned_addr learned my addr 172.21.15.67:0/1563428666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): get_auth_request con 0x7fbcd4098a50 auth_method 0 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 10 monclient(hunting): get_auth_request con 0x7fbcd4098140 auth_method 0 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd41360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): get_auth_request con 0x7fbcd40a0370 auth_method 0 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 msgr2=0x7fbcd4135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 msgr2=0x7fbcd4135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:54.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd259c640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient(hunting): handle_auth_done global_id 4915 payload 824 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 msgr2=0x7fbcd4135640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 msgr2=0x7fbcd41360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd41360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 monclient: found mon.noname-b 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.170+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc8009390 con 0x7fbcd4098a50 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd1d9b640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 msgr2=0x7fbcd4135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd1d9b640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009360 tx=0x7fbcc8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8031b10 con 0x7fbcd4098a50 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: dump: 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.187 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc8031c70 con 0x7fbcd4098a50 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 5 monclient: authenticate success, global_id 4915 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 10 monclient: _renew_subs 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd4137510 con 0x7fbcd4098a50 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:54.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc803d440 con 0x7fbcd4098a50 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcc27fc640 10 monclient: dump: 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.189 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd0d99640 4 set_mon_vals no callback set 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 10 monclient: _renew_subs 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.173+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd4137b70 con 0x7fbcd4098a50 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbcc8039040 con 0x7fbcd4098a50 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd8834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd8834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd8834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd404bf60 con 0x7fbcd4098a50 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcc27fc640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fbcc806b830 con 0x7fbcd4098a50 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd259c640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.174+0000 7fbcd259c640 10 monclient: get_auth_request con 0x7fbcb4038740 auth_method 0 2024-09-15T01:53:54.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.177+0000 7fbcd259c640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 0x7fbcb403abf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4007d10 tx=0x7fbcc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.178+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbcc8030990 con 0x7fbcd4098a50 2024-09-15T01:53:54.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.178+0000 7fbcc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.178+0000 7fbcc27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:54.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.345+0000 7fbcd8834640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.345+0000 7fbcd8834640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.345+0000 7fbcd8834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.345+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fbcd4098ef0 con 0x7fbcd4098a50 2024-09-15T01:53:54.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.346+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v55) v1 ==== 139+0+0 (secure 0 0 0) 0x7fbcd4098ef0 con 0x7fbcd4098a50 2024-09-15T01:53:54.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.346+0000 7fbcc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.347+0000 7fbcc27fc640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-15T01:53:54.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.450+0000 7fbcd8834640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.450+0000 7fbcd8834640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.450+0000 7fbcd8834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.450+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fbcd4137510 con 0x7fbcd4098a50 2024-09-15T01:53:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.451+0000 7fbcc27fc640 1 -- 172.21.15.67:0/1563428666 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v55) v1 ==== 139+0+0 (secure 0 0 0) 0x7fbcd4137510 con 0x7fbcd4098a50 2024-09-15T01:53:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.451+0000 7fbcc27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T01:53:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.451+0000 7fbcc27fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-15T01:53:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'foo-asdf' 2024-09-15T01:53:54.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 msgr2=0x7fbcb403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 0x7fbcb403abf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4007d10 tx=0x7fbcc4009040 comp rx=0 tx=0).stop 2024-09-15T01:53:54.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 10 monclient: shutdown 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 msgr2=0x7fbcd4135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009360 tx=0x7fbcc8032000 comp rx=0 tx=0).stop 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 shutdown_connections 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd40a0370 0x7fbcd41360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4098a50 0x7fbcd4135b80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 --2- 172.21.15.67:0/1563428666 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd4098140 0x7fbcd4135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 >> 172.21.15.67:0/1563428666 conn(0x7fbcd40a5780 msgr2=0x7fbcd409e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 shutdown_connections 2024-09-15T01:53:54.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.453+0000 7fbcd8834640 1 -- 172.21.15.67:0/1563428666 wait complete. 2024-09-15T01:53:54.464 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-15T01:53:54.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.540+0000 7fea8959d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.540+0000 7fea8959d640 10 monclient: build_initial_monmap 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.540+0000 7fea8959d640 10 monclient: monmap: 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:54.541747+0000 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:54.541747+0000 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:54.543 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:54.544 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:54.544 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 Processor -- start 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 -- start start 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient: init 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fea8406e210 0x7fea8406b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-a con 0x7fea8406e210 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea8406e5e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-b con 0x7fea8409e220 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:54.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8406aa60 con 0x7fea8406e5e0 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8406a190 con 0x7fea8406e210 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea84069bd0 con 0x7fea8409e220 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea8959d640 10 monclient(hunting): authenticate will time out at 867.010716s 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea837fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea82ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fea8406e210 0x7fea8406b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea837fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:34646/0 (socket says 172.21.15.67:34646) 2024-09-15T01:53:54.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.541+0000 7fea82ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fea8406e210 0x7fea8406b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:42530/0 (socket says 172.21.15.67:42530) 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 1 -- 172.21.15.67:0/2708237853 learned_addr learned my addr 172.21.15.67:0/2708237853 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 10 monclient(hunting): get_auth_request con 0x7fea8409e220 auth_method 0 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): get_auth_request con 0x7fea8406e210 auth_method 0 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 1 -- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 msgr2=0x7fea84139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 1 -- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 msgr2=0x7fea84139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:54.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea837fe640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient(hunting): handle_auth_done global_id 44355 payload 274 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 1 -- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 msgr2=0x7fea84139d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 1 -- 172.21.15.67:0/2708237853 >> v1:172.21.15.17:6789/0 conn(0x7fea8406e5e0 legacy=0x7fea8409db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 1 monclient: found mon.noname-a 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:54.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.542+0000 7fea82ffd640 1 -- 172.21.15.67:0/2708237853 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840698f0 con 0x7fea8406e210 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea82ffd640 1 -- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8406e210 msgr2=0x7fea8406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8406e210 0x7fea8406b9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea74009850 tx=0x7fea740090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c135d1d065675e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 1 -- 172.21.15.67:0/2708237853 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74032440 con 0x7fea8406e210 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: dump: 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.559 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea8959d640 5 monclient: authenticate success, global_id 44355 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea8959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:54.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 1 -- 172.21.15.67:0/2708237853 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea740325a0 con 0x7fea8406e210 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea8959d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea8959d640 4 set_mon_vals no callback set 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 1 -- 172.21.15.67:0/2708237853 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74032940 con 0x7fea8406e210 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.545+0000 7fea81ffb640 10 monclient: dump: 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.561 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 10 monclient: shutdown 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 -- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8406e210 msgr2=0x7fea8406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8406e210 0x7fea8406b9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea74009850 tx=0x7fea740090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 -- 172.21.15.67:0/2708237853 shutdown_connections 2024-09-15T01:53:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8409e220 0x7fea84139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 --2- 172.21.15.67:0/2708237853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8406e210 0x7fea8406b9b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 -- 172.21.15.67:0/2708237853 >> 172.21.15.67:0/2708237853 conn(0x7fea84070ff0 msgr2=0x7fea840a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 -- 172.21.15.67:0/2708237853 shutdown_connections 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 1 -- 172.21.15.67:0/2708237853 wait complete. 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 10 monclient: build_initial_monmap 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.546+0000 7fea8959d640 10 monclient: monmap: 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.563 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 Processor -- start 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 -- start start 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient: init 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-c con 0x7fea8406e210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-b con 0x7fea8406e5e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:54.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8409e220 0x7fea841415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): picked mon.noname-a con 0x7fea8409e220 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84122860 con 0x7fea8409e220 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84122a60 con 0x7fea8406e5e0 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea84123020 con 0x7fea8406e210 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea8959d640 10 monclient(hunting): authenticate will time out at 867.016691s 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56128/0 (socket says 172.21.15.67:56128) 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 1 -- 172.21.15.67:0/2020345784 learned_addr learned my addr 172.21.15.67:0/2020345784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): get_auth_request con 0x7fea8406e5e0 auth_method 0 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea82ffd640 10 monclient(hunting): get_auth_request con 0x7fea8406e210 auth_method 0 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea837fe640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8409e220 0x7fea841415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.547+0000 7fea82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): get_auth_request con 0x7fea8409e220 auth_method 0 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea82ffd640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 msgr2=0x7fea8413e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea82ffd640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 msgr2=0x7fea8413e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 10 monclient(hunting): handle_auth_done global_id 4918 payload 824 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 msgr2=0x7fea8413e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8409e220 msgr2=0x7fea841415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:54.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8409e220 0x7fea841415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 monclient: found mon.noname-b 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.548+0000 7fea827fc640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74009390 con 0x7fea8406e5e0 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea827fc640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 msgr2=0x7fea8413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea827fc640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c5e0 tx=0x7fea78007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78012840 con 0x7fea8406e5e0 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: dump: 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.568 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: _finish_auth 0 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea780129a0 con 0x7fea8406e5e0 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 5 monclient: authenticate success, global_id 4918 2024-09-15T01:53:54.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 10 monclient: _renew_subs 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea84122d40 con 0x7fea8406e5e0 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78012da0 con 0x7fea8406e5e0 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea6affd640 10 monclient: dump: 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:54.570 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea817fa640 4 set_mon_vals no callback set 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 10 monclient: _renew_subs 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.550+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8413dc20 con 0x7fea8406e5e0 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fea78011590 con 0x7fea8406e5e0 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea8959d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea8959d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea8959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8404bf60 con 0x7fea8406e5e0 2024-09-15T01:53:54.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea6affd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 0x7fea7003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fea7804bd70 con 0x7fea8406e5e0 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 0x7fea7003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.551+0000 7fea82ffd640 10 monclient: get_auth_request con 0x7fea70038740 auth_method 0 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.554+0000 7fea82ffd640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 0x7fea7003abf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea74031000 tx=0x7fea74032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.556+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fea78010ac0 con 0x7fea8406e5e0 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.556+0000 7fea6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:54.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.556+0000 7fea6affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:54.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.725+0000 7fea8959d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:54.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.725+0000 7fea8959d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:54.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.725+0000 7fea8959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:54.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.725+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fea8409f980 con 0x7fea8406e5e0 2024-09-15T01:53:54.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.970+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v56) v1 ==== 110+0+0 (secure 0 0 0) 0x7fea78016020 con 0x7fea8406e5e0 2024-09-15T01:53:54.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.970+0000 7fea6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:54.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:54.970+0000 7fea6affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.074+0000 7fea8959d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:55.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.074+0000 7fea8959d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:55.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.074+0000 7fea8959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.074+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fea84068a20 con 0x7fea8406e5e0 2024-09-15T01:53:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.076+0000 7fea6affd640 1 -- 172.21.15.67:0/2020345784 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v56) v1 ==== 139+0+0 (secure 0 0 0) 0x7fea780102e0 con 0x7fea8406e5e0 2024-09-15T01:53:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.076+0000 7fea6affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T01:53:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.076+0000 7fea6affd640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-15T01:53:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'bar-asdf' 2024-09-15T01:53:55.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 msgr2=0x7fea7003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 0x7fea7003abf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea74031000 tx=0x7fea74032000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 10 monclient: shutdown 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 msgr2=0x7fea8413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c5e0 tx=0x7fea78007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 shutdown_connections 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fea70038740 0x7fea7003abf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fea8409e220 0x7fea841415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea8406e5e0 0x7fea8413d6e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 --2- 172.21.15.67:0/2020345784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fea8406e210 0x7fea8413e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 >> 172.21.15.67:0/2020345784 conn(0x7fea84070ff0 msgr2=0x7fea840a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 shutdown_connections 2024-09-15T01:53:55.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.078+0000 7fea8959d640 1 -- 172.21.15.67:0/2020345784 wait complete. 2024-09-15T01:53:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-asdf 2024-09-15T01:53:55.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.162+0000 7f8923880640 10 monclient: get_monmap_and_config 2024-09-15T01:53:55.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.162+0000 7f8923880640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.162+0000 7f8923880640 10 monclient: monmap: 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:55.163433+0000 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:55.163433+0000 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:55.165 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.162+0000 7f8923880640 1 Processor -- start 2024-09-15T01:53:55.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 1 -- start start 2024-09-15T01:53:55.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient: init 2024-09-15T01:53:55.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f891c098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f891c0a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f891c13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): picked mon.noname-b con 0x7f891c098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891c0a0b80 con 0x7f891c0a0370 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891c1328f0 con 0x7f891c098140 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0a1730 con 0x7f891c098c20 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8923880640 10 monclient(hunting): authenticate will time out at 867.632672s 2024-09-15T01:53:55.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f892161c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f891c098140 0x7f891c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57942/0 (socket says 172.21.15.67:57942) 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f892161c640 1 -- 172.21.15.67:0/537885791 learned_addr learned my addr 172.21.15.67:0/537885791 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f891c13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8921e1d640 10 monclient(hunting): get_auth_request con 0x7f891c098c20 auth_method 0 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8921e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8921e1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.163+0000 7f8921e1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.164+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 msgr2=0x7f891c13a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.164+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 msgr2=0x7f891c13a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.164+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f891c13a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.164+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f891c13a0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.165+0000 7f892161c640 1 -- 172.21.15.67:0/537885791 >> v1:172.21.15.67:6789/0 conn(0x7f891c098140 legacy=0x7f891c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064567310 0 0) 0x7f891c1328f0 con 0x7f891c098140 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): my global_id is 4921 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 1 -- 172.21.15.67:0/537885791 >> v1:172.21.15.17:6789/0 conn(0x7f891c0a0370 legacy=0x7f891c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89040034e0 con 0x7f891c098140 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342637911 0 0) 0x7f891c0a0b80 con 0x7f891c0a0370 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): my global_id is 44361 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c1328f0 con 0x7f891c0a0370 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044832476 0 0) 0x7f89040034e0 con 0x7f891c098140 2024-09-15T01:53:55.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 msgr2=0x7f891c13a0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f891c13a0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 >> v1:172.21.15.17:6789/0 conn(0x7f891c0a0370 legacy=0x7f891c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c0a00b0 con 0x7f891c098140 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f8914001720 con 0x7f891c098140 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8923880640 5 monclient: authenticate success, global_id 4921 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8923880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:55.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: dump: 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 >> v1:172.21.15.67:6789/0 conn(0x7f891c098140 legacy=0x7f891c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f8904003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): picked mon.c con 0x7f891c098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8904004190 0x7f8904004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): picked mon.a con 0x7f8904004190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 0x7f8904005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): picked mon.b con 0x7f8904004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0a1730 con 0x7f8904004190 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890400bd00 con 0x7f8904004de0 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890400bec0 con 0x7f891c098c20 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f891bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8923880640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 0x7f8904005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): get_auth_request con 0x7f8904004de0 auth_method 0 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f8904003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8904004190 0x7f8904004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): get_auth_request con 0x7f891c098c20 auth_method 0 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 10 monclient(hunting): get_auth_request con 0x7f8904004190 auth_method 0 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f892161c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient(hunting): handle_auth_done global_id 4921 payload 293 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 msgr2=0x7f8904003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f891c098c20 0x7f8904003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8904004190 msgr2=0x7f8904004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8904004190 0x7f8904004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 monclient: found mon.b 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89040034e0 con 0x7f8904004de0 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8921e1d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.166+0000 7f8920e1b640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8904004190 0x7f8904004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:55.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 msgr2=0x7f8904005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f8921e1d640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 0x7f8904005250 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f891c096ab0 tx=0x7f890c0096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2322e0c90156f45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c030680 con 0x7f8904004de0 2024-09-15T01:53:55.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: dump: 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c0307e0 con 0x7f8904004de0 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f8923880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 1 -- 172.21.15.67:0/537885791 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c030ba0 con 0x7f8904004de0 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f8923880640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f8923880640 4 set_mon_vals no callback set 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.169+0000 7f891bfff640 10 monclient: dump: 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 10 monclient: shutdown 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 -- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 msgr2=0x7f8904005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 --2- 172.21.15.67:0/537885791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8904004de0 0x7f8904005250 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f891c096ab0 tx=0x7f890c0096f0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8921e1d640 1 -- 172.21.15.67:0/537885791 reap_dead start 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 -- 172.21.15.67:0/537885791 shutdown_connections 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 -- 172.21.15.67:0/537885791 >> 172.21.15.67:0/537885791 conn(0x7f891c0a5780 msgr2=0x7f891c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 -- 172.21.15.67:0/537885791 shutdown_connections 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 1 -- 172.21.15.67:0/537885791 wait complete. 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.170+0000 7f8923880640 10 monclient: monmap: 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 Processor -- start 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 -- start start 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient: init 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): picked mon.noname-c con 0x7f8904004de0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f891c1357f0 0x7f891c135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): picked mon.noname-a con 0x7f891c1357f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:55.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): picked mon.noname-b con 0x7f891c136100 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0a4000 con 0x7f891c1357f0 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c11c880 con 0x7f891c136100 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c137c10 con 0x7f8904004de0 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.171+0000 7f8923880640 10 monclient(hunting): authenticate will time out at 867.640968s 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56140/0 (socket says 172.21.15.67:56140) 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 -- 172.21.15.67:0/1839603401 learned_addr learned my addr 172.21.15.67:0/1839603401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:55.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): get_auth_request con 0x7f891c136100 auth_method 0 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 10 monclient(hunting): get_auth_request con 0x7f8904004de0 auth_method 0 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f891c1357f0 0x7f891c135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): get_auth_request con 0x7f891c1357f0 auth_method 0 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 msgr2=0x7f891c1352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 msgr2=0x7f891c1352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f892161c640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8920e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient(hunting): handle_auth_done global_id 4924 payload 824 2024-09-15T01:53:55.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 msgr2=0x7f891c1352b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f891c1357f0 msgr2=0x7f891c135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f891c1357f0 0x7f891c135bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 monclient: found mon.noname-b 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.172+0000 7f8921e1d640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c11c5a0 con 0x7f891c136100 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8921e1d640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 msgr2=0x7f891c147490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8921e1d640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f890c009750 tx=0x7f890c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c03f070 con 0x7f891c136100 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:55.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: dump: 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c02f3d0 con 0x7f891c136100 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c02f790 con 0x7f891c136100 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 5 monclient: authenticate success, global_id 4924 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 10 monclient: _renew_subs 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c11cb60 con 0x7f891c136100 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f89197fa640 10 monclient: dump: 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.197 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f891b7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 10 monclient: _renew_subs 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.174+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891c136ec0 con 0x7f891c136100 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.175+0000 7f8923880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.175+0000 7f8923880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.175+0000 7f8923880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.175+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f890c02fb80 con 0x7f891c136100 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.175+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891c04bf60 con 0x7f891c136100 2024-09-15T01:53:55.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.176+0000 7f89197fa640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 0x7f890803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.176+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f890c06c7d0 con 0x7f891c136100 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.176+0000 7f892161c640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 0x7f890803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.176+0000 7f892161c640 10 monclient: get_auth_request con 0x7f89080387c0 auth_method 0 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.178+0000 7f892161c640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 0x7f890803ac70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f89140018d0 tx=0x7f8914009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.180+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f890c02fe30 con 0x7f891c136100 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.180+0000 7f89197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.180+0000 7f89197fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:55.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f8923880640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f8923880640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f8923880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f891c071450 con 0x7f891c136100 2024-09-15T01:53:55.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+70 (secure 0 0 0) 0x7f890c034060 con 0x7f891c136100 2024-09-15T01:53:55.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f89197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.347+0000 7f89197fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:55.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f8923880640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f8923880640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f8923880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f891c0008d0 con 0x7f891c136100 2024-09-15T01:53:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f89197fa640 1 -- 172.21.15.67:0/1839603401 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+70 (secure 0 0 0) 0x7f890c048030 con 0x7f891c136100 2024-09-15T01:53:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f89197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.448+0000 7f89197fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 msgr2=0x7f890803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 0x7f890803ac70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f89140018d0 tx=0x7f8914009040 comp rx=0 tx=0).stop 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 10 monclient: shutdown 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 msgr2=0x7f891c147490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f890c009750 tx=0x7f890c036000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 shutdown_connections 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f89080387c0 0x7f890803ac70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f891c1357f0 0x7f891c135bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c136100 0x7f891c147490 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 --2- 172.21.15.67:0/1839603401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8904004de0 0x7f891c1352b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 >> 172.21.15.67:0/1839603401 conn(0x7f891c0a5780 msgr2=0x7f891c09bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 shutdown_connections 2024-09-15T01:53:55.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.450+0000 7f8923880640 1 -- 172.21.15.67:0/1839603401 wait complete. 2024-09-15T01:53:55.461 INFO:tasks.workunit.client.0.smithi067.stdout:foo-asdf 2024-09-15T01:53:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:55.462 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep bar-asdf 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 10 monclient: monmap: 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:55.527949+0000 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:55.527949+0000 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:55.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 1 Processor -- start 2024-09-15T01:53:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 1 -- start start 2024-09-15T01:53:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.527+0000 7fad1999d640 10 monclient: init 2024-09-15T01:53:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fad140a5780 0x7fad1409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-a con 0x7fad140a5780 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad1409ccf0 0x7fad14069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-c con 0x7fad1409ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad14069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140a5b50 con 0x7fad14069ac0 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad140a18b0 con 0x7fad140a5780 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1409c5f0 con 0x7fad1409ccf0 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad1999d640 10 monclient(hunting): authenticate will time out at 895.524391s 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fad140a5780 0x7fad1409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fad140a5780 0x7fad1409bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35876/0 (socket says 172.21.15.17:35876) 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad137fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fad14069ac0 0x7fad14138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52206/0 (socket says 172.21.15.17:52206) 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 -- 172.21.15.17:0/494564555 learned_addr learned my addr 172.21.15.17:0/494564555 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): get_auth_request con 0x7fad140a5780 auth_method 0 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 1 --2- 172.21.15.17:0/494564555 >> v2:172.21.15.67:3300/0 conn(0x7fad1409ccf0 0x7fad14069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): get_auth_request con 0x7fad1409ccf0 auth_method 0 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient(hunting): handle_auth_done global_id 44367 payload 274 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 -- 172.21.15.17:0/494564555 >> v2:172.21.15.67:3300/0 conn(0x7fad1409ccf0 msgr2=0x7fad14069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 --2- 172.21.15.17:0/494564555 >> v2:172.21.15.67:3300/0 conn(0x7fad1409ccf0 0x7fad14069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 -- 172.21.15.17:0/494564555 >> v1:172.21.15.17:6789/0 conn(0x7fad14069ac0 legacy=0x7fad14138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 monclient: found mon.noname-a 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.528+0000 7fad12ffd640 1 -- 172.21.15.17:0/494564555 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1409d210 con 0x7fad140a5780 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad12ffd640 1 -- 172.21.15.17:0/494564555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad140a5780 msgr2=0x7fad1409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad12ffd640 1 --2- 172.21.15.17:0/494564555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad140a5780 0x7fad1409bc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad14068e70 tx=0x7fad040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48370be58b013183 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad11ffb640 1 -- 172.21.15.17:0/494564555 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04032440 con 0x7fad140a5780 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.531+0000 7fad11ffb640 10 monclient: dump: 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 1 -- 172.21.15.17:0/494564555 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040325a0 con 0x7fad140a5780 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 5 monclient: authenticate success, global_id 44367 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 1 -- 172.21.15.17:0/494564555 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04032980 con 0x7fad140a5780 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 4 set_mon_vals no callback set 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad11ffb640 10 monclient: dump: 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 10 monclient: shutdown 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 -- 172.21.15.17:0/494564555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad140a5780 msgr2=0x7fad1409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 --2- 172.21.15.17:0/494564555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad140a5780 0x7fad1409bc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad14068e70 tx=0x7fad040091d0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 -- 172.21.15.17:0/494564555 shutdown_connections 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 --2- 172.21.15.17:0/494564555 >> v2:172.21.15.67:3300/0 conn(0x7fad1409ccf0 0x7fad14069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 --2- 172.21.15.17:0/494564555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad140a5780 0x7fad1409bc90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 -- 172.21.15.17:0/494564555 >> 172.21.15.17:0/494564555 conn(0x7fad14070e10 msgr2=0x7fad140a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 -- 172.21.15.17:0/494564555 shutdown_connections 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.532+0000 7fad1999d640 1 -- 172.21.15.17:0/494564555 wait complete. 2024-09-15T01:53:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.533+0000 7fad1999d640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.533+0000 7fad1999d640 10 monclient: monmap: 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.533+0000 7fad1999d640 1 Processor -- start 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.533+0000 7fad1999d640 1 -- start start 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.533+0000 7fad1999d640 10 monclient: init 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-c con 0x7fad14069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-a con 0x7fad1409ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140a5780 0x7fad141413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): picked mon.noname-b con 0x7fad140a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14120b80 con 0x7fad1409ccf0 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1409b7a0 con 0x7fad140a5780 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad140a08f0 con 0x7fad14069ac0 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad1999d640 10 monclient(hunting): authenticate will time out at 895.530115s 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39338/0 (socket says 172.21.15.17:39338) 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 -- 172.21.15.17:0/223658379 learned_addr learned my addr 172.21.15.17:0/223658379 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 10 monclient(hunting): get_auth_request con 0x7fad14069ac0 auth_method 0 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35880/0 (socket says 172.21.15.17:35880) 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): get_auth_request con 0x7fad1409ccf0 auth_method 0 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140a5780 0x7fad141413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): get_auth_request con 0x7fad140a5780 auth_method 0 2024-09-15T01:53:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 msgr2=0x7fad1413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 msgr2=0x7fad1413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad12ffd640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient(hunting): handle_auth_done global_id 44370 payload 824 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 msgr2=0x7fad1413de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140a5780 msgr2=0x7fad141413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140a5780 0x7fad141413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.534+0000 7fad127fc640 1 -- 172.21.15.17:0/223658379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04009450 con 0x7fad1409ccf0 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7fad127fc640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 msgr2=0x7fad1413d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7fad127fc640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fad0800e960 tx=0x7fad080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08009540 con 0x7fad1409ccf0 2024-09-15T01:53:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 10 monclient: dump: 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0800cb50 con 0x7fad1409ccf0 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08013470 con 0x7fad1409ccf0 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.536+0000 7fad1999d640 5 monclient: authenticate success, global_id 44370 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 10 monclient: _renew_subs 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad141228c0 con 0x7fad1409ccf0 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7facfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad117fa640 4 set_mon_vals no callback set 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7facfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7facfaffd640 10 monclient: dump: 2024-09-15T01:53:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 10 monclient: _renew_subs 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.537+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1413da80 con 0x7fad1409ccf0 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad08013820 con 0x7fad1409ccf0 2024-09-15T01:53:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad1999d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad1999d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad1999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad140724a0 con 0x7fad1409ccf0 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7facfaffd640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 0x7fad0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7fad0804bf10 con 0x7fad1409ccf0 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad12ffd640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 0x7fad0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.538+0000 7fad12ffd640 10 monclient: get_auth_request con 0x7fad00038740 auth_method 0 2024-09-15T01:53:55.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.541+0000 7fad12ffd640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 0x7fad0003abf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fad04009420 tx=0x7fad04032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.543+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fad080234d0 con 0x7fad1409ccf0 2024-09-15T01:53:55.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.543+0000 7facfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.543+0000 7facfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.535+0000 7f6033909640 10 monclient: get_monmap_and_config 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.535+0000 7f6033909640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.535+0000 7f6033909640 10 monclient: monmap: 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:55.536426+0000 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:55.536426+0000 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.535+0000 7f6033909640 1 Processor -- start 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 1 -- start start 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient: init 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f602c098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f602c0a0370 0x7f602c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): picked mon.noname-c con 0x7f602c0a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f602c098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c098ef0 con 0x7f602c098a50 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c0a0b80 con 0x7f602c098140 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c1328f0 con 0x7f602c0a0370 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6033909640 10 monclient(hunting): authenticate will time out at 868.005561s 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f602c0a0370 0x7f602c098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f60316a5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f602c098140 0x7f602c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57958/0 (socket says 172.21.15.67:57958) 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f602c0a0370 0x7f602c098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56150/0 (socket says 172.21.15.67:56150) 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f60316a5640 1 -- 172.21.15.67:0/3076574061 learned_addr learned my addr 172.21.15.67:0/3076574061 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:55.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): get_auth_request con 0x7f602c0a0370 auth_method 0 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient(hunting): handle_auth_done global_id 4933 payload 274 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 -- 172.21.15.67:0/3076574061 >> v1:172.21.15.67:6789/0 conn(0x7f602c098140 legacy=0x7f602c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 -- 172.21.15.67:0/3076574061 >> v1:172.21.15.17:6789/0 conn(0x7f602c098a50 legacy=0x7f602c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 monclient: found mon.noname-c 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.536+0000 7f6030ea4640 1 -- 172.21.15.67:0/3076574061 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0a00b0 con 0x7f602c0a0370 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.539+0000 7f6030ea4640 1 -- 172.21.15.67:0/3076574061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0a0370 msgr2=0x7f602c098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.539+0000 7f6030ea4640 1 --2- 172.21.15.67:0/3076574061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0a0370 0x7f602c098510 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6020009850 tx=0x7f60200090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=648ed7c929b7ea7e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.539+0000 7f602bfff640 1 -- 172.21.15.67:0/3076574061 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020032440 con 0x7f602c0a0370 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.539+0000 7f602bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.539+0000 7f602bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: dump: 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 5 monclient: authenticate success, global_id 4933 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 1 -- 172.21.15.67:0/3076574061 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60200325a0 con 0x7f602c0a0370 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 4 set_mon_vals no callback set 2024-09-15T01:53:55.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 1 -- 172.21.15.67:0/3076574061 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020032940 con 0x7f602c0a0370 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f602bfff640 10 monclient: dump: 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 10 monclient: shutdown 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 -- 172.21.15.67:0/3076574061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0a0370 msgr2=0x7f602c098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 --2- 172.21.15.67:0/3076574061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0a0370 0x7f602c098510 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6020009850 tx=0x7f60200090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 -- 172.21.15.67:0/3076574061 shutdown_connections 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 --2- 172.21.15.67:0/3076574061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0a0370 0x7f602c098510 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 -- 172.21.15.67:0/3076574061 >> 172.21.15.67:0/3076574061 conn(0x7f602c0a5780 msgr2=0x7f602c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 -- 172.21.15.67:0/3076574061 shutdown_connections 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.540+0000 7f6033909640 1 -- 172.21.15.67:0/3076574061 wait complete. 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 10 monclient: monmap: 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 1 Processor -- start 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 1 -- start start 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 10 monclient: init 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.541+0000 7f6033909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): picked mon.noname-c con 0x7f602c098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): picked mon.noname-b con 0x7f602c098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 0x7f602c1307f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): picked mon.noname-a con 0x7f602c0a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c131e80 con 0x7f602c0a0370 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c133bb0 con 0x7f602c098a50 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c13a210 con 0x7f602c098140 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6033909640 10 monclient(hunting): authenticate will time out at 868.011070s 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56166/0 (socket says 172.21.15.67:56166) 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 -- 172.21.15.67:0/1392928315 learned_addr learned my addr 172.21.15.67:0/1392928315 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): get_auth_request con 0x7f602c098a50 auth_method 0 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 0x7f602c1307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 10 monclient(hunting): get_auth_request con 0x7f602c098140 auth_method 0 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): get_auth_request con 0x7f602c0a0370 auth_method 0 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 msgr2=0x7f602c12fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 msgr2=0x7f602c12fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:55.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f60316a5640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient(hunting): handle_auth_done global_id 4936 payload 824 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 msgr2=0x7f602c12fd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 msgr2=0x7f602c1307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 0x7f602c1307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 monclient: found mon.noname-b 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6030ea4640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020009390 con 0x7f602c098a50 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.542+0000 7f6031ea6640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 0x7f602c1307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:55.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6030ea4640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 msgr2=0x7f602c1302b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6030ea4640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6020009850 tx=0x7f6020032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60200318c0 con 0x7f602c098a50 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: dump: 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.576 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.577 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.577 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6020031a20 con 0x7f602c098a50 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020031de0 con 0x7f602c098a50 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f60297fa640 10 monclient: dump: 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6033909640 5 monclient: authenticate success, global_id 4936 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6033909640 10 monclient: _renew_subs 2024-09-15T01:53:55.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6033909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c099090 con 0x7f602c098a50 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.544+0000 7f602b7fe640 4 set_mon_vals no callback set 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 10 monclient: _renew_subs 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c138dd0 con 0x7f602c098a50 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6020039040 con 0x7f602c098a50 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c04bf60 con 0x7f602c098a50 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.545+0000 7f60297fa640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 0x7f601803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.546+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f602006bd80 con 0x7f602c098a50 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.546+0000 7f60316a5640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 0x7f601803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.546+0000 7f60316a5640 10 monclient: get_auth_request con 0x7f6018038790 auth_method 0 2024-09-15T01:53:55.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.548+0000 7f60316a5640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 0x7f601803ac40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6024005140 tx=0x7f6024009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.550+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f60200306c0 con 0x7f602c098a50 2024-09-15T01:53:55.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.550+0000 7f60297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.550+0000 7f60297fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.715+0000 7f6033909640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.715+0000 7f6033909640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.715+0000 7f6033909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.715+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f602c04bf60 con 0x7f602c098a50 2024-09-15T01:53:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.716+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+70 (secure 0 0 0) 0x7f602003f020 con 0x7f602c098a50 2024-09-15T01:53:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.716+0000 7f60297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.716+0000 7f60297fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7fad1999d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7fad1999d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7fad1999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad1406f4f0 con 0x7fad1409ccf0 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7facfaffd640 1 -- 172.21.15.17:0/223658379 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fad08021030 con 0x7fad1409ccf0 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7facfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:55.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.771+0000 7facfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:55.774 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:55.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 msgr2=0x7fad0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 0x7fad0003abf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fad04009420 tx=0x7fad04032000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 10 monclient: shutdown 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 msgr2=0x7fad1413d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fad0800e960 tx=0x7fad080090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 shutdown_connections 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fad00038740 0x7fad0003abf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fad1409ccf0 0x7fad1413d540 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140a5780 0x7fad141413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 --2- 172.21.15.17:0/223658379 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fad14069ac0 0x7fad1413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 >> 172.21.15.17:0/223658379 conn(0x7fad14070e10 msgr2=0x7fad14098c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 shutdown_connections 2024-09-15T01:53:55.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.773+0000 7fad1999d640 1 -- 172.21.15.17:0/223658379 wait complete. 2024-09-15T01:53:55.785 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:53:55.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:53:55.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:53:55.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f6033909640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f6033909640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f6033909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f602c098ef0 con 0x7f602c098a50 2024-09-15T01:53:55.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f60297fa640 1 -- 172.21.15.67:0/1392928315 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+70 (secure 0 0 0) 0x7f602c098ef0 con 0x7f602c098a50 2024-09-15T01:53:55.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f60297fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:55.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.815+0000 7f60297fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 msgr2=0x7f601803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 0x7f601803ac40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6024005140 tx=0x7f6024009040 comp rx=0 tx=0).stop 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 10 monclient: shutdown 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 msgr2=0x7f602c1302b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6020009850 tx=0x7f6020032000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 shutdown_connections 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6018038790 0x7f601803ac40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f602c0a0370 0x7f602c1307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c098a50 0x7f602c1302b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 --2- 172.21.15.67:0/1392928315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f602c098140 0x7f602c12fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 >> 172.21.15.67:0/1392928315 conn(0x7f602c0a5780 msgr2=0x7f602c0a3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 shutdown_connections 2024-09-15T01:53:55.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:55.817+0000 7f6033909640 1 -- 172.21.15.67:0/1392928315 wait complete. 2024-09-15T01:53:55.828 INFO:tasks.workunit.client.0.smithi067.stdout:bar-asdf 2024-09-15T01:53:55.828 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm foo 2024-09-15T01:53:55.828 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'does not exist' 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.881+0000 7faafd2e8640 10 monclient: get_monmap_and_config 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.881+0000 7faafd2e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.881+0000 7faafd2e8640 10 monclient: monmap: 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:55.882271+0000 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:55.882271+0000 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.883 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:55.884 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 Processor -- start 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 -- start start 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient: init 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faaf809c300 0x7faaf809c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-c con 0x7faaf809c300 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf809a920 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-b con 0x7faaf809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf8099f30 con 0x7faaf809a920 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf806a300 con 0x7faaf809c300 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806a5c0 con 0x7faaf809e970 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faafd2e8640 10 monclient(hunting): authenticate will time out at 895.878594s 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39346/0 (socket says 172.21.15.17:39346) 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 -- 172.21.15.17:0/2587126201 learned_addr learned my addr 172.21.15.17:0/2587126201 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 10 monclient(hunting): get_auth_request con 0x7faaf809e970 auth_method 0 2024-09-15T01:53:55.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 -- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 msgr2=0x7faaf806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 -- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 msgr2=0x7faaf806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2587126201 >> v2:172.21.15.67:3300/0 conn(0x7faaf809c300 0x7faaf809c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request con 0x7faaf809c300 auth_method 0 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.882+0000 7faaf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_done global_id 4939 payload 274 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 msgr2=0x7faaf806f240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2587126201 >> v1:172.21.15.67:6789/0 conn(0x7faaf809a920 legacy=0x7faaf806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 1 monclient: found mon.noname-c 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.883+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2587126201 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf809cc10 con 0x7faaf809c300 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2587126201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 msgr2=0x7faaf809c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf809c6d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7faae8009e60 tx=0x7faae8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23e5ade7e8626389 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 1 -- 172.21.15.17:0/2587126201 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8007760 con 0x7faaf809c300 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: dump: 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.896 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 1 -- 172.21.15.17:0/2587126201 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae802d050 con 0x7faaf809c300 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 1 -- 172.21.15.17:0/2587126201 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8007af0 con 0x7faaf809c300 2024-09-15T01:53:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faaf5ffb640 10 monclient: dump: 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 5 monclient: authenticate success, global_id 4939 2024-09-15T01:53:55.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 4 set_mon_vals no callback set 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 10 monclient: shutdown 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 -- 172.21.15.17:0/2587126201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 msgr2=0x7faaf809c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf809c6d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7faae8009e60 tx=0x7faae8009780 comp rx=0 tx=0).stop 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 -- 172.21.15.17:0/2587126201 shutdown_connections 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809e970 0x7faaf806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2587126201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf809c6d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 -- 172.21.15.17:0/2587126201 >> 172.21.15.17:0/2587126201 conn(0x7faaf80a2f70 msgr2=0x7faaf80a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 -- 172.21.15.17:0/2587126201 shutdown_connections 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.886+0000 7faafd2e8640 1 -- 172.21.15.17:0/2587126201 wait complete. 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.887+0000 7faafd2e8640 10 monclient: build_initial_monmap 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.887+0000 7faafd2e8640 10 monclient: monmap: 2024-09-15T01:53:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.887+0000 7faafd2e8640 1 Processor -- start 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.887+0000 7faafd2e8640 1 -- start start 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.887+0000 7faafd2e8640 10 monclient: init 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:55.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-c con 0x7faaf809a920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf8131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-b con 0x7faaf809c300 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf809e970 0x7faaf8097aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): picked mon.noname-a con 0x7faaf809e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806ea30 con 0x7faaf809e970 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80995f0 con 0x7faaf809c300 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf811eee0 con 0x7faaf809a920 2024-09-15T01:53:55.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faafd2e8640 10 monclient(hunting): authenticate will time out at 895.884359s 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39360/0 (socket says 172.21.15.17:39360) 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 learned_addr learned my addr 172.21.15.17:0/2479498906 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request con 0x7faaf809a920 auth_method 0 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf809e970 0x7faaf8097aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): get_auth_request con 0x7faaf809e970 auth_method 0 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf8131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): get_auth_request con 0x7faaf809c300 auth_method 0 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 msgr2=0x7faaf8141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 msgr2=0x7faaf8141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient(hunting): handle_auth_done global_id 4942 payload 824 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 msgr2=0x7faaf8141220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faaf809a920 0x7faaf8141220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf809e970 msgr2=0x7faaf8097aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf809e970 0x7faaf8097aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 monclient: found mon.noname-b 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf67fc640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae8009a20 con 0x7faaf809c300 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.888+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf809e970 0x7faaf8097aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaf67fc640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 msgr2=0x7faaf8131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaf67fc640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf8131dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faae000c4e0 tx=0x7faae0007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00117a0 con 0x7faaf809c300 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: dump: 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0011920 con 0x7faaf809c300 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 5 monclient: authenticate success, global_id 4942 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 10 monclient: _renew_subs 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf811f1c0 con 0x7faaf809c300 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0011d20 con 0x7faaf809c300 2024-09-15T01:53:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaeeffd640 10 monclient: dump: 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faaf4ff9640 4 set_mon_vals no callback set 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 10 monclient: _renew_subs 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.891+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8141ff0 con 0x7faaf809c300 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faafd2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 0x7faaf8097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8063b90 con 0x7faaf8141760 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faae0012630 con 0x7faaf809c300 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 0x7faaf8097fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient: get_auth_request con 0x7faaf8141760 auth_method 0 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaeeffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faad8038740 0x7faad803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_done global_id 44379 payload 824 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7faae004b910 con 0x7faaf809c300 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faaf811f3e0 con 0x7faaf8141760 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faad8038740 0x7faad803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.892+0000 7faaf77fe640 10 monclient: get_auth_request con 0x7faad8038740 auth_method 0 2024-09-15T01:53:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.894+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 msgr2=0x7faaf8097fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.894+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 0x7faaf8097fe0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faae8009e60 tx=0x7faae8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae803db30 con 0x7faaf8141760 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 10 monclient: dump: 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.912 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaf77fe640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faad8038740 0x7faad803abf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faaf0006a70 tx=0x7faaf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faae803dc90 con 0x7faaf8141760 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 msgr2=0x7faaf8097fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 0x7faaf8097fe0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faae8009e60 tx=0x7faae8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 msgr2=0x7faaf8097fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.895+0000 7faaeeffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8141760 0x7faaf8097fe0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faae8009e60 tx=0x7faae8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faafd2e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 0x7faaf8098520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faafd2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80724a0 con 0x7faaf8132300 2024-09-15T01:53:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 0x7faaf8098520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient: get_auth_request con 0x7faaf8132300 auth_method 0 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 10 monclient(hunting): handle_auth_done global_id 44382 payload 824 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.901+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faaf8067220 con 0x7faaf8132300 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaf6ffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 msgr2=0x7faaf8098520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaf6ffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 0x7faaf8098520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faae8044000 tx=0x7faae803f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8051a00 con 0x7faaf8132300 2024-09-15T01:53:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 10 monclient: dump: 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7faae8031400 con 0x7faaf8132300 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.903+0000 7faaeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 msgr2=0x7faaf8098520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaeeffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 0x7faaf8098520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faae8044000 tx=0x7faae803f040 comp rx=0 tx=0).stop 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaeeffd640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 msgr2=0x7faaf8098520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaeeffd640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faaf8132300 0x7faaf8098520 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faae8044000 tx=0x7faae803f040 comp rx=0 tx=0).stop 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:53:55.916 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:55.917 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:55.918 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:55.919 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:55.920 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.921 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:55.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:55.925 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:55.926 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:55.927 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:55.928 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:55.929 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faad8038740 msgr2=0x7faad803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faad8038740 0x7faad803abf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faaf0006a70 tx=0x7faaf0008040 comp rx=0 tx=0).stop 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 10 monclient: shutdown 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 msgr2=0x7faaf8131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 --2- 172.21.15.17:0/2479498906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf809c300 0x7faaf8131dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faae000c4e0 tx=0x7faae0007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaf77fe640 1 -- 172.21.15.17:0/2479498906 reap_dead start 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faaf77fe640 1 -- 172.21.15.17:0/2479498906 reap_dead start 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 shutdown_connections 2024-09-15T01:53:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 >> 172.21.15.17:0/2479498906 conn(0x7faaf80a2f70 msgr2=0x7faaf806ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 shutdown_connections 2024-09-15T01:53:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:55.904+0000 7faafd2e8640 1 -- 172.21.15.17:0/2479498906 wait complete. 2024-09-15T01:53:55.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:55.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:53:55.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 10 monclient: monmap: 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:56.027862+0000 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:56.027862+0000 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:56.030 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 1 Processor -- start 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 1 -- start start 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.027+0000 7fd9c5a6f640 10 monclient: init 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9c0131b20 0x7fd9c009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c0131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9c009f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00a0040 con 0x7fd9c009f1e0 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c009bff0 con 0x7fd9c0131b20 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00a14a0 con 0x7fd9c00a0280 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9c5a6f640 10 monclient(hunting): authenticate will time out at 896.024361s 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd9c009f1e0 0x7fd9c0068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52222/0 (socket says 172.21.15.17:52222) 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bffff640 1 -- 172.21.15.17:0/1117274415 learned_addr learned my addr 172.21.15.17:0/1117274415 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 10 monclient(hunting): get_auth_request con 0x7fd9c00a0280 auth_method 0 2024-09-15T01:53:56.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1117274415 >> v2:172.21.15.67:3300/0 conn(0x7fd9c0131b20 0x7fd9c009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd9c0131b20 auth_method 0 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 1 -- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 msgr2=0x7fd9c0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 1 -- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 msgr2=0x7fd9c0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:56.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9beffd640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_done global_id 4951 payload 274 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 msgr2=0x7fd9c0131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1117274415 >> v1:172.21.15.17:6789/0 conn(0x7fd9c009f1e0 legacy=0x7fd9c0068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 monclient: found mon.noname-c 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.028+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1117274415 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0132430 con 0x7fd9c0131b20 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.031+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1117274415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 msgr2=0x7fd9c009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.031+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c009eca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009850 tx=0x7fd9b00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bf1dcccb7edb31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 1 -- 172.21.15.17:0/1117274415 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b002adf0 con 0x7fd9c0131b20 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: dump: 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 5 monclient: authenticate success, global_id 4951 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 1 -- 172.21.15.17:0/1117274415 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b0032440 con 0x7fd9c0131b20 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 4 set_mon_vals no callback set 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 1 -- 172.21.15.17:0/1117274415 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00327e0 con 0x7fd9c0131b20 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9be7fc640 10 monclient: dump: 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 10 monclient: shutdown 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1117274415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 msgr2=0x7fd9c009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c009eca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009850 tx=0x7fd9b00090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1117274415 shutdown_connections 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c00a0280 0x7fd9c0131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1117274415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c009eca0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1117274415 >> 172.21.15.17:0/1117274415 conn(0x7fd9c0071270 msgr2=0x7fd9c00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1117274415 shutdown_connections 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.032+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1117274415 wait complete. 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.033+0000 7fd9c5a6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.033+0000 7fd9c5a6f640 10 monclient: monmap: 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.033+0000 7fd9c5a6f640 1 Processor -- start 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.033+0000 7fd9c5a6f640 1 -- start start 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.033+0000 7fd9c5a6f640 10 monclient: init 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9c00a0280 0x7fd9c012ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9c00a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c0098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c0131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c0141620 con 0x7fd9c00a0280 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c011bcf0 con 0x7fd9c0131b20 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c009ab30 con 0x7fd9c009f1e0 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9c5a6f640 10 monclient(hunting): authenticate will time out at 896.030165s 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9c00a0280 0x7fd9c012ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39384/0 (socket says 172.21.15.17:39384) 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 learned_addr learned my addr 172.21.15.17:0/1702332481 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd9c009f1e0 auth_method 0 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): get_auth_request con 0x7fd9c00a0280 auth_method 0 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c0098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 msgr2=0x7fd9c01410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 msgr2=0x7fd9c01410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): get_auth_request con 0x7fd9c0131b20 auth_method 0 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient(hunting): handle_auth_done global_id 4954 payload 824 2024-09-15T01:53:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 msgr2=0x7fd9c01410e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009f1e0 0x7fd9c01410e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9c00a0280 msgr2=0x7fd9c012ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9c00a0280 0x7fd9c012ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 monclient: found mon.noname-b 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b0009390 con 0x7fd9c0131b20 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.034+0000 7fd9beffd640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9c00a0280 0x7fd9c012ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.036+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 msgr2=0x7fd9c0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9bffff640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c0098a70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800cbc0 tx=0x7fd9b80052b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b800a850 con 0x7fd9c0131b20 2024-09-15T01:53:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: dump: 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 5 monclient: authenticate success, global_id 4954 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b800a9d0 con 0x7fd9c0131b20 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 10 monclient: _renew_subs 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c0131850 con 0x7fd9c0131b20 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b800ad90 con 0x7fd9c0131b20 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9a77fe640 10 monclient: dump: 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 10 monclient: _renew_subs 2024-09-15T01:53:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c0141e20 con 0x7fd9c0131b20 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9bdffb640 4 set_mon_vals no callback set 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.037+0000 7fd9c5a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c0063a40 con 0x7fd9c009b180 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd9b8007300 con 0x7fd9c0131b20 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 10 monclient: get_auth_request con 0x7fd9c009b180 auth_method 0 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9a77fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd9ac038790 0x7fd9ac03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7fd9b804bc90 con 0x7fd9c0131b20 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9beffd640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd9ac038790 0x7fd9ac03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.038+0000 7fd9beffd640 10 monclient: get_auth_request con 0x7fd9ac038790 auth_method 0 2024-09-15T01:53:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.041+0000 7fd9beffd640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd9ac038790 0x7fd9ac03ac40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4006fd0 tx=0x7fd9b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.201 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.086+0000 7f02127fc640 1 -- 172.21.15.67:0/4294723601 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v56) v1 ==== 103+0+0 (secure 0 0 0) 0x7f022411e090 con 0x7f0224098140 2024-09-15T01:53:56.201 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.086+0000 7f02127fc640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-15T01:53:56.201 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.188+0000 7f02127fc640 1 -- 172.21.15.67:0/4294723601 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v56) v1 ==== 103+0+0 (secure 0 0 0) 0x7f022411e090 con 0x7f0224098140 2024-09-15T01:53:56.201 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.188+0000 7f02127fc640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T01:53:56.201 INFO:tasks.workunit.client.0.smithi067.stdout:rule foo does not exist 2024-09-15T01:53:56.202 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm bar 2024-09-15T01:53:56.202 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'does not exist' 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 10 monclient: get_auth_request con 0x7fd9c009b180 auth_method 0 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.239+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:53:56.573 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.459+0000 7f7326ffd640 1 -- 172.21.15.67:0/2511135490 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v56) v1 ==== 103+0+0 (secure 0 0 0) 0x7f732001dbb0 con 0x7f733006a640 2024-09-15T01:53:56.573 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.460+0000 7f7326ffd640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-15T01:53:56.574 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.560+0000 7f7326ffd640 1 -- 172.21.15.67:0/2511135490 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v56) v1 ==== 103+0+0 (secure 0 0 0) 0x7f732001dd90 con 0x7f733006a640 2024-09-15T01:53:56.574 INFO:tasks.workunit.client.0.smithi067.stdout:2024-09-15T01:53:56.560+0000 7f7326ffd640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-15T01:53:56.574 INFO:tasks.workunit.client.0.smithi067.stdout:rule bar does not exist 2024-09-15T01:53:56.574 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 10 monclient: get_auth_request con 0x7fd9c009b180 auth_method 0 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:56.640+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:53:56.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.648+0000 7f5c638dd640 10 monclient: get_monmap_and_config 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.648+0000 7f5c638dd640 10 monclient: build_initial_monmap 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.648+0000 7f5c638dd640 10 monclient: monmap: 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:56.649762+0000 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:56.649762+0000 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:56.651 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:56.652 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:56.652 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:56.652 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 Processor -- start 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 -- start start 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient: init 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:53:56.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c5c131cf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5c5c1323f0 0x7f5c5c137020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-a con 0x7f5c5c1323f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c137560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c096a70 con 0x7f5c5c131cf0 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c0a1950 con 0x7f5c5c1323f0 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c1320c0 con 0x7f5c5c137560 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:56.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.649+0000 7f5c638dd640 10 monclient(hunting): authenticate will time out at 869.118940s 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61679640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c5c131cf0 0x7f5c5c098420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57982/0 (socket says 172.21.15.67:57982) 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61679640 1 -- 172.21.15.67:0/46960033 learned_addr learned my addr 172.21.15.67:0/46960033 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 --2- 172.21.15.67:0/46960033 >> v2:172.21.15.17:3300/0 conn(0x7f5c5c1323f0 0x7f5c5c137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): get_auth_request con 0x7f5c5c1323f0 auth_method 0 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 10 monclient(hunting): get_auth_request con 0x7f5c5c137560 auth_method 0 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 1 -- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 msgr2=0x7f5c5c139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 1 -- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 msgr2=0x7f5c5c139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c61e7a640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient(hunting): handle_auth_done global_id 44397 payload 274 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 -- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 msgr2=0x7f5c5c139940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:53:56.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 -- 172.21.15.67:0/46960033 >> v1:172.21.15.67:6789/0 conn(0x7f5c5c131cf0 legacy=0x7f5c5c098420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient: _finish_hunting 0 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 monclient: found mon.noname-a 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.650+0000 7f5c60e78640 1 -- 172.21.15.67:0/46960033 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c132ad0 con 0x7f5c5c1323f0 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c60e78640 1 -- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c1323f0 msgr2=0x7f5c5c137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c60e78640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c137020 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009850 tx=0x7f5c4c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76659197b2de9fa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 1 -- 172.21.15.67:0/46960033 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c02adf0 con 0x7f5c5c1323f0 2024-09-15T01:53:56.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 10 monclient: dump: 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.653+0000 7f5c53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 5 monclient: authenticate success, global_id 44397 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 1 -- 172.21.15.67:0/46960033 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c032440 con 0x7f5c5c1323f0 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 1 -- 172.21.15.67:0/46960033 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0327e0 con 0x7f5c5c1323f0 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c53fff640 10 monclient: dump: 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.664 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.665 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 4 set_mon_vals no callback set 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 10 monclient: shutdown 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 -- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c1323f0 msgr2=0x7f5c5c137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c137020 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009850 tx=0x7f5c4c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 -- 172.21.15.67:0/46960033 shutdown_connections 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c137560 0x7f5c5c139940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 --2- 172.21.15.67:0/46960033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c137020 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 -- 172.21.15.67:0/46960033 >> 172.21.15.67:0/46960033 conn(0x7f5c5c071440 msgr2=0x7f5c5c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 -- 172.21.15.67:0/46960033 shutdown_connections 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.654+0000 7f5c638dd640 1 -- 172.21.15.67:0/46960033 wait complete. 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.655+0000 7f5c638dd640 10 monclient: build_initial_monmap 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.655+0000 7f5c638dd640 10 monclient: monmap: 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:56.672 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.655+0000 7f5c638dd640 1 Processor -- start 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.655+0000 7f5c638dd640 1 -- start start 2024-09-15T01:53:56.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.655+0000 7f5c638dd640 10 monclient: init 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5c131cf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c1323f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 0x7f5c5c1343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): picked mon.noname-a con 0x7f5c5c137560 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c130940 con 0x7f5c5c137560 2024-09-15T01:53:56.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0a0b30 con 0x7f5c5c1323f0 2024-09-15T01:53:56.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:56.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0a32e0 con 0x7f5c5c131cf0 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c638dd640 10 monclient(hunting): authenticate will time out at 869.125191s 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56194/0 (socket says 172.21.15.67:56194) 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 -- 172.21.15.67:0/3258649812 learned_addr learned my addr 172.21.15.67:0/3258649812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): get_auth_request con 0x7f5c5c1323f0 auth_method 0 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 10 monclient(hunting): get_auth_request con 0x7f5c5c131cf0 auth_method 0 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 0x7f5c5c1343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 10 monclient(hunting): get_auth_request con 0x7f5c5c137560 auth_method 0 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:56.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 msgr2=0x7f5c5c141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 msgr2=0x7f5c5c141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61679640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient(hunting): handle_auth_done global_id 4963 payload 824 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 msgr2=0x7f5c5c141c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 msgr2=0x7f5c5c1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 0x7f5c5c1343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:56.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient: _finish_hunting 0 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 monclient: found mon.noname-b 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c60e78640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009390 con 0x7f5c5c1323f0 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.656+0000 7f5c61e7a640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 0x7f5c5c1343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.658+0000 7f5c60e78640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 msgr2=0x7f5c5c142140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.658+0000 7f5c60e78640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009850 tx=0x7f5c4c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c032ea0 con 0x7f5c5c1323f0 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:56.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: dump: 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: _finish_auth 0 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 5 monclient: authenticate success, global_id 4963 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c03b820 con 0x7f5c5c1323f0 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _renew_subs 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c03bbc0 con 0x7f5c5c1323f0 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c11c250 con 0x7f5c5c1323f0 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 10 monclient: dump: 2024-09-15T01:53:56.682 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c537fe640 4 set_mon_vals no callback set 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _renew_subs 2024-09-15T01:53:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c11c810 con 0x7f5c5c1323f0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5c4c039040 con 0x7f5c5c1323f0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.659+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c11c490 con 0x7f5c5c1323f0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.660+0000 7f5c517fa640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 0x7f5c4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.660+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f5c4c06bd90 con 0x7f5c5c1323f0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.660+0000 7f5c61679640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 0x7f5c4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.660+0000 7f5c61679640 10 monclient: get_auth_request con 0x7f5c40038740 auth_method 0 2024-09-15T01:53:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.662+0000 7f5c61679640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 0x7f5c4003abf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5c58006fd0 tx=0x7f5c58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:56.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.664+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5c4c04b300 con 0x7f5c5c1323f0 2024-09-15T01:53:56.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.664+0000 7f5c517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:56.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.664+0000 7f5c517fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:56.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.829+0000 7f5c638dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:56.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.829+0000 7f5c638dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:56.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.829+0000 7f5c638dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:56.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:56.829+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f5c5c0639e0 con 0x7f5c5c1323f0 2024-09-15T01:53:57.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.014+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v57) v1 ==== 110+0+0 (secure 0 0 0) 0x7f5c4c03f070 con 0x7f5c5c1323f0 2024-09-15T01:53:57.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.014+0000 7f5c517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.014+0000 7f5c517fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:57.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.114+0000 7f5c638dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.114+0000 7f5c638dd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.114+0000 7f5c638dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.114+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f5c5c0639e0 con 0x7f5c5c1323f0 2024-09-15T01:53:57.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.115+0000 7f5c517fa640 1 -- 172.21.15.67:0/3258649812 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v57) v1 ==== 134+0+0 (secure 0 0 0) 0x7f5c4c037030 con 0x7f5c5c1323f0 2024-09-15T01:53:57.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.115+0000 7f5c517fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.115+0000 7f5c517fa640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-15T01:53:57.118 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'foo' 2024-09-15T01:53:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 msgr2=0x7f5c4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 0x7f5c4003abf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5c58006fd0 tx=0x7f5c58008040 comp rx=0 tx=0).stop 2024-09-15T01:53:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 10 monclient: shutdown 2024-09-15T01:53:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 msgr2=0x7f5c5c142140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009850 tx=0x7f5c4c032000 comp rx=0 tx=0).stop 2024-09-15T01:53:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 shutdown_connections 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c40038740 0x7f5c4003abf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c5c137560 0x7f5c5c1343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c5c1323f0 0x7f5c5c142140 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 --2- 172.21.15.67:0/3258649812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c5c131cf0 0x7f5c5c141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 >> 172.21.15.67:0/3258649812 conn(0x7f5c5c071440 msgr2=0x7f5c5c097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 shutdown_connections 2024-09-15T01:53:57.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.117+0000 7f5c638dd640 1 -- 172.21.15.67:0/3258649812 wait complete. 2024-09-15T01:53:57.128 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.204+0000 7f159155d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.204+0000 7f159155d640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.204+0000 7f159155d640 10 monclient: monmap: 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:57.205537+0000 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:57.205537+0000 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.207 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.204+0000 7f159155d640 1 Processor -- start 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 1 -- start start 2024-09-15T01:53:57.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient: init 2024-09-15T01:53:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:53:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f158c09ce90 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c09d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): picked mon.noname-b con 0x7f158c09d260 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f158c137090 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158c0954d0 con 0x7f158c09ce90 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158c06f300 con 0x7f158c137090 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c098490 con 0x7f158c09d260 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f159155d640 10 monclient(hunting): authenticate will time out at 869.674518s 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f158c137090 0x7f158c139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57996/0 (socket says 172.21.15.67:57996) 2024-09-15T01:53:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158b7fe640 1 -- 172.21.15.67:0/946414460 learned_addr learned my addr 172.21.15.67:0/946414460 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 1 --2- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c09d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): get_auth_request con 0x7f158c09d260 auth_method 0 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.205+0000 7f158a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 10 monclient(hunting): handle_auth_done global_id 54101 payload 274 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 1 -- 172.21.15.67:0/946414460 >> v1:172.21.15.67:6789/0 conn(0x7f158c137090 legacy=0x7f158c139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 1 -- 172.21.15.67:0/946414460 >> v1:172.21.15.17:6789/0 conn(0x7f158c09ce90 legacy=0x7f158c0958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 1 monclient: found mon.noname-b 2024-09-15T01:53:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.206+0000 7f158a7fc640 1 -- 172.21.15.67:0/946414460 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c06ed60 con 0x7f158c09d260 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f158a7fc640 1 -- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 msgr2=0x7f158c09d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f158a7fc640 1 --2- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c09d630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1580009880 tx=0x7f15800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa89e804dd9a5c91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 1 -- 172.21.15.67:0/946414460 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580032440 con 0x7f158c09d260 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: dump: 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.219 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 1 -- 172.21.15.67:0/946414460 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15800325a0 con 0x7f158c09d260 2024-09-15T01:53:57.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 5 monclient: authenticate success, global_id 54101 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 1 -- 172.21.15.67:0/946414460 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580032940 con 0x7f158c09d260 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 4 set_mon_vals no callback set 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f1589ffb640 10 monclient: dump: 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 10 monclient: shutdown 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 -- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 msgr2=0x7f158c09d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 --2- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c09d630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1580009880 tx=0x7f15800090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 -- 172.21.15.67:0/946414460 shutdown_connections 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 --2- 172.21.15.67:0/946414460 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c09d630 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 -- 172.21.15.67:0/946414460 >> 172.21.15.67:0/946414460 conn(0x7f158c0a1f20 msgr2=0x7f158c09df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 -- 172.21.15.67:0/946414460 shutdown_connections 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.209+0000 7f159155d640 1 -- 172.21.15.67:0/946414460 wait complete. 2024-09-15T01:53:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.210+0000 7f159155d640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.210+0000 7f159155d640 10 monclient: monmap: 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:57.223 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.210+0000 7f159155d640 1 Processor -- start 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.210+0000 7f159155d640 1 -- start start 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.210+0000 7f159155d640 10 monclient: init 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 0x7f158c09b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): picked mon.noname-a con 0x7f158c09ce90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c12c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): picked mon.noname-c con 0x7f158c09d260 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): picked mon.noname-b con 0x7f158c137090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c12f770 con 0x7f158c09ce90 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c130810 con 0x7f158c137090 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c136c50 con 0x7f158c09d260 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f159155d640 10 monclient(hunting): authenticate will time out at 869.680407s 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56200/0 (socket says 172.21.15.67:56200) 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 -- 172.21.15.67:0/3579167714 learned_addr learned my addr 172.21.15.67:0/3579167714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): get_auth_request con 0x7f158c137090 auth_method 0 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 0x7f158c09b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158a7fc640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c12c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158a7fc640 10 monclient(hunting): get_auth_request con 0x7f158c09d260 auth_method 0 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 10 monclient(hunting): get_auth_request con 0x7f158c09ce90 auth_method 0 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient(hunting): handle_auth_done global_id 4966 payload 824 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 -- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 msgr2=0x7f158c12c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c12c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 -- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 msgr2=0x7f158c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 0x7f158c09b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158b7fe640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580009390 con 0x7f158c137090 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.211+0000 7f158affd640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 0x7f158c09b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f158b7fe640 1 -- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 msgr2=0x7f158c12ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f158b7fe640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f158400cbf0 tx=0x7f1584005340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158400ce10 con 0x7f158c137090 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: dump: 2024-09-15T01:53:57.228 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158400aa60 con 0x7f158c137090 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 5 monclient: authenticate success, global_id 4966 2024-09-15T01:53:57.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158400ae20 con 0x7f158c137090 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 10 monclient: _renew_subs 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158c12f490 con 0x7f158c137090 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f1572ffd640 10 monclient: dump: 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.230 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f15897fa640 4 set_mon_vals no callback set 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 10 monclient: _renew_subs 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.214+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158c12dce0 con 0x7f158c137090 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f159155d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f159155d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f159155d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158c0639e0 con 0x7f158c137090 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1584007300 con 0x7f158c137090 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f1572ffd640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 0x7f157803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f158404be20 con 0x7f158c137090 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f158affd640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 0x7f157803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.215+0000 7f158affd640 10 monclient: get_auth_request con 0x7f1578038740 auth_method 0 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.218+0000 7f158affd640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 0x7f157803abf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f157c002860 tx=0x7f157c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.219+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f158401b400 con 0x7f158c137090 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.219+0000 7f1572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.219+0000 7f1572ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.386+0000 7f159155d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.386+0000 7f159155d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.386+0000 7f159155d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.386+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f158c0676e0 con 0x7f158c137090 2024-09-15T01:53:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.387+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v57) v1 ==== 134+0+0 (secure 0 0 0) 0x7f1584010300 con 0x7f158c137090 2024-09-15T01:53:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.387+0000 7f1572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.387+0000 7f1572ffd640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient: get_auth_request con 0x7fd9c009b180 auth_method 0 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_done global_id 54107 payload 824 2024-09-15T01:53:57.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.441+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9c00724a0 con 0x7fd9c009b180 2024-09-15T01:53:57.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.443+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.443+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009130 tx=0x7fd9b0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b003c9f0 con 0x7fd9c009b180 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 10 monclient: dump: 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.446 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd9b003cb50 con 0x7fd9c009b180 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:57.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009130 tx=0x7fd9b0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:57.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 msgr2=0x7fd9c0098fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:57.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.444+0000 7fd9a77fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c009b180 0x7fd9c0098fb0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009130 tx=0x7fd9b0032000 comp rx=0 tx=0).stop 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.450+0000 7fd9c5a6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 0x7fd9c00994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9c5a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c011c190 con 0x7fd9c012f110 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 0x7fd9c00994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient: get_auth_request con 0x7fd9c012f110 auth_method 0 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 10 monclient(hunting): handle_auth_done global_id 54110 payload 824 2024-09-15T01:53:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.451+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9c00a09b0 con 0x7fd9c012f110 2024-09-15T01:53:57.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9bf7fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 msgr2=0x7fd9c00994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9bf7fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 0x7fd9c00994f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9c011c160 tx=0x7fd9b0040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b003b940 con 0x7fd9c012f110 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.453+0000 7fd9a77fe640 10 monclient: dump: 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.456 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7fd9b003baa0 con 0x7fd9c012f110 2024-09-15T01:53:57.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:57.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:57.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 msgr2=0x7fd9c00994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 0x7fd9c00994f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9c011c160 tx=0x7fd9b0040040 comp rx=0 tx=0).stop 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 msgr2=0x7fd9c00994f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9a77fe640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd9c012f110 0x7fd9c00994f0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9c011c160 tx=0x7fd9b0040040 comp rx=0 tx=0).stop 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:57.458 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:57.459 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:57.460 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.461 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:57.462 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.463 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:57.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:57.465 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.466 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.467 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.468 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:57.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:57.472 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:57.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd9ac038790 msgr2=0x7fd9ac03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd9ac038790 0x7fd9ac03ac40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4006fd0 tx=0x7fd9b4008040 comp rx=0 tx=0).stop 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 10 monclient: shutdown 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 msgr2=0x7fd9c0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 --2- 172.21.15.17:0/1702332481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c0131b20 0x7fd9c0098a70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800cbc0 tx=0x7fd9b80052b0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 reap_dead start 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9bffff640 1 -- 172.21.15.17:0/1702332481 reap_dead start 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 shutdown_connections 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 >> 172.21.15.17:0/1702332481 conn(0x7fd9c0071270 msgr2=0x7fd9c00a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.454+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 shutdown_connections 2024-09-15T01:53:57.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.455+0000 7fd9c5a6f640 1 -- 172.21.15.17:0/1702332481 wait complete. 2024-09-15T01:53:57.475 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:57.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:53:57.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:53:57.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.489+0000 7f159155d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.489+0000 7f159155d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.489+0000 7f159155d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:57.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.489+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f158c12d1f0 con 0x7f158c137090 2024-09-15T01:53:57.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.490+0000 7f1572ffd640 1 -- 172.21.15.67:0/3579167714 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v57) v1 ==== 134+0+0 (secure 0 0 0) 0x7f158402a0b0 con 0x7f158c137090 2024-09-15T01:53:57.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.490+0000 7f1572ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T01:53:57.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.490+0000 7f1572ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-15T01:53:57.493 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'foo' 2024-09-15T01:53:57.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 msgr2=0x7f157803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 0x7f157803abf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f157c002860 tx=0x7f157c009040 comp rx=0 tx=0).stop 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 10 monclient: shutdown 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 msgr2=0x7f158c12ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f158400cbf0 tx=0x7f1584005340 comp rx=0 tx=0).stop 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 shutdown_connections 2024-09-15T01:53:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1578038740 0x7f157803abf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158c137090 0x7f158c12ccb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f158c09d260 0x7f158c12c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 --2- 172.21.15.67:0/3579167714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f158c09ce90 0x7f158c09b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 >> 172.21.15.67:0/3579167714 conn(0x7f158c0a1f20 msgr2=0x7f158c0a0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 shutdown_connections 2024-09-15T01:53:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.492+0000 7f159155d640 1 -- 172.21.15.67:0/3579167714 wait complete. 2024-09-15T01:53:57.503 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-15T01:53:57.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.577+0000 7fbc23e2b640 10 monclient: get_monmap_and_config 2024-09-15T01:53:57.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.577+0000 7fbc23e2b640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.577+0000 7fbc23e2b640 10 monclient: monmap: 2024-09-15T01:53:57.580 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:57.578373+0000 2024-09-15T01:53:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:57.578373+0000 2024-09-15T01:53:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.582 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:57.582 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:57.582 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:57.582 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:57.583 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:57.583 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 Processor -- start 2024-09-15T01:53:57.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 -- start start 2024-09-15T01:53:57.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient: init 2024-09-15T01:53:57.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T01:53:57.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc1c070a40 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:57.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc1c06a550 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 0x7fbc1c13a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1c138140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c06eb80 con 0x7fbc1c070a40 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c09dd00 con 0x7fbc1c06a550 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c09bc40 con 0x7fbc1c138140 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.578+0000 7fbc23e2b640 10 monclient(hunting): authenticate will time out at 897.574883s 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc21bc7640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbc1c070a40 0x7fbc1c09ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52230/0 (socket says 172.21.15.17:52230) 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc21bc7640 1 -- 172.21.15.17:0/3972192641 learned_addr learned my addr 172.21.15.17:0/3972192641 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 1 --2- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 0x7fbc1c13a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): get_auth_request con 0x7fbc1c138140 auth_method 0 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient(hunting): handle_auth_done global_id 54113 payload 274 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 1 -- 172.21.15.17:0/3972192641 >> v1:172.21.15.67:6789/0 conn(0x7fbc1c06a550 legacy=0x7fbc1c09b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 1 -- 172.21.15.17:0/3972192641 >> v1:172.21.15.17:6789/0 conn(0x7fbc1c070a40 legacy=0x7fbc1c09ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 1 monclient: found mon.noname-b 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.579+0000 7fbc223c8640 1 -- 172.21.15.17:0/3972192641 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c070f30 con 0x7fbc1c138140 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc223c8640 1 -- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 msgr2=0x7fbc1c13a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc223c8640 1 --2- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 0x7fbc1c13a520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009e90 tx=0x7fbc14009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621e47525b414097 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 1 -- 172.21.15.17:0/3972192641 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14007840 con 0x7fbc1c138140 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: dump: 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 1 -- 172.21.15.17:0/3972192641 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1402d050 con 0x7fbc1c138140 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 5 monclient: authenticate success, global_id 54113 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:57.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 4 set_mon_vals no callback set 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 1 -- 172.21.15.17:0/3972192641 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14007bd0 con 0x7fbc1c138140 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc20bc5640 10 monclient: dump: 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.604 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 10 monclient: shutdown 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 -- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 msgr2=0x7fbc1c13a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 0x7fbc1c13a520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009e90 tx=0x7fbc14009780 comp rx=0 tx=0).stop 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 -- 172.21.15.17:0/3972192641 shutdown_connections 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/3972192641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c138140 0x7fbc1c13a520 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 -- 172.21.15.17:0/3972192641 >> 172.21.15.17:0/3972192641 conn(0x7fbc1c067ae0 msgr2=0x7fbc1c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 -- 172.21.15.17:0/3972192641 shutdown_connections 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.582+0000 7fbc23e2b640 1 -- 172.21.15.17:0/3972192641 wait complete. 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.583+0000 7fbc23e2b640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.583+0000 7fbc23e2b640 10 monclient: monmap: 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.583+0000 7fbc23e2b640 1 Processor -- start 2024-09-15T01:53:57.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.583+0000 7fbc23e2b640 1 -- start start 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.583+0000 7fbc23e2b640 10 monclient: init 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c06a550 0x7fbc1c11e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1c06a550 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 0x7fbc1c11eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1c070a40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc1c138140 0x7fbc1c11f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1c138140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1206c0 con 0x7fbc1c138140 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c06deb0 con 0x7fbc1c06a550 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c06d0c0 con 0x7fbc1c070a40 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc23e2b640 10 monclient(hunting): authenticate will time out at 897.580277s 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 0x7fbc1c11eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc1c138140 0x7fbc1c11f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c06a550 0x7fbc1c11e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 0x7fbc1c11eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39462/0 (socket says 172.21.15.17:39462) 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc1c138140 0x7fbc1c11f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35916/0 (socket says 172.21.15.17:35916) 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 -- 172.21.15.17:0/4272343824 learned_addr learned my addr 172.21.15.17:0/4272343824 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): get_auth_request con 0x7fbc1c070a40 auth_method 0 2024-09-15T01:53:57.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): get_auth_request con 0x7fbc1c06a550 auth_method 0 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): get_auth_request con 0x7fbc1c138140 auth_method 0 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc223c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient(hunting): handle_auth_done global_id 54116 payload 824 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c06a550 msgr2=0x7fbc1c11e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c06a550 0x7fbc1c11e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc1c138140 msgr2=0x7fbc1c11f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbc1c138140 0x7fbc1c11f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 monclient: found mon.noname-c 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.584+0000 7fbc213c6640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc14009a20 con 0x7fbc1c070a40 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc213c6640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 msgr2=0x7fbc1c11eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc213c6640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 0x7fbc1c11eaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c5e0 tx=0x7fbc0c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c012840 con 0x7fbc1c070a40 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: dump: 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.611 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c0129a0 con 0x7fbc1c070a40 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 5 monclient: authenticate success, global_id 54116 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 10 monclient: _renew_subs 2024-09-15T01:53:57.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c121c00 con 0x7fbc1c070a40 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c012da0 con 0x7fbc1c070a40 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc11ffb640 10 monclient: dump: 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.613 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc13fff640 4 set_mon_vals no callback set 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 10 monclient: _renew_subs 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.587+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c120d10 con 0x7fbc1c070a40 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc23e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbc0c011590 con 0x7fbc1c070a40 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 0x7fbc1c11ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c063b00 con 0x7fbc1c11f570 2024-09-15T01:53:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 0x7fbc1c11ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient: get_auth_request con 0x7fbc1c11f570 auth_method 0 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc11ffb640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc00038740 0x7fbc0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fbc0c04bd70 con 0x7fbc1c070a40 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_done global_id 4975 payload 824 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc223c8640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc00038740 0x7fbc0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc21bc7640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc1c120f90 con 0x7fbc1c11f570 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.588+0000 7fbc223c8640 10 monclient: get_auth_request con 0x7fbc00038740 auth_method 0 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc21bc7640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 msgr2=0x7fbc1c11ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc21bc7640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 0x7fbc1c11ad70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800e710 tx=0x7fbc18007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc18004b50 con 0x7fbc1c11f570 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 10 monclient: dump: 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.616 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc223c8640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc00038740 0x7fbc0003abf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbc14034000 tx=0x7fbc14037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbc18004cb0 con 0x7fbc1c11f570 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 msgr2=0x7fbc1c11ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 0x7fbc1c11ad70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800e710 tx=0x7fbc18007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 msgr2=0x7fbc1c11ad70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:57.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.591+0000 7fbc11ffb640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11f570 0x7fbc1c11ad70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.578+0000 7f7136689640 10 monclient: get_monmap_and_config 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.578+0000 7f7136689640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.579+0000 7f7136689640 10 monclient: monmap: 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:57.580165+0000 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:57.580165+0000 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:57.618 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.579+0000 7f7136689640 1 Processor -- start 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.579+0000 7f7136689640 1 -- start start 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.579+0000 7f7136689640 10 monclient: init 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7130098190 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7130098560 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7130098960 0x7f713013a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): picked mon.noname-a con 0x7f7130098960 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:57.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71300a16c0 con 0x7f7130098190 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71301328c0 con 0x7f7130098560 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71300a0700 con 0x7f7130098960 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7136689640 10 monclient(hunting): authenticate will time out at 870.049311s 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f712f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7130098560 0x7f7130136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58004/0 (socket says 172.21.15.67:58004) 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f712f7fe640 1 -- 172.21.15.67:0/3987885975 learned_addr learned my addr 172.21.15.67:0/3987885975 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 1 --2- 172.21.15.67:0/3987885975 >> v2:172.21.15.17:3300/0 conn(0x7f7130098960 0x7f713013a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): get_auth_request con 0x7f7130098960 auth_method 0 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.580+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 10 monclient(hunting): handle_auth_done global_id 44406 payload 274 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 1 -- 172.21.15.67:0/3987885975 >> v1:172.21.15.67:6789/0 conn(0x7f7130098560 legacy=0x7f7130136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 1 -- 172.21.15.67:0/3987885975 >> v1:172.21.15.17:6789/0 conn(0x7f7130098190 legacy=0x7f71301321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 1 monclient: found mon.noname-a 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.581+0000 7f7134c26640 1 -- 172.21.15.67:0/3987885975 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71300a13e0 con 0x7f7130098960 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.583+0000 7f7134c26640 1 -- 172.21.15.67:0/3987885975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 msgr2=0x7f713013a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.583+0000 7f7134c26640 1 --2- 172.21.15.67:0/3987885975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713013a540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7130096c20 tx=0x7f7128009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe67d208aa8128b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 1 -- 172.21.15.67:0/3987885975 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128007840 con 0x7f7130098960 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: dump: 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.622 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 1 -- 172.21.15.67:0/3987885975 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712802d050 con 0x7f7130098960 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 5 monclient: authenticate success, global_id 44406 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 1 -- 172.21.15.67:0/3987885975 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128007bd0 con 0x7f7130098960 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 4 set_mon_vals no callback set 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f712effd640 10 monclient: dump: 2024-09-15T01:53:57.623 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 10 monclient: shutdown 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 -- 172.21.15.67:0/3987885975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 msgr2=0x7f713013a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 --2- 172.21.15.67:0/3987885975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713013a540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7130096c20 tx=0x7f7128009780 comp rx=0 tx=0).stop 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 -- 172.21.15.67:0/3987885975 shutdown_connections 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 --2- 172.21.15.67:0/3987885975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713013a540 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 -- 172.21.15.67:0/3987885975 >> 172.21.15.67:0/3987885975 conn(0x7f71300711e0 msgr2=0x7f71300a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 -- 172.21.15.67:0/3987885975 shutdown_connections 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.584+0000 7f7136689640 1 -- 172.21.15.67:0/3987885975 wait complete. 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.585+0000 7f7136689640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.585+0000 7f7136689640 10 monclient: monmap: 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:57.625 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.585+0000 7f7136689640 1 Processor -- start 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.585+0000 7f7136689640 1 -- start start 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.585+0000 7f7136689640 10 monclient: init 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): picked mon.noname-c con 0x7f7130098190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7130098560 0x7f7130130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): picked mon.noname-b con 0x7f7130098560 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): picked mon.noname-a con 0x7f7130098960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130141f30 con 0x7f7130098960 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713009f1a0 con 0x7f7130098560 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7130134b10 con 0x7f7130098190 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7136689640 10 monclient(hunting): authenticate will time out at 870.055289s 2024-09-15T01:53:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7130098560 0x7f7130130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:34778/0 (socket says 172.21.15.67:34778) 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 -- 172.21.15.67:0/4186874482 learned_addr learned my addr 172.21.15.67:0/4186874482 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): get_auth_request con 0x7f7130098190 auth_method 0 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): get_auth_request con 0x7f7130098560 auth_method 0 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): get_auth_request con 0x7f7130098960 auth_method 0 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f7134c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient(hunting): handle_auth_done global_id 54119 payload 824 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 -- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7130098560 msgr2=0x7f7130130b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7130098560 0x7f7130130b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 -- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 msgr2=0x7f713011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713011b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 monclient: found mon.noname-c 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.586+0000 7f712ffff640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128009a20 con 0x7f7130098190 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.588+0000 7f712ffff640 1 -- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 msgr2=0x7f713009e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.588+0000 7f712ffff640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f712000c5e0 tx=0x7f7120007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71200117a0 con 0x7f7130098190 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: dump: 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120011900 con 0x7f7130098190 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 5 monclient: authenticate success, global_id 54119 2024-09-15T01:53:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 10 monclient: _renew_subs 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713009f620 con 0x7f7130098190 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120011d00 con 0x7f7130098190 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 10 monclient: dump: 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.633 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 10 monclient: _renew_subs 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f712e7fc640 4 set_mon_vals no callback set 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713011cd70 con 0x7f7130098190 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.589+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7120012630 con 0x7f7130098190 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7136689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7136689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7136689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7127fff640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 0x7f711003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f712004b950 con 0x7f7130098190 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.590+0000 7f712f7fe640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 0x7f711003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.591+0000 7f712f7fe640 10 monclient: get_auth_request con 0x7f7110038790 auth_method 0 2024-09-15T01:53:57.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.593+0000 7f712f7fe640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 0x7f711003ac40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f713009ed10 tx=0x7f7118007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.598+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:57.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.598+0000 7f7127fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.598+0000 7f7127fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.600+0000 7fbc23e2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.600+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 0x7fbc1c11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.600+0000 7fbc23e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.600+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c0724a0 con 0x7fbc1c11b2b0 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 0x7fbc1c11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient: get_auth_request con 0x7fbc1c11b2b0 auth_method 0 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 10 monclient(hunting): handle_auth_done global_id 4978 payload 824 2024-09-15T01:53:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.601+0000 7fbc21bc7640 1 -- 172.21.15.17:0/4272343824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc1c11f940 con 0x7fbc1c11b2b0 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc21bc7640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 msgr2=0x7fbc1c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc21bc7640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 0x7fbc1c11b6b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800e710 tx=0x7fbc18019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1802a410 con 0x7fbc1c11b2b0 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.603+0000 7fbc11ffb640 10 monclient: dump: 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7fbc1800c770 con 0x7fbc1c11b2b0 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 msgr2=0x7fbc1c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 0x7fbc1c11b6b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800e710 tx=0x7fbc18019000 comp rx=0 tx=0).stop 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 msgr2=0x7fbc1c11b6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc11ffb640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc1c11b2b0 0x7fbc1c11b6b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.640 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.641 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.642 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:57.644 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:57.645 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:57.646 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.650 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:57.651 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.652 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.653 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:57.654 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:57.655 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc00038740 msgr2=0x7fbc0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbc00038740 0x7fbc0003abf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbc14034000 tx=0x7fbc14037000 comp rx=0 tx=0).stop 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 10 monclient: shutdown 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 msgr2=0x7fbc1c11eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 --2- 172.21.15.17:0/4272343824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbc1c070a40 0x7fbc1c11eaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c5e0 tx=0x7fbc0c007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc223c8640 1 -- 172.21.15.17:0/4272343824 reap_dead start 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc223c8640 1 -- 172.21.15.17:0/4272343824 reap_dead start 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 shutdown_connections 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 >> 172.21.15.17:0/4272343824 conn(0x7fbc1c067ae0 msgr2=0x7fbc1c06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 shutdown_connections 2024-09-15T01:53:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.604+0000 7fbc23e2b640 1 -- 172.21.15.17:0/4272343824 wait complete. 2024-09-15T01:53:57.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:57.657 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:53:57.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 10 monclient: get_monmap_and_config 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 10 monclient: monmap: 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:57.753001+0000 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:57.753001+0000 2024-09-15T01:53:57.754 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:57.755 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 1 Processor -- start 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 1 -- start start 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.752+0000 7ff491c46640 10 monclient: init 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff48c0708a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff48c06a510 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 0x7ff48c13a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c138140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c070c70 con 0x7ff48c06a510 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c09bc40 con 0x7ff48c0708a0 2024-09-15T01:53:57.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c09da40 con 0x7ff48c138140 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff491c46640 10 monclient(hunting): authenticate will time out at 897.749476s 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 0x7ff48c13a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48affd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff48c06a510 0x7ff48c09b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52246/0 (socket says 172.21.15.17:52246) 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48affd640 1 -- 172.21.15.17:0/4107328542 learned_addr learned my addr 172.21.15.17:0/4107328542 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): get_auth_request con 0x7ff48c138140 auth_method 0 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient(hunting): handle_auth_done global_id 54122 payload 274 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 1 -- 172.21.15.17:0/4107328542 >> v1:172.21.15.67:6789/0 conn(0x7ff48c0708a0 legacy=0x7ff48c09ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 1 -- 172.21.15.17:0/4107328542 >> v1:172.21.15.17:6789/0 conn(0x7ff48c06a510 legacy=0x7ff48c09b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 1 monclient: found mon.noname-b 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.753+0000 7ff48bfff640 1 -- 172.21.15.17:0/4107328542 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c070f30 con 0x7ff48c138140 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.756+0000 7ff48bfff640 1 -- 172.21.15.17:0/4107328542 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 msgr2=0x7ff48c13a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.756+0000 7ff48bfff640 1 --2- 172.21.15.17:0/4107328542 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 0x7ff48c13a520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff484009e90 tx=0x7ff484009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9568a4d5bcc6e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 1 -- 172.21.15.17:0/4107328542 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484007840 con 0x7ff48c138140 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: dump: 2024-09-15T01:53:57.764 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 1 -- 172.21.15.17:0/4107328542 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48402d050 con 0x7ff48c138140 2024-09-15T01:53:57.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 5 monclient: authenticate success, global_id 54122 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 4 set_mon_vals no callback set 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 1 -- 172.21.15.17:0/4107328542 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484007bd0 con 0x7ff48c138140 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff48a7fc640 10 monclient: dump: 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.766 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 10 monclient: shutdown 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 -- 172.21.15.17:0/4107328542 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 msgr2=0x7ff48c13a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 --2- 172.21.15.17:0/4107328542 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 0x7ff48c13a520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff484009e90 tx=0x7ff484009780 comp rx=0 tx=0).stop 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 -- 172.21.15.17:0/4107328542 shutdown_connections 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 --2- 172.21.15.17:0/4107328542 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c138140 0x7ff48c13a520 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 -- 172.21.15.17:0/4107328542 >> 172.21.15.17:0/4107328542 conn(0x7ff48c067ae0 msgr2=0x7ff48c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:57.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 -- 172.21.15.17:0/4107328542 shutdown_connections 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.757+0000 7ff491c46640 1 -- 172.21.15.17:0/4107328542 wait complete. 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.758+0000 7ff491c46640 10 monclient: build_initial_monmap 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.758+0000 7ff491c46640 10 monclient: monmap: 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.758+0000 7ff491c46640 1 Processor -- start 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.758+0000 7ff491c46640 1 -- start start 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.758+0000 7ff491c46640 10 monclient: init 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-a con 0x7ff48c06a510 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c0708a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff48c138140 0x7ff48c11f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c138140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c120d90 con 0x7ff48c06a510 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c120740 con 0x7ff48c138140 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c06d5e0 con 0x7ff48c0708a0 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff491c46640 10 monclient(hunting): authenticate will time out at 897.755163s 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35932/0 (socket says 172.21.15.17:35932) 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 -- 172.21.15.17:0/631195897 learned_addr learned my addr 172.21.15.17:0/631195897 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request con 0x7ff48c06a510 auth_method 0 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39490/0 (socket says 172.21.15.17:39490) 2024-09-15T01:53:57.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): get_auth_request con 0x7ff48c0708a0 auth_method 0 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff48c138140 0x7ff48c11f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): get_auth_request con 0x7ff48c138140 auth_method 0 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_done global_id 44415 payload 824 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 -- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 msgr2=0x7ff48c11eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 -- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff48c138140 msgr2=0x7ff48c11f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff48c138140 0x7ff48c11f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48b7fe640 1 -- 172.21.15.17:0/631195897 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484009a20 con 0x7ff48c06a510 2024-09-15T01:53:57.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.759+0000 7ff48affd640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.761+0000 7ff48b7fe640 1 -- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 msgr2=0x7ff48c11e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.761+0000 7ff48b7fe640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00c5e0 tx=0x7ff47c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.761+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c012840 con 0x7ff48c06a510 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.761+0000 7ff4737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: dump: 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: _finish_auth 0 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 5 monclient: authenticate success, global_id 44415 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _renew_subs 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c0129a0 con 0x7ff48c06a510 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c06deb0 con 0x7ff48c06a510 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c012d60 con 0x7ff48c06a510 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 10 monclient: dump: 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:57.775 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:57.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff489ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:57.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _renew_subs 2024-09-15T01:53:57.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:57.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c121d30 con 0x7ff48c06a510 2024-09-15T01:53:57.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.767+0000 7f7136689640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:57.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.767+0000 7f7136689640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:57.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.767+0000 7f7136689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:57.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:57.767+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff47c011590 con 0x7ff48c06a510 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.762+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c0724a0 con 0x7ff48c06a510 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.763+0000 7ff4737fe640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 0x7ff47803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.763+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5146+0+0 (secure 0 0 0) 0x7ff47c04bd30 con 0x7ff48c06a510 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.763+0000 7ff48affd640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 0x7ff47803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.763+0000 7ff48affd640 10 monclient: get_auth_request con 0x7ff478038740 auth_method 0 2024-09-15T01:53:57.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.766+0000 7ff48affd640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 0x7ff47803abf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff48c1205f0 tx=0x7ff480007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:57.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.767+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff47c010aa0 con 0x7ff48c06a510 2024-09-15T01:53:57.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.767+0000 7ff4737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:57.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.767+0000 7ff4737fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.998+0000 7ff491c46640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:53:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.998+0000 7ff491c46640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:53:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.998+0000 7ff491c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.998+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7ff48c122060 con 0x7ff48c06a510 2024-09-15T01:53:58.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.999+0000 7ff4737fe640 1 -- 172.21.15.17:0/631195897 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7ff47c016020 con 0x7ff48c06a510 2024-09-15T01:53:58.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.999+0000 7ff4737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:53:58.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:57.999+0000 7ff4737fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:58.002 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.002 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=455565311,config=3140514009,health=3172947059,kv=922137073,logm=856148608}) 2024-09-15T01:53:58.002 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=455565311,config=3140514009,health=1886805112,kv=1529054990,logm=4054667929}) 2024-09-15T01:53:58.002 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.002 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=455565311,config=3140514009,health=3172947059,kv=922137073,logm=856148608}) 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:57.999+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=8,logm=68} crc {auth=2383738110,config=3140514009,health=138247344,kv=1529054990,logm=3741021818}) 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960616000}) 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159872010}) 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960616000}) 2024-09-15T01:53:58.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.000+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533697896}) 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450912917}) 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=380241552}) 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450912917}) 2024-09-15T01:53:58.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.001+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556695738}) 2024-09-15T01:53:58.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 msgr2=0x7ff47803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 0x7ff47803abf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff48c1205f0 tx=0x7ff480007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:58.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 10 monclient: shutdown 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 msgr2=0x7ff48c11e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00c5e0 tx=0x7ff47c007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 shutdown_connections 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff478038740 0x7ff47803abf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff48c138140 0x7ff48c11f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff48c0708a0 0x7ff48c11eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 --2- 172.21.15.17:0/631195897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff48c06a510 0x7ff48c11e630 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.001+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 >> 172.21.15.17:0/631195897 conn(0x7ff48c067ae0 msgr2=0x7ff48c06bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.002+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 shutdown_connections 2024-09-15T01:53:58.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:58.002+0000 7ff491c46640 1 -- 172.21.15.17:0/631195897 wait complete. 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595250807}) 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674606833}) 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595250807}) 2024-09-15T01:53:58.009 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.002+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166751923}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170844993}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855437771}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170844993}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.003+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416671630}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3894608136}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2598566945}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3894608136}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4121902858}) 2024-09-15T01:53:58.010 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613446939}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964028764}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613446939}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.004+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968047560}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228462328}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1469707401}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228462328}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.005+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531617628}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=18} crc {logm=2284489358,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2670784270,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2170749948,osd_pg_creating=3942925033,osd_snap=1063953654,osdmap=1033212499}) 2024-09-15T01:53:58.011 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=21} crc {logm=1051885922,mdsmap=3585761405,mgr=2138800014,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=1632613521,mon_config_key=1383706935,monmap=3178194974,osd_metadata=612523271,osd_pg_creating=3942925033,osd_snap=1063953654,osdmap=3861096944}) 2024-09-15T01:53:58.012 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.012 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=18} crc {logm=2284489358,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2670784270,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2170749948,osd_pg_creating=3942925033,osd_snap=1063953654,osdmap=1033212499}) 2024-09-15T01:53:58.012 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.006+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=803843693,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235355,mgrstat=1632613521,mon_config_key=1583035879,monmap=3178194975,osd_metadata=1149731095,osd_pg_creating=3942925033,osd_snap=1063953654,osdmap=604442232}) 2024-09-15T01:53:58.012 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.013 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=3371283882}) 2024-09-15T01:53:58.013 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=74} crc {osdmap=622697594}) 2024-09-15T01:53:58.013 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:53:58.013 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=3371283882}) 2024-09-15T01:53:58.013 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:53:58.008+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=79} crc {osdmap=1018999850}) 2024-09-15T01:53:58.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:53:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.180+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v58) v1 ==== 110+0+0 (secure 0 0 0) 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.180+0000 7f7127fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.180+0000 7f7127fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:58.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.280+0000 7f7136689640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:58.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.280+0000 7f7136689640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:58.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.280+0000 7f7136689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:58.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.280+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.281+0000 7f7127fff640 1 -- 172.21.15.67:0/4186874482 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v58) v1 ==== 134+0+0 (secure 0 0 0) 0x7f713011de00 con 0x7f7130098190 2024-09-15T01:53:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.281+0000 7f7127fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T01:53:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.281+0000 7f7127fff640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-15T01:53:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:already renamed to 'bar' 2024-09-15T01:53:58.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 msgr2=0x7f711003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 0x7f711003ac40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f713009ed10 tx=0x7f7118007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:58.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 10 monclient: shutdown 2024-09-15T01:53:58.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 msgr2=0x7f713009e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f712000c5e0 tx=0x7f7120007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 shutdown_connections 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7110038790 0x7f711003ac40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7130098960 0x7f713011b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7130098560 0x7f7130130b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 --2- 172.21.15.67:0/4186874482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7130098190 0x7f713009e510 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.283+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 >> 172.21.15.67:0/4186874482 conn(0x7f71300711e0 msgr2=0x7f7130096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.284+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 shutdown_connections 2024-09-15T01:53:58.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.284+0000 7f7136689640 1 -- 172.21.15.67:0/4186874482 wait complete. 2024-09-15T01:53:58.295 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:58.295 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false grep foo-asdf 2024-09-15T01:53:58.295 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:58.295 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo-asdf 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.367+0000 7fcfe7e0f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.367+0000 7fcfe7e0f640 10 monclient: build_initial_monmap 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.367+0000 7fcfe7e0f640 10 monclient: monmap: 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:58.369132+0000 2024-09-15T01:53:58.370 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:58.369132+0000 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:58.371 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.368+0000 7fcfe7e0f640 1 Processor -- start 2024-09-15T01:53:58.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.368+0000 7fcfe7e0f640 1 -- start start 2024-09-15T01:53:58.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.368+0000 7fcfe7e0f640 10 monclient: init 2024-09-15T01:53:58.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfe00a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfe009eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 0x7fcfe013a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe009f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00a61a0 con 0x7fcfe009eea0 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00a04c0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00a1610 con 0x7fcfe009f2c0 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe7e0f640 10 monclient(hunting): authenticate will time out at 870.838317s 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe5bab640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcfe00a0b50 0x7fcfe0069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58016/0 (socket says 172.21.15.67:58016) 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe5bab640 1 -- 172.21.15.67:0/1017910534 learned_addr learned my addr 172.21.15.67:0/1017910534 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:58.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 1 --2- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 0x7fcfe013a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): get_auth_request con 0x7fcfe009f2c0 auth_method 0 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_done global_id 54128 payload 274 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 1 -- 172.21.15.67:0/1017910534 >> v1:172.21.15.67:6789/0 conn(0x7fcfe00a0b50 legacy=0x7fcfe0069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 1 -- 172.21.15.67:0/1017910534 >> v1:172.21.15.17:6789/0 conn(0x7fcfe009eea0 legacy=0x7fcfe0136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient: _finish_hunting 0 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 1 monclient: found mon.noname-b 2024-09-15T01:53:58.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:58.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:58.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.369+0000 7fcfe63ac640 1 -- 172.21.15.67:0/1017910534 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0069a90 con 0x7fcfe009f2c0 2024-09-15T01:53:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.372+0000 7fcfe63ac640 1 -- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 msgr2=0x7fcfe013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.372+0000 7fcfe63ac640 1 --2- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 0x7fcfe013a570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009e90 tx=0x7fcfd8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489f543f862c29f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 1 -- 172.21.15.67:0/1017910534 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8007840 con 0x7fcfe009f2c0 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: dump: 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.382 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: _finish_auth 0 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 5 monclient: authenticate success, global_id 54128 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 1 -- 172.21.15.67:0/1017910534 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd802d050 con 0x7fcfe009f2c0 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 4 set_mon_vals no callback set 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 1 -- 172.21.15.67:0/1017910534 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8007bd0 con 0x7fcfe009f2c0 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe4ba9640 10 monclient: dump: 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 10 monclient: shutdown 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:58.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 msgr2=0x7fcfe013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 0x7fcfe013a570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009e90 tx=0x7fcfd8009780 comp rx=0 tx=0).stop 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/1017910534 shutdown_connections 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/1017910534 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009f2c0 0x7fcfe013a570 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/1017910534 >> 172.21.15.67:0/1017910534 conn(0x7fcfe0071330 msgr2=0x7fcfe00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/1017910534 shutdown_connections 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.373+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/1017910534 wait complete. 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient: build_initial_monmap 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient: monmap: 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:58.385 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 Processor -- start 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 -- start start 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient: init 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009eea0 0x7fcfe0140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfe009f2c0 0x7fcfe0131d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe009f2c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe00a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe011ed60 con 0x7fcfe009f2c0 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0120160 con 0x7fcfe00a0b50 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe011e830 con 0x7fcfe009eea0 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.374+0000 7fcfe7e0f640 10 monclient(hunting): authenticate will time out at 870.843958s 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56220/0 (socket says 172.21.15.67:56220) 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 -- 172.21.15.67:0/3740857156 learned_addr learned my addr 172.21.15.67:0/3740857156 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:58.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): get_auth_request con 0x7fcfe00a0b50 auth_method 0 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009eea0 0x7fcfe0140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfe009f2c0 0x7fcfe0131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): get_auth_request con 0x7fcfe009f2c0 auth_method 0 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): get_auth_request con 0x7fcfe009eea0 auth_method 0 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe53aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe5bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient(hunting): handle_auth_done global_id 4984 payload 824 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 -- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009eea0 msgr2=0x7fcfe0140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009eea0 0x7fcfe0140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 -- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfe009f2c0 msgr2=0x7fcfe0131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfe009f2c0 0x7fcfe0131d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient: _finish_hunting 0 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 monclient: found mon.noname-b 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.375+0000 7fcfe63ac640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8009a20 con 0x7fcfe00a0b50 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe63ac640 1 -- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 msgr2=0x7fcfe011c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe63ac640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8008000 tx=0x7fcfd8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8031bb0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: dump: 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd8040070 con 0x7fcfe00a0b50 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8045430 con 0x7fcfe00a0b50 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd5ffb640 10 monclient: dump: 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.392 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe7e0f640 5 monclient: authenticate success, global_id 4984 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe7e0f640 10 monclient: _renew_subs 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe7e0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfd7fff640 4 set_mon_vals no callback set 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.377+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe011fe50 con 0x7fcfe00a0b50 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.378+0000 7fcfe7e0f640 10 monclient: _renew_subs 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.378+0000 7fcfe7e0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.378+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0141dd0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.378+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcfd8042030 con 0x7fcfe00a0b50 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe7e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe7e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe7e0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe004bf60 con 0x7fcfe00a0b50 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfd5ffb640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 0x7fcfc403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fcfd806d6c0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe5bab640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 0x7fcfc403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.379+0000 7fcfe5bab640 10 monclient: get_auth_request con 0x7fcfc4038740 auth_method 0 2024-09-15T01:53:58.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.381+0000 7fcfe5bab640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 0x7fcfc403abf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc007fd0 tx=0x7fcfdc007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.383+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcfd803b420 con 0x7fcfe00a0b50 2024-09-15T01:53:58.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.383+0000 7fcfd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.383+0000 7fcfd5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfe7e0f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfe7e0f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfe7e0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fcfe004bf60 con 0x7fcfe00a0b50 2024-09-15T01:53:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7fcfd80443d0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.550+0000 7fcfd5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.648+0000 7fcfe7e0f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.648+0000 7fcfe7e0f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.648+0000 7fcfe7e0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.648+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fcfe004bf60 con 0x7fcfe00a0b50 2024-09-15T01:53:58.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.649+0000 7fcfd5ffb640 1 -- 172.21.15.67:0/3740857156 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7fcfd80445b0 con 0x7fcfe00a0b50 2024-09-15T01:53:58.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.649+0000 7fcfd5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.649+0000 7fcfd5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 msgr2=0x7fcfc403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 0x7fcfc403abf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc007fd0 tx=0x7fcfdc007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 10 monclient: shutdown 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 msgr2=0x7fcfe011c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8008000 tx=0x7fcfd8037000 comp rx=0 tx=0).stop 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 shutdown_connections 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcfc4038740 0x7fcfc403abf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfe009f2c0 0x7fcfe0131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe00a0b50 0x7fcfe011c910 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 --2- 172.21.15.67:0/3740857156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcfe009eea0 0x7fcfe0140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 >> 172.21.15.67:0/3740857156 conn(0x7fcfe0071330 msgr2=0x7fcfe009ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:58.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 shutdown_connections 2024-09-15T01:53:58.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.651+0000 7fcfe7e0f640 1 -- 172.21.15.67:0/3740857156 wait complete. 2024-09-15T01:53:58.662 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:58.662 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule ls 2024-09-15T01:53:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false grep bar-asdf 2024-09-15T01:53:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:53:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep bar-asdf 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.735+0000 7fc650b9d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.735+0000 7fc650b9d640 10 monclient: build_initial_monmap 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.735+0000 7fc650b9d640 10 monclient: monmap: 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:58.736883+0000 2024-09-15T01:53:58.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:58.736883+0000 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:58.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.736+0000 7fc650b9d640 1 Processor -- start 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.736+0000 7fc650b9d640 1 -- start start 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.736+0000 7fc650b9d640 10 monclient: init 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.736+0000 7fc650b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.736+0000 7fc650b9d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc64c0a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc64c09eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 0x7fc64c13a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc64c09f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0a61a0 con 0x7fc64c09eea0 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0a04c0 con 0x7fc64c0a0b50 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0a1610 con 0x7fc64c09f2c0 2024-09-15T01:53:58.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc650b9d640 10 monclient(hunting): authenticate will time out at 871.206131s 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc64c0a0b50 0x7fc64c069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58032/0 (socket says 172.21.15.67:58032) 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64a59c640 1 -- 172.21.15.67:0/4186324728 learned_addr learned my addr 172.21.15.67:0/4186324728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 1 --2- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 0x7fc64c13a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): get_auth_request con 0x7fc64c09f2c0 auth_method 0 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_done global_id 54134 payload 274 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 1 -- 172.21.15.67:0/4186324728 >> v1:172.21.15.67:6789/0 conn(0x7fc64c0a0b50 legacy=0x7fc64c069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 1 -- 172.21.15.67:0/4186324728 >> v1:172.21.15.17:6789/0 conn(0x7fc64c09eea0 legacy=0x7fc64c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 1 monclient: found mon.noname-b 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.737+0000 7fc64ad9d640 1 -- 172.21.15.67:0/4186324728 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c069a90 con 0x7fc64c09f2c0 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.740+0000 7fc64ad9d640 1 -- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 msgr2=0x7fc64c13a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.740+0000 7fc64ad9d640 1 --2- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 0x7fc64c13a570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc644009e90 tx=0x7fc644009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73530000cf39f4c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 1 -- 172.21.15.67:0/4186324728 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644007840 con 0x7fc64c09f2c0 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: dump: 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: _finish_auth 0 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 1 -- 172.21.15.67:0/4186324728 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64402d050 con 0x7fc64c09f2c0 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 1 -- 172.21.15.67:0/4186324728 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644007bd0 con 0x7fc64c09f2c0 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 5 monclient: authenticate success, global_id 54134 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 4 set_mon_vals no callback set 2024-09-15T01:53:58.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc64959a640 10 monclient: dump: 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 10 monclient: shutdown 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 -- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 msgr2=0x7fc64c13a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 --2- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 0x7fc64c13a570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc644009e90 tx=0x7fc644009780 comp rx=0 tx=0).stop 2024-09-15T01:53:58.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 -- 172.21.15.67:0/4186324728 shutdown_connections 2024-09-15T01:53:58.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 --2- 172.21.15.67:0/4186324728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09f2c0 0x7fc64c13a570 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 -- 172.21.15.67:0/4186324728 >> 172.21.15.67:0/4186324728 conn(0x7fc64c071330 msgr2=0x7fc64c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:58.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 -- 172.21.15.67:0/4186324728 shutdown_connections 2024-09-15T01:53:58.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.741+0000 7fc650b9d640 1 -- 172.21.15.67:0/4186324728 wait complete. 2024-09-15T01:53:58.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.742+0000 7fc650b9d640 10 monclient: build_initial_monmap 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.742+0000 7fc650b9d640 10 monclient: monmap: 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.753 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.742+0000 7fc650b9d640 1 Processor -- start 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.742+0000 7fc650b9d640 1 -- start start 2024-09-15T01:53:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.742+0000 7fc650b9d640 10 monclient: init 2024-09-15T01:53:58.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09eea0 0x7fc64c140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc64c09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc64c09f2c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc64c0a0b50 0x7fc64c11c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc64c0a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c11e400 con 0x7fc64c0a0b50 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0a1260 con 0x7fc64c09f2c0 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:58.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c120180 con 0x7fc64c09eea0 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc650b9d640 10 monclient(hunting): authenticate will time out at 871.212262s 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56228/0 (socket says 172.21.15.67:56228) 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 -- 172.21.15.67:0/548257259 learned_addr learned my addr 172.21.15.67:0/548257259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): get_auth_request con 0x7fc64c09f2c0 auth_method 0 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09eea0 0x7fc64c140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc64c0a0b50 0x7fc64c11c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): get_auth_request con 0x7fc64c09eea0 auth_method 0 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): get_auth_request con 0x7fc64c0a0b50 auth_method 0 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc64a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient(hunting): handle_auth_done global_id 4987 payload 824 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 -- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09eea0 msgr2=0x7fc64c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09eea0 0x7fc64c140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 -- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc64c0a0b50 msgr2=0x7fc64c11c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc64c0a0b50 0x7fc64c11c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 monclient: found mon.noname-b 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.743+0000 7fc649d9b640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644009a20 con 0x7fc64c09f2c0 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.745+0000 7fc649d9b640 1 -- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 msgr2=0x7fc64c131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.745+0000 7fc649d9b640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc64000c640 tx=0x7fc640007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640012840 con 0x7fc64c09f2c0 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: dump: 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:58.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 5 monclient: authenticate success, global_id 4987 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 10 monclient: _renew_subs 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6400129a0 con 0x7fc64c09f2c0 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c141e50 con 0x7fc64c09f2c0 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640012d60 con 0x7fc64c09f2c0 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:58.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc63a7fc640 10 monclient: dump: 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:58.766 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc648d99640 4 set_mon_vals no callback set 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 10 monclient: _renew_subs 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.746+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c11ef40 con 0x7fc64c09f2c0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc650b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc650b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc650b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c04bf60 con 0x7fc64c09f2c0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc640011590 con 0x7fc64c09f2c0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc63a7fc640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 0x7fc62c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fc64004ba90 con 0x7fc64c09f2c0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc64a59c640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 0x7fc62c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:58.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.747+0000 7fc64a59c640 10 monclient: get_auth_request con 0x7fc62c038740 auth_method 0 2024-09-15T01:53:58.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.750+0000 7fc64a59c640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 0x7fc62c03abf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc63c007fd0 tx=0x7fc63c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:58.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.751+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc640021a90 con 0x7fc64c09f2c0 2024-09-15T01:53:58.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.751+0000 7fc63a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:58.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.751+0000 7fc63a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.916+0000 7fc650b9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.916+0000 7fc650b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.916+0000 7fc650b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.916+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fc64c11e8e0 con 0x7fc64c09f2c0 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.917+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7fc64c11e8e0 con 0x7fc64c09f2c0 2024-09-15T01:53:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.917+0000 7fc63a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:58.917+0000 7fc63a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:59.015 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:53:59.016 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc650b9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc650b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc650b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fc64c11e8e0 con 0x7fc64c09f2c0 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc63a7fc640 1 -- 172.21.15.67:0/548257259 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7fc64c11e8e0 con 0x7fc64c09f2c0 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc63a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T01:53:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.016+0000 7fc63a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 msgr2=0x7fc62c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 0x7fc62c03abf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc63c007fd0 tx=0x7fc63c007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 10 monclient: shutdown 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 msgr2=0x7fc64c131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc64000c640 tx=0x7fc640007ed0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 shutdown_connections 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc62c038740 0x7fc62c03abf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc64c0a0b50 0x7fc64c11c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c09f2c0 0x7fc64c131d70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 --2- 172.21.15.67:0/548257259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc64c09eea0 0x7fc64c140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 >> 172.21.15.67:0/548257259 conn(0x7fc64c071330 msgr2=0x7fc64c09ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 shutdown_connections 2024-09-15T01:53:59.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.018+0000 7fc650b9d640 1 -- 172.21.15.67:0/548257259 wait complete. 2024-09-15T01:53:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:53:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm foo 2024-09-15T01:53:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 10 monclient: monmap: 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:53:59.105224+0000 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:53:59.105224+0000 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.108 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 1 Processor -- start 2024-09-15T01:53:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 1 -- start start 2024-09-15T01:53:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.104+0000 7fde0051d640 10 monclient: init 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 0x7fddf81321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-a con 0x7fddf8098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddf80a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddf8098c20 0x7fddf81390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-c con 0x7fddf8098c20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80a0b80 con 0x7fddf80a0370 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf80a00b0 con 0x7fddf8098140 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf8071390 con 0x7fddf8098c20 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fde0051d640 10 monclient(hunting): authenticate will time out at 871.574397s 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddf8098c20 0x7fddf81390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfdab8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddf80a0370 0x7fddf8098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58048/0 (socket says 172.21.15.67:58048) 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfdab8640 1 -- 172.21.15.67:0/4044964052 learned_addr learned my addr 172.21.15.67:0/4044964052 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): get_auth_request con 0x7fddf8098c20 auth_method 0 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 1 --2- 172.21.15.67:0/4044964052 >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 0x7fddf81321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 10 monclient(hunting): get_auth_request con 0x7fddf8098140 auth_method 0 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient(hunting): handle_auth_done global_id 4990 payload 274 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 -- 172.21.15.67:0/4044964052 >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 msgr2=0x7fddf81321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 --2- 172.21.15.67:0/4044964052 >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 0x7fddf81321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 -- 172.21.15.67:0/4044964052 >> v1:172.21.15.67:6789/0 conn(0x7fddf80a0370 legacy=0x7fddf8098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 monclient: found mon.noname-c 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfeaba640 1 -- 172.21.15.67:0/4044964052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8071500 con 0x7fddf8098c20 2024-09-15T01:53:59.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.105+0000 7fddfe2b9640 1 --2- 172.21.15.67:0/4044964052 >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 0x7fddf81321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:59.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfeaba640 1 -- 172.21.15.67:0/4044964052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098c20 msgr2=0x7fddf81390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfeaba640 1 --2- 172.21.15.67:0/4044964052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098c20 0x7fddf81390c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fddf80a1b60 tx=0x7fddf0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24c28e31af0117d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 1 -- 172.21.15.67:0/4044964052 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0007760 con 0x7fddf8098c20 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: dump: 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 1 -- 172.21.15.67:0/4044964052 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf002d050 con 0x7fddf8098c20 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fde0051d640 5 monclient: authenticate success, global_id 4990 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fde0051d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fde0051d640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 1 -- 172.21.15.67:0/4044964052 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0007b30 con 0x7fddf8098c20 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fde0051d640 4 set_mon_vals no callback set 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.108+0000 7fddfd2b7640 10 monclient: dump: 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 10 monclient: shutdown 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 -- 172.21.15.67:0/4044964052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098c20 msgr2=0x7fddf81390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 --2- 172.21.15.67:0/4044964052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098c20 0x7fddf81390c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fddf80a1b60 tx=0x7fddf0009780 comp rx=0 tx=0).stop 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 -- 172.21.15.67:0/4044964052 shutdown_connections 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 --2- 172.21.15.67:0/4044964052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098c20 0x7fddf81390c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 --2- 172.21.15.67:0/4044964052 >> v2:172.21.15.17:3300/0 conn(0x7fddf8098140 0x7fddf81321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 -- 172.21.15.67:0/4044964052 >> 172.21.15.67:0/4044964052 conn(0x7fddf80a5780 msgr2=0x7fddf80a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 -- 172.21.15.67:0/4044964052 shutdown_connections 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 1 -- 172.21.15.67:0/4044964052 wait complete. 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.109+0000 7fde0051d640 10 monclient: monmap: 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:53:59.122 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 Processor -- start 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 -- start start 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient: init 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:53:59.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-b con 0x7fddf8098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 0x7fddf8130c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-a con 0x7fddf8098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddf80a0370 0x7fddf81311d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): picked mon.noname-c con 0x7fddf80a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8132e70 con 0x7fddf8098c20 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf81337c0 con 0x7fddf8098140 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80a3b40 con 0x7fddf80a0370 2024-09-15T01:53:59.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fde0051d640 10 monclient(hunting): authenticate will time out at 871.579756s 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56254/0 (socket says 172.21.15.67:56254) 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 1 -- 172.21.15.67:0/1449591926 learned_addr learned my addr 172.21.15.67:0/1449591926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): get_auth_request con 0x7fddf8098140 auth_method 0 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfdab8640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 0x7fddf8130c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfdab8640 10 monclient(hunting): get_auth_request con 0x7fddf8098c20 auth_method 0 2024-09-15T01:53:59.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfdab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfdab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfdab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfe2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.110+0000 7fddfeaba640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddf80a0370 0x7fddf81311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfdab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfdab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfdab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): get_auth_request con 0x7fddf80a0370 auth_method 0 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfeaba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 10 monclient(hunting): handle_auth_done global_id 4993 payload 824 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 -- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddf80a0370 msgr2=0x7fddf81311d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddf80a0370 0x7fddf81311d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 -- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 msgr2=0x7fddf8130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 0x7fddf8130c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 monclient: found mon.noname-b 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfe2b9640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf0009a20 con 0x7fddf8098140 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.111+0000 7fddfdab8640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 0x7fddf8130c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddfe2b9640 1 -- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 msgr2=0x7fddf8138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddfe2b9640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fddf400d840 tx=0x7fddf4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf40098b0 con 0x7fddf8098140 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: dump: 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4009a10 con 0x7fddf8098140 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 5 monclient: authenticate success, global_id 4993 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4009dd0 con 0x7fddf8098140 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 10 monclient: _renew_subs 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf8134b90 con 0x7fddf8098140 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddfcab6640 4 set_mon_vals no callback set 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fddea7fc640 10 monclient: dump: 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.130 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 10 monclient: _renew_subs 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.113+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf8137d20 con 0x7fddf8098140 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fde0051d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fde0051d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fde0051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf81348b0 con 0x7fddf8098140 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fddf4014640 con 0x7fddf8098140 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fddea7fc640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fddf404c030 con 0x7fddf8098140 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fddfdab8640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.114+0000 7fddfdab8640 10 monclient: get_auth_request con 0x7fdddc038740 auth_method 0 2024-09-15T01:53:59.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.117+0000 7fddfdab8640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 0x7fdddc03abf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fddec007fd0 tx=0x7fddec007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.118+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fddf407de40 con 0x7fddf8098140 2024-09-15T01:53:59.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.119+0000 7fddea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.119+0000 7fddea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:59.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.122+0000 7f6ee1634640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.122+0000 7f6ee1634640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.122+0000 7f6ee1634640 10 monclient: monmap: 2024-09-15T01:53:59.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:59.123767+0000 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:59.123767+0000 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.123+0000 7f6ee1634640 1 Processor -- start 2024-09-15T01:53:59.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.123+0000 7f6ee1634640 1 -- start start 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.123+0000 7f6ee1634640 10 monclient: init 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6edc068f20 0x7f6edc1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-c con 0x7f6edc068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6edc0692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 0x7f6edc1393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-b con 0x7f6edc136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc0a1950 con 0x7f6edc0692f0 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc0a1670 con 0x7f6edc068f20 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0a0020 con 0x7f6edc136fe0 2024-09-15T01:53:59.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ee1634640 10 monclient(hunting): authenticate will time out at 899.120356s 2024-09-15T01:53:59.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 0x7f6edc1393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ed3fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6edc0692f0 0x7f6edc0696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52250/0 (socket says 172.21.15.17:52250) 2024-09-15T01:53:59.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3687040205 learned_addr learned my addr 172.21.15.17:0/3687040205 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): get_auth_request con 0x7f6edc136fe0 auth_method 0 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 1 --2- 172.21.15.17:0/3687040205 >> v2:172.21.15.67:3300/0 conn(0x7f6edc068f20 0x7f6edc1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): get_auth_request con 0x7f6edc068f20 auth_method 0 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_done global_id 54143 payload 274 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 -- 172.21.15.17:0/3687040205 >> v2:172.21.15.67:3300/0 conn(0x7f6edc068f20 msgr2=0x7f6edc1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 --2- 172.21.15.17:0/3687040205 >> v2:172.21.15.67:3300/0 conn(0x7f6edc068f20 0x7f6edc1321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 -- 172.21.15.17:0/3687040205 >> v1:172.21.15.17:6789/0 conn(0x7f6edc0692f0 legacy=0x7f6edc0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 monclient: found mon.noname-b 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.124+0000 7f6edb7fe640 1 -- 172.21.15.17:0/3687040205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0a1390 con 0x7f6edc136fe0 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6edb7fe640 1 -- 172.21.15.17:0/3687040205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 msgr2=0x7f6edc1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6edb7fe640 1 --2- 172.21.15.17:0/3687040205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 0x7f6edc1393c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009e90 tx=0x7f6ed4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2509216b60be1337 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 1 -- 172.21.15.17:0/3687040205 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4007760 con 0x7f6edc136fe0 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: dump: 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.146 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 1 -- 172.21.15.17:0/3687040205 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed402d050 con 0x7f6edc136fe0 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 1 -- 172.21.15.17:0/3687040205 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4007af0 con 0x7f6edc136fe0 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6eda7fc640 10 monclient: dump: 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.147 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6ee1634640 5 monclient: authenticate success, global_id 54143 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6ee1634640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.127+0000 7f6ee1634640 4 set_mon_vals no callback set 2024-09-15T01:53:59.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 10 monclient: shutdown 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 -- 172.21.15.17:0/3687040205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 msgr2=0x7f6edc1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3687040205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 0x7f6edc1393c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009e90 tx=0x7f6ed4009780 comp rx=0 tx=0).stop 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 -- 172.21.15.17:0/3687040205 shutdown_connections 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3687040205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc136fe0 0x7f6edc1393c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3687040205 >> v2:172.21.15.67:3300/0 conn(0x7f6edc068f20 0x7f6edc1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 -- 172.21.15.17:0/3687040205 >> 172.21.15.17:0/3687040205 conn(0x7f6edc070fb0 msgr2=0x7f6edc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 -- 172.21.15.17:0/3687040205 shutdown_connections 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.128+0000 7f6ee1634640 1 -- 172.21.15.17:0/3687040205 wait complete. 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.129+0000 7f6ee1634640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.129+0000 7f6ee1634640 10 monclient: monmap: 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.149 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.129+0000 7f6ee1634640 1 Processor -- start 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.129+0000 7f6ee1634640 1 -- start start 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.129+0000 7f6ee1634640 10 monclient: init 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:59.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6edc068f20 0x7f6edc099d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-b con 0x7f6edc068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-c con 0x7f6edc0692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 0x7f6edc09a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): picked mon.noname-a con 0x7f6edc136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc11da70 con 0x7f6edc136fe0 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0a0750 con 0x7f6edc068f20 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc09c550 con 0x7f6edc0692f0 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ee1634640 10 monclient(hunting): authenticate will time out at 899.126273s 2024-09-15T01:53:59.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 0x7f6edc09a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39514/0 (socket says 172.21.15.17:39514) 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6edc068f20 0x7f6edc099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3818763650 learned_addr learned my addr 172.21.15.17:0/3818763650 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request con 0x7f6edc0692f0 auth_method 0 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): get_auth_request con 0x7f6edc068f20 auth_method 0 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): get_auth_request con 0x7f6edc136fe0 auth_method 0 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_done global_id 54146 payload 824 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6edc068f20 msgr2=0x7f6edc099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6edc068f20 0x7f6edc099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 msgr2=0x7f6edc09a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 0x7f6edc09a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 monclient: found mon.noname-c 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3818763650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4009a20 con 0x7f6edc0692f0 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.130+0000 7f6edb7fe640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 0x7f6edc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.132+0000 7f6ed3fff640 1 -- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 msgr2=0x7f6edc09a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.132+0000 7f6ed3fff640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400c5e0 tx=0x7f6ec4004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4005cb0 con 0x7f6edc0692f0 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: dump: 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.155 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4005e30 con 0x7f6edc0692f0 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 5 monclient: authenticate success, global_id 54146 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 10 monclient: _renew_subs 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc11c7e0 con 0x7f6edc0692f0 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4007320 con 0x7f6edc0692f0 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 10 monclient: dump: 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed9ffb640 4 set_mon_vals no callback set 2024-09-15T01:53:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 10 monclient: _renew_subs 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc11b880 con 0x7f6edc0692f0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.133+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6ec401b3b0 con 0x7f6edc0692f0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ee1634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ee1634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ee1634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc09b560 con 0x7f6edc0692f0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ed2ffd640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 0x7f6ebc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f6ec404b250 con 0x7f6edc0692f0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6edaffd640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 0x7f6ebc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.134+0000 7f6edaffd640 10 monclient: get_auth_request con 0x7f6ebc038790 auth_method 0 2024-09-15T01:53:59.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.136+0000 7f6edaffd640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 0x7f6ebc03ac40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009c80 tx=0x7f6ecc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.139+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6edc09b560 con 0x7f6edc0692f0 2024-09-15T01:53:59.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.139+0000 7f6ed2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.139+0000 7f6ed2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:59.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.286+0000 7fde0051d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:53:59.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.286+0000 7fde0051d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:53:59.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.286+0000 7fde0051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:53:59.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:53:59.286+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fddf804bf60 con 0x7fddf8098140 2024-09-15T01:53:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ee1634640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:53:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ee1634640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ee1634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6edc09b560 con 0x7f6edc0692f0 2024-09-15T01:53:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ed2ffd640 1 -- 172.21.15.17:0/3818763650 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f6edc09b560 con 0x7f6edc0692f0 2024-09-15T01:53:59.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ed2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:53:59.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.368+0000 7f6ed2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:59.371 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 msgr2=0x7f6ebc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 0x7f6ebc03ac40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009c80 tx=0x7f6ecc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 10 monclient: shutdown 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 msgr2=0x7f6edc09a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400c5e0 tx=0x7f6ec4004960 comp rx=0 tx=0).stop 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 shutdown_connections 2024-09-15T01:53:59.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ebc038790 0x7f6ebc03ac40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6edc136fe0 0x7f6edc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6edc0692f0 0x7f6edc09a240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 --2- 172.21.15.17:0/3818763650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6edc068f20 0x7f6edc099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.370+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 >> 172.21.15.17:0/3818763650 conn(0x7f6edc070fb0 msgr2=0x7f6edc09e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.371+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 shutdown_connections 2024-09-15T01:53:59.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.371+0000 7f6ee1634640 1 -- 172.21.15.17:0/3818763650 wait complete. 2024-09-15T01:53:59.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:53:59.383 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:53:59.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:53:59.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 10 monclient: monmap: 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:59.479889+0000 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:59.479889+0000 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 1 Processor -- start 2024-09-15T01:53:59.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 1 -- start start 2024-09-15T01:53:59.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.479+0000 7fa194a6f640 10 monclient: init 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa190131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 0x7fa190131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1900a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa19009f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1900a0040 con 0x7fa19009f1e0 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa19009bff0 con 0x7fa1900a0280 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900a14a0 con 0x7fa190131b20 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa194a6f640 10 monclient(hunting): authenticate will time out at 899.476447s 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39524/0 (socket says 172.21.15.17:39524) 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 -- 172.21.15.17:0/1478878038 learned_addr learned my addr 172.21.15.17:0/1478878038 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): get_auth_request con 0x7fa190131b20 auth_method 0 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18ed9d640 1 --1- 172.21.15.17:0/1478878038 >> v1:172.21.15.17:6789/0 conn(0x7fa19009f1e0 0x7fa190068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52252/0 (socket says 172.21.15.17:52252) 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 1 --2- 172.21.15.17:0/1478878038 >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 0x7fa190131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 10 monclient(hunting): get_auth_request con 0x7fa1900a0280 auth_method 0 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient(hunting): handle_auth_done global_id 54149 payload 274 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 -- 172.21.15.17:0/1478878038 >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 msgr2=0x7fa190131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 --2- 172.21.15.17:0/1478878038 >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 0x7fa190131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 -- 172.21.15.17:0/1478878038 >> v1:172.21.15.17:6789/0 conn(0x7fa19009f1e0 legacy=0x7fa190068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 monclient: found mon.noname-b 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18e59c640 1 -- 172.21.15.17:0/1478878038 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190132430 con 0x7fa190131b20 2024-09-15T01:53:59.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.480+0000 7fa18dd9b640 1 --2- 172.21.15.17:0/1478878038 >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 0x7fa190131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:53:59.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18e59c640 1 -- 172.21.15.17:0/1478878038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 msgr2=0x7fa19009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18e59c640 1 --2- 172.21.15.17:0/1478878038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa180009f20 tx=0x7fa180009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b82e71c601a7ff59 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 1 -- 172.21.15.17:0/1478878038 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180007610 con 0x7fa190131b20 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: dump: 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.494 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 1 -- 172.21.15.17:0/1478878038 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18002d050 con 0x7fa190131b20 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa194a6f640 5 monclient: authenticate success, global_id 54149 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa194a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 1 -- 172.21.15.17:0/1478878038 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800079a0 con 0x7fa190131b20 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa194a6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa194a6f640 4 set_mon_vals no callback set 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.483+0000 7fa18d59a640 10 monclient: dump: 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 10 monclient: shutdown 2024-09-15T01:53:59.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 -- 172.21.15.17:0/1478878038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 msgr2=0x7fa19009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 --2- 172.21.15.17:0/1478878038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa180009f20 tx=0x7fa180009840 comp rx=0 tx=0).stop 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 -- 172.21.15.17:0/1478878038 shutdown_connections 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 --2- 172.21.15.17:0/1478878038 >> v2:172.21.15.67:3300/0 conn(0x7fa1900a0280 0x7fa190131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 --2- 172.21.15.17:0/1478878038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa190131b20 0x7fa19009eca0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 -- 172.21.15.17:0/1478878038 >> 172.21.15.17:0/1478878038 conn(0x7fa190071270 msgr2=0x7fa1900a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 -- 172.21.15.17:0/1478878038 shutdown_connections 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.484+0000 7fa194a6f640 1 -- 172.21.15.17:0/1478878038 wait complete. 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient: monmap: 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:59.497 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 Processor -- start 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 -- start start 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient: init 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19009f1e0 0x7fa190140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa19009f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 0x7fa19012ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1900a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 0x7fa190098af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa190131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19011c060 con 0x7fa190131b20 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900a09b0 con 0x7fa19009f1e0 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900a5330 con 0x7fa1900a0280 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.485+0000 7fa194a6f640 10 monclient(hunting): authenticate will time out at 899.481925s 2024-09-15T01:53:59.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 0x7fa190098af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 0x7fa19012ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 0x7fa190098af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35958/0 (socket says 172.21.15.17:35958) 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 1 -- 172.21.15.17:0/3462005596 learned_addr learned my addr 172.21.15.17:0/3462005596 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): get_auth_request con 0x7fa1900a0280 auth_method 0 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): get_auth_request con 0x7fa190131b20 auth_method 0 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19009f1e0 0x7fa190140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): get_auth_request con 0x7fa19009f1e0 auth_method 0 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient(hunting): handle_auth_done global_id 54152 payload 824 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19009f1e0 msgr2=0x7fa190140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19009f1e0 0x7fa190140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 msgr2=0x7fa190098af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 0x7fa190098af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 monclient: found mon.noname-c 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18dd9b640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009ae0 con 0x7fa1900a0280 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.486+0000 7fa18ed9d640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190131b20 0x7fa190098af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa18dd9b640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 msgr2=0x7fa19012ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa18dd9b640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 0x7fa19012ec60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa18400d860 tx=0x7fa184008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18400b8d0 con 0x7fa1900a0280 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: dump: 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.503 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18400ba30 con 0x7fa1900a0280 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18400bdb0 con 0x7fa1900a0280 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 5 monclient: authenticate success, global_id 54152 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa17e7fc640 10 monclient: dump: 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.504 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 10 monclient: _renew_subs 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa18cd99640 4 set_mon_vals no callback set 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190141820 con 0x7fa1900a0280 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 10 monclient: _renew_subs 2024-09-15T01:53:59.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.488+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1901414a0 con 0x7fa1900a0280 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa184014640 con 0x7fa1900a0280 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa194a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa194a6f640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 0x7fa190099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19011c400 con 0x7fa19012f1a0 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 0x7fa190099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient: get_auth_request con 0x7fa19012f1a0 auth_method 0 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa17e7fc640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa170038790 0x7fa17003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fa18404c210 con 0x7fa1900a0280 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.489+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.490+0000 7fa18e59c640 10 monclient(hunting): handle_auth_done global_id 44436 payload 824 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.490+0000 7fa18e59c640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1900724a0 con 0x7fa19012f1a0 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.490+0000 7fa18ed9d640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa170038790 0x7fa17003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.490+0000 7fa18ed9d640 10 monclient: get_auth_request con 0x7fa170038790 auth_method 0 2024-09-15T01:53:59.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa18e59c640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 msgr2=0x7fa190099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa18e59c640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 0x7fa190099030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa180033000 tx=0x7fa180036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800079d0 con 0x7fa19012f1a0 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 10 monclient: dump: 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.510 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa18ed9d640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa170038790 0x7fa17003ac40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa188007fd0 tx=0x7fa188007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa180041070 con 0x7fa19012f1a0 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 msgr2=0x7fa190099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 0x7fa190099030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa180033000 tx=0x7fa180036000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 msgr2=0x7fa190099030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.492+0000 7fa17e7fc640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa19012f1a0 0x7fa190099030 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa180033000 tx=0x7fa180036000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa194a6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa194a6f640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 0x7fa1900a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa194a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190063a40 con 0x7fa190099570 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 0x7fa1900a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient: get_auth_request con 0x7fa190099570 auth_method 0 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 10 monclient(hunting): handle_auth_done global_id 44439 payload 824 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.498+0000 7fa18e59c640 1 -- 172.21.15.17:0/3462005596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa190099970 con 0x7fa190099570 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa18e59c640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 msgr2=0x7fa1900a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa18e59c640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 0x7fa1900a2f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa180009880 tx=0x7fa18003f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180032c30 con 0x7fa190099570 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 10 monclient: dump: 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.513 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7fa18003d400 con 0x7fa190099570 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 msgr2=0x7fa1900a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 0x7fa1900a2f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa180009880 tx=0x7fa18003f070 comp rx=0 tx=0).stop 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 msgr2=0x7fa1900a2f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa17e7fc640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa190099570 0x7fa1900a2f90 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa180009880 tx=0x7fa18003f070 comp rx=0 tx=0).stop 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:53:59.515 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.516 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.517 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:59.518 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.519 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.521 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:59.524 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:59.525 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:59.527 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:59.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa170038790 msgr2=0x7fa17003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa170038790 0x7fa17003ac40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa188007fd0 tx=0x7fa188007f00 comp rx=0 tx=0).stop 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 10 monclient: shutdown 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 msgr2=0x7fa19012ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 --2- 172.21.15.17:0/3462005596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa1900a0280 0x7fa19012ec60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa18400d860 tx=0x7fa184008040 comp rx=0 tx=0).stop 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa18ed9d640 1 -- 172.21.15.17:0/3462005596 reap_dead start 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa18ed9d640 1 -- 172.21.15.17:0/3462005596 reap_dead start 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 shutdown_connections 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.501+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 >> 172.21.15.17:0/3462005596 conn(0x7fa190071270 msgr2=0x7fa19009d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.502+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 shutdown_connections 2024-09-15T01:53:59.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.502+0000 7fa194a6f640 1 -- 172.21.15.17:0/3462005596 wait complete. 2024-09-15T01:53:59.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:59.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:53:59.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:53:59.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 10 monclient: monmap: 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:59.626181+0000 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:59.626181+0000 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.628 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.629 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.629 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 1 Processor -- start 2024-09-15T01:53:59.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 1 -- start start 2024-09-15T01:53:59.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.625+0000 7f6360909640 10 monclient: init 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f635c09c1b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f635c09e9e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 0x7f635c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): picked mon.noname-b con 0x7f635c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c09f790 con 0x7f635c09e9e0 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c068b60 con 0x7f635c09c1b0 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c069d40 con 0x7f635c06eb30 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6360909640 10 monclient(hunting): authenticate will time out at 899.622669s 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6359d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f635c09e9e0 0x7f635c09c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52264/0 (socket says 172.21.15.17:52264) 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f6359d9b640 1 -- 172.21.15.17:0/814990430 learned_addr learned my addr 172.21.15.17:0/814990430 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 1 --2- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 0x7f635c06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): get_auth_request con 0x7f635c06eb30 auth_method 0 2024-09-15T01:53:59.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.626+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 10 monclient(hunting): handle_auth_done global_id 54155 payload 274 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 1 -- 172.21.15.17:0/814990430 >> v1:172.21.15.67:6789/0 conn(0x7f635c09c1b0 legacy=0x7f635c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 1 -- 172.21.15.17:0/814990430 >> v1:172.21.15.17:6789/0 conn(0x7f635c09e9e0 legacy=0x7f635c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 1 monclient: found mon.noname-b 2024-09-15T01:53:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.627+0000 7f635ad9d640 1 -- 172.21.15.17:0/814990430 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c069aa0 con 0x7f635c06eb30 2024-09-15T01:53:59.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.629+0000 7f635ad9d640 1 -- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 msgr2=0x7f635c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.629+0000 7f635ad9d640 1 --2- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 0x7f635c06ef90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e90 tx=0x7f6354009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9a9f42ce34d6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 1 -- 172.21.15.17:0/814990430 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354007840 con 0x7f635c06eb30 2024-09-15T01:53:59.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: dump: 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 1 -- 172.21.15.17:0/814990430 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635402d050 con 0x7f635c06eb30 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 5 monclient: authenticate success, global_id 54155 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 4 set_mon_vals no callback set 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 1 -- 172.21.15.17:0/814990430 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354007bd0 con 0x7f635c06eb30 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f635959a640 10 monclient: dump: 2024-09-15T01:53:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 10 monclient: shutdown 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 -- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 msgr2=0x7f635c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 --2- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 0x7f635c06ef90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e90 tx=0x7f6354009780 comp rx=0 tx=0).stop 2024-09-15T01:53:59.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 -- 172.21.15.17:0/814990430 shutdown_connections 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 --2- 172.21.15.17:0/814990430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c06eb30 0x7f635c06ef90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 -- 172.21.15.17:0/814990430 >> 172.21.15.17:0/814990430 conn(0x7f635c071300 msgr2=0x7f635c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 -- 172.21.15.17:0/814990430 shutdown_connections 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.630+0000 7f6360909640 1 -- 172.21.15.17:0/814990430 wait complete. 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient: monmap: 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 Processor -- start 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 -- start start 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient: init 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f635c06eb30 0x7f635c141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): picked mon.noname-a con 0x7f635c06eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c09c1b0 0x7f635c133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): picked mon.noname-b con 0x7f635c09c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 0x7f635c11da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): picked mon.noname-c con 0x7f635c09e9e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c11f5a0 con 0x7f635c06eb30 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1212b0 con 0x7f635c09c1b0 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c142030 con 0x7f635c09e9e0 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.631+0000 7f6360909640 10 monclient(hunting): authenticate will time out at 899.627995s 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 0x7f635c11da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f635c06eb30 0x7f635c141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 0x7f635c11da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39542/0 (socket says 172.21.15.17:39542) 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 learned_addr learned my addr 172.21.15.17:0/1863036904 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): get_auth_request con 0x7f635c06eb30 auth_method 0 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): get_auth_request con 0x7f635c09e9e0 auth_method 0 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f6359d9b640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c09c1b0 0x7f635c133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f6359d9b640 10 monclient(hunting): get_auth_request con 0x7f635c09c1b0 auth_method 0 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f6359d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f6359d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f6359d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient(hunting): handle_auth_done global_id 54158 payload 824 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c09c1b0 msgr2=0x7f635c133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c09c1b0 0x7f635c133eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f635c06eb30 msgr2=0x7f635c141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f635c06eb30 0x7f635c141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 monclient: found mon.noname-c 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354009a20 con 0x7f635c09e9e0 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.632+0000 7f635a59c640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f635c06eb30 0x7f635c141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 msgr2=0x7f635c11da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f635ad9d640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 0x7f635c11da50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e90 tx=0x7f6354037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354007ec0 con 0x7f635c09e9e0 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: dump: 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6354040070 con 0x7f635c09e9e0 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354045430 con 0x7f635c09e9e0 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f634a7fc640 10 monclient: dump: 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.650 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 5 monclient: authenticate success, global_id 54158 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 10 monclient: _renew_subs 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c121030 con 0x7f635c09e9e0 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6358d99640 4 set_mon_vals no callback set 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 10 monclient: _renew_subs 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.634+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c141860 con 0x7f635c09e9e0 2024-09-15T01:53:59.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6354042030 con 0x7f635c09e9e0 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f6360909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f6360909640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 0x7f635c11df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c063bb0 con 0x7f635c121470 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f634a7fc640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f633c0387c0 0x7f633c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f635a59c640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 0x7f635c11df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f635a59c640 10 monclient: get_auth_request con 0x7f635c121470 auth_method 0 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f635a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f635a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f635a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.635+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f635406d780 con 0x7f635c09e9e0 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f6359d9b640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f633c0387c0 0x7f633c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f635a59c640 10 monclient(hunting): handle_auth_done global_id 54161 payload 824 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f635a59c640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f635c120d50 con 0x7f635c121470 2024-09-15T01:53:59.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.636+0000 7f6359d9b640 10 monclient: get_auth_request con 0x7f633c0387c0 auth_method 0 2024-09-15T01:53:59.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f635a59c640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 msgr2=0x7f635c11df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f635a59c640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 0x7f635c11df90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f634c00fa50 tx=0x7f634c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00dda0 con 0x7f635c121470 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 10 monclient: dump: 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f6359d9b640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f633c0387c0 0x7f633c03ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6350007fd0 tx=0x7f6350009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f634c014470 con 0x7f635c121470 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 msgr2=0x7f635c11df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 0x7f635c11df90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f634c00fa50 tx=0x7f634c007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 msgr2=0x7f635c11df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.638+0000 7f634a7fc640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c121470 0x7f635c11df90 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f634c00fa50 tx=0x7f634c007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f6360909640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f6360909640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 0x7f635c11e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f6360909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0724a0 con 0x7f635c1343f0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 0x7f635c11e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient: get_auth_request con 0x7f635c1343f0 auth_method 0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 10 monclient(hunting): handle_auth_done global_id 54164 payload 824 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.644+0000 7f635a59c640 1 -- 172.21.15.17:0/1863036904 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f635c067220 con 0x7f635c1343f0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f635a59c640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 msgr2=0x7f635c11e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f635a59c640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 0x7f635c11e4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f634c019000 tx=0x7f634c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c01ed20 con 0x7f635c1343f0 2024-09-15T01:53:59.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 10 monclient: dump: 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f634c020410 con 0x7f635c1343f0 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 msgr2=0x7f635c11e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 0x7f635c11e4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f634c019000 tx=0x7f634c01a000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 msgr2=0x7f635c11e4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f634a7fc640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c1343f0 0x7f635c11e4d0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f634c019000 tx=0x7f634c01a000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:59.672 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:59.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f633c0387c0 msgr2=0x7f633c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f633c0387c0 0x7f633c03ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6350007fd0 tx=0x7f6350009040 comp rx=0 tx=0).stop 2024-09-15T01:53:59.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 10 monclient: shutdown 2024-09-15T01:53:59.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 msgr2=0x7f635c11da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f6360909640 1 --2- 172.21.15.17:0/1863036904 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f635c09e9e0 0x7f635c11da50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e90 tx=0x7f6354037000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 reap_dead start 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.647+0000 7f635ad9d640 1 -- 172.21.15.17:0/1863036904 reap_dead start 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.648+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 shutdown_connections 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.648+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 >> 172.21.15.17:0/1863036904 conn(0x7f635c071300 msgr2=0x7f635c134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.648+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 shutdown_connections 2024-09-15T01:53:59.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.648+0000 7f6360909640 1 -- 172.21.15.17:0/1863036904 wait complete. 2024-09-15T01:53:59.676 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:59.676 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:53:59.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:53:59.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.772+0000 7faf2d08b640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.772+0000 7faf2d08b640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.772+0000 7faf2d08b640 10 monclient: monmap: 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:59.773603+0000 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:59.773603+0000 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.776 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 Processor -- start 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 -- start start 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient: init 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf28131bd0 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf2809eb10 0x7faf2809ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-c con 0x7faf2809eb10 addr v2:172.21.15.67:3300/0 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faf28136cd0 0x7faf281390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-a con 0x7faf28136cd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2809fd00 con 0x7faf28131bd0 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf281326b0 con 0x7faf28136cd0 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280a15f0 con 0x7faf2809eb10 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2d08b640 10 monclient(hunting): authenticate will time out at 899.769866s 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.773+0000 7faf2759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faf28136cd0 0x7faf281390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faf28136cd0 0x7faf281390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35994/0 (socket says 172.21.15.17:35994) 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 -- 172.21.15.17:0/1184831343 learned_addr learned my addr 172.21.15.17:0/1184831343 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): get_auth_request con 0x7faf28136cd0 auth_method 0 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 1 --2- 172.21.15.17:0/1184831343 >> v2:172.21.15.67:3300/0 conn(0x7faf2809eb10 0x7faf2809ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): get_auth_request con 0x7faf2809eb10 auth_method 0 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient(hunting): handle_auth_done global_id 44445 payload 274 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 -- 172.21.15.17:0/1184831343 >> v2:172.21.15.67:3300/0 conn(0x7faf2809eb10 msgr2=0x7faf2809ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 --2- 172.21.15.17:0/1184831343 >> v2:172.21.15.67:3300/0 conn(0x7faf2809eb10 0x7faf2809ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 -- 172.21.15.17:0/1184831343 >> v1:172.21.15.67:6789/0 conn(0x7faf28131bd0 legacy=0x7faf28131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 monclient: found mon.noname-a 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.774+0000 7faf2759e640 1 -- 172.21.15.17:0/1184831343 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280a1310 con 0x7faf28136cd0 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2759e640 1 -- 172.21.15.17:0/1184831343 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28136cd0 msgr2=0x7faf281390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2759e640 1 --2- 172.21.15.17:0/1184831343 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28136cd0 0x7faf281390b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faf28068b00 tx=0x7faf20009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f057c1346154034 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 1 -- 172.21.15.17:0/1184831343 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20007610 con 0x7faf28136cd0 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: dump: 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.787 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 1 -- 172.21.15.17:0/1184831343 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2002d050 con 0x7faf28136cd0 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 5 monclient: authenticate success, global_id 44445 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 1 -- 172.21.15.17:0/1184831343 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200079a0 con 0x7faf28136cd0 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 4 set_mon_vals no callback set 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf25d9b640 10 monclient: dump: 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 10 monclient: shutdown 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 -- 172.21.15.17:0/1184831343 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28136cd0 msgr2=0x7faf281390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 --2- 172.21.15.17:0/1184831343 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28136cd0 0x7faf281390b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faf28068b00 tx=0x7faf20009840 comp rx=0 tx=0).stop 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 -- 172.21.15.17:0/1184831343 shutdown_connections 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 --2- 172.21.15.17:0/1184831343 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28136cd0 0x7faf281390b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 --2- 172.21.15.17:0/1184831343 >> v2:172.21.15.67:3300/0 conn(0x7faf2809eb10 0x7faf2809ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 -- 172.21.15.17:0/1184831343 >> 172.21.15.17:0/1184831343 conn(0x7faf28071320 msgr2=0x7faf280a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 -- 172.21.15.17:0/1184831343 shutdown_connections 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.777+0000 7faf2d08b640 1 -- 172.21.15.17:0/1184831343 wait complete. 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.778+0000 7faf2d08b640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.778+0000 7faf2d08b640 10 monclient: monmap: 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.791 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.778+0000 7faf2d08b640 1 Processor -- start 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.778+0000 7faf2d08b640 1 -- start start 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.778+0000 7faf2d08b640 10 monclient: init 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf2809eb10 0x7faf28131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-c con 0x7faf2809eb10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 0x7faf28096e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-a con 0x7faf28131bd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28136cd0 0x7faf28097390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): picked mon.noname-b con 0x7faf28136cd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2809bec0 con 0x7faf28131bd0 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280a2550 con 0x7faf28136cd0 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf28098f40 con 0x7faf2809eb10 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2d08b640 10 monclient(hunting): authenticate will time out at 899.775257s 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 0x7faf28096e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf2809eb10 0x7faf28131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 0x7faf28096e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:36006/0 (socket says 172.21.15.17:36006) 2024-09-15T01:53:59.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 -- 172.21.15.17:0/2926226477 learned_addr learned my addr 172.21.15.17:0/2926226477 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): get_auth_request con 0x7faf28131bd0 auth_method 0 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): get_auth_request con 0x7faf2809eb10 auth_method 0 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28136cd0 0x7faf28097390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): get_auth_request con 0x7faf28136cd0 auth_method 0 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient(hunting): handle_auth_done global_id 44448 payload 824 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf2809eb10 msgr2=0x7faf28131250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf2809eb10 0x7faf28131250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28136cd0 msgr2=0x7faf28097390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28136cd0 0x7faf28097390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 monclient: found mon.noname-a 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.779+0000 7faf2659c640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20009ae0 con 0x7faf28131bd0 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.781+0000 7faf2659c640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 msgr2=0x7faf28096e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.781+0000 7faf2659c640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 0x7faf28096e50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00ea20 tx=0x7faf1c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c009600 con 0x7faf28131bd0 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:59.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: dump: 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1c00cc20 con 0x7faf28131bd0 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 5 monclient: authenticate success, global_id 44448 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 10 monclient: _renew_subs 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c013470 con 0x7faf28131bd0 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf28098a30 con 0x7faf28131bd0 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf0effd640 10 monclient: dump: 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2559a640 4 set_mon_vals no callback set 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 10 monclient: _renew_subs 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.782+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2809b1b0 con 0x7faf28131bd0 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2d08b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2d08b640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 0x7faf280978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf28063a20 con 0x7faf2809b670 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faf1c013860 con 0x7faf28131bd0 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 0x7faf280978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient: get_auth_request con 0x7faf2809b670 auth_method 0 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf0effd640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf14038790 0x7faf1403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7faf1c04bf80 con 0x7faf28131bd0 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2759e640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf14038790 0x7faf1403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf2759e640 10 monclient: get_auth_request con 0x7faf14038790 auth_method 0 2024-09-15T01:53:59.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 10 monclient(hunting): handle_auth_done global_id 5017 payload 824 2024-09-15T01:53:59.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.783+0000 7faf26d9d640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf2809a600 con 0x7faf2809b670 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf26d9d640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 msgr2=0x7faf280978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf26d9d640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 0x7faf280978d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1800d8b0 con 0x7faf2809b670 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 10 monclient: dump: 2024-09-15T01:53:59.803 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf2759e640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf14038790 0x7faf1403ac40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf280a5fd0 tx=0x7faf20036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faf1800da10 con 0x7faf2809b670 2024-09-15T01:53:59.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 msgr2=0x7faf280978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 0x7faf280978d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 msgr2=0x7faf280978d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.786+0000 7faf0effd640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf2809b670 0x7faf280978d0 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18007e10 comp rx=0 tx=0).stop 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf2d08b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf2d08b640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 0x7faf28132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf2d08b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280724a0 con 0x7faf281329f0 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 0x7faf28132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient: get_auth_request con 0x7faf281329f0 auth_method 0 2024-09-15T01:53:59.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.792+0000 7faf26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.793+0000 7faf26d9d640 10 monclient(hunting): handle_auth_done global_id 5020 payload 824 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.793+0000 7faf26d9d640 1 -- 172.21.15.17:0/2926226477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faf2806efc0 con 0x7faf281329f0 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf26d9d640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 msgr2=0x7faf28132df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf26d9d640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 0x7faf28132df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18007030 con 0x7faf281329f0 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:53:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.795+0000 7faf0effd640 10 monclient: dump: 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7faf18007190 con 0x7faf281329f0 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 msgr2=0x7faf28132df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 0x7faf28132df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18019000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 msgr2=0x7faf28132df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf0effd640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281329f0 0x7faf28132df0 secure :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf1800f780 tx=0x7faf18019000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 32, 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:53:59.808 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.809 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.810 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:53:59.811 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:53:59.812 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:53:59.813 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.815 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:53:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.819 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:53:59.820 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:53:59.821 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf14038790 msgr2=0x7faf1403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf14038790 0x7faf1403ac40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf280a5fd0 tx=0x7faf20036000 comp rx=0 tx=0).stop 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 10 monclient: shutdown 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 msgr2=0x7faf28096e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 --2- 172.21.15.17:0/2926226477 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf28131bd0 0x7faf28096e50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00ea20 tx=0x7faf1c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2759e640 1 -- 172.21.15.17:0/2926226477 reap_dead start 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2759e640 1 -- 172.21.15.17:0/2926226477 reap_dead start 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 shutdown_connections 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 >> 172.21.15.17:0/2926226477 conn(0x7faf28071320 msgr2=0x7faf2809d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 shutdown_connections 2024-09-15T01:53:59.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.796+0000 7faf2d08b640 1 -- 172.21.15.17:0/2926226477 wait complete. 2024-09-15T01:53:59.824 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:53:59.824 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T01:53:59.824 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:53:59.824 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T01:53:59.824 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:53:59.825 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T01:53:59.825 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:53:59.834 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T01:53:59.834 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:53:59.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 10 monclient: get_monmap_and_config 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 10 monclient: monmap: 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.969 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:53:59.968069+0000 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:53:59.968069+0000 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:53:59.970 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 1 Processor -- start 2024-09-15T01:53:59.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 1 -- start start 2024-09-15T01:53:59.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.967+0000 7f7a15899640 10 monclient: init 2024-09-15T01:53:59.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7a10067090 0x7f7a101321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-a con 0x7f7a10067090 addr v2:172.21.15.17:3300/0 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a10067460 addr v1:172.21.15.67:6789/0 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a10136f90 addr v1:172.21.15.17:6789/0 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1009cbc0 con 0x7f7a10136f90 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1009e550 con 0x7f7a10067460 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a10067f40 con 0x7f7a10067090 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a15899640 10 monclient(hunting): authenticate will time out at 899.964380s 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7a10067090 0x7f7a101321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7a10136f90 0x7f7a10139370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52266/0 (socket says 172.21.15.17:52266) 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7a10067090 0x7f7a101321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:36022/0 (socket says 172.21.15.17:36022) 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0f7fe640 1 -- 172.21.15.17:0/1123405982 learned_addr learned my addr 172.21.15.17:0/1123405982 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): get_auth_request con 0x7f7a10067090 auth_method 0 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient(hunting): handle_auth_done global_id 44451 payload 274 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 -- 172.21.15.17:0/1123405982 >> v1:172.21.15.67:6789/0 conn(0x7f7a10067460 legacy=0x7f7a10067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 -- 172.21.15.17:0/1123405982 >> v1:172.21.15.17:6789/0 conn(0x7f7a10136f90 legacy=0x7f7a10139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:53:59.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 monclient: found mon.noname-a 2024-09-15T01:53:59.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.968+0000 7f7a0effd640 1 -- 172.21.15.17:0/1123405982 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1009e1e0 con 0x7f7a10067090 2024-09-15T01:53:59.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.971+0000 7f7a0effd640 1 -- 172.21.15.17:0/1123405982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067090 msgr2=0x7f7a101321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.971+0000 7f7a0effd640 1 --2- 172.21.15.17:0/1123405982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067090 0x7f7a101321a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009850 tx=0x7f7a000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5f901a9a5fe4c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 1 -- 172.21.15.17:0/1123405982 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00032440 con 0x7f7a10067090 2024-09-15T01:53:59.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: dump: 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.981 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 1 -- 172.21.15.17:0/1123405982 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a000325a0 con 0x7f7a10067090 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 1 -- 172.21.15.17:0/1123405982 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00032940 con 0x7f7a10067090 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 5 monclient: authenticate success, global_id 44451 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 10 monclient: get_monmap_and_config success 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 4 set_mon_vals no callback set 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a0dffb640 10 monclient: dump: 2024-09-15T01:53:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 10 monclient: shutdown 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 -- 172.21.15.17:0/1123405982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067090 msgr2=0x7f7a101321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 --2- 172.21.15.17:0/1123405982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067090 0x7f7a101321a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009850 tx=0x7f7a000090f0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 -- 172.21.15.17:0/1123405982 shutdown_connections 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 --2- 172.21.15.17:0/1123405982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067090 0x7f7a101321a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 -- 172.21.15.17:0/1123405982 >> 172.21.15.17:0/1123405982 conn(0x7f7a100a1ee0 msgr2=0x7f7a100a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 -- 172.21.15.17:0/1123405982 shutdown_connections 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.972+0000 7f7a15899640 1 -- 172.21.15.17:0/1123405982 wait complete. 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.973+0000 7f7a15899640 10 monclient: build_initial_monmap 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.973+0000 7f7a15899640 10 monclient: monmap: 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.973+0000 7f7a15899640 1 Processor -- start 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.973+0000 7f7a15899640 1 -- start start 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.973+0000 7f7a15899640 10 monclient: init 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient: _reopen_session rank -1 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 0x7f7a10141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-c con 0x7f7a10067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-a con 0x7f7a10067460 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10136f90 0x7f7a10118940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): picked mon.noname-b con 0x7f7a10136f90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011c180 con 0x7f7a10067460 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011bea0 con 0x7f7a10136f90 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0effd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 msgr2=0x7f7a10141630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011c460 con 0x7f7a10067090 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): _renew_subs 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a15899640 10 monclient(hunting): authenticate will time out at 899.970148s 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 0x7f7a10141630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:53:59.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:36024/0 (socket says 172.21.15.17:36024) 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 -- 172.21.15.17:0/921022410 learned_addr learned my addr 172.21.15.17:0/921022410 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): get_auth_request con 0x7f7a10067460 auth_method 0 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0f7fe640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10136f90 0x7f7a10118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0f7fe640 10 monclient(hunting): get_auth_request con 0x7f7a10136f90 auth_method 0 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:53:59.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient(hunting): handle_auth_done global_id 44454 payload 824 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 -- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 msgr2=0x7f7a10141630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 0x7f7a10141630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 -- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10136f90 msgr2=0x7f7a10118940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10136f90 0x7f7a10118940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 monclient: found mon.noname-a 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.974+0000 7f7a0e7fc640 1 -- 172.21.15.17:0/921022410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00009390 con 0x7f7a10067460 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a0e7fc640 1 -- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 msgr2=0x7f7a10141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:53:59.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a0e7fc640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c5e0 tx=0x7f7a04007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04012840 con 0x7f7a10067460 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: dump: 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.989 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a040129a0 con 0x7f7a10067460 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04012da0 con 0x7f7a10067460 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 5 monclient: authenticate success, global_id 44454 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f79f6ffd640 10 monclient: dump: 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:53:59.990 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 10 monclient: _renew_subs 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a0d7fa640 4 set_mon_vals no callback set 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1011af40 con 0x7f7a10067460 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 10 monclient: _renew_subs 2024-09-15T01:53:59.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.976+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a101420b0 con 0x7f7a10067460 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a04011590 con 0x7f7a10067460 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f7a15899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f7a15899640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f7a15899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1011c890 con 0x7f7a10067460 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f79f6ffd640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 0x7f79fc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.977+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f7a0404bde0 con 0x7f7a10067460 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.978+0000 7f7a0effd640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 0x7f79fc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:53:59.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.978+0000 7f7a0effd640 10 monclient: get_auth_request con 0x7f79fc0386f0 auth_method 0 2024-09-15T01:53:59.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.980+0000 7f7a0effd640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 0x7f79fc03aba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a10131ea0 tx=0x7f7a00032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:53:59.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.982+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7a04010aa0 con 0x7f7a10067460 2024-09-15T01:53:59.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.982+0000 7f79f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:53:59.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:53:59.982+0000 7f79f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.210+0000 7f7a15899640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.210+0000 7f7a15899640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.210+0000 7f7a15899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.210+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a100724a0 con 0x7f7a10067460 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.210+0000 7f79f6ffd640 1 -- 172.21.15.17:0/921022410 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f7a04016070 con 0x7f7a10067460 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.211+0000 7f79f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.211+0000 7f79f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:00.213 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:00.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 msgr2=0x7f79fc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:00.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 0x7f79fc03aba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a10131ea0 tx=0x7f7a00032000 comp rx=0 tx=0).stop 2024-09-15T01:54:00.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 10 monclient: shutdown 2024-09-15T01:54:00.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:00.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 msgr2=0x7f7a10141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c5e0 tx=0x7f7a04007ed0 comp rx=0 tx=0).stop 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 shutdown_connections 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f79fc0386f0 0x7f79fc03aba0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a10067460 0x7f7a10141b70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10136f90 0x7f7a10118940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 --2- 172.21.15.17:0/921022410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a10067090 0x7f7a10141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 >> 172.21.15.17:0/921022410 conn(0x7f7a100a1ee0 msgr2=0x7f7a10097ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 shutdown_connections 2024-09-15T01:54:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:00.213+0000 7f7a15899640 1 -- 172.21.15.17:0/921022410 wait complete. 2024-09-15T01:54:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:02.110+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:02.110+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:02.110+0000 7fdde97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:03.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:03.111+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:03.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:03.111+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:03.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 10 monclient: build_initial_monmap 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 10 monclient: monmap: 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:03.333011+0000 2024-09-15T01:54:03.334 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:03.333011+0000 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:03.335 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 1 Processor -- start 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 1 -- start start 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.332+0000 7efda095d640 10 monclient: init 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd9c131b20 0x7efd9c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efd9c0a0280 0x7efd9c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c0a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd9c09f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0a0040 con 0x7efd9c09f1e0 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c09bff0 con 0x7efd9c0a0280 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0a14a0 con 0x7efd9c131b20 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efda095d640 10 monclient(hunting): authenticate will time out at 903.329566s 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efd9c0a0280 0x7efd9c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9ad9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7efd9c09f1e0 0x7efd9c068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34676/0 (socket says 172.21.15.17:34676) 2024-09-15T01:54:03.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9ad9d640 1 -- 172.21.15.17:0/664615749 learned_addr learned my addr 172.21.15.17:0/664615749 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): get_auth_request con 0x7efd9c0a0280 auth_method 0 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9a59c640 1 --2- 172.21.15.17:0/664615749 >> v2:172.21.15.67:3300/0 conn(0x7efd9c131b20 0x7efd9c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9a59c640 10 monclient(hunting): get_auth_request con 0x7efd9c131b20 auth_method 0 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.333+0000 7efd9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:03.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 10 monclient(hunting): handle_auth_done global_id 44457 payload 274 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 1 -- 172.21.15.17:0/664615749 >> v2:172.21.15.67:3300/0 conn(0x7efd9c131b20 msgr2=0x7efd9c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 1 --2- 172.21.15.17:0/664615749 >> v2:172.21.15.67:3300/0 conn(0x7efd9c131b20 0x7efd9c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 1 -- 172.21.15.17:0/664615749 >> v1:172.21.15.17:6789/0 conn(0x7efd9c09f1e0 legacy=0x7efd9c068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 1 monclient: found mon.noname-a 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.334+0000 7efd99d9b640 1 -- 172.21.15.17:0/664615749 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c132430 con 0x7efd9c0a0280 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.336+0000 7efd99d9b640 1 -- 172.21.15.17:0/664615749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c0a0280 msgr2=0x7efd9c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.336+0000 7efd99d9b640 1 --2- 172.21.15.17:0/664615749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c0a0280 0x7efd9c131ef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd9c09f710 tx=0x7efd900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9933b91540abe968 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 1 -- 172.21.15.17:0/664615749 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9002ae00 con 0x7efd9c0a0280 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:03.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: dump: 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: _finish_auth 0 2024-09-15T01:54:03.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 1 -- 172.21.15.17:0/664615749 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90032440 con 0x7efd9c0a0280 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 5 monclient: authenticate success, global_id 44457 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 1 -- 172.21.15.17:0/664615749 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90032820 con 0x7efd9c0a0280 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 4 set_mon_vals no callback set 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:03.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efd9959a640 10 monclient: dump: 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 10 monclient: shutdown 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 -- 172.21.15.17:0/664615749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c0a0280 msgr2=0x7efd9c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 --2- 172.21.15.17:0/664615749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c0a0280 0x7efd9c131ef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd9c09f710 tx=0x7efd900090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 -- 172.21.15.17:0/664615749 shutdown_connections 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 --2- 172.21.15.17:0/664615749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c0a0280 0x7efd9c131ef0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 --2- 172.21.15.17:0/664615749 >> v2:172.21.15.67:3300/0 conn(0x7efd9c131b20 0x7efd9c09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 -- 172.21.15.17:0/664615749 >> 172.21.15.17:0/664615749 conn(0x7efd9c071270 msgr2=0x7efd9c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 -- 172.21.15.17:0/664615749 shutdown_connections 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.337+0000 7efda095d640 1 -- 172.21.15.17:0/664615749 wait complete. 2024-09-15T01:54:03.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.338+0000 7efda095d640 10 monclient: build_initial_monmap 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.338+0000 7efda095d640 10 monclient: monmap: 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.338+0000 7efda095d640 1 Processor -- start 2024-09-15T01:54:03.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.338+0000 7efda095d640 1 -- start start 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.338+0000 7efda095d640 10 monclient: init 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 0x7efd9c1410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c0a0280 0x7efd9c12eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): picked mon.noname-b con 0x7efd9c0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9a59c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 msgr2=0x7efd9c1410d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c11bfc0 con 0x7efd9c131b20 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0a09b0 con 0x7efd9c0a0280 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 0x7efd9c1410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c141fb0 con 0x7efd9c09f1e0 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efda095d640 10 monclient(hunting): authenticate will time out at 903.335248s 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47248/0 (socket says 172.21.15.17:47248) 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 -- 172.21.15.17:0/1590092537 learned_addr learned my addr 172.21.15.17:0/1590092537 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): get_auth_request con 0x7efd9c131b20 auth_method 0 2024-09-15T01:54:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c0a0280 0x7efd9c12eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): get_auth_request con 0x7efd9c0a0280 auth_method 0 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient(hunting): handle_auth_done global_id 44460 payload 824 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 -- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 msgr2=0x7efd9c1410d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 0x7efd9c1410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 -- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c0a0280 msgr2=0x7efd9c12eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c0a0280 0x7efd9c12eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 monclient: found mon.noname-a 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.339+0000 7efd9ad9d640 1 -- 172.21.15.17:0/1590092537 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd90009390 con 0x7efd9c131b20 2024-09-15T01:54:03.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.341+0000 7efd9ad9d640 1 -- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 msgr2=0x7efd9c098d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.341+0000 7efd9ad9d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd9400ccc0 tx=0x7efd94005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400ce80 con 0x7efd9c131b20 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: dump: 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9400aa60 con 0x7efd9c131b20 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 5 monclient: authenticate success, global_id 44460 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 10 monclient: _renew_subs 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c0a5360 con 0x7efd9c131b20 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400ae00 con 0x7efd9c131b20 2024-09-15T01:54:03.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd8a7fc640 10 monclient: dump: 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:03.358 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 10 monclient: _renew_subs 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efd98d99640 4 set_mon_vals no callback set 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.342+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c09a860 con 0x7efd9c131b20 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efd94007300 con 0x7efd9c131b20 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efda095d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efda095d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efda095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c0724a0 con 0x7efd9c131b20 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efd8a7fc640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 0x7efd7c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7efd9404bdf0 con 0x7efd9c131b20 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efd9a59c640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 0x7efd7c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:03.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.343+0000 7efd9a59c640 10 monclient: get_auth_request con 0x7efd7c0386f0 auth_method 0 2024-09-15T01:54:03.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.346+0000 7efd9a59c640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 0x7efd7c03aba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd8c006f90 tx=0x7efd8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:03.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.347+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efd9404f030 con 0x7efd9c131b20 2024-09-15T01:54:03.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.347+0000 7efd8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:03.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.347+0000 7efd8a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.573+0000 7efda095d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.573+0000 7efda095d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.573+0000 7efda095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.573+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efd9c06efc0 con 0x7efd9c131b20 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.574+0000 7efd8a7fc640 1 -- 172.21.15.17:0/1590092537 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7efd940075b0 con 0x7efd9c131b20 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.574+0000 7efd8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.574+0000 7efd8a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:03.576 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 msgr2=0x7efd7c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 0x7efd7c03aba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd8c006f90 tx=0x7efd8c008040 comp rx=0 tx=0).stop 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 10 monclient: shutdown 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 msgr2=0x7efd9c098d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd9400ccc0 tx=0x7efd94005370 comp rx=0 tx=0).stop 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 shutdown_connections 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efd7c0386f0 0x7efd7c03aba0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efd9c131b20 0x7efd9c098d70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c0a0280 0x7efd9c12eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 --2- 172.21.15.17:0/1590092537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efd9c09f1e0 0x7efd9c1410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 >> 172.21.15.17:0/1590092537 conn(0x7efd9c071270 msgr2=0x7efd9c09d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 shutdown_connections 2024-09-15T01:54:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:03.576+0000 7efda095d640 1 -- 172.21.15.17:0/1590092537 wait complete. 2024-09-15T01:54:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:04.111+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:04.111+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:05.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:05.111+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:05.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:05.111+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:06.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:06.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:06.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:06.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:06.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:06.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.694+0000 7f4606080640 10 monclient: get_monmap_and_config 2024-09-15T01:54:06.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.694+0000 7f4606080640 10 monclient: build_initial_monmap 2024-09-15T01:54:06.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.694+0000 7f4606080640 10 monclient: monmap: 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:06.696345+0000 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:06.696345+0000 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:06.698 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:06.699 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:06.699 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:06.699 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:06.699 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 Processor -- start 2024-09-15T01:54:06.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 -- start start 2024-09-15T01:54:06.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient: init 2024-09-15T01:54:06.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:06.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 0x7f460009a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): picked mon.noname-b con 0x7f460006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f460006ef00 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f460006f2d0 0x7f460009e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): picked mon.noname-a con 0x7f460006f2d0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4600069f20 con 0x7f460006ef00 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4600099f40 con 0x7f460006f2d0 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460006a550 con 0x7f460006eb30 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f4606080640 10 monclient(hunting): authenticate will time out at 906.692872s 2024-09-15T01:54:06.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f45ff7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 msgr2=0x7f460009a960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.695+0000 7f45ff7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 0x7f460009a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f460006f2d0 0x7f460009e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f460006f2d0 0x7f460009e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47258/0 (socket says 172.21.15.17:47258) 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 -- 172.21.15.17:0/4084546610 learned_addr learned my addr 172.21.15.17:0/4084546610 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): get_auth_request con 0x7f460006f2d0 auth_method 0 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:06.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient(hunting): handle_auth_done global_id 44463 payload 274 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 -- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 msgr2=0x7f460009a960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 --2- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 0x7f460009a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 -- 172.21.15.17:0/4084546610 >> v1:172.21.15.67:6789/0 conn(0x7f460006ef00 legacy=0x7f460009bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 monclient: found mon.noname-a 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.696+0000 7f45fffff640 1 -- 172.21.15.17:0/4084546610 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600069930 con 0x7f460006f2d0 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fffff640 1 -- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006f2d0 msgr2=0x7f460009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fffff640 1 --2- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006f2d0 0x7f460009e970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f460006a6f0 tx=0x7f45f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02f1bd7a8e0a86a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 1 -- 172.21.15.17:0/4084546610 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f8007610 con 0x7f460006f2d0 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: dump: 2024-09-15T01:54:06.711 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:06.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 1 -- 172.21.15.17:0/4084546610 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f802d050 con 0x7f460006f2d0 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 5 monclient: authenticate success, global_id 44463 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 4 set_mon_vals no callback set 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 1 -- 172.21.15.17:0/4084546610 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f80079e0 con 0x7f460006f2d0 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f45fe7fc640 10 monclient: dump: 2024-09-15T01:54:06.713 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:06.714 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 10 monclient: shutdown 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 -- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006f2d0 msgr2=0x7f460009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 --2- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006f2d0 0x7f460009e970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f460006a6f0 tx=0x7f45f8009840 comp rx=0 tx=0).stop 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 -- 172.21.15.17:0/4084546610 shutdown_connections 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 --2- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006f2d0 0x7f460009e970 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 --2- 172.21.15.17:0/4084546610 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006eb30 0x7f460009a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.699+0000 7f4606080640 1 -- 172.21.15.17:0/4084546610 >> 172.21.15.17:0/4084546610 conn(0x7f46000712f0 msgr2=0x7f46000a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.700+0000 7f4606080640 1 -- 172.21.15.17:0/4084546610 shutdown_connections 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.700+0000 7f4606080640 1 -- 172.21.15.17:0/4084546610 wait complete. 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.700+0000 7f4606080640 10 monclient: build_initial_monmap 2024-09-15T01:54:06.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.700+0000 7f4606080640 10 monclient: monmap: 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 Processor -- start 2024-09-15T01:54:06.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 -- start start 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient: init 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): picked mon.noname-a con 0x7f460006eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460006ef00 0x7f4600141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): picked mon.noname-b con 0x7f460006ef00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 0x7f46001415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): picked mon.noname-c con 0x7f460006f2d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4600141b00 con 0x7f460006eb30 2024-09-15T01:54:06.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460009ccd0 con 0x7f460006ef00 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460009b490 con 0x7f460006f2d0 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f4606080640 10 monclient(hunting): authenticate will time out at 906.698632s 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45fffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 msgr2=0x7f46001415c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45fffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 0x7f46001415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47272/0 (socket says 172.21.15.17:47272) 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 1 -- 172.21.15.17:0/2534021208 learned_addr learned my addr 172.21.15.17:0/2534021208 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): get_auth_request con 0x7f460006eb30 auth_method 0 2024-09-15T01:54:06.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460006ef00 0x7f4600141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 10 monclient(hunting): get_auth_request con 0x7f460006ef00 auth_method 0 2024-09-15T01:54:06.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.701+0000 7f45feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 10 monclient(hunting): handle_auth_done global_id 44466 payload 824 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 -- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 msgr2=0x7f46001415c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 0x7f46001415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 -- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460006ef00 msgr2=0x7f4600141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460006ef00 0x7f4600141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.702+0000 7f45ff7fe640 1 -- 172.21.15.17:0/2534021208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f8009ae0 con 0x7f460006eb30 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45ff7fe640 1 -- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 msgr2=0x7f4600134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45ff7fe640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45f000b5c0 tx=0x7f45f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0003d90 con 0x7f460006eb30 2024-09-15T01:54:06.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: dump: 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f0006150 con 0x7f460006eb30 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0006550 con 0x7f460006eb30 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 5 monclient: authenticate success, global_id 44466 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 10 monclient: _renew_subs 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46001204f0 con 0x7f460006eb30 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45e77fe640 10 monclient: dump: 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:06.723 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f45fdffb640 4 set_mon_vals no callback set 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 10 monclient: _renew_subs 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.704+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46001353d0 con 0x7f460006eb30 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f45f00105d0 con 0x7f460006eb30 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f4606080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f4606080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f4606080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46000724a0 con 0x7f460006eb30 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f45e77fe640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 0x7f45ec03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f45f004b3b0 con 0x7f460006eb30 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.705+0000 7f45feffd640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 0x7f45ec03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.706+0000 7f45feffd640 10 monclient: get_auth_request con 0x7f45ec0386f0 auth_method 0 2024-09-15T01:54:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.708+0000 7f45feffd640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 0x7f45ec03aba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4600120180 tx=0x7f45f4007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:06.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.708+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:54:06.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.710+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f45f000e8c0 con 0x7f460006eb30 2024-09-15T01:54:06.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.710+0000 7f45e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:06.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.710+0000 7f45e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f4606080640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f4606080640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f4606080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4600121c20 con 0x7f460006eb30 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f45e77fe640 1 -- 172.21.15.17:0/2534021208 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f45f0017080 con 0x7f460006eb30 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f45e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.939+0000 7f45e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:06.943 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:06.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 msgr2=0x7f45ec03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 0x7f45ec03aba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4600120180 tx=0x7f45f4007e10 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 10 monclient: shutdown 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 msgr2=0x7f4600134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45f000b5c0 tx=0x7f45f0009120 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 shutdown_connections 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f45ec0386f0 0x7f45ec03aba0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f460006f2d0 0x7f46001415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460006ef00 0x7f4600141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 --2- 172.21.15.17:0/2534021208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f460006eb30 0x7f4600134e90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 >> 172.21.15.17:0/2534021208 conn(0x7f46000712f0 msgr2=0x7f460009e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:06.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 shutdown_connections 2024-09-15T01:54:06.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:06.941+0000 7f4606080640 1 -- 172.21.15.17:0/2534021208 wait complete. 2024-09-15T01:54:07.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:07.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:07.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:07.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:08.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:08.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:08.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:08.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:09.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:09.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:09.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:10.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.061+0000 7f8ac4f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.061+0000 7f8ac4f9d640 10 monclient: build_initial_monmap 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.061+0000 7f8ac4f9d640 10 monclient: monmap: 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:10.063743+0000 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:10.063743+0000 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:10.065 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:10.066 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:10.066 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:10.066 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.062+0000 7f8ac4f9d640 1 Processor -- start 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.062+0000 7f8ac4f9d640 1 -- start start 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.062+0000 7f8ac4f9d640 10 monclient: init 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac0068f20 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac00692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac012e900 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac00a1950 con 0x7f8ac0068f20 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac00a1670 con 0x7f8ac012e900 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00a0020 con 0x7f8ac00692f0 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8ac4f9d640 10 monclient(hunting): authenticate will time out at 910.060296s 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abdd9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 msgr2=0x7f8ac00696c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:10.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47282/0 (socket says 172.21.15.17:47282) 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 learned_addr learned my addr 172.21.15.17:0/134618386 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 10 monclient(hunting): get_auth_request con 0x7f8ac0068f20 auth_method 0 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 10 monclient(hunting): get_auth_request con 0x7f8ac012e900 auth_method 0 2024-09-15T01:54:10.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.063+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:10.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:10.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abdd9b640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 msgr2=0x7f8ac00696c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abdd9b640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 10 monclient(hunting): get_auth_request con 0x7f8ac0068f20 auth_method 0 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 10 monclient(hunting): get_auth_request con 0x7f8ac012e900 auth_method 0 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.264+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:54:10.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abdd9b640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 msgr2=0x7f8ac00696c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:10.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abdd9b640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:54:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 10 monclient(hunting): get_auth_request con 0x7f8ac0068f20 auth_method 0 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 10 monclient(hunting): get_auth_request con 0x7f8ac012e900 auth_method 0 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:10.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:10.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:10.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:10.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:10.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:10.665+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:54:11.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:11.112+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:11.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:11.112+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abdd9b640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 msgr2=0x7f8ac00696c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abdd9b640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:54:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 10 monclient(hunting): get_auth_request con 0x7f8ac0068f20 auth_method 0 2024-09-15T01:54:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 10 monclient(hunting): get_auth_request con 0x7f8ac012e900 auth_method 0 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:11.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:54:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:11.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:11.466+0000 7f8abed9d640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:54:12.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:12.113+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:12.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:12.113+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:12.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:12.113+0000 7fdde97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:13.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient(hunting): tick 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 monclient(hunting): continuing hunt 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 msgr2=0x7f8ac00696c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8ac00692f0 0x7f8ac00696c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 msgr2=0x7f8ac012ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.67:3300/0 conn(0x7f8ac012e900 0x7f8ac012ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 -- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 msgr2=0x7f8ac01321c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 --2- 172.21.15.17:0/134618386 >> v2:172.21.15.17:3300/0 conn(0x7f8ac0068f20 0x7f8ac01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient: _add_conns ranks=[0,1] 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8aa0003670 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:13.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8aa0007310 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 -- 172.21.15.17:0/134618386 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00a1390 con 0x7f8aa0003670 2024-09-15T01:54:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 1 -- 172.21.15.17:0/134618386 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa000c040 con 0x7f8aa0007310 2024-09-15T01:54:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.064+0000 7f8ab7fff640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.065+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> v1:172.21.15.17:6789/0 conn(0x7f8aa0003670 legacy=0x7f8aa0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:13.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:13.066+0000 7f8abdd9b640 1 -- 172.21.15.17:0/134618386 >> v1:172.21.15.67:6789/0 conn(0x7f8aa0007310 legacy=0x7f8aa0009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:13.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:13.113+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:13.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:13.113+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:14.113+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:14.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:14.113+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.188+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666624345 0 0) 0x7f8ac00a1390 con 0x7f8aa0003670 2024-09-15T01:54:14.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.188+0000 7f8abd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.188+0000 7f8abd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.188+0000 7f8abd59a640 10 monclient(hunting): my global_id is 44469 2024-09-15T01:54:14.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.188+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac0034e0 con 0x7f8aa0003670 2024-09-15T01:54:14.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3249447193 0 0) 0x7f8aac0034e0 con 0x7f8aa0003670 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 >> v1:172.21.15.67:6789/0 conn(0x7f8aa0007310 legacy=0x7f8aa0009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa000d240 con 0x7f8aa0003670 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f8ab000c680 con 0x7f8aa0003670 2024-09-15T01:54:14.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8ac4f9d640 5 monclient: authenticate success, global_id 44469 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8ac4f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: dump: 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.196 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 >> v1:172.21.15.17:6789/0 conn(0x7f8aa0003670 legacy=0x7f8aa0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abed9d640 1 -- 172.21.15.17:0/134618386 reap_dead start 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 0x7f8aac003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): picked mon.b con 0x7f8aac003a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 0x7f8aac004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): picked mon.c con 0x7f8aac004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac005330 0x7f8aac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): picked mon.a con 0x7f8aac005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac00c230 con 0x7f8aac005330 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac00c3a0 con 0x7f8aac003a00 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac00c560 con 0x7f8aac004780 2024-09-15T01:54:14.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abd59a640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8ac4f9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abdd9b640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 msgr2=0x7f8aac004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abdd9b640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 0x7f8aac004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 0x7f8aac003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): get_auth_request con 0x7f8aac003a00 auth_method 0 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.189+0000 7f8abe59c640 10 monclient(hunting): handle_auth_done global_id 44469 payload 293 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 msgr2=0x7f8aac004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 0x7f8aac004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac005330 msgr2=0x7f8aac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac005330 0x7f8aac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 monclient: found mon.b 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0034e0 con 0x7f8aac003a00 2024-09-15T01:54:14.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.190+0000 7f8abe59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.193+0000 7f8abe59c640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 msgr2=0x7f8aac003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.193+0000 7f8abe59c640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 0x7f8aac003df0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0006fd0 tx=0x7f8ab0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b78b4bc75995bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0034410 con 0x7f8aac003a00 2024-09-15T01:54:14.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: dump: 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab002e030 con 0x7f8aac003a00 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8ac4f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8ac4f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 1 -- 172.21.15.17:0/134618386 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00345b0 con 0x7f8aac003a00 2024-09-15T01:54:14.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8ac4f9d640 4 set_mon_vals no callback set 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.194+0000 7f8abd59a640 10 monclient: dump: 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.209 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 10 monclient: shutdown 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 msgr2=0x7f8aac003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 0x7f8aac003df0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0006fd0 tx=0x7f8ab0008040 comp rx=0 tx=0).stop 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/134618386 shutdown_connections 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac005330 0x7f8aac00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac004780 0x7f8aac004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/134618386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac003a00 0x7f8aac003df0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/134618386 >> 172.21.15.17:0/134618386 conn(0x7f8ac0070fb0 msgr2=0x7f8ac00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/134618386 shutdown_connections 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.195+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/134618386 wait complete. 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.196+0000 7f8ac4f9d640 10 monclient: build_initial_monmap 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.196+0000 7f8ac4f9d640 10 monclient: monmap: 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.211 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.196+0000 7f8ac4f9d640 1 Processor -- start 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.196+0000 7f8ac4f9d640 1 -- start start 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.196+0000 7f8ac4f9d640 10 monclient: init 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 0x7f8ac0140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8aac003a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:14.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8aac004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 0x7f8ac009a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8aac005330 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abe59c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 msgr2=0x7f8ac0140d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00a0750 con 0x7f8aac004780 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00a45b0 con 0x7f8aac005330 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac011dd20 con 0x7f8aac003a00 2024-09-15T01:54:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abe59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 0x7f8ac0140d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8ac4f9d640 10 monclient(hunting): authenticate will time out at 914.194459s 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51792/0 (socket says 172.21.15.17:51792) 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 1 -- 172.21.15.17:0/3906702045 learned_addr learned my addr 172.21.15.17:0/3906702045 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): get_auth_request con 0x7f8aac004780 auth_method 0 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 0x7f8ac009a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): get_auth_request con 0x7f8aac005330 auth_method 0 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient(hunting): handle_auth_done global_id 5035 payload 824 2024-09-15T01:54:14.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 -- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 msgr2=0x7f8ac0140d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 0x7f8ac0140d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 -- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 msgr2=0x7f8ac0141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 monclient: found mon.noname-b 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abed9d640 1 -- 172.21.15.17:0/3906702045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac011c870 con 0x7f8aac005330 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.197+0000 7f8abdd9b640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8abed9d640 1 -- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 msgr2=0x7f8ac009a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8abed9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 0x7f8ac009a930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8ac009c290 tx=0x7f8ab800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8017850 con 0x7f8aac005330 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: dump: 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.217 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab800f040 con 0x7f8aac005330 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8017c40 con 0x7f8aac005330 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 5 monclient: authenticate success, global_id 5035 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 10 monclient: _renew_subs 2024-09-15T01:54:14.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac011d7c0 con 0x7f8aac005330 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8abcd99640 4 set_mon_vals no callback set 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ab67fc640 10 monclient: dump: 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.219 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 10 monclient: _renew_subs 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.200+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac011e910 con 0x7f8aac005330 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.201+0000 7f8ac4f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.201+0000 7f8ac4f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.201+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8ab8003040 con 0x7f8aac005330 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.201+0000 7f8ac4f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.201+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac011d9a0 con 0x7f8aac005330 2024-09-15T01:54:14.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.202+0000 7f8ab67fc640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 0x7f8aa003b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.202+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f8ab804bef0 con 0x7f8aac005330 2024-09-15T01:54:14.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.202+0000 7f8abe59c640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 0x7f8aa003b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.202+0000 7f8abe59c640 10 monclient: get_auth_request con 0x7f8aa0039140 auth_method 0 2024-09-15T01:54:14.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.205+0000 7f8abe59c640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 0x7f8aa003b5f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0033000 tx=0x7f8ab0039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.207+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8ac011d9a0 con 0x7f8aac005330 2024-09-15T01:54:14.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.207+0000 7f8ab67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.207+0000 7f8ab67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:14.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.439+0000 7f8ac4f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:14.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.439+0000 7f8ac4f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:14.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.439+0000 7f8ac4f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:14.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.439+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ac011d9a0 con 0x7f8aac005330 2024-09-15T01:54:14.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.440+0000 7f8ab67fc640 1 -- 172.21.15.17:0/3906702045 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f8ac011d9a0 con 0x7f8aac005330 2024-09-15T01:54:14.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.440+0000 7f8ab67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:14.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.440+0000 7f8ab67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:14.443 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 msgr2=0x7f8aa003b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 0x7f8aa003b5f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0033000 tx=0x7f8ab0039000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 10 monclient: shutdown 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 msgr2=0x7f8ac009a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.441+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 0x7f8ac009a930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8ac009c290 tx=0x7f8ab800b040 comp rx=0 tx=0).stop 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 shutdown_connections 2024-09-15T01:54:14.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8aa0039140 0x7f8aa003b5f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac004780 0x7f8ac0141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac005330 0x7f8ac009a930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 --2- 172.21.15.17:0/3906702045 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8aac003a00 0x7f8ac0140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 >> 172.21.15.17:0/3906702045 conn(0x7f8ac0070fb0 msgr2=0x7f8ac012fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 shutdown_connections 2024-09-15T01:54:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.442+0000 7f8ac4f9d640 1 -- 172.21.15.17:0/3906702045 wait complete. 2024-09-15T01:54:14.455 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:54:14.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:54:14.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:54:14.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.549+0000 7f587ecaf640 10 monclient: get_monmap_and_config 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.549+0000 7f587ecaf640 10 monclient: build_initial_monmap 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.549+0000 7f587ecaf640 10 monclient: monmap: 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:14.551618+0000 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:14.551618+0000 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:14.553 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:14.554 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:14.554 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.550+0000 7f587ecaf640 1 Processor -- start 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.550+0000 7f587ecaf640 1 -- start start 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.550+0000 7f587ecaf640 10 monclient: init 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.550+0000 7f587ecaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.550+0000 7f587ecaf640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f587809c2c0 0x7f587809a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-a con 0x7f587809c2c0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:14.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f587809c690 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 0x7f587806f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-c con 0x7f587809e970 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587806a5c0 con 0x7f587809c690 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587809d190 con 0x7f587809c2c0 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5878099f50 con 0x7f587809e970 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ecaf640 10 monclient(hunting): authenticate will time out at 914.548065s 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f587809c2c0 0x7f587809a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f587809c2c0 0x7f587809a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51800/0 (socket says 172.21.15.17:51800) 2024-09-15T01:54:14.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 -- 172.21.15.17:0/87050478 learned_addr learned my addr 172.21.15.17:0/87050478 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): get_auth_request con 0x7f587809c2c0 auth_method 0 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient(hunting): handle_auth_done global_id 44475 payload 274 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 -- 172.21.15.17:0/87050478 >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 msgr2=0x7f587806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587d24c640 1 --2- 172.21.15.17:0/87050478 >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 0x7f587806f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 --2- 172.21.15.17:0/87050478 >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 0x7f587806f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 -- 172.21.15.17:0/87050478 >> v1:172.21.15.67:6789/0 conn(0x7f587809c690 legacy=0x7f587806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 monclient: found mon.noname-a 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587ca4b640 1 -- 172.21.15.17:0/87050478 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878068ac0 con 0x7f587809c2c0 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587d24c640 10 monclient: get_auth_request con 0x7f587809e970 auth_method 0 2024-09-15T01:54:14.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.551+0000 7f587d24c640 1 --2- 172.21.15.17:0/87050478 >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 0x7f587806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ca4b640 1 -- 172.21.15.17:0/87050478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809c2c0 msgr2=0x7f587809a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ca4b640 1 --2- 172.21.15.17:0/87050478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809c2c0 0x7f587809a920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f58780a0db0 tx=0x7f58740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaa576367f78b25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 1 -- 172.21.15.17:0/87050478 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874032440 con 0x7f587809c2c0 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: dump: 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.567 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 1 -- 172.21.15.17:0/87050478 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58740325a0 con 0x7f587809c2c0 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ecaf640 5 monclient: authenticate success, global_id 44475 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ecaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ecaf640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 1 -- 172.21.15.17:0/87050478 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874032940 con 0x7f587809c2c0 2024-09-15T01:54:14.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f587ecaf640 4 set_mon_vals no callback set 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.554+0000 7f586f7fe640 10 monclient: dump: 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.569 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 10 monclient: shutdown 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 -- 172.21.15.17:0/87050478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809c2c0 msgr2=0x7f587809a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 --2- 172.21.15.17:0/87050478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809c2c0 0x7f587809a920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f58780a0db0 tx=0x7f58740091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 -- 172.21.15.17:0/87050478 shutdown_connections 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 --2- 172.21.15.17:0/87050478 >> v2:172.21.15.67:3300/0 conn(0x7f587809e970 0x7f587806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 --2- 172.21.15.17:0/87050478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809c2c0 0x7f587809a920 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 -- 172.21.15.17:0/87050478 >> 172.21.15.17:0/87050478 conn(0x7f58780a2f70 msgr2=0x7f58780a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 -- 172.21.15.17:0/87050478 shutdown_connections 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 1 -- 172.21.15.17:0/87050478 wait complete. 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 10 monclient: build_initial_monmap 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.555+0000 7f587ecaf640 10 monclient: monmap: 2024-09-15T01:54:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 Processor -- start 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 -- start start 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient: init 2024-09-15T01:54:14.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f587809c2c0 0x7f587811fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-b con 0x7f587809c2c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f587809c690 0x7f587813d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-c con 0x7f587809c690 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 0x7f5878140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): picked mon.noname-a con 0x7f587809e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587809ce60 con 0x7f587809e970 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5878122440 con 0x7f587809c2c0 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5878122160 con 0x7f587809c690 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587ecaf640 10 monclient(hunting): authenticate will time out at 914.553826s 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f586ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f587809c690 msgr2=0x7f587813d0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f586ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f587809c690 0x7f587813d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 0x7f5878140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 0x7f5878140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51816/0 (socket says 172.21.15.17:51816) 2024-09-15T01:54:14.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 learned_addr learned my addr 172.21.15.17:0/3599126228 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 10 monclient(hunting): get_auth_request con 0x7f587809e970 auth_method 0 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.556+0000 7f587d24c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f587809c2c0 0x7f587811fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): get_auth_request con 0x7f587809c2c0 auth_method 0 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient(hunting): handle_auth_done global_id 44478 payload 824 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f587809c690 msgr2=0x7f587813d0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f587809c690 0x7f587813d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f587809c2c0 msgr2=0x7f587811fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f587809c2c0 0x7f587811fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 monclient: found mon.noname-a 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.557+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874009450 con 0x7f587809e970 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 msgr2=0x7f5878140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587d24c640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 0x7f5878140f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f587000a380 tx=0x7f5870007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587000d860 con 0x7f587809e970 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: dump: 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.576 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587000d9c0 con 0x7f587809e970 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 5 monclient: authenticate success, global_id 44478 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 10 monclient: _renew_subs 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5878122720 con 0x7f587809e970 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587000dd80 con 0x7f587809e970 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586cff9640 10 monclient: dump: 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 10 monclient: _renew_subs 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587813dc40 con 0x7f587809e970 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.559+0000 7f586e7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ecaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ecaf640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 0x7f58780298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f587001e3b0 con 0x7f587809e970 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5878063b90 con 0x7f5878121240 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 0x7f58780298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient: get_auth_request con 0x7f5878121240 auth_method 0 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.560+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f586cff9640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58640386f0 0x7f586403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f587004c600 con 0x7f587809e970 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f587ca4b640 10 monclient(hunting): handle_auth_done global_id 44481 payload 824 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f587ca4b640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58780724a0 con 0x7f5878121240 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f586ffff640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58640386f0 0x7f586403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.561+0000 7f586ffff640 10 monclient: get_auth_request con 0x7f58640386f0 auth_method 0 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f587ca4b640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 msgr2=0x7f58780298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f587ca4b640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 0x7f58780298c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5874034000 tx=0x7f5874032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587403baa0 con 0x7f5878121240 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 10 monclient: dump: 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.581 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5874040070 con 0x7f5878121240 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586ffff640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58640386f0 0x7f586403aba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5868007fd0 tx=0x7f5868007a70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 msgr2=0x7f58780298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 0x7f58780298c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5874034000 tx=0x7f5874032000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 msgr2=0x7f58780298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.563+0000 7f586cff9640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878121240 0x7f58780298c0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5874034000 tx=0x7f5874032000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ecaf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ecaf640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 0x7f587802a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ecaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587806aa70 con 0x7f5878029e00 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 0x7f587802a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient: get_auth_request con 0x7f5878029e00 auth_method 0 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.569+0000 7f587ca4b640 10 monclient(hunting): handle_auth_done global_id 44484 payload 824 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.570+0000 7f587ca4b640 1 -- 172.21.15.17:0/3599126228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f587806d1b0 con 0x7f5878029e00 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ca4b640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 msgr2=0x7f587802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ca4b640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 0x7f587802a200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5874003080 tx=0x7f587403e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874046c30 con 0x7f5878029e00 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 10 monclient: dump: 2024-09-15T01:54:14.585 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4556 (secure 0 0 0) 0x7f587403b890 con 0x7f5878029e00 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 msgr2=0x7f587802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 0x7f587802a200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5874003080 tx=0x7f587403e070 comp rx=0 tx=0).stop 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 msgr2=0x7f587802a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f586cff9640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5878029e00 0x7f587802a200 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5874003080 tx=0x7f587403e070 comp rx=0 tx=0).stop 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 36, 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:14.587 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.588 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.589 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:14.590 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.591 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.592 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.593 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.594 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:14.596 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.598 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:14.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:14.600 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:14.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58640386f0 msgr2=0x7f586403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58640386f0 0x7f586403aba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5868007fd0 tx=0x7f5868007a70 comp rx=0 tx=0).stop 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 10 monclient: shutdown 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 msgr2=0x7f5878140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587ecaf640 1 --2- 172.21.15.17:0/3599126228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f587809e970 0x7f5878140f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f587000a380 tx=0x7f5870007900 comp rx=0 tx=0).stop 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 reap_dead start 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.572+0000 7f587d24c640 1 -- 172.21.15.17:0/3599126228 reap_dead start 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.573+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 shutdown_connections 2024-09-15T01:54:14.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.573+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 >> 172.21.15.17:0/3599126228 conn(0x7f58780a2f70 msgr2=0x7f5878098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.573+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 shutdown_connections 2024-09-15T01:54:14.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.573+0000 7f587ecaf640 1 -- 172.21.15.17:0/3599126228 wait complete. 2024-09-15T01:54:14.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:54:14.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:54:14.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:54:14.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.697+0000 7fba44b5d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:14.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.697+0000 7fba44b5d640 10 monclient: build_initial_monmap 2024-09-15T01:54:14.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.697+0000 7fba44b5d640 10 monclient: monmap: 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:14.699320+0000 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:14.699320+0000 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:14.701 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 Processor -- start 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 -- start start 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient: init 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba400a5780 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 0x7fba4009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fba4009ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba40069c90 0x7fba40139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fba40069c90 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:14.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400a5b50 con 0x7fba400a5780 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400a18b0 con 0x7fba40069c90 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4009c5f0 con 0x7fba4009ccf0 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba44b5d640 10 monclient(hunting): authenticate will time out at 914.695789s 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3ed9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba40069c90 0x7fba40139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3e59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fba400a5780 0x7fba40069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:53700/0 (socket says 172.21.15.17:53700) 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3e59c640 1 -- 172.21.15.17:0/2837605114 learned_addr learned my addr 172.21.15.17:0/2837605114 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3ed9d640 10 monclient(hunting): get_auth_request con 0x7fba40069c90 auth_method 0 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.698+0000 7fba3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba37fff640 1 --2- 172.21.15.17:0/2837605114 >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 0x7fba4009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient(hunting): handle_auth_done global_id 44487 payload 274 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2837605114 >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 msgr2=0x7fba4009bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 1 --2- 172.21.15.17:0/2837605114 >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 0x7fba4009bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2837605114 >> v1:172.21.15.17:6789/0 conn(0x7fba400a5780 legacy=0x7fba40069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 1 monclient: found mon.noname-a 2024-09-15T01:54:14.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2837605114 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4009d210 con 0x7fba40069c90 2024-09-15T01:54:14.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba37fff640 10 monclient: get_auth_request con 0x7fba4009ccf0 auth_method 0 2024-09-15T01:54:14.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.699+0000 7fba37fff640 1 --2- 172.21.15.17:0/2837605114 >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 0x7fba4009bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:14.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2837605114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 msgr2=0x7fba40139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3ed9d640 1 --2- 172.21.15.17:0/2837605114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40139880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fba40068e70 tx=0x7fba38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10007a255ea4d168 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 1 -- 172.21.15.17:0/2837605114 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38007840 con 0x7fba40069c90 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: dump: 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 1 -- 172.21.15.17:0/2837605114 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3802d050 con 0x7fba40069c90 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 5 monclient: authenticate success, global_id 44487 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:14.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 4 set_mon_vals no callback set 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 1 -- 172.21.15.17:0/2837605114 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38007c10 con 0x7fba40069c90 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba3dd9b640 10 monclient: dump: 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 10 monclient: shutdown 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 -- 172.21.15.17:0/2837605114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 msgr2=0x7fba40139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2837605114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40139880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fba40068e70 tx=0x7fba38009780 comp rx=0 tx=0).stop 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 -- 172.21.15.17:0/2837605114 shutdown_connections 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2837605114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40139880 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2837605114 >> v2:172.21.15.67:3300/0 conn(0x7fba4009ccf0 0x7fba4009bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 -- 172.21.15.17:0/2837605114 >> 172.21.15.17:0/2837605114 conn(0x7fba40070e10 msgr2=0x7fba400a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 -- 172.21.15.17:0/2837605114 shutdown_connections 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.702+0000 7fba44b5d640 1 -- 172.21.15.17:0/2837605114 wait complete. 2024-09-15T01:54:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 10 monclient: build_initial_monmap 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 10 monclient: monmap: 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 1 Processor -- start 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 1 -- start start 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 10 monclient: init 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.703+0000 7fba44b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fba40069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4009ccf0 0x7fba40131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fba4009ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba400a5780 0x7fba4011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fba400a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4011d740 con 0x7fba40069c90 2024-09-15T01:54:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4009b4c0 con 0x7fba4009ccf0 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4011d1c0 con 0x7fba400a5780 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba44b5d640 10 monclient(hunting): authenticate will time out at 914.701070s 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3ed9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba400a5780 msgr2=0x7fba4011b6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba400a5780 0x7fba4011b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51862/0 (socket says 172.21.15.17:51862) 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 -- 172.21.15.17:0/2393554026 learned_addr learned my addr 172.21.15.17:0/2393554026 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): get_auth_request con 0x7fba40069c90 auth_method 0 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba37fff640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4009ccf0 0x7fba40131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba37fff640 10 monclient(hunting): get_auth_request con 0x7fba4009ccf0 auth_method 0 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient(hunting): handle_auth_done global_id 44490 payload 824 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba400a5780 msgr2=0x7fba4011b6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba400a5780 0x7fba4011b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4009ccf0 msgr2=0x7fba40131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4009ccf0 0x7fba40131b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:14.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 monclient: found mon.noname-a 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.704+0000 7fba3e59c640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38009a20 con 0x7fba40069c90 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba3e59c640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 msgr2=0x7fba40141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba3e59c640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40141320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba3000c5e0 tx=0x7fba30004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30005cb0 con 0x7fba40069c90 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: dump: 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.721 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba30005e30 con 0x7fba40069c90 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 5 monclient: authenticate success, global_id 44490 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 10 monclient: _renew_subs 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba400a0920 con 0x7fba40069c90 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30007340 con 0x7fba40069c90 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba367fc640 10 monclient: dump: 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.723 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba3d59a640 4 set_mon_vals no callback set 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 10 monclient: _renew_subs 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.706+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba40141860 con 0x7fba40069c90 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba44b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 0x7fba4011bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba40063b20 con 0x7fba401320a0 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fba3001b400 con 0x7fba40069c90 2024-09-15T01:54:14.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 0x7fba4011bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient: get_auth_request con 0x7fba401320a0 auth_method 0 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba367fc640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba20038740 0x7fba2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.707+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fba3004b7c0 con 0x7fba40069c90 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.708+0000 7fba3ed9d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba20038740 0x7fba2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.708+0000 7fba37fff640 10 monclient(hunting): handle_auth_done global_id 5044 payload 824 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.708+0000 7fba37fff640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba4011f700 con 0x7fba401320a0 2024-09-15T01:54:14.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.708+0000 7fba3ed9d640 10 monclient: get_auth_request con 0x7fba20038740 auth_method 0 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba37fff640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 msgr2=0x7fba4011bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba37fff640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 0x7fba4011bc10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fba4011cdc0 tx=0x7fba28008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba3ed9d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba20038740 0x7fba2003abf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba38034000 tx=0x7fba38037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28014980 con 0x7fba401320a0 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 10 monclient: dump: 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.728 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fba2800c030 con 0x7fba401320a0 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 msgr2=0x7fba4011bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 0x7fba4011bc10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fba4011cdc0 tx=0x7fba28008040 comp rx=0 tx=0).stop 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 msgr2=0x7fba4011bc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.710+0000 7fba367fc640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba401320a0 0x7fba4011bc10 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fba4011cdc0 tx=0x7fba28008040 comp rx=0 tx=0).stop 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.716+0000 7fba44b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.716+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 0x7fba40096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.716+0000 7fba44b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.716+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4011f010 con 0x7fba4011c150 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 0x7fba40096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient: get_auth_request con 0x7fba4011c150 auth_method 0 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:14.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 10 monclient(hunting): handle_auth_done global_id 5047 payload 824 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.717+0000 7fba37fff640 1 -- 172.21.15.17:0/2393554026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fba4011c550 con 0x7fba4011c150 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.719+0000 7fba37fff640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 msgr2=0x7fba40096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.719+0000 7fba37fff640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 0x7fba40096a10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba2801a000 tx=0x7fba2801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28022070 con 0x7fba4011c150 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 10 monclient: dump: 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:14.731 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4724 (secure 0 0 0) 0x7fba28013800 con 0x7fba4011c150 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 msgr2=0x7fba40096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 0x7fba40096a10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba2801a000 tx=0x7fba2801d000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 msgr2=0x7fba40096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:14.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.720+0000 7fba367fc640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4011c150 0x7fba40096a10 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba2801a000 tx=0x7fba2801d000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 36, 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:14.733 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.734 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:14.736 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:14.737 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.740 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:14.741 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:14.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.744 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:14.745 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:14.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba20038740 msgr2=0x7fba2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba20038740 0x7fba2003abf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba38034000 tx=0x7fba38037000 comp rx=0 tx=0).stop 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 10 monclient: shutdown 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 msgr2=0x7fba40141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 --2- 172.21.15.17:0/2393554026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba40069c90 0x7fba40141320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba3000c5e0 tx=0x7fba30004960 comp rx=0 tx=0).stop 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2393554026 reap_dead start 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba3ed9d640 1 -- 172.21.15.17:0/2393554026 reap_dead start 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 shutdown_connections 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 >> 172.21.15.17:0/2393554026 conn(0x7fba40070e10 msgr2=0x7fba4009d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 shutdown_connections 2024-09-15T01:54:14.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:14.721+0000 7fba44b5d640 1 -- 172.21.15.17:0/2393554026 wait complete. 2024-09-15T01:54:14.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:54:14.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:54:15.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.113+0000 7fdde97fa640 10 monclient: tick 2024-09-15T01:54:15.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.113+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:15.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.194+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7fddf40148f0 con 0x7fddf8098140 2024-09-15T01:54:15.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.194+0000 7fddea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.194+0000 7fddea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.298+0000 7fde0051d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.298+0000 7fde0051d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.298+0000 7fde0051d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.298+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fddf8132720 con 0x7fddf8098140 2024-09-15T01:54:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.300+0000 7fddea7fc640 1 -- 172.21.15.67:0/1449591926 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v59) v1 ==== 103+0+0 (secure 0 0 0) 0x7fddf4000ed0 con 0x7fddf8098140 2024-09-15T01:54:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.300+0000 7fddea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.300+0000 7fddea7fc640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T01:54:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo does not exist 2024-09-15T01:54:15.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 msgr2=0x7fdddc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 0x7fdddc03abf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fddec007fd0 tx=0x7fddec007f00 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 10 monclient: shutdown 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 msgr2=0x7fddf8138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fddf400d840 tx=0x7fddf4008040 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 shutdown_connections 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fddf8098c20 0x7fddf8130c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fddf80a0370 0x7fddf81311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 --2- 172.21.15.67:0/1449591926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8098140 0x7fddf8138810 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 >> 172.21.15.67:0/1449591926 conn(0x7fddf80a5780 msgr2=0x7fddf809ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 shutdown_connections 2024-09-15T01:54:15.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.302+0000 7fde0051d640 1 -- 172.21.15.67:0/1449591926 wait complete. 2024-09-15T01:54:15.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm foo 2024-09-15T01:54:15.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.390+0000 7f1bbd71d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.390+0000 7f1bbd71d640 10 monclient: build_initial_monmap 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.390+0000 7f1bbd71d640 10 monclient: monmap: 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:15.391548+0000 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:15.391548+0000 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:15.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:15.394 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:15.394 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:15.394 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:15.394 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.390+0000 7f1bbd71d640 1 Processor -- start 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 -- start start 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient: init 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bb8131d80 0x7f1bb8132150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb8131d80 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb8098400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb81329f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bb8098940 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb80a0440 con 0x7f1bb8098940 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb80a01e0 con 0x7f1bb8131d80 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80a19b0 con 0x7f1bb81329f0 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bbd71d640 10 monclient(hunting): authenticate will time out at 887.860515s 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bb8131d80 0x7f1bb8132150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bb8098940 0x7f1bb8068e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43206/0 (socket says 172.21.15.67:43206) 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb67fc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 msgr2=0x7f1bb8098400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb77fe640 1 -- 172.21.15.67:0/3621944587 learned_addr learned my addr 172.21.15.67:0/3621944587 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:15.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb8131d80 auth_method 0 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb67fc640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb8098400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_done global_id 5050 payload 274 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 -- 172.21.15.67:0/3621944587 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 msgr2=0x7f1bb8098400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb8098400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 -- 172.21.15.67:0/3621944587 >> v1:172.21.15.67:6789/0 conn(0x7f1bb8098940 legacy=0x7f1bb8068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 monclient: found mon.noname-c 2024-09-15T01:54:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:15.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.391+0000 7f1bb6ffd640 1 -- 172.21.15.67:0/3621944587 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8096d30 con 0x7f1bb8131d80 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb6ffd640 1 -- 172.21.15.67:0/3621944587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 msgr2=0x7f1bb8132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb6ffd640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb8132150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009850 tx=0x7f1ba80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fdd869791eb1f77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 1 -- 172.21.15.67:0/3621944587 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8032440 con 0x7f1bb8131d80 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: dump: 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 1 -- 172.21.15.67:0/3621944587 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba80325a0 con 0x7f1bb8131d80 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bbd71d640 5 monclient: authenticate success, global_id 5050 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bbd71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bbd71d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bbd71d640 4 set_mon_vals no callback set 2024-09-15T01:54:15.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 1 -- 172.21.15.67:0/3621944587 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8032940 con 0x7f1bb8131d80 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.394+0000 7f1bb5ffb640 10 monclient: dump: 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 10 monclient: shutdown 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 -- 172.21.15.67:0/3621944587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 msgr2=0x7f1bb8132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb8132150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009850 tx=0x7f1ba80090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 -- 172.21.15.67:0/3621944587 shutdown_connections 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb8098400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/3621944587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb8132150 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 -- 172.21.15.67:0/3621944587 >> 172.21.15.67:0/3621944587 conn(0x7f1bb80712f0 msgr2=0x7f1bb80a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 -- 172.21.15.67:0/3621944587 shutdown_connections 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.395+0000 7f1bbd71d640 1 -- 172.21.15.67:0/3621944587 wait complete. 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 10 monclient: build_initial_monmap 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 10 monclient: monmap: 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:15.408 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 1 Processor -- start 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 1 -- start start 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 10 monclient: init 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.396+0000 7f1bbd71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bb8098940 0x7f1bb8141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb8098940 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb8131d80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb80994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb81329f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80a0d80 con 0x7f1bb8098940 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80a4a30 con 0x7f1bb8131d80 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb811c400 con 0x7f1bb81329f0 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bbd71d640 10 monclient(hunting): authenticate will time out at 887.866118s 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb77fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 msgr2=0x7f1bb80994f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60584/0 (socket says 172.21.15.67:60584) 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 -- 172.21.15.67:0/1310545420 learned_addr learned my addr 172.21.15.67:0/1310545420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): get_auth_request con 0x7f1bb8131d80 auth_method 0 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb77fe640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb80994f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bb8098940 0x7f1bb8141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb8098940 auth_method 0 2024-09-15T01:54:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient(hunting): handle_auth_done global_id 5053 payload 824 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 -- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 msgr2=0x7f1bb80994f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb80994f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 -- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bb8098940 msgr2=0x7f1bb8141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bb8098940 0x7f1bb8141500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 monclient: found mon.noname-b 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.397+0000 7f1bb67fc640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8009390 con 0x7f1bb8131d80 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bb67fc640 1 -- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 msgr2=0x7f1bb812f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bb67fc640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00b600 tx=0x7f1bac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac015070 con 0x7f1bb8131d80 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 10 monclient: dump: 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 10 monclient: _finish_auth 0 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bbd71d640 5 monclient: authenticate success, global_id 5053 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac003d90 con 0x7f1bb8131d80 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bbd71d640 10 monclient: _renew_subs 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bbd71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb811c120 con 0x7f1bb8131d80 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac006380 con 0x7f1bb8131d80 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1b9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1b9effd640 10 monclient: dump: 2024-09-15T01:54:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 10 monclient: _renew_subs 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.399+0000 7f1bb57fa640 4 set_mon_vals no callback set 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb8141a40 con 0x7f1bb8131d80 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1bac010440 con 0x7f1bb8131d80 2024-09-15T01:54:15.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.400+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb811c5c0 con 0x7f1bb8131d80 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.401+0000 7f1b9effd640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.401+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5086+0+0 (secure 0 0 0) 0x7f1bac04b650 con 0x7f1bb8131d80 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.401+0000 7f1bb6ffd640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.401+0000 7f1bb6ffd640 10 monclient: get_auth_request con 0x7f1ba40386f0 auth_method 0 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.403+0000 7f1bb6ffd640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80a30f0 tx=0x7f1ba8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.405+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1bac018910 con 0x7f1bb8131d80 2024-09-15T01:54:15.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.405+0000 7f1b9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.405+0000 7f1b9effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.573+0000 7f1bbd71d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.573+0000 7f1bbd71d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.573+0000 7f1bbd71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.573+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f1bb804bf60 con 0x7f1bb8131d80 2024-09-15T01:54:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.575+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v59) v1 ==== 103+0+0 (secure 0 0 0) 0x7f1bac018310 con 0x7f1bb8131d80 2024-09-15T01:54:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.575+0000 7f1b9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.575+0000 7f1b9effd640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-15T01:54:15.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.674+0000 7f1bbd71d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.674+0000 7f1bbd71d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.674+0000 7f1bbd71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.674+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f1bb8067a20 con 0x7f1bb8131d80 2024-09-15T01:54:15.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.676+0000 7f1b9effd640 1 -- 172.21.15.67:0/1310545420 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v59) v1 ==== 103+0+0 (secure 0 0 0) 0x7f1bac0184f0 con 0x7f1bb8131d80 2024-09-15T01:54:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.676+0000 7f1b9effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T01:54:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.676+0000 7f1b9effd640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T01:54:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo does not exist 2024-09-15T01:54:15.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 msgr2=0x7f1ba403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80a30f0 tx=0x7f1ba8032000 comp rx=0 tx=0).stop 2024-09-15T01:54:15.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 10 monclient: shutdown 2024-09-15T01:54:15.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:15.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 msgr2=0x7f1bb812f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00b600 tx=0x7f1bac0091a0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 shutdown_connections 2024-09-15T01:54:15.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bb81329f0 0x7f1bb80994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bb8131d80 0x7f1bb812f000 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 --2- 172.21.15.67:0/1310545420 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bb8098940 0x7f1bb8141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 >> 172.21.15.67:0/1310545420 conn(0x7f1bb80712f0 msgr2=0x7f1bb80971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:15.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 shutdown_connections 2024-09-15T01:54:15.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.678+0000 7f1bbd71d640 1 -- 172.21.15.67:0/1310545420 wait complete. 2024-09-15T01:54:15.689 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm bar 2024-09-15T01:54:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.763+0000 7f2b1b86f640 10 monclient: get_monmap_and_config 2024-09-15T01:54:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.763+0000 7f2b1b86f640 10 monclient: build_initial_monmap 2024-09-15T01:54:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.763+0000 7f2b1b86f640 10 monclient: monmap: 2024-09-15T01:54:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:15.764851+0000 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:15.764851+0000 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 Processor -- start 2024-09-15T01:54:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 -- start start 2024-09-15T01:54:15.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient: init 2024-09-15T01:54:15.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:15.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T01:54:15.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b140a22f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:15.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2b1406e160 0x7f2b14136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b1406e160 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b1406e600 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140710b0 con 0x7f2b140a22f0 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1412f550 con 0x7f2b1406e600 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1412ee80 con 0x7f2b1406e160 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.764+0000 7f2b1b86f640 10 monclient(hunting): authenticate will time out at 888.233966s 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b19e0c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b1406e600 0x7f2b14139480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43222/0 (socket says 172.21.15.67:43222) 2024-09-15T01:54:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b19e0c640 1 -- 172.21.15.67:0/4092085125 learned_addr learned my addr 172.21.15.67:0/4092085125 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 1 --2- 172.21.15.67:0/4092085125 >> v2:172.21.15.17:3300/0 conn(0x7f2b1406e160 0x7f2b14136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): get_auth_request con 0x7f2b1406e160 auth_method 0 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient(hunting): handle_auth_done global_id 44496 payload 274 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 1 -- 172.21.15.67:0/4092085125 >> v1:172.21.15.67:6789/0 conn(0x7f2b1406e600 legacy=0x7f2b14139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 1 -- 172.21.15.67:0/4092085125 >> v1:172.21.15.17:6789/0 conn(0x7f2b140a22f0 legacy=0x7f2b1412e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient: _finish_hunting 0 2024-09-15T01:54:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 1 monclient: found mon.noname-a 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.765+0000 7f2b18e0a640 1 -- 172.21.15.67:0/4092085125 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140a26c0 con 0x7f2b1406e160 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.768+0000 7f2b18e0a640 1 -- 172.21.15.67:0/4092085125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b1406e160 msgr2=0x7f2b14136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.768+0000 7f2b18e0a640 1 --2- 172.21.15.67:0/4092085125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b1406e160 0x7f2b14136f30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009850 tx=0x7f2b040090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a230b077cc87051 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 1 -- 172.21.15.67:0/4092085125 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04032440 con 0x7f2b1406e160 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: dump: 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.789 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: _finish_auth 0 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 5 monclient: authenticate success, global_id 44496 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 1 -- 172.21.15.67:0/4092085125 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b040325a0 con 0x7f2b1406e160 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:15.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 4 set_mon_vals no callback set 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 1 -- 172.21.15.67:0/4092085125 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04032940 con 0x7f2b1406e160 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b0bfff640 10 monclient: dump: 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 10 monclient: shutdown 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 -- 172.21.15.67:0/4092085125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b1406e160 msgr2=0x7f2b14136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/4092085125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b1406e160 0x7f2b14136f30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009850 tx=0x7f2b040090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 -- 172.21.15.67:0/4092085125 shutdown_connections 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/4092085125 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b1406e160 0x7f2b14136f30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 -- 172.21.15.67:0/4092085125 >> 172.21.15.67:0/4092085125 conn(0x7f2b14067ae0 msgr2=0x7f2b140a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 -- 172.21.15.67:0/4092085125 shutdown_connections 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.769+0000 7f2b1b86f640 1 -- 172.21.15.67:0/4092085125 wait complete. 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.770+0000 7f2b1b86f640 10 monclient: build_initial_monmap 2024-09-15T01:54:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.770+0000 7f2b1b86f640 10 monclient: monmap: 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.770+0000 7f2b1b86f640 1 Processor -- start 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.770+0000 7f2b1b86f640 1 -- start start 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.770+0000 7f2b1b86f640 10 monclient: init 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1406e160 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 0x7f2b14141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b1406e600 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 0x7f2b14141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b140a22f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1409cf20 con 0x7f2b140a22f0 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14134d80 con 0x7f2b1406e160 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1406ca20 con 0x7f2b1406e600 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1b86f640 10 monclient(hunting): authenticate will time out at 888.240203s 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b18e0a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 msgr2=0x7f2b14141500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60588/0 (socket says 172.21.15.67:60588) 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 -- 172.21.15.67:0/2749391632 learned_addr learned my addr 172.21.15.67:0/2749391632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): get_auth_request con 0x7f2b1406e160 auth_method 0 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b18e0a640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 0x7f2b14141500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b19e0c640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 0x7f2b14141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient(hunting): handle_auth_done global_id 5056 payload 824 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 -- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 msgr2=0x7f2b14141500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 0x7f2b14141500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 -- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 msgr2=0x7f2b14141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 0x7f2b14141a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient: _finish_hunting 0 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 monclient: found mon.noname-b 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b1960b640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04009390 con 0x7f2b1406e160 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b19e0c640 10 monclient: get_auth_request con 0x7f2b140a22f0 auth_method 0 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.771+0000 7f2b19e0c640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 0x7f2b14141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.773+0000 7f2b1960b640 1 -- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 msgr2=0x7f2b1412b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.773+0000 7f2b1960b640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c5e0 tx=0x7f2b10007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10012920 con 0x7f2b1406e160 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: dump: 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.798 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10012a80 con 0x7f2b1406e160 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 5 monclient: authenticate success, global_id 5056 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 10 monclient: _renew_subs 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10012e80 con 0x7f2b1406e160 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b141183e0 con 0x7f2b1406e160 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:15.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b0b7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 10 monclient: dump: 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 10 monclient: _renew_subs 2024-09-15T01:54:15.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14117fc0 con 0x7f2b1406e160 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.774+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2b10011590 con 0x7f2b1406e160 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b097fa640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 0x7f2af803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b1b86f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b1b86f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b1b86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5086+0+0 (secure 0 0 0) 0x7f2b1004be50 con 0x7f2b1406e160 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1404bf60 con 0x7f2b1406e160 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b18e0a640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 0x7f2af803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.775+0000 7f2b18e0a640 10 monclient: get_auth_request con 0x7f2af8038720 auth_method 0 2024-09-15T01:54:15.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.777+0000 7f2b18e0a640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 0x7f2af803abd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b04034000 tx=0x7f2b04032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:15.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.779+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2b10016030 con 0x7f2b1406e160 2024-09-15T01:54:15.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.779+0000 7f2b097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:15.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.779+0000 7f2b097fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:15.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.945+0000 7f2b1b86f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:15.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.945+0000 7f2b1b86f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:15.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.945+0000 7f2b1b86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:15.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:15.945+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f2b14071330 con 0x7f2b1406e160 2024-09-15T01:54:16.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.202+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v60) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2b10029090 con 0x7f2b1406e160 2024-09-15T01:54:16.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.202+0000 7f2b097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:16.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.202+0000 7f2b097fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:16.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.302+0000 7f2b1b86f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.302+0000 7f2b1b86f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.302+0000 7f2b1b86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.302+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f2b141186c0 con 0x7f2b1406e160 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.303+0000 7f2b097fa640 1 -- 172.21.15.67:0/2749391632 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f2b10016030 con 0x7f2b1406e160 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.303+0000 7f2b097fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.303+0000 7f2b097fa640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-15T01:54:16.305 INFO:tasks.workunit.client.0.smithi067.stderr:rule bar does not exist 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 msgr2=0x7f2af803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 0x7f2af803abd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b04034000 tx=0x7f2b04032000 comp rx=0 tx=0).stop 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 10 monclient: shutdown 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 msgr2=0x7f2b1412b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c5e0 tx=0x7f2b10007ed0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 shutdown_connections 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2af8038720 0x7f2af803abd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b140a22f0 0x7f2b14141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b1406e600 0x7f2b14141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 --2- 172.21.15.67:0/2749391632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1406e160 0x7f2b1412b510 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 >> 172.21.15.67:0/2749391632 conn(0x7f2b14067ae0 msgr2=0x7f2b140a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:16.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 shutdown_connections 2024-09-15T01:54:16.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.305+0000 7f2b1b86f640 1 -- 172.21.15.67:0/2749391632 wait complete. 2024-09-15T01:54:16.316 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create pinning_pool 1 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.390+0000 7ff13e46f640 10 monclient: get_monmap_and_config 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.390+0000 7ff13e46f640 10 monclient: build_initial_monmap 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.390+0000 7ff13e46f640 10 monclient: monmap: 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:16.391972+0000 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:16.391972+0000 2024-09-15T01:54:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:16.394 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.391+0000 7ff13e46f640 1 Processor -- start 2024-09-15T01:54:16.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.391+0000 7ff13e46f640 1 -- start start 2024-09-15T01:54:16.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.391+0000 7ff13e46f640 10 monclient: init 2024-09-15T01:54:16.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1380a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff13809eea0 0x7ff13809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-a con 0x7ff13809eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 0x7ff138139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-b con 0x7ff138136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13806a270 con 0x7ff1380a0b50 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13806f290 con 0x7ff13809eea0 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13809f7e0 con 0x7ff138136e40 2024-09-15T01:54:16.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13e46f640 10 monclient(hunting): authenticate will time out at 888.861219s 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13ca0c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 msgr2=0x7ff138139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff13ca0c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 0x7ff138139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff13809eea0 0x7ff13809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff137fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff1380a0b50 0x7ff138069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:49330/0 (socket says 172.21.15.67:49330) 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff137fff640 1 -- 172.21.15.67:0/316024303 learned_addr learned my addr 172.21.15.67:0/316024303 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): get_auth_request con 0x7ff13809eea0 auth_method 0 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:16.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 10 monclient(hunting): handle_auth_done global_id 44499 payload 274 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.392+0000 7ff12ffff640 1 -- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 msgr2=0x7ff138139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 1 --2- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 0x7ff138139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 1 -- 172.21.15.67:0/316024303 >> v1:172.21.15.17:6789/0 conn(0x7ff1380a0b50 legacy=0x7ff138069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 1 monclient: found mon.noname-a 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:16.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.393+0000 7ff12ffff640 1 -- 172.21.15.67:0/316024303 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380a0f20 con 0x7ff13809eea0 2024-09-15T01:54:16.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff12ffff640 1 -- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff13809eea0 msgr2=0x7ff13809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:16.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff12ffff640 1 --2- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff13809eea0 0x7ff13809f2a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff120009880 tx=0x7ff1200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d18c68536c12998 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:16.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 1 -- 172.21.15.67:0/316024303 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120032440 con 0x7ff13809eea0 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: dump: 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:16.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 1 -- 172.21.15.67:0/316024303 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1200325a0 con 0x7ff13809eea0 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 1 -- 172.21.15.67:0/316024303 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120032940 con 0x7ff13809eea0 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff1377fe640 10 monclient: dump: 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 5 monclient: authenticate success, global_id 44499 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 4 set_mon_vals no callback set 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 10 monclient: shutdown 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 -- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff13809eea0 msgr2=0x7ff13809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 --2- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff13809eea0 0x7ff13809f2a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff120009880 tx=0x7ff1200090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 -- 172.21.15.67:0/316024303 shutdown_connections 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 --2- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff138136e40 0x7ff138139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 --2- 172.21.15.67:0/316024303 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff13809eea0 0x7ff13809f2a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 -- 172.21.15.67:0/316024303 >> 172.21.15.67:0/316024303 conn(0x7ff138071290 msgr2=0x7ff1380a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 -- 172.21.15.67:0/316024303 shutdown_connections 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.396+0000 7ff13e46f640 1 -- 172.21.15.67:0/316024303 wait complete. 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.397+0000 7ff13e46f640 10 monclient: build_initial_monmap 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.397+0000 7ff13e46f640 10 monclient: monmap: 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.397+0000 7ff13e46f640 1 Processor -- start 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.397+0000 7ff13e46f640 1 -- start start 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.397+0000 7ff13e46f640 10 monclient: init 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 0x7ff138134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-c con 0x7ff13809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-b con 0x7ff1380a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff138136e40 0x7ff13811dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): picked mon.noname-a con 0x7ff138136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138137720 con 0x7ff138136e40 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1380a12d0 con 0x7ff1380a0b50 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1380a3450 con 0x7ff13809eea0 2024-09-15T01:54:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13e46f640 10 monclient(hunting): authenticate will time out at 888.867226s 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff137fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 msgr2=0x7ff138134be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff137fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 0x7ff138134be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60598/0 (socket says 172.21.15.67:60598) 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 -- 172.21.15.67:0/3674991549 learned_addr learned my addr 172.21.15.67:0/3674991549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): get_auth_request con 0x7ff1380a0b50 auth_method 0 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13ca0c640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff138136e40 0x7ff13811dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13ca0c640 10 monclient(hunting): get_auth_request con 0x7ff138136e40 auth_method 0 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13ca0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13ca0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff13ca0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient(hunting): handle_auth_done global_id 5059 payload 824 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 -- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 msgr2=0x7ff138134be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 0x7ff138134be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 -- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff138136e40 msgr2=0x7ff13811dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff138136e40 0x7ff13811dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 monclient: found mon.noname-b 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:16.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.398+0000 7ff12ffff640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff120009390 con 0x7ff1380a0b50 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.400+0000 7ff12ffff640 1 -- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 msgr2=0x7ff13811d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.400+0000 7ff12ffff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff120009880 tx=0x7ff120032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff12ffff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff120009880 tx=0x7ff120032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7ff1200318c0 mon_map magic: 0 v1 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200318c0 con 0x7ff1380a0b50 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: dump: 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120031a20 con 0x7ff1380a0b50 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 5 monclient: authenticate success, global_id 5059 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 10 monclient: _renew_subs 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138120fd0 con 0x7ff1380a0b50 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 10 monclient: _renew_subs 2024-09-15T01:54:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff136ffd640 4 set_mon_vals no callback set 2024-09-15T01:54:16.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:16.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff138142070 con 0x7ff1380a0b50 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120031dc0 con 0x7ff1380a0b50 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.401+0000 7ff134ff9640 10 monclient: dump: 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:16.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff120043030 con 0x7ff1380a0b50 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff13e46f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff13e46f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff13e46f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13804bf60 con 0x7ff1380a0b50 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff134ff9640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 0x7ff11803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff137fff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 0x7ff11803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.402+0000 7ff137fff640 10 monclient: get_auth_request con 0x7ff118038740 auth_method 0 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.404+0000 7ff137fff640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 0x7ff11803abf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1280047d0 tx=0x7ff128007af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:16.406+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5031+0+0 (secure 0 0 0) 0x7ff120041070 con 0x7ff1380a0b50 2024-09-15T01:54:17.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.025+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff120030940 con 0x7ff1380a0b50 2024-09-15T01:54:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.025+0000 7ff134ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.025+0000 7ff134ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:17.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.192+0000 7ff13e46f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:17.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.192+0000 7ff13e46f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:17.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.192+0000 7ff13e46f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:17.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:17.192+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7ff13811f230 con 0x7ff1380a0b50 2024-09-15T01:54:18.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.200+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v62) v1 ==== 127+0+0 (secure 0 0 0) 0x7ff1200370a0 con 0x7ff1380a0b50 2024-09-15T01:54:18.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.200+0000 7ff134ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:18.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.200+0000 7ff134ff9640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-15T01:54:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.302+0000 7ff13e46f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.302+0000 7ff13e46f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.302+0000 7ff13e46f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.302+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7ff13811ff10 con 0x7ff1380a0b50 2024-09-15T01:54:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.303+0000 7ff134ff9640 1 -- 172.21.15.67:0/3674991549 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v62) v1 ==== 134+0+0 (secure 0 0 0) 0x7ff13811ff10 con 0x7ff1380a0b50 2024-09-15T01:54:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.303+0000 7ff134ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T01:54:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.303+0000 7ff134ff9640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-15T01:54:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'pinning_pool' already exists 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 msgr2=0x7ff11803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 0x7ff11803abf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1280047d0 tx=0x7ff128007af0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 10 monclient: shutdown 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 msgr2=0x7ff13811d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff120009880 tx=0x7ff120032000 comp rx=0 tx=0).stop 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 shutdown_connections 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff118038740 0x7ff11803abf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff138136e40 0x7ff13811dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1380a0b50 0x7ff13811d6a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 --2- 172.21.15.67:0/3674991549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff13809eea0 0x7ff138134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 >> 172.21.15.67:0/3674991549 conn(0x7ff138071290 msgr2=0x7ff138135e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:18.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 shutdown_connections 2024-09-15T01:54:18.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.305+0000 7ff13e46f640 1 -- 172.21.15.67:0/3674991549 wait complete. 2024-09-15T01:54:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-15T01:54:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.392+0000 7fdaae11d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.392+0000 7fdaae11d640 10 monclient: build_initial_monmap 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.392+0000 7fdaae11d640 10 monclient: monmap: 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:18.393804+0000 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:18.393804+0000 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.395 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:18.396 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:18.396 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:18.396 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:18.396 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:18.396 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 1 Processor -- start 2024-09-15T01:54:18.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 1 -- start start 2024-09-15T01:54:18.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient: init 2024-09-15T01:54:18.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:18.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdaa8097ba0 0x7fdaa8097f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa8097ba0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaa80672c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdaa8137000 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.393+0000 7fdaae11d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa809e430 con 0x7fdaa8137000 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaae11d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80977a0 con 0x7fdaa80672c0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaae11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaae11d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa809e150 con 0x7fdaa8097ba0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaae11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaae11d640 10 monclient(hunting): authenticate will time out at 890.863024s 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdaa8097ba0 0x7fdaa8097f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdaa8097ba0 0x7fdaa8097f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60610/0 (socket says 172.21.15.67:60610) 2024-09-15T01:54:18.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 -- 172.21.15.67:0/3371699723 learned_addr learned my addr 172.21.15.67:0/3371699723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request con 0x7fdaa8097ba0 auth_method 0 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa6ffd640 1 --1- 172.21.15.67:0/3371699723 >> v1:172.21.15.67:6789/0 conn(0x7fdaa80672c0 0x7fdaa8097090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43236/0 (socket says 172.21.15.67:43236) 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_done global_id 5062 payload 274 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 -- 172.21.15.67:0/3371699723 >> v1:172.21.15.67:6789/0 conn(0x7fdaa80672c0 legacy=0x7fdaa8097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 -- 172.21.15.67:0/3371699723 >> v1:172.21.15.17:6789/0 conn(0x7fdaa8137000 legacy=0x7fdaa81393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:18.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:18.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:18.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:18.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.394+0000 7fdaa77fe640 1 -- 172.21.15.67:0/3371699723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa809e710 con 0x7fdaa8097ba0 2024-09-15T01:54:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa77fe640 1 -- 172.21.15.67:0/3371699723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 msgr2=0x7fdaa8097f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa77fe640 1 --2- 172.21.15.67:0/3371699723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa8097f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda98009910 tx=0x7fda980091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98742acf037eb4c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 1 -- 172.21.15.67:0/3371699723 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98032440 con 0x7fdaa8097ba0 2024-09-15T01:54:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: dump: 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 1 -- 172.21.15.67:0/3371699723 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda980325a0 con 0x7fdaa8097ba0 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 5 monclient: authenticate success, global_id 5062 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 1 -- 172.21.15.67:0/3371699723 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98032940 con 0x7fdaa8097ba0 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 4 set_mon_vals no callback set 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaa67fc640 10 monclient: dump: 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 10 monclient: shutdown 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 -- 172.21.15.67:0/3371699723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 msgr2=0x7fdaa8097f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 --2- 172.21.15.67:0/3371699723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa8097f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda98009910 tx=0x7fda980091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 -- 172.21.15.67:0/3371699723 shutdown_connections 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 --2- 172.21.15.67:0/3371699723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa8097f70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 -- 172.21.15.67:0/3371699723 >> 172.21.15.67:0/3371699723 conn(0x7fdaa80a2110 msgr2=0x7fdaa80a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 -- 172.21.15.67:0/3371699723 shutdown_connections 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 -- 172.21.15.67:0/3371699723 wait complete. 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 10 monclient: build_initial_monmap 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 10 monclient: monmap: 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.398+0000 7fdaae11d640 1 Processor -- start 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 -- start start 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient: init 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdaa80672c0 0x7fdaa806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa80672c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa81355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa8097ba0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdaa8137000 0x7fdaa8135b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:18.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa8137000 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8136ac0 con 0x7fdaa8137000 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa806e440 con 0x7fdaa8097ba0 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa806b960 con 0x7fdaa80672c0 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaae11d640 10 monclient(hunting): authenticate will time out at 890.868471s 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa81355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa77fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdaa80672c0 msgr2=0x7fdaa806df00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa81355e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60626/0 (socket says 172.21.15.67:60626) 2024-09-15T01:54:18.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 -- 172.21.15.67:0/18655954 learned_addr learned my addr 172.21.15.67:0/18655954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request con 0x7fdaa8097ba0 auth_method 0 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa77fe640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdaa80672c0 0x7fdaa806df00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_done global_id 5065 payload 824 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 -- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdaa80672c0 msgr2=0x7fdaa806df00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdaa80672c0 0x7fdaa806df00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 -- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdaa8137000 msgr2=0x7fdaa8135b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa7fff640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdaa8137000 0x7fdaa8135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdaa8137000 0x7fdaa8135b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 monclient: found mon.noname-b 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.399+0000 7fdaa6ffd640 1 -- 172.21.15.67:0/18655954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98009450 con 0x7fdaa8097ba0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaa6ffd640 1 -- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 msgr2=0x7fdaa81355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaa6ffd640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa81355e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b510 tx=0x7fda9c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c011840 con 0x7fdaa8097ba0 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:18.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: dump: 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c0119a0 con 0x7fdaa8097ba0 2024-09-15T01:54:18.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c011d60 con 0x7fdaa8097ba0 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fda8f7fe640 10 monclient: dump: 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 5 monclient: authenticate success, global_id 5065 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 10 monclient: _renew_subs 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaa5ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 1 -- 172.21.15.67:0/18655954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8067ad0 con 0x7fdaa8097ba0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 10 monclient: _renew_subs 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.402+0000 7fdaae11d640 1 -- 172.21.15.67:0/18655954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa80a0c60 con 0x7fdaa8097ba0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fda9c010590 con 0x7fdaa8097ba0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fda8f7fe640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda940386f0 0x7fda9403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fda9c04be50 con 0x7fdaa8097ba0 2024-09-15T01:54:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaae11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaae11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaae11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaae11d640 1 -- 172.21.15.67:0/18655954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa804bf60 con 0x7fdaa8097ba0 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaa77fe640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda940386f0 0x7fda9403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.403+0000 7fdaa77fe640 10 monclient: get_auth_request con 0x7fda940386f0 auth_method 0 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.405+0000 7fdaa77fe640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda940386f0 0x7fda9403aba0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80a31e0 tx=0x7fda98032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:18.407+0000 7fdaa6ffd640 1 --2- 172.21.15.67:0/18655954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa8097ba0 0x7fdaa81355e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b510 tx=0x7fda9c0090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fda9c015020 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:54:19.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.236+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fda9c015020 con 0x7fdaa8097ba0 2024-09-15T01:54:19.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.236+0000 7fda8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:19.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.236+0000 7fda8f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:19.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.407+0000 7fdaae11d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T01:54:19.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.407+0000 7fdaae11d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T01:54:19.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.407+0000 7fdaae11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.407+0000 7fdaae11d640 1 -- 172.21.15.67:0/18655954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7fdaa8067840 con 0x7fdaa8097ba0 2024-09-15T01:54:19.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.424+0000 7fda8f7fe640 1 -- 172.21.15.67:0/18655954 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v63) v1 ==== 132+0+0 (secure 0 0 0) 0x7fda9c010840 con 0x7fdaa8097ba0 2024-09-15T01:54:19.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.424+0000 7fda8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T01:54:19.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.424+0000 7fda8f7fe640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-15T01:54:19.427 INFO:tasks.workunit.client.0.smithi067.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-15T01:54:19.429 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:19.430 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-15T01:54:19.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.505+0000 7f245b39d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:19.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.505+0000 7f245b39d640 10 monclient: build_initial_monmap 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.505+0000 7f245b39d640 10 monclient: monmap: 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:19.506392+0000 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:19.506392+0000 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.508 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:19.509 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:19.509 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:19.509 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:19.509 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:19.509 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.505+0000 7f245b39d640 1 Processor -- start 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 -- start start 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient: init 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24540a0b50 0x7f2454069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f24540a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f245409eea0 0x7f245409f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f245409eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 0x7f2454131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f2454131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245406f290 con 0x7f245409eea0 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245409f7e0 con 0x7f24540a0b50 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245406a270 con 0x7f2454131b10 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245b39d640 10 monclient(hunting): authenticate will time out at 891.975514s 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245993a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 msgr2=0x7f2454131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:19.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f245993a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 0x7f2454131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24540a0b50 0x7f2454069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24540a0b50 0x7f2454069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60632/0 (socket says 172.21.15.67:60632) 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 -- 172.21.15.67:0/378321662 learned_addr learned my addr 172.21.15.67:0/378321662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): get_auth_request con 0x7f24540a0b50 auth_method 0 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 1 --2- 172.21.15.67:0/378321662 >> v2:172.21.15.17:3300/0 conn(0x7f245409eea0 0x7f245409f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:19.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): get_auth_request con 0x7f245409eea0 auth_method 0 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:19.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2458938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient(hunting): handle_auth_done global_id 5068 payload 274 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 -- 172.21.15.67:0/378321662 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 msgr2=0x7f2454131f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 --2- 172.21.15.67:0/378321662 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 0x7f2454131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 -- 172.21.15.67:0/378321662 >> v2:172.21.15.17:3300/0 conn(0x7f245409eea0 msgr2=0x7f245409f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 --2- 172.21.15.67:0/378321662 >> v2:172.21.15.17:3300/0 conn(0x7f245409eea0 0x7f245409f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient: _finish_hunting 0 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 monclient: found mon.noname-c 2024-09-15T01:54:19.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.506+0000 7f2459139640 1 -- 172.21.15.67:0/378321662 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24540a0f20 con 0x7f24540a0b50 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f2459139640 1 -- 172.21.15.67:0/378321662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24540a0b50 msgr2=0x7f2454069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f2459139640 1 --2- 172.21.15.67:0/378321662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24540a0b50 0x7f2454069340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f245406a450 tx=0x7f24500091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b7687b397d2317 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 1 -- 172.21.15.67:0/378321662 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450032440 con 0x7f24540a0b50 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:19.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: dump: 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: _finish_auth 0 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 1 -- 172.21.15.67:0/378321662 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24500325a0 con 0x7f24540a0b50 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 5 monclient: authenticate success, global_id 5068 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 1 -- 172.21.15.67:0/378321662 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450032940 con 0x7f24540a0b50 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f244bfff640 10 monclient: dump: 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.530 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 4 set_mon_vals no callback set 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 10 monclient: shutdown 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 -- 172.21.15.67:0/378321662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24540a0b50 msgr2=0x7f2454069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 --2- 172.21.15.67:0/378321662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24540a0b50 0x7f2454069340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f245406a450 tx=0x7f24500091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 -- 172.21.15.67:0/378321662 shutdown_connections 2024-09-15T01:54:19.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 --2- 172.21.15.67:0/378321662 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2454131b10 0x7f2454131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 --2- 172.21.15.67:0/378321662 >> v2:172.21.15.17:3300/0 conn(0x7f245409eea0 0x7f245409f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 --2- 172.21.15.67:0/378321662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24540a0b50 0x7f2454069340 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 -- 172.21.15.67:0/378321662 >> 172.21.15.67:0/378321662 conn(0x7f2454071290 msgr2=0x7f24540a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 -- 172.21.15.67:0/378321662 shutdown_connections 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.510+0000 7f245b39d640 1 -- 172.21.15.67:0/378321662 wait complete. 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 10 monclient: build_initial_monmap 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 10 monclient: monmap: 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 1 Processor -- start 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 1 -- start start 2024-09-15T01:54:19.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 10 monclient: init 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.511+0000 7f245b39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 0x7f2454141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f245409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24540a0b50 0x7f2454141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f24540a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:19.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f2454131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2454141970 con 0x7f24540a0b50 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245411f6b0 con 0x7f2454131b10 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24540a1260 con 0x7f245409eea0 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245b39d640 10 monclient(hunting): authenticate will time out at 891.981063s 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f2459139640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 msgr2=0x7f2454141cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60638/0 (socket says 172.21.15.67:60638) 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 -- 172.21.15.67:0/3239312789 learned_addr learned my addr 172.21.15.67:0/3239312789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): get_auth_request con 0x7f2454131b10 auth_method 0 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f2459139640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 0x7f2454141cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f2458938640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24540a0b50 0x7f2454141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient(hunting): handle_auth_done global_id 5071 payload 824 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 -- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 msgr2=0x7f2454141cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 0x7f2454141cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 -- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24540a0b50 msgr2=0x7f2454141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24540a0b50 0x7f2454141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient: _finish_hunting 0 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 monclient: found mon.noname-b 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.512+0000 7f245993a640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450009450 con 0x7f2454131b10 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245993a640 1 -- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 msgr2=0x7f2454130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245993a640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f244c00bb10 tx=0x7f244c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:19.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c00cbc0 con 0x7f2454131b10 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: dump: 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:19.538 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c00cd40 con 0x7f2454131b10 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 5 monclient: authenticate success, global_id 5071 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 10 monclient: _renew_subs 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f245411e1f0 con 0x7f2454131b10 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c006150 con 0x7f2454131b10 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:19.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f24497fa640 10 monclient: dump: 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f244b7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:19.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 10 monclient: _renew_subs 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.514+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245411d5c0 con 0x7f2454131b10 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f244c012020 con 0x7f2454131b10 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f245b39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f245b39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f245b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245404bf60 con 0x7f2454131b10 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f24497fa640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 0x7f243803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.515+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f244c04bcf0 con 0x7f2454131b10 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.516+0000 7f2459139640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 0x7f243803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.516+0000 7f2459139640 10 monclient: get_auth_request con 0x7f2438038720 auth_method 0 2024-09-15T01:54:19.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.518+0000 7f2459139640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 0x7f243803abd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f24540a08b0 tx=0x7f2450032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:19.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.520+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f244c051020 con 0x7f2454131b10 2024-09-15T01:54:19.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.520+0000 7f24497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:19.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.520+0000 7f24497fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:19.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.685+0000 7f245b39d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:19.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.685+0000 7f245b39d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:19.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.685+0000 7f245b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:19.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:19.685+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f245409eba0 con 0x7f2454131b10 2024-09-15T01:54:20.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.246+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v64) v1 ==== 172+0+0 (secure 0 0 0) 0x7f244c01da80 con 0x7f2454131b10 2024-09-15T01:54:20.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.246+0000 7f24497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:20.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.246+0000 7f24497fa640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-15T01:54:20.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.345+0000 7f245b39d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.345+0000 7f245b39d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.345+0000 7f245b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.345+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f245406f000 con 0x7f2454131b10 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.346+0000 7f24497fa640 1 -- 172.21.15.67:0/3239312789 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v64) v1 ==== 179+0+0 (secure 0 0 0) 0x7f244c04a480 con 0x7f2454131b10 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.346+0000 7f24497fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.346+0000 7f24497fa640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-15T01:54:20.347 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'pinning_pool' does not exist 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 msgr2=0x7f243803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 0x7f243803abd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f24540a08b0 tx=0x7f2450032000 comp rx=0 tx=0).stop 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 10 monclient: shutdown 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 msgr2=0x7f2454130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f244c00bb10 tx=0x7f244c0097d0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 shutdown_connections 2024-09-15T01:54:20.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2438038720 0x7f243803abd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24540a0b50 0x7f2454141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2454131b10 0x7f2454130fc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 --2- 172.21.15.67:0/3239312789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f245409eea0 0x7f2454141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 >> 172.21.15.67:0/3239312789 conn(0x7f2454071290 msgr2=0x7f24540a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 shutdown_connections 2024-09-15T01:54:20.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.348+0000 7f245b39d640 1 -- 172.21.15.67:0/3239312789 wait complete. 2024-09-15T01:54:20.358 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-15T01:54:20.358 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:20.359 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket foo osd 2024-09-15T01:54:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.434+0000 7f40b0e2b640 10 monclient: get_monmap_and_config 2024-09-15T01:54:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.434+0000 7f40b0e2b640 10 monclient: build_initial_monmap 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.434+0000 7f40b0e2b640 10 monclient: monmap: 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:20.434564+0000 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:20.434564+0000 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:20.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.434+0000 7f40b0e2b640 1 Processor -- start 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 -- start start 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient: init 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40ac0a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac09eea0 0x7f40ac09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f40ac09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f40ac136e40 0x7f40ac139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-a con 0x7f40ac136e40 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac06a270 con 0x7f40ac0a0b50 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac06f290 con 0x7f40ac136e40 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac09f7e0 con 0x7f40ac09eea0 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40b0e2b640 10 monclient(hunting): authenticate will time out at 892.903691s 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac09eea0 0x7f40ac09f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac09eea0 0x7f40ac09f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41762/0 (socket says 172.21.15.67:41762) 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 1 -- 172.21.15.67:0/4038107326 learned_addr learned my addr 172.21.15.67:0/4038107326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): get_auth_request con 0x7f40ac09eea0 auth_method 0 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40aad9d640 1 --2- 172.21.15.67:0/4038107326 >> v2:172.21.15.17:3300/0 conn(0x7f40ac136e40 0x7f40ac139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.435+0000 7f40a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): get_auth_request con 0x7f40ac136e40 auth_method 0 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 10 monclient(hunting): handle_auth_done global_id 5074 payload 274 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 1 -- 172.21.15.67:0/4038107326 >> v2:172.21.15.17:3300/0 conn(0x7f40ac136e40 msgr2=0x7f40ac139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 1 --2- 172.21.15.67:0/4038107326 >> v2:172.21.15.17:3300/0 conn(0x7f40ac136e40 0x7f40ac139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 1 -- 172.21.15.67:0/4038107326 >> v1:172.21.15.17:6789/0 conn(0x7f40ac0a0b50 legacy=0x7f40ac069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 1 monclient: found mon.noname-c 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.436+0000 7f40a9d9b640 1 -- 172.21.15.67:0/4038107326 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0a0f20 con 0x7f40ac09eea0 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.438+0000 7f40a9d9b640 1 -- 172.21.15.67:0/4038107326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 msgr2=0x7f40ac09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.438+0000 7f40a9d9b640 1 --2- 172.21.15.67:0/4038107326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac09f2a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0a08b0 tx=0x7f409c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64b63d4bde0d09df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 1 -- 172.21.15.67:0/4038107326 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c032440 con 0x7f40ac09eea0 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: dump: 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.448 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: _finish_auth 0 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 1 -- 172.21.15.67:0/4038107326 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c0325a0 con 0x7f40ac09eea0 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 5 monclient: authenticate success, global_id 5074 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:20.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 1 -- 172.21.15.67:0/4038107326 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c032980 con 0x7f40ac09eea0 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 4 set_mon_vals no callback set 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40a959a640 10 monclient: dump: 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.450 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 10 monclient: shutdown 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 -- 172.21.15.67:0/4038107326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 msgr2=0x7f40ac09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 --2- 172.21.15.67:0/4038107326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac09f2a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0a08b0 tx=0x7f409c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 -- 172.21.15.67:0/4038107326 shutdown_connections 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 --2- 172.21.15.67:0/4038107326 >> v2:172.21.15.17:3300/0 conn(0x7f40ac136e40 0x7f40ac139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 --2- 172.21.15.67:0/4038107326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac09f2a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 -- 172.21.15.67:0/4038107326 >> 172.21.15.67:0/4038107326 conn(0x7f40ac071290 msgr2=0x7f40ac0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 -- 172.21.15.67:0/4038107326 shutdown_connections 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.439+0000 7f40b0e2b640 1 -- 172.21.15.67:0/4038107326 wait complete. 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.440+0000 7f40b0e2b640 10 monclient: build_initial_monmap 2024-09-15T01:54:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.440+0000 7f40b0e2b640 10 monclient: monmap: 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.440+0000 7f40b0e2b640 1 Processor -- start 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.440+0000 7f40b0e2b640 1 -- start start 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.440+0000 7f40b0e2b640 10 monclient: init 2024-09-15T01:54:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-b con 0x7f40ac09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0a0b50 0x7f40ac11e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f40ac0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136e40 0x7f40ac11ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): picked mon.noname-a con 0x7f40ac136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac0a1260 con 0x7f40ac136e40 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac09e280 con 0x7f40ac09eea0 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac121b40 con 0x7f40ac0a0b50 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40b0e2b640 10 monclient(hunting): authenticate will time out at 892.909087s 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac135d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41764/0 (socket says 172.21.15.67:41764) 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 -- 172.21.15.67:0/1832959146 learned_addr learned my addr 172.21.15.67:0/1832959146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): get_auth_request con 0x7f40ac09eea0 auth_method 0 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40a9d9b640 1 -- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0a0b50 msgr2=0x7f40ac11e810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40a9d9b640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0a0b50 0x7f40ac11e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:20.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136e40 0x7f40ac11ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 10 monclient(hunting): get_auth_request con 0x7f40ac136e40 auth_method 0 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient(hunting): handle_auth_done global_id 5077 payload 824 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 -- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0a0b50 msgr2=0x7f40ac11e810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0a0b50 0x7f40ac11e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 -- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136e40 msgr2=0x7f40ac11ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136e40 0x7f40ac11ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 monclient: found mon.noname-b 2024-09-15T01:54:20.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aa59c640 1 -- 172.21.15.67:0/1832959146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c009450 con 0x7f40ac09eea0 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.441+0000 7f40aad9d640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136e40 0x7f40ac11ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40aa59c640 1 -- 172.21.15.67:0/1832959146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 msgr2=0x7f40ac135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40aa59c640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac09eea0 0x7f40ac135d20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f409400c5e0 tx=0x7f4094007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094012840 con 0x7f40ac09eea0 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: dump: 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.456 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 5 monclient: authenticate success, global_id 5077 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40940129a0 con 0x7f40ac09eea0 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 10 monclient: _renew_subs 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 1 -- 172.21.15.67:0/1832959146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac120f30 con 0x7f40ac09eea0 2024-09-15T01:54:20.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094012d60 con 0x7f40ac09eea0 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a27fc640 10 monclient: dump: 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.458 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 10 monclient: _renew_subs 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40b0e2b640 1 -- 172.21.15.67:0/1832959146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac120300 con 0x7f40ac09eea0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.443+0000 7f40a8d99640 4 set_mon_vals no callback set 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.444+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4094011590 con 0x7f40ac09eea0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.444+0000 7f40b0e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.444+0000 7f40b0e2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.444+0000 7f40b0e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.444+0000 7f40b0e2b640 1 -- 172.21.15.67:0/1832959146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac1222b0 con 0x7f40ac09eea0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.445+0000 7f40a27fc640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f408c038740 0x7f408c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.445+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f409404bd30 con 0x7f40ac09eea0 2024-09-15T01:54:20.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.445+0000 7f40a9d9b640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f408c038740 0x7f408c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.445+0000 7f40a9d9b640 10 monclient: get_auth_request con 0x7f408c038740 auth_method 0 2024-09-15T01:54:20.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.447+0000 7f40a9d9b640 1 --2- 172.21.15.67:0/1832959146 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f408c038740 0x7f408c03abf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f409c034000 tx=0x7f409c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.449+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4094010aa0 con 0x7f40ac09eea0 2024-09-15T01:54:20.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.449+0000 7f40a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.449+0000 7f40a27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:20.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.613+0000 7f40b0e2b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T01:54:20.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.613+0000 7f40b0e2b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T01:54:20.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.613+0000 7f40b0e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.613+0000 7f40b0e2b640 1 -- 172.21.15.67:0/1832959146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7f40ac04bf60 con 0x7f40ac09eea0 2024-09-15T01:54:20.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.615+0000 7f40a27fc640 1 -- 172.21.15.67:0/1832959146 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v64) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4094016070 con 0x7f40ac09eea0 2024-09-15T01:54:20.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.615+0000 7f40a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T01:54:20.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.615+0000 7f40a27fc640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-15T01:54:20.616 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-15T01:54:20.620 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:20.620 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket foo root 2024-09-15T01:54:20.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 10 monclient: get_monmap_and_config 2024-09-15T01:54:20.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 10 monclient: build_initial_monmap 2024-09-15T01:54:20.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 10 monclient: monmap: 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:20.695211+0000 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:20.695211+0000 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:20.697 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 1 Processor -- start 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 1 -- start start 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.695+0000 7f7a8224f640 10 monclient: init 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a7c098140 0x7f7a7c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7c098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7a7c0a0370 0x7f7a7c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-a con 0x7f7a7c0a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a7c098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c098ef0 con 0x7f7a7c098a50 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0a0b80 con 0x7f7a7c0a0370 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0a00b0 con 0x7f7a7c098140 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a8224f640 10 monclient(hunting): authenticate will time out at 893.164445s 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a7c098140 0x7f7a7c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a7c098140 0x7f7a7c1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41774/0 (socket says 172.21.15.67:41774) 2024-09-15T01:54:20.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 -- 172.21.15.67:0/4156635175 learned_addr learned my addr 172.21.15.67:0/4156635175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f7a7c098140 auth_method 0 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient(hunting): handle_auth_done global_id 5080 payload 274 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 -- 172.21.15.67:0/4156635175 >> v2:172.21.15.17:3300/0 conn(0x7f7a7c0a0370 msgr2=0x7f7a7c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 --2- 172.21.15.67:0/4156635175 >> v2:172.21.15.17:3300/0 conn(0x7f7a7c0a0370 0x7f7a7c098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 -- 172.21.15.67:0/4156635175 >> v1:172.21.15.17:6789/0 conn(0x7f7a7c098a50 legacy=0x7f7a7c0695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:20.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:20.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:20.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.696+0000 7f7a7b7fe640 1 -- 172.21.15.67:0/4156635175 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c071390 con 0x7f7a7c098140 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.699+0000 7f7a7b7fe640 1 -- 172.21.15.67:0/4156635175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c098140 msgr2=0x7f7a7c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.699+0000 7f7a7b7fe640 1 --2- 172.21.15.67:0/4156635175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c098140 0x7f7a7c1321e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a5340 tx=0x7f7a6c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bbda6e9b13c6c29 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 1 -- 172.21.15.67:0/4156635175 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c032440 con 0x7f7a7c098140 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:20.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: dump: 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.706 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 1 -- 172.21.15.67:0/4156635175 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c0325a0 con 0x7f7a7c098140 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 5 monclient: authenticate success, global_id 5080 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 1 -- 172.21.15.67:0/4156635175 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c032940 con 0x7f7a7c098140 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 4 set_mon_vals no callback set 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:20.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a7a7fc640 10 monclient: dump: 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 10 monclient: shutdown 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:20.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 -- 172.21.15.67:0/4156635175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c098140 msgr2=0x7f7a7c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 --2- 172.21.15.67:0/4156635175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c098140 0x7f7a7c1321e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a5340 tx=0x7f7a6c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 -- 172.21.15.67:0/4156635175 shutdown_connections 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 --2- 172.21.15.67:0/4156635175 >> v2:172.21.15.17:3300/0 conn(0x7f7a7c0a0370 0x7f7a7c098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 --2- 172.21.15.67:0/4156635175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c098140 0x7f7a7c1321e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 -- 172.21.15.67:0/4156635175 >> 172.21.15.67:0/4156635175 conn(0x7f7a7c0a5780 msgr2=0x7f7a7c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 -- 172.21.15.67:0/4156635175 shutdown_connections 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.700+0000 7f7a8224f640 1 -- 172.21.15.67:0/4156635175 wait complete. 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.701+0000 7f7a8224f640 10 monclient: build_initial_monmap 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.701+0000 7f7a8224f640 10 monclient: monmap: 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:20.709 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.701+0000 7f7a8224f640 1 Processor -- start 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.701+0000 7f7a8224f640 1 -- start start 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.701+0000 7f7a8224f640 10 monclient: init 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c098140 0x7f7a7c09e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-a con 0x7f7a7c098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 0x7f7a7c130c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7c098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): picked mon.noname-b con 0x7f7a7c0a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c099090 con 0x7f7a7c098140 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c131c40 con 0x7f7a7c0a0370 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c11dda0 con 0x7f7a7c098a50 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a8224f640 10 monclient(hunting): authenticate will time out at 893.170192s 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7affd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 msgr2=0x7f7a7c130c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41782/0 (socket says 172.21.15.67:41782) 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 0x7f7a7c130c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 -- 172.21.15.67:0/1258037717 learned_addr learned my addr 172.21.15.67:0/1258037717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): get_auth_request con 0x7f7a7c0a0370 auth_method 0 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7b7fe640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c098140 0x7f7a7c09e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient(hunting): handle_auth_done global_id 5083 payload 824 2024-09-15T01:54:20.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 -- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 msgr2=0x7f7a7c130c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 0x7f7a7c130c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 -- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c098140 msgr2=0x7f7a7c09e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c098140 0x7f7a7c09e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 monclient: found mon.noname-b 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.702+0000 7f7a7bfff640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c009450 con 0x7f7a7c0a0370 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a7bfff640 1 -- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 msgr2=0x7f7a7c11ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a7bfff640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b7f0 tx=0x7f7a74009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400c6a0 con 0x7f7a7c0a0370 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:20.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 10 monclient: dump: 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:20.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7400c820 con 0x7f7a7c0a0370 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a8224f640 5 monclient: authenticate success, global_id 5083 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.704+0000 7f7a8224f640 10 monclient: _renew_subs 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a8224f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c09f280 con 0x7f7a7c0a0370 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400cbe0 con 0x7f7a7c0a0370 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 10 monclient: dump: 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a8224f640 10 monclient: _renew_subs 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a8224f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c11ceb0 con 0x7f7a7c0a0370 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a79ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.705+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a74012070 con 0x7f7a7c0a0370 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a8224f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a8224f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a8224f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c11e220 con 0x7f7a7c0a0370 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a637fe640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 0x7f7a6803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f7a7404b640 con 0x7f7a7c0a0370 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a7b7fe640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 0x7f7a6803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.706+0000 7f7a7b7fe640 10 monclient: get_auth_request con 0x7f7a680386f0 auth_method 0 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.708+0000 7f7a7b7fe640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 0x7f7a6803aba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a1b30 tx=0x7f7a6c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.710+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7a7400ce00 con 0x7f7a7c0a0370 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.710+0000 7f7a637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:20.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.710+0000 7f7a637fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:20.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.875+0000 7f7a8224f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:20.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.875+0000 7f7a8224f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:20.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.875+0000 7f7a8224f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:20.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:20.875+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f7a7c04bf60 con 0x7f7a7c0a0370 2024-09-15T01:54:21.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.249+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7a7401e3a0 con 0x7f7a7c0a0370 2024-09-15T01:54:21.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.249+0000 7f7a637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:21.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.249+0000 7f7a637fe640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-15T01:54:21.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.355+0000 7f7a8224f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:21.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.355+0000 7f7a8224f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:21.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.355+0000 7f7a8224f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.355+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f7a7c11e410 con 0x7f7a7c0a0370 2024-09-15T01:54:21.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.356+0000 7f7a637fe640 1 -- 172.21.15.67:0/1258037717 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v65) v1 ==== 126+0+0 (secure 0 0 0) 0x7f7a74013370 con 0x7f7a7c0a0370 2024-09-15T01:54:21.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.356+0000 7f7a637fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T01:54:21.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.356+0000 7f7a637fe640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-15T01:54:21.358 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'foo' already exists 2024-09-15T01:54:21.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 msgr2=0x7f7a6803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:21.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 0x7f7a6803aba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a1b30 tx=0x7f7a6c032000 comp rx=0 tx=0).stop 2024-09-15T01:54:21.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 10 monclient: shutdown 2024-09-15T01:54:21.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:21.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 msgr2=0x7f7a7c11ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b7f0 tx=0x7f7a74009840 comp rx=0 tx=0).stop 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 shutdown_connections 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a680386f0 0x7f7a6803aba0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c0a0370 0x7f7a7c11ba40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c098a50 0x7f7a7c130c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 --2- 172.21.15.67:0/1258037717 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c098140 0x7f7a7c09e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 >> 172.21.15.67:0/1258037717 conn(0x7f7a7c0a5780 msgr2=0x7f7a7c135020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 shutdown_connections 2024-09-15T01:54:21.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.358+0000 7f7a8224f640 1 -- 172.21.15.67:0/1258037717 wait complete. 2024-09-15T01:54:21.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 2024-09-15T01:54:21.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.441+0000 7f5105b5d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:21.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.441+0000 7f5105b5d640 10 monclient: build_initial_monmap 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.441+0000 7f5105b5d640 10 monclient: monmap: 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:21.441652+0000 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:21.441652+0000 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:21.443 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:21.444 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:21.444 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 Processor -- start 2024-09-15T01:54:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 -- start start 2024-09-15T01:54:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient: init 2024-09-15T01:54:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000a3770 0x7f510006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f51000a3770 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 0x7f510006e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f510006e130 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f510009da30 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510009a000 con 0x7f510009da30 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5100068b20 con 0x7f510006e130 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510009f6e0 con 0x7f51000a3770 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f5105b5d640 10 monclient(hunting): authenticate will time out at 893.910777s 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000a3770 0x7f510006b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000a3770 0x7f510006b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41796/0 (socket says 172.21.15.67:41796) 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 1 -- 172.21.15.67:0/1141783104 learned_addr learned my addr 172.21.15.67:0/1141783104 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 10 monclient(hunting): get_auth_request con 0x7f51000a3770 auth_method 0 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.442+0000 7f50ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_done global_id 5086 payload 274 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50f6ffd640 1 --2- 172.21.15.67:0/1141783104 >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 0x7f510006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 1 -- 172.21.15.67:0/1141783104 >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 msgr2=0x7f510006e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 1 --2- 172.21.15.67:0/1141783104 >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 0x7f510006e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 1 -- 172.21.15.67:0/1141783104 >> v1:172.21.15.17:6789/0 conn(0x7f510009da30 legacy=0x7f510009de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50ff7fe640 1 -- 172.21.15.67:0/1141783104 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000a3b40 con 0x7f51000a3770 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50f6ffd640 10 monclient: get_auth_request con 0x7f510006e130 auth_method 0 2024-09-15T01:54:21.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.443+0000 7f50f6ffd640 1 --2- 172.21.15.67:0/1141783104 >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 0x7f510006e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50ff7fe640 1 -- 172.21.15.67:0/1141783104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000a3770 msgr2=0x7f510006b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50ff7fe640 1 --2- 172.21.15.67:0/1141783104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000a3770 0x7f510006b9b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f510009eb50 tx=0x7f50f00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc606e7c4c6d7a80 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 1 -- 172.21.15.67:0/1141783104 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0032440 con 0x7f51000a3770 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: dump: 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.455 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 1 -- 172.21.15.67:0/1141783104 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00325a0 con 0x7f51000a3770 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 1 -- 172.21.15.67:0/1141783104 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0032940 con 0x7f51000a3770 2024-09-15T01:54:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 5 monclient: authenticate success, global_id 5086 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 4 set_mon_vals no callback set 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f50feffd640 10 monclient: dump: 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 10 monclient: shutdown 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 -- 172.21.15.67:0/1141783104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000a3770 msgr2=0x7f510006b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 --2- 172.21.15.67:0/1141783104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000a3770 0x7f510006b9b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f510009eb50 tx=0x7f50f00091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 -- 172.21.15.67:0/1141783104 shutdown_connections 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 --2- 172.21.15.67:0/1141783104 >> v2:172.21.15.17:3300/0 conn(0x7f510006e130 0x7f510006e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 --2- 172.21.15.67:0/1141783104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000a3770 0x7f510006b9b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 -- 172.21.15.67:0/1141783104 >> 172.21.15.67:0/1141783104 conn(0x7f5100070fc0 msgr2=0x7f51000a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 -- 172.21.15.67:0/1141783104 shutdown_connections 2024-09-15T01:54:21.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.446+0000 7f5105b5d640 1 -- 172.21.15.67:0/1141783104 wait complete. 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.447+0000 7f5105b5d640 10 monclient: build_initial_monmap 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.447+0000 7f5105b5d640 10 monclient: monmap: 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.447+0000 7f5105b5d640 1 Processor -- start 2024-09-15T01:54:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.447+0000 7f5105b5d640 1 -- start start 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.447+0000 7f5105b5d640 10 monclient: init 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f510006e130 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f510009da30 0x7f5100133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f510009da30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 0x7f510011e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f51000a3770 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51001204e0 con 0x7f510009da30 2024-09-15T01:54:21.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510006ea90 con 0x7f510006e130 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51001221a0 con 0x7f51000a3770 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f5105b5d640 10 monclient(hunting): authenticate will time out at 893.916260s 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41802/0 (socket says 172.21.15.67:41802) 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 -- 172.21.15.67:0/3572456072 learned_addr learned my addr 172.21.15.67:0/3572456072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): get_auth_request con 0x7f510006e130 auth_method 0 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:21.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50fffff640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 msgr2=0x7f510011e900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50fffff640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 0x7f510011e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient(hunting): handle_auth_done global_id 5089 payload 824 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50f6ffd640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f510009da30 0x7f5100133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:21.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 msgr2=0x7f510011e900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 0x7f510011e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f510009da30 msgr2=0x7f5100133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f510009da30 0x7f5100133d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 monclient: found mon.noname-b 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.448+0000 7f50ff7fe640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0009450 con 0x7f510006e130 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.450+0000 7f50ff7fe640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 msgr2=0x7f5100140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.450+0000 7f50ff7fe640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009910 tx=0x7f50f0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0031ca0 con 0x7f510006e130 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:21.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: dump: 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 5 monclient: authenticate success, global_id 5089 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f0031e00 con 0x7f510006e130 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f003c650 con 0x7f510006e130 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 10 monclient: _renew_subs 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5100121ec0 con 0x7f510006e130 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50fe7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f50f7fff640 10 monclient: dump: 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:21.466 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 10 monclient: _renew_subs 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.451+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5100121170 con 0x7f510006e130 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f50f003ca80 con 0x7f510006e130 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f5105b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f5105b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f5105b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510004bf60 con 0x7f510006e130 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f50f7fff640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 0x7f50e003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5058+0+0 (secure 0 0 0) 0x7f50f006bf50 con 0x7f510006e130 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f50f6ffd640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 0x7f50e003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:21.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.452+0000 7f50f6ffd640 10 monclient: get_auth_request con 0x7f50e00386f0 auth_method 0 2024-09-15T01:54:21.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.454+0000 7f50f6ffd640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 0x7f50e003aba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f50e80089b0 tx=0x7f50e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:21.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.456+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f50f0042300 con 0x7f510006e130 2024-09-15T01:54:21.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.456+0000 7f50f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:21.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.456+0000 7f50f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:21.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.633+0000 7f5105b5d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:54:21.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.633+0000 7f5105b5d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T01:54:21.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.633+0000 7f5105b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:21.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:21.633+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f5100067140 con 0x7f510006e130 2024-09-15T01:54:22.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.261+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v66) v1 ==== 58+0+2 (secure 0 0 0) 0x7f50f0037020 con 0x7f510006e130 2024-09-15T01:54:22.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.261+0000 7f50f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T01:54:22.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.261+0000 7f50f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:22.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.373+0000 7f5105b5d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:54:22.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.373+0000 7f5105b5d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T01:54:22.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.373+0000 7f5105b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:22.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:22.373+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f51000008d0 con 0x7f510006e130 2024-09-15T01:54:23.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.262+0000 7f50f7fff640 1 -- 172.21.15.67:0/3572456072 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) v1 ==== 58+0+2 (secure 0 0 0) 0x7f50f003a300 con 0x7f510006e130 2024-09-15T01:54:23.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.262+0000 7f50f7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T01:54:23.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.262+0000 7f50f7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:23.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 msgr2=0x7f50e003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:23.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 0x7f50e003aba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f50e80089b0 tx=0x7f50e8008040 comp rx=0 tx=0).stop 2024-09-15T01:54:23.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 10 monclient: shutdown 2024-09-15T01:54:23.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:23.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 msgr2=0x7f5100140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:23.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009910 tx=0x7f50f0032000 comp rx=0 tx=0).stop 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 shutdown_connections 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50e00386f0 0x7f50e003aba0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f510009da30 0x7f5100133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51000a3770 0x7f510011e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 --2- 172.21.15.67:0/3572456072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006e130 0x7f5100140f60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 >> 172.21.15.67:0/3572456072 conn(0x7f5100070fc0 msgr2=0x7f5100098a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 shutdown_connections 2024-09-15T01:54:23.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.265+0000 7f5105b5d640 1 -- 172.21.15.67:0/3572456072 wait complete. 2024-09-15T01:54:23.277 INFO:tasks.workunit.client.0.smithi067.stderr:+ o1=7 2024-09-15T01:54:23.278 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 2024-09-15T01:54:23.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 10 monclient: get_monmap_and_config 2024-09-15T01:54:23.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 10 monclient: build_initial_monmap 2024-09-15T01:54:23.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 10 monclient: monmap: 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:23.356210+0000 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:23.356210+0000 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:23.358 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 1 Processor -- start 2024-09-15T01:54:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 1 -- start start 2024-09-15T01:54:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.356+0000 7f9aa02f8640 10 monclient: init 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a98098370 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 0x7f9a98098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a98098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a98136ea0 0x7f9a98139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a98136ea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a980a17b0 con 0x7f9a98098370 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a98132a70 con 0x7f9a98136ea0 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9809feb0 con 0x7f9a98098740 2024-09-15T01:54:23.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9aa02f8640 10 monclient(hunting): authenticate will time out at 895.825436s 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a98136ea0 0x7f9a98139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9d893640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 msgr2=0x7f9a98098b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9d893640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 0x7f9a98098b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a98136ea0 0x7f9a98139280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41804/0 (socket says 172.21.15.67:41804) 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 -- 172.21.15.67:0/3440724038 learned_addr learned my addr 172.21.15.67:0/3440724038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): get_auth_request con 0x7f9a98136ea0 auth_method 0 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:23.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient(hunting): handle_auth_done global_id 5092 payload 274 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 -- 172.21.15.67:0/3440724038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 msgr2=0x7f9a98098b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 --2- 172.21.15.67:0/3440724038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 0x7f9a98098b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 -- 172.21.15.67:0/3440724038 >> v1:172.21.15.17:6789/0 conn(0x7f9a98098370 legacy=0x7f9a98132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient: _finish_hunting 0 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 monclient: found mon.noname-c 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.357+0000 7f9a9e895640 1 -- 172.21.15.67:0/3440724038 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a980a00f0 con 0x7f9a98136ea0 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.360+0000 7f9a9e895640 1 -- 172.21.15.67:0/3440724038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98136ea0 msgr2=0x7f9a98139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.360+0000 7f9a9e895640 1 --2- 172.21.15.67:0/3440724038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98136ea0 0x7f9a98139280 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9a9809fe40 tx=0x7f9a90009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8cbed2a2551c679 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.360+0000 7f9a9d092640 1 -- 172.21.15.67:0/3440724038 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90007610 con 0x7f9a98136ea0 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: dump: 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:23.368 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: _finish_auth 0 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 1 -- 172.21.15.67:0/3440724038 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9002d050 con 0x7f9a98136ea0 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 5 monclient: authenticate success, global_id 5092 2024-09-15T01:54:23.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 1 -- 172.21.15.67:0/3440724038 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900079a0 con 0x7f9a98136ea0 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 4 set_mon_vals no callback set 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9a9d092640 10 monclient: dump: 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 10 monclient: shutdown 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 -- 172.21.15.67:0/3440724038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98136ea0 msgr2=0x7f9a98139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/3440724038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98136ea0 0x7f9a98139280 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9a9809fe40 tx=0x7f9a90009840 comp rx=0 tx=0).stop 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 -- 172.21.15.67:0/3440724038 shutdown_connections 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/3440724038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98136ea0 0x7f9a98139280 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/3440724038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098740 0x7f9a98098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 -- 172.21.15.67:0/3440724038 >> 172.21.15.67:0/3440724038 conn(0x7f9a980a59a0 msgr2=0x7f9a980a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 -- 172.21.15.67:0/3440724038 shutdown_connections 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.361+0000 7f9aa02f8640 1 -- 172.21.15.67:0/3440724038 wait complete. 2024-09-15T01:54:23.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient: build_initial_monmap 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient: monmap: 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 1 Processor -- start 2024-09-15T01:54:23.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 1 -- start start 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient: init 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 0x7f9a98136780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a98098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.362+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a98098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 0x7f9a981331b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): picked mon.noname-a con 0x7f9a98136ea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98131a00 con 0x7f9a98136ea0 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:23.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98136cc0 con 0x7f9a98098740 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a980a0820 con 0x7f9a98098370 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9aa02f8640 10 monclient(hunting): authenticate will time out at 895.831028s 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e094640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 msgr2=0x7f9a98136780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e094640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 0x7f9a98136780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41816/0 (socket says 172.21.15.67:41816) 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 -- 172.21.15.67:0/2780485512 learned_addr learned my addr 172.21.15.67:0/2780485512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): get_auth_request con 0x7f9a98098740 auth_method 0 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:23.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 0x7f9a981331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 10 monclient(hunting): get_auth_request con 0x7f9a98136ea0 auth_method 0 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient(hunting): handle_auth_done global_id 5095 payload 824 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 -- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 msgr2=0x7f9a98136780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 0x7f9a98136780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 -- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 msgr2=0x7f9a981331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 0x7f9a981331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:23.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient: _finish_hunting 0 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 monclient: found mon.noname-b 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9d893640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90009ae0 con 0x7f9a98098740 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.363+0000 7f9a9e895640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 0x7f9a981331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a9d893640 1 -- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 msgr2=0x7f9a98132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a9d893640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b600 tx=0x7f9a880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88015070 con 0x7f9a98098740 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: dump: 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:23.376 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a88003d90 con 0x7f9a98098740 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 5 monclient: authenticate success, global_id 5095 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 10 monclient: _renew_subs 2024-09-15T01:54:23.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9809f390 con 0x7f9a98098740 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88006380 con 0x7f9a98098740 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a8e7fc640 10 monclient: dump: 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:23.378 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9a9c891640 4 set_mon_vals no callback set 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 10 monclient: _renew_subs 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.365+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a98135ac0 con 0x7f9a98098740 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.366+0000 7f9aa02f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.366+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9a88010440 con 0x7f9a98098740 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.366+0000 7f9aa02f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.366+0000 7f9aa02f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.366+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9804bf60 con 0x7f9a98098740 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.367+0000 7f9a8e7fc640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 0x7f9a7c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:23.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.367+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f9a8804b720 con 0x7f9a98098740 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.367+0000 7f9a9e094640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 0x7f9a7c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.367+0000 7f9a9e094640 10 monclient: get_auth_request con 0x7f9a7c038720 auth_method 0 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.369+0000 7f9a9e094640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 0x7f9a7c03abd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007fd0 tx=0x7f9a94007a70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.371+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a88018910 con 0x7f9a98098740 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.371+0000 7f9a8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.371+0000 7f9a8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:23.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.545+0000 7f9aa02f8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:54:23.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.545+0000 7f9aa02f8640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T01:54:23.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.545+0000 7f9aa02f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:23.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:23.545+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f9a98135e50 con 0x7f9a98098740 2024-09-15T01:54:24.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.268+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v68) v1 ==== 58+0+2 (secure 0 0 0) 0x7f9a88018310 con 0x7f9a98098740 2024-09-15T01:54:24.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.268+0000 7f9a8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T01:54:24.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.268+0000 7f9a8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:24.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.379+0000 7f9aa02f8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:54:24.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.379+0000 7f9aa02f8640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T01:54:24.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.379+0000 7f9aa02f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:24.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:24.379+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f9a980008d0 con 0x7f9a98098740 2024-09-15T01:54:25.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.267+0000 7f9a8e7fc640 1 -- 172.21.15.67:0/2780485512 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v69) v1 ==== 58+0+2 (secure 0 0 0) 0x7f9a880184f0 con 0x7f9a98098740 2024-09-15T01:54:25.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.267+0000 7f9a8e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T01:54:25.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.267+0000 7f9a8e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 msgr2=0x7f9a7c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 0x7f9a7c03abd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007fd0 tx=0x7f9a94007a70 comp rx=0 tx=0).stop 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 10 monclient: shutdown 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 msgr2=0x7f9a98132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b600 tx=0x7f9a880091a0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 shutdown_connections 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a7c038720 0x7f9a7c03abd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a98136ea0 0x7f9a981331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a98098740 0x7f9a98132c70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 --2- 172.21.15.67:0/2780485512 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a98098370 0x7f9a98136780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 >> 172.21.15.67:0/2780485512 conn(0x7f9a980a59a0 msgr2=0x7f9a98138c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 shutdown_connections 2024-09-15T01:54:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.270+0000 7f9aa02f8640 1 -- 172.21.15.67:0/2780485512 wait complete. 2024-09-15T01:54:25.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ o2=9 2024-09-15T01:54:25.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-15T01:54:25.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.367+0000 7f16c571d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:25.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.367+0000 7f16c571d640 10 monclient: build_initial_monmap 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.367+0000 7f16c571d640 10 monclient: monmap: 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:25.367539+0000 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:25.367539+0000 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:25.373 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.367+0000 7f16c571d640 1 Processor -- start 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 -- start start 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient: init 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16c00a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f16c009eea0 0x7f16c009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-a con 0x7f16c009eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c0136e40 0x7f16c0139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-c con 0x7f16c0136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c00a61a0 con 0x7f16c00a0b50 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c009f800 con 0x7f16c009eea0 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c00a1610 con 0x7f16c0136e40 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16c571d640 10 monclient(hunting): authenticate will time out at 897.836678s 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c0136e40 0x7f16c0139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c0136e40 0x7f16c0139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41826/0 (socket says 172.21.15.67:41826) 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 1 -- 172.21.15.67:0/2064373582 learned_addr learned my addr 172.21.15.67:0/2064373582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:25.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): get_auth_request con 0x7f16c0136e40 auth_method 0 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16be7fc640 1 --2- 172.21.15.67:0/2064373582 >> v2:172.21.15.17:3300/0 conn(0x7f16c009eea0 0x7f16c009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16be7fc640 10 monclient(hunting): get_auth_request con 0x7f16c009eea0 auth_method 0 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.368+0000 7f16be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:25.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 10 monclient(hunting): handle_auth_done global_id 5098 payload 274 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 1 -- 172.21.15.67:0/2064373582 >> v2:172.21.15.17:3300/0 conn(0x7f16c009eea0 msgr2=0x7f16c009f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 1 --2- 172.21.15.67:0/2064373582 >> v2:172.21.15.17:3300/0 conn(0x7f16c009eea0 0x7f16c009f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 1 -- 172.21.15.67:0/2064373582 >> v1:172.21.15.17:6789/0 conn(0x7f16c00a0b50 legacy=0x7f16c0069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.369+0000 7f16bf7fe640 1 -- 172.21.15.67:0/2064373582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c00a04c0 con 0x7f16c0136e40 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.371+0000 7f16bf7fe640 1 -- 172.21.15.67:0/2064373582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0136e40 msgr2=0x7f16c0139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.371+0000 7f16bf7fe640 1 --2- 172.21.15.67:0/2064373582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0136e40 0x7f16c0139220 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16c00a1200 tx=0x7f16b8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42245f503b4ac791 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 1 -- 172.21.15.67:0/2064373582 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8007610 con 0x7f16c0136e40 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: dump: 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.383 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 1 -- 172.21.15.67:0/2064373582 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b802d050 con 0x7f16c0136e40 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 5 monclient: authenticate success, global_id 5098 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 1 -- 172.21.15.67:0/2064373582 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b80079a0 con 0x7f16c0136e40 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 4 set_mon_vals no callback set 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16bdffb640 10 monclient: dump: 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:25.385 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 10 monclient: shutdown 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 -- 172.21.15.67:0/2064373582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0136e40 msgr2=0x7f16c0139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 --2- 172.21.15.67:0/2064373582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0136e40 0x7f16c0139220 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16c00a1200 tx=0x7f16b8009840 comp rx=0 tx=0).stop 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 -- 172.21.15.67:0/2064373582 shutdown_connections 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 --2- 172.21.15.67:0/2064373582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0136e40 0x7f16c0139220 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 --2- 172.21.15.67:0/2064373582 >> v2:172.21.15.17:3300/0 conn(0x7f16c009eea0 0x7f16c009f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 -- 172.21.15.67:0/2064373582 >> 172.21.15.67:0/2064373582 conn(0x7f16c0071330 msgr2=0x7f16c00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 -- 172.21.15.67:0/2064373582 shutdown_connections 2024-09-15T01:54:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.372+0000 7f16c571d640 1 -- 172.21.15.67:0/2064373582 wait complete. 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.373+0000 7f16c571d640 10 monclient: build_initial_monmap 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.373+0000 7f16c571d640 10 monclient: monmap: 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.373+0000 7f16c571d640 1 Processor -- start 2024-09-15T01:54:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.373+0000 7f16c571d640 1 -- start start 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.373+0000 7f16c571d640 10 monclient: init 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-b con 0x7f16c009eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 0x7f16c011be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-a con 0x7f16c00a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 0x7f16c011c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): picked mon.noname-c con 0x7f16c0136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c011e020 con 0x7f16c00a0b50 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:25.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c009b020 con 0x7f16c009eea0 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00a3310 con 0x7f16c0136e40 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16c571d640 10 monclient(hunting): authenticate will time out at 897.842401s 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41834/0 (socket says 172.21.15.67:41834) 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 -- 172.21.15.67:0/2772849956 learned_addr learned my addr 172.21.15.67:0/2772849956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): get_auth_request con 0x7f16c009eea0 auth_method 0 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16bf7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 msgr2=0x7f16c011c3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:25.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16bf7fe640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 0x7f16c011c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 0x7f16c011be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 10 monclient(hunting): get_auth_request con 0x7f16c00a0b50 auth_method 0 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient(hunting): handle_auth_done global_id 5101 payload 824 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 -- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 msgr2=0x7f16c011c3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 0x7f16c011c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 -- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 msgr2=0x7f16c011be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 0x7f16c011be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 monclient: found mon.noname-b 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16beffd640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8009ae0 con 0x7f16c009eea0 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.374+0000 7f16be7fc640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 0x7f16c011be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16beffd640 1 -- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 msgr2=0x7f16c011b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16beffd640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c5e0 tx=0x7f16b0007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0012840 con 0x7f16c009eea0 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: dump: 2024-09-15T01:54:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00129a0 con 0x7f16c009eea0 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 5 monclient: authenticate success, global_id 5101 2024-09-15T01:54:25.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 10 monclient: _renew_subs 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c011dc40 con 0x7f16c009eea0 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0012d60 con 0x7f16c009eea0 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16a6ffd640 10 monclient: dump: 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:25.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 10 monclient: _renew_subs 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16bd7fa640 4 set_mon_vals no callback set 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c009a6a0 con 0x7f16c009eea0 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16c571d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16c571d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16c571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f16b0011590 con 0x7f16c009eea0 2024-09-15T01:54:25.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c004bf60 con 0x7f16c009eea0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16a6ffd640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 0x7f16ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5518+0+0 (secure 0 0 0) 0x7f16b004bee0 con 0x7f16c009eea0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16be7fc640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 0x7f16ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.378+0000 7f16be7fc640 10 monclient: get_auth_request con 0x7f16ac038740 auth_method 0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.381+0000 7f16be7fc640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 0x7f16ac03abf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f16c011d890 tx=0x7f16b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.382+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f16b0010ac0 con 0x7f16c009eea0 2024-09-15T01:54:25.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.382+0000 7f16a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:25.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.382+0000 7f16a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:25.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.549+0000 7f16c571d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:25.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.549+0000 7f16c571d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:25.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.549+0000 7f16c571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:25.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:25.549+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f16c011d9c0 con 0x7f16c009eea0 2024-09-15T01:54:26.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.271+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v70) v1 ==== 203+0+0 (secure 0 0 0) 0x7f16b0016070 con 0x7f16c009eea0 2024-09-15T01:54:26.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.272+0000 7f16a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.272+0000 7f16a6ffd640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-15T01:54:26.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.373+0000 7f16c571d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.373+0000 7f16c571d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.373+0000 7f16c571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.373+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f16c006f000 con 0x7f16c009eea0 2024-09-15T01:54:26.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.375+0000 7f16a6ffd640 1 -- 172.21.15.67:0/2772849956 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7f16b00102e0 con 0x7f16c009eea0 2024-09-15T01:54:26.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.375+0000 7f16a6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.375+0000 7f16a6ffd640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T01:54:26.377 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T01:54:26.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 msgr2=0x7f16ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 0x7f16ac03abf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f16c011d890 tx=0x7f16b4008040 comp rx=0 tx=0).stop 2024-09-15T01:54:26.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 10 monclient: shutdown 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 msgr2=0x7f16c011b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c5e0 tx=0x7f16b0007ed0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 shutdown_connections 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16ac038740 0x7f16ac03abf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c00a0b50 0x7f16c011be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0136e40 0x7f16c011c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 --2- 172.21.15.67:0/2772849956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c009eea0 0x7f16c011b940 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 >> 172.21.15.67:0/2772849956 conn(0x7f16c0071330 msgr2=0x7f16c009dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 shutdown_connections 2024-09-15T01:54:26.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.377+0000 7f16c571d640 1 -- 172.21.15.67:0/2772849956 wait complete. 2024-09-15T01:54:26.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.465+0000 7f8c2f699640 10 monclient: get_monmap_and_config 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.465+0000 7f8c2f699640 10 monclient: build_initial_monmap 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.465+0000 7f8c2f699640 10 monclient: monmap: 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:26.466704+0000 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:26.466704+0000 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:26.468 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:26.469 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:26.469 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:26.469 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:26.469 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 Processor -- start 2024-09-15T01:54:26.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 -- start start 2024-09-15T01:54:26.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient: init 2024-09-15T01:54:26.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c280a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8c2809eea0 0x7f8c2809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2809eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28136e40 0x7f8c28139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-c con 0x7f8c28136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2806a270 con 0x7f8c280a0b50 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2806f290 con 0x7f8c2809eea0 2024-09-15T01:54:26.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2809f7e0 con 0x7f8c28136e40 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2f699640 10 monclient(hunting): authenticate will time out at 898.935803s 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28136e40 0x7f8c28139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28136e40 0x7f8c28139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41842/0 (socket says 172.21.15.67:41842) 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 1 -- 172.21.15.67:0/2664410995 learned_addr learned my addr 172.21.15.67:0/2664410995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 10 monclient(hunting): get_auth_request con 0x7f8c28136e40 auth_method 0 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.466+0000 7f8c2dc36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2cc34640 1 --2- 172.21.15.67:0/2664410995 >> v2:172.21.15.17:3300/0 conn(0x7f8c2809eea0 0x7f8c2809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2cc34640 10 monclient(hunting): get_auth_request con 0x7f8c2809eea0 auth_method 0 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2cc34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2cc34640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2cc34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient(hunting): handle_auth_done global_id 5104 payload 274 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 1 -- 172.21.15.67:0/2664410995 >> v2:172.21.15.17:3300/0 conn(0x7f8c2809eea0 msgr2=0x7f8c2809f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 1 --2- 172.21.15.67:0/2664410995 >> v2:172.21.15.17:3300/0 conn(0x7f8c2809eea0 0x7f8c2809f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 1 -- 172.21.15.67:0/2664410995 >> v1:172.21.15.17:6789/0 conn(0x7f8c280a0b50 legacy=0x7f8c28069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient: _finish_hunting 0 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 1 monclient: found mon.noname-c 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.467+0000 7f8c2dc36640 1 -- 172.21.15.67:0/2664410995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280a0f20 con 0x7f8c28136e40 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2dc36640 1 -- 172.21.15.67:0/2664410995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28136e40 msgr2=0x7f8c28139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2dc36640 1 --2- 172.21.15.67:0/2664410995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28136e40 0x7f8c28139220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c2809cdb0 tx=0x7f8c20009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e68ea9f7a64d80b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 1 -- 172.21.15.67:0/2664410995 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20007610 con 0x7f8c28136e40 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: dump: 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.480 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 1 -- 172.21.15.67:0/2664410995 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2002d050 con 0x7f8c28136e40 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:26.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 1 -- 172.21.15.67:0/2664410995 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c200079a0 con 0x7f8c28136e40 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 5 monclient: authenticate success, global_id 5104 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 4 set_mon_vals no callback set 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c1ffff640 10 monclient: dump: 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.482 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 10 monclient: shutdown 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 -- 172.21.15.67:0/2664410995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28136e40 msgr2=0x7f8c28139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2664410995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28136e40 0x7f8c28139220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c2809cdb0 tx=0x7f8c20009840 comp rx=0 tx=0).stop 2024-09-15T01:54:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 -- 172.21.15.67:0/2664410995 shutdown_connections 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2664410995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28136e40 0x7f8c28139220 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2664410995 >> v2:172.21.15.17:3300/0 conn(0x7f8c2809eea0 0x7f8c2809f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 -- 172.21.15.67:0/2664410995 >> 172.21.15.67:0/2664410995 conn(0x7f8c28071290 msgr2=0x7f8c280a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 -- 172.21.15.67:0/2664410995 shutdown_connections 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.470+0000 7f8c2f699640 1 -- 172.21.15.67:0/2664410995 wait complete. 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.471+0000 7f8c2f699640 10 monclient: build_initial_monmap 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.471+0000 7f8c2f699640 10 monclient: monmap: 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:26.484 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.471+0000 7f8c2f699640 1 Processor -- start 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.471+0000 7f8c2f699640 1 -- start start 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.471+0000 7f8c2f699640 10 monclient: init 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 0x7f8c2811e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-b con 0x7f8c280a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c28136e40 0x7f8c2811f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): picked mon.noname-a con 0x7f8c28136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280a1260 con 0x7f8c28136e40 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2809e5b0 con 0x7f8c280a0b50 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28121840 con 0x7f8c2809eea0 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2f699640 10 monclient(hunting): authenticate will time out at 898.941459s 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2d435640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 msgr2=0x7f8c2811e5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2d435640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 0x7f8c2811e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41856/0 (socket says 172.21.15.67:41856) 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 -- 172.21.15.67:0/2153677499 learned_addr learned my addr 172.21.15.67:0/2153677499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): get_auth_request con 0x7f8c280a0b50 auth_method 0 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient(hunting): handle_auth_done global_id 5107 payload 824 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2dc36640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c28136e40 0x7f8c2811f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 -- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 msgr2=0x7f8c2811e5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 0x7f8c2811e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 -- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c28136e40 msgr2=0x7f8c2811f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c28136e40 0x7f8c2811f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient: _finish_hunting 0 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 monclient: found mon.noname-b 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.472+0000 7f8c2cc34640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c20009ae0 con 0x7f8c280a0b50 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2cc34640 1 -- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 msgr2=0x7f8c2811eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2cc34640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800d8c0 tx=0x7f8c18008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800de50 con 0x7f8c280a0b50 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: dump: 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:26.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 5 monclient: authenticate success, global_id 5107 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1800bae0 con 0x7f8c280a0b50 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 10 monclient: _renew_subs 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c28122100 con 0x7f8c280a0b50 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800be80 con 0x7f8c280a0b50 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1d7fa640 10 monclient: dump: 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.490 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 10 monclient: _renew_subs 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c1f7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.475+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28120be0 con 0x7f8c280a0b50 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8c18014640 con 0x7f8c280a0b50 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2f699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2f699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2f699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c28121b40 con 0x7f8c280a0b50 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c1d7fa640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 0x7f8c0c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f8c1804c560 con 0x7f8c280a0b50 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2d435640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 0x7f8c0c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.476+0000 7f8c2d435640 10 monclient: get_auth_request con 0x7f8c0c038740 auth_method 0 2024-09-15T01:54:26.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.478+0000 7f8c2d435640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 0x7f8c0c03abf0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c24007fd0 tx=0x7f8c24007a70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.478+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:54:26.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.480+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8c18011570 con 0x7f8c280a0b50 2024-09-15T01:54:26.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.481+0000 7f8c1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.481+0000 7f8c1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:26.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.648+0000 7f8c2f699640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.648+0000 7f8c2f699640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.648+0000 7f8c2f699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.648+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f8c2804bf60 con 0x7f8c280a0b50 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.649+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7f8c1801dbb0 con 0x7f8c280a0b50 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.649+0000 7f8c1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.649+0000 7f8c1d7fa640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T01:54:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.750+0000 7f8c2f699640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.750+0000 7f8c2f699640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.750+0000 7f8c2f699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.750+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f8c2806f000 con 0x7f8c280a0b50 2024-09-15T01:54:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.751+0000 7f8c1d7fa640 1 -- 172.21.15.67:0/2153677499 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7f8c18011340 con 0x7f8c280a0b50 2024-09-15T01:54:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.751+0000 7f8c1d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T01:54:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.751+0000 7f8c1d7fa640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T01:54:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 msgr2=0x7f8c0c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 0x7f8c0c03abf0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c24007fd0 tx=0x7f8c24007a70 comp rx=0 tx=0).stop 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 10 monclient: shutdown 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 msgr2=0x7f8c2811eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800d8c0 tx=0x7f8c18008040 comp rx=0 tx=0).stop 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 shutdown_connections 2024-09-15T01:54:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c0c038740 0x7f8c0c03abf0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c28136e40 0x7f8c2811f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280a0b50 0x7f8c2811eb10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 --2- 172.21.15.67:0/2153677499 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2809eea0 0x7f8c2811e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 >> 172.21.15.67:0/2153677499 conn(0x7f8c28071290 msgr2=0x7f8c2809ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 shutdown_connections 2024-09-15T01:54:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.753+0000 7f8c2f699640 1 -- 172.21.15.67:0/2153677499 wait complete. 2024-09-15T01:54:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.840+0000 7f22bea20640 10 monclient: get_monmap_and_config 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.840+0000 7f22bea20640 10 monclient: build_initial_monmap 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.840+0000 7f22bea20640 10 monclient: monmap: 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:26.841723+0000 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:26.841723+0000 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:26.843 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:26.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 Processor -- start 2024-09-15T01:54:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 -- start start 2024-09-15T01:54:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient: init 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f22b8068dc0 0x7f22b8132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-a con 0x7f22b8068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22b8069190 0x7f22b8069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-c con 0x7f22b8069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22b812e960 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b8096c10 con 0x7f22b812e960 2024-09-15T01:54:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b80a14d0 con 0x7f22b8068dc0 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b809ff20 con 0x7f22b8069190 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bea20640 10 monclient(hunting): authenticate will time out at 899.310837s 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22b77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22b8069190 0x7f22b8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bcfbd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f22b812e960 0x7f22b812ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44886/0 (socket says 172.21.15.67:44886) 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22bcfbd640 1 -- 172.21.15.67:0/720317637 learned_addr learned my addr 172.21.15.67:0/720317637 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22b77fe640 10 monclient(hunting): get_auth_request con 0x7f22b8069190 auth_method 0 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.841+0000 7f22b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient(hunting): handle_auth_done global_id 5110 payload 274 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b7fff640 1 --2- 172.21.15.67:0/720317637 >> v2:172.21.15.17:3300/0 conn(0x7f22b8068dc0 0x7f22b8132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 1 -- 172.21.15.67:0/720317637 >> v2:172.21.15.17:3300/0 conn(0x7f22b8068dc0 msgr2=0x7f22b8132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 1 --2- 172.21.15.67:0/720317637 >> v2:172.21.15.17:3300/0 conn(0x7f22b8068dc0 0x7f22b8132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 1 -- 172.21.15.67:0/720317637 >> v1:172.21.15.67:6789/0 conn(0x7f22b812e960 legacy=0x7f22b812ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.842+0000 7f22b77fe640 1 -- 172.21.15.67:0/720317637 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80a0180 con 0x7f22b8069190 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b77fe640 1 -- 172.21.15.67:0/720317637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 msgr2=0x7f22b8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b77fe640 1 --2- 172.21.15.67:0/720317637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b8069590 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f22b806a680 tx=0x7f22ac0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=893104dabda98eed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 1 -- 172.21.15.67:0/720317637 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac032440 con 0x7f22b8069190 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: dump: 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 1 -- 172.21.15.67:0/720317637 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ac0325a0 con 0x7f22b8069190 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 1 -- 172.21.15.67:0/720317637 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac032940 con 0x7f22b8069190 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22b6ffd640 10 monclient: dump: 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.857 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 5 monclient: authenticate success, global_id 5110 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 4 set_mon_vals no callback set 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 10 monclient: shutdown 2024-09-15T01:54:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 -- 172.21.15.67:0/720317637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 msgr2=0x7f22b8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 --2- 172.21.15.67:0/720317637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b8069590 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f22b806a680 tx=0x7f22ac0090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 -- 172.21.15.67:0/720317637 shutdown_connections 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 --2- 172.21.15.67:0/720317637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b8069590 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 --2- 172.21.15.67:0/720317637 >> v2:172.21.15.17:3300/0 conn(0x7f22b8068dc0 0x7f22b8132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 -- 172.21.15.67:0/720317637 >> 172.21.15.67:0/720317637 conn(0x7f22b80711b0 msgr2=0x7f22b80a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 -- 172.21.15.67:0/720317637 shutdown_connections 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.845+0000 7f22bea20640 1 -- 172.21.15.67:0/720317637 wait complete. 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.846+0000 7f22bea20640 10 monclient: build_initial_monmap 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.846+0000 7f22bea20640 10 monclient: monmap: 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:26.859 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.846+0000 7f22bea20640 1 Processor -- start 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.846+0000 7f22bea20640 1 -- start start 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.846+0000 7f22bea20640 10 monclient: init 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:26.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 0x7f22b8141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-c con 0x7f22b8068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-b con 0x7f22b8069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22b812e960 0x7f22b8098fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): picked mon.noname-a con 0x7f22b812e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b811bc80 con 0x7f22b812e960 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b809b530 con 0x7f22b8069190 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80a0890 con 0x7f22b8068dc0 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:26.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bea20640 10 monclient(hunting): authenticate will time out at 899.316317s 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b7fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 msgr2=0x7f22b8141170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 0x7f22b8141170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41878/0 (socket says 172.21.15.67:41878) 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 -- 172.21.15.67:0/4029734704 learned_addr learned my addr 172.21.15.67:0/4029734704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): get_auth_request con 0x7f22b8069190 auth_method 0 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:26.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22b812e960 0x7f22b8098fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): get_auth_request con 0x7f22b812e960 auth_method 0 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22bcfbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient(hunting): handle_auth_done global_id 5113 payload 824 2024-09-15T01:54:26.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 -- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 msgr2=0x7f22b8141170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 0x7f22b8141170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 -- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22b812e960 msgr2=0x7f22b8098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22b812e960 0x7f22b8098fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 monclient: found mon.noname-b 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.847+0000 7f22b77fe640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac009390 con 0x7f22b8069190 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.850+0000 7f22b77fe640 1 -- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 msgr2=0x7f22b812db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.850+0000 7f22b77fe640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009360 tx=0x7f22ac032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.850+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac031c70 con 0x7f22b8069190 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: dump: 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.865 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ac031dd0 con 0x7f22b8069190 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 5 monclient: authenticate success, global_id 5113 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 10 monclient: _renew_subs 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b809b040 con 0x7f22b8069190 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac03d670 con 0x7f22b8069190 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:26.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f229ffff640 10 monclient: dump: 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22b67fc640 4 set_mon_vals no callback set 2024-09-15T01:54:26.867 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 10 monclient: _renew_subs 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.852+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b809ab30 con 0x7f22b8069190 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f22ac039040 con 0x7f22b8069190 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22bea20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22bea20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22bea20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b804bf60 con 0x7f22b8069190 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f229ffff640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 0x7f22a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f22ac06be40 con 0x7f22b8069190 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22b7fff640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 0x7f22a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:26.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.853+0000 7f22b7fff640 10 monclient: get_auth_request con 0x7f22a4038740 auth_method 0 2024-09-15T01:54:26.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.855+0000 7f22b7fff640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 0x7f22a403abf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22a8006fd0 tx=0x7f22a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:26.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.857+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f22ac030950 con 0x7f22b8069190 2024-09-15T01:54:26.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.857+0000 7f229ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:26.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:26.857+0000 7f229ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:27.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.023+0000 7f22bea20640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.023+0000 7f22bea20640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.023+0000 7f22bea20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.023+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f22b8069ad0 con 0x7f22b8069190 2024-09-15T01:54:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.195+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v71) v1 ==== 203+0+0 (secure 0 0 0) 0x7f22ac0370c0 con 0x7f22b8069190 2024-09-15T01:54:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.195+0000 7f229ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.195+0000 7f229ffff640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-15T01:54:27.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.297+0000 7f22bea20640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.297+0000 7f22bea20640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.297+0000 7f22bea20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.297+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f22b8069ad0 con 0x7f22b8069190 2024-09-15T01:54:27.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.322+0000 7f229ffff640 1 -- 172.21.15.67:0/4029734704 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (secure 0 0 0) 0x7f22ac06a430 con 0x7f22b8069190 2024-09-15T01:54:27.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.322+0000 7f229ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:27.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.322+0000 7f229ffff640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T01:54:27.325 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 msgr2=0x7f22a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 0x7f22a403abf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22a8006fd0 tx=0x7f22a8008040 comp rx=0 tx=0).stop 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 10 monclient: shutdown 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 msgr2=0x7f22b812db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009360 tx=0x7f22ac032000 comp rx=0 tx=0).stop 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 shutdown_connections 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f22a4038740 0x7f22a403abf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22b812e960 0x7f22b8098fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b8069190 0x7f22b812db50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 --2- 172.21.15.67:0/4029734704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f22b8068dc0 0x7f22b8141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 >> 172.21.15.67:0/4029734704 conn(0x7f22b80711b0 msgr2=0x7f22b8130b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:27.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 shutdown_connections 2024-09-15T01:54:27.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.324+0000 7f22bea20640 1 -- 172.21.15.67:0/4029734704 wait complete. 2024-09-15T01:54:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-15T01:54:27.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 10 monclient: get_monmap_and_config 2024-09-15T01:54:27.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 10 monclient: build_initial_monmap 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 10 monclient: monmap: 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:27.413332+0000 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:27.413332+0000 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:27.415 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:27.416 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:27.416 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 1 Processor -- start 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 1 -- start start 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.412+0000 7f85f2d00640 10 monclient: init 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 0x7f85ec1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-a con 0x7f85ec098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85ec098a50 0x7f85ec0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec098a50 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:27.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0a1730 con 0x7f85ec098140 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec098ef0 con 0x7f85ec098a50 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0a0b80 con 0x7f85ec0a0370 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f2d00640 10 monclient(hunting): authenticate will time out at 899.882468s 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85ec098a50 0x7f85ec0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85ebfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 msgr2=0x7f85ec098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85ec098a50 0x7f85ec0695c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41880/0 (socket says 172.21.15.67:41880) 2024-09-15T01:54:27.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85ebfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 -- 172.21.15.67:0/3108946234 learned_addr learned my addr 172.21.15.67:0/3108946234 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): get_auth_request con 0x7f85ec098a50 auth_method 0 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient(hunting): handle_auth_done global_id 5116 payload 274 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/3108946234 >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 0x7f85ec1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 -- 172.21.15.67:0/3108946234 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 msgr2=0x7f85ec098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:27.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 --2- 172.21.15.67:0/3108946234 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 -- 172.21.15.67:0/3108946234 >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 msgr2=0x7f85ec1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 --2- 172.21.15.67:0/3108946234 >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 0x7f85ec1321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 monclient: found mon.noname-c 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f129d640 1 -- 172.21.15.67:0/3108946234 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0a00b0 con 0x7f85ec098a50 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f0a9c640 10 monclient: get_auth_request con 0x7f85ec098140 auth_method 0 2024-09-15T01:54:27.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.413+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/3108946234 >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 0x7f85ec1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:27.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.416+0000 7f85f129d640 1 -- 172.21.15.67:0/3108946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098a50 msgr2=0x7f85ec0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:27.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.416+0000 7f85f129d640 1 --2- 172.21.15.67:0/3108946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098a50 0x7f85ec0695c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0a1660 tx=0x7f85e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58400958474c5e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:27.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 1 -- 172.21.15.67:0/3108946234 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4007610 con 0x7f85ec098a50 2024-09-15T01:54:27.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:27.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: dump: 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:27.428 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 5 monclient: authenticate success, global_id 5116 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 1 -- 172.21.15.67:0/3108946234 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e402d050 con 0x7f85ec098a50 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 4 set_mon_vals no callback set 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 1 -- 172.21.15.67:0/3108946234 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e40079a0 con 0x7f85ec098a50 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:27.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85eb7fe640 10 monclient: dump: 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 10 monclient: shutdown 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 -- 172.21.15.67:0/3108946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098a50 msgr2=0x7f85ec0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 --2- 172.21.15.67:0/3108946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098a50 0x7f85ec0695c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0a1660 tx=0x7f85e4009840 comp rx=0 tx=0).stop 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 -- 172.21.15.67:0/3108946234 shutdown_connections 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 --2- 172.21.15.67:0/3108946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098a50 0x7f85ec0695c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 --2- 172.21.15.67:0/3108946234 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 --2- 172.21.15.67:0/3108946234 >> v2:172.21.15.17:3300/0 conn(0x7f85ec098140 0x7f85ec1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 -- 172.21.15.67:0/3108946234 >> 172.21.15.67:0/3108946234 conn(0x7f85ec0a5780 msgr2=0x7f85ec0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 -- 172.21.15.67:0/3108946234 shutdown_connections 2024-09-15T01:54:27.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.417+0000 7f85f2d00640 1 -- 172.21.15.67:0/3108946234 wait complete. 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.418+0000 7f85f2d00640 10 monclient: build_initial_monmap 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.418+0000 7f85f2d00640 10 monclient: monmap: 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:27.432 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.418+0000 7f85f2d00640 1 Processor -- start 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.418+0000 7f85f2d00640 1 -- start start 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.418+0000 7f85f2d00640 10 monclient: init 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85ec098a50 0x7f85ec11d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-a con 0x7f85ec098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:27.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec11dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec120fd0 con 0x7f85ec098a50 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1332d0 con 0x7f85ec098140 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec11f230 con 0x7f85ec0a0370 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f2d00640 10 monclient(hunting): authenticate will time out at 899.888129s 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:27.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41894/0 (socket says 172.21.15.67:41894) 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 -- 172.21.15.67:0/173279736 learned_addr learned my addr 172.21.15.67:0/173279736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): get_auth_request con 0x7f85ec098140 auth_method 0 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f129d640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 msgr2=0x7f85ec11dbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f129d640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec11dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:27.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient(hunting): handle_auth_done global_id 5119 payload 824 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 msgr2=0x7f85ec11dbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec11dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85ec098a50 msgr2=0x7f85ec11d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85ec098a50 0x7f85ec11d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 monclient: found mon.noname-b 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.419+0000 7f85f0a9c640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e4009ae0 con 0x7f85ec098140 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f0a9c640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 msgr2=0x7f85ec130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:27.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c880 tx=0x7f85dc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00ce10 con 0x7f85ec098140 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f0a9c640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c880 tx=0x7f85dc008040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f85dc007e60 mon_map magic: 0 v1 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 10 monclient: dump: 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:27.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 5 monclient: authenticate success, global_id 5119 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc007aa0 con 0x7f85ec098140 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 10 monclient: _renew_subs 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec11f9b0 con 0x7f85ec098140 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 10 monclient: _renew_subs 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85eaffd640 4 set_mon_vals no callback set 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.421+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec120cf0 con 0x7f85ec098140 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc007e60 con 0x7f85ec098140 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 10 monclient: dump: 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:27.439 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85dc011600 con 0x7f85ec098140 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85f2d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85f2d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85f2d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:27.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec141d00 con 0x7f85ec098140 2024-09-15T01:54:27.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85e8ff9640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 0x7f85d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:27.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85ebfff640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 0x7f85d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:27.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.423+0000 7f85ebfff640 10 monclient: get_auth_request con 0x7f85d80386f0 auth_method 0 2024-09-15T01:54:27.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.425+0000 7f85ebfff640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 0x7f85d803aba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0a0760 tx=0x7f85e0006cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:27.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:27.427+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6044+0+0 (secure 0 0 0) 0x7f85dc04bff0 con 0x7f85ec098140 2024-09-15T01:54:28.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.044+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85dc017ae0 con 0x7f85ec098140 2024-09-15T01:54:28.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.044+0000 7f85e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:28.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.044+0000 7f85e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:28.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.213+0000 7f85f2d00640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.213+0000 7f85f2d00640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.213+0000 7f85f2d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.213+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f85ec11f9b0 con 0x7f85ec098140 2024-09-15T01:54:28.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.214+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (secure 0 0 0) 0x7f85ec11f9b0 con 0x7f85ec098140 2024-09-15T01:54:28.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.215+0000 7f85e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.215+0000 7f85e8ff9640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T01:54:28.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.317+0000 7f85f2d00640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.317+0000 7f85f2d00640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.317+0000 7f85f2d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.317+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f85ec11f950 con 0x7f85ec098140 2024-09-15T01:54:28.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.318+0000 7f85e8ff9640 1 -- 172.21.15.67:0/173279736 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (secure 0 0 0) 0x7f85dc017300 con 0x7f85ec098140 2024-09-15T01:54:28.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.318+0000 7f85e8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T01:54:28.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.318+0000 7f85e8ff9640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T01:54:28.321 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 msgr2=0x7f85d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 0x7f85d803aba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0a0760 tx=0x7f85e0006cc0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 10 monclient: shutdown 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 msgr2=0x7f85ec130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c880 tx=0x7f85dc008040 comp rx=0 tx=0).stop 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 shutdown_connections 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85d80386f0 0x7f85d803aba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85ec098a50 0x7f85ec11d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85ec0a0370 0x7f85ec11dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 --2- 172.21.15.67:0/173279736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ec098140 0x7f85ec130ba0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 >> 172.21.15.67:0/173279736 conn(0x7f85ec0a5780 msgr2=0x7f85ec0a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:28.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 shutdown_connections 2024-09-15T01:54:28.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.320+0000 7f85f2d00640 1 -- 172.21.15.67:0/173279736 wait complete. 2024-09-15T01:54:28.332 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket bar root 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.406+0000 7f97e7edd640 10 monclient: get_monmap_and_config 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.406+0000 7f97e7edd640 10 monclient: build_initial_monmap 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.406+0000 7f97e7edd640 10 monclient: monmap: 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:28.407819+0000 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:28.407819+0000 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.409 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:28.410 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:28.410 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:28.410 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:28.410 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:28.410 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 Processor -- start 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 -- start start 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient: init 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e009cb00 0x7f97e00672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-c con 0x7f97e009cb00 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:28.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e00971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-b con 0x7f97e0096e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97e009ced0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e0067f20 con 0x7f97e009ced0 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e009dff0 con 0x7f97e009cb00 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0096a70 con 0x7f97e0096e00 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.407+0000 7f97e7edd640 10 monclient(hunting): authenticate will time out at 900.876977s 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e009cb00 0x7f97e00672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5478640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 msgr2=0x7f97e00971d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5478640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e00971d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:28.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e009cb00 0x7f97e00672c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41906/0 (socket says 172.21.15.67:41906) 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e647a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f97e009ced0 0x7f97e009d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44902/0 (socket says 172.21.15.67:44902) 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 -- 172.21.15.67:0/1156879781 learned_addr learned my addr 172.21.15.67:0/1156879781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): get_auth_request con 0x7f97e009cb00 auth_method 0 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient(hunting): handle_auth_done global_id 5122 payload 274 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 -- 172.21.15.67:0/1156879781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 msgr2=0x7f97e00971d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 --2- 172.21.15.67:0/1156879781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e00971d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 -- 172.21.15.67:0/1156879781 >> v1:172.21.15.67:6789/0 conn(0x7f97e009ced0 legacy=0x7f97e009d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient: _finish_hunting 0 2024-09-15T01:54:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 monclient: found mon.noname-c 2024-09-15T01:54:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.408+0000 7f97e5c79640 1 -- 172.21.15.67:0/1156879781 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e009e4e0 con 0x7f97e009cb00 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e5c79640 1 -- 172.21.15.67:0/1156879781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 msgr2=0x7f97e00672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e5c79640 1 --2- 172.21.15.67:0/1156879781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e00672c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009910 tx=0x7f97dc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccd9bf93a3fd3dc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 1 -- 172.21.15.67:0/1156879781 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc032440 con 0x7f97e009cb00 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: dump: 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.420 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: _finish_auth 0 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 1 -- 172.21.15.67:0/1156879781 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc0325a0 con 0x7f97e009cb00 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 5 monclient: authenticate success, global_id 5122 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:28.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 1 -- 172.21.15.67:0/1156879781 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc032940 con 0x7f97e009cb00 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 4 set_mon_vals no callback set 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e4c77640 10 monclient: dump: 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:28.422 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 10 monclient: shutdown 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 -- 172.21.15.67:0/1156879781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 msgr2=0x7f97e00672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 --2- 172.21.15.67:0/1156879781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e00672c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009910 tx=0x7f97dc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 -- 172.21.15.67:0/1156879781 shutdown_connections 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 --2- 172.21.15.67:0/1156879781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e00971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 --2- 172.21.15.67:0/1156879781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e00672c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 -- 172.21.15.67:0/1156879781 >> 172.21.15.67:0/1156879781 conn(0x7f97e00a2110 msgr2=0x7f97e00a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 -- 172.21.15.67:0/1156879781 shutdown_connections 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.411+0000 7f97e7edd640 1 -- 172.21.15.67:0/1156879781 wait complete. 2024-09-15T01:54:28.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.412+0000 7f97e7edd640 10 monclient: build_initial_monmap 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.412+0000 7f97e7edd640 10 monclient: monmap: 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.412+0000 7f97e7edd640 1 Processor -- start 2024-09-15T01:54:28.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.412+0000 7f97e7edd640 1 -- start start 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.412+0000 7f97e7edd640 10 monclient: init 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e011e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-c con 0x7f97e0096e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-b con 0x7f97e009cb00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e009ced0 0x7f97e011f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): picked mon.noname-a con 0x7f97e009ced0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0138d60 con 0x7f97e009ced0 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0067bf0 con 0x7f97e009cb00 2024-09-15T01:54:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00a0a00 con 0x7f97e0096e00 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e7edd640 10 monclient(hunting): authenticate will time out at 900.882195s 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41910/0 (socket says 172.21.15.67:41910) 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5c79640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 msgr2=0x7f97e011e890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 -- 172.21.15.67:0/182428673 learned_addr learned my addr 172.21.15.67:0/182428673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): get_auth_request con 0x7f97e009cb00 auth_method 0 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5c79640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e011e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:28.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e647a640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e009ced0 0x7f97e011f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e647a640 10 monclient(hunting): get_auth_request con 0x7f97e009ced0 auth_method 0 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e647a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e647a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e647a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient(hunting): handle_auth_done global_id 5125 payload 824 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 -- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 msgr2=0x7f97e011e890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e011e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 -- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e009ced0 msgr2=0x7f97e011f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e009ced0 0x7f97e011f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:28.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient: _finish_hunting 0 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 monclient: found mon.noname-b 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.413+0000 7f97e5478640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc009450 con 0x7f97e009cb00 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.415+0000 7f97e5478640 1 -- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 msgr2=0x7f97e011edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.415+0000 7f97e5478640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f97d000b600 tx=0x7f97d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.415+0000 7f97e5478640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f97d000b600 tx=0x7f97d00091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f97d0015070 mon_map magic: 0 v1 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.415+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0015070 con 0x7f97e009cb00 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 10 monclient: dump: 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:28.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 5 monclient: authenticate success, global_id 5125 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 10 monclient: _renew_subs 2024-09-15T01:54:28.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0003d90 con 0x7f97e009cb00 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e0121cd0 con 0x7f97e009cb00 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 10 monclient: _renew_subs 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e0120d70 con 0x7f97e009cb00 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.416+0000 7f97d7fff640 4 set_mon_vals no callback set 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e7edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e7edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e7edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0006380 con 0x7f97e009cb00 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e01227b0 con 0x7f97e009cb00 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:28.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 10 monclient: dump: 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97d0010440 con 0x7f97e009cb00 2024-09-15T01:54:28.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97d5ffb640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 0x7f97c403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:28.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e5c79640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 0x7f97c403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:28.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.417+0000 7f97e5c79640 10 monclient: get_auth_request con 0x7f97c4038770 auth_method 0 2024-09-15T01:54:28.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.419+0000 7f97e5c79640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 0x7f97c403ac20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f97e00a3110 tx=0x7f97dc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:28.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:28.421+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6044+0+0 (secure 0 0 0) 0x7f97d004bb20 con 0x7f97e009cb00 2024-09-15T01:54:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.038+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f97e01227b0 con 0x7f97e009cb00 2024-09-15T01:54:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.038+0000 7f97d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:29.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.038+0000 7f97d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:29.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.207+0000 7f97e7edd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.207+0000 7f97e7edd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.207+0000 7f97e7edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.207+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f97e004bf60 con 0x7f97e009cb00 2024-09-15T01:54:29.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.285+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v72) v1 ==== 138+0+0 (secure 0 0 0) 0x7f97d0006770 con 0x7f97e009cb00 2024-09-15T01:54:29.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.285+0000 7f97d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.285+0000 7f97d5ffb640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-15T01:54:29.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.384+0000 7f97e7edd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.384+0000 7f97e7edd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.384+0000 7f97e7edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.384+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f97e0139180 con 0x7f97e009cb00 2024-09-15T01:54:29.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.386+0000 7f97d5ffb640 1 -- 172.21.15.67:0/182428673 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (secure 0 0 0) 0x7f97d0018780 con 0x7f97e009cb00 2024-09-15T01:54:29.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.386+0000 7f97d5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:29.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.386+0000 7f97d5ffb640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-15T01:54:29.388 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'bar' already exists 2024-09-15T01:54:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 msgr2=0x7f97c403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 0x7f97c403ac20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f97e00a3110 tx=0x7f97dc032000 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 10 monclient: shutdown 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 msgr2=0x7f97e011edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f97d000b600 tx=0x7f97d00091a0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 shutdown_connections 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038770 0x7f97c403ac20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e009ced0 0x7f97e011f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e009cb00 0x7f97e011edd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 --2- 172.21.15.67:0/182428673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e0096e00 0x7f97e011e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 >> 172.21.15.67:0/182428673 conn(0x7f97e00a2110 msgr2=0x7f97e009ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 shutdown_connections 2024-09-15T01:54:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.388+0000 7f97e7edd640 1 -- 172.21.15.67:0/182428673 wait complete. 2024-09-15T01:54:29.399 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket bar root 2024-09-15T01:54:29.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.473+0000 7f36c055d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:29.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.473+0000 7f36c055d640 10 monclient: build_initial_monmap 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.473+0000 7f36c055d640 10 monclient: monmap: 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:29.474574+0000 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:29.474574+0000 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:29.476 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:29.477 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.473+0000 7f36c055d640 1 Processor -- start 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 -- start start 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient: init 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36b80a0b50 0x7f36b8069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-c con 0x7f36b80a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 0x7f36b809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-a con 0x7f36b809eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36b8131b10 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b806a270 con 0x7f36b8131b10 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b806f290 con 0x7f36b809eea0 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b809f7e0 con 0x7f36b80a0b50 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36c055d640 10 monclient(hunting): authenticate will time out at 901.943736s 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36b80a0b50 0x7f36b8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36b80a0b50 0x7f36b8069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41924/0 (socket says 172.21.15.67:41924) 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 1 -- 172.21.15.67:0/3570220245 learned_addr learned my addr 172.21.15.67:0/3570220245 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 10 monclient(hunting): get_auth_request con 0x7f36b80a0b50 auth_method 0 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.474+0000 7f36be2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient(hunting): handle_auth_done global_id 5128 payload 274 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36bdaf8640 1 --2- 172.21.15.67:0/3570220245 >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 0x7f36b809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 1 -- 172.21.15.67:0/3570220245 >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 msgr2=0x7f36b809f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 1 --2- 172.21.15.67:0/3570220245 >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 0x7f36b809f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 1 -- 172.21.15.67:0/3570220245 >> v1:172.21.15.17:6789/0 conn(0x7f36b8131b10 legacy=0x7f36b8131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient: _finish_hunting 0 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 1 monclient: found mon.noname-c 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36be2f9640 1 -- 172.21.15.67:0/3570220245 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80a0f20 con 0x7f36b80a0b50 2024-09-15T01:54:29.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36bdaf8640 10 monclient: get_auth_request con 0x7f36b809eea0 auth_method 0 2024-09-15T01:54:29.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.475+0000 7f36bdaf8640 1 --2- 172.21.15.67:0/3570220245 >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 0x7f36b809f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36be2f9640 1 -- 172.21.15.67:0/3570220245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b80a0b50 msgr2=0x7f36b8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36be2f9640 1 --2- 172.21.15.67:0/3570220245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b80a0b50 0x7f36b8069340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36b806a450 tx=0x7f36b40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3091353258677f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 1 -- 172.21.15.67:0/3570220245 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4032440 con 0x7f36b80a0b50 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:29.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: dump: 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: _finish_auth 0 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 1 -- 172.21.15.67:0/3570220245 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b40325a0 con 0x7f36b80a0b50 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 5 monclient: authenticate success, global_id 5128 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 4 set_mon_vals no callback set 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 1 -- 172.21.15.67:0/3570220245 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4032940 con 0x7f36b80a0b50 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36bd2f7640 10 monclient: dump: 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 10 monclient: shutdown 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 -- 172.21.15.67:0/3570220245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b80a0b50 msgr2=0x7f36b8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 --2- 172.21.15.67:0/3570220245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b80a0b50 0x7f36b8069340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36b806a450 tx=0x7f36b40091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 -- 172.21.15.67:0/3570220245 shutdown_connections 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 --2- 172.21.15.67:0/3570220245 >> v2:172.21.15.17:3300/0 conn(0x7f36b809eea0 0x7f36b809f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 --2- 172.21.15.67:0/3570220245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b80a0b50 0x7f36b8069340 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.478+0000 7f36c055d640 1 -- 172.21.15.67:0/3570220245 >> 172.21.15.67:0/3570220245 conn(0x7f36b8071290 msgr2=0x7f36b80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 1 -- 172.21.15.67:0/3570220245 shutdown_connections 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 1 -- 172.21.15.67:0/3570220245 wait complete. 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 10 monclient: build_initial_monmap 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 10 monclient: monmap: 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:29.490 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 1 Processor -- start 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 1 -- start start 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.479+0000 7f36c055d640 10 monclient: init 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 0x7f36b809a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-c con 0x7f36b809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b80a0b50 0x7f36b8130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-a con 0x7f36b80a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): picked mon.noname-b con 0x7f36b8131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:29.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b8136570 con 0x7f36b80a0b50 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b80a12d0 con 0x7f36b8131b10 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b811df60 con 0x7f36b809eea0 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36c055d640 10 monclient(hunting): authenticate will time out at 901.949289s 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36be2f9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 msgr2=0x7f36b809a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36be2f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 0x7f36b809a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41938/0 (socket says 172.21.15.67:41938) 2024-09-15T01:54:29.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 -- 172.21.15.67:0/518012173 learned_addr learned my addr 172.21.15.67:0/518012173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): get_auth_request con 0x7f36b8131b10 auth_method 0 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36bdaf8640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b80a0b50 0x7f36b8130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient(hunting): handle_auth_done global_id 5131 payload 824 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 -- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 msgr2=0x7f36b809a790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 0x7f36b809a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 -- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b80a0b50 msgr2=0x7f36b8130b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:29.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b80a0b50 0x7f36b8130b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient: _finish_hunting 0 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 monclient: found mon.noname-b 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.480+0000 7f36beafa640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4009450 con 0x7f36b8131b10 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.482+0000 7f36beafa640 1 -- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 msgr2=0x7f36b811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.482+0000 7f36beafa640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f36b000c8c0 tx=0x7f36b0002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.482+0000 7f36beafa640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f36b000c8c0 tx=0x7f36b0002640 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f36b000aa00 mon_map magic: 0 v1 2024-09-15T01:54:29.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b000aa00 con 0x7f36b8131b10 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: dump: 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b000ab80 con 0x7f36b8131b10 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 5 monclient: authenticate success, global_id 5131 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _renew_subs 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b8131980 con 0x7f36b8131b10 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _renew_subs 2024-09-15T01:54:29.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36bcaf6640 4 set_mon_vals no callback set 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b811dc80 con 0x7f36b8131b10 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0012420 con 0x7f36b8131b10 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 10 monclient: dump: 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f36b0012810 con 0x7f36b8131b10 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.483+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b8141f30 con 0x7f36b8131b10 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.484+0000 7f36aa7fc640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 0x7f369c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.484+0000 7f36be2f9640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 0x7f369c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.484+0000 7f36be2f9640 10 monclient: get_auth_request con 0x7f369c038740 auth_method 0 2024-09-15T01:54:29.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.486+0000 7f36be2f9640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 0x7f369c03abf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36b809cdb0 tx=0x7f36b4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:29.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:29.488+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f36b004c640 con 0x7f36b8131b10 2024-09-15T01:54:30.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.106+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36b0022d40 con 0x7f36b8131b10 2024-09-15T01:54:30.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.106+0000 7f36aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:30.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.106+0000 7f36aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:30.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.274+0000 7f36c055d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.274+0000 7f36c055d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.274+0000 7f36c055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.274+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f36b804bf60 con 0x7f36b8131b10 2024-09-15T01:54:30.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.275+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (secure 0 0 0) 0x7f36b0022770 con 0x7f36b8131b10 2024-09-15T01:54:30.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.275+0000 7f36aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.275+0000 7f36aa7fc640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-15T01:54:30.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.374+0000 7f36c055d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.374+0000 7f36c055d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.374+0000 7f36c055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.374+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f36b80a04e0 con 0x7f36b8131b10 2024-09-15T01:54:30.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.426+0000 7f36aa7fc640 1 -- 172.21.15.67:0/518012173 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (secure 0 0 0) 0x7f36b0016030 con 0x7f36b8131b10 2024-09-15T01:54:30.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.426+0000 7f36aa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T01:54:30.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.426+0000 7f36aa7fc640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-15T01:54:30.428 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'bar' already exists 2024-09-15T01:54:30.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 msgr2=0x7f369c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:30.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 0x7f369c03abf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36b809cdb0 tx=0x7f36b4032000 comp rx=0 tx=0).stop 2024-09-15T01:54:30.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 10 monclient: shutdown 2024-09-15T01:54:30.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:30.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 msgr2=0x7f36b811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f36b000c8c0 tx=0x7f36b0002640 comp rx=0 tx=0).stop 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 shutdown_connections 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f369c038740 0x7f369c03abf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b80a0b50 0x7f36b8130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b8131b10 0x7f36b811b940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 --2- 172.21.15.67:0/518012173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36b809eea0 0x7f36b809a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.427+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 >> 172.21.15.67:0/518012173 conn(0x7f36b8071290 msgr2=0x7f36b8134e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.428+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 shutdown_connections 2024-09-15T01:54:30.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.428+0000 7f36c055d640 1 -- 172.21.15.67:0/518012173 wait complete. 2024-09-15T01:54:30.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush link host1 root=bar 2024-09-15T01:54:30.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.513+0000 7f072855d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:30.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.513+0000 7f072855d640 10 monclient: build_initial_monmap 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.513+0000 7f072855d640 10 monclient: monmap: 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:30.514588+0000 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:30.514588+0000 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:30.516 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:30.517 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 Processor -- start 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 -- start start 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient: init 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07200a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f072009eea0 0x7f072009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): picked mon.noname-c con 0x7f072009eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0720136e40 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200a61a0 con 0x7f0720136e40 2024-09-15T01:54:30.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200a04c0 con 0x7f07200a0b50 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072009f800 con 0x7f072009eea0 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f072855d640 10 monclient(hunting): authenticate will time out at 902.983871s 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f07262f9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07200a0b50 0x7f0720069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60980/0 (socket says 172.21.15.67:60980) 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f0725af8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f072009eea0 0x7f072009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f07262f9640 1 -- 172.21.15.67:0/3549697350 learned_addr learned my addr 172.21.15.67:0/3549697350 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f0725af8640 10 monclient(hunting): get_auth_request con 0x7f072009eea0 auth_method 0 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f0725af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f0725af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.514+0000 7f0725af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:30.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient(hunting): handle_auth_done global_id 5134 payload 274 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 1 -- 172.21.15.67:0/3549697350 >> v1:172.21.15.67:6789/0 conn(0x7f07200a0b50 legacy=0x7f0720069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 1 -- 172.21.15.67:0/3549697350 >> v1:172.21.15.17:6789/0 conn(0x7f0720136e40 legacy=0x7f0720139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 1 monclient: found mon.noname-c 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.515+0000 7f0725af8640 1 -- 172.21.15.67:0/3549697350 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720069a90 con 0x7f072009eea0 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f0725af8640 1 -- 172.21.15.67:0/3549697350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f072009eea0 msgr2=0x7f072009f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f0725af8640 1 --2- 172.21.15.67:0/3549697350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f072009eea0 0x7f072009f2c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0714009850 tx=0x7f07140090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fff1efc049a2a7d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 1 -- 172.21.15.67:0/3549697350 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714032440 con 0x7f072009eea0 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: dump: 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:30.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: _finish_auth 0 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 5 monclient: authenticate success, global_id 5134 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 1 -- 172.21.15.67:0/3549697350 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07140325a0 con 0x7f072009eea0 2024-09-15T01:54:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 4 set_mon_vals no callback set 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 1 -- 172.21.15.67:0/3549697350 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714032940 con 0x7f072009eea0 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f07252f7640 10 monclient: dump: 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 10 monclient: shutdown 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 -- 172.21.15.67:0/3549697350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f072009eea0 msgr2=0x7f072009f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 --2- 172.21.15.67:0/3549697350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f072009eea0 0x7f072009f2c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0714009850 tx=0x7f07140090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 -- 172.21.15.67:0/3549697350 shutdown_connections 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 --2- 172.21.15.67:0/3549697350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f072009eea0 0x7f072009f2c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 -- 172.21.15.67:0/3549697350 >> 172.21.15.67:0/3549697350 conn(0x7f0720071330 msgr2=0x7f07200a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 -- 172.21.15.67:0/3549697350 shutdown_connections 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.518+0000 7f072855d640 1 -- 172.21.15.67:0/3549697350 wait complete. 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.519+0000 7f072855d640 10 monclient: build_initial_monmap 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.519+0000 7f072855d640 10 monclient: monmap: 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:30.531 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.519+0000 7f072855d640 1 Processor -- start 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.519+0000 7f072855d640 1 -- start start 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.519+0000 7f072855d640 10 monclient: init 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f072009eea0 0x7f072009b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): picked mon.noname-a con 0x7f072009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): picked mon.noname-b con 0x7f07200a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 0x7f0720097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): picked mon.noname-c con 0x7f0720136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:30.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200a1260 con 0x7f072009eea0 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200a3370 con 0x7f07200a0b50 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0720098510 con 0x7f0720136e40 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f072855d640 10 monclient(hunting): authenticate will time out at 902.989187s 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48496/0 (socket says 172.21.15.67:48496) 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 -- 172.21.15.67:0/565069781 learned_addr learned my addr 172.21.15.67:0/565069781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): get_auth_request con 0x7f07200a0b50 auth_method 0 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:30.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0726afa640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 msgr2=0x7f0720097600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0726afa640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 0x7f0720097600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f072009eea0 0x7f072009b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): get_auth_request con 0x7f072009eea0 auth_method 0 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:30.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f07262f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient(hunting): handle_auth_done global_id 5137 payload 824 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 msgr2=0x7f0720097600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 0x7f0720097600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f072009eea0 msgr2=0x7f072009b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f072009eea0 0x7f072009b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 monclient: found mon.noname-b 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.520+0000 7f0725af8640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0714009390 con 0x7f07200a0b50 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.522+0000 7f0725af8640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 msgr2=0x7f07200970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.522+0000 7f0725af8640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0714009360 tx=0x7f0714032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140318c0 con 0x7f07200a0b50 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: dump: 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714031a20 con 0x7f07200a0b50 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 5 monclient: authenticate success, global_id 5137 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _renew_subs 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072009ad30 con 0x7f07200a0b50 2024-09-15T01:54:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714031dc0 con 0x7f07200a0b50 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 10 monclient: dump: 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _renew_subs 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0720098c50 con 0x7f07200a0b50 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f0724af6640 4 set_mon_vals no callback set 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0714039040 con 0x7f07200a0b50 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.523+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072004bf60 con 0x7f07200a0b50 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.524+0000 7f07127fc640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 0x7f070403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.524+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f071406c270 con 0x7f07200a0b50 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.524+0000 7f07262f9640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 0x7f070403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.524+0000 7f07262f9640 10 monclient: get_auth_request con 0x7f07040386f0 auth_method 0 2024-09-15T01:54:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.526+0000 7f07262f9640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 0x7f070403aba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f071c007fd0 tx=0x7f071c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.528+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f071404b530 con 0x7f07200a0b50 2024-09-15T01:54:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.528+0000 7f07127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.528+0000 7f07127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:30.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.695+0000 7f072855d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:30.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.695+0000 7f072855d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:30.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.695+0000 7f072855d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:30.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:30.695+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f072004bf60 con 0x7f07200a0b50 2024-09-15T01:54:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.293+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 168+0+0 (secure 0 0 0) 0x7f071403f0a0 con 0x7f07200a0b50 2024-09-15T01:54:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.293+0000 7f07127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.293+0000 7f07127fc640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.398+0000 7f072855d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.398+0000 7f072855d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.398+0000 7f072855d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.398+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f072004bf60 con 0x7f07200a0b50 2024-09-15T01:54:31.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.400+0000 7f07127fc640 1 -- 172.21.15.67:0/565069781 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7f0714037020 con 0x7f07200a0b50 2024-09-15T01:54:31.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.400+0000 7f07127fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.400+0000 7f07127fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.402 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 msgr2=0x7f070403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 0x7f070403aba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f071c007fd0 tx=0x7f071c007f00 comp rx=0 tx=0).stop 2024-09-15T01:54:31.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 10 monclient: shutdown 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 msgr2=0x7f07200970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0714009360 tx=0x7f0714032000 comp rx=0 tx=0).stop 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 shutdown_connections 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07040386f0 0x7f070403aba0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0720136e40 0x7f0720097600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07200a0b50 0x7f07200970c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 --2- 172.21.15.67:0/565069781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f072009eea0 0x7f072009b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 >> 172.21.15.67:0/565069781 conn(0x7f0720071330 msgr2=0x7f072009e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 shutdown_connections 2024-09-15T01:54:31.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.402+0000 7f072855d640 1 -- 172.21.15.67:0/565069781 wait complete. 2024-09-15T01:54:31.413 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush link host1 root=bar 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.488+0000 7fc222f00640 10 monclient: get_monmap_and_config 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.488+0000 7fc222f00640 10 monclient: build_initial_monmap 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.488+0000 7fc222f00640 10 monclient: monmap: 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:31.490055+0000 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:31.490055+0000 2024-09-15T01:54:31.491 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:31.492 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.489+0000 7fc222f00640 1 Processor -- start 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.489+0000 7fc222f00640 1 -- start start 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.489+0000 7fc222f00640 10 monclient: init 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 0x7fc21c069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc21c09eea0 0x7fc21c09f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-c con 0x7fc21c09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 0x7fc21c131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c131b10 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c06f290 con 0x7fc21c131b10 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c09f7b0 con 0x7fc21c09eea0 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c06a270 con 0x7fc21c0a0b50 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc222f00640 10 monclient(hunting): authenticate will time out at 903.959218s 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc21c09eea0 0x7fc21c09f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc220c9c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 msgr2=0x7fc21c069340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc220c9c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 0x7fc21c069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:31.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc21c09eea0 0x7fc21c09f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48506/0 (socket says 172.21.15.67:48506) 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 -- 172.21.15.67:0/2337720631 learned_addr learned my addr 172.21.15.67:0/2337720631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): get_auth_request con 0x7fc21c09eea0 auth_method 0 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc22149d640 1 --2- 172.21.15.67:0/2337720631 >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 0x7fc21c131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient(hunting): handle_auth_done global_id 5140 payload 274 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 -- 172.21.15.67:0/2337720631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 msgr2=0x7fc21c069340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2337720631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 0x7fc21c069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 -- 172.21.15.67:0/2337720631 >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 msgr2=0x7fc21c131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2337720631 >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 0x7fc21c131f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 monclient: found mon.noname-c 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc21bfff640 1 -- 172.21.15.67:0/2337720631 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c0a0f20 con 0x7fc21c09eea0 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc22149d640 10 monclient: get_auth_request con 0x7fc21c131b10 auth_method 0 2024-09-15T01:54:31.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.490+0000 7fc22149d640 1 --2- 172.21.15.67:0/2337720631 >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 0x7fc21c131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21bfff640 1 -- 172.21.15.67:0/2337720631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c09eea0 msgr2=0x7fc21c09f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2337720631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c09eea0 0x7fc21c09f270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc21c06a480 tx=0x7fc2100091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9cb22f7c7a02185 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 1 -- 172.21.15.67:0/2337720631 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210032440 con 0x7fc21c09eea0 2024-09-15T01:54:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: dump: 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 1 -- 172.21.15.67:0/2337720631 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2100325a0 con 0x7fc21c09eea0 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc222f00640 5 monclient: authenticate success, global_id 5140 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc222f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc222f00640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 1 -- 172.21.15.67:0/2337720631 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210032980 con 0x7fc21c09eea0 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc222f00640 4 set_mon_vals no callback set 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.493+0000 7fc21b7fe640 10 monclient: dump: 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.505 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.506 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.507 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 10 monclient: shutdown 2024-09-15T01:54:31.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 -- 172.21.15.67:0/2337720631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c09eea0 msgr2=0x7fc21c09f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 --2- 172.21.15.67:0/2337720631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c09eea0 0x7fc21c09f270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc21c06a480 tx=0x7fc2100091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 -- 172.21.15.67:0/2337720631 shutdown_connections 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 --2- 172.21.15.67:0/2337720631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c09eea0 0x7fc21c09f270 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 --2- 172.21.15.67:0/2337720631 >> v2:172.21.15.17:3300/0 conn(0x7fc21c131b10 0x7fc21c131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 --2- 172.21.15.67:0/2337720631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c0a0b50 0x7fc21c069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 -- 172.21.15.67:0/2337720631 >> 172.21.15.67:0/2337720631 conn(0x7fc21c071290 msgr2=0x7fc21c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 -- 172.21.15.67:0/2337720631 shutdown_connections 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 1 -- 172.21.15.67:0/2337720631 wait complete. 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 10 monclient: build_initial_monmap 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.494+0000 7fc222f00640 10 monclient: monmap: 2024-09-15T01:54:31.508 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 Processor -- start 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 -- start start 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient: init 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 0x7fc21c09a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-c con 0x7fc21c09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c0a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc21c131b10 0x7fc21c11b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c131b10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c141f40 con 0x7fc21c131b10 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c11e270 con 0x7fc21c0a0b50 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c11dd20 con 0x7fc21c09eea0 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc222f00640 10 monclient(hunting): authenticate will time out at 903.964816s 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc220c9c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 msgr2=0x7fc21c09a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc220c9c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 0x7fc21c09a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48520/0 (socket says 172.21.15.67:48520) 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 1 -- 172.21.15.67:0/2838791742 learned_addr learned my addr 172.21.15.67:0/2838791742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 10 monclient(hunting): get_auth_request con 0x7fc21c0a0b50 auth_method 0 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.495+0000 7fc21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:31.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc22149d640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc21c131b10 0x7fc21c11b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc22149d640 10 monclient(hunting): get_auth_request con 0x7fc21c131b10 auth_method 0 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc22149d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc22149d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc22149d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient(hunting): handle_auth_done global_id 5143 payload 824 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 -- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 msgr2=0x7fc21c09a790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 0x7fc21c09a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 -- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc21c131b10 msgr2=0x7fc21c11b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc21c131b10 0x7fc21c11b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 monclient: found mon.noname-b 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:31.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.496+0000 7fc21bfff640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc210009450 con 0x7fc21c0a0b50 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc21bfff640 1 -- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 msgr2=0x7fc21c130b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc21bfff640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc210009940 tx=0x7fc210032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210000d10 con 0x7fc21c0a0b50 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: dump: 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.513 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210031c70 con 0x7fc21c0a0b50 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 5 monclient: authenticate success, global_id 5143 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 10 monclient: _renew_subs 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c11cdb0 con 0x7fc21c0a0b50 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21003b400 con 0x7fc21c0a0b50 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc218ff9640 10 monclient: dump: 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 10 monclient: _renew_subs 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc21affd640 4 set_mon_vals no callback set 2024-09-15T01:54:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.498+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c1364b0 con 0x7fc21c0a0b50 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc222f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc222f00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc210039040 con 0x7fc21c0a0b50 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc222f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c04bf60 con 0x7fc21c0a0b50 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc218ff9640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 0x7fc20803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.499+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6197+0+0 (secure 0 0 0) 0x7fc21006bf50 con 0x7fc21c0a0b50 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.500+0000 7fc220c9c640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 0x7fc20803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.500+0000 7fc220c9c640 10 monclient: get_auth_request con 0x7fc2080386f0 auth_method 0 2024-09-15T01:54:31.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.502+0000 7fc220c9c640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 0x7fc20803aba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0038f0 tx=0x7fc20c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.504+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc210039ad0 con 0x7fc21c0a0b50 2024-09-15T01:54:31.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.504+0000 7fc218ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:31.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.504+0000 7fc218ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:31.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.670+0000 7fc222f00640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.670+0000 7fc222f00640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.670+0000 7fc222f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.670+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fc21c06f000 con 0x7fc21c0a0b50 2024-09-15T01:54:31.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.671+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc21003f070 con 0x7fc21c0a0b50 2024-09-15T01:54:31.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.671+0000 7fc218ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.671+0000 7fc218ff9640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.774+0000 7fc222f00640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.774+0000 7fc222f00640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.774+0000 7fc222f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.774+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fc21c1310d0 con 0x7fc21c0a0b50 2024-09-15T01:54:31.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.776+0000 7fc218ff9640 1 -- 172.21.15.67:0/2838791742 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc210037030 con 0x7fc21c0a0b50 2024-09-15T01:54:31.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.776+0000 7fc218ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T01:54:31.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.776+0000 7fc218ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.778 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T01:54:31.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.777+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 msgr2=0x7fc20803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.777+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 0x7fc20803aba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0038f0 tx=0x7fc20c006d80 comp rx=0 tx=0).stop 2024-09-15T01:54:31.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.777+0000 7fc222f00640 10 monclient: shutdown 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 msgr2=0x7fc21c130b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc210009940 tx=0x7fc210032000 comp rx=0 tx=0).stop 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 shutdown_connections 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc2080386f0 0x7fc20803aba0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc21c131b10 0x7fc21c11b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21c0a0b50 0x7fc21c130b90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 --2- 172.21.15.67:0/2838791742 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc21c09eea0 0x7fc21c09a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 >> 172.21.15.67:0/2838791742 conn(0x7fc21c071290 msgr2=0x7fc21c134dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 shutdown_connections 2024-09-15T01:54:31.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.778+0000 7fc222f00640 1 -- 172.21.15.67:0/2838791742 wait complete. 2024-09-15T01:54:31.789 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush link host2 root=bar 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 10 monclient: build_initial_monmap 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 10 monclient: monmap: 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.866 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:31.865257+0000 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:31.865257+0000 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:31.867 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 1 Processor -- start 2024-09-15T01:54:31.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 1 -- start start 2024-09-15T01:54:31.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.864+0000 7f7d32f9d640 10 monclient: init 2024-09-15T01:54:31.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:31.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:54:31.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d2c098140 0x7f7d2c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d2c098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d2c0a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 0x7f7d2c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2c098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c0a0b80 con 0x7f7d2c0a0370 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c0a1730 con 0x7f7d2c098140 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0a00b0 con 0x7f7d2c098c20 2024-09-15T01:54:31.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d32f9d640 10 monclient(hunting): authenticate will time out at 904.334387s 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d2c098140 0x7f7d2c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d3153a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 msgr2=0x7f7d2c1390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d2c098140 0x7f7d2c1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48532/0 (socket says 172.21.15.67:48532) 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d3153a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 0x7f7d2c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 -- 172.21.15.67:0/1858817845 learned_addr learned my addr 172.21.15.67:0/1858817845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): get_auth_request con 0x7f7d2c098140 auth_method 0 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:31.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient(hunting): handle_auth_done global_id 5146 payload 274 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 -- 172.21.15.67:0/1858817845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 msgr2=0x7f7d2c1390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 --2- 172.21.15.67:0/1858817845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 0x7f7d2c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 -- 172.21.15.67:0/1858817845 >> v1:172.21.15.17:6789/0 conn(0x7f7d2c0a0370 legacy=0x7f7d2c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient: _finish_hunting 0 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 monclient: found mon.noname-c 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.865+0000 7f7d30d39640 1 -- 172.21.15.67:0/1858817845 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c071390 con 0x7f7d2c098140 2024-09-15T01:54:31.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.868+0000 7f7d30d39640 1 -- 172.21.15.67:0/1858817845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c098140 msgr2=0x7f7d2c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.868+0000 7f7d30d39640 1 --2- 172.21.15.67:0/1858817845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c098140 0x7f7d2c1321e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009f20 tx=0x7f7d24009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f80dd246ca35ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.870+0000 7f7d2b7fe640 1 -- 172.21.15.67:0/1858817845 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24007610 con 0x7f7d2c098140 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: dump: 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.880 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 5 monclient: authenticate success, global_id 5146 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 1 -- 172.21.15.67:0/1858817845 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2402d050 con 0x7f7d2c098140 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 1 -- 172.21.15.67:0/1858817845 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240079a0 con 0x7f7d2c098140 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 4 set_mon_vals no callback set 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:31.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d2b7fe640 10 monclient: dump: 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 10 monclient: shutdown 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 -- 172.21.15.67:0/1858817845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c098140 msgr2=0x7f7d2c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/1858817845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c098140 0x7f7d2c1321e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009f20 tx=0x7f7d24009840 comp rx=0 tx=0).stop 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 -- 172.21.15.67:0/1858817845 shutdown_connections 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/1858817845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098c20 0x7f7d2c1390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/1858817845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c098140 0x7f7d2c1321e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 -- 172.21.15.67:0/1858817845 >> 172.21.15.67:0/1858817845 conn(0x7f7d2c0a5780 msgr2=0x7f7d2c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 -- 172.21.15.67:0/1858817845 shutdown_connections 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.871+0000 7f7d32f9d640 1 -- 172.21.15.67:0/1858817845 wait complete. 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient: build_initial_monmap 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient: monmap: 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 1 Processor -- start 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 1 -- start start 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient: init 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 0x7f7d2c12ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.872+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d2c098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:31.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d2c098c20 0x7f7d2c12d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d2c098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2c0a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c133dd0 con 0x7f7d2c098c20 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c133fe0 con 0x7f7d2c0a0370 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1342c0 con 0x7f7d2c098140 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d32f9d640 10 monclient(hunting): authenticate will time out at 904.342058s 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d30d39640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 msgr2=0x7f7d2c12ccf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d30d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 0x7f7d2c12ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48546/0 (socket says 172.21.15.67:48546) 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 -- 172.21.15.67:0/2495143412 learned_addr learned my addr 172.21.15.67:0/2495143412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): get_auth_request con 0x7f7d2c0a0370 auth_method 0 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d2bfff640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d2c098c20 0x7f7d2c12d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d2bfff640 10 monclient(hunting): get_auth_request con 0x7f7d2c098c20 auth_method 0 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient(hunting): handle_auth_done global_id 5149 payload 824 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 -- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 msgr2=0x7f7d2c12ccf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 0x7f7d2c12ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 -- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d2c098c20 msgr2=0x7f7d2c12d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d2c098c20 0x7f7d2c12d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient: _finish_hunting 0 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 monclient: found mon.noname-b 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.873+0000 7f7d3153a640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24009ae0 con 0x7f7d2c0a0370 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d3153a640 1 -- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 msgr2=0x7f7d2c12ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d3153a640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b4c0 tx=0x7f7d1c006f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c003d10 con 0x7f7d2c0a0370 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: dump: 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.888 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c003e70 con 0x7f7d2c0a0370 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 5 monclient: authenticate success, global_id 5149 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 10 monclient: _renew_subs 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c1330f0 con 0x7f7d2c0a0370 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c006380 con 0x7f7d2c0a0370 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d28ff9640 10 monclient: dump: 2024-09-15T01:54:31.889 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 10 monclient: _renew_subs 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d2affd640 4 set_mon_vals no callback set 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.875+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c131ae0 con 0x7f7d2c0a0370 2024-09-15T01:54:31.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.876+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7d1c01a3b0 con 0x7f7d2c0a0370 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.876+0000 7f7d32f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.876+0000 7f7d32f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.876+0000 7f7d32f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.876+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c04bf60 con 0x7f7d2c0a0370 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.877+0000 7f7d28ff9640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 0x7f7d1803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.877+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f7d1c04b520 con 0x7f7d2c0a0370 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.877+0000 7f7d30d39640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 0x7f7d1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.877+0000 7f7d30d39640 10 monclient: get_auth_request con 0x7f7d180386f0 auth_method 0 2024-09-15T01:54:31.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.879+0000 7f7d30d39640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 0x7f7d1803aba0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d24030000 tx=0x7f7d24036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:31.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:31.881+0000 7f7d3153a640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b4c0 tx=0x7f7d1c006f30 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f7d1c017820 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:54:32.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.709+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7d1c017820 con 0x7f7d2c0a0370 2024-09-15T01:54:32.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.709+0000 7f7d28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:32.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.709+0000 7f7d28ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:32.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.883+0000 7f7d32f9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:32.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.883+0000 7f7d32f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:32.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.883+0000 7f7d32f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:32.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:32.883+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7d2c0990f0 con 0x7f7d2c0a0370 2024-09-15T01:54:33.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.298+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7d1c0175c0 con 0x7f7d2c0a0370 2024-09-15T01:54:33.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.298+0000 7f7d28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.298+0000 7f7d28ff9640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.405+0000 7f7d32f9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.405+0000 7f7d32f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.405+0000 7f7d32f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.405+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7d2c09f7f0 con 0x7f7d2c0a0370 2024-09-15T01:54:33.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.406+0000 7f7d28ff9640 1 -- 172.21.15.67:0/2495143412 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7d1c016030 con 0x7f7d2c0a0370 2024-09-15T01:54:33.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.406+0000 7f7d28ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.406+0000 7f7d28ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.408 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 msgr2=0x7f7d1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 0x7f7d1803aba0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d24030000 tx=0x7f7d24036000 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 10 monclient: shutdown 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 msgr2=0x7f7d2c12ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b4c0 tx=0x7f7d1c006f30 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 shutdown_connections 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7d180386f0 0x7f7d1803aba0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d2c098c20 0x7f7d2c12d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2c0a0370 0x7f7d2c12ff90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 --2- 172.21.15.67:0/2495143412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7d2c098140 0x7f7d2c12ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 >> 172.21.15.67:0/2495143412 conn(0x7f7d2c0a5780 msgr2=0x7f7d2c09b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 shutdown_connections 2024-09-15T01:54:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.408+0000 7f7d32f9d640 1 -- 172.21.15.67:0/2495143412 wait complete. 2024-09-15T01:54:33.419 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush link host2 root=bar 2024-09-15T01:54:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 10 monclient: build_initial_monmap 2024-09-15T01:54:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 10 monclient: monmap: 2024-09-15T01:54:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:33.495328+0000 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:33.495328+0000 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:33.497 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:33.498 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 1 Processor -- start 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 1 -- start start 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.494+0000 7fd77b35d640 10 monclient: init 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd774098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7740a0370 0x7fd774098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7740a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd774098a50 0x7fd7741390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-a con 0x7fd774098a50 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd774098ef0 con 0x7fd774098140 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7740a0b80 con 0x7fd774098a50 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7741328f0 con 0x7fd7740a0370 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd77b35d640 10 monclient(hunting): authenticate will time out at 905.964455s 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7740a0370 0x7fd774098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7740a0370 0x7fd774098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48552/0 (socket says 172.21.15.67:48552) 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 -- 172.21.15.67:0/2114788763 learned_addr learned my addr 172.21.15.67:0/2114788763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7790f9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd774098140 0x7fd7741321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60994/0 (socket says 172.21.15.67:60994) 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): get_auth_request con 0x7fd7740a0370 auth_method 0 2024-09-15T01:54:33.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient(hunting): handle_auth_done global_id 5152 payload 274 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7798fa640 1 --2- 172.21.15.67:0/2114788763 >> v2:172.21.15.17:3300/0 conn(0x7fd774098a50 0x7fd7741390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 -- 172.21.15.67:0/2114788763 >> v2:172.21.15.17:3300/0 conn(0x7fd774098a50 msgr2=0x7fd7741390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 --2- 172.21.15.67:0/2114788763 >> v2:172.21.15.17:3300/0 conn(0x7fd774098a50 0x7fd7741390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 -- 172.21.15.67:0/2114788763 >> v1:172.21.15.67:6789/0 conn(0x7fd774098140 legacy=0x7fd7741321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 monclient: found mon.noname-c 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:33.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.495+0000 7fd7788f8640 1 -- 172.21.15.67:0/2114788763 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740a00b0 con 0x7fd7740a0370 2024-09-15T01:54:33.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd7788f8640 1 -- 172.21.15.67:0/2114788763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 msgr2=0x7fd774098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:33.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd7788f8640 1 --2- 172.21.15.67:0/2114788763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd774098510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd7740717a0 tx=0x7fd7640091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99b7ad376cbde1c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 1 -- 172.21.15.67:0/2114788763 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764032440 con 0x7fd7740a0370 2024-09-15T01:54:33.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: dump: 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: _finish_auth 0 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 1 -- 172.21.15.67:0/2114788763 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7640325a0 con 0x7fd7740a0370 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd77b35d640 5 monclient: authenticate success, global_id 5152 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd77b35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd77b35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 1 -- 172.21.15.67:0/2114788763 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764032940 con 0x7fd7740a0370 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd77b35d640 4 set_mon_vals no callback set 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.498+0000 7fd76bfff640 10 monclient: dump: 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.511 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 10 monclient: shutdown 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 -- 172.21.15.67:0/2114788763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 msgr2=0x7fd774098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 --2- 172.21.15.67:0/2114788763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd774098510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd7740717a0 tx=0x7fd7640091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 -- 172.21.15.67:0/2114788763 shutdown_connections 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 --2- 172.21.15.67:0/2114788763 >> v2:172.21.15.17:3300/0 conn(0x7fd774098a50 0x7fd7741390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 --2- 172.21.15.67:0/2114788763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd774098510 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 -- 172.21.15.67:0/2114788763 >> 172.21.15.67:0/2114788763 conn(0x7fd7740a5780 msgr2=0x7fd7740a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 -- 172.21.15.67:0/2114788763 shutdown_connections 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 1 -- 172.21.15.67:0/2114788763 wait complete. 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.499+0000 7fd77b35d640 10 monclient: build_initial_monmap 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 10 monclient: monmap: 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:33.513 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 1 Processor -- start 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 1 -- start start 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 10 monclient: init 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 0x7fd77412f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.500+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-c con 0x7fd774098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 0x7fd77412f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-a con 0x7fd774098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7740a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd774099090 con 0x7fd774098a50 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77409f500 con 0x7fd7740a0370 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd774131d90 con 0x7fd774098140 2024-09-15T01:54:33.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd77b35d640 10 monclient(hunting): authenticate will time out at 905.970039s 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7790f9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 msgr2=0x7fd77412f0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7790f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 0x7fd77412f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48556/0 (socket says 172.21.15.67:48556) 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 -- 172.21.15.67:0/278829533 learned_addr learned my addr 172.21.15.67:0/278829533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): get_auth_request con 0x7fd7740a0370 auth_method 0 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:33.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 0x7fd77412f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 10 monclient(hunting): get_auth_request con 0x7fd774098a50 auth_method 0 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient(hunting): handle_auth_done global_id 5155 payload 824 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 -- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 msgr2=0x7fd77412f0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 0x7fd77412f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 -- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 msgr2=0x7fd77412f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 0x7fd77412f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient: _finish_hunting 0 2024-09-15T01:54:33.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 monclient: found mon.noname-b 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7798fa640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764009450 con 0x7fd7740a0370 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.501+0000 7fd7788f8640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 0x7fd77412f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7798fa640 1 -- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 msgr2=0x7fd77412fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7798fa640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00ceb0 tx=0x7fd76c007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00e610 con 0x7fd7740a0370 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: dump: 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.518 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c00e770 con 0x7fd7740a0370 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd77b35d640 5 monclient: authenticate success, global_id 5155 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd77b35d640 10 monclient: _renew_subs 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd77b35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77409c140 con 0x7fd7740a0370 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00eb30 con 0x7fd7740a0370 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd7697fa640 10 monclient: dump: 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.520 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.503+0000 7fd76b7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 10 monclient: _renew_subs 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7741317d0 con 0x7fd7740a0370 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd76c009030 con 0x7fd7740a0370 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.504+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77404bf60 con 0x7fd7740a0370 2024-09-15T01:54:33.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.505+0000 7fd7697fa640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 0x7fd75803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.505+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7fd76c012070 con 0x7fd7740a0370 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.505+0000 7fd7790f9640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 0x7fd75803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.505+0000 7fd7790f9640 10 monclient: get_auth_request con 0x7fd7580386f0 auth_method 0 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.507+0000 7fd7790f9640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 0x7fd75803aba0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd770004880 tx=0x7fd770007ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.509+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd76c0153d0 con 0x7fd7740a0370 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.509+0000 7fd7697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.509+0000 7fd7697fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:33.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.679+0000 7fd77b35d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.679+0000 7fd77b35d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.679+0000 7fd77b35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.679+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fd774071390 con 0x7fd7740a0370 2024-09-15T01:54:33.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.681+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (secure 0 0 0) 0x7fd76c014330 con 0x7fd7740a0370 2024-09-15T01:54:33.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.681+0000 7fd7697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.681+0000 7fd7697fa640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.787+0000 7fd77b35d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.787+0000 7fd77b35d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.787+0000 7fd77b35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.787+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fd77406f000 con 0x7fd7740a0370 2024-09-15T01:54:33.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.788+0000 7fd7697fa640 1 -- 172.21.15.67:0/278829533 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (secure 0 0 0) 0x7fd76c0410a0 con 0x7fd7740a0370 2024-09-15T01:54:33.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.788+0000 7fd7697fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T01:54:33.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.788+0000 7fd7697fa640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.790 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 msgr2=0x7fd75803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 0x7fd75803aba0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd770004880 tx=0x7fd770007ac0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 10 monclient: shutdown 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 msgr2=0x7fd77412fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00ceb0 tx=0x7fd76c007870 comp rx=0 tx=0).stop 2024-09-15T01:54:33.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 shutdown_connections 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7580386f0 0x7fd75803aba0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd774098a50 0x7fd77412f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7740a0370 0x7fd77412fb70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 --2- 172.21.15.67:0/278829533 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd774098140 0x7fd77412f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 >> 172.21.15.67:0/278829533 conn(0x7fd7740a5780 msgr2=0x7fd77412dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 shutdown_connections 2024-09-15T01:54:33.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.790+0000 7fd77b35d640 1 -- 172.21.15.67:0/278829533 wait complete. 2024-09-15T01:54:33.802 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:33.802 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c osd.7 2024-09-15T01:54:33.802 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 2 2024-09-15T01:54:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.873+0000 7f164c220640 10 monclient: get_monmap_and_config 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.873+0000 7f164c220640 10 monclient: build_initial_monmap 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.873+0000 7f164c220640 10 monclient: monmap: 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:33.874660+0000 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:33.874660+0000 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:33.876 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:33.877 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 Processor -- start 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 -- start start 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient: init 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T01:54:33.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1644098370 0x7f1644132350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): picked mon.noname-c con 0x7f1644098370 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1644098740 0x7f1644098b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): picked mon.noname-a con 0x7f1644098740 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f16440692a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): picked mon.noname-b con 0x7f1644068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16440a0160 con 0x7f1644098740 2024-09-15T01:54:33.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1644096fc0 con 0x7f1644098370 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440a17b0 con 0x7f1644068e70 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164c220640 10 monclient(hunting): authenticate will time out at 906.343881s 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f1649fbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1644098370 0x7f1644132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1644098370 0x7f1644132350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48564/0 (socket says 172.21.15.67:48564) 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.874+0000 7f164a7bd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 msgr2=0x7f16440692a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 -- 172.21.15.67:0/3294323877 learned_addr learned my addr 172.21.15.67:0/3294323877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): get_auth_request con 0x7f1644098370 auth_method 0 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f164a7bd640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f16440692a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 1 --2- 172.21.15.67:0/3294323877 >> v2:172.21.15.17:3300/0 conn(0x7f1644098740 0x7f1644098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): get_auth_request con 0x7f1644098740 auth_method 0 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f16497bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient(hunting): handle_auth_done global_id 5158 payload 274 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 -- 172.21.15.67:0/3294323877 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 msgr2=0x7f16440692a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f16440692a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 -- 172.21.15.67:0/3294323877 >> v2:172.21.15.17:3300/0 conn(0x7f1644098740 msgr2=0x7f1644098b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 --2- 172.21.15.67:0/3294323877 >> v2:172.21.15.17:3300/0 conn(0x7f1644098740 0x7f1644098b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient: _finish_hunting 0 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 monclient: found mon.noname-c 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.875+0000 7f1649fbc640 1 -- 172.21.15.67:0/3294323877 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164409fef0 con 0x7f1644098370 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1649fbc640 1 -- 172.21.15.67:0/3294323877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098370 msgr2=0x7f1644132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1649fbc640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098370 0x7f1644132350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f164406a170 tx=0x7f1634009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0afcfdcaa334160 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 1 -- 172.21.15.67:0/3294323877 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634007610 con 0x7f1644098370 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: dump: 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.886 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: _finish_auth 0 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 5 monclient: authenticate success, global_id 5158 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 1 -- 172.21.15.67:0/3294323877 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163402d050 con 0x7f1644098370 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 4 set_mon_vals no callback set 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 1 -- 172.21.15.67:0/3294323877 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16340079e0 con 0x7f1644098370 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f1648fba640 10 monclient: dump: 2024-09-15T01:54:33.887 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 10 monclient: shutdown 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 -- 172.21.15.67:0/3294323877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098370 msgr2=0x7f1644132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098370 0x7f1644132350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f164406a170 tx=0x7f1634009840 comp rx=0 tx=0).stop 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 -- 172.21.15.67:0/3294323877 shutdown_connections 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f16440692a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 --2- 172.21.15.67:0/3294323877 >> v2:172.21.15.17:3300/0 conn(0x7f1644098740 0x7f1644098b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 --2- 172.21.15.67:0/3294323877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098370 0x7f1644132350 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 -- 172.21.15.67:0/3294323877 >> 172.21.15.67:0/3294323877 conn(0x7f1644071150 msgr2=0x7f16440a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 -- 172.21.15.67:0/3294323877 shutdown_connections 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.878+0000 7f164c220640 1 -- 172.21.15.67:0/3294323877 wait complete. 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.879+0000 7f164c220640 10 monclient: build_initial_monmap 2024-09-15T01:54:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.879+0000 7f164c220640 10 monclient: monmap: 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:33.890 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.879+0000 7f164c220640 1 Processor -- start 2024-09-15T01:54:33.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.879+0000 7f164c220640 1 -- start start 2024-09-15T01:54:33.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.879+0000 7f164c220640 10 monclient: init 2024-09-15T01:54:33.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f164409e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): picked mon.noname-c con 0x7f1644068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1644098370 0x7f1644130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): picked mon.noname-a con 0x7f1644098370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): picked mon.noname-b con 0x7f1644098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164411df00 con 0x7f1644098370 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164411dc20 con 0x7f1644098740 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164411e1e0 con 0x7f1644068e70 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164c220640 10 monclient(hunting): authenticate will time out at 906.349173s 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f1649fbc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 msgr2=0x7f164409e330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48578/0 (socket says 172.21.15.67:48578) 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 -- 172.21.15.67:0/442679621 learned_addr learned my addr 172.21.15.67:0/442679621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): get_auth_request con 0x7f1644098740 auth_method 0 2024-09-15T01:54:33.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f1649fbc640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f164409e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient(hunting): handle_auth_done global_id 5161 payload 824 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f16497bb640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1644098370 0x7f1644130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 -- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 msgr2=0x7f164409e330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f164409e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 -- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1644098370 msgr2=0x7f1644130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1644098370 0x7f1644130b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:33.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 monclient: found mon.noname-b 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.880+0000 7f164a7bd640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009ae0 con 0x7f1644098740 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164a7bd640 1 -- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 msgr2=0x7f16441410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164a7bd640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f163c00b4e0 tx=0x7f163c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c003d40 con 0x7f1644098740 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:33.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 10 monclient: dump: 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:33.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c006150 con 0x7f1644098740 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164c220640 5 monclient: authenticate success, global_id 5161 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164c220640 10 monclient: _renew_subs 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.882+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164411c7e0 con 0x7f1644098740 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1641ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c006510 con 0x7f1644098740 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:33.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1641ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1641ffb640 10 monclient: dump: 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f164c220640 10 monclient: _renew_subs 2024-09-15T01:54:33.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f1643fff640 4 set_mon_vals no callback set 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f164c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.883+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164411ced0 con 0x7f1644098740 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f163c01b3b0 con 0x7f1644098740 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f1641ffb640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 0x7f163003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f164c220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f164c220640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f164c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164404bf60 con 0x7f1644098740 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f163c04b680 con 0x7f1644098740 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f1649fbc640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 0x7f163003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.884+0000 7f1649fbc640 10 monclient: get_auth_request con 0x7f16300386f0 auth_method 0 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.886+0000 7f1649fbc640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 0x7f163003aba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1634033000 tx=0x7f1634036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:33.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.888+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f163c017030 con 0x7f1644098740 2024-09-15T01:54:33.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.888+0000 7f1641ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:33.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:33.888+0000 7f1641ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:34.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.053+0000 7f164c220640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.053+0000 7f164c220640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.053+0000 7f164c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.053+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f164406f0b0 con 0x7f1644098740 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.054+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f163c018800 con 0x7f1644098740 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.054+0000 7f1641ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.054+0000 7f1641ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:34.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f164c220640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f164c220640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f164c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f164411bd40 con 0x7f1644098740 2024-09-15T01:54:34.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f1641ffb640 1 -- 172.21.15.67:0/442679621 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f163c023360 con 0x7f1644098740 2024-09-15T01:54:34.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f1641ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.151+0000 7f1641ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.153+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 msgr2=0x7f163003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.153+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 0x7f163003aba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1634033000 tx=0x7f1634036000 comp rx=0 tx=0).stop 2024-09-15T01:54:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 10 monclient: shutdown 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 msgr2=0x7f16441410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f163c00b4e0 tx=0x7f163c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 shutdown_connections 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16300386f0 0x7f163003aba0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1644098370 0x7f1644130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1644098740 0x7f16441410b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 --2- 172.21.15.67:0/442679621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1644068e70 0x7f164409e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 >> 172.21.15.67:0/442679621 conn(0x7f1644071150 msgr2=0x7f16440a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 shutdown_connections 2024-09-15T01:54:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.154+0000 7f164c220640 1 -- 172.21.15.67:0/442679621 wait complete. 2024-09-15T01:54:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c host1 2024-09-15T01:54:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 2 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.237+0000 7f5e5666f640 10 monclient: get_monmap_and_config 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.237+0000 7f5e5666f640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.237+0000 7f5e5666f640 10 monclient: monmap: 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:34.238730+0000 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:34.238730+0000 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:34.240 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:34.241 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:34.241 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:34.241 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:34.241 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 Processor -- start 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 -- start start 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient: init 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:54:34.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e50130e00 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 0x7f5e501315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e501311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5e50136f10 0x7f5e501392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-a con 0x7f5e50136f10 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5009ff20 con 0x7f5e50130e00 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e501320b0 con 0x7f5e50136f10 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500a17b0 con 0x7f5e501311d0 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e5666f640 10 monclient(hunting): authenticate will time out at 906.707856s 2024-09-15T01:54:34.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.238+0000 7f5e4f7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 msgr2=0x7f5e501315c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e4f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 0x7f5e501315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5e50136f10 0x7f5e501392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e4ffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5e50130e00 0x7f5e50132430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:39066/0 (socket says 172.21.15.67:39066) 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e4ffff640 1 -- 172.21.15.67:0/3743604681 learned_addr learned my addr 172.21.15.67:0/3743604681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 --2- 172.21.15.67:0/3743604681 >> v2:172.21.15.17:3300/0 conn(0x7f5e50136f10 0x7f5e501392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43216/0 (socket says 172.21.15.67:43216) 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): get_auth_request con 0x7f5e50136f10 auth_method 0 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient(hunting): handle_auth_done global_id 44547 payload 274 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 -- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 msgr2=0x7f5e501315c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 --2- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 0x7f5e501315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 -- 172.21.15.67:0/3743604681 >> v1:172.21.15.17:6789/0 conn(0x7f5e50130e00 legacy=0x7f5e50132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 monclient: found mon.noname-a 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.239+0000 7f5e54c0c640 1 -- 172.21.15.67:0/3743604681 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500a00f0 con 0x7f5e50136f10 2024-09-15T01:54:34.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e54c0c640 1 -- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50136f10 msgr2=0x7f5e501392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e54c0c640 1 --2- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50136f10 0x7f5e501392f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009e60 tx=0x7f5e48009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1858f83290e4826 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 1 -- 172.21.15.67:0/3743604681 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48007840 con 0x7f5e50136f10 2024-09-15T01:54:34.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 10 monclient: dump: 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.253 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 1 -- 172.21.15.67:0/3743604681 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4802d050 con 0x7f5e50136f10 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e4effd640 1 -- 172.21.15.67:0/3743604681 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48007bd0 con 0x7f5e50136f10 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.242+0000 7f5e5666f640 5 monclient: authenticate success, global_id 44547 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 4 set_mon_vals no callback set 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e4effd640 10 monclient: dump: 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.254 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 10 monclient: shutdown 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 -- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50136f10 msgr2=0x7f5e501392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 --2- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50136f10 0x7f5e501392f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009e60 tx=0x7f5e48009780 comp rx=0 tx=0).stop 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 -- 172.21.15.67:0/3743604681 shutdown_connections 2024-09-15T01:54:34.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 --2- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50136f10 0x7f5e501392f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 --2- 172.21.15.67:0/3743604681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e501311d0 0x7f5e501315c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 -- 172.21.15.67:0/3743604681 >> 172.21.15.67:0/3743604681 conn(0x7f5e500710f0 msgr2=0x7f5e500a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 -- 172.21.15.67:0/3743604681 shutdown_connections 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.243+0000 7f5e5666f640 1 -- 172.21.15.67:0/3743604681 wait complete. 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient: monmap: 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.256 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 1 Processor -- start 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 1 -- start start 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient: init 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 0x7f5e50097bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.244+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-a con 0x7f5e50130e00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e501311d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 0x7f5e50098630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): picked mon.noname-c con 0x7f5e50136f10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5009b350 con 0x7f5e50130e00 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5009b070 con 0x7f5e501311d0 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5009b630 con 0x7f5e50136f10 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e5666f640 10 monclient(hunting): authenticate will time out at 906.714065s 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e54c0c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 msgr2=0x7f5e50098630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e54c0c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 0x7f5e50098630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48592/0 (socket says 172.21.15.67:48592) 2024-09-15T01:54:34.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 -- 172.21.15.67:0/4011771034 learned_addr learned my addr 172.21.15.67:0/4011771034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 0x7f5e50097bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f5e501311d0 auth_method 0 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): get_auth_request con 0x7f5e50130e00 auth_method 0 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient(hunting): handle_auth_done global_id 5164 payload 824 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 -- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 msgr2=0x7f5e50098630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 0x7f5e50098630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 -- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 msgr2=0x7f5e50097bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 0x7f5e50097bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 monclient: found mon.noname-b 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4f7fe640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48009a20 con 0x7f5e501311d0 2024-09-15T01:54:34.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.245+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 0x7f5e50097bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e4f7fe640 1 -- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 msgr2=0x7f5e500980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e4f7fe640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400b600 tx=0x7f5e440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44015070 con 0x7f5e501311d0 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 10 monclient: dump: 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.261 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44003d90 con 0x7f5e501311d0 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e5666f640 5 monclient: authenticate success, global_id 5164 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e5666f640 10 monclient: _renew_subs 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44006380 con 0x7f5e501311d0 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e5666f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e50099540 con 0x7f5e501311d0 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.247+0000 7f5e4e7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e37fff640 10 monclient: dump: 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.263 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e5666f640 10 monclient: _renew_subs 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e5666f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.248+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5009bd30 con 0x7f5e501311d0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5e44010440 con 0x7f5e501311d0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e5666f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e5666f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e5666f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5004bf60 con 0x7f5e501311d0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e37fff640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 0x7f5e3c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f5e4404bad0 con 0x7f5e501311d0 2024-09-15T01:54:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 0x7f5e3c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.249+0000 7f5e4ffff640 10 monclient: get_auth_request con 0x7f5e3c0386f0 auth_method 0 2024-09-15T01:54:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.251+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 0x7f5e3c03aba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007fd0 tx=0x7f5e40007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.253+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5e44018ab0 con 0x7f5e501311d0 2024-09-15T01:54:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.253+0000 7f5e37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.253+0000 7f5e37fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:34.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.418+0000 7f5e5666f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.418+0000 7f5e5666f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.418+0000 7f5e5666f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.418+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5e500a14d0 con 0x7f5e501311d0 2024-09-15T01:54:34.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.419+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f5e44018690 con 0x7f5e501311d0 2024-09-15T01:54:34.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.419+0000 7f5e37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.419+0000 7f5e37fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:34.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.516+0000 7f5e5666f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.516+0000 7f5e5666f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.516+0000 7f5e5666f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.516+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5e5009b9d0 con 0x7f5e501311d0 2024-09-15T01:54:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.517+0000 7f5e37fff640 1 -- 172.21.15.67:0/4011771034 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f5e44018690 con 0x7f5e501311d0 2024-09-15T01:54:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.517+0000 7f5e37fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.517+0000 7f5e37fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:34.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 msgr2=0x7f5e3c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 0x7f5e3c03aba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007fd0 tx=0x7f5e40007f00 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 10 monclient: shutdown 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 msgr2=0x7f5e500980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400b600 tx=0x7f5e440091a0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 shutdown_connections 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e3c0386f0 0x7f5e3c03aba0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e50136f10 0x7f5e50098630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e501311d0 0x7f5e500980f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 --2- 172.21.15.67:0/4011771034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e50130e00 0x7f5e50097bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 >> 172.21.15.67:0/4011771034 conn(0x7f5e500710f0 msgr2=0x7f5e500a3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 shutdown_connections 2024-09-15T01:54:34.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.519+0000 7f5e5666f640 1 -- 172.21.15.67:0/4011771034 wait complete. 2024-09-15T01:54:34.531 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:34.531 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c osd.9 2024-09-15T01:54:34.531 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 2 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.602+0000 7f2df6494640 10 monclient: get_monmap_and_config 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.602+0000 7f2df6494640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.602+0000 7f2df6494640 10 monclient: monmap: 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:34.604098+0000 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:34.604098+0000 2024-09-15T01:54:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:34.606 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.603+0000 7f2df6494640 1 Processor -- start 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.603+0000 7f2df6494640 1 -- start start 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.603+0000 7f2df6494640 10 monclient: init 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2df0130e00 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2df01311d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2df01315f0 0x7f2df013a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-a con 0x7f2df01315f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:34.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df009ff20 con 0x7f2df01311d0 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df00a00f0 con 0x7f2df0130e00 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df01320b0 con 0x7f2df01315f0 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df6494640 10 monclient(hunting): authenticate will time out at 907.073361s 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2deffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2df0130e00 0x7f2df0132430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60998/0 (socket says 172.21.15.67:60998) 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2deffff640 1 -- 172.21.15.67:0/742675542 learned_addr learned my addr 172.21.15.67:0/742675542 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 1 --2- 172.21.15.67:0/742675542 >> v2:172.21.15.17:3300/0 conn(0x7f2df01315f0 0x7f2df013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): get_auth_request con 0x7f2df01315f0 auth_method 0 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient(hunting): handle_auth_done global_id 44553 payload 274 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 1 -- 172.21.15.67:0/742675542 >> v1:172.21.15.67:6789/0 conn(0x7f2df0130e00 legacy=0x7f2df0132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 1 -- 172.21.15.67:0/742675542 >> v1:172.21.15.17:6789/0 conn(0x7f2df01311d0 legacy=0x7f2df0136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 1 monclient: found mon.noname-a 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:34.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.604+0000 7f2df4a31640 1 -- 172.21.15.67:0/742675542 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df00a14d0 con 0x7f2df01315f0 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.607+0000 7f2df4a31640 1 -- 172.21.15.67:0/742675542 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01315f0 msgr2=0x7f2df013a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.607+0000 7f2df4a31640 1 --2- 172.21.15.67:0/742675542 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01315f0 0x7f2df013a640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009e60 tx=0x7f2de8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea672fd58f51d6ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 1 -- 172.21.15.67:0/742675542 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8007630 con 0x7f2df01315f0 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: dump: 2024-09-15T01:54:34.617 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 1 -- 172.21.15.67:0/742675542 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de802d050 con 0x7f2df01315f0 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 1 -- 172.21.15.67:0/742675542 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de80079c0 con 0x7f2df01315f0 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2deeffd640 10 monclient: dump: 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.619 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 5 monclient: authenticate success, global_id 44553 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 4 set_mon_vals no callback set 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 10 monclient: shutdown 2024-09-15T01:54:34.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 1 -- 172.21.15.67:0/742675542 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01315f0 msgr2=0x7f2df013a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 1 --2- 172.21.15.67:0/742675542 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01315f0 0x7f2df013a640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009e60 tx=0x7f2de8009780 comp rx=0 tx=0).stop 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 1 -- 172.21.15.67:0/742675542 shutdown_connections 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 1 --2- 172.21.15.67:0/742675542 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01315f0 0x7f2df013a640 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.608+0000 7f2df6494640 1 -- 172.21.15.67:0/742675542 >> 172.21.15.67:0/742675542 conn(0x7f2df00710f0 msgr2=0x7f2df00a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.609+0000 7f2df6494640 1 -- 172.21.15.67:0/742675542 shutdown_connections 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.609+0000 7f2df6494640 1 -- 172.21.15.67:0/742675542 wait complete. 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.609+0000 7f2df6494640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.609+0000 7f2df6494640 10 monclient: monmap: 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:34.621 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.609+0000 7f2df6494640 1 Processor -- start 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 -- start start 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient: init 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 0x7f2df0141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-c con 0x7f2df0130e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01311d0 0x7f2df012ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-a con 0x7f2df01311d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): picked mon.noname-b con 0x7f2df01315f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df011c420 con 0x7f2df01311d0 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df011c670 con 0x7f2df01315f0 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00980f0 con 0x7f2df0130e00 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df6494640 10 monclient(hunting): authenticate will time out at 907.079527s 2024-09-15T01:54:34.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2deffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 msgr2=0x7f2df0141230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48598/0 (socket says 172.21.15.67:48598) 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 -- 172.21.15.67:0/2309736341 learned_addr learned my addr 172.21.15.67:0/2309736341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): get_auth_request con 0x7f2df01315f0 auth_method 0 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2deffff640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 0x7f2df0141230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01311d0 0x7f2df012ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): get_auth_request con 0x7f2df01311d0 auth_method 0 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2def7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient(hunting): handle_auth_done global_id 5167 payload 824 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 -- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 msgr2=0x7f2df0141230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 0x7f2df0141230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 -- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01311d0 msgr2=0x7f2df012ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01311d0 0x7f2df012ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 monclient: found mon.noname-b 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.610+0000 7f2df4a31640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de8009a20 con 0x7f2df01315f0 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df4a31640 1 -- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 msgr2=0x7f2df009c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df4a31640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009e60 tx=0x7f2de8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8007ec0 con 0x7f2df01315f0 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: dump: 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.626 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 5 monclient: authenticate success, global_id 5167 2024-09-15T01:54:34.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de8040070 con 0x7f2df01315f0 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 10 monclient: _renew_subs 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df009e760 con 0x7f2df01315f0 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8031d10 con 0x7f2df01315f0 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dd7fff640 10 monclient: dump: 2024-09-15T01:54:34.628 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 10 monclient: _renew_subs 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2dee7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.613+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df00a08a0 con 0x7f2df01315f0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2de8042030 con 0x7f2df01315f0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2df6494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2df6494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2df6494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df011c7b0 con 0x7f2df01315f0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2dd7fff640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 0x7f2ddc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f2de806d7b0 con 0x7f2df01315f0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2deffff640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 0x7f2ddc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.614+0000 7f2deffff640 10 monclient: get_auth_request con 0x7f2ddc0386f0 auth_method 0 2024-09-15T01:54:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.617+0000 7f2deffff640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 0x7f2ddc03aba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2de00047d0 tx=0x7f2de0007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.618+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2df011c7b0 con 0x7f2df01315f0 2024-09-15T01:54:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.618+0000 7f2dd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.618+0000 7f2dd7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:34.748 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:54:34.749 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T01:54:34.749 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:54:34.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:54:34.752 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:54:34.752 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:54:34.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.784+0000 7f2df6494640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.784+0000 7f2df6494640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.784+0000 7f2df6494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.784+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f2df011c1d0 con 0x7f2df01315f0 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.785+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f2de803ab80 con 0x7f2df01315f0 2024-09-15T01:54:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.785+0000 7f2dd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.785+0000 7f2dd7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.858+0000 7f11ed080640 10 monclient: get_monmap_and_config 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.858+0000 7f11ed080640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.858+0000 7f11ed080640 10 monclient: monmap: 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:34.859652+0000 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:34.859652+0000 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:34.862 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 Processor -- start 2024-09-15T01:54:34.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 -- start start 2024-09-15T01:54:34.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient: init 2024-09-15T01:54:34.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-b con 0x7f11e80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11e809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11e8069c90 0x7f11e8139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-c con 0x7f11e8069c90 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e80a5b50 con 0x7f11e809ccf0 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e80a18b0 con 0x7f11e8069c90 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.859+0000 7f11ed080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e809c5f0 con 0x7f11e80a5780 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11ed080640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11ed080640 10 monclient(hunting): authenticate will time out at 934.856012s 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:32928/0 (socket says 172.21.15.17:32928) 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 -- 172.21.15.17:0/563339129 learned_addr learned my addr 172.21.15.17:0/563339129 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 10 monclient(hunting): get_auth_request con 0x7f11e80a5780 auth_method 0 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 --2- 172.21.15.17:0/563339129 >> v2:172.21.15.67:3300/0 conn(0x7f11e8069c90 0x7f11e8139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): get_auth_request con 0x7f11e8069c90 auth_method 0 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 -- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 msgr2=0x7f11e809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 -- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 msgr2=0x7f11e809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient(hunting): handle_auth_done global_id 5170 payload 274 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 -- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 msgr2=0x7f11e809bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 -- 172.21.15.17:0/563339129 >> v1:172.21.15.67:6789/0 conn(0x7f11e809ccf0 legacy=0x7f11e8069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 monclient: found mon.noname-c 2024-09-15T01:54:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.860+0000 7f11e759e640 1 -- 172.21.15.17:0/563339129 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e809d210 con 0x7f11e8069c90 2024-09-15T01:54:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e759e640 1 -- 172.21.15.17:0/563339129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e8069c90 msgr2=0x7f11e8139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e759e640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e8069c90 0x7f11e8139880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009e60 tx=0x7f11e0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be91f66bc75db97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 1 -- 172.21.15.17:0/563339129 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0007760 con 0x7f11e8069c90 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: dump: 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.874 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 1 -- 172.21.15.17:0/563339129 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e002d050 con 0x7f11e8069c90 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 1 -- 172.21.15.17:0/563339129 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0007af0 con 0x7f11e8069c90 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11ed080640 5 monclient: authenticate success, global_id 5170 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11ed080640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11ed080640 4 set_mon_vals no callback set 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.863+0000 7f11e5d9b640 10 monclient: dump: 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 10 monclient: shutdown 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 -- 172.21.15.17:0/563339129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e8069c90 msgr2=0x7f11e8139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e8069c90 0x7f11e8139880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009e60 tx=0x7f11e0009780 comp rx=0 tx=0).stop 2024-09-15T01:54:34.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 -- 172.21.15.17:0/563339129 shutdown_connections 2024-09-15T01:54:34.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e8069c90 0x7f11e8139880 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 --2- 172.21.15.17:0/563339129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e80a5780 0x7f11e809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 -- 172.21.15.17:0/563339129 >> 172.21.15.17:0/563339129 conn(0x7f11e8070e10 msgr2=0x7f11e80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 -- 172.21.15.17:0/563339129 shutdown_connections 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.864+0000 7f11ed080640 1 -- 172.21.15.17:0/563339129 wait complete. 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 10 monclient: monmap: 2024-09-15T01:54:34.880 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 1 Processor -- start 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 1 -- start start 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 10 monclient: init 2024-09-15T01:54:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.865+0000 7f11ed080640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-c con 0x7f11e8069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 0x7f11e8131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-a con 0x7f11e809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 0x7f11e811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): picked mon.noname-b con 0x7f11e80a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80a0990 con 0x7f11e809ccf0 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e809b5f0 con 0x7f11e80a5780 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8142100 con 0x7f11e8069c90 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:32934/0 (socket says 172.21.15.17:32934) 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11ed080640 10 monclient(hunting): authenticate will time out at 934.862208s 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 -- 172.21.15.17:0/53185236 learned_addr learned my addr 172.21.15.17:0/53185236 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 10 monclient(hunting): get_auth_request con 0x7f11e8069c90 auth_method 0 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 0x7f11e8131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 0x7f11e811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): get_auth_request con 0x7f11e809ccf0 auth_method 0 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): get_auth_request con 0x7f11e80a5780 auth_method 0 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 msgr2=0x7f11e8141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 msgr2=0x7f11e8141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient(hunting): handle_auth_done global_id 5173 payload 824 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 msgr2=0x7f11e8141290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 msgr2=0x7f11e8131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 0x7f11e8131b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 monclient: found mon.noname-b 2024-09-15T01:54:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e759e640 1 -- 172.21.15.17:0/53185236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0009a20 con 0x7f11e80a5780 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.866+0000 7f11e659c640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 0x7f11e8131b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.868+0000 7f11e759e640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 msgr2=0x7f11e811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.868+0000 7f11e759e640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 0x7f11e811b6d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f11e0031000 tx=0x7f11e0037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0031e20 con 0x7f11e80a5780 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: dump: 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.886 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0045460 con 0x7f11e80a5780 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0045820 con 0x7f11e80a5780 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 5 monclient: authenticate success, global_id 5173 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ceffd640 10 monclient: dump: 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 10 monclient: _renew_subs 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e811ec20 con 0x7f11e80a5780 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11e559a640 4 set_mon_vals no callback set 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 10 monclient: _renew_subs 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.869+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81417d0 con 0x7f11e80a5780 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ed080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f11e0045c50 con 0x7f11e80a5780 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ed080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ed080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e80724a0 con 0x7f11e80a5780 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ceffd640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 0x7f11d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f11e006dd00 con 0x7f11e80a5780 2024-09-15T01:54:34.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 0x7f11d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.870+0000 7f11e6d9d640 10 monclient: get_auth_request con 0x7f11d4038740 auth_method 0 2024-09-15T01:54:34.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.872+0000 7f11e6d9d640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 0x7f11d403abf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006f30 tx=0x7f11d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.874+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f11e00305a0 con 0x7f11e80a5780 2024-09-15T01:54:34.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.874+0000 7f11ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:34.874+0000 7f11ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:34.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2df6494640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2df6494640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2df6494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f2df011c1d0 con 0x7f2df01315f0 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2dd7fff640 1 -- 172.21.15.67:0/2309736341 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f2de803a3a0 con 0x7f2df01315f0 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2dd7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.883+0000 7f2dd7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 msgr2=0x7f2ddc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 0x7f2ddc03aba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2de00047d0 tx=0x7f2de0007b80 comp rx=0 tx=0).stop 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 10 monclient: shutdown 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 msgr2=0x7f2df009c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009e60 tx=0x7f2de8037000 comp rx=0 tx=0).stop 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 shutdown_connections 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ddc0386f0 0x7f2ddc03aba0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df01311d0 0x7f2df012ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df01315f0 0x7f2df009c880 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 --2- 172.21.15.67:0/2309736341 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0130e00 0x7f2df0141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 >> 172.21.15.67:0/2309736341 conn(0x7f2df00710f0 msgr2=0x7f2df0132ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 shutdown_connections 2024-09-15T01:54:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.885+0000 7f2df6494640 1 -- 172.21.15.67:0/2309736341 wait complete. 2024-09-15T01:54:34.897 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:34.897 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c host2 2024-09-15T01:54:34.897 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 2 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.968+0000 7f5f7a100640 10 monclient: get_monmap_and_config 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.968+0000 7f5f7a100640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.968+0000 7f5f7a100640 10 monclient: monmap: 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:34.970095+0000 2024-09-15T01:54:34.971 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:34.970095+0000 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:34.972 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.969+0000 7f5f7a100640 1 Processor -- start 2024-09-15T01:54:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.969+0000 7f5f7a100640 1 -- start start 2024-09-15T01:54:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.969+0000 7f5f7a100640 10 monclient: init 2024-09-15T01:54:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5f74130e00 0x7f5f74132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-a con 0x7f5f74130e00 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-b con 0x7f5f741311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f74068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7409ff20 con 0x7f5f74068dc0 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f741320b0 con 0x7f5f74130e00 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740a17b0 con 0x7f5f741311d0 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f7a100640 10 monclient(hunting): authenticate will time out at 907.439239s 2024-09-15T01:54:34.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5f74130e00 0x7f5f74132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f73fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5f74068dc0 0x7f5f74069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:39084/0 (socket says 172.21.15.67:39084) 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f73fff640 1 -- 172.21.15.67:0/2869127839 learned_addr learned my addr 172.21.15.67:0/2869127839 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 10 monclient(hunting): get_auth_request con 0x7f5f741311d0 auth_method 0 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): get_auth_request con 0x7f5f74130e00 auth_method 0 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 1 -- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 msgr2=0x7f5f741315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 1 -- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 msgr2=0x7f5f741315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f72ffd640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient(hunting): handle_auth_done global_id 44562 payload 274 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 -- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 msgr2=0x7f5f741315c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 -- 172.21.15.67:0/2869127839 >> v1:172.21.15.17:6789/0 conn(0x7f5f74068dc0 legacy=0x7f5f74069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:34.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:34.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:34.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.970+0000 7f5f737fe640 1 -- 172.21.15.67:0/2869127839 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740a00f0 con 0x7f5f74130e00 2024-09-15T01:54:34.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f737fe640 1 -- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74130e00 msgr2=0x7f5f74132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f737fe640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74130e00 0x7f5f74132430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009850 tx=0x7f5f640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb524dbde398a48 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 1 -- 172.21.15.67:0/2869127839 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6402adf0 con 0x7f5f74130e00 2024-09-15T01:54:34.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: dump: 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 1 -- 172.21.15.67:0/2869127839 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f64032440 con 0x7f5f74130e00 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f7a100640 5 monclient: authenticate success, global_id 44562 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f7a100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f7a100640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 1 -- 172.21.15.67:0/2869127839 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f640327e0 con 0x7f5f74130e00 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f7a100640 4 set_mon_vals no callback set 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.973+0000 7f5f727fc640 10 monclient: dump: 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.985 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 10 monclient: shutdown 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 -- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74130e00 msgr2=0x7f5f74132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74130e00 0x7f5f74132430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009850 tx=0x7f5f640090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 -- 172.21.15.67:0/2869127839 shutdown_connections 2024-09-15T01:54:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f741311d0 0x7f5f741315c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2869127839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74130e00 0x7f5f74132430 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 -- 172.21.15.67:0/2869127839 >> 172.21.15.67:0/2869127839 conn(0x7f5f740710f0 msgr2=0x7f5f740a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 -- 172.21.15.67:0/2869127839 shutdown_connections 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 1 -- 172.21.15.67:0/2869127839 wait complete. 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 10 monclient: build_initial_monmap 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.974+0000 7f5f7a100640 10 monclient: monmap: 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.987 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 Processor -- start 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 -- start start 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient: init 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-a con 0x7f5f74068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-c con 0x7f5f74130e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:34.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 0x7f5f7409cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): picked mon.noname-b con 0x7f5f741311d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740a4a80 con 0x7f5f74068dc0 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740a0820 con 0x7f5f741311d0 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f741364e0 con 0x7f5f74130e00 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.975+0000 7f5f7a100640 10 monclient(hunting): authenticate will time out at 907.444868s 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 0x7f5f7409cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43276/0 (socket says 172.21.15.67:43276) 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:60448/0 (socket says 172.21.15.67:60448) 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 1 -- 172.21.15.67:0/2446135393 learned_addr learned my addr 172.21.15.67:0/2446135393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): get_auth_request con 0x7f5f741311d0 auth_method 0 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): get_auth_request con 0x7f5f74068dc0 auth_method 0 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 10 monclient(hunting): get_auth_request con 0x7f5f74130e00 auth_method 0 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:34.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 msgr2=0x7f5f7409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 msgr2=0x7f5f7409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f72ffd640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient(hunting): handle_auth_done global_id 5176 payload 824 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 msgr2=0x7f5f7409c6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:34.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 msgr2=0x7f5f74141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 monclient: found mon.noname-b 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f73fff640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64009390 con 0x7f5f741311d0 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.976+0000 7f5f737fe640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f73fff640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 msgr2=0x7f5f7409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f73fff640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 0x7f5f7409cbe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00bf50 tx=0x7f5f6c009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c014440 con 0x7f5f741311d0 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: dump: 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:34.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c0145e0 con 0x7f5f741311d0 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0149a0 con 0x7f5f741311d0 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f5b7fe640 10 monclient: dump: 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:34.994 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 5 monclient: authenticate success, global_id 5176 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 10 monclient: _renew_subs 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7411bcf0 con 0x7f5f741311d0 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f71ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 10 monclient: _renew_subs 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.978+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7409e180 con 0x7f5f741311d0 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5f6c01d3b0 con 0x7f5f741311d0 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f7a100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f7a100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f7a100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7411c1d0 con 0x7f5f741311d0 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f5b7fe640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 0x7f5f6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f5f6c04c1c0 con 0x7f5f741311d0 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f737fe640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 0x7f5f6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.979+0000 7f5f737fe640 10 monclient: get_auth_request con 0x7f5f60038740 auth_method 0 2024-09-15T01:54:34.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.982+0000 7f5f737fe640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 0x7f5f6003abf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009130 tx=0x7f5f64032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:34.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.984+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5f6c0112a0 con 0x7f5f741311d0 2024-09-15T01:54:34.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.984+0000 7f5f5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:34.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:34.984+0000 7f5f5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:35.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:35.105+0000 7f11ed080640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:35.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:35.105+0000 7f11ed080640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:35.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:35.105+0000 7f11ed080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:35.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:35.105+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f11e81419c0 con 0x7f11e80a5780 2024-09-15T01:54:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:35.149+0000 7f5f7a100640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:35.149+0000 7f5f7a100640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:35.150+0000 7f5f7a100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:35.150+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5f7404bf60 con 0x7f5f741311d0 2024-09-15T01:54:37.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:37.866+0000 7f11cdffb640 10 monclient: tick 2024-09-15T01:54:37.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:37.866+0000 7f11cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:37.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:37.866+0000 7f11cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:37.975+0000 7f5f5a7fc640 10 monclient: tick 2024-09-15T01:54:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:37.975+0000 7f5f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:37.975+0000 7f5f5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:38.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:38.866+0000 7f11cdffb640 10 monclient: tick 2024-09-15T01:54:38.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:38.866+0000 7f11cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:38.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:38.976+0000 7f5f5a7fc640 10 monclient: tick 2024-09-15T01:54:38.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:38.976+0000 7f5f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:39.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:39.866+0000 7f11cdffb640 10 monclient: tick 2024-09-15T01:54:39.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:39.866+0000 7f11cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:39.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:39.976+0000 7f5f5a7fc640 10 monclient: tick 2024-09-15T01:54:39.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:39.976+0000 7f5f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:40.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.066+0000 7f11ceffd640 1 -- 172.21.15.17:0/53185236 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f11e004a070 con 0x7f11e80a5780 2024-09-15T01:54:40.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.066+0000 7f11ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:40.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.066+0000 7f11ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:40.069 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:40.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.067+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f5f6c019070 con 0x7f5f741311d0 2024-09-15T01:54:40.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.067+0000 7f5f5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:40.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.067+0000 7f5f5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:40.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 msgr2=0x7f11d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 0x7f11d403abf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006f30 tx=0x7f11d8008040 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 10 monclient: shutdown 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 msgr2=0x7f11e811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 0x7f11e811b6d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f11e0031000 tx=0x7f11e0037000 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 shutdown_connections 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f11d4038740 0x7f11d403abf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f11e809ccf0 0x7f11e8131b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e80a5780 0x7f11e811b6d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 --2- 172.21.15.17:0/53185236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11e8069c90 0x7f11e8141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 >> 172.21.15.17:0/53185236 conn(0x7f11e8070e10 msgr2=0x7f11e809d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.069+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 shutdown_connections 2024-09-15T01:54:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:40.070+0000 7f11ed080640 1 -- 172.21.15.17:0/53185236 wait complete. 2024-09-15T01:54:40.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f7a100640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:40.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f7a100640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:40.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f7a100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5f740a14d0 con 0x7f5f741311d0 2024-09-15T01:54:40.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f5b7fe640 1 -- 172.21.15.67:0/2446135393 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f5f6c007a90 con 0x7f5f741311d0 2024-09-15T01:54:40.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f5b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:40.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.174+0000 7f5f5b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 msgr2=0x7f5f6003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 0x7f5f6003abf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009130 tx=0x7f5f64032000 comp rx=0 tx=0).stop 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 10 monclient: shutdown 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 msgr2=0x7f5f7409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 0x7f5f7409cbe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00bf50 tx=0x7f5f6c009720 comp rx=0 tx=0).stop 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 shutdown_connections 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f60038740 0x7f5f6003abf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f741311d0 0x7f5f7409cbe0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f74130e00 0x7f5f7409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 --2- 172.21.15.67:0/2446135393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f74068dc0 0x7f5f74141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 >> 172.21.15.67:0/2446135393 conn(0x7f5f740710f0 msgr2=0x7f5f7412fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:40.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 shutdown_connections 2024-09-15T01:54:40.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.176+0000 7f5f7a100640 1 -- 172.21.15.67:0/2446135393 wait complete. 2024-09-15T01:54:40.188 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-15T01:54:40.189 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:40.189 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm host1 foo 2024-09-15T01:54:40.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.264+0000 7f7e12434640 10 monclient: get_monmap_and_config 2024-09-15T01:54:40.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.264+0000 7f7e12434640 10 monclient: build_initial_monmap 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.264+0000 7f7e12434640 10 monclient: monmap: 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:40.265929+0000 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:40.265929+0000 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:40.268 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.265+0000 7f7e12434640 1 Processor -- start 2024-09-15T01:54:40.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.265+0000 7f7e12434640 1 -- start start 2024-09-15T01:54:40.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.265+0000 7f7e12434640 10 monclient: init 2024-09-15T01:54:40.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.265+0000 7f7e12434640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.265+0000 7f7e12434640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e0c0a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e0c09eea0 0x7f7e0c09f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e0c136e40 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c0a61a0 con 0x7f7e0c136e40 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c0a04c0 con 0x7f7e0c0a0b50 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c09f800 con 0x7f7e0c09eea0 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e12434640 10 monclient(hunting): authenticate will time out at 912.735081s 2024-09-15T01:54:40.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e0c09eea0 0x7f7e0c09f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e0c0a0b50 0x7f7e0c069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32772/0 (socket says 172.21.15.67:32772) 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0bfff640 1 -- 172.21.15.67:0/204807561 learned_addr learned my addr 172.21.15.67:0/204807561 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e0c09eea0 auth_method 0 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient(hunting): handle_auth_done global_id 5179 payload 274 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/204807561 >> v1:172.21.15.67:6789/0 conn(0x7f7e0c0a0b50 legacy=0x7f7e0c069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/204807561 >> v1:172.21.15.17:6789/0 conn(0x7f7e0c136e40 legacy=0x7f7e0c139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:40.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:40.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:40.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:40.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.266+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/204807561 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c069a90 con 0x7f7e0c09eea0 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/204807561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 msgr2=0x7f7e0c09f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/204807561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09f2c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009850 tx=0x7f7e000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ecbe5921d6b69a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 1 -- 172.21.15.67:0/204807561 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00032440 con 0x7f7e0c09eea0 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: dump: 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.278 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: _finish_auth 0 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 1 -- 172.21.15.67:0/204807561 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e000325a0 con 0x7f7e0c09eea0 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e12434640 5 monclient: authenticate success, global_id 5179 2024-09-15T01:54:40.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e12434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e12434640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 1 -- 172.21.15.67:0/204807561 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00032940 con 0x7f7e0c09eea0 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e12434640 4 set_mon_vals no callback set 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.269+0000 7f7e0affd640 10 monclient: dump: 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.280 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 10 monclient: shutdown 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 -- 172.21.15.67:0/204807561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 msgr2=0x7f7e0c09f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 --2- 172.21.15.67:0/204807561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09f2c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009850 tx=0x7f7e000090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 -- 172.21.15.67:0/204807561 shutdown_connections 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 --2- 172.21.15.67:0/204807561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09f2c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 -- 172.21.15.67:0/204807561 >> 172.21.15.67:0/204807561 conn(0x7f7e0c071330 msgr2=0x7f7e0c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:40.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 -- 172.21.15.67:0/204807561 shutdown_connections 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 -- 172.21.15.67:0/204807561 wait complete. 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 10 monclient: build_initial_monmap 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 10 monclient: monmap: 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:40.282 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.270+0000 7f7e12434640 1 Processor -- start 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 -- start start 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient: init 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0c09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 0x7f7e0c0970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e0c136e40 0x7f7e0c097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0c136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c0a1260 con 0x7f7e0c136e40 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c0a3370 con 0x7f7e0c09eea0 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c098510 con 0x7f7e0c0a0b50 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e12434640 10 monclient(hunting): authenticate will time out at 912.740537s 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48628/0 (socket says 172.21.15.67:48628) 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 -- 172.21.15.67:0/2828098582 learned_addr learned my addr 172.21.15.67:0/2828098582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): get_auth_request con 0x7f7e0c09eea0 auth_method 0 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e0c136e40 0x7f7e0c097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 0x7f7e0c0970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): get_auth_request con 0x7f7e0c136e40 auth_method 0 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e0c0a0b50 auth_method 0 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e109d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 msgr2=0x7f7e0c0970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 1 -- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 msgr2=0x7f7e0c0970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 0x7f7e0c0970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient(hunting): handle_auth_done global_id 5182 payload 824 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 0x7f7e0c0970c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 -- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 msgr2=0x7f7e0c0970c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e0c0a0b50 0x7f7e0c0970c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 -- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e0c136e40 msgr2=0x7f7e0c097600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e0c136e40 0x7f7e0c097600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 monclient: found mon.noname-b 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.271+0000 7f7e0bfff640 1 -- 172.21.15.67:0/2828098582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009390 con 0x7f7e0c09eea0 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e0bfff640 1 -- 172.21.15.67:0/2828098582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 msgr2=0x7f7e0c09b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e0bfff640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0c09eea0 0x7f7e0c09b3d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c5e0 tx=0x7f7dfc004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc005cb0 con 0x7f7e0c09eea0 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: dump: 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 5 monclient: authenticate success, global_id 5182 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 10 monclient: _renew_subs 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 1 -- 172.21.15.67:0/2828098582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c099160 con 0x7f7e0c09eea0 2024-09-15T01:54:40.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc005e30 con 0x7f7e0c09eea0 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc007340 con 0x7f7e0c09eea0 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7df3fff640 10 monclient: dump: 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.289 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 10 monclient: _renew_subs 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e12434640 1 -- 172.21.15.67:0/2828098582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c098c50 con 0x7f7e0c09eea0 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.274+0000 7f7e0a7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7dfc01b400 con 0x7f7e0c09eea0 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e12434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e12434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e12434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e12434640 1 -- 172.21.15.67:0/2828098582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0c04bf60 con 0x7f7e0c09eea0 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7df3fff640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7df8038740 0x7f7df803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f7dfc04bc30 con 0x7f7e0c09eea0 2024-09-15T01:54:40.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7df8038740 0x7f7df803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.275+0000 7f7e0b7fe640 10 monclient: get_auth_request con 0x7f7df8038740 auth_method 0 2024-09-15T01:54:40.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.278+0000 7f7e0b7fe640 1 --2- 172.21.15.67:0/2828098582 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7df8038740 0x7f7df803abf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7e00031000 tx=0x7f7e00032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.280+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7dfc01b6b0 con 0x7f7e0c09eea0 2024-09-15T01:54:40.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.280+0000 7f7df3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.280+0000 7f7df3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:40.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.449+0000 7f7e12434640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:40.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.449+0000 7f7e12434640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:40.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.449+0000 7f7e12434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.449+0000 7f7e12434640 1 -- 172.21.15.67:0/2828098582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f7e0c09a910 con 0x7f7e0c09eea0 2024-09-15T01:54:40.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.450+0000 7f7df3fff640 1 -- 172.21.15.67:0/2828098582 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v74) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7dfc01a360 con 0x7f7e0c09eea0 2024-09-15T01:54:40.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.450+0000 7f7df3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:40.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.450+0000 7f7df3fff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T01:54:40.453 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T01:54:40.456 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:40.456 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush unlink host1 foo 2024-09-15T01:54:40.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.530+0000 7f2ff651d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:40.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.531+0000 7f2ff651d640 10 monclient: build_initial_monmap 2024-09-15T01:54:40.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.531+0000 7f2ff651d640 10 monclient: monmap: 2024-09-15T01:54:40.533 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:40.532176+0000 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:40.532176+0000 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:40.534 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.531+0000 7f2ff651d640 1 Processor -- start 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.531+0000 7f2ff651d640 1 -- start start 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.531+0000 7f2ff651d640 10 monclient: init 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff00a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 0x7f2ff009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff009eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ff0136e40 0x7f2ff0139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff0136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00a61a0 con 0x7f2ff00a0b50 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff009f800 con 0x7f2ff009eea0 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00a1610 con 0x7f2ff0136e40 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff651d640 10 monclient(hunting): authenticate will time out at 913.001373s 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ff0136e40 0x7f2ff0139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ff0136e40 0x7f2ff0139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57708/0 (socket says 172.21.15.67:57708) 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 -- 172.21.15.67:0/4173650602 learned_addr learned my addr 172.21.15.67:0/4173650602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:40.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): get_auth_request con 0x7f2ff0136e40 auth_method 0 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 1 --2- 172.21.15.67:0/4173650602 >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 0x7f2ff009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request con 0x7f2ff009eea0 auth_method 0 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_done global_id 5185 payload 274 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 -- 172.21.15.67:0/4173650602 >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 msgr2=0x7f2ff009f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 --2- 172.21.15.67:0/4173650602 >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 0x7f2ff009f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 -- 172.21.15.67:0/4173650602 >> v1:172.21.15.17:6789/0 conn(0x7f2ff00a0b50 legacy=0x7f2ff0069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient: _finish_hunting 0 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 monclient: found mon.noname-c 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2ff4aba640 1 -- 172.21.15.67:0/4173650602 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff00a04c0 con 0x7f2ff0136e40 2024-09-15T01:54:40.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.532+0000 7f2fef7fe640 1 --2- 172.21.15.67:0/4173650602 >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 0x7f2ff009f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:40.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff4aba640 1 -- 172.21.15.67:0/4173650602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 msgr2=0x7f2ff0139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff4aba640 1 --2- 172.21.15.67:0/4173650602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff0139220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00a1200 tx=0x7f2fe8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beaf159c36ae2586 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 1 -- 172.21.15.67:0/4173650602 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8007610 con 0x7f2ff0136e40 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: dump: 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.546 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 1 -- 172.21.15.67:0/4173650602 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe802d050 con 0x7f2ff0136e40 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff651d640 5 monclient: authenticate success, global_id 5185 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff651d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 1 -- 172.21.15.67:0/4173650602 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80079a0 con 0x7f2ff0136e40 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2feeffd640 10 monclient: dump: 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.547 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff651d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.535+0000 7f2ff651d640 4 set_mon_vals no callback set 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 10 monclient: shutdown 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 -- 172.21.15.67:0/4173650602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 msgr2=0x7f2ff0139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 --2- 172.21.15.67:0/4173650602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff0139220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00a1200 tx=0x7f2fe8009840 comp rx=0 tx=0).stop 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 -- 172.21.15.67:0/4173650602 shutdown_connections 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 --2- 172.21.15.67:0/4173650602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff0139220 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 --2- 172.21.15.67:0/4173650602 >> v2:172.21.15.17:3300/0 conn(0x7f2ff009eea0 0x7f2ff009f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 -- 172.21.15.67:0/4173650602 >> 172.21.15.67:0/4173650602 conn(0x7f2ff0071330 msgr2=0x7f2ff00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 -- 172.21.15.67:0/4173650602 shutdown_connections 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 1 -- 172.21.15.67:0/4173650602 wait complete. 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.536+0000 7f2ff651d640 10 monclient: build_initial_monmap 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 10 monclient: monmap: 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:40.549 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 1 Processor -- start 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 1 -- start start 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 10 monclient: init 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ff009eea0 0x7f2ff009a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:40.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.537+0000 7f2ff651d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff00a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff0136e40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00a1260 con 0x7f2ff009eea0 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00a50e0 con 0x7f2ff0136e40 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff011db90 con 0x7f2ff00a0b50 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff651d640 10 monclient(hunting): authenticate will time out at 913.007060s 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57710/0 (socket says 172.21.15.67:57710) 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 -- 172.21.15.67:0/1627728488 learned_addr learned my addr 172.21.15.67:0/1627728488 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): get_auth_request con 0x7f2ff0136e40 auth_method 0 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ff009eea0 0x7f2ff009a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): get_auth_request con 0x7f2ff009eea0 auth_method 0 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request con 0x7f2ff00a0b50 auth_method 0 2024-09-15T01:54:40.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 msgr2=0x7f2ff009ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 msgr2=0x7f2ff009ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2fef7fe640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:40.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient(hunting): handle_auth_done global_id 5188 payload 824 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 msgr2=0x7f2ff009ab50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ff009eea0 msgr2=0x7f2ff009a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ff009eea0 0x7f2ff009a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient: _finish_hunting 0 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 monclient: found mon.noname-b 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.538+0000 7f2ff4aba640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8009ae0 con 0x7f2ff0136e40 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2ff4aba640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 msgr2=0x7f2ff009b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2ff4aba640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009ab0 tx=0x7f2fe8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8032cc0 con 0x7f2ff0136e40 2024-09-15T01:54:40.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: dump: 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2ff651d640 5 monclient: authenticate success, global_id 5188 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe8032e40 con 0x7f2ff0136e40 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe803b660 con 0x7f2ff0136e40 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fd7fff640 10 monclient: dump: 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:40.556 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 10 monclient: _renew_subs 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.540+0000 7f2fee7fc640 4 set_mon_vals no callback set 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff011eaa0 con 0x7f2ff0136e40 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 10 monclient: _renew_subs 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.541+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff011c920 con 0x7f2ff0136e40 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2fe8003020 con 0x7f2ff0136e40 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2ff651d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2ff651d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2ff651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff004bf60 con 0x7f2ff0136e40 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2fd7fff640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 0x7f2fdc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f2fe806cb90 con 0x7f2ff0136e40 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2feffff640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 0x7f2fdc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.542+0000 7f2feffff640 10 monclient: get_auth_request con 0x7f2fdc038770 auth_method 0 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.544+0000 7f2feffff640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 0x7f2fdc03ac20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0007fd0 tx=0x7f2fe0007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:40.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.546+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2fe8039ae0 con 0x7f2ff0136e40 2024-09-15T01:54:40.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.546+0000 7f2fd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:40.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.546+0000 7f2fd7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:40.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.715+0000 7f2ff651d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.715+0000 7f2ff651d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.715+0000 7f2ff651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:40.715+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f2ff011dcd0 con 0x7f2ff0136e40 2024-09-15T01:54:41.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.615+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v75) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2fe803f070 con 0x7f2ff0136e40 2024-09-15T01:54:41.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.615+0000 7f2fd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:41.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.615+0000 7f2fd7fff640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T01:54:41.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.721+0000 7f2ff651d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:41.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.721+0000 7f2ff651d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:41.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.721+0000 7f2ff651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:41.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.721+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f2ff006f000 con 0x7f2ff0136e40 2024-09-15T01:54:41.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.722+0000 7f2fd7fff640 1 -- 172.21.15.67:0/1627728488 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2fe8039300 con 0x7f2ff0136e40 2024-09-15T01:54:41.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.722+0000 7f2fd7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:41.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.722+0000 7f2fd7fff640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T01:54:41.724 INFO:tasks.workunit.client.0.smithi067.stderr:item -11 does not appear in that position 2024-09-15T01:54:41.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 msgr2=0x7f2fdc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:41.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 0x7f2fdc03ac20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0007fd0 tx=0x7f2fe0007e10 comp rx=0 tx=0).stop 2024-09-15T01:54:41.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 10 monclient: shutdown 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 msgr2=0x7f2ff009b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009ab0 tx=0x7f2fe8036000 comp rx=0 tx=0).stop 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 shutdown_connections 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2fdc038770 0x7f2fdc03ac20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff0136e40 0x7f2ff009b090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ff00a0b50 0x7f2ff009ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 --2- 172.21.15.67:0/1627728488 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ff009eea0 0x7f2ff009a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 >> 172.21.15.67:0/1627728488 conn(0x7f2ff0071330 msgr2=0x7f2ff009dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 shutdown_connections 2024-09-15T01:54:41.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.724+0000 7f2ff651d640 1 -- 172.21.15.67:0/1627728488 wait complete. 2024-09-15T01:54:41.736 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush unlink host1 foo 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 10 monclient: get_monmap_and_config 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 10 monclient: build_initial_monmap 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 10 monclient: monmap: 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:41.814 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:41.813187+0000 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:41.813187+0000 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:41.815 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 1 Processor -- start 2024-09-15T01:54:41.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 1 -- start start 2024-09-15T01:54:41.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.812+0000 7f052f2a8640 10 monclient: init 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0528068dc0 0x7f0528132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0528068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0528069190 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0528137150 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0528096c10 con 0x7f0528137150 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052809ff20 con 0x7f0528069190 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f052806a2a0 con 0x7f0528068dc0 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052f2a8640 10 monclient(hunting): authenticate will time out at 914.282312s 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052c843640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0528069190 0x7f0528069590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55210/0 (socket says 172.21.15.67:55210) 2024-09-15T01:54:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052c843640 1 -- 172.21.15.67:0/2544840504 learned_addr learned my addr 172.21.15.67:0/2544840504 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 1 --2- 172.21.15.67:0/2544840504 >> v2:172.21.15.17:3300/0 conn(0x7f0528068dc0 0x7f0528132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): get_auth_request con 0x7f0528068dc0 auth_method 0 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.813+0000 7f052d044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 10 monclient(hunting): handle_auth_done global_id 44577 payload 274 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 1 -- 172.21.15.67:0/2544840504 >> v1:172.21.15.67:6789/0 conn(0x7f0528069190 legacy=0x7f0528069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 1 -- 172.21.15.67:0/2544840504 >> v1:172.21.15.17:6789/0 conn(0x7f0528137150 legacy=0x7f0528139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 10 monclient: _finish_hunting 0 2024-09-15T01:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 1 monclient: found mon.noname-a 2024-09-15T01:54:41.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:41.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:41.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.814+0000 7f052d044640 1 -- 172.21.15.67:0/2544840504 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05280a0180 con 0x7f0528068dc0 2024-09-15T01:54:41.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.816+0000 7f052d044640 1 -- 172.21.15.67:0/2544840504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 msgr2=0x7f0528132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.816+0000 7f052d044640 1 --2- 172.21.15.67:0/2544840504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f0528132250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0520009850 tx=0x7f05200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9af3dbd9957835 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 1 -- 172.21.15.67:0/2544840504 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520032440 con 0x7f0528068dc0 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: dump: 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 5 monclient: authenticate success, global_id 44577 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 1 -- 172.21.15.67:0/2544840504 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05200325a0 con 0x7f0528068dc0 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 4 set_mon_vals no callback set 2024-09-15T01:54:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 1 -- 172.21.15.67:0/2544840504 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520032940 con 0x7f0528068dc0 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f0527fff640 10 monclient: dump: 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:41.826 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 10 monclient: shutdown 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 -- 172.21.15.67:0/2544840504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 msgr2=0x7f0528132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 --2- 172.21.15.67:0/2544840504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f0528132250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0520009850 tx=0x7f05200090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 -- 172.21.15.67:0/2544840504 shutdown_connections 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 --2- 172.21.15.67:0/2544840504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f0528132250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 -- 172.21.15.67:0/2544840504 >> 172.21.15.67:0/2544840504 conn(0x7f05280711b0 msgr2=0x7f05280a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 -- 172.21.15.67:0/2544840504 shutdown_connections 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.817+0000 7f052f2a8640 1 -- 172.21.15.67:0/2544840504 wait complete. 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.818+0000 7f052f2a8640 10 monclient: build_initial_monmap 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.818+0000 7f052f2a8640 10 monclient: monmap: 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:41.827 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.818+0000 7f052f2a8640 1 Processor -- start 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.818+0000 7f052f2a8640 1 -- start start 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.818+0000 7f052f2a8640 10 monclient: init 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f05281415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0528068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0528069190 0x7f0528141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f0528069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f0528137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052809b460 con 0x7f0528068dc0 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05280a0890 con 0x7f0528137150 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0528099930 con 0x7f0528069190 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052f2a8640 10 monclient(hunting): authenticate will time out at 914.288340s 2024-09-15T01:54:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57724/0 (socket says 172.21.15.67:57724) 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 -- 172.21.15.67:0/1440217407 learned_addr learned my addr 172.21.15.67:0/1440217407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): get_auth_request con 0x7f0528137150 auth_method 0 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f05281415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052c843640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0528069190 0x7f0528141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 10 monclient(hunting): get_auth_request con 0x7f0528068dc0 auth_method 0 2024-09-15T01:54:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052c843640 10 monclient(hunting): get_auth_request con 0x7f0528069190 auth_method 0 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052c843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052c843640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052c843640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient(hunting): handle_auth_done global_id 5191 payload 824 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 -- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0528069190 msgr2=0x7f0528141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0528069190 0x7f0528141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 -- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 msgr2=0x7f05281415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f05281415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient: _finish_hunting 0 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 monclient: found mon.noname-b 2024-09-15T01:54:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:41.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:41.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d845640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520009390 con 0x7f0528137150 2024-09-15T01:54:41.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.819+0000 7f052d044640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f05281415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052d845640 1 -- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 msgr2=0x7f0528142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052d845640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f051800c810 tx=0x7f0518002620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051800ce00 con 0x7f0528137150 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: dump: 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:41.834 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051800aa60 con 0x7f0528137150 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 5 monclient: authenticate success, global_id 5191 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 10 monclient: _renew_subs 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052811be50 con 0x7f0528137150 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051800ae20 con 0x7f0528137150 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05257fa640 10 monclient: dump: 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 10 monclient: _renew_subs 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052809a500 con 0x7f0528137150 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.822+0000 7f05277fe640 4 set_mon_vals no callback set 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0518012630 con 0x7f0528137150 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052f2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052f2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052f2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052804bf60 con 0x7f0528137150 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f05257fa640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 0x7f051403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f051804c500 con 0x7f0528137150 2024-09-15T01:54:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052d044640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 0x7f051403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.823+0000 7f052d044640 10 monclient: get_auth_request con 0x7f0514038740 auth_method 0 2024-09-15T01:54:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.826+0000 7f052d044640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 0x7f051403abf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0520034000 tx=0x7f0520032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.827+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0518022ae0 con 0x7f0528137150 2024-09-15T01:54:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.827+0000 7f05257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.827+0000 7f05257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:41.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.997+0000 7f052f2a8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:41.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.997+0000 7f052f2a8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:42.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.997+0000 7f052f2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.997+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f0528131e40 con 0x7f0528137150 2024-09-15T01:54:42.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.998+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0518016020 con 0x7f0528137150 2024-09-15T01:54:42.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.998+0000 7f05257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:42.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:41.998+0000 7f05257fa640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-15T01:54:42.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.100+0000 7f052f2a8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:42.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.100+0000 7f052f2a8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:42.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.100+0000 7f052f2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.100+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f05280008d0 con 0x7f0528137150 2024-09-15T01:54:42.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.101+0000 7f05257fa640 1 -- 172.21.15.67:0/1440217407 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0518006b10 con 0x7f0528137150 2024-09-15T01:54:42.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.101+0000 7f05257fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T01:54:42.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.101+0000 7f05257fa640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T01:54:42.104 INFO:tasks.workunit.client.0.smithi067.stderr:item -11 does not appear in that position 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 msgr2=0x7f051403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 0x7f051403abf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0520034000 tx=0x7f0520032000 comp rx=0 tx=0).stop 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 10 monclient: shutdown 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 msgr2=0x7f0528142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f051800c810 tx=0x7f0518002620 comp rx=0 tx=0).stop 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 shutdown_connections 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0514038740 0x7f051403abf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528137150 0x7f0528142040 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0528069190 0x7f0528141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 --2- 172.21.15.67:0/1440217407 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0528068dc0 0x7f05281415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 >> 172.21.15.67:0/1440217407 conn(0x7f05280711b0 msgr2=0x7f05280a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:42.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 shutdown_connections 2024-09-15T01:54:42.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.103+0000 7f052f2a8640 1 -- 172.21.15.67:0/1440217407 wait complete. 2024-09-15T01:54:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c host1 2024-09-15T01:54:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 1 2024-09-15T01:54:42.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 10 monclient: get_monmap_and_config 2024-09-15T01:54:42.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 10 monclient: monmap: 2024-09-15T01:54:42.189 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.189 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:42.188205+0000 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:42.188205+0000 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:42.190 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 1 Processor -- start 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 1 -- start start 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.187+0000 7f97ea817640 10 monclient: init 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97e4098370 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97e4098740 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e4098b10 0x7f97e413a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-c con 0x7f97e4098b10 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40a17b0 con 0x7f97e4098740 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e409feb0 con 0x7f97e4098370 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e4132a70 con 0x7f97e4098b10 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97ea817640 10 monclient(hunting): authenticate will time out at 914.657497s 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e4098b10 0x7f97e413a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97e4098b10 0x7f97e413a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57736/0 (socket says 172.21.15.67:57736) 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 -- 172.21.15.67:0/2640010775 learned_addr learned my addr 172.21.15.67:0/2640010775 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): get_auth_request con 0x7f97e4098b10 auth_method 0 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e3fff640 1 --1- 172.21.15.67:0/2640010775 >> v1:172.21.15.67:6789/0 conn(0x7f97e4098370 0x7f97e4132360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55216/0 (socket says 172.21.15.67:55216) 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient(hunting): handle_auth_done global_id 5194 payload 274 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 -- 172.21.15.67:0/2640010775 >> v1:172.21.15.67:6789/0 conn(0x7f97e4098370 legacy=0x7f97e4132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 -- 172.21.15.67:0/2640010775 >> v1:172.21.15.17:6789/0 conn(0x7f97e4098740 legacy=0x7f97e4136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 monclient: found mon.noname-c 2024-09-15T01:54:42.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.188+0000 7f97e8db4640 1 -- 172.21.15.67:0/2640010775 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4098f00 con 0x7f97e4098b10 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.191+0000 7f97e8db4640 1 -- 172.21.15.67:0/2640010775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 msgr2=0x7f97e413a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.191+0000 7f97e8db4640 1 --2- 172.21.15.67:0/2640010775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e413a5d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97e40a0780 tx=0x7f97dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b757df62f6d0194 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 1 -- 172.21.15.67:0/2640010775 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc007610 con 0x7f97e4098b10 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: dump: 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 1 -- 172.21.15.67:0/2640010775 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc02d050 con 0x7f97e4098b10 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 5 monclient: authenticate success, global_id 5194 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 1 -- 172.21.15.67:0/2640010775 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0079a0 con 0x7f97e4098b10 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 4 set_mon_vals no callback set 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97e37fe640 10 monclient: dump: 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 10 monclient: shutdown 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 -- 172.21.15.67:0/2640010775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 msgr2=0x7f97e413a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 --2- 172.21.15.67:0/2640010775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e413a5d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97e40a0780 tx=0x7f97dc009840 comp rx=0 tx=0).stop 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 -- 172.21.15.67:0/2640010775 shutdown_connections 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 --2- 172.21.15.67:0/2640010775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e413a5d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 -- 172.21.15.67:0/2640010775 >> 172.21.15.67:0/2640010775 conn(0x7f97e40a59a0 msgr2=0x7f97e40a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 -- 172.21.15.67:0/2640010775 shutdown_connections 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.192+0000 7f97ea817640 1 -- 172.21.15.67:0/2640010775 wait complete. 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient: monmap: 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:42.203 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 Processor -- start 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 -- start start 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient: init 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e4098370 0x7f97e409aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-c con 0x7f97e4098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e4098740 0x7f97e41309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-a con 0x7f97e4098740 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): picked mon.noname-b con 0x7f97e4098b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4141e30 con 0x7f97e4098740 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40a3870 con 0x7f97e4098b10 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e411cd00 con 0x7f97e4098370 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.193+0000 7f97ea817640 10 monclient(hunting): authenticate will time out at 914.662992s 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57752/0 (socket says 172.21.15.67:57752) 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 -- 172.21.15.67:0/2094695083 learned_addr learned my addr 172.21.15.67:0/2094695083 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): get_auth_request con 0x7f97e4098b10 auth_method 0 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e3fff640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e4098370 0x7f97e409aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e3fff640 10 monclient(hunting): get_auth_request con 0x7f97e4098370 auth_method 0 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97dbfff640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e4098740 0x7f97e41309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97dbfff640 10 monclient(hunting): get_auth_request con 0x7f97e4098740 auth_method 0 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient(hunting): handle_auth_done global_id 5197 payload 824 2024-09-15T01:54:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 -- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e4098370 msgr2=0x7f97e409aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e4098370 0x7f97e409aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 -- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e4098740 msgr2=0x7f97e41309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e4098740 0x7f97e41309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 monclient: found mon.noname-b 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.194+0000 7f97e8db4640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc009ae0 con 0x7f97e4098b10 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e8db4640 1 -- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 msgr2=0x7f97e4130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e8db4640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009ab0 tx=0x7f97dc036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc031cd0 con 0x7f97e4098b10 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: dump: 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 5 monclient: authenticate success, global_id 5197 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc031e30 con 0x7f97e4098b10 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 10 monclient: _renew_subs 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e411dbf0 con 0x7f97e4098b10 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc03c650 con 0x7f97e4098b10 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e0ff9640 10 monclient: dump: 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.209 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 10 monclient: _renew_subs 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e409ba50 con 0x7f97e4098b10 2024-09-15T01:54:42.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.196+0000 7f97e2ffd640 4 set_mon_vals no callback set 2024-09-15T01:54:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.197+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97dc03ca40 con 0x7f97e4098b10 2024-09-15T01:54:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97e0ff9640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 0x7f97c403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97ea817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97ea817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97ea817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e40a0890 con 0x7f97e4098b10 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f97dc06d460 con 0x7f97e4098b10 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97e3fff640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 0x7f97c403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.198+0000 7f97e3fff640 10 monclient: get_auth_request con 0x7f97c4038740 auth_method 0 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.200+0000 7f97e3fff640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 0x7f97c403abf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97d4007110 tx=0x7f97d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.202+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f97e40a0890 con 0x7f97e4098b10 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.202+0000 7f97e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.202+0000 7f97e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.366+0000 7f97ea817640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.366+0000 7f97ea817640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.366+0000 7f97ea817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.366+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f97e404bf60 con 0x7f97e4098b10 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.367+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f97dc04a070 con 0x7f97e4098b10 2024-09-15T01:54:42.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.367+0000 7f97e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:42.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.367+0000 7f97e0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:42.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97ea817640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97ea817640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97ea817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f97e40a14d0 con 0x7f97e4098b10 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97e0ff9640 1 -- 172.21.15.67:0/2094695083 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f97dc040030 con 0x7f97e4098b10 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97e0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:42.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.465+0000 7f97e0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 msgr2=0x7f97c403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 0x7f97c403abf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97d4007110 tx=0x7f97d4009040 comp rx=0 tx=0).stop 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 10 monclient: shutdown 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 msgr2=0x7f97e4130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009ab0 tx=0x7f97dc036000 comp rx=0 tx=0).stop 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 shutdown_connections 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97c4038740 0x7f97c403abf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97e4098740 0x7f97e41309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97e4098b10 0x7f97e4130f10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 --2- 172.21.15.67:0/2094695083 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97e4098370 0x7f97e409aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 >> 172.21.15.67:0/2094695083 conn(0x7f97e40a59a0 msgr2=0x7f97e409e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:42.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 shutdown_connections 2024-09-15T01:54:42.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.467+0000 7f97ea817640 1 -- 172.21.15.67:0/2094695083 wait complete. 2024-09-15T01:54:42.479 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rm foo 2024-09-15T01:54:42.479 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:42.479 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm foo 2024-09-15T01:54:42.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.553+0000 7fae854a8640 10 monclient: get_monmap_and_config 2024-09-15T01:54:42.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.553+0000 7fae854a8640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.554+0000 7fae854a8640 10 monclient: monmap: 2024-09-15T01:54:42.556 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.556 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:42.555172+0000 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:42.555172+0000 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:42.557 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.554+0000 7fae854a8640 1 Processor -- start 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.554+0000 7fae854a8640 1 -- start start 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.554+0000 7fae854a8640 10 monclient: init 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:54:42.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae800a3850 0x7fae8006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-b con 0x7fae800a3850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fae8006a6e0 0x7fae8009db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-a con 0x7fae8006a6e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae8009e050 0x7fae8009e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-c con 0x7fae8009e050 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8009e9d0 con 0x7fae8006a6e0 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae80069c20 con 0x7fae8009e050 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800696c0 con 0x7fae800a3850 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae854a8640 10 monclient(hunting): authenticate will time out at 915.024142s 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae8009e050 0x7fae8009e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae8009e050 0x7fae8009e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57756/0 (socket says 172.21.15.67:57756) 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 -- 172.21.15.67:0/2311506510 learned_addr learned my addr 172.21.15.67:0/2311506510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): get_auth_request con 0x7fae8009e050 auth_method 0 2024-09-15T01:54:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/2311506510 >> v2:172.21.15.17:3300/0 conn(0x7fae8006a6e0 0x7fae8009db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7effd640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae800a3850 0x7fae8006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient(hunting): handle_auth_done global_id 5200 payload 274 2024-09-15T01:54:42.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 -- 172.21.15.67:0/2311506510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae800a3850 msgr2=0x7fae8006e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae800a3850 0x7fae8006e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 -- 172.21.15.67:0/2311506510 >> v2:172.21.15.17:3300/0 conn(0x7fae8006a6e0 msgr2=0x7fae8009db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/2311506510 >> v2:172.21.15.17:3300/0 conn(0x7fae8006a6e0 0x7fae8009db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 monclient: found mon.noname-c 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.555+0000 7fae7f7fe640 1 -- 172.21.15.67:0/2311506510 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8009f9a0 con 0x7fae8009e050 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7f7fe640 1 -- 172.21.15.67:0/2311506510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8009e050 msgr2=0x7fae8009e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8009e050 0x7fae8009e490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fae8009d730 tx=0x7fae78009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4101a9f4247445cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 1 -- 172.21.15.67:0/2311506510 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78007610 con 0x7fae8009e050 2024-09-15T01:54:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: dump: 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae854a8640 5 monclient: authenticate success, global_id 5200 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae854a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 1 -- 172.21.15.67:0/2311506510 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7802d050 con 0x7fae8009e050 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae854a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae854a8640 4 set_mon_vals no callback set 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 1 -- 172.21.15.67:0/2311506510 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780079a0 con 0x7fae8009e050 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.558+0000 7fae7dffb640 10 monclient: dump: 2024-09-15T01:54:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 10 monclient: shutdown 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 -- 172.21.15.67:0/2311506510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8009e050 msgr2=0x7fae8009e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8009e050 0x7fae8009e490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fae8009d730 tx=0x7fae78009840 comp rx=0 tx=0).stop 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 -- 172.21.15.67:0/2311506510 shutdown_connections 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8009e050 0x7fae8009e490 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 --2- 172.21.15.67:0/2311506510 >> v2:172.21.15.17:3300/0 conn(0x7fae8006a6e0 0x7fae8009db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 --2- 172.21.15.67:0/2311506510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae800a3850 0x7fae8006e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 -- 172.21.15.67:0/2311506510 >> 172.21.15.67:0/2311506510 conn(0x7fae80070ff0 msgr2=0x7fae800a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 -- 172.21.15.67:0/2311506510 shutdown_connections 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.559+0000 7fae854a8640 1 -- 172.21.15.67:0/2311506510 wait complete. 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient: monmap: 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:42.570 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 Processor -- start 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 -- start start 2024-09-15T01:54:42.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient: init 2024-09-15T01:54:42.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8006a6e0 0x7fae80140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-b con 0x7fae8006a6e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 0x7fae8013d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-c con 0x7fae8009e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 0x7fae8013d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): picked mon.noname-a con 0x7fae800a3850 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae80122420 con 0x7fae800a3850 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8011ff00 con 0x7fae8006a6e0 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae80122140 con 0x7fae8009e050 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae854a8640 10 monclient(hunting): authenticate will time out at 915.029890s 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8006a6e0 0x7fae80140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8006a6e0 0x7fae80140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57758/0 (socket says 172.21.15.67:57758) 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 1 -- 172.21.15.67:0/3320325422 learned_addr learned my addr 172.21.15.67:0/3320325422 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 10 monclient(hunting): get_auth_request con 0x7fae8006a6e0 auth_method 0 2024-09-15T01:54:42.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.560+0000 7fae7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 0x7fae8013d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 0x7fae8013d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 10 monclient(hunting): get_auth_request con 0x7fae8009e050 auth_method 0 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 10 monclient(hunting): get_auth_request con 0x7fae800a3850 auth_method 0 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 msgr2=0x7fae8013d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 msgr2=0x7fae8013d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 0x7fae8013d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7f7fe640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 0x7fae8013d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient(hunting): handle_auth_done global_id 5203 payload 824 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 msgr2=0x7fae8013d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 msgr2=0x7fae8013d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:42.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 0x7fae8013d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 msgr2=0x7fae8013d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 0x7fae8013d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fae8009e050 0x7fae8013d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 msgr2=0x7fae8013d9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fae800a3850 0x7fae8013d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 monclient: found mon.noname-b 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.561+0000 7fae7effd640 1 -- 172.21.15.67:0/3320325422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78009ae0 con 0x7fae8006a6e0 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae7effd640 1 -- 172.21.15.67:0/3320325422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8006a6e0 msgr2=0x7fae80140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae7effd640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8006a6e0 0x7fae80140f60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c880 tx=0x7fae68008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6800ce10 con 0x7fae8006a6e0 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: dump: 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.583 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68007a40 con 0x7fae8006a6e0 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68007de0 con 0x7fae8006a6e0 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 5 monclient: authenticate success, global_id 5203 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 10 monclient: _renew_subs 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 1 -- 172.21.15.67:0/3320325422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae80122700 con 0x7fae8006a6e0 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae7d7fa640 4 set_mon_vals no callback set 2024-09-15T01:54:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae76ffd640 10 monclient: dump: 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.585 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 10 monclient: _renew_subs 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.563+0000 7fae854a8640 1 -- 172.21.15.67:0/3320325422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8013d760 con 0x7fae8006a6e0 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fae68011600 con 0x7fae8006a6e0 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae854a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae854a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae854a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae854a8640 1 -- 172.21.15.67:0/3320325422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8004bf60 con 0x7fae8006a6e0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae76ffd640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae60038740 0x7fae6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7fae6804c2e0 con 0x7fae8006a6e0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae60038740 0x7fae6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.564+0000 7fae7e7fc640 10 monclient: get_auth_request con 0x7fae60038740 auth_method 0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.567+0000 7fae7e7fc640 1 --2- 172.21.15.67:0/3320325422 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fae60038740 0x7fae6003abf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fae7000a160 tx=0x7fae70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.569+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fae68017af0 con 0x7fae8006a6e0 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.569+0000 7fae76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.569+0000 7fae76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:42.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.739+0000 7fae854a8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:54:42.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.739+0000 7fae854a8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:54:42.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.739+0000 7fae854a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.739+0000 7fae854a8640 1 -- 172.21.15.67:0/3320325422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fae80069110 con 0x7fae8006a6e0 2024-09-15T01:54:42.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.740+0000 7fae76ffd640 1 -- 172.21.15.67:0/3320325422 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v75) v1 ==== 99+0+0 (secure 0 0 0) 0x7fae6801e020 con 0x7fae8006a6e0 2024-09-15T01:54:42.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.740+0000 7fae76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:54:42.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.740+0000 7fae76ffd640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T01:54:42.742 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T01:54:42.746 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:42.746 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rm bar 2024-09-15T01:54:42.747 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:42.747 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm bar 2024-09-15T01:54:42.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.820+0000 7f57cd43f640 10 monclient: get_monmap_and_config 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.820+0000 7f57cd43f640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.820+0000 7f57cd43f640 10 monclient: monmap: 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:42.821643+0000 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:42.821643+0000 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:42.823 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:42.824 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:42.824 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 Processor -- start 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 -- start start 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient: init 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57c80a0b50 0x7f57c8069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:42.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57c809eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-b con 0x7f57c809f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80a61a0 con 0x7f57c809eea0 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c80a1610 con 0x7f57c80a0b50 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c80a04c0 con 0x7f57c809f2c0 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57cd43f640 10 monclient(hunting): authenticate will time out at 915.290765s 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57c80a0b50 0x7f57c8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57c80a0b50 0x7f57c8069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57770/0 (socket says 172.21.15.67:57770) 2024-09-15T01:54:42.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 1 -- 172.21.15.67:0/3393880149 learned_addr learned my addr 172.21.15.67:0/3393880149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 10 monclient(hunting): get_auth_request con 0x7f57c80a0b50 auth_method 0 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.821+0000 7f57c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 10 monclient(hunting): get_auth_request con 0x7f57c809f2c0 auth_method 0 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient(hunting): handle_auth_done global_id 5206 payload 274 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 1 -- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 msgr2=0x7f57c8132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 1 -- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 msgr2=0x7f57c8132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 1 -- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 msgr2=0x7f57c8132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c77fe640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 1 -- 172.21.15.67:0/3393880149 >> v1:172.21.15.17:6789/0 conn(0x7f57c809eea0 legacy=0x7f57c8131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 1 monclient: found mon.noname-c 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.822+0000 7f57c6ffd640 1 -- 172.21.15.67:0/3393880149 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c80a0f20 con 0x7f57c80a0b50 2024-09-15T01:54:42.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c6ffd640 1 -- 172.21.15.67:0/3393880149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 msgr2=0x7f57c8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c6ffd640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c8069380 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009910 tx=0x7f57b80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f547915c861627f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 1 -- 172.21.15.67:0/3393880149 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b802aea0 con 0x7f57c80a0b50 2024-09-15T01:54:42.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: dump: 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 5 monclient: authenticate success, global_id 5206 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 1 -- 172.21.15.67:0/3393880149 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b8032440 con 0x7f57c80a0b50 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 1 -- 172.21.15.67:0/3393880149 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8032820 con 0x7f57c80a0b50 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 4 set_mon_vals no callback set 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57c67fc640 10 monclient: dump: 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.836 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 10 monclient: shutdown 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 -- 172.21.15.67:0/3393880149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 msgr2=0x7f57c8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c8069380 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009910 tx=0x7f57b80091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 -- 172.21.15.67:0/3393880149 shutdown_connections 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 --2- 172.21.15.67:0/3393880149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c8069380 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 -- 172.21.15.67:0/3393880149 >> 172.21.15.67:0/3393880149 conn(0x7f57c8071330 msgr2=0x7f57c80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 -- 172.21.15.67:0/3393880149 shutdown_connections 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.825+0000 7f57cd43f640 1 -- 172.21.15.67:0/3393880149 wait complete. 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.826+0000 7f57cd43f640 10 monclient: build_initial_monmap 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.826+0000 7f57cd43f640 10 monclient: monmap: 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.826+0000 7f57cd43f640 1 Processor -- start 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.826+0000 7f57cd43f640 1 -- start start 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.826+0000 7f57cd43f640 10 monclient: init 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57c809eea0 0x7f57c81415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-a con 0x7f57c809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-c con 0x7f57c809f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c81324e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): picked mon.noname-b con 0x7f57c80a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c811d690 con 0x7f57c809eea0 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c80a1260 con 0x7f57c80a0b50 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8134d10 con 0x7f57c809f2c0 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57cd43f640 10 monclient(hunting): authenticate will time out at 915.296182s 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c81324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c81324e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57772/0 (socket says 172.21.15.67:57772) 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 -- 172.21.15.67:0/1357188032 learned_addr learned my addr 172.21.15.67:0/1357188032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): get_auth_request con 0x7f57c80a0b50 auth_method 0 2024-09-15T01:54:42.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57bffff640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient(hunting): handle_auth_done global_id 5209 payload 824 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c6ffd640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57c809eea0 0x7f57c81415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 -- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 msgr2=0x7f57c8131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 -- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57c809eea0 msgr2=0x7f57c81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57c809eea0 0x7f57c81415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:42.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 monclient: found mon.noname-b 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57c77fe640 1 -- 172.21.15.67:0/1357188032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8009450 con 0x7f57c80a0b50 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57bffff640 10 monclient: get_auth_request con 0x7f57c809f2c0 auth_method 0 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.827+0000 7f57bffff640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57c809f2c0 0x7f57c8131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57c77fe640 1 -- 172.21.15.67:0/1357188032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 msgr2=0x7f57c81324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57c77fe640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c80a0b50 0x7f57c81324e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f57c0007ef0 tx=0x7f57c0009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0014420 con 0x7f57c80a0b50 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 10 monclient: dump: 2024-09-15T01:54:42.842 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c0014580 con 0x7f57c80a0b50 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57cd43f640 5 monclient: authenticate success, global_id 5209 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57cd43f640 10 monclient: _renew_subs 2024-09-15T01:54:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57cd43f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.829+0000 7f57cd43f640 1 -- 172.21.15.67:0/1357188032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c811f0c0 con 0x7f57c80a0b50 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0014940 con 0x7f57c80a0b50 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57c5ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 10 monclient: dump: 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 10 monclient: _renew_subs 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 1 -- 172.21.15.67:0/1357188032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8141b20 con 0x7f57c80a0b50 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f57c0014d30 con 0x7f57c80a0b50 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57cd43f640 1 -- 172.21.15.67:0/1357188032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c8068690 con 0x7f57c80a0b50 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.830+0000 7f57beffd640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f57a8038740 0x7f57a803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:42.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.831+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f57c004c8f0 con 0x7f57c80a0b50 2024-09-15T01:54:42.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.831+0000 7f57c6ffd640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f57a8038740 0x7f57a803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:42.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.831+0000 7f57c6ffd640 10 monclient: get_auth_request con 0x7f57a8038740 auth_method 0 2024-09-15T01:54:42.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.833+0000 7f57c6ffd640 1 --2- 172.21.15.67:0/1357188032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f57a8038740 0x7f57a803abf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f57b8034000 tx=0x7f57b8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:42.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.835+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f57c8068690 con 0x7f57c80a0b50 2024-09-15T01:54:42.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.835+0000 7f57beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:42.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:42.835+0000 7f57beffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:43.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.003+0000 7f57cd43f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:54:43.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.003+0000 7f57cd43f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:54:43.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.003+0000 7f57cd43f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.003+0000 7f57cd43f640 1 -- 172.21.15.67:0/1357188032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f57c811f4c0 con 0x7f57c80a0b50 2024-09-15T01:54:43.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.004+0000 7f57beffd640 1 -- 172.21.15.67:0/1357188032 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v75) v1 ==== 99+0+0 (secure 0 0 0) 0x7f57c00126a0 con 0x7f57c80a0b50 2024-09-15T01:54:43.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.004+0000 7f57beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:54:43.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.004+0000 7f57beffd640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T01:54:43.008 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T01:54:43.010 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:43.011 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush unlink host1 bar 2024-09-15T01:54:43.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:43.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.084+0000 7f5b40ec9640 10 monclient: get_monmap_and_config 2024-09-15T01:54:43.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.084+0000 7f5b40ec9640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.084+0000 7f5b40ec9640 10 monclient: monmap: 2024-09-15T01:54:43.087 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:43.087 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.088 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:43.086076+0000 2024-09-15T01:54:43.088 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:43.086076+0000 2024-09-15T01:54:43.088 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.088 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.088 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:43.089 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:43.089 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:43.089 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:43.089 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:43.089 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.085+0000 7f5b40ec9640 1 Processor -- start 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.085+0000 7f5b40ec9640 1 -- start start 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.085+0000 7f5b40ec9640 10 monclient: init 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b3c0a1a00 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:43.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b3c09b1f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b3c09d5a0 0x7f5b3c137250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5b3c09d5a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3c09cc40 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3c0a1dd0 con 0x7f5b3c0a1a00 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3c09cf80 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b40ec9640 10 monclient(hunting): authenticate will time out at 915.555207s 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b3c0a1a00 0x7f5b3c09aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55222/0 (socket says 172.21.15.67:55222) 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3a59c640 1 -- 172.21.15.67:0/1640382474 learned_addr learned my addr 172.21.15.67:0/1640382474 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 1 --2- 172.21.15.67:0/1640382474 >> v2:172.21.15.67:3300/0 conn(0x7f5b3c09d5a0 0x7f5b3c137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): get_auth_request con 0x7f5b3c09d5a0 auth_method 0 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient(hunting): handle_auth_done global_id 5212 payload 274 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 1 -- 172.21.15.67:0/1640382474 >> v1:172.21.15.67:6789/0 conn(0x7f5b3c0a1a00 legacy=0x7f5b3c09aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 1 -- 172.21.15.67:0/1640382474 >> v1:172.21.15.17:6789/0 conn(0x7f5b3c09b1f0 legacy=0x7f5b3c133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 1 monclient: found mon.noname-c 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.086+0000 7f5b3ad9d640 1 -- 172.21.15.67:0/1640382474 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c12efa0 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3ad9d640 1 -- 172.21.15.67:0/1640382474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09d5a0 msgr2=0x7f5b3c137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3ad9d640 1 --2- 172.21.15.67:0/1640382474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09d5a0 0x7f5b3c137250 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009e60 tx=0x7f5b34009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b792e25f3dccc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 1 -- 172.21.15.67:0/1640382474 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b34007630 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: dump: 2024-09-15T01:54:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 1 -- 172.21.15.67:0/1640382474 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3402d050 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 1 -- 172.21.15.67:0/1640382474 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b340079c0 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b40ec9640 5 monclient: authenticate success, global_id 5212 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b40ec9640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b40ec9640 4 set_mon_vals no callback set 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.089+0000 7f5b3959a640 10 monclient: dump: 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 10 monclient: shutdown 2024-09-15T01:54:43.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 -- 172.21.15.67:0/1640382474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09d5a0 msgr2=0x7f5b3c137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/1640382474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09d5a0 0x7f5b3c137250 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009e60 tx=0x7f5b34009780 comp rx=0 tx=0).stop 2024-09-15T01:54:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 -- 172.21.15.67:0/1640382474 shutdown_connections 2024-09-15T01:54:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/1640382474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09d5a0 0x7f5b3c137250 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 -- 172.21.15.67:0/1640382474 >> 172.21.15.67:0/1640382474 conn(0x7f5b3c071340 msgr2=0x7f5b3c0a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 -- 172.21.15.67:0/1640382474 shutdown_connections 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.090+0000 7f5b40ec9640 1 -- 172.21.15.67:0/1640382474 wait complete. 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient: monmap: 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:43.111 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 Processor -- start 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 -- start start 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient: init 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:43.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f5b3c09b1f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5b3c09d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 0x7f5b3c12e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f5b3c0a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c13fd10 con 0x7f5b3c0a1a00 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c11c520 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c11bfb0 con 0x7f5b3c09d5a0 2024-09-15T01:54:43.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.091+0000 7f5b40ec9640 10 monclient(hunting): authenticate will time out at 915.560957s 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57800/0 (socket says 172.21.15.67:57800) 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 -- 172.21.15.67:0/502087033 learned_addr learned my addr 172.21.15.67:0/502087033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): get_auth_request con 0x7f5b3c09b1f0 auth_method 0 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 0x7f5b3c12e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 10 monclient(hunting): get_auth_request con 0x7f5b3c09d5a0 auth_method 0 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 10 monclient(hunting): get_auth_request con 0x7f5b3c0a1a00 auth_method 0 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 msgr2=0x7f5b3c12db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 msgr2=0x7f5b3c12db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient(hunting): handle_auth_done global_id 5215 payload 824 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b39d9b640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 msgr2=0x7f5b3c12db00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 msgr2=0x7f5b3c12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 0x7f5b3c12e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 monclient: found mon.noname-b 2024-09-15T01:54:43.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3a59c640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b34009a20 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.092+0000 7f5b3ad9d640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 0x7f5b3c12e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b3a59c640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 msgr2=0x7f5b3c09b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b3a59c640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400c5e0 tx=0x7f5b24004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24005cb0 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: dump: 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.117 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b24005e30 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24007340 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 5 monclient: authenticate success, global_id 5215 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b327fc640 10 monclient: dump: 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 10 monclient: _renew_subs 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b38d99640 4 set_mon_vals no callback set 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3c11bc50 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 10 monclient: _renew_subs 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.094+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3c09c370 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5b2401b400 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b40ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b40ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b40ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3c04bf60 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b327fc640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 0x7f5b1c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.095+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f5b2404bc60 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.096+0000 7f5b39d9b640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 0x7f5b1c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.096+0000 7f5b39d9b640 10 monclient: get_auth_request con 0x7f5b1c038790 auth_method 0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.098+0000 7f5b39d9b640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 0x7f5b1c03ac40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c09b5e0 tx=0x7f5b2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.100+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5b2401ab40 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.100+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.100+0000 7f5b327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 10 monclient: monmap: 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:43.196961+0000 2024-09-15T01:54:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:43.196961+0000 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:43.199 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 1 Processor -- start 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 1 -- start start 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.196+0000 7f902199d640 10 monclient: init 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f901c131b20 0x7f901c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): picked mon.noname-c con 0x7f901c131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:43.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f901c0a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): picked mon.noname-b con 0x7f901c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0a0040 con 0x7f901c0a0280 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c09bff0 con 0x7f901c131b20 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0a14a0 con 0x7f901c09f1e0 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f902199d640 10 monclient(hunting): authenticate will time out at 943.193476s 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f901c131b20 0x7f901c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901a7fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f901c0a0280 0x7f901c131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48568/0 (socket says 172.21.15.17:48568) 2024-09-15T01:54:43.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901a7fc640 1 -- 172.21.15.17:0/3854702144 learned_addr learned my addr 172.21.15.17:0/3854702144 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 10 monclient(hunting): get_auth_request con 0x7f901c09f1e0 auth_method 0 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): get_auth_request con 0x7f901c131b20 auth_method 0 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 1 -- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 1 -- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.197+0000 7f901b7fe640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 10 monclient(hunting): handle_auth_done global_id 5218 payload 274 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 1 -- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c138e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 1 -- 172.21.15.17:0/3854702144 >> v1:172.21.15.17:6789/0 conn(0x7f901c0a0280 legacy=0x7f901c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 1 monclient: found mon.noname-c 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.198+0000 7f901affd640 1 -- 172.21.15.17:0/3854702144 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c132600 con 0x7f901c131b20 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.200+0000 7f901affd640 1 -- 172.21.15.17:0/3854702144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c131b20 msgr2=0x7f901c09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.200+0000 7f901affd640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c131b20 0x7f901c09eca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f900c009850 tx=0x7f900c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c93bc445c0cdedc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 1 -- 172.21.15.17:0/3854702144 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c02adf0 con 0x7f901c131b20 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: dump: 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.210 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 5 monclient: authenticate success, global_id 5218 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 1 -- 172.21.15.17:0/3854702144 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c032440 con 0x7f901c131b20 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 4 set_mon_vals no callback set 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 1 -- 172.21.15.17:0/3854702144 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0327e0 con 0x7f901c131b20 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f9019ffb640 10 monclient: dump: 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.212 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 10 monclient: shutdown 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 -- 172.21.15.17:0/3854702144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c131b20 msgr2=0x7f901c09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c131b20 0x7f901c09eca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f900c009850 tx=0x7f900c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 -- 172.21.15.17:0/3854702144 shutdown_connections 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 --2- 172.21.15.17:0/3854702144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c131b20 0x7f901c09eca0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 -- 172.21.15.17:0/3854702144 >> 172.21.15.17:0/3854702144 conn(0x7f901c071270 msgr2=0x7f901c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 -- 172.21.15.17:0/3854702144 shutdown_connections 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.201+0000 7f902199d640 1 -- 172.21.15.17:0/3854702144 wait complete. 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.202+0000 7f902199d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.202+0000 7f902199d640 10 monclient: monmap: 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.202+0000 7f902199d640 1 Processor -- start 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 -- start start 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient: init 2024-09-15T01:54:43.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): picked mon.noname-c con 0x7f901c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0a0280 0x7f901c132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): picked mon.noname-b con 0x7f901c0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 0x7f901c132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): picked mon.noname-a con 0x7f901c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c137f40 con 0x7f901c131b20 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0a09b0 con 0x7f901c0a0280 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c1308e0 con 0x7f901c09f1e0 2024-09-15T01:54:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f902199d640 10 monclient(hunting): authenticate will time out at 943.199557s 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 0x7f901c132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34188/0 (socket says 172.21.15.17:34188) 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 -- 172.21.15.17:0/989096790 learned_addr learned my addr 172.21.15.17:0/989096790 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 10 monclient(hunting): get_auth_request con 0x7f901c09f1e0 auth_method 0 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): get_auth_request con 0x7f901c131b20 auth_method 0 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0a0280 0x7f901c132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c09b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c09b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901affd640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): get_auth_request con 0x7f901c0a0280 auth_method 0 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 10 monclient(hunting): handle_auth_done global_id 44589 payload 824 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 msgr2=0x7f901c09b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0a0280 msgr2=0x7f901c132950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.203+0000 7f901b7fe640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0a0280 0x7f901c132950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.204+0000 7f901b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.204+0000 7f901b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.204+0000 7f901b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.204+0000 7f901b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.204+0000 7f901b7fe640 1 -- 172.21.15.17:0/989096790 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c009390 con 0x7f901c131b20 2024-09-15T01:54:43.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f901b7fe640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 msgr2=0x7f901c132e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f901b7fe640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 0x7f901c132e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9014007e40 tx=0x7f90140097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014014420 con 0x7f901c131b20 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: dump: 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.219 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014014580 con 0x7f901c131b20 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 5 monclient: authenticate success, global_id 44589 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 10 monclient: _renew_subs 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c135840 con 0x7f901c131b20 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014014940 con 0x7f901c131b20 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:43.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f9002ffd640 10 monclient: dump: 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 10 monclient: _renew_subs 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c133da0 con 0x7f901c131b20 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.206+0000 7f90197fa640 4 set_mon_vals no callback set 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f901401e460 con 0x7f901c131b20 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f902199d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f902199d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f902199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c135cc0 con 0x7f901c131b20 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f9002ffd640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 0x7f900803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f901404c750 con 0x7f901c131b20 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f901affd640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 0x7f900803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.207+0000 7f901affd640 10 monclient: get_auth_request con 0x7f9008038740 auth_method 0 2024-09-15T01:54:43.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.210+0000 7f901affd640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 0x7f900803abf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f900c000e10 tx=0x7f900c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.212+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f90140128e0 con 0x7f901c131b20 2024-09-15T01:54:43.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.212+0000 7f9002ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.212+0000 7f9002ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:43.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.269+0000 7f5b40ec9640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:43.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.269+0000 7f5b40ec9640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:43.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.269+0000 7f5b40ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:43.269+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f5b3c12ecc0 con 0x7f5b3c09b1f0 2024-09-15T01:54:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f902199d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f902199d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f902199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f901c0724a0 con 0x7f901c131b20 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f9002ffd640 1 -- 172.21.15.17:0/989096790 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f901401b310 con 0x7f901c131b20 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f9002ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.442+0000 7f9002ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:43.445 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 msgr2=0x7f900803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 0x7f900803abf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f900c000e10 tx=0x7f900c032000 comp rx=0 tx=0).stop 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 10 monclient: shutdown 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 msgr2=0x7f901c132e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 0x7f901c132e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9014007e40 tx=0x7f90140097e0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 shutdown_connections 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9008038740 0x7f900803abf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f901c131b20 0x7f901c132e90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0a0280 0x7f901c132950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 --2- 172.21.15.17:0/989096790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f901c09f1e0 0x7f901c09b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.444+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 >> 172.21.15.17:0/989096790 conn(0x7f901c071270 msgr2=0x7f901c09e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.445+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 shutdown_connections 2024-09-15T01:54:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.445+0000 7f902199d640 1 -- 172.21.15.17:0/989096790 wait complete. 2024-09-15T01:54:43.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:54:43.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:54:43.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 10 monclient: monmap: 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.554 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:43.552952+0000 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:43.552952+0000 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 1 Processor -- start 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 1 -- start start 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.552+0000 7f35a751d640 10 monclient: init 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35a00a58b0 0x7f35a0132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-c con 0x7f35a00a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35a00980a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35a0136dd0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00a01e0 con 0x7f35a0136dd0 2024-09-15T01:54:43.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00a1580 con 0x7f35a00980a0 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a00a5c80 con 0x7f35a00a58b0 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a751d640 10 monclient(hunting): authenticate will time out at 943.549394s 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a5aba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f35a0136dd0 0x7f35a01391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48570/0 (socket says 172.21.15.17:48570) 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a5aba640 1 -- 172.21.15.17:0/2706459072 learned_addr learned my addr 172.21.15.17:0/2706459072 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 1 --2- 172.21.15.17:0/2706459072 >> v2:172.21.15.67:3300/0 conn(0x7f35a00a58b0 0x7f35a0132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): get_auth_request con 0x7f35a00a58b0 auth_method 0 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient(hunting): handle_auth_done global_id 5224 payload 274 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 1 -- 172.21.15.17:0/2706459072 >> v1:172.21.15.67:6789/0 conn(0x7f35a00980a0 legacy=0x7f35a00984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 1 -- 172.21.15.17:0/2706459072 >> v1:172.21.15.17:6789/0 conn(0x7f35a0136dd0 legacy=0x7f35a01391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 1 monclient: found mon.noname-c 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.553+0000 7f35a52b9640 1 -- 172.21.15.17:0/2706459072 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0098bb0 con 0x7f35a00a58b0 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.556+0000 7f35a52b9640 1 -- 172.21.15.17:0/2706459072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00a58b0 msgr2=0x7f35a0132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.556+0000 7f35a52b9640 1 --2- 172.21.15.17:0/2706459072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00a58b0 0x7f35a0132040 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3598009850 tx=0x7f35980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2bccde947f4c687 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 1 -- 172.21.15.17:0/2706459072 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598032440 con 0x7f35a00a58b0 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: dump: 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.563 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 1 -- 172.21.15.17:0/2706459072 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35980325a0 con 0x7f35a00a58b0 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 1 -- 172.21.15.17:0/2706459072 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598032940 con 0x7f35a00a58b0 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 5 monclient: authenticate success, global_id 5224 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 4 set_mon_vals no callback set 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f359ffff640 10 monclient: dump: 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 10 monclient: shutdown 2024-09-15T01:54:43.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 -- 172.21.15.17:0/2706459072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00a58b0 msgr2=0x7f35a0132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 --2- 172.21.15.17:0/2706459072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00a58b0 0x7f35a0132040 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3598009850 tx=0x7f35980090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 -- 172.21.15.17:0/2706459072 shutdown_connections 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 --2- 172.21.15.17:0/2706459072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00a58b0 0x7f35a0132040 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 -- 172.21.15.17:0/2706459072 >> 172.21.15.17:0/2706459072 conn(0x7f35a0070ed0 msgr2=0x7f35a00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 -- 172.21.15.17:0/2706459072 shutdown_connections 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.557+0000 7f35a751d640 1 -- 172.21.15.17:0/2706459072 wait complete. 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.558+0000 7f35a751d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.558+0000 7f35a751d640 10 monclient: monmap: 2024-09-15T01:54:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.558+0000 7f35a751d640 1 Processor -- start 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.558+0000 7f35a751d640 1 -- start start 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.558+0000 7f35a751d640 10 monclient: init 2024-09-15T01:54:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00980a0 0x7f35a0132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-b con 0x7f35a00980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 0x7f35a0133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-a con 0x7f35a00a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): picked mon.noname-c con 0x7f35a0136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a009ed00 con 0x7f35a00a58b0 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00a32a0 con 0x7f35a00980a0 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0134ec0 con 0x7f35a0136dd0 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a751d640 10 monclient(hunting): authenticate will time out at 943.555177s 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 0x7f35a0133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34202/0 (socket says 172.21.15.17:34202) 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 -- 172.21.15.17:0/3328980647 learned_addr learned my addr 172.21.15.17:0/3328980647 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 10 monclient(hunting): get_auth_request con 0x7f35a0136dd0 auth_method 0 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): get_auth_request con 0x7f35a00a58b0 auth_method 0 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 msgr2=0x7f35a01335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 msgr2=0x7f35a01335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00980a0 0x7f35a0132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a5aba640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): get_auth_request con 0x7f35a00980a0 auth_method 0 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient(hunting): handle_auth_done global_id 44592 payload 824 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 msgr2=0x7f35a01335a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35a0136dd0 0x7f35a01335a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00980a0 msgr2=0x7f35a0132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a00980a0 0x7f35a0132b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 monclient: found mon.noname-a 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.559+0000 7f35a4ab8640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598009390 con 0x7f35a00a58b0 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.561+0000 7f35a4ab8640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 msgr2=0x7f35a0133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.561+0000 7f35a4ab8640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 0x7f35a0133060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f359400b4c0 tx=0x7f3594006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400f720 con 0x7f35a00a58b0 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: dump: 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359400f880 con 0x7f35a00a58b0 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400fc40 con 0x7f35a00a58b0 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359d7fa640 10 monclient: dump: 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 5 monclient: authenticate success, global_id 44592 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 monclient: _renew_subs 2024-09-15T01:54:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a0134be0 con 0x7f35a00a58b0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f359f7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 monclient: _renew_subs 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a009e2e0 con 0x7f35a00a58b0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 0x7f35a002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.562+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0063b20 con 0x7f35a0133ae0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 0x7f35a002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3594011610 con 0x7f35a00a58b0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient: get_auth_request con 0x7f35a0133ae0 auth_method 0 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 10 monclient(hunting): handle_auth_done global_id 44595 payload 824 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a52b9640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35a0135100 con 0x7f35a0133ae0 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f359d7fa640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f358c038740 0x7f358c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f359404b590 con 0x7f35a00a58b0 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a5aba640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f358c038740 0x7f358c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.563+0000 7f35a5aba640 10 monclient: get_auth_request con 0x7f358c038740 auth_method 0 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f35a52b9640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 msgr2=0x7f35a002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f35a52b9640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 0x7f35a002aa40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3598034000 tx=0x7f3598032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598031a70 con 0x7f35a0133ae0 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 10 monclient: dump: 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3598031bd0 con 0x7f35a0133ae0 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 msgr2=0x7f35a002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 0x7f35a002aa40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3598034000 tx=0x7f3598032000 comp rx=0 tx=0).stop 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 msgr2=0x7f35a002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f359d7fa640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a0133ae0 0x7f35a002aa40 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3598034000 tx=0x7f3598032000 comp rx=0 tx=0).stop 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.565+0000 7f35a5aba640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f358c038740 0x7f358c03abf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35a01355c0 tx=0x7f3590009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a751d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a751d640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 0x7f35a005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a751d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0136010 con 0x7f35a005e310 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 0x7f35a005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient: get_auth_request con 0x7f35a005e310 auth_method 0 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 10 monclient(hunting): handle_auth_done global_id 44598 payload 824 2024-09-15T01:54:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.571+0000 7f35a52b9640 1 -- 172.21.15.17:0/3328980647 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f35a00724a0 con 0x7f35a005e310 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a52b9640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 msgr2=0x7f35a005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a52b9640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 0x7f35a005e710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3598009130 tx=0x7f359803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598045910 con 0x7f35a005e310 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 10 monclient: dump: 2024-09-15T01:54:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f3598045a70 con 0x7f35a005e310 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 msgr2=0x7f35a005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 0x7f35a005e710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3598009130 tx=0x7f359803d000 comp rx=0 tx=0).stop 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 msgr2=0x7f35a005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f359d7fa640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a005e310 0x7f35a005e710 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3598009130 tx=0x7f359803d000 comp rx=0 tx=0).stop 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:43.585 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:43.586 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:43.587 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:43.588 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:43.591 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:43.594 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:43.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:43.596 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:43.597 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f358c038740 msgr2=0x7f358c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f358c038740 0x7f358c03abf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35a01355c0 tx=0x7f3590009040 comp rx=0 tx=0).stop 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 10 monclient: shutdown 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 msgr2=0x7f35a0133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a751d640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 0x7f35a0133060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f359400b4c0 tx=0x7f3594006f50 comp rx=0 tx=0).stop 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.574+0000 7f35a5aba640 1 -- 172.21.15.17:0/3328980647 reap_dead start 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.575+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 shutdown_connections 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.575+0000 7f35a751d640 1 --2- 172.21.15.17:0/3328980647 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35a00a58b0 0x7f35a0133060 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.575+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 >> 172.21.15.17:0/3328980647 conn(0x7f35a0070ed0 msgr2=0x7f35a0139340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.575+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 shutdown_connections 2024-09-15T01:54:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.575+0000 7f35a751d640 1 -- 172.21.15.17:0/3328980647 wait complete. 2024-09-15T01:54:43.600 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:43.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:54:43.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:54:43.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.694+0000 7f08dc15d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.694+0000 7f08dc15d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.694+0000 7f08dc15d640 10 monclient: monmap: 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:43.695685+0000 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:43.695685+0000 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:43.698 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.695+0000 7f08dc15d640 1 Processor -- start 2024-09-15T01:54:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.695+0000 7f08dc15d640 1 -- start start 2024-09-15T01:54:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.695+0000 7f08dc15d640 10 monclient: init 2024-09-15T01:54:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08d4068f20 0x7f08d41321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d4068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08d40692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d4136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40a1950 con 0x7f08d40692f0 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d40a1670 con 0x7f08d4068f20 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40a0020 con 0x7f08d4136fe0 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08dc15d640 10 monclient(hunting): authenticate will time out at 943.692130s 2024-09-15T01:54:43.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34212/0 (socket says 172.21.15.17:34212) 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 -- 172.21.15.17:0/2201479280 learned_addr learned my addr 172.21.15.17:0/2201479280 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 10 monclient(hunting): get_auth_request con 0x7f08d4136fe0 auth_method 0 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d96f8640 1 --1- 172.21.15.17:0/2201479280 >> v1:172.21.15.17:6789/0 conn(0x7f08d40692f0 0x7f08d40696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48576/0 (socket says 172.21.15.17:48576) 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 -- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 msgr2=0x7f08d41393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 -- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 msgr2=0x7f08d41393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08da6fa640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/2201479280 >> v2:172.21.15.67:3300/0 conn(0x7f08d4068f20 0x7f08d41321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request con 0x7f08d4068f20 auth_method 0 2024-09-15T01:54:43.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_done global_id 5230 payload 274 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 -- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 msgr2=0x7f08d41393c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 -- 172.21.15.17:0/2201479280 >> v1:172.21.15.17:6789/0 conn(0x7f08d40692f0 legacy=0x7f08d40696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 monclient: found mon.noname-c 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:43.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.696+0000 7f08d9ef9640 1 -- 172.21.15.17:0/2201479280 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d40a1390 con 0x7f08d4068f20 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.699+0000 7f08d9ef9640 1 -- 172.21.15.17:0/2201479280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4068f20 msgr2=0x7f08d41321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.699+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4068f20 0x7f08d41321c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009880 tx=0x7f08c80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=945854d414f364fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 1 -- 172.21.15.17:0/2201479280 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c802aea0 con 0x7f08d4068f20 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: dump: 2024-09-15T01:54:43.709 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 1 -- 172.21.15.17:0/2201479280 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8032440 con 0x7f08d4068f20 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 1 -- 172.21.15.17:0/2201479280 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c80327e0 con 0x7f08d4068f20 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 5 monclient: authenticate success, global_id 5230 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 4 set_mon_vals no callback set 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08d8ef7640 10 monclient: dump: 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.711 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 10 monclient: shutdown 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 1 -- 172.21.15.17:0/2201479280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4068f20 msgr2=0x7f08d41321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.700+0000 7f08dc15d640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4068f20 0x7f08d41321c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009880 tx=0x7f08c80090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 -- 172.21.15.17:0/2201479280 shutdown_connections 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4136fe0 0x7f08d41393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 --2- 172.21.15.17:0/2201479280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4068f20 0x7f08d41321c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 -- 172.21.15.17:0/2201479280 >> 172.21.15.17:0/2201479280 conn(0x7f08d4070fb0 msgr2=0x7f08d40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 -- 172.21.15.17:0/2201479280 shutdown_connections 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 -- 172.21.15.17:0/2201479280 wait complete. 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 10 monclient: build_initial_monmap 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 10 monclient: monmap: 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:43.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 Processor -- start 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 1 -- start start 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.701+0000 7f08dc15d640 10 monclient: init 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d4068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 0x7f08d4130790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-a con 0x7f08d40692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4136fe0 0x7f08d4099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d4136fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d411cd30 con 0x7f08d40692f0 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40a07c0 con 0x7f08d4136fe0 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d411ca50 con 0x7f08d4068f20 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08dc15d640 10 monclient(hunting): authenticate will time out at 943.698362s 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34222/0 (socket says 172.21.15.17:34222) 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 learned_addr learned my addr 172.21.15.17:0/270741856 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 0x7f08d4130790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request con 0x7f08d4068f20 auth_method 0 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): get_auth_request con 0x7f08d40692f0 auth_method 0 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 msgr2=0x7f08d4130250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 msgr2=0x7f08d4130250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08da6fa640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4136fe0 0x7f08d4099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08da6fa640 10 monclient(hunting): get_auth_request con 0x7f08d4136fe0 auth_method 0 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08da6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08da6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08da6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient(hunting): handle_auth_done global_id 44601 payload 824 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 msgr2=0x7f08d4130250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d4068f20 0x7f08d4130250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4136fe0 msgr2=0x7f08d4099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d4136fe0 0x7f08d4099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 monclient: found mon.noname-a 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.702+0000 7f08d96f8640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c8009390 con 0x7f08d40692f0 2024-09-15T01:54:43.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08d96f8640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 msgr2=0x7f08d4130790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08d96f8640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 0x7f08d4130790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c490 tx=0x7f08d0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00074f0 con 0x7f08d40692f0 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: dump: 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.718 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 5 monclient: authenticate success, global_id 44601 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 10 monclient: _renew_subs 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d409ca20 con 0x7f08d40692f0 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d0007670 con 0x7f08d40692f0 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0005a20 con 0x7f08d40692f0 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c5ffb640 10 monclient: dump: 2024-09-15T01:54:43.719 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08c7fff640 4 set_mon_vals no callback set 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 10 monclient: _renew_subs 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:43.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.705+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d411c6f0 con 0x7f08d40692f0 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08dc15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08dc15d640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4063b90 con 0x7f08d409a180 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f08d001b3b0 con 0x7f08d40692f0 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 10 monclient: get_auth_request con 0x7f08d409a180 auth_method 0 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08c5ffb640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08b8038740 0x7f08b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:43.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:43.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f08d004b560 con 0x7f08d40692f0 2024-09-15T01:54:43.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08da6fa640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08b8038740 0x7f08b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.706+0000 7f08da6fa640 10 monclient: get_auth_request con 0x7f08b8038740 auth_method 0 2024-09-15T01:54:43.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.709+0000 7f08da6fa640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08b8038740 0x7f08b803abf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009650 tx=0x7f08cc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:43.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 10 monclient: get_auth_request con 0x7f08d409a180 auth_method 0 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:43.907+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:54:44.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.202+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v76) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5b2401a360 con 0x7f5b3c09b1f0 2024-09-15T01:54:44.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.202+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:44.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.202+0000 7f5b327fc640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T01:54:44.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 10 monclient: get_auth_request con 0x7f08d409a180 auth_method 0 2024-09-15T01:54:44.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:44.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:44.308+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.308+0000 7f5b40ec9640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.308+0000 7f5b40ec9640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.308+0000 7f5b40ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.308+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f5b3c0008d0 con 0x7f5b3c09b1f0 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.309+0000 7f5b327fc640 1 -- 172.21.15.67:0/502087033 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v76) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5b2401a540 con 0x7f5b3c09b1f0 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.309+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.309+0000 7f5b327fc640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T01:54:44.312 INFO:tasks.workunit.client.0.smithi067.stderr:item -11 does not appear in that position 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 msgr2=0x7f5b1c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 0x7f5b1c03ac40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c09b5e0 tx=0x7f5b2c008040 comp rx=0 tx=0).stop 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 10 monclient: shutdown 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 msgr2=0x7f5b3c09b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400c5e0 tx=0x7f5b24004960 comp rx=0 tx=0).stop 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 shutdown_connections 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5b1c038790 0x7f5b1c03ac40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5b3c0a1a00 0x7f5b3c12e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5b3c09d5a0 0x7f5b3c12db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 --2- 172.21.15.67:0/502087033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c09b1f0 0x7f5b3c09b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 >> 172.21.15.67:0/502087033 conn(0x7f5b3c071340 msgr2=0x7f5b3c0a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:44.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 shutdown_connections 2024-09-15T01:54:44.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.311+0000 7f5b40ec9640 1 -- 172.21.15.67:0/502087033 wait complete. 2024-09-15T01:54:44.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:44.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c host1 2024-09-15T01:54:44.324 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 1 2024-09-15T01:54:44.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.396+0000 7f5d9d55d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:44.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.396+0000 7f5d9d55d640 10 monclient: build_initial_monmap 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.396+0000 7f5d9d55d640 10 monclient: monmap: 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:44.397692+0000 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:44.397692+0000 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:44.399 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 Processor -- start 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 -- start start 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient: init 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d98130e00 0x7f5d98132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-c con 0x7f5d98130e00 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d981311d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d981315f0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9809ff20 con 0x7f5d981311d0 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d98069be0 con 0x7f5d981315f0 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d981320b0 con 0x7f5d98130e00 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d9d55d640 10 monclient(hunting): authenticate will time out at 916.866885s 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.397+0000 7f5d96ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d98130e00 0x7f5d98132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d977fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5d981315f0 0x7f5d980694d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55232/0 (socket says 172.21.15.67:55232) 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d977fe640 1 -- 172.21.15.67:0/3386835497 learned_addr learned my addr 172.21.15.67:0/3386835497 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d98130e00 0x7f5d98132430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57806/0 (socket says 172.21.15.67:57806) 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): get_auth_request con 0x7f5d98130e00 auth_method 0 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:44.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_done global_id 5236 payload 274 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 1 -- 172.21.15.67:0/3386835497 >> v1:172.21.15.67:6789/0 conn(0x7f5d981315f0 legacy=0x7f5d980694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 1 -- 172.21.15.67:0/3386835497 >> v1:172.21.15.17:6789/0 conn(0x7f5d981311d0 legacy=0x7f5d98068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 1 monclient: found mon.noname-c 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.398+0000 7f5d96ffd640 1 -- 172.21.15.67:0/3386835497 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d980a00f0 con 0x7f5d98130e00 2024-09-15T01:54:44.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d96ffd640 1 -- 172.21.15.67:0/3386835497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 msgr2=0x7f5d98132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d96ffd640 1 --2- 172.21.15.67:0/3386835497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d98132430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009850 tx=0x7f5d880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d3bfef4e6d0da86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 1 -- 172.21.15.67:0/3386835497 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88032440 con 0x7f5d98130e00 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: dump: 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.410 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 1 -- 172.21.15.67:0/3386835497 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d880325a0 con 0x7f5d98130e00 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 5 monclient: authenticate success, global_id 5236 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 1 -- 172.21.15.67:0/3386835497 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88032940 con 0x7f5d98130e00 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d967fc640 10 monclient: dump: 2024-09-15T01:54:44.411 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 4 set_mon_vals no callback set 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 10 monclient: shutdown 2024-09-15T01:54:44.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 1 -- 172.21.15.67:0/3386835497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 msgr2=0x7f5d98132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.401+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/3386835497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d98132430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009850 tx=0x7f5d880090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 -- 172.21.15.67:0/3386835497 shutdown_connections 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/3386835497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d98132430 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 -- 172.21.15.67:0/3386835497 >> 172.21.15.67:0/3386835497 conn(0x7f5d980710f0 msgr2=0x7f5d980a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 -- 172.21.15.67:0/3386835497 shutdown_connections 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 -- 172.21.15.67:0/3386835497 wait complete. 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 10 monclient: build_initial_monmap 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 10 monclient: monmap: 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:44.413 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 Processor -- start 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 1 -- start start 2024-09-15T01:54:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.402+0000 7f5d9d55d640 10 monclient: init 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-b con 0x7f5d98130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 0x7f5d9812ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-a con 0x7f5d981311d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 0x7f5d9809c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): picked mon.noname-c con 0x7f5d981315f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9809e830 con 0x7f5d981311d0 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9811be50 con 0x7f5d98130e00 2024-09-15T01:54:44.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980a0820 con 0x7f5d981315f0 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d9d55d640 10 monclient(hunting): authenticate will time out at 916.872426s 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57814/0 (socket says 172.21.15.67:57814) 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 -- 172.21.15.67:0/675711274 learned_addr learned my addr 172.21.15.67:0/675711274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): get_auth_request con 0x7f5d98130e00 auth_method 0 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:44.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d8ffff640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 0x7f5d9812ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient(hunting): handle_auth_done global_id 5239 payload 824 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d977fe640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 0x7f5d9809c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 -- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 msgr2=0x7f5d9809c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 0x7f5d9809c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 -- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 msgr2=0x7f5d9812ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 0x7f5d9812ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 monclient: found mon.noname-b 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d96ffd640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88009390 con 0x7f5d98130e00 2024-09-15T01:54:44.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d8ffff640 10 monclient: get_auth_request con 0x7f5d981311d0 auth_method 0 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d8ffff640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 0x7f5d9812ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d977fe640 10 monclient: get_auth_request con 0x7f5d981315f0 auth_method 0 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.403+0000 7f5d977fe640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 0x7f5d9809c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d96ffd640 1 -- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 msgr2=0x7f5d981412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d96ffd640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009850 tx=0x7f5d88032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88031940 con 0x7f5d98130e00 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: dump: 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.418 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: _finish_auth 0 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d88031aa0 con 0x7f5d98130e00 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88031e40 con 0x7f5d98130e00 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 5 monclient: authenticate success, global_id 5239 2024-09-15T01:54:44.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 10 monclient: _renew_subs 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d980a4b70 con 0x7f5d98130e00 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d95ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d8effd640 10 monclient: dump: 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.420 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 10 monclient: _renew_subs 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.406+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9809ee80 con 0x7f5d98130e00 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d9d55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d9d55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d9d55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9804bf60 con 0x7f5d98130e00 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5d88039040 con 0x7f5d98130e00 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d8effd640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 0x7f5d7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f5d8806c4b0 con 0x7f5d98130e00 2024-09-15T01:54:44.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d8ffff640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 0x7f5d7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.407+0000 7f5d8ffff640 10 monclient: get_auth_request con 0x7f5d78038740 auth_method 0 2024-09-15T01:54:44.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.410+0000 7f5d8ffff640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 0x7f5d7803abf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5d800027e0 tx=0x7f5d80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.411+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5d8804b530 con 0x7f5d98130e00 2024-09-15T01:54:44.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.411+0000 7f5d8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.411+0000 7f5d8effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:44.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.575+0000 7f5d9d55d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:44.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.575+0000 7f5d9d55d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:44.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.575+0000 7f5d9d55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.576+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5d9804bf60 con 0x7f5d98130e00 2024-09-15T01:54:44.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.576+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f5d8803f070 con 0x7f5d98130e00 2024-09-15T01:54:44.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.576+0000 7f5d8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:44.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.576+0000 7f5d8effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:44.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d9d55d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:44.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d9d55d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:44.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d9d55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5d9804bf60 con 0x7f5d98130e00 2024-09-15T01:54:44.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d8effd640 1 -- 172.21.15.67:0/675711274 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f5d88042e80 con 0x7f5d98130e00 2024-09-15T01:54:44.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d8effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:44.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.676+0000 7f5d8effd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 msgr2=0x7f5d7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 0x7f5d7803abf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5d800027e0 tx=0x7f5d80009040 comp rx=0 tx=0).stop 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 10 monclient: shutdown 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 msgr2=0x7f5d981412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009850 tx=0x7f5d88032000 comp rx=0 tx=0).stop 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 shutdown_connections 2024-09-15T01:54:44.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5d78038740 0x7f5d7803abf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5d981311d0 0x7f5d9812ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5d981315f0 0x7f5d9809c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 --2- 172.21.15.67:0/675711274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d98130e00 0x7f5d981412a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 >> 172.21.15.67:0/675711274 conn(0x7f5d980710f0 msgr2=0x7f5d9812fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 shutdown_connections 2024-09-15T01:54:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.678+0000 7f5d9d55d640 1 -- 172.21.15.67:0/675711274 wait complete. 2024-09-15T01:54:44.690 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.7 host1 2024-09-15T01:54:44.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.764+0000 7f451819d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.764+0000 7f451819d640 10 monclient: build_initial_monmap 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.764+0000 7f451819d640 10 monclient: monmap: 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:44.765546+0000 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:44.765546+0000 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:44.767 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:44.768 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 Processor -- start 2024-09-15T01:54:44.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 -- start start 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient: init 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f4510069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f451009eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f451009f290 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100a61a0 con 0x7f451009f290 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100a04c0 con 0x7f451009eea0 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45100a1610 con 0x7f45100a0b50 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:44.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f451819d640 10 monclient(hunting): authenticate will time out at 917.234843s 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f4515738640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f451009eea0 0x7f4510131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55248/0 (socket says 172.21.15.67:55248) 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.765+0000 7f4515738640 1 -- 172.21.15.67:0/3657569362 learned_addr learned my addr 172.21.15.67:0/3657569362 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f4510069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 10 monclient(hunting): get_auth_request con 0x7f45100a0b50 auth_method 0 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 msgr2=0x7f4510069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 msgr2=0x7f4510069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f4510069380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:44.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.766+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f4510069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:44.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 1 -- 172.21.15.67:0/3657569362 >> v1:172.21.15.67:6789/0 conn(0x7f451009eea0 legacy=0x7f4510131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616813200 0 0) 0x7f45100a04c0 con 0x7f451009eea0 2024-09-15T01:54:44.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): my global_id is 5242 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc0034e0 con 0x7f451009eea0 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 1 -- 172.21.15.67:0/3657569362 >> v1:172.21.15.17:6789/0 conn(0x7f451009f290 legacy=0x7f4510132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3669313751 0 0) 0x7f44fc0034e0 con 0x7f451009eea0 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 msgr2=0x7f4510069380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f4510069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 >> v1:172.21.15.17:6789/0 conn(0x7f451009f290 legacy=0x7f4510132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: _finish_hunting 0 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:54:44.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100a0f20 con 0x7f451009eea0 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: _finish_auth 0 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4500001720 con 0x7f451009eea0 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451819d640 5 monclient: authenticate success, global_id 5242 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451819d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: dump: 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.777 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 >> v1:172.21.15.67:6789/0 conn(0x7f451009eea0 legacy=0x7f4510131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f44fc003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): picked mon.b con 0x7f44fc003ad0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f44fc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): picked mon.c con 0x7f45100a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:44.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f44fc004de0 0x7f44fc005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): picked mon.a con 0x7f44fc004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45100a1610 con 0x7f44fc004de0 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc00bcd0 con 0x7f44fc003ad0 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc00be90 con 0x7f45100a0b50 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4514f37640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451819d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f44fc003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): get_auth_request con 0x7f44fc003ad0 auth_method 0 2024-09-15T01:54:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f44fc004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): get_auth_request con 0x7f45100a0b50 auth_method 0 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f44fc004de0 0x7f44fc005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 10 monclient(hunting): get_auth_request con 0x7f44fc004de0 auth_method 0 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f451673a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.768+0000 7f4515738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515738640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 10 monclient(hunting): handle_auth_done global_id 5242 payload 293 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 msgr2=0x7f44fc004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0b50 0x7f44fc004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f44fc004de0 msgr2=0x7f44fc005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f44fc004de0 0x7f44fc005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 10 monclient: _finish_hunting 0 2024-09-15T01:54:44.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 monclient: found mon.b 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0034e0 con 0x7f44fc003ad0 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 10 monclient: _finish_auth 0 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f4515f39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.769+0000 7f451673a640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f44fc004de0 0x7f44fc005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4515f39640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 msgr2=0x7f44fc003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4515f39640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f44fc003ea0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f450c006fd0 tx=0x7f450c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df4a72b2a43a3646 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c02add0 con 0x7f44fc003ad0 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:44.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 10 monclient: dump: 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c02e070 con 0x7f44fc003ad0 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f4514f37640 1 -- 172.21.15.67:0/3657569362 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c034660 con 0x7f44fc003ad0 2024-09-15T01:54:44.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.771+0000 7f451819d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 4 set_mon_vals no callback set 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f4514f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f4514f37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f4514f37640 10 monclient: dump: 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 10 monclient: shutdown 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 -- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 msgr2=0x7f44fc003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 --2- 172.21.15.67:0/3657569362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f44fc003ea0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f450c006fd0 tx=0x7f450c008040 comp rx=0 tx=0).stop 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451673a640 1 -- 172.21.15.67:0/3657569362 reap_dead start 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 -- 172.21.15.67:0/3657569362 shutdown_connections 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 -- 172.21.15.67:0/3657569362 >> 172.21.15.67:0/3657569362 conn(0x7f4510071330 msgr2=0x7f45100a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 -- 172.21.15.67:0/3657569362 shutdown_connections 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.772+0000 7f451819d640 1 -- 172.21.15.67:0/3657569362 wait complete. 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient: build_initial_monmap 2024-09-15T01:54:44.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient: monmap: 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 1 Processor -- start 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 1 -- start start 2024-09-15T01:54:44.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient: init 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 10 monclient(hunting): picked mon.noname-b con 0x7f44fc003ad0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.773+0000 7f451819d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 0x7f4510132050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): picked mon.noname-c con 0x7f4510131c80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f451011e930 0x7f451011ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): picked mon.noname-a con 0x7f451011e930 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510132590 con 0x7f451011e930 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510120aa0 con 0x7f44fc003ad0 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45100a1260 con 0x7f4510131c80 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451819d640 10 monclient(hunting): authenticate will time out at 917.243048s 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57832/0 (socket says 172.21.15.67:57832) 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 -- 172.21.15.67:0/2008874027 learned_addr learned my addr 172.21.15.67:0/2008874027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): get_auth_request con 0x7f44fc003ad0 auth_method 0 2024-09-15T01:54:44.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451673a640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f451011e930 0x7f451011ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451673a640 10 monclient(hunting): get_auth_request con 0x7f451011e930 auth_method 0 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451673a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f451673a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient(hunting): handle_auth_done global_id 5245 payload 824 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515738640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 0x7f4510132050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 -- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 msgr2=0x7f4510132050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 0x7f4510132050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 -- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f451011e930 msgr2=0x7f451011ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f451011e930 0x7f451011ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient: _finish_hunting 0 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 monclient: found mon.noname-b 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515f39640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451009ea50 con 0x7f44fc003ad0 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515738640 10 monclient: get_auth_request con 0x7f4510131c80 auth_method 0 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.774+0000 7f4515738640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 0x7f4510132050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4515f39640 1 -- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 msgr2=0x7f4510140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4515f39640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f450c02a890 tx=0x7f450c035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c0335c0 con 0x7f44fc003ad0 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: dump: 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.794 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c033740 con 0x7f44fc003ad0 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f451819d640 5 monclient: authenticate success, global_id 5245 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f451819d640 10 monclient: _renew_subs 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f451819d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4510120e70 con 0x7f44fc003ad0 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c033b00 con 0x7f44fc003ad0 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4505ffb640 10 monclient: dump: 2024-09-15T01:54:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.776+0000 7f4507fff640 4 set_mon_vals no callback set 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.777+0000 7f451819d640 10 monclient: _renew_subs 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.777+0000 7f451819d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.777+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45101202a0 con 0x7f44fc003ad0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.777+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f450c049460 con 0x7f44fc003ad0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f4505ffb640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 0x7f44f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f450c06cca0 con 0x7f44fc003ad0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f451819d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f451819d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f451819d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451004bf60 con 0x7f44fc003ad0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f4515738640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 0x7f44f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.778+0000 7f4515738640 10 monclient: get_auth_request con 0x7f44f4038740 auth_method 0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.780+0000 7f4515738640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 0x7f44f403abf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4500001ed0 tx=0x7f4500009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.782+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f450c03f440 con 0x7f44fc003ad0 2024-09-15T01:54:44.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.782+0000 7f4505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:44.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.782+0000 7f4505ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:44.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.950+0000 7f451819d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:44.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.950+0000 7f451819d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:44.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.950+0000 7f451819d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:44.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:44.950+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f451004bf60 con 0x7f44fc003ad0 2024-09-15T01:54:45.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient: get_auth_request con 0x7f08d409a180 auth_method 0 2024-09-15T01:54:45.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:45.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.109+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.110+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_done global_id 64100 payload 824 2024-09-15T01:54:45.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.110+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08d411d1a0 con 0x7f08d409a180 2024-09-15T01:54:45.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009130 tx=0x7f08c8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c802aea0 con 0x7f08d409a180 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.112+0000 7f08c5ffb640 10 monclient: dump: 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f08c803cd70 con 0x7f08d409a180 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009130 tx=0x7f08c8032000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 msgr2=0x7f08d409a550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:45.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.113+0000 7f08c5ffb640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409a180 0x7f08d409a550 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009130 tx=0x7f08c8032000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08dc15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08dc15d640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 0x7f08d4134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08dc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40724a0 con 0x7f08d409aa90 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 0x7f08d4134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 10 monclient: get_auth_request con 0x7f08d409aa90 auth_method 0 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.119+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.120+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.120+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.120+0000 7f08d9ef9640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-15T01:54:45.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.120+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08d411cf50 con 0x7f08d409aa90 2024-09-15T01:54:45.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08d9ef9640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 msgr2=0x7f08d4134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08d9ef9640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 0x7f08d4134ab0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08d411d420 tx=0x7f08c803e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c803bba0 con 0x7f08d409aa90 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 10 monclient: dump: 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.125 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f08c803bd00 con 0x7f08d409aa90 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 msgr2=0x7f08d4134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 0x7f08d4134ab0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08d411d420 tx=0x7f08c803e040 comp rx=0 tx=0).stop 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 msgr2=0x7f08d4134ab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:45.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.122+0000 7f08c5ffb640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08d409aa90 0x7f08d4134ab0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.127 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.128 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:45.130 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.131 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:45.137 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.138 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:45.139 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:45.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08b8038740 msgr2=0x7f08b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08b8038740 0x7f08b803abf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009650 tx=0x7f08cc00b040 comp rx=0 tx=0).stop 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 10 monclient: shutdown 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 msgr2=0x7f08d4130790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 --2- 172.21.15.17:0/270741856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f08d40692f0 0x7f08d4130790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c490 tx=0x7f08d0007e40 comp rx=0 tx=0).stop 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08da6fa640 1 -- 172.21.15.17:0/270741856 reap_dead start 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08da6fa640 1 -- 172.21.15.17:0/270741856 reap_dead start 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 shutdown_connections 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 >> 172.21.15.17:0/270741856 conn(0x7f08d4070fb0 msgr2=0x7f08d409dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 shutdown_connections 2024-09-15T01:54:45.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.123+0000 7f08dc15d640 1 -- 172.21.15.17:0/270741856 wait complete. 2024-09-15T01:54:45.142 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:45.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:54:45.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:54:45.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.247+0000 7f63d0646640 10 monclient: get_monmap_and_config 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.247+0000 7f63d0646640 10 monclient: build_initial_monmap 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.248+0000 7f63d0646640 10 monclient: monmap: 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:45.248816+0000 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:45.248816+0000 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:45.251 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:45.251 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:45.251 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:45.251 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:45.251 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.248+0000 7f63d0646640 1 Processor -- start 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.248+0000 7f63d0646640 1 -- start start 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.248+0000 7f63d0646640 10 monclient: init 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-a con 0x7f63c8068f20 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c80692f0 0x7f63c80696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-b con 0x7f63c80692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:45.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63c812e900 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80a1950 con 0x7f63c812e900 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c80a1670 con 0x7f63c8068f20 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80a0020 con 0x7f63c80692f0 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63d0646640 10 monclient(hunting): authenticate will time out at 945.245326s 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c80692f0 0x7f63c80696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40846/0 (socket says 172.21.15.17:40846) 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 learned_addr learned my addr 172.21.15.17:0/3186368664 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:45.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request con 0x7f63c80692f0 auth_method 0 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request con 0x7f63c8068f20 auth_method 0 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_done global_id 44610 payload 274 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c80692f0 msgr2=0x7f63c80696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c80692f0 0x7f63c80696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 >> v1:172.21.15.67:6789/0 conn(0x7f63c812e900 legacy=0x7f63c812ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient: _finish_hunting 0 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 monclient: found mon.noname-a 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:45.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80a1390 con 0x7f63c8068f20 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.249+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c80692f0 0x7f63c80696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.250+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 msgr2=0x7f63c81321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.250+0000 7f63ce3e2640 1 -- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 msgr2=0x7f63c81321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.250+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f63c4009910 tx=0x7f63c40091d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:45.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.250+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:45.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63ce3e2640 10 monclient: get_auth_request con 0x7f63c8068f20 auth_method 0 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63ce3e2640 0 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient: ms_handle_reset current mon v2:172.21.15.17:3300/0 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 msgr2=0x7f63c81321c0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.17:3300/0 conn(0x7f63c8068f20 0x7f63c81321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63b40034c0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.67:3300/0 conn(0x7f63b4007440 0x7f63b40098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient(hunting): picked mon.noname-c con 0x7f63b4007440 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b400a200 con 0x7f63b40034c0 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b400b400 con 0x7f63b4007440 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cd3e0640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.17:3300/0 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/3186368664 >> v2:172.21.15.67:3300/0 conn(0x7f63b4007440 0x7f63b40098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request con 0x7f63b4007440 auth_method 0 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:45.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.451+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_done global_id 44610 payload 293 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 1 -- 172.21.15.17:0/3186368664 >> v1:172.21.15.17:6789/0 conn(0x7f63b40034c0 legacy=0x7f63b4005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 10 monclient: _finish_hunting 0 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 1 monclient: found mon.noname-c 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:45.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.452+0000 7f63cdbe1640 1 -- 172.21.15.17:0/3186368664 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b400b630 con 0x7f63b4007440 2024-09-15T01:54:45.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cdbe1640 1 -- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 msgr2=0x7f63b40098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63b40098f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63bc006e70 tx=0x7f63bc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1837f37aa1c4ad0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00ee70 con 0x7f63b4007440 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63b40098f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63bc006e70 tx=0x7f63bc0090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f63bc0136c0 mon_map magic: 0 v1 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 10 monclient: dump: 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.464 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 10 monclient: _finish_auth 0 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0096b0 con 0x7f63b4007440 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63d0646640 5 monclient: authenticate success, global_id 44610 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63d0646640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:45.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63cd3e0640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63d0646640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.454+0000 7f63d0646640 4 set_mon_vals no callback set 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 10 monclient: shutdown 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 -- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 msgr2=0x7f63b40098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63b40098f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63bc006e70 tx=0x7f63bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63cd3e0640 1 -- 172.21.15.17:0/3186368664 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f63bc0136c0 con 0x7f63b4007440 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63cd3e0640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63cebe3640 1 -- 172.21.15.17:0/3186368664 reap_dead start 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 -- 172.21.15.17:0/3186368664 shutdown_connections 2024-09-15T01:54:45.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 --2- 172.21.15.17:0/3186368664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63b40098f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 -- 172.21.15.17:0/3186368664 >> 172.21.15.17:0/3186368664 conn(0x7f63c8070fb0 msgr2=0x7f63c80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 -- 172.21.15.17:0/3186368664 shutdown_connections 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.455+0000 7f63d0646640 1 -- 172.21.15.17:0/3186368664 wait complete. 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.456+0000 7f63d0646640 10 monclient: build_initial_monmap 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.456+0000 7f63d0646640 10 monclient: monmap: 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.467 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.456+0000 7f63d0646640 1 Processor -- start 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.456+0000 7f63d0646640 1 -- start start 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.456+0000 7f63d0646640 10 monclient: init 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63c81412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-b con 0x7f63b4007440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:45.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 0x7f63c812dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-a con 0x7f63c8141820 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c812e1a0 0x7f63c8097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): picked mon.noname-c con 0x7f63c812e1a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80a0750 con 0x7f63c8141820 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8130e00 con 0x7f63b4007440 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c809b930 con 0x7f63c812e1a0 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:45.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63d0646640 10 monclient(hunting): authenticate will time out at 945.453142s 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 0x7f63c812dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 0x7f63c812dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40850/0 (socket says 172.21.15.17:40850) 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c812e1a0 0x7f63c8097fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 -- 172.21.15.17:0/1513203571 learned_addr learned my addr 172.21.15.17:0/1513203571 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request con 0x7f63c8141820 auth_method 0 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): get_auth_request con 0x7f63c812e1a0 auth_method 0 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63c81412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request con 0x7f63b4007440 auth_method 0 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient(hunting): handle_auth_done global_id 44613 payload 824 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c812e1a0 msgr2=0x7f63c8097fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c812e1a0 0x7f63c8097fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 msgr2=0x7f63c81412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63c81412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient: _finish_hunting 0 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 monclient: found mon.noname-a 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cdbe1640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c8099b20 con 0x7f63c8141820 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63b4007440 0x7f63c81412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:54:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.457+0000 7f63cebe3640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c812e1a0 0x7f63c8097fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.459+0000 7f63cdbe1640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 msgr2=0x7f63c812dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.459+0000 7f63cdbe1640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 0x7f63c812dc60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00ea80 tx=0x7f63bc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc01b600 con 0x7f63c8141820 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: dump: 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 5 monclient: authenticate success, global_id 44613 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc01b7a0 con 0x7f63c8141820 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 monclient: _renew_subs 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c809b650 con 0x7f63c8141820 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc01bb60 con 0x7f63c8141820 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ba7fc640 10 monclient: dump: 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 monclient: _renew_subs 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c809a6f0 con 0x7f63c8141820 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63ccbdf640 4 set_mon_vals no callback set 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 0x7f63c80988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.460+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8063b90 con 0x7f63c8098520 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f63bc026440 con 0x7f63c8141820 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 0x7f63c80988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ba7fc640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63ac038740 0x7f63ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient: get_auth_request con 0x7f63c8098520 auth_method 0 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f63bc04da50 con 0x7f63c8141820 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63cebe3640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63ac038740 0x7f63ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63cebe3640 10 monclient: get_auth_request con 0x7f63ac038740 auth_method 0 2024-09-15T01:54:45.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_done global_id 5251 payload 824 2024-09-15T01:54:45.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.461+0000 7f63ce3e2640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63c811b860 con 0x7f63c8098520 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ce3e2640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 msgr2=0x7f63c80988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 0x7f63c80988f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009910 tx=0x7f63c4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63cebe3640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63ac038740 0x7f63ac03abf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63c8096fa0 tx=0x7f63c00023b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c402ae80 con 0x7f63c8098520 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:45.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 10 monclient: dump: 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f63c4036860 con 0x7f63c8098520 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 msgr2=0x7f63c80988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 0x7f63c80988f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009910 tx=0x7f63c4008040 comp rx=0 tx=0).stop 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 msgr2=0x7f63c80988f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.464+0000 7f63ba7fc640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8098520 0x7f63c80988f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63d0646640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63d0646640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 0x7f63c809c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63d0646640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8098e30 con 0x7f63c809c550 2024-09-15T01:54:45.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 0x7f63c809c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient: get_auth_request con 0x7f63c809c550 auth_method 0 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.470+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.471+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.471+0000 7f63ce3e2640 10 monclient(hunting): handle_auth_done global_id 5254 payload 824 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.471+0000 7f63ce3e2640 1 -- 172.21.15.17:0/1513203571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f63c80724a0 con 0x7f63c809c550 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ce3e2640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 msgr2=0x7f63c809c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ce3e2640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 0x7f63c809c950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f63c8071880 tx=0x7f63c4039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4045070 con 0x7f63c809c550 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 10 monclient: dump: 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.484 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f63c4036650 con 0x7f63c809c550 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 msgr2=0x7f63c809c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 0x7f63c809c950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f63c8071880 tx=0x7f63c4039000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 msgr2=0x7f63c809c950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:45.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.473+0000 7f63ba7fc640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c809c550 0x7f63c809c950 secure :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f63c8071880 tx=0x7f63c4039000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:45.501 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63ac038740 msgr2=0x7f63ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63ac038740 0x7f63ac03abf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63c8096fa0 tx=0x7f63c00023b0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 10 monclient: shutdown 2024-09-15T01:54:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 msgr2=0x7f63c812dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 --2- 172.21.15.17:0/1513203571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c8141820 0x7f63c812dc60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00ea80 tx=0x7f63bc019000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63cebe3640 1 -- 172.21.15.17:0/1513203571 reap_dead start 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63cebe3640 1 -- 172.21.15.17:0/1513203571 reap_dead start 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 shutdown_connections 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 >> 172.21.15.17:0/1513203571 conn(0x7f63c8070fb0 msgr2=0x7f63c8131080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 shutdown_connections 2024-09-15T01:54:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.474+0000 7f63d0646640 1 -- 172.21.15.17:0/1513203571 wait complete. 2024-09-15T01:54:45.504 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:45.504 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:54:45.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:54:45.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.599+0000 7f1f8e15d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.599+0000 7f1f8e15d640 10 monclient: build_initial_monmap 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.599+0000 7f1f8e15d640 10 monclient: monmap: 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:45.600706+0000 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:45.600706+0000 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:45.603 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.600+0000 7f1f8e15d640 1 Processor -- start 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.600+0000 7f1f8e15d640 1 -- start start 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.600+0000 7f1f8e15d640 10 monclient: init 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1f88068f20 0x7f1f881321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f88068f20 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f880692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 0x7f1f881393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f88136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:45.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f880a1950 con 0x7f1f880692f0 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f880a1670 con 0x7f1f88068f20 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f880a0020 con 0x7f1f88136fe0 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f8e15d640 10 monclient(hunting): authenticate will time out at 945.597218s 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 0x7f1f881393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1f88068f20 0x7f1f881321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f86ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1f880692f0 0x7f1f880696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48594/0 (socket says 172.21.15.17:48594) 2024-09-15T01:54:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f86ffd640 1 -- 172.21.15.17:0/1961170453 learned_addr learned my addr 172.21.15.17:0/1961170453 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): get_auth_request con 0x7f1f88136fe0 auth_method 0 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): get_auth_request con 0x7f1f88068f20 auth_method 0 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient(hunting): handle_auth_done global_id 44616 payload 274 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 -- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 msgr2=0x7f1f881393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 0x7f1f881393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 -- 172.21.15.17:0/1961170453 >> v1:172.21.15.17:6789/0 conn(0x7f1f880692f0 legacy=0x7f1f880696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f877fe640 1 -- 172.21.15.17:0/1961170453 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880a1390 con 0x7f1f88068f20 2024-09-15T01:54:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.601+0000 7f1f87fff640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 0x7f1f881393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f877fe640 1 -- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88068f20 msgr2=0x7f1f881321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f877fe640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88068f20 0x7f1f881321c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009880 tx=0x7f1f780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58950028da286850 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 1 -- 172.21.15.17:0/1961170453 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78032440 con 0x7f1f88068f20 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: dump: 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 1 -- 172.21.15.17:0/1961170453 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f780325a0 con 0x7f1f88068f20 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 1 -- 172.21.15.17:0/1961170453 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78032940 con 0x7f1f88068f20 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f8e15d640 5 monclient: authenticate success, global_id 44616 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f8e15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f8e15d640 4 set_mon_vals no callback set 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.604+0000 7f1f867fc640 10 monclient: dump: 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 10 monclient: shutdown 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 -- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88068f20 msgr2=0x7f1f881321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88068f20 0x7f1f881321c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009880 tx=0x7f1f780090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 -- 172.21.15.17:0/1961170453 shutdown_connections 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f88136fe0 0x7f1f881393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/1961170453 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88068f20 0x7f1f881321c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 -- 172.21.15.17:0/1961170453 >> 172.21.15.17:0/1961170453 conn(0x7f1f88070fb0 msgr2=0x7f1f880a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 -- 172.21.15.17:0/1961170453 shutdown_connections 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 1 -- 172.21.15.17:0/1961170453 wait complete. 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 10 monclient: build_initial_monmap 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.605+0000 7f1f8e15d640 10 monclient: monmap: 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 Processor -- start 2024-09-15T01:54:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 -- start start 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient: init 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f88068f20 0x7f1f881416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f88068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f880692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f88136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8811ccc0 con 0x7f1f88136fe0 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f880a0750 con 0x7f1f88068f20 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8809c9b0 con 0x7f1f880692f0 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f8e15d640 10 monclient(hunting): authenticate will time out at 945.602887s 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f86ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.606+0000 7f1f87fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34326/0 (socket says 172.21.15.17:34326) 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 1 -- 172.21.15.17:0/2735138228 learned_addr learned my addr 172.21.15.17:0/2735138228 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40858/0 (socket says 172.21.15.17:40858) 2024-09-15T01:54:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): get_auth_request con 0x7f1f880692f0 auth_method 0 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): get_auth_request con 0x7f1f88136fe0 auth_method 0 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f877fe640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f88068f20 0x7f1f881416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f877fe640 10 monclient(hunting): get_auth_request con 0x7f1f88068f20 auth_method 0 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient(hunting): handle_auth_done global_id 44619 payload 824 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 -- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 msgr2=0x7f1f88141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 -- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f88068f20 msgr2=0x7f1f881416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f88068f20 0x7f1f881416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 monclient: found mon.noname-a 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f87fff640 1 -- 172.21.15.17:0/2735138228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80009a20 con 0x7f1f88136fe0 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.607+0000 7f1f86ffd640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f87fff640 1 -- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 msgr2=0x7f1f88099c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f87fff640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009f10 tx=0x7f1f8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80014480 con 0x7f1f88136fe0 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: dump: 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8000f040 con 0x7f1f88136fe0 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 5 monclient: authenticate success, global_id 44619 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 10 monclient: _renew_subs 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78009390 con 0x7f1f88136fe0 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80014850 con 0x7f1f88136fe0 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f6f7fe640 10 monclient: dump: 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 10 monclient: _renew_subs 2024-09-15T01:54:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f85ffb640 4 set_mon_vals no callback set 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.609+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8809b770 con 0x7f1f88136fe0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1f80003020 con 0x7f1f88136fe0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f8e15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f8e15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f8e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f880724a0 con 0x7f1f88136fe0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f6f7fe640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 0x7f1f7403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.610+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f1f8004cb70 con 0x7f1f88136fe0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.611+0000 7f1f877fe640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 0x7f1f7403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.611+0000 7f1f877fe640 10 monclient: get_auth_request con 0x7f1f74038740 auth_method 0 2024-09-15T01:54:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.613+0000 7f1f877fe640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 0x7f1f7403abf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009850 tx=0x7f1f78032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.615+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1f800112c0 con 0x7f1f88136fe0 2024-09-15T01:54:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.615+0000 7f1f6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.615+0000 7f1f6f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:45.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.840+0000 7f1f8e15d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:54:45.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.840+0000 7f1f8e15d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:54:45.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.840+0000 7f1f8e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:45.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.840+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1f88131880 con 0x7f1f88136fe0 2024-09-15T01:54:45.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.841+0000 7f1f6f7fe640 1 -- 172.21.15.17:0/2735138228 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1f80012340 con 0x7f1f88136fe0 2024-09-15T01:54:45.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.841+0000 7f1f6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:54:45.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.841+0000 7f1f6f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=1471503653,config=3140514009,health=1791238353,kv=922137073,logm=1654279859}) 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=200703689,config=3140514009,health=1535303502,kv=1529054990,logm=1769290869}) 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=1471503653,config=3140514009,health=1791238353,kv=922137073,logm=1654279859}) 2024-09-15T01:54:45.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=1677533968,config=3140514009,health=1535303502,kv=1529054990,logm=524547301}) 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103225977}) 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544717038}) 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.842+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103225977}) 2024-09-15T01:54:45.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775558239}) 2024-09-15T01:54:45.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 msgr2=0x7f1f7403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 0x7f1f7403abf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009850 tx=0x7f1f78032000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 10 monclient: shutdown 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 msgr2=0x7f1f88099c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009f10 tx=0x7f1f8000b040 comp rx=0 tx=0).stop 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 shutdown_connections 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1f74038740 0x7f1f7403abf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1f88136fe0 0x7f1f88099c90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1f880692f0 0x7f1f88141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 --2- 172.21.15.17:0/2735138228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f88068f20 0x7f1f881416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 >> 172.21.15.17:0/2735138228 conn(0x7f1f88070fb0 msgr2=0x7f1f8809e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 shutdown_connections 2024-09-15T01:54:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:45.843+0000 7f1f8e15d640 1 -- 172.21.15.17:0/2735138228 wait complete. 2024-09-15T01:54:45.848 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.848 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705482481}) 2024-09-15T01:54:45.849 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391431251}) 2024-09-15T01:54:45.849 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.849 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705482481}) 2024-09-15T01:54:45.849 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.845+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725047770}) 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754647186}) 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723637593}) 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754647186}) 2024-09-15T01:54:45.850 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.848+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425140445}) 2024-09-15T01:54:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.850+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v77) v1 ==== 143+0+0 (secure 0 0 0) 0x7f450c047360 con 0x7f44fc003ad0 2024-09-15T01:54:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.850+0000 7f4505ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.850+0000 7f4505ffb640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-15T01:54:45.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:54:45.929 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.929 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974744545}) 2024-09-15T01:54:45.929 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970403066}) 2024-09-15T01:54:45.929 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974744545}) 2024-09-15T01:54:45.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.927+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817377543}) 2024-09-15T01:54:45.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660789223}) 2024-09-15T01:54:45.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000467303}) 2024-09-15T01:54:45.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660789223}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103623140}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595166556}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629542820}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595166556}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.928+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452197018}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173420253}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3460421409}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173420253}) 2024-09-15T01:54:45.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.929+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392579362}) 2024-09-15T01:54:45.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544660758}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2515898678}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544660758}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893062134}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.930+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362588852}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806036593}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362588852}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277098706}) 2024-09-15T01:54:45.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579422728}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029130429}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579422728}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.931+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385682936}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3849601380,mdsmap=1228700967,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2031851056}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=4265985035,mdsmap=1228700967,mgr=1655507991,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2790965718}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3849601380,mdsmap=1228700967,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2031851056}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.932+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=1956610551,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2579346497}) 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mgrstat=3242478683,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2868232970,osd_pg_creating=2506423872,osd_snap=1063953655,osdmap=233557072}) 2024-09-15T01:54:45.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=84} crc {mgrstat=3242478683,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2170749948,osd_pg_creating=2506423872,osd_snap=2673976383,osdmap=2060528893}) 2024-09-15T01:54:45.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:45.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mgrstat=3242478683,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2868232970,osd_pg_creating=2506423872,osd_snap=1063953655,osdmap=233557072}) 2024-09-15T01:54:45.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:45.933+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mgrstat=3242478683,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1149731095,osd_pg_creating=2506423872,osd_snap=1063953654,osdmap=3487829275}) 2024-09-15T01:54:45.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.956+0000 7f451819d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:45.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.956+0000 7f451819d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:45.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.956+0000 7f451819d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:45.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.956+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f451004bf60 con 0x7f44fc003ad0 2024-09-15T01:54:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.957+0000 7f4505ffb640 1 -- 172.21.15.67:0/2008874027 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v77) v1 ==== 145+0+0 (secure 0 0 0) 0x7f450c03e3a0 con 0x7f44fc003ad0 2024-09-15T01:54:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.957+0000 7f4505ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T01:54:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.957+0000 7f4505ffb640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-15T01:54:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.7' does not appear in the crush map 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 msgr2=0x7f44f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 0x7f44f403abf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4500001ed0 tx=0x7f4500009040 comp rx=0 tx=0).stop 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 10 monclient: shutdown 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 msgr2=0x7f4510140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f450c02a890 tx=0x7f450c035000 comp rx=0 tx=0).stop 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 shutdown_connections 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f4038740 0x7f44f403abf0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f451011e930 0x7f451011ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510131c80 0x7f4510132050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 --2- 172.21.15.67:0/2008874027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44fc003ad0 0x7f4510140d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:45.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 >> 172.21.15.67:0/2008874027 conn(0x7f4510071330 msgr2=0x7f45100a3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:45.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 shutdown_connections 2024-09-15T01:54:45.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:45.959+0000 7f451819d640 1 -- 172.21.15.67:0/2008874027 wait complete. 2024-09-15T01:54:45.971 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm host1 2024-09-15T01:54:46.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.045+0000 7fba3e59d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:46.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.045+0000 7fba3e59d640 10 monclient: build_initial_monmap 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.045+0000 7fba3e59d640 10 monclient: monmap: 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:46.046345+0000 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:46.046345+0000 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:46.048 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:46.049 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:46.049 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.045+0000 7fba3e59d640 1 Processor -- start 2024-09-15T01:54:46.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 -- start start 2024-09-15T01:54:46.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient: init 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba38098140 0x7fba381321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-a con 0x7fba38098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba38098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-b con 0x7fba380a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba38098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38098ef0 con 0x7fba38098a50 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba380a1730 con 0x7fba38098140 2024-09-15T01:54:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba380a0b80 con 0x7fba380a0370 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3e59d640 10 monclient(hunting): authenticate will time out at 918.515610s 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba38098140 0x7fba381321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba38098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3cb3a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fba38098a50 0x7fba380695c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:41210/0 (socket says 172.21.15.67:41210) 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba3cb3a640 1 -- 172.21.15.67:0/2434936903 learned_addr learned my addr 172.21.15.67:0/2434936903 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): get_auth_request con 0x7fba38098140 auth_method 0 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): get_auth_request con 0x7fba380a0370 auth_method 0 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.046+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 10 monclient(hunting): handle_auth_done global_id 44622 payload 274 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 1 -- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 msgr2=0x7fba38098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba38098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 1 -- 172.21.15.67:0/2434936903 >> v1:172.21.15.17:6789/0 conn(0x7fba38098a50 legacy=0x7fba380695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 1 monclient: found mon.noname-a 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba37fff640 1 -- 172.21.15.67:0/2434936903 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380a00b0 con 0x7fba38098140 2024-09-15T01:54:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.047+0000 7fba377fe640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba38098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.049+0000 7fba37fff640 1 -- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098140 msgr2=0x7fba381321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.049+0000 7fba37fff640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098140 0x7fba381321e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fba380a5340 tx=0x7fba280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d35370cac8ffd8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 1 -- 172.21.15.67:0/2434936903 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2802adb0 con 0x7fba38098140 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: dump: 2024-09-15T01:54:46.061 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:46.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 1 -- 172.21.15.67:0/2434936903 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28032440 con 0x7fba38098140 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 5 monclient: authenticate success, global_id 44622 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 1 -- 172.21.15.67:0/2434936903 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba280327e0 con 0x7fba38098140 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 4 set_mon_vals no callback set 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba36ffd640 10 monclient: dump: 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 10 monclient: shutdown 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 -- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098140 msgr2=0x7fba381321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098140 0x7fba381321e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fba380a5340 tx=0x7fba280090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 -- 172.21.15.67:0/2434936903 shutdown_connections 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba38098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 --2- 172.21.15.67:0/2434936903 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098140 0x7fba381321e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 -- 172.21.15.67:0/2434936903 >> 172.21.15.67:0/2434936903 conn(0x7fba380a5780 msgr2=0x7fba380a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 -- 172.21.15.67:0/2434936903 shutdown_connections 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.050+0000 7fba3e59d640 1 -- 172.21.15.67:0/2434936903 wait complete. 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.051+0000 7fba3e59d640 10 monclient: build_initial_monmap 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.051+0000 7fba3e59d640 10 monclient: monmap: 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.051+0000 7fba3e59d640 1 Processor -- start 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.051+0000 7fba3e59d640 1 -- start start 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.051+0000 7fba3e59d640 10 monclient: init 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-b con 0x7fba38098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 0x7fba38130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-a con 0x7fba38098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba3811ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): picked mon.noname-c con 0x7fba380a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3811e150 con 0x7fba38098a50 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38142010 con 0x7fba38098140 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3809f140 con 0x7fba380a0370 2024-09-15T01:54:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3e59d640 10 monclient(hunting): authenticate will time out at 918.521388s 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57844/0 (socket says 172.21.15.67:57844) 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 -- 172.21.15.67:0/1154093919 learned_addr learned my addr 172.21.15.67:0/1154093919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): get_auth_request con 0x7fba38098140 auth_method 0 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba377fe640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 0x7fba38130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient(hunting): handle_auth_done global_id 5260 payload 824 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 -- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 msgr2=0x7fba3811ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba3cb3a640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba3811ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba3811ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 -- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 msgr2=0x7fba38130d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 0x7fba38130d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 monclient: found mon.noname-b 2024-09-15T01:54:46.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba37fff640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009390 con 0x7fba38098140 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba377fe640 10 monclient: get_auth_request con 0x7fba38098a50 auth_method 0 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.052+0000 7fba377fe640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 0x7fba38130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.054+0000 7fba37fff640 1 -- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 msgr2=0x7fba3809e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba37fff640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fba28009880 tx=0x7fba28032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28031d50 con 0x7fba38098140 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: dump: 2024-09-15T01:54:46.070 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:46.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2803c400 con 0x7fba38098140 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 5 monclient: authenticate success, global_id 5260 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 10 monclient: _renew_subs 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28009390 con 0x7fba38098140 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2803c7a0 con 0x7fba38098140 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba2ffff640 10 monclient: dump: 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba367fc640 4 set_mon_vals no callback set 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 10 monclient: _renew_subs 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:46.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.055+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3811e430 con 0x7fba38098140 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba3e59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba3e59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba3e59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fba2803cc10 con 0x7fba38098140 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3804bf60 con 0x7fba38098140 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba2ffff640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 0x7fba1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fba2806cb10 con 0x7fba38098140 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba377fe640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 0x7fba1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.056+0000 7fba377fe640 10 monclient: get_auth_request con 0x7fba18038740 auth_method 0 2024-09-15T01:54:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.059+0000 7fba377fe640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 0x7fba1803abf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fba20009360 tx=0x7fba20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.061+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fba2803b3a0 con 0x7fba38098140 2024-09-15T01:54:46.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.061+0000 7fba2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.061+0000 7fba2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:46.203 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:46.204 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=2747576344}) 2024-09-15T01:54:46.204 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=45} crc {osdmap=1715041789}) 2024-09-15T01:54:46.204 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:54:46.204 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=2747576344}) 2024-09-15T01:54:46.204 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:54:46.201+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=3241834663}) 2024-09-15T01:54:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.227+0000 7fba3e59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.227+0000 7fba3e59d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.227+0000 7fba3e59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:46.227+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7fba38098ef0 con 0x7fba38098140 2024-09-15T01:54:46.858 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:54:46.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 10 monclient: get_monmap_and_config 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 10 monclient: build_initial_monmap 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 10 monclient: monmap: 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.966 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:46.965110+0000 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:46.965110+0000 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:46.967 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 1 Processor -- start 2024-09-15T01:54:46.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 1 -- start start 2024-09-15T01:54:46.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.964+0000 7efed1354640 10 monclient: init 2024-09-15T01:54:46.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:46.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efecc096c10 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 0x7efecc096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): picked mon.noname-c con 0x7efecc09d3b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efecc097520 0x7efecc135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): picked mon.noname-a con 0x7efecc097520 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc0983b0 con 0x7efecc096c10 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc099260 con 0x7efecc097520 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc097920 con 0x7efecc09d3b0 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efed1354640 10 monclient(hunting): authenticate will time out at 946.961450s 2024-09-15T01:54:46.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efecc097520 0x7efecc135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efecc097520 0x7efecc135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40874/0 (socket says 172.21.15.17:40874) 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 -- 172.21.15.17:0/2336571025 learned_addr learned my addr 172.21.15.17:0/2336571025 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): get_auth_request con 0x7efecc097520 auth_method 0 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efeca7fc640 1 --2- 172.21.15.17:0/2336571025 >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 0x7efecc096fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient(hunting): handle_auth_done global_id 44625 payload 274 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 -- 172.21.15.17:0/2336571025 >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 msgr2=0x7efecc096fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 --2- 172.21.15.17:0/2336571025 >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 0x7efecc096fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 -- 172.21.15.17:0/2336571025 >> v1:172.21.15.67:6789/0 conn(0x7efecc096c10 legacy=0x7efecc099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:46.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:46.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efecb7fe640 1 -- 172.21.15.17:0/2336571025 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc097a60 con 0x7efecc097520 2024-09-15T01:54:46.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efeca7fc640 10 monclient: get_auth_request con 0x7efecc09d3b0 auth_method 0 2024-09-15T01:54:46.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.965+0000 7efeca7fc640 1 --2- 172.21.15.17:0/2336571025 >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 0x7efecc096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:46.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.968+0000 7efecb7fe640 1 -- 172.21.15.17:0/2336571025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc097520 msgr2=0x7efecc135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.968+0000 7efecb7fe640 1 --2- 172.21.15.17:0/2336571025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc097520 0x7efecc135b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efecc065ca0 tx=0x7efec400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65acd14b530f9137 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 1 -- 172.21.15.17:0/2336571025 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4009610 con 0x7efecc097520 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: dump: 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.979 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 5 monclient: authenticate success, global_id 44625 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 1 -- 172.21.15.17:0/2336571025 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec4009770 con 0x7efecc097520 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 4 set_mon_vals no callback set 2024-09-15T01:54:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 1 -- 172.21.15.17:0/2336571025 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4009b10 con 0x7efecc097520 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efec9ffb640 10 monclient: dump: 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.981 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 10 monclient: shutdown 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 -- 172.21.15.17:0/2336571025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc097520 msgr2=0x7efecc135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 --2- 172.21.15.17:0/2336571025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc097520 0x7efecc135b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efecc065ca0 tx=0x7efec400ba50 comp rx=0 tx=0).stop 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 -- 172.21.15.17:0/2336571025 shutdown_connections 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 --2- 172.21.15.17:0/2336571025 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc097520 0x7efecc135b80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 --2- 172.21.15.17:0/2336571025 >> v2:172.21.15.67:3300/0 conn(0x7efecc09d3b0 0x7efecc096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 -- 172.21.15.17:0/2336571025 >> 172.21.15.17:0/2336571025 conn(0x7efecc0a1000 msgr2=0x7efecc0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 -- 172.21.15.17:0/2336571025 shutdown_connections 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.969+0000 7efed1354640 1 -- 172.21.15.17:0/2336571025 wait complete. 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.970+0000 7efed1354640 10 monclient: build_initial_monmap 2024-09-15T01:54:46.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.970+0000 7efed1354640 10 monclient: monmap: 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.970+0000 7efed1354640 1 Processor -- start 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.970+0000 7efed1354640 1 -- start start 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.970+0000 7efed1354640 10 monclient: init 2024-09-15T01:54:46.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): picked mon.noname-a con 0x7efecc096c10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc097520 0x7efecc118910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): picked mon.noname-b con 0x7efecc097520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efecc09d3b0 0x7efecc118e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): picked mon.noname-c con 0x7efecc09d3b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc11c190 con 0x7efecc096c10 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc098e50 con 0x7efecc097520 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:46.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc11a430 con 0x7efecc09d3b0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efed1354640 10 monclient(hunting): authenticate will time out at 946.967398s 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40878/0 (socket says 172.21.15.17:40878) 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 -- 172.21.15.17:0/3528470482 learned_addr learned my addr 172.21.15.17:0/3528470482 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): get_auth_request con 0x7efecc096c10 auth_method 0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efecc09d3b0 0x7efecc118e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): get_auth_request con 0x7efecc09d3b0 auth_method 0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc097520 0x7efecc118910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): get_auth_request con 0x7efecc097520 auth_method 0 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efeca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient(hunting): handle_auth_done global_id 44628 payload 824 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 -- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efecc09d3b0 msgr2=0x7efecc118e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efecc09d3b0 0x7efecc118e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 -- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc097520 msgr2=0x7efecc118910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc097520 0x7efecc118910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 monclient: found mon.noname-a 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.971+0000 7efecaffd640 1 -- 172.21.15.17:0/3528470482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec400bcf0 con 0x7efecc096c10 2024-09-15T01:54:46.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efecaffd640 1 -- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 msgr2=0x7efecc12ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efecaffd640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b5e0 tx=0x7efeb40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb400f720 con 0x7efecc096c10 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: dump: 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.988 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb400f8a0 con 0x7efecc096c10 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb400fc60 con 0x7efecc096c10 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 5 monclient: authenticate success, global_id 44628 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 10 monclient: _renew_subs 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc11a900 con 0x7efecc096c10 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec97fa640 4 set_mon_vals no callback set 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:46.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efec2ffd640 10 monclient: dump: 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 10 monclient: _renew_subs 2024-09-15T01:54:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.974+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc12e410 con 0x7efecc096c10 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efeb4011610 con 0x7efecc096c10 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efec2ffd640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 0x7efeac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efed1354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efed1354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efed1354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7efeb404bab0 con 0x7efecc096c10 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc11c910 con 0x7efecc096c10 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efeca7fc640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 0x7efeac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:46.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.975+0000 7efeca7fc640 10 monclient: get_auth_request con 0x7efeac038740 auth_method 0 2024-09-15T01:54:46.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.978+0000 7efeca7fc640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 0x7efeac03abf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efebc00acc0 tx=0x7efebc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:46.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.980+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efeb4017880 con 0x7efecc096c10 2024-09-15T01:54:46.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.980+0000 7efec2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:46.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:46.980+0000 7efec2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efed1354640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efed1354640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efed1354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efecc11c3d0 con 0x7efecc096c10 2024-09-15T01:54:47.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efec2ffd640 1 -- 172.21.15.17:0/3528470482 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7efeb4028610 con 0x7efecc096c10 2024-09-15T01:54:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efec2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:47.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.209+0000 7efec2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:47.212 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 msgr2=0x7efeac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 0x7efeac03abf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efebc00acc0 tx=0x7efebc008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 10 monclient: shutdown 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 msgr2=0x7efecc12ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b5e0 tx=0x7efeb40090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 shutdown_connections 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeac038740 0x7efeac03abf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efecc09d3b0 0x7efecc118e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc097520 0x7efecc118910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 --2- 172.21.15.17:0/3528470482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efecc096c10 0x7efecc12ef70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 >> 172.21.15.17:0/3528470482 conn(0x7efecc0a1000 msgr2=0x7efecc069b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.211+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 shutdown_connections 2024-09-15T01:54:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.212+0000 7efed1354640 1 -- 172.21.15.17:0/3528470482 wait complete. 2024-09-15T01:54:47.224 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:54:47.224 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:54:47.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:54:47.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:54:47.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.319+0000 7fa87859d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:47.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.319+0000 7fa87859d640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.319+0000 7fa87859d640 10 monclient: monmap: 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:47.320323+0000 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:47.320323+0000 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.322 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:47.323 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:47.323 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:47.323 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:47.323 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:47.323 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 Processor -- start 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 -- start start 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient: init 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 0x7fa870131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-b con 0x7fa8701319a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8701322b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa87009f000 0x7fa870068e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-c con 0x7fa87009f000 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8700a1530 con 0x7fa8701322b0 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa870132680 con 0x7fa87009f000 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87009fca0 con 0x7fa8701319a0 2024-09-15T01:54:47.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa87859d640 10 monclient(hunting): authenticate will time out at 947.316805s 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa876339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 0x7fa870131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40880/0 (socket says 172.21.15.17:40880) 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 1 -- 172.21.15.17:0/685396593 learned_addr learned my addr 172.21.15.17:0/685396593 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 10 monclient(hunting): get_auth_request con 0x7fa8701322b0 auth_method 0 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa875b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa876339640 10 monclient(hunting): get_auth_request con 0x7fa8701319a0 auth_method 0 2024-09-15T01:54:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa876339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa876339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.320+0000 7fa876339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876b3a640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.67:3300/0 conn(0x7fa87009f000 0x7fa870068e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876b3a640 10 monclient(hunting): get_auth_request con 0x7fa87009f000 auth_method 0 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient(hunting): handle_auth_done global_id 64124 payload 274 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 -- 172.21.15.17:0/685396593 >> v2:172.21.15.67:3300/0 conn(0x7fa87009f000 msgr2=0x7fa870068e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.67:3300/0 conn(0x7fa87009f000 0x7fa870068e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 -- 172.21.15.17:0/685396593 >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 msgr2=0x7fa87009eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 monclient: found mon.noname-b 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa876339640 1 -- 172.21.15.17:0/685396593 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8700a00e0 con 0x7fa8701319a0 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.321+0000 7fa875b38640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa876339640 1 -- 172.21.15.17:0/685396593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 msgr2=0x7fa870131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa876339640 1 --2- 172.21.15.17:0/685396593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 0x7fa870131d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa8700a0030 tx=0x7fa860009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede53f4c13abf620 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 1 -- 172.21.15.17:0/685396593 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860007760 con 0x7fa8701319a0 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: dump: 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.335 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 1 -- 172.21.15.17:0/685396593 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86002d050 con 0x7fa8701319a0 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 5 monclient: authenticate success, global_id 64124 2024-09-15T01:54:47.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 1 -- 172.21.15.17:0/685396593 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860007af0 con 0x7fa8701319a0 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 4 set_mon_vals no callback set 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa875337640 10 monclient: dump: 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.337 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 10 monclient: shutdown 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 -- 172.21.15.17:0/685396593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 msgr2=0x7fa870131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 --2- 172.21.15.17:0/685396593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 0x7fa870131d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa8700a0030 tx=0x7fa860009780 comp rx=0 tx=0).stop 2024-09-15T01:54:47.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 -- 172.21.15.17:0/685396593 shutdown_connections 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.67:3300/0 conn(0x7fa87009f000 0x7fa870068e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 --2- 172.21.15.17:0/685396593 >> v2:172.21.15.17:3300/0 conn(0x7fa8701322b0 0x7fa87009eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 --2- 172.21.15.17:0/685396593 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8701319a0 0x7fa870131d70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 -- 172.21.15.17:0/685396593 >> 172.21.15.17:0/685396593 conn(0x7fa8700a5000 msgr2=0x7fa8700a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 -- 172.21.15.17:0/685396593 shutdown_connections 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.324+0000 7fa87859d640 1 -- 172.21.15.17:0/685396593 wait complete. 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.325+0000 7fa87859d640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.325+0000 7fa87859d640 10 monclient: monmap: 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.325+0000 7fa87859d640 1 Processor -- start 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.325+0000 7fa87859d640 1 -- start start 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.325+0000 7fa87859d640 10 monclient: init 2024-09-15T01:54:47.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 0x7fa87009ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-c con 0x7fa87009f000 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8701319a0 0x7fa8701308e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-b con 0x7fa8701319a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8701322b0 0x7fa87011b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8701322b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87011dd70 con 0x7fa8701322b0 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87009bc90 con 0x7fa8701319a0 2024-09-15T01:54:47.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87011da00 con 0x7fa87009f000 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa87859d640 10 monclient(hunting): authenticate will time out at 947.322217s 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 0x7fa87009ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8701322b0 0x7fa87011b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 0x7fa87009ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34340/0 (socket says 172.21.15.17:34340) 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 -- 172.21.15.17:0/2091503941 learned_addr learned my addr 172.21.15.17:0/2091503941 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): get_auth_request con 0x7fa87009f000 auth_method 0 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): get_auth_request con 0x7fa8701322b0 auth_method 0 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8701319a0 0x7fa8701308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient(hunting): handle_auth_done global_id 64127 payload 824 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8701319a0 msgr2=0x7fa8701308e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8701319a0 0x7fa8701308e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8701322b0 msgr2=0x7fa87011b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8701322b0 0x7fa87011b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 monclient: found mon.noname-c 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876339640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864009450 con 0x7fa87009f000 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa875b38640 10 monclient: get_auth_request con 0x7fa8701319a0 auth_method 0 2024-09-15T01:54:47.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8701319a0 0x7fa8701308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:54:47.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.326+0000 7fa876b3a640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa8701322b0 0x7fa87011b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:47.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.328+0000 7fa876339640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 msgr2=0x7fa87009ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.328+0000 7fa876339640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 0x7fa87009ac00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa860009e90 tx=0x7fa860037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860031890 con 0x7fa87009f000 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: dump: 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.350 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa860040070 con 0x7fa87009f000 2024-09-15T01:54:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 5 monclient: authenticate success, global_id 64127 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 10 monclient: _renew_subs 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa860009a20 con 0x7fa87009f000 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860031c60 con 0x7fa87009f000 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa86e7fc640 10 monclient: dump: 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.353 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa874b36640 4 set_mon_vals no callback set 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 10 monclient: _renew_subs 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.329+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87011cb10 con 0x7fa87009f000 2024-09-15T01:54:47.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa860042030 con 0x7fa87009f000 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa87859d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa87859d640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 0x7fa87011bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa870063b00 con 0x7fa870130e20 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 0x7fa87011bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient: get_auth_request con 0x7fa870130e20 auth_method 0 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa86e7fc640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa85c038790 0x7fa85c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fa86006d640 con 0x7fa87009f000 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 10 monclient(hunting): handle_auth_done global_id 44637 payload 824 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa875b38640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8700a4500 con 0x7fa870130e20 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa876b3a640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa85c038790 0x7fa85c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.330+0000 7fa876b3a640 10 monclient: get_auth_request con 0x7fa85c038790 auth_method 0 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa875b38640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 msgr2=0x7fa87011bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 0x7fa87011bba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa864008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86400ed80 con 0x7fa870130e20 2024-09-15T01:54:47.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 10 monclient: dump: 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.357 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa86401c400 con 0x7fa870130e20 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 msgr2=0x7fa87011bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa876b3a640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa85c038790 0x7fa85c03ac40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fa0 tx=0x7fa868008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 0x7fa87011bba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa864008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 msgr2=0x7fa87011bba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.333+0000 7fa86e7fc640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870130e20 0x7fa87011bba0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa864008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa87859d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa87859d640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 0x7fa870096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa87859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8701311f0 con 0x7fa87011c0e0 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 0x7fa870096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient: get_auth_request con 0x7fa87011c0e0 auth_method 0 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 10 monclient(hunting): handle_auth_done global_id 44640 payload 824 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.339+0000 7fa875b38640 1 -- 172.21.15.17:0/2091503941 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa87011c4e0 con 0x7fa87011c0e0 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.341+0000 7fa875b38640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 msgr2=0x7fa870096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.341+0000 7fa875b38640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 0x7fa870096a10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa86401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864006250 con 0x7fa87011c0e0 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 10 monclient: dump: 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.360 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7fa86401ce90 con 0x7fa87011c0e0 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 msgr2=0x7fa870096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 0x7fa870096a10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa86401c000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 msgr2=0x7fa870096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa86e7fc640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa87011c0e0 0x7fa870096a10 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa87009b380 tx=0x7fa86401c000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:47.362 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.363 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:47.364 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.365 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:47.366 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.367 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:47.368 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:47.370 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.372 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:47.373 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.375 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.376 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.377 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:47.377 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:47.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa85c038790 msgr2=0x7fa85c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa85c038790 0x7fa85c03ac40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fa0 tx=0x7fa868008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 10 monclient: shutdown 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 msgr2=0x7fa87009ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 --2- 172.21.15.17:0/2091503941 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa87009f000 0x7fa87009ac00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa860009e90 tx=0x7fa860037000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa876b3a640 1 -- 172.21.15.17:0/2091503941 reap_dead start 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa876b3a640 1 -- 172.21.15.17:0/2091503941 reap_dead start 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 shutdown_connections 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 >> 172.21.15.17:0/2091503941 conn(0x7fa8700a5000 msgr2=0x7fa870134bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 shutdown_connections 2024-09-15T01:54:47.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.342+0000 7fa87859d640 1 -- 172.21.15.17:0/2091503941 wait complete. 2024-09-15T01:54:47.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:47.380 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:54:47.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.474+0000 7fcc0a289640 10 monclient: get_monmap_and_config 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.474+0000 7fcc0a289640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.474+0000 7fcc0a289640 10 monclient: monmap: 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:47.475544+0000 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:47.475544+0000 2024-09-15T01:54:47.477 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 1 Processor -- start 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 1 -- start start 2024-09-15T01:54:47.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 10 monclient: init 2024-09-15T01:54:47.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T01:54:47.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-b con 0x7fcc04131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcc040a0280 0x7fcc04131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.475+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-a con 0x7fcc040a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc0409f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc040a0040 con 0x7fcc0409f1e0 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0409bff0 con 0x7fcc040a0280 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040a14a0 con 0x7fcc04131b20 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc0a289640 10 monclient(hunting): authenticate will time out at 947.472047s 2024-09-15T01:54:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcc040a0280 0x7fcc04131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcc040a0280 0x7fcc04131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40914/0 (socket says 172.21.15.17:40914) 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34342/0 (socket says 172.21.15.17:34342) 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 -- 172.21.15.17:0/3185396830 learned_addr learned my addr 172.21.15.17:0/3185396830 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): get_auth_request con 0x7fcc040a0280 auth_method 0 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): get_auth_request con 0x7fcc04131b20 auth_method 0 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient(hunting): handle_auth_done global_id 44643 payload 274 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 -- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 msgr2=0x7fcc0409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 -- 172.21.15.17:0/3185396830 >> v1:172.21.15.67:6789/0 conn(0x7fcc0409f1e0 legacy=0x7fcc04068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 monclient: found mon.noname-a 2024-09-15T01:54:47.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc037fe640 1 -- 172.21.15.17:0/3185396830 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04132430 con 0x7fcc040a0280 2024-09-15T01:54:47.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.476+0000 7fcc03fff640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:47.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc037fe640 1 -- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 msgr2=0x7fcc04131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc037fe640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc04131ef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009910 tx=0x7fcbf80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ffdd171619f9ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 1 -- 172.21.15.17:0/3185396830 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8032440 con 0x7fcc040a0280 2024-09-15T01:54:47.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: dump: 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.490 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc0a289640 5 monclient: authenticate success, global_id 44643 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 1 -- 172.21.15.17:0/3185396830 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf80325a0 con 0x7fcc040a0280 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc0a289640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc0a289640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 1 -- 172.21.15.17:0/3185396830 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8032980 con 0x7fcc040a0280 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc0a289640 4 set_mon_vals no callback set 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.479+0000 7fcc02ffd640 10 monclient: dump: 2024-09-15T01:54:47.491 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 10 monclient: shutdown 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 -- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 msgr2=0x7fcc04131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc04131ef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009910 tx=0x7fcbf80091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 -- 172.21.15.17:0/3185396830 shutdown_connections 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc04131ef0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 --2- 172.21.15.17:0/3185396830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc0409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 -- 172.21.15.17:0/3185396830 >> 172.21.15.17:0/3185396830 conn(0x7fcc04071270 msgr2=0x7fcc040a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 -- 172.21.15.17:0/3185396830 shutdown_connections 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 1 -- 172.21.15.17:0/3185396830 wait complete. 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.480+0000 7fcc0a289640 10 monclient: monmap: 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 Processor -- start 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 -- start start 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient: init 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0409f1e0 0x7fcc04141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0409f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc0412ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-a con 0x7fcc040a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc04098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): picked mon.noname-c con 0x7fcc04131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040a09b0 con 0x7fcc040a0280 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04131910 con 0x7fcc0409f1e0 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0411c310 con 0x7fcc04131b20 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc0a289640 10 monclient(hunting): authenticate will time out at 947.477798s 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc0412ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc08826640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc04098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc0412ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40916/0 (socket says 172.21.15.17:40916) 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 1 -- 172.21.15.17:0/1636275665 learned_addr learned my addr 172.21.15.17:0/1636275665 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): get_auth_request con 0x7fcc040a0280 auth_method 0 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc08826640 10 monclient(hunting): get_auth_request con 0x7fcc04131b20 auth_method 0 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc08826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc08826640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc08826640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.481+0000 7fcc037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0409f1e0 0x7fcc04141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): get_auth_request con 0x7fcc0409f1e0 auth_method 0 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-15T01:54:47.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0409f1e0 msgr2=0x7fcc04141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0409f1e0 0x7fcc04141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 msgr2=0x7fcc0412ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc0412ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 monclient: found mon.noname-c 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc08826640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf4009390 con 0x7fcc04131b20 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.482+0000 7fcc037fe640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc040a0280 0x7fcc0412ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc08826640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 msgr2=0x7fcc04098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc08826640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc04098df0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c7e0 tx=0x7fcbfc0025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00a850 con 0x7fcc04131b20 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: dump: 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc00a9d0 con 0x7fcc04131b20 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 5 monclient: authenticate success, global_id 64133 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 10 monclient: _renew_subs 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf8009450 con 0x7fcc04131b20 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00ad90 con 0x7fcc04131b20 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcbebfff640 10 monclient: dump: 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc027fc640 4 set_mon_vals no callback set 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 10 monclient: _renew_subs 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.484+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0409b4c0 con 0x7fcc04131b20 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc0a289640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc0a289640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 0x7fcc0412f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0411c1d0 con 0x7fcc041417b0 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcbfc012630 con 0x7fcc04131b20 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 0x7fcc0412f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient: get_auth_request con 0x7fcc041417b0 auth_method 0 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcbebfff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcbf0038770 0x7fcbf003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fcbfc04c340 con 0x7fcc04131b20 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 10 monclient(hunting): handle_auth_done global_id 64136 payload 824 2024-09-15T01:54:47.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.485+0000 7fcc03fff640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc040724a0 con 0x7fcc041417b0 2024-09-15T01:54:47.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.486+0000 7fcc037fe640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcbf0038770 0x7fcbf003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.486+0000 7fcc037fe640 10 monclient: get_auth_request con 0x7fcbf0038770 auth_method 0 2024-09-15T01:54:47.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.490+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v78) v1 ==== 124+0+0 (secure 0 0 0) 0x7fba38098ef0 con 0x7fba38098140 2024-09-15T01:54:47.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.490+0000 7fba2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:47.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.490+0000 7fba2ffff640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcc03fff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 msgr2=0x7fcc0412f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcc03fff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 0x7fcc0412f450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400fe90 con 0x7fcc041417b0 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 10 monclient: dump: 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcbf4014650 con 0x7fcc041417b0 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 msgr2=0x7fcc0412f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 0x7fcc0412f450 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf40090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 msgr2=0x7fcc0412f450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcbebfff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc041417b0 0x7fcc0412f450 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf40090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.488+0000 7fcc037fe640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcbf0038770 0x7fcbf003ac20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009420 tx=0x7fcbf8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.495+0000 7fcc0a289640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.495+0000 7fcc0a289640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 0x7fcc04099730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.495+0000 7fcc0a289640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.495+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04063a40 con 0x7fcc04099330 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 0x7fcc04099730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient: get_auth_request con 0x7fcc04099330 auth_method 0 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 10 monclient(hunting): handle_auth_done global_id 64139 payload 824 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.496+0000 7fcc03fff640 1 -- 172.21.15.17:0/1636275665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc04099c70 con 0x7fcc04099330 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcc03fff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 msgr2=0x7fcc04099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcc03fff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 0x7fcc04099730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4025040 con 0x7fcc04099330 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcbebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcbebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.498+0000 7fcbebfff640 10 monclient: dump: 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.509 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7fcbf401c3e0 con 0x7fcc04099330 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 msgr2=0x7fcc04099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 0x7fcc04099730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf401d000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 msgr2=0x7fcc04099730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcbebfff640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04099330 0x7fcc04099730 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011bf0 tx=0x7fcbf401d000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:54:47.511 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.512 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:47.514 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.515 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.517 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:47.518 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.519 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.521 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.522 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:47.523 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcbf0038770 msgr2=0x7fcbf003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcbf0038770 0x7fcbf003ac20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009420 tx=0x7fcbf8032000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 10 monclient: shutdown 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 msgr2=0x7fcc04098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc04098df0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c7e0 tx=0x7fcbfc0025b0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc08826640 1 -- 172.21.15.17:0/1636275665 reap_dead start 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 shutdown_connections 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 --2- 172.21.15.17:0/1636275665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc04131b20 0x7fcc04098df0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 >> 172.21.15.17:0/1636275665 conn(0x7fcc04071270 msgr2=0x7fcc040a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 shutdown_connections 2024-09-15T01:54:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.499+0000 7fcc0a289640 1 -- 172.21.15.17:0/1636275665 wait complete. 2024-09-15T01:54:47.526 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:47.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:54:47.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:54:47.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.595+0000 7fba3e59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:47.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.595+0000 7fba3e59d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:47.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.595+0000 7fba3e59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.595+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7fba38098ef0 con 0x7fba38098140 2024-09-15T01:54:47.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.596+0000 7fba2ffff640 1 -- 172.21.15.67:0/1154093919 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v78) v1 ==== 124+0+0 (secure 0 0 0) 0x7fba38098ef0 con 0x7fba38098140 2024-09-15T01:54:47.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.596+0000 7fba2ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T01:54:47.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.596+0000 7fba2ffff640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-15T01:54:47.599 INFO:tasks.workunit.client.0.smithi067.stderr:device 'host1' does not appear in the crush map 2024-09-15T01:54:47.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 msgr2=0x7fba1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 0x7fba1803abf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fba20009360 tx=0x7fba20008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 10 monclient: shutdown 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 msgr2=0x7fba3809e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fba28009880 tx=0x7fba28032000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 shutdown_connections 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba18038740 0x7fba1803abf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba38098a50 0x7fba38130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba380a0370 0x7fba3811ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 --2- 172.21.15.67:0/1154093919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba38098140 0x7fba3809e540 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.598+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 >> 172.21.15.67:0/1154093919 conn(0x7fba380a5780 msgr2=0x7fba3809b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.599+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 shutdown_connections 2024-09-15T01:54:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.599+0000 7fba3e59d640 1 -- 172.21.15.67:0/1154093919 wait complete. 2024-09-15T01:54:47.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:54:47.611 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c host1 2024-09-15T01:54:47.611 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 0 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.621+0000 7f171f99d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.621+0000 7f171f99d640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.621+0000 7f171f99d640 10 monclient: monmap: 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:47.622030+0000 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:47.622030+0000 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.623 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:47.624 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.621+0000 7f171f99d640 1 Processor -- start 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 -- start start 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient: init 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T01:54:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-b con 0x7f1718131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17180a0280 0x7f1718131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-c con 0x7f17180a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f171809f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180a0040 con 0x7f171809f1e0 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171809bff0 con 0x7f17180a0280 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17180a14a0 con 0x7f1718131b20 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171f99d640 10 monclient(hunting): authenticate will time out at 947.618837s 2024-09-15T01:54:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34380/0 (socket says 172.21.15.17:34380) 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 1 -- 172.21.15.17:0/727289429 learned_addr learned my addr 172.21.15.17:0/727289429 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 10 monclient(hunting): get_auth_request con 0x7f1718131b20 auth_method 0 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.622+0000 7f171d739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 1 --2- 172.21.15.17:0/727289429 >> v2:172.21.15.67:3300/0 conn(0x7f17180a0280 0x7f1718131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): get_auth_request con 0x7f17180a0280 auth_method 0 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient(hunting): handle_auth_done global_id 64142 payload 274 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 1 -- 172.21.15.17:0/727289429 >> v2:172.21.15.67:3300/0 conn(0x7f17180a0280 msgr2=0x7f1718131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 1 --2- 172.21.15.17:0/727289429 >> v2:172.21.15.67:3300/0 conn(0x7f17180a0280 0x7f1718131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 1 -- 172.21.15.17:0/727289429 >> v1:172.21.15.67:6789/0 conn(0x7f171809f1e0 legacy=0x7f1718068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 1 monclient: found mon.noname-b 2024-09-15T01:54:47.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.623+0000 7f171d739640 1 -- 172.21.15.17:0/727289429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718132430 con 0x7f1718131b20 2024-09-15T01:54:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.625+0000 7f171d739640 1 -- 172.21.15.17:0/727289429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 msgr2=0x7f171809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.625+0000 7f171d739640 1 --2- 172.21.15.17:0/727289429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1714009910 tx=0x7f17140091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccd8dbe42f389e84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 1 -- 172.21.15.17:0/727289429 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171402aea0 con 0x7f1718131b20 2024-09-15T01:54:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: dump: 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 5 monclient: authenticate success, global_id 64142 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 1 -- 172.21.15.17:0/727289429 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1714032440 con 0x7f1718131b20 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 4 set_mon_vals no callback set 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 1 -- 172.21.15.17:0/727289429 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714032820 con 0x7f1718131b20 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f170ffff640 10 monclient: dump: 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 10 monclient: shutdown 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 -- 172.21.15.17:0/727289429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 msgr2=0x7f171809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 --2- 172.21.15.17:0/727289429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1714009910 tx=0x7f17140091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 -- 172.21.15.17:0/727289429 shutdown_connections 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 --2- 172.21.15.17:0/727289429 >> v2:172.21.15.67:3300/0 conn(0x7f17180a0280 0x7f1718131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 --2- 172.21.15.17:0/727289429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f171809eca0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 -- 172.21.15.17:0/727289429 >> 172.21.15.17:0/727289429 conn(0x7f1718071270 msgr2=0x7f17180a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 -- 172.21.15.17:0/727289429 shutdown_connections 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.626+0000 7f171f99d640 1 -- 172.21.15.17:0/727289429 wait complete. 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.627+0000 7f171f99d640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.627+0000 7f171f99d640 10 monclient: monmap: 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.627+0000 7f171f99d640 1 Processor -- start 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.627+0000 7f171f99d640 1 -- start start 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.627+0000 7f171f99d640 10 monclient: init 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171809f1e0 0x7f17181411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-b con 0x7f171809f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 0x7f171812ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-a con 0x7f17180a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f1718098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): picked mon.noname-c con 0x7f1718131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171811c180 con 0x7f17180a0280 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17180a5310 con 0x7f171809f1e0 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171811bea0 con 0x7f1718131b20 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171f99d640 10 monclient(hunting): authenticate will time out at 947.624390s 2024-09-15T01:54:47.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 0x7f171812ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f1718098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 0x7f171812ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40926/0 (socket says 172.21.15.17:40926) 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 1 -- 172.21.15.17:0/3068029715 learned_addr learned my addr 172.21.15.17:0/3068029715 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): get_auth_request con 0x7f17180a0280 auth_method 0 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171809f1e0 0x7f17181411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): get_auth_request con 0x7f1718131b20 auth_method 0 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): get_auth_request con 0x7f171809f1e0 auth_method 0 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171809f1e0 msgr2=0x7f17181411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171809f1e0 0x7f17181411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 msgr2=0x7f171812ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 0x7f171812ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 monclient: found mon.noname-c 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171df3a640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009450 con 0x7f1718131b20 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.628+0000 7f171cf38640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17180a0280 0x7f171812ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171df3a640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 msgr2=0x7f1718098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171df3a640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f1718098c10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f171000c810 tx=0x7f17100025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171000a850 con 0x7f1718131b20 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: dump: 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 5 monclient: authenticate success, global_id 64145 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171000a9b0 con 0x7f1718131b20 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 10 monclient: _renew_subs 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171809b350 con 0x7f1718131b20 2024-09-15T01:54:47.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171000ad70 con 0x7f1718131b20 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170d7fa640 10 monclient: dump: 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.649 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f170f7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 10 monclient: _renew_subs 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.631+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171809a780 con 0x7f1718131b20 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1710012630 con 0x7f1718131b20 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171f99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171f99d640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 0x7f1718099150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1718063a40 con 0x7f171812f2c0 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f170d7fa640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16fc038740 0x7f16fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 0x7f1718099150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient: get_auth_request con 0x7f171812f2c0 auth_method 0 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f171004c390 con 0x7f1718131b20 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171cf38640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16fc038740 0x7f16fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171cf38640 10 monclient: get_auth_request con 0x7f16fc038740 auth_method 0 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 10 monclient(hunting): handle_auth_done global_id 5278 payload 824 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.632+0000 7f171d739640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17180724a0 con 0x7f171812f2c0 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f171d739640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 msgr2=0x7f1718099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f171d739640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 0x7f1718099150 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17180a2f90 tx=0x7f1714032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f171cf38640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16fc038740 0x7f16fc03abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1718132b80 tx=0x7f17080090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171402aea0 con 0x7f171812f2c0 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 10 monclient: dump: 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.652 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1714031d30 con 0x7f171812f2c0 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 msgr2=0x7f1718099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 0x7f1718099150 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f17180a2f90 tx=0x7f1714032000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 msgr2=0x7f1718099150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.635+0000 7f170d7fa640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171812f2c0 0x7f1718099150 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171f99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171f99d640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 0x7f17180a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171f99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171812f690 con 0x7f1718099690 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 0x7f17180a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient: get_auth_request con 0x7f1718099690 auth_method 0 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 10 monclient(hunting): handle_auth_done global_id 5281 payload 824 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.641+0000 7f171d739640 1 -- 172.21.15.17:0/3068029715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f171811c010 con 0x7f1718099690 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f171d739640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 msgr2=0x7f17180a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f171d739640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 0x7f17180a2f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f17181421d0 tx=0x7f171403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171404b070 con 0x7f1718099690 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 10 monclient: dump: 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f1714031b20 con 0x7f1718099690 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 msgr2=0x7f17180a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 0x7f17180a2f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f17181421d0 tx=0x7f171403d000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 msgr2=0x7f17180a2f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:54:47.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.644+0000 7f170d7fa640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1718099690 0x7f17180a2f90 secure :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f17181421d0 tx=0x7f171403d000 comp rx=0 tx=0).stop 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 42, 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:54:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:54:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:54:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:54:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:54:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:54:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:54:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:54:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:54:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:54:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:54:47.672 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16fc038740 msgr2=0x7f16fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16fc038740 0x7f16fc03abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1718132b80 tx=0x7f17080090f0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 10 monclient: shutdown 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 msgr2=0x7f1718098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f1718098c10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f171000c810 tx=0x7f17100025b0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171df3a640 1 -- 172.21.15.17:0/3068029715 reap_dead start 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 shutdown_connections 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 --2- 172.21.15.17:0/3068029715 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1718131b20 0x7f1718098c10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 >> 172.21.15.17:0/3068029715 conn(0x7f1718071270 msgr2=0x7f171809d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 shutdown_connections 2024-09-15T01:54:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.645+0000 7f171f99d640 1 -- 172.21.15.17:0/3068029715 wait complete. 2024-09-15T01:54:47.674 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:54:47.674 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T01:54:47.674 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:54:47.675 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T01:54:47.675 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:54:47.675 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T01:54:47.675 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:54:47.685 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T01:54:47.685 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:54:47.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:47.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.685+0000 7f88cccaf640 10 monclient: get_monmap_and_config 2024-09-15T01:54:47.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.685+0000 7f88cccaf640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.685+0000 7f88cccaf640 10 monclient: monmap: 2024-09-15T01:54:47.687 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:47.687 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:47.686475+0000 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:47.686475+0000 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:47.688 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.685+0000 7f88cccaf640 1 Processor -- start 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 -- start start 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient: init 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f88c80981b0 0x7f88c81322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-a con 0x7f88c80981b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88c8098580 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88c8098980 0x7f88c80694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-c con 0x7f88c8098980 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80a07b0 con 0x7f88c8098580 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c8098d90 con 0x7f88c80981b0 2024-09-15T01:54:47.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80a16e0 con 0x7f88c8098980 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88cccaf640 10 monclient(hunting): authenticate will time out at 920.155712s 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88c8098980 0x7f88c80694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88c8098980 0x7f88c80694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57848/0 (socket says 172.21.15.67:57848) 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 1 -- 172.21.15.67:0/1642935728 learned_addr learned my addr 172.21.15.67:0/1642935728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c5d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88c8098580 0x7f88c8068dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55264/0 (socket says 172.21.15.67:55264) 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): get_auth_request con 0x7f88c8098980 auth_method 0 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.686+0000 7f88c659c640 1 --2- 172.21.15.67:0/1642935728 >> v2:172.21.15.17:3300/0 conn(0x7f88c80981b0 0x7f88c81322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c659c640 10 monclient(hunting): get_auth_request con 0x7f88c80981b0 auth_method 0 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_done global_id 5284 payload 274 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 1 -- 172.21.15.67:0/1642935728 >> v2:172.21.15.17:3300/0 conn(0x7f88c80981b0 msgr2=0x7f88c81322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 1 --2- 172.21.15.67:0/1642935728 >> v2:172.21.15.17:3300/0 conn(0x7f88c80981b0 0x7f88c81322c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 1 -- 172.21.15.67:0/1642935728 >> v1:172.21.15.67:6789/0 conn(0x7f88c8098580 legacy=0x7f88c8068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 1 monclient: found mon.noname-c 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.687+0000 7f88c6d9d640 1 -- 172.21.15.67:0/1642935728 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80a1400 con 0x7f88c8098980 2024-09-15T01:54:47.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.689+0000 7f88c6d9d640 1 -- 172.21.15.67:0/1642935728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098980 msgr2=0x7f88c80694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.689+0000 7f88c6d9d640 1 --2- 172.21.15.67:0/1642935728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098980 0x7f88c80694d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f88c806a890 tx=0x7f88c0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1c0b7897fd0d33 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.689+0000 7f88c559a640 1 -- 172.21.15.67:0/1642935728 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0007610 con 0x7f88c8098980 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: dump: 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.700 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 5 monclient: authenticate success, global_id 5284 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 1 -- 172.21.15.67:0/1642935728 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c002d050 con 0x7f88c8098980 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:47.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 4 set_mon_vals no callback set 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 1 -- 172.21.15.67:0/1642935728 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00079a0 con 0x7f88c8098980 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88c559a640 10 monclient: dump: 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.702 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 10 monclient: shutdown 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 -- 172.21.15.67:0/1642935728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098980 msgr2=0x7f88c80694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1642935728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098980 0x7f88c80694d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f88c806a890 tx=0x7f88c0009840 comp rx=0 tx=0).stop 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 -- 172.21.15.67:0/1642935728 shutdown_connections 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1642935728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098980 0x7f88c80694d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1642935728 >> v2:172.21.15.17:3300/0 conn(0x7f88c80981b0 0x7f88c81322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 -- 172.21.15.67:0/1642935728 >> 172.21.15.67:0/1642935728 conn(0x7f88c8071150 msgr2=0x7f88c80a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 -- 172.21.15.67:0/1642935728 shutdown_connections 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.690+0000 7f88cccaf640 1 -- 172.21.15.67:0/1642935728 wait complete. 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient: monmap: 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 Processor -- start 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 -- start start 2024-09-15T01:54:47.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient: init 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 0x7f88c811bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-c con 0x7f88c80981b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-b con 0x7f88c8098580 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88c8098980 0x7f88c8141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): picked mon.noname-a con 0x7f88c8098980 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c811cc30 con 0x7f88c8098980 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80a46d0 con 0x7f88c8098580 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c811ded0 con 0x7f88c80981b0 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88cccaf640 10 monclient(hunting): authenticate will time out at 920.160911s 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88c659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 msgr2=0x7f88c811bd70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.691+0000 7f88c659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 0x7f88c811bd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57864/0 (socket says 172.21.15.67:57864) 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 -- 172.21.15.67:0/1170632535 learned_addr learned my addr 172.21.15.67:0/1170632535 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): get_auth_request con 0x7f88c8098580 auth_method 0 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88c8098980 0x7f88c8141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): get_auth_request con 0x7f88c8098980 auth_method 0 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_done global_id 5287 payload 824 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 -- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 msgr2=0x7f88c811bd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 0x7f88c811bd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 -- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88c8098980 msgr2=0x7f88c8141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88c8098980 0x7f88c8141140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 monclient: found mon.noname-b 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.692+0000 7f88c5d9b640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0009ae0 con 0x7f88c8098580 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88c5d9b640 1 -- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 msgr2=0x7f88c8130db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88c5d9b640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c5b0 tx=0x7f88b0007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0007510 con 0x7f88c8098580 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: dump: 2024-09-15T01:54:47.708 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b00057d0 con 0x7f88c8098580 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 5 monclient: authenticate success, global_id 5287 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 10 monclient: _renew_subs 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c811e1b0 con 0x7f88c8098580 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0007690 con 0x7f88c8098580 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88be7fc640 10 monclient: dump: 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.710 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 10 monclient: _renew_subs 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8134f40 con 0x7f88c8098580 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.694+0000 7f88c4d99640 4 set_mon_vals no callback set 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f88b001b400 con 0x7f88c8098580 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88cccaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88cccaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88cccaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88be7fc640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 0x7f88a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8098f40 con 0x7f88c8098580 2024-09-15T01:54:47.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.695+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f88b004ba60 con 0x7f88c8098580 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.696+0000 7f88c659c640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 0x7f88a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.696+0000 7f88c659c640 10 monclient: get_auth_request con 0x7f88a80386f0 auth_method 0 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.698+0000 7f88c659c640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 0x7f88a803aba0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f88b8006f40 tx=0x7f88b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.700+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f88c8098f40 con 0x7f88c8098580 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.700+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.700+0000 7f88be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 10 monclient: get_monmap_and_config 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 10 monclient: monmap: 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:47.818904+0000 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:47.818904+0000 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.820 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:47.821 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 1 Processor -- start 2024-09-15T01:54:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 1 -- start start 2024-09-15T01:54:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.818+0000 7f40fa42b640 10 monclient: init 2024-09-15T01:54:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40f40a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40f4132040 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40f413a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-b con 0x7f40f40987b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40a1580 con 0x7f40f40a58b0 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40a5c80 con 0x7f40f4132040 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4132440 con 0x7f40f40987b0 2024-09-15T01:54:47.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40fa42b640 10 monclient(hunting): authenticate will time out at 947.815537s 2024-09-15T01:54:47.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40f89c8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 msgr2=0x7f40f413a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:47.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40f89c8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40f413a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:47.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40f3fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f40f40a58b0 0x7f40f40980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48596/0 (socket says 172.21.15.17:48596) 2024-09-15T01:54:47.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.819+0000 7f40f3fff640 1 -- 172.21.15.17:0/3250110077 learned_addr learned my addr 172.21.15.17:0/3250110077 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f3fff640 1 -- 172.21.15.17:0/3250110077 >> v1:172.21.15.17:6789/0 conn(0x7f40f40a58b0 legacy=0x7f40f40980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f37fe640 1 -- 172.21.15.17:0/3250110077 >> v1:172.21.15.67:6789/0 conn(0x7f40f4132040 legacy=0x7f40f4136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131153515 0 0) 0x7f40f40a1580 con 0x7f40f40a58b0 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f2ffd640 10 monclient(hunting): my global_id is 44658 2024-09-15T01:54:47.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.821+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e00034e0 con 0x7f40f40a58b0 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183571750 0 0) 0x7f40f40a5c80 con 0x7f40f4132040 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): my global_id is 5290 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f40a1580 con 0x7f40f4132040 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1001235860 0 0) 0x7f40e00034e0 con 0x7f40f40a58b0 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 msgr2=0x7f40f413a500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40f413a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 >> v1:172.21.15.67:6789/0 conn(0x7f40f4132040 legacy=0x7f40f4136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:54:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4098bd0 con 0x7f40f40a58b0 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f40e4001720 con 0x7f40f40a58b0 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40fa42b640 5 monclient: authenticate success, global_id 44658 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40fa42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: dump: 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 >> v1:172.21.15.17:6789/0 conn(0x7f40f40a58b0 legacy=0x7f40f40980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0003b90 0x7f40e0003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): picked mon.b con 0x7f40e0003b90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40e0004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): picked mon.c con 0x7f40f40987b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 0x7f40e0005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): picked mon.a con 0x7f40e0004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4132440 con 0x7f40e0004de0 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e000bd10 con 0x7f40e0003b90 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e000bed0 con 0x7f40f40987b0 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:54:47.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40fa42b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f37fe640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 msgr2=0x7f40e0004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f37fe640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40e0004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 0x7f40e0005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0003b90 0x7f40e0003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): get_auth_request con 0x7f40e0004de0 auth_method 0 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): get_auth_request con 0x7f40e0003b90 auth_method 0 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient(hunting): handle_auth_done global_id 44658 payload 293 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 msgr2=0x7f40e0004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f40987b0 0x7f40e0004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0003b90 msgr2=0x7f40e0003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0003b90 0x7f40e0003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 monclient: found mon.a 2024-09-15T01:54:47.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 1 -- 172.21.15.17:0/3250110077 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00034e0 con 0x7f40e0004de0 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.822+0000 7f40f89c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f89c8640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 msgr2=0x7f40e0005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f89c8640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 0x7f40e0005250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009f90 tx=0x7f40ec0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceac190bce93c5f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec032430 con 0x7f40e0004de0 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: dump: 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.838 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec02d070 con 0x7f40e0004de0 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40fa42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40fa42b640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 1 -- 172.21.15.17:0/3250110077 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec032820 con 0x7f40e0004de0 2024-09-15T01:54:47.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40fa42b640 4 set_mon_vals no callback set 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.825+0000 7f40f2ffd640 10 monclient: dump: 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 10 monclient: shutdown 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 -- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 msgr2=0x7f40e0005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 --2- 172.21.15.17:0/3250110077 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40e0004de0 0x7f40e0005250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009f90 tx=0x7f40ec0095d0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40f89c8640 1 -- 172.21.15.17:0/3250110077 reap_dead start 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 -- 172.21.15.17:0/3250110077 shutdown_connections 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 -- 172.21.15.17:0/3250110077 >> 172.21.15.17:0/3250110077 conn(0x7f40f4070ed0 msgr2=0x7f40f40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 -- 172.21.15.17:0/3250110077 shutdown_connections 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 1 -- 172.21.15.17:0/3250110077 wait complete. 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 10 monclient: build_initial_monmap 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.826+0000 7f40fa42b640 10 monclient: monmap: 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:47.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 Processor -- start 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 -- start start 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient: init 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:47.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0004de0 0x7f40f409ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-b con 0x7f40e0004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 0x7f40f4130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-c con 0x7f40f411bca0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): picked mon.noname-a con 0x7f40f4131170 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f411e030 con 0x7f40f4131170 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f411c7c0 con 0x7f40e0004de0 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f409e7f0 con 0x7f40f411bca0 2024-09-15T01:54:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40fa42b640 10 monclient(hunting): authenticate will time out at 947.823763s 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f37fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 msgr2=0x7f40f4130c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 0x7f40f4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40934/0 (socket says 172.21.15.17:40934) 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 1 -- 172.21.15.17:0/2936205691 learned_addr learned my addr 172.21.15.17:0/2936205691 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): get_auth_request con 0x7f40f4131170 auth_method 0 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f3fff640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0004de0 0x7f40f409ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f3fff640 10 monclient(hunting): get_auth_request con 0x7f40e0004de0 auth_method 0 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.827+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 10 monclient(hunting): handle_auth_done global_id 44661 payload 824 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 -- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 msgr2=0x7f40f4130c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 0x7f40f4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 -- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0004de0 msgr2=0x7f40f409ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0004de0 0x7f40f409ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 monclient: found mon.noname-a 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.828+0000 7f40f89c8640 1 -- 172.21.15.17:0/2936205691 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f411dd50 con 0x7f40f4131170 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40f89c8640 1 -- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 msgr2=0x7f40f4141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40f89c8640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f40ec02ad80 tx=0x7f40ec035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec0093a0 con 0x7f40f4131170 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: dump: 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: _finish_auth 0 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec0315e0 con 0x7f40f4131170 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 5 monclient: authenticate success, global_id 44661 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 10 monclient: _renew_subs 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f411ce60 con 0x7f40f4131170 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec0319a0 con 0x7f40f4131170 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40ebfff640 10 monclient: dump: 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:47.850 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 10 monclient: _renew_subs 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40f27fc640 4 set_mon_vals no callback set 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.830+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f41416e0 con 0x7f40f4131170 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f40ec039070 con 0x7f40f4131170 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40fa42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40fa42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40fa42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40ebfff640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 0x7f40d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f411c070 con 0x7f40f4131170 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.831+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f40ec006e40 con 0x7f40f4131170 2024-09-15T01:54:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.832+0000 7f40f3fff640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 0x7f40d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.832+0000 7f40f3fff640 10 monclient: get_auth_request con 0x7f40d40386f0 auth_method 0 2024-09-15T01:54:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.834+0000 7f40f3fff640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 0x7f40d403aba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40e4001900 tx=0x7f40f4096b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.836+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f40ec09ee40 con 0x7f40f4131170 2024-09-15T01:54:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.836+0000 7f40ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:47.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:47.836+0000 7f40ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88cccaf640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88cccaf640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88cccaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f88c804bf60 con 0x7f88c8098580 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (secure 0 0 0) 0x7f88b001a720 con 0x7f88c8098580 2024-09-15T01:54:47.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:54:47.867 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.864+0000 7f88be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88cccaf640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:54:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88cccaf640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88cccaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:47.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f88c80a19c0 con 0x7f88c8098580 2024-09-15T01:54:47.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88be7fc640 1 -- 172.21.15.67:0/1170632535 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (secure 0 0 0) 0x7f88b001a360 con 0x7f88c8098580 2024-09-15T01:54:47.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:54:47.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.962+0000 7f88be7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:54:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 msgr2=0x7f88a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 0x7f88a803aba0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f88b8006f40 tx=0x7f88b8008040 comp rx=0 tx=0).stop 2024-09-15T01:54:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 10 monclient: shutdown 2024-09-15T01:54:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:47.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 msgr2=0x7f88c8130db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c5b0 tx=0x7f88b0007f00 comp rx=0 tx=0).stop 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 shutdown_connections 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88a80386f0 0x7f88a803aba0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88c8098980 0x7f88c8141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88c8098580 0x7f88c8130db0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 --2- 172.21.15.67:0/1170632535 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88c80981b0 0x7f88c811bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 >> 172.21.15.67:0/1170632535 conn(0x7f88c8071150 msgr2=0x7f88c809b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 shutdown_connections 2024-09-15T01:54:47.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:47.964+0000 7f88cccaf640 1 -- 172.21.15.67:0/1170632535 wait complete. 2024-09-15T01:54:47.975 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd tree-from host1 2024-09-15T01:54:47.975 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:54:47.975 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree-from host1 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.047+0000 7f67cf2b8640 10 monclient: get_monmap_and_config 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.047+0000 7f67cf2b8640 10 monclient: build_initial_monmap 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.047+0000 7f67cf2b8640 10 monclient: monmap: 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:48.048828+0000 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:48.048828+0000 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:48.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:48.051 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 Processor -- start 2024-09-15T01:54:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 -- start start 2024-09-15T01:54:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient: init 2024-09-15T01:54:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T01:54:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67c8096a70 0x7f67c809d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-c con 0x7f67c8096a70 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67c8096e40 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f67c8134dd0 0x7f67c81371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-a con 0x7f67c8134dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c809bf70 con 0x7f67c8096e40 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c809c500 con 0x7f67c8134dd0 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c809ba10 con 0x7f67c8096a70 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.048+0000 7f67cf2b8640 10 monclient(hunting): authenticate will time out at 920.517987s 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67c8096a70 0x7f67c809d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67c8096a70 0x7f67c809d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57880/0 (socket says 172.21.15.67:57880) 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 -- 172.21.15.67:0/1632969820 learned_addr learned my addr 172.21.15.67:0/1632969820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): get_auth_request con 0x7f67c8096a70 auth_method 0 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 1 --2- 172.21.15.67:0/1632969820 >> v2:172.21.15.17:3300/0 conn(0x7f67c8134dd0 0x7f67c81371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): get_auth_request con 0x7f67c8134dd0 auth_method 0 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd855640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient(hunting): handle_auth_done global_id 5296 payload 274 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 -- 172.21.15.67:0/1632969820 >> v2:172.21.15.17:3300/0 conn(0x7f67c8134dd0 msgr2=0x7f67c81371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 --2- 172.21.15.67:0/1632969820 >> v2:172.21.15.17:3300/0 conn(0x7f67c8134dd0 0x7f67c81371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 -- 172.21.15.67:0/1632969820 >> v1:172.21.15.17:6789/0 conn(0x7f67c8096e40 legacy=0x7f67c80972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient: _finish_hunting 0 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 monclient: found mon.noname-c 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.049+0000 7f67cd054640 1 -- 172.21.15.67:0/1632969820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c809c220 con 0x7f67c8096a70 2024-09-15T01:54:48.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cd054640 1 -- 172.21.15.67:0/1632969820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 msgr2=0x7f67c809d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cd054640 1 --2- 172.21.15.67:0/1632969820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c809d0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f67c8067cf0 tx=0x7f67c40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1a9f8e7d5b6f9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 1 -- 172.21.15.67:0/1632969820 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4032440 con 0x7f67c8096a70 2024-09-15T01:54:48.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: dump: 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.064 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 1 -- 172.21.15.67:0/1632969820 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c40325a0 con 0x7f67c8096a70 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 5 monclient: authenticate success, global_id 5296 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 4 set_mon_vals no callback set 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 1 -- 172.21.15.67:0/1632969820 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4032940 con 0x7f67c8096a70 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67bffff640 10 monclient: dump: 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 10 monclient: shutdown 2024-09-15T01:54:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 -- 172.21.15.67:0/1632969820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 msgr2=0x7f67c809d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 --2- 172.21.15.67:0/1632969820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c809d0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f67c8067cf0 tx=0x7f67c40091d0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 -- 172.21.15.67:0/1632969820 shutdown_connections 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 --2- 172.21.15.67:0/1632969820 >> v2:172.21.15.17:3300/0 conn(0x7f67c8134dd0 0x7f67c81371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 --2- 172.21.15.67:0/1632969820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c809d0e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.052+0000 7f67cf2b8640 1 -- 172.21.15.67:0/1632969820 >> 172.21.15.67:0/1632969820 conn(0x7f67c80a1020 msgr2=0x7f67c80a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 1 -- 172.21.15.67:0/1632969820 shutdown_connections 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 1 -- 172.21.15.67:0/1632969820 wait complete. 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 10 monclient: build_initial_monmap 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 10 monclient: monmap: 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 1 Processor -- start 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 1 -- start start 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.053+0000 7f67cf2b8640 10 monclient: init 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c8135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-b con 0x7f67c8096a70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f67c8096e40 0x7f67c8131a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-a con 0x7f67c8096e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f67c8134dd0 0x7f67c8098580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): picked mon.noname-c con 0x7f67c8134dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c809bcb0 con 0x7f67c8096e40 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c811fca0 con 0x7f67c8096a70 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c811f9c0 con 0x7f67c8134dd0 2024-09-15T01:54:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cf2b8640 10 monclient(hunting): authenticate will time out at 920.523444s 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c8135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd855640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f67c8134dd0 msgr2=0x7f67c8098580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd855640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f67c8134dd0 0x7f67c8098580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c8135670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57894/0 (socket says 172.21.15.67:57894) 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 -- 172.21.15.67:0/3573857053 learned_addr learned my addr 172.21.15.67:0/3573857053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): get_auth_request con 0x7f67c8096a70 auth_method 0 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f67c8096e40 0x7f67c8131a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): get_auth_request con 0x7f67c8096e40 auth_method 0 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cc853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient(hunting): handle_auth_done global_id 5299 payload 824 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 -- 172.21.15.67:0/3573857053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f67c8134dd0 msgr2=0x7f67c8098580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f67c8134dd0 0x7f67c8098580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 -- 172.21.15.67:0/3573857053 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f67c8096e40 msgr2=0x7f67c8131a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f67c8096e40 0x7f67c8131a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient: _finish_hunting 0 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 monclient: found mon.noname-b 2024-09-15T01:54:48.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.054+0000 7f67cd054640 1 -- 172.21.15.67:0/3573857053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c4009450 con 0x7f67c8096a70 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cd054640 1 -- 172.21.15.67:0/3573857053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 msgr2=0x7f67c8135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cd054640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c8096a70 0x7f67c8135670 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009210 tx=0x7f67c4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c40319c0 con 0x7f67c8096a70 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: dump: 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.072 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c4031b40 con 0x7f67c8096a70 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 5 monclient: authenticate success, global_id 5299 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c403d440 con 0x7f67c8096a70 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 10 monclient: _renew_subs 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 1 -- 172.21.15.67:0/3573857053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c811ff80 con 0x7f67c8096a70 2024-09-15T01:54:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bf7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67bd7fa640 10 monclient: dump: 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.074 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 10 monclient: _renew_subs 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.057+0000 7f67cf2b8640 1 -- 172.21.15.67:0/3573857053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c8132810 con 0x7f67c8096a70 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f67c4039040 con 0x7f67c8096a70 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cf2b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cf2b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cf2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cf2b8640 1 -- 172.21.15.67:0/3573857053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c8120140 con 0x7f67c8096a70 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67bd7fa640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f67ac038720 0x7f67ac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f67c406ba90 con 0x7f67c8096a70 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cc853640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f67ac038720 0x7f67ac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.058+0000 7f67cc853640 10 monclient: get_auth_request con 0x7f67ac038720 auth_method 0 2024-09-15T01:54:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.060+0000 7f67cc853640 1 --2- 172.21.15.67:0/3573857053 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f67ac038720 0x7f67ac03abd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f67b8007fd0 tx=0x7f67b8007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.062+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f67c402f2a0 con 0x7f67c8096a70 2024-09-15T01:54:48.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.062+0000 7f67bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.062+0000 7f67bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40fa42b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40fa42b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40fa42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f40f4098e30 con 0x7f40f4131170 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40ebfff640 1 -- 172.21.15.17:0/2936205691 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f40f4098e30 con 0x7f40f4131170 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.060+0000 7f40ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:48.077 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 msgr2=0x7f40d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 0x7f40d403aba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40e4001900 tx=0x7f40f4096b10 comp rx=0 tx=0).stop 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 10 monclient: shutdown 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 msgr2=0x7f40f4141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f40ec02ad80 tx=0x7f40ec035000 comp rx=0 tx=0).stop 2024-09-15T01:54:48.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 shutdown_connections 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f40d40386f0 0x7f40d403aba0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40f4131170 0x7f40f4141000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40f411bca0 0x7f40f4130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 --2- 172.21.15.17:0/2936205691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e0004de0 0x7f40f409ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 >> 172.21.15.17:0/2936205691 conn(0x7f40f4070ed0 msgr2=0x7f40f409ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 shutdown_connections 2024-09-15T01:54:48.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:48.062+0000 7f40fa42b640 1 -- 172.21.15.17:0/2936205691 wait complete. 2024-09-15T01:54:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.227+0000 7f67cf2b8640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T01:54:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.227+0000 7f67cf2b8640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T01:54:48.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.227+0000 7f67cf2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.227+0000 7f67cf2b8640 1 -- 172.21.15.67:0/3573857053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7f67c804bf60 con 0x7f67c8096a70 2024-09-15T01:54:48.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.228+0000 7f67bd7fa640 1 -- 172.21.15.67:0/3573857053 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v78) v1 ==== 109+0+0 (secure 0 0 0) 0x7f67c4037090 con 0x7f67c8096a70 2024-09-15T01:54:48.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.228+0000 7f67bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T01:54:48.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.228+0000 7f67bd7fa640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-15T01:54:48.231 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-15T01:54:48.233 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:54:48.234 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree-from host2 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.306+0000 7faf6539d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.306+0000 7faf6539d640 10 monclient: build_initial_monmap 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.306+0000 7faf6539d640 10 monclient: monmap: 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:54:48.307879+0000 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:54:48.307879+0000 2024-09-15T01:54:48.309 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:48.310 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 Processor -- start 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 -- start start 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient: init 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf60130e00 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf601315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-b con 0x7faf601311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf60136f10 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6009ff20 con 0x7faf60130e00 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600a00f0 con 0x7faf60136f10 2024-09-15T01:54:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf601320b0 con 0x7faf601311d0 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.307+0000 7faf6539d640 10 monclient(hunting): authenticate will time out at 920.776978s 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.308+0000 7faf57fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 msgr2=0x7faf601315c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.308+0000 7faf57fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf601315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.308+0000 7faf5f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faf60136f10 0x7faf601392f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55276/0 (socket says 172.21.15.67:55276) 2024-09-15T01:54:48.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.308+0000 7faf5f7fe640 1 -- 172.21.15.67:0/3862545661 learned_addr learned my addr 172.21.15.67:0/3862545661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5f7fe640 1 -- 172.21.15.67:0/3862545661 >> v1:172.21.15.67:6789/0 conn(0x7faf60136f10 legacy=0x7faf601392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940411893 0 0) 0x7faf600a00f0 con 0x7faf60136f10 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): my global_id is 5302 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c0034e0 con 0x7faf60136f10 2024-09-15T01:54:48.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5effd640 1 -- 172.21.15.67:0/3862545661 >> v1:172.21.15.17:6789/0 conn(0x7faf60130e00 legacy=0x7faf60132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915930891 0 0) 0x7faf4c0034e0 con 0x7faf60136f10 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 msgr2=0x7faf601315c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf601315c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 >> v1:172.21.15.17:6789/0 conn(0x7faf60130e00 legacy=0x7faf60132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600a14d0 con 0x7faf60136f10 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf6539d640 5 monclient: authenticate success, global_id 5302 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7faf58001aa0 con 0x7faf60136f10 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf6539d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: dump: 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 >> v1:172.21.15.67:6789/0 conn(0x7faf60136f10 legacy=0x7faf601392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf4c003a50 0x7faf4c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): picked mon.a con 0x7faf4c003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 0x7faf4c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): picked mon.b con 0x7faf4c004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf4c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): picked mon.c con 0x7faf601311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf601320b0 con 0x7faf4c003a50 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c005630 con 0x7faf4c004780 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c00bf90 con 0x7faf601311d0 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:48.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf6539d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5f7fe640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 msgr2=0x7faf4c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf5f7fe640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf4c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf57fff640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 0x7faf4c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf57fff640 10 monclient(hunting): get_auth_request con 0x7faf4c004780 auth_method 0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.310+0000 7faf57fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf4c003a50 0x7faf4c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): get_auth_request con 0x7faf4c003a50 auth_method 0 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf5effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient(hunting): handle_auth_done global_id 5302 payload 293 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 msgr2=0x7faf4c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:48.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf601311d0 0x7faf4c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf4c003a50 msgr2=0x7faf4c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf4c003a50 0x7faf4c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 monclient: found mon.b 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 1 -- 172.21.15.67:0/3862545661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0034e0 con 0x7faf4c004780 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 10 monclient: _finish_auth 0 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.311+0000 7faf57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.313+0000 7faf57fff640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 msgr2=0x7faf4c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.313+0000 7faf57fff640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 0x7faf4c004bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf480099c0 tx=0x7faf48009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47ec233420471967 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48003b80 con 0x7faf4c004780 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: dump: 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf48003d20 con 0x7faf4c004780 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 4 set_mon_vals no callback set 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 1 -- 172.21.15.67:0/3862545661 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4802ab80 con 0x7faf4c004780 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5e7fc640 10 monclient: dump: 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.329 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 10 monclient: shutdown 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 -- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 msgr2=0x7faf4c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 --2- 172.21.15.67:0/3862545661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c004780 0x7faf4c004bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf480099c0 tx=0x7faf48009060 comp rx=0 tx=0).stop 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf5f7fe640 1 -- 172.21.15.67:0/3862545661 reap_dead start 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 -- 172.21.15.67:0/3862545661 shutdown_connections 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 -- 172.21.15.67:0/3862545661 >> 172.21.15.67:0/3862545661 conn(0x7faf600710f0 msgr2=0x7faf600a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:48.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 -- 172.21.15.67:0/3862545661 shutdown_connections 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.314+0000 7faf6539d640 1 -- 172.21.15.67:0/3862545661 wait complete. 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient: build_initial_monmap 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient: monmap: 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:48.331 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 1 Processor -- start 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 1 -- start start 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient: init 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 0x7faf6012e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.315+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-b con 0x7faf6012e7b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf6012eb80 0x7faf60098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): picked mon.noname-a con 0x7faf6012eb80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60099cc0 con 0x7faf6012eb80 2024-09-15T01:54:48.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600a3b40 con 0x7faf6012e7b0 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600a0820 con 0x7faf4c004780 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf6539d640 10 monclient(hunting): authenticate will time out at 920.785028s 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57906/0 (socket says 172.21.15.67:57906) 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 -- 172.21.15.67:0/912339032 learned_addr learned my addr 172.21.15.67:0/912339032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): get_auth_request con 0x7faf6012e7b0 auth_method 0 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5effd640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 msgr2=0x7faf6012e270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5effd640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 0x7faf6012e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5f7fe640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf6012eb80 0x7faf60098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5f7fe640 10 monclient(hunting): get_auth_request con 0x7faf6012eb80 auth_method 0 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient(hunting): handle_auth_done global_id 5305 payload 824 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 msgr2=0x7faf6012e270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 0x7faf6012e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf6012eb80 msgr2=0x7faf60098220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf6012eb80 0x7faf60098220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:48.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient: _finish_hunting 0 2024-09-15T01:54:48.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 monclient: found mon.noname-b 2024-09-15T01:54:48.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:48.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.316+0000 7faf57fff640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6009a310 con 0x7faf6012e7b0 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.318+0000 7faf57fff640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 msgr2=0x7faf60097ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.318+0000 7faf57fff640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7faf48009a20 tx=0x7faf48030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.318+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4803b070 con 0x7faf6012e7b0 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: dump: 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.337 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: _finish_auth 0 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf48005690 con 0x7faf6012e7b0 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf480336a0 con 0x7faf6012e7b0 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf56ffd640 10 monclient: dump: 2024-09-15T01:54:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 5 monclient: authenticate success, global_id 5305 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 10 monclient: _renew_subs 2024-09-15T01:54:48.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6009b270 con 0x7faf6012e7b0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf5dffb640 4 set_mon_vals no callback set 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 10 monclient: _renew_subs 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.320+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6009af10 con 0x7faf6012e7b0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faf4803e070 con 0x7faf6012e7b0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf6539d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf6539d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf6539d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6004bf60 con 0x7faf6012e7b0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.321+0000 7faf56ffd640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 0x7faf4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.322+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7faf4806aed0 con 0x7faf6012e7b0 2024-09-15T01:54:48.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.322+0000 7faf5effd640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 0x7faf4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:48.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.322+0000 7faf5effd640 10 monclient: get_auth_request con 0x7faf40038740 auth_method 0 2024-09-15T01:54:48.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.324+0000 7faf5effd640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 0x7faf4003abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf50007fd0 tx=0x7faf50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:48.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.326+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faf4803eb50 con 0x7faf6012e7b0 2024-09-15T01:54:48.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.326+0000 7faf56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:48.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.326+0000 7faf56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:48.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf6539d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf6539d640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf6539d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7faf6006f0b0 con 0x7faf6012e7b0 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (secure 0 0 0) 0x7faf4803e320 con 0x7faf6012e7b0 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:54:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.490+0000 7faf56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.587+0000 7faf6539d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:54:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.587+0000 7faf6539d640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:54:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.587+0000 7faf6539d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:54:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:48.587+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7faf60098760 con 0x7faf6012e7b0 2024-09-15T01:54:51.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:51.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.188+0000 7f99c075d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:51.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.188+0000 7f99c075d640 10 monclient: build_initial_monmap 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.188+0000 7f99c075d640 10 monclient: monmap: 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:51.189477+0000 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:51.189477+0000 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:51.191 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:51.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 1 Processor -- start 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 1 -- start start 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 10 monclient: init 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:54:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f99b806e920 0x7f99b812fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-a con 0x7f99b806e920 addr v2:172.21.15.17:3300/0 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.189+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99b806ecf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8097320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-b con 0x7f99b806f110 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b809c860 con 0x7f99b806ecf0 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b809d090 con 0x7f99b806e920 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b809d4b0 con 0x7f99b806f110 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99c075d640 10 monclient(hunting): authenticate will time out at 951.186051s 2024-09-15T01:54:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99becfa640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 msgr2=0x7f99b8097320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99becfa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f99b806e920 0x7f99b812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f99b806e920 0x7f99b812fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40944/0 (socket says 172.21.15.17:40944) 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 -- 172.21.15.17:0/2585941121 learned_addr learned my addr 172.21.15.17:0/2585941121 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): get_auth_request con 0x7f99b806e920 auth_method 0 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:51.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient(hunting): handle_auth_done global_id 44676 payload 274 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 -- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 msgr2=0x7f99b8097320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 --2- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8097320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 -- 172.21.15.17:0/2585941121 >> v1:172.21.15.67:6789/0 conn(0x7f99b806ecf0 legacy=0x7f99b8096c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient: _finish_hunting 0 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 monclient: found mon.noname-a 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:51.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.190+0000 7f99be4f9640 1 -- 172.21.15.17:0/2585941121 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b809cad0 con 0x7f99b806e920 2024-09-15T01:54:51.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99be4f9640 1 -- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806e920 msgr2=0x7f99b812fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:51.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99be4f9640 1 --2- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806e920 0x7f99b812fc30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009f20 tx=0x7f99b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad11b20615394673 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:51.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 1 -- 172.21.15.17:0/2585941121 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4007610 con 0x7f99b806e920 2024-09-15T01:54:51.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: dump: 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: _finish_auth 0 2024-09-15T01:54:51.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 1 -- 172.21.15.17:0/2585941121 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b402d050 con 0x7f99b806e920 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 1 -- 172.21.15.17:0/2585941121 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b40079a0 con 0x7f99b806e920 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99c075d640 5 monclient: authenticate success, global_id 44676 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99c075d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99c075d640 4 set_mon_vals no callback set 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.193+0000 7f99bd4f7640 10 monclient: dump: 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.205 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 10 monclient: shutdown 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 -- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806e920 msgr2=0x7f99b812fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 --2- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806e920 0x7f99b812fc30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009f20 tx=0x7f99b4009840 comp rx=0 tx=0).stop 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 -- 172.21.15.17:0/2585941121 shutdown_connections 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 --2- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8097320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 --2- 172.21.15.17:0/2585941121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806e920 0x7f99b812fc30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 -- 172.21.15.17:0/2585941121 >> 172.21.15.17:0/2585941121 conn(0x7f99b80a1000 msgr2=0x7f99b80a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 -- 172.21.15.17:0/2585941121 shutdown_connections 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 -- 172.21.15.17:0/2585941121 wait complete. 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 10 monclient: build_initial_monmap 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 10 monclient: monmap: 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:51.207 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.194+0000 7f99c075d640 1 Processor -- start 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 -- start start 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient: init 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99b806e920 0x7f99b813f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-b con 0x7f99b806e920 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-a con 0x7f99b806ecf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8118650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): picked mon.noname-c con 0x7f99b806f110 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:51.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b811c1a0 con 0x7f99b806ecf0 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b806f640 con 0x7f99b806e920 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b811bec0 con 0x7f99b806f110 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99c075d640 10 monclient(hunting): authenticate will time out at 951.191535s 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99becfa640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 msgr2=0x7f99b8118650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99becfa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8118650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99b806e920 0x7f99b813f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40958/0 (socket says 172.21.15.17:40958) 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 -- 172.21.15.17:0/1762880613 learned_addr learned my addr 172.21.15.17:0/1762880613 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): get_auth_request con 0x7f99b806ecf0 auth_method 0 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): get_auth_request con 0x7f99b806e920 auth_method 0 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:51.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99be4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient(hunting): handle_auth_done global_id 44679 payload 824 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 -- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 msgr2=0x7f99b8118650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8118650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 -- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99b806e920 msgr2=0x7f99b813f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99b806e920 0x7f99b813f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient: _finish_hunting 0 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 monclient: found mon.noname-a 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.195+0000 7f99bdcf8640 1 -- 172.21.15.17:0/1762880613 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b4009ae0 con 0x7f99b806ecf0 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99bdcf8640 1 -- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 msgr2=0x7f99b812eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:51.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99bdcf8640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c5e0 tx=0x7f99a8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8012840 con 0x7f99b806ecf0 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: dump: 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:51.212 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: _finish_auth 0 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 5 monclient: authenticate success, global_id 44679 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a80129c0 con 0x7f99b806ecf0 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 10 monclient: _renew_subs 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b811ac80 con 0x7f99b806ecf0 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8012d80 con 0x7f99b806ecf0 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:51.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99ae7fc640 10 monclient: dump: 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 10 monclient: _renew_subs 2024-09-15T01:54:51.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99bccf6640 4 set_mon_vals no callback set 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.198+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b811a0b0 con 0x7f99b806ecf0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f99a8011590 con 0x7f99b806ecf0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99c075d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99c075d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99c075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b811c350 con 0x7f99b806ecf0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99ae7fc640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 0x7f999c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f99a804c0e0 con 0x7f99b806ecf0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99be4f9640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 0x7f999c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.199+0000 7f99be4f9640 10 monclient: get_auth_request con 0x7f999c038740 auth_method 0 2024-09-15T01:54:51.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.202+0000 7f99be4f9640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 0x7f999c03abf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f99b8067140 tx=0x7f99b4036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:51.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.203+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f99b811c350 con 0x7f99b806ecf0 2024-09-15T01:54:51.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.203+0000 7f99ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:51.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.203+0000 7f99ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:51.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:51.316+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:51.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:51.316+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:51.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:51.316+0000 7faf55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:51.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.427+0000 7f99c075d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:51.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.427+0000 7f99c075d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.427+0000 7f99c075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.427+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f99b80724a0 con 0x7f99b806ecf0 2024-09-15T01:54:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.428+0000 7f99ae7fc640 1 -- 172.21.15.17:0/1762880613 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f99a8016080 con 0x7f99b806ecf0 2024-09-15T01:54:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.428+0000 7f99ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:51.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.428+0000 7f99ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:51.431 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 msgr2=0x7f999c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 0x7f999c03abf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f99b8067140 tx=0x7f99b4036000 comp rx=0 tx=0).stop 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 10 monclient: shutdown 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 msgr2=0x7f99b812eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.429+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c5e0 tx=0x7f99a8007ed0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 shutdown_connections 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f999c038740 0x7f999c03abf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99b806ecf0 0x7f99b812eac0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99b806f110 0x7f99b8118650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 --2- 172.21.15.17:0/1762880613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99b806e920 0x7f99b813f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 >> 172.21.15.17:0/1762880613 conn(0x7f99b80a1000 msgr2=0x7f99b8132bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 shutdown_connections 2024-09-15T01:54:51.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:51.430+0000 7f99c075d640 1 -- 172.21.15.17:0/1762880613 wait complete. 2024-09-15T01:54:52.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:52.316+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:52.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:52.316+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:53.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:53.316+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:53.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:53.316+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:54.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:54.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:54.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:54.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:54.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.549+0000 7f4d5f35d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.549+0000 7f4d5f35d640 10 monclient: build_initial_monmap 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.549+0000 7f4d5f35d640 10 monclient: monmap: 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:54.550714+0000 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:54.550714+0000 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:54.552 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:54.553 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:54.553 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:54.553 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:54.553 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.550+0000 7f4d5f35d640 1 Processor -- start 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.550+0000 7f4d5f35d640 1 -- start start 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.550+0000 7f4d5f35d640 10 monclient: init 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d58097320 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d5809a590 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d581350f0 0x7f4d581374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:54.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f4d581350f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d580985d0 con 0x7f4d58097320 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5809d200 con 0x7f4d5809a590 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d580992f0 con 0x7f4d581350f0 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5f35d640 10 monclient(hunting): authenticate will time out at 954.547284s 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d0f9640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4d58097320 0x7f4d580976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51894/0 (socket says 172.21.15.17:51894) 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d0f9640 1 -- 172.21.15.17:0/3171586129 learned_addr learned my addr 172.21.15.17:0/3171586129 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 1 --2- 172.21.15.17:0/3171586129 >> v2:172.21.15.67:3300/0 conn(0x7f4d581350f0 0x7f4d581374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): get_auth_request con 0x7f4d581350f0 auth_method 0 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:54.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_done global_id 5311 payload 274 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3171586129 >> v1:172.21.15.67:6789/0 conn(0x7f4d5809a590 legacy=0x7f4d581339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3171586129 >> v1:172.21.15.17:6789/0 conn(0x7f4d58097320 legacy=0x7f4d580976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient: _finish_hunting 0 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 1 monclient: found mon.noname-c 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:54:54.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.551+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3171586129 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5809cb90 con 0x7f4d581350f0 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.554+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3171586129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d581350f0 msgr2=0x7f4d581374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.554+0000 7f4d5d8fa640 1 --2- 172.21.15.17:0/3171586129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d581350f0 0x7f4d581374d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009e60 tx=0x7f4d50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc0037979485fa09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 1 -- 172.21.15.17:0/3171586129 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50007630 con 0x7f4d581350f0 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: dump: 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.561 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: _finish_auth 0 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 1 -- 172.21.15.17:0/3171586129 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5002d050 con 0x7f4d581350f0 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 1 -- 172.21.15.17:0/3171586129 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500079c0 con 0x7f4d581350f0 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d4ffff640 10 monclient: dump: 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 5 monclient: authenticate success, global_id 5311 2024-09-15T01:54:54.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 4 set_mon_vals no callback set 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 10 monclient: shutdown 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3171586129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d581350f0 msgr2=0x7f4d581374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3171586129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d581350f0 0x7f4d581374d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009e60 tx=0x7f4d50009780 comp rx=0 tx=0).stop 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3171586129 shutdown_connections 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3171586129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d581350f0 0x7f4d581374d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.555+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3171586129 >> 172.21.15.17:0/3171586129 conn(0x7f4d580a1000 msgr2=0x7f4d580a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3171586129 shutdown_connections 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3171586129 wait complete. 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 10 monclient: build_initial_monmap 2024-09-15T01:54:54.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 10 monclient: monmap: 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 1 Processor -- start 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 1 -- start start 2024-09-15T01:54:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.556+0000 7f4d5f35d640 10 monclient: init 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 0x7f4d5812eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f4d58097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d5809a590 0x7f4d5809b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-b con 0x7f4d5809a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): picked mon.noname-a con 0x7f4d581350f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d58130b10 con 0x7f4d581350f0 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5809ad50 con 0x7f4d5809a590 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:54.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d58134d30 con 0x7f4d58097320 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5f35d640 10 monclient(hunting): authenticate will time out at 954.553550s 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d0f9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 msgr2=0x7f4d5812eec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d0f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 0x7f4d5812eec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35704/0 (socket says 172.21.15.17:35704) 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3351544818 learned_addr learned my addr 172.21.15.17:0/3351544818 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): get_auth_request con 0x7f4d581350f0 auth_method 0 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d5809a590 0x7f4d5809b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:54.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): get_auth_request con 0x7f4d5809a590 auth_method 0 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.557+0000 7f4d5c8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 10 monclient(hunting): handle_auth_done global_id 44682 payload 824 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 msgr2=0x7f4d5812eec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 0x7f4d5812eec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d5809a590 msgr2=0x7f4d5809b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d5809a590 0x7f4d5809b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 10 monclient: _finish_hunting 0 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 monclient: found mon.noname-a 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:54:54.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.558+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3351544818 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d50009a20 con 0x7f4d581350f0 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.560+0000 7f4d5d8fa640 1 -- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 msgr2=0x7f4d5809bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.560+0000 7f4d5d8fa640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009ec0 tx=0x7f4d50037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.560+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50007ec0 con 0x7f4d581350f0 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: dump: 2024-09-15T01:54:54.571 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50040070 con 0x7f4d581350f0 2024-09-15T01:54:54.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50031cf0 con 0x7f4d581350f0 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4d7fa640 10 monclient: dump: 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:54:54.573 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 5 monclient: authenticate success, global_id 44682 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 10 monclient: _renew_subs 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5812fb20 con 0x7f4d581350f0 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d4f7fe640 4 set_mon_vals no callback set 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 10 monclient: _renew_subs 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.562+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5806dea0 con 0x7f4d581350f0 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4d50042030 con 0x7f4d581350f0 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d5f35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d5f35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:54.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d5f35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d580724a0 con 0x7f4d581350f0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d4d7fa640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 0x7f4d3c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.563+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f4d5006d700 con 0x7f4d581350f0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.564+0000 7f4d5d0f9640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 0x7f4d3c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.564+0000 7f4d5d0f9640 10 monclient: get_auth_request con 0x7f4d3c038740 auth_method 0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.566+0000 7f4d5d0f9640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 0x7f4d3c03abf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d480047d0 tx=0x7f4d48007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.568+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4d5003a9a0 con 0x7f4d581350f0 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.568+0000 7f4d4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:54:54.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.568+0000 7f4d4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:54:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d5f35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d5f35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d5f35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4d580724a0 con 0x7f4d581350f0 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d4d7fa640 1 -- 172.21.15.17:0/3351544818 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f4d500393c0 con 0x7f4d581350f0 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.792+0000 7f4d4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:54:54.795 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 msgr2=0x7f4d3c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 0x7f4d3c03abf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d480047d0 tx=0x7f4d48007b80 comp rx=0 tx=0).stop 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 10 monclient: shutdown 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 msgr2=0x7f4d5809bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009ec0 tx=0x7f4d50037000 comp rx=0 tx=0).stop 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 shutdown_connections 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d3c038740 0x7f4d3c03abf0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d581350f0 0x7f4d5809bd90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d5809a590 0x7f4d5809b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 --2- 172.21.15.17:0/3351544818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d58097320 0x7f4d5812eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:54:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 >> 172.21.15.17:0/3351544818 conn(0x7f4d580a1000 msgr2=0x7f4d58135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:54:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 shutdown_connections 2024-09-15T01:54:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:54.794+0000 7f4d5f35d640 1 -- 172.21.15.17:0/3351544818 wait complete. 2024-09-15T01:54:55.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:55.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:55.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:55.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:56.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:56.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:56.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:56.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:57.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:57.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:57.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.914+0000 7f38f9b5d640 10 monclient: get_monmap_and_config 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.914+0000 7f38f9b5d640 10 monclient: build_initial_monmap 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.914+0000 7f38f9b5d640 10 monclient: monmap: 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:54:57.915148+0000 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:54:57.915148+0000 2024-09-15T01:54:57.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:54:57.917 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 Processor -- start 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 -- start start 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient: init 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38f4098190 addr v1:172.21.15.17:6789/0 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38f409cfa0 addr v1:172.21.15.67:6789/0 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f38f4137e80 addr v2:172.21.15.67:3300/0 2024-09-15T01:54:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f409e380 con 0x7f38f4098190 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f4098c70 con 0x7f38f409cfa0 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f409d730 con 0x7f38f4137e80 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f9b5d640 10 monclient(hunting): authenticate will time out at 957.911719s 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f37fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f38f4098190 0x7f38f4131fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51910/0 (socket says 172.21.15.17:51910) 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.915+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 learned_addr learned my addr 172.21.15.17:0/3077512424 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:57.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.916+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.918+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v1:172.21.15.17:6789/0 conn(0x7f38f4098190 legacy=0x7f38f4131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:57.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:57.918+0000 7f38eaffd640 1 -- 172.21.15.17:0/3077512424 >> v1:172.21.15.67:6789/0 conn(0x7f38f409cfa0 legacy=0x7f38f4098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:58.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.117+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:54:58.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:58.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:58.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:58.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:58.518+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:54:59.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:59.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:54:59.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:54:59.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:54:59.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:54:59.320+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:55:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:00.317+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:55:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:00.317+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:00.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): tick 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38f413a260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38f413a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 >> v1:172.21.15.67:6789/0 conn(0x7f38f409cfa0 legacy=0x7f38f4098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 >> v1:172.21.15.17:6789/0 conn(0x7f38f4098190 legacy=0x7f38f4131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient: _add_conns ranks=[3,2] 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f38d40037a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f38d40062d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f409d730 con 0x7f38d40037a0 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4008e90 con 0x7f38d40062d0 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f1ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38eaffd640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38eaffd640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d40037a0 auth_method 0 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:00.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:01.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38eaffd640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:01.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38eaffd640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:01.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d40037a0 auth_method 0 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:01.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.117+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:01.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:01.318+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:55:01.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:01.318+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:01.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:01.318+0000 7faf55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:01.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38eaffd640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:01.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38eaffd640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d40037a0 auth_method 0 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:01.518+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.318+0000 7faf55ffb640 10 monclient: tick 2024-09-15T01:55:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.318+0000 7faf55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:02.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38eaffd640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:02.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38eaffd640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:55:02.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:02.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d40037a0 auth_method 0 2024-09-15T01:55:02.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:02.319+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:55:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.505+0000 7faf56ffd640 1 -- 172.21.15.67:0/912339032 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (secure 0 0 0) 0x7faf4803e970 con 0x7faf6012e7b0 2024-09-15T01:55:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.505+0000 7faf56ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T01:55:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.505+0000 7faf56ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:02.508 INFO:tasks.workunit.client.0.smithi067.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T01:55:02.508 INFO:tasks.workunit.client.0.smithi067.stdout:-16 1.00000 host host2 2024-09-15T01:55:02.509 INFO:tasks.workunit.client.0.smithi067.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-15T01:55:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 msgr2=0x7faf4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 0x7faf4003abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf50007fd0 tx=0x7faf50009040 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 10 monclient: shutdown 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 msgr2=0x7faf60097ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7faf48009a20 tx=0x7faf48030000 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 shutdown_connections 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf40038740 0x7faf4003abf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf6012eb80 0x7faf60098220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf6012e7b0 0x7faf60097ce0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 --2- 172.21.15.67:0/912339032 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf4c004780 0x7faf6012e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 >> 172.21.15.67:0/912339032 conn(0x7faf600710f0 msgr2=0x7faf6009e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 shutdown_connections 2024-09-15T01:55:02.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.509+0000 7faf6539d640 1 -- 172.21.15.67:0/912339032 wait complete. 2024-09-15T01:55:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-15T01:55:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:55:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree-from osd.9 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 10 monclient: build_initial_monmap 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 10 monclient: monmap: 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:02.603 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:02.602050+0000 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:02.602050+0000 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:02.604 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 1 Processor -- start 2024-09-15T01:55:02.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 1 -- start start 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.601+0000 7f59ca64f640 10 monclient: init 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59c4130e00 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f59c41311d0 0x7f59c41315f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-a con 0x7f59c41311d0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 0x7f59c41392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-b con 0x7f59c4136f10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c409ff20 con 0x7f59c4130e00 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c41320b0 con 0x7f59c41311d0 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:02.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40a17b0 con 0x7f59c4136f10 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59ca64f640 10 monclient(hunting): authenticate will time out at 935.071351s 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f59c4130e00 0x7f59c4132430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49838/0 (socket says 172.21.15.67:49838) 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c3fff640 1 -- 172.21.15.67:0/2896518075 learned_addr learned my addr 172.21.15.67:0/2896518075 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c8bec640 1 -- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 msgr2=0x7f59c41392f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c8bec640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 0x7f59c41392f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 1 --2- 172.21.15.67:0/2896518075 >> v2:172.21.15.17:3300/0 conn(0x7f59c41311d0 0x7f59c41315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): get_auth_request con 0x7f59c41311d0 auth_method 0 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:02.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 10 monclient(hunting): handle_auth_done global_id 44685 payload 274 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.602+0000 7f59c37fe640 1 -- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 msgr2=0x7f59c41392f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 0x7f59c41392f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 1 -- 172.21.15.67:0/2896518075 >> v1:172.21.15.67:6789/0 conn(0x7f59c4130e00 legacy=0x7f59c4132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.603+0000 7f59c37fe640 1 -- 172.21.15.67:0/2896518075 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c40a00f0 con 0x7f59c41311d0 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.605+0000 7f59c37fe640 1 -- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c41311d0 msgr2=0x7f59c41315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.605+0000 7f59c37fe640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c41311d0 0x7f59c41315f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009850 tx=0x7f59ac0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba1baf9bf6e5843 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 1 -- 172.21.15.67:0/2896518075 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac032440 con 0x7f59c41311d0 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: dump: 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 1 -- 172.21.15.67:0/2896518075 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac0325a0 con 0x7f59c41311d0 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 1 -- 172.21.15.67:0/2896518075 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac032940 con 0x7f59c41311d0 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59c2ffd640 10 monclient: dump: 2024-09-15T01:55:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 5 monclient: authenticate success, global_id 44685 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 4 set_mon_vals no callback set 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 10 monclient: shutdown 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 -- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c41311d0 msgr2=0x7f59c41315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c41311d0 0x7f59c41315f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009850 tx=0x7f59ac0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 -- 172.21.15.67:0/2896518075 shutdown_connections 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c4136f10 0x7f59c41392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 --2- 172.21.15.67:0/2896518075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c41311d0 0x7f59c41315f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 -- 172.21.15.67:0/2896518075 >> 172.21.15.67:0/2896518075 conn(0x7f59c40710f0 msgr2=0x7f59c40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.606+0000 7f59ca64f640 1 -- 172.21.15.67:0/2896518075 shutdown_connections 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.607+0000 7f59ca64f640 1 -- 172.21.15.67:0/2896518075 wait complete. 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.607+0000 7f59ca64f640 10 monclient: build_initial_monmap 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.607+0000 7f59ca64f640 10 monclient: monmap: 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.607+0000 7f59ca64f640 1 Processor -- start 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 -- start start 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient: init 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 0x7f59c412ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-b con 0x7f59c4130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c41311d0 0x7f59c4097da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-c con 0x7f59c41311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c4136f10 0x7f59c40982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): picked mon.noname-a con 0x7f59c4136f10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40a0820 con 0x7f59c4136f10 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40a3e80 con 0x7f59c4130e00 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c41421c0 con 0x7f59c41311d0 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59ca64f640 10 monclient(hunting): authenticate will time out at 935.077555s 2024-09-15T01:55:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 0x7f59c412ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 0x7f59c412ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48434/0 (socket says 172.21.15.67:48434) 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 -- 172.21.15.67:0/3705779198 learned_addr learned my addr 172.21.15.67:0/3705779198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): get_auth_request con 0x7f59c4130e00 auth_method 0 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c37fe640 1 -- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c41311d0 msgr2=0x7f59c4097da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c37fe640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c41311d0 0x7f59c4097da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c8bec640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c4136f10 0x7f59c40982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c8bec640 10 monclient(hunting): get_auth_request con 0x7f59c4136f10 auth_method 0 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c8bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c8bec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c8bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient(hunting): handle_auth_done global_id 5317 payload 824 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 -- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c41311d0 msgr2=0x7f59c4097da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59c41311d0 0x7f59c4097da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 -- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c4136f10 msgr2=0x7f59c40982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59c4136f10 0x7f59c40982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 monclient: found mon.noname-b 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.608+0000 7f59c3fff640 1 -- 172.21.15.67:0/3705779198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac009390 con 0x7f59c4130e00 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59c3fff640 1 -- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 msgr2=0x7f59c412ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59c3fff640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 0x7f59c412ead0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f59b400c640 tx=0x7f59b4004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4005cb0 con 0x7f59c4130e00 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: dump: 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: _finish_auth 0 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 5 monclient: authenticate success, global_id 5317 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b4005e30 con 0x7f59c4130e00 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 10 monclient: _renew_subs 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 1 -- 172.21.15.67:0/3705779198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c4099ec0 con 0x7f59c4130e00 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4007340 con 0x7f59c4130e00 2024-09-15T01:55:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59bbfff640 10 monclient: dump: 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59c27fc640 4 set_mon_vals no callback set 2024-09-15T01:55:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 10 monclient: _renew_subs 2024-09-15T01:55:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.611+0000 7f59ca64f640 1 -- 172.21.15.67:0/3705779198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c40999b0 con 0x7f59c4130e00 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f59b401b400 con 0x7f59c4130e00 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59ca64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59ca64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59ca64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59ca64f640 1 -- 172.21.15.67:0/3705779198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c404bf60 con 0x7f59c4130e00 2024-09-15T01:55:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59bbfff640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59a40386f0 0x7f59a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f59b404bbb0 con 0x7f59c4130e00 2024-09-15T01:55:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59c37fe640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59a40386f0 0x7f59a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.612+0000 7f59c37fe640 10 monclient: get_auth_request con 0x7f59a40386f0 auth_method 0 2024-09-15T01:55:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.615+0000 7f59c37fe640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59a40386f0 0x7f59a403aba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f59ac031000 tx=0x7f59ac032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:02.616+0000 7f59c3fff640 1 --2- 172.21.15.67:0/3705779198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59c4130e00 0x7f59c412ead0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f59b400c640 tx=0x7f59b4004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f59b40076f0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:55:03.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.445+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f59b40076f0 con 0x7f59c4130e00 2024-09-15T01:55:03.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.445+0000 7f59bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.445+0000 7f59bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:03.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59ca64f640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59ca64f640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59ca64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59ca64f640 1 -- 172.21.15.67:0/3705779198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) v1 -- 0x7f59c40a14d0 con 0x7f59c4130e00 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59bbfff640 1 -- 172.21.15.67:0/3705779198 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v78) v1 ==== 103+0+0 (secure 0 0 0) 0x7f59b401a360 con 0x7f59c4130e00 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.617+0000 7f59bbfff640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-15T01:55:03.619 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-15T01:55:03.622 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:55:03.622 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rm bar 2024-09-15T01:55:03.622 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:55:03.622 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm bar 2024-09-15T01:55:03.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.696+0000 7f617035d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.696+0000 7f617035d640 10 monclient: build_initial_monmap 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.696+0000 7f617035d640 10 monclient: monmap: 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:03.696857+0000 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:03.696857+0000 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:03.698 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:03.699 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:03.699 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:03.699 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 1 Processor -- start 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 1 -- start start 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient: init 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61680a0b50 0x7f6168069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient(hunting): picked mon.noname-c con 0x7f61680a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient(hunting): picked mon.noname-b con 0x7f616809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.697+0000 7f617035d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6168131b10 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61680a61a0 con 0x7f6168131b10 2024-09-15T01:55:03.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61680a1610 con 0x7f61680a0b50 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616809f7d0 con 0x7f616809eea0 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f617035d640 10 monclient(hunting): authenticate will time out at 936.166030s 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61680a0b50 0x7f6168069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616d8f8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 msgr2=0x7f616809f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e8fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6168131b10 0x7f6168131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49844/0 (socket says 172.21.15.67:49844) 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61680a0b50 0x7f6168069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48438/0 (socket says 172.21.15.67:48438) 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e8fa640 1 -- 172.21.15.67:0/2640796077 learned_addr learned my addr 172.21.15.67:0/2640796077 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616d8f8640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): get_auth_request con 0x7f61680a0b50 auth_method 0 2024-09-15T01:55:03.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient(hunting): handle_auth_done global_id 5320 payload 274 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 -- 172.21.15.67:0/2640796077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 msgr2=0x7f616809f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 -- 172.21.15.67:0/2640796077 >> v1:172.21.15.67:6789/0 conn(0x7f6168131b10 legacy=0x7f6168131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 monclient: found mon.noname-c 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.698+0000 7f616e0f9640 1 -- 172.21.15.67:0/2640796077 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61680a04c0 con 0x7f61680a0b50 2024-09-15T01:55:03.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616e0f9640 1 -- 172.21.15.67:0/2640796077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 msgr2=0x7f6168069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616e0f9640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168069380 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6164009850 tx=0x7f61640090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c38998ac6ee97ec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 1 -- 172.21.15.67:0/2640796077 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164032440 con 0x7f61680a0b50 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: dump: 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.708 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 1 -- 172.21.15.67:0/2640796077 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640325a0 con 0x7f61680a0b50 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 5 monclient: authenticate success, global_id 5320 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 4 set_mon_vals no callback set 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 1 -- 172.21.15.67:0/2640796077 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164032940 con 0x7f61680a0b50 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f616d0f7640 10 monclient: dump: 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 10 monclient: shutdown 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 -- 172.21.15.67:0/2640796077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 msgr2=0x7f6168069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168069380 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6164009850 tx=0x7f61640090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 -- 172.21.15.67:0/2640796077 shutdown_connections 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 --2- 172.21.15.67:0/2640796077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168069380 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.701+0000 7f617035d640 1 -- 172.21.15.67:0/2640796077 >> 172.21.15.67:0/2640796077 conn(0x7f6168071330 msgr2=0x7f61680a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 1 -- 172.21.15.67:0/2640796077 shutdown_connections 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 1 -- 172.21.15.67:0/2640796077 wait complete. 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 10 monclient: build_initial_monmap 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 10 monclient: monmap: 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 1 Processor -- start 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 1 -- start start 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.702+0000 7f617035d640 10 monclient: init 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): picked mon.noname-c con 0x7f616809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): picked mon.noname-b con 0x7f61680a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6168131b10 0x7f616811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): picked mon.noname-a con 0x7f6168131b10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61680a1260 con 0x7f6168131b10 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6168134c30 con 0x7f61680a0b50 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6168141f50 con 0x7f616809eea0 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f617035d640 10 monclient(hunting): authenticate will time out at 936.171264s 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616e0f9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 msgr2=0x7f616809a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616e0f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809a820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168130ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48454/0 (socket says 172.21.15.67:48454) 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 -- 172.21.15.67:0/170365992 learned_addr learned my addr 172.21.15.67:0/170365992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): get_auth_request con 0x7f61680a0b50 auth_method 0 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient(hunting): handle_auth_done global_id 5323 payload 824 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616e8fa640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6168131b10 0x7f616811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 -- 172.21.15.67:0/170365992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 msgr2=0x7f616809a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f616809eea0 0x7f616809a820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 -- 172.21.15.67:0/170365992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6168131b10 msgr2=0x7f616811b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6168131b10 0x7f616811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 monclient: found mon.noname-b 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.703+0000 7f616d8f8640 1 -- 172.21.15.67:0/170365992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009390 con 0x7f61680a0b50 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.705+0000 7f616d8f8640 1 -- 172.21.15.67:0/170365992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 msgr2=0x7f6168130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.705+0000 7f616d8f8640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680a0b50 0x7f6168130ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f615c00b630 tx=0x7f615c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c015070 con 0x7f61680a0b50 2024-09-15T01:55:03.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: dump: 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c003d90 con 0x7f61680a0b50 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 5 monclient: authenticate success, global_id 5323 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 10 monclient: _renew_subs 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 1 -- 172.21.15.67:0/170365992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616811dcc0 con 0x7f61680a0b50 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c006380 con 0x7f61680a0b50 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f615a7fc640 10 monclient: dump: 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.717 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f616c8f6640 4 set_mon_vals no callback set 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 10 monclient: _renew_subs 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.706+0000 7f617035d640 1 -- 172.21.15.67:0/170365992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6168136520 con 0x7f61680a0b50 2024-09-15T01:55:03.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f615c010440 con 0x7f61680a0b50 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f615a7fc640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f614c0386f0 0x7f614c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f615c04b9f0 con 0x7f61680a0b50 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f617035d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f617035d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f617035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f616e0f9640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f614c0386f0 0x7f614c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f617035d640 1 -- 172.21.15.67:0/170365992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616811e160 con 0x7f61680a0b50 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.707+0000 7f616e0f9640 10 monclient: get_auth_request con 0x7f614c0386f0 auth_method 0 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.710+0000 7f616e0f9640 1 --2- 172.21.15.67:0/170365992 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f614c0386f0 0x7f614c03aba0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f616809ffe0 tx=0x7f6164032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.711+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f615c006770 con 0x7f61680a0b50 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.712+0000 7f615a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.712+0000 7f615a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:03.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.877+0000 7f617035d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:03.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.877+0000 7f617035d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:03.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.877+0000 7f617035d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.877+0000 7f617035d640 1 -- 172.21.15.67:0/170365992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f616804bf60 con 0x7f61680a0b50 2024-09-15T01:55:03.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.878+0000 7f615a7fc640 1 -- 172.21.15.67:0/170365992 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v78) v1 ==== 99+0+0 (secure 0 0 0) 0x7f615c018a70 con 0x7f61680a0b50 2024-09-15T01:55:03.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.878+0000 7f615a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:03.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.878+0000 7f615a7fc640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T01:55:03.880 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T01:55:03.883 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:55:03.883 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush unlink host2 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): tick 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d40062d0 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 msgr2=0x7f38d4005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d40037a0 0x7f38d4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d4009290 0x7f38d400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 reap_dead start 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f38d4009290 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38d400bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f38f4137e80 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 0x7f38d4008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f38d4005c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f409d730 con 0x7f38d4009290 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d4008e90 con 0x7f38f4137e80 2024-09-15T01:55:03.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 1 -- 172.21.15.17:0/3077512424 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d400c280 con 0x7f38d4005c20 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f1ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f3fff640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f3fff640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.17:3300/0 conn(0x7f38d4009290 0x7f38d400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d4009290 auth_method 0 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38d400bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient(hunting): handle_auth_done global_id 44691 payload 274 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 msgr2=0x7f38d4008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 0x7f38d4008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 msgr2=0x7f38d400bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38d400bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:03.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.916+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f406eff0 con 0x7f38d4009290 2024-09-15T01:55:03.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.919+0000 7f38f37fe640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38d4009290 msgr2=0x7f38d400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.919+0000 7f38f37fe640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38d4009290 0x7f38d400b670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38d4003be0 tx=0x7f38e4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe62ce45182107aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 1 -- 172.21.15.17:0/3077512424 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4033440 con 0x7f38d4009290 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: dump: 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.929 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 1 -- 172.21.15.17:0/3077512424 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e40335a0 con 0x7f38d4009290 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 1 -- 172.21.15.17:0/3077512424 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4033960 con 0x7f38d4009290 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 5 monclient: authenticate success, global_id 44691 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 4 set_mon_vals no callback set 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:03.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f2ffd640 10 monclient: dump: 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 10 monclient: shutdown 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:03.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 1 -- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38d4009290 msgr2=0x7f38d400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38d4009290 0x7f38d400b670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38d4003be0 tx=0x7f38e4009040 comp rx=0 tx=0).stop 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 1 -- 172.21.15.17:0/3077512424 shutdown_connections 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.920+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/3077512424 >> v2:172.21.15.67:3300/0 conn(0x7f38f4137e80 0x7f38d400bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4005c20 0x7f38d4008760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/3077512424 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38d4009290 0x7f38d400b670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 1 -- 172.21.15.17:0/3077512424 >> 172.21.15.17:0/3077512424 conn(0x7f38f40a2100 msgr2=0x7f38f40a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 1 -- 172.21.15.17:0/3077512424 shutdown_connections 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 1 -- 172.21.15.17:0/3077512424 wait complete. 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 10 monclient: build_initial_monmap 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.921+0000 7f38f9b5d640 10 monclient: monmap: 2024-09-15T01:55:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 Processor -- start 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 -- start start 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient: init 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 0x7f38f4145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f38d4005c20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 0x7f38f4145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f38d4009290 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f38f4137e80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f409d900 con 0x7f38f4137e80 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f40a0b10 con 0x7f38d4005c20 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f406c1c0 con 0x7f38d4009290 2024-09-15T01:55:03.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.922+0000 7f38f9b5d640 10 monclient(hunting): authenticate will time out at 963.918966s 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38eaffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 msgr2=0x7f38f4145a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 0x7f38f4145a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33112/0 (socket says 172.21.15.17:33112) 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 1 -- 172.21.15.17:0/2954226210 learned_addr learned my addr 172.21.15.17:0/2954226210 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f4137e80 auth_method 0 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 0x7f38f4145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38d4005c20 auth_method 0 2024-09-15T01:55:03.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient(hunting): handle_auth_done global_id 5329 payload 824 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 -- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 msgr2=0x7f38f4145a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 0x7f38f4145a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 -- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 msgr2=0x7f38f4135a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f37fe640 1 -- 172.21.15.17:0/2954226210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e400f040 con 0x7f38d4005c20 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.923+0000 7f38f3fff640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f37fe640 1 -- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 msgr2=0x7f38f4145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f37fe640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 0x7f38f4145550 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38e4036000 tx=0x7f38e4039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4005470 con 0x7f38d4005c20 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: dump: 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:03.937 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e40055f0 con 0x7f38d4005c20 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e40049f0 con 0x7f38d4005c20 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 5 monclient: authenticate success, global_id 5329 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 10 monclient: _renew_subs 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f406bb90 con 0x7f38d4005c20 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f27fc640 4 set_mon_vals no callback set 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38ebfff640 10 monclient: dump: 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 10 monclient: _renew_subs 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.926+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f406ac30 con 0x7f38d4005c20 2024-09-15T01:55:03.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f38e403d410 con 0x7f38d4005c20 2024-09-15T01:55:03.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38f9b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38f9b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38f9b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f40724a0 con 0x7f38d4005c20 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38ebfff640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 0x7f38d403c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f38e406cf50 con 0x7f38d4005c20 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38eaffd640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 0x7f38d403c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.927+0000 7f38eaffd640 10 monclient: get_auth_request con 0x7f38d403a010 auth_method 0 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.930+0000 7f38eaffd640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 0x7f38d403c4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f38f409d6a0 tx=0x7f38dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.932+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f38e404d580 con 0x7f38d4005c20 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.932+0000 7f38ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:03.932+0000 7f38ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 10 monclient: get_monmap_and_config 2024-09-15T01:55:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 10 monclient: build_initial_monmap 2024-09-15T01:55:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 10 monclient: monmap: 2024-09-15T01:55:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:03.958305+0000 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:03.958305+0000 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:03.960 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 1 Processor -- start 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 1 -- start start 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.958+0000 7f76eed54640 10 monclient: init 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 0x7f76e81321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-b con 0x7f76e8098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76e80a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f76e8098c20 0x7f76e81390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-a con 0x7f76e8098c20 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:03.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80a0b80 con 0x7f76e80a0370 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80a1730 con 0x7f76e8098c20 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e80a00b0 con 0x7f76e8098140 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76eed54640 10 monclient(hunting): authenticate will time out at 936.427519s 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ecaf0640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 msgr2=0x7f76e81321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ecaf0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 0x7f76e81321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76e7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76e80a0370 0x7f76e8098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49850/0 (socket says 172.21.15.67:49850) 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76e7fff640 1 -- 172.21.15.67:0/4232526034 learned_addr learned my addr 172.21.15.67:0/4232526034 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:03.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 1 --2- 172.21.15.67:0/4232526034 >> v2:172.21.15.17:3300/0 conn(0x7f76e8098c20 0x7f76e81390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): get_auth_request con 0x7f76e8098c20 auth_method 0 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.959+0000 7f76ed2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 10 monclient(hunting): handle_auth_done global_id 44697 payload 274 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 1 -- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 msgr2=0x7f76e81321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 1 --2- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 0x7f76e81321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 1 -- 172.21.15.67:0/4232526034 >> v1:172.21.15.67:6789/0 conn(0x7f76e80a0370 legacy=0x7f76e8098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 1 monclient: found mon.noname-a 2024-09-15T01:55:03.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:03.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.960+0000 7f76ed2f1640 1 -- 172.21.15.67:0/4232526034 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8071390 con 0x7f76e8098c20 2024-09-15T01:55:03.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.962+0000 7f76ed2f1640 1 -- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 msgr2=0x7f76e81390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.962+0000 7f76ed2f1640 1 --2- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e81390c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009850 tx=0x7f76d00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0f98530b77e4d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 1 -- 172.21.15.67:0/4232526034 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0032440 con 0x7f76e8098c20 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: dump: 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 5 monclient: authenticate success, global_id 44697 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 1 -- 172.21.15.67:0/4232526034 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d00325a0 con 0x7f76e8098c20 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 1 -- 172.21.15.67:0/4232526034 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0032940 con 0x7f76e8098c20 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 4 set_mon_vals no callback set 2024-09-15T01:55:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76e77fe640 10 monclient: dump: 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 10 monclient: shutdown 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 -- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 msgr2=0x7f76e81390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 --2- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e81390c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009850 tx=0x7f76d00090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 -- 172.21.15.67:0/4232526034 shutdown_connections 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 --2- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e81390c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 --2- 172.21.15.67:0/4232526034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e8098140 0x7f76e81321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 -- 172.21.15.67:0/4232526034 >> 172.21.15.67:0/4232526034 conn(0x7f76e80a5780 msgr2=0x7f76e80a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 -- 172.21.15.67:0/4232526034 shutdown_connections 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.963+0000 7f76eed54640 1 -- 172.21.15.67:0/4232526034 wait complete. 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.964+0000 7f76eed54640 10 monclient: build_initial_monmap 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.964+0000 7f76eed54640 10 monclient: monmap: 2024-09-15T01:55:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.964+0000 7f76eed54640 1 Processor -- start 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.964+0000 7f76eed54640 1 -- start start 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.964+0000 7f76eed54640 10 monclient: init 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:03.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-b con 0x7f76e8098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e8130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-a con 0x7f76e8098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 0x7f76e8131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): picked mon.noname-c con 0x7f76e80a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8133440 con 0x7f76e8098c20 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8099090 con 0x7f76e8098140 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e809f1f0 con 0x7f76e80a0370 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76eed54640 10 monclient(hunting): authenticate will time out at 936.433375s 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ed2f1640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 msgr2=0x7f76e8131250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ed2f1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 0x7f76e8131250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48460/0 (socket says 172.21.15.67:48460) 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 -- 172.21.15.67:0/3889014091 learned_addr learned my addr 172.21.15.67:0/3889014091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): get_auth_request con 0x7f76e8098140 auth_method 0 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e8130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): get_auth_request con 0x7f76e8098c20 auth_method 0 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient(hunting): handle_auth_done global_id 5332 payload 824 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 -- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 msgr2=0x7f76e8131250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 0x7f76e8131250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 -- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 msgr2=0x7f76e8130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e8130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient: _finish_hunting 0 2024-09-15T01:55:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 monclient: found mon.noname-b 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.965+0000 7f76ecaf0640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0009390 con 0x7f76e8098140 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.967+0000 7f76ecaf0640 1 -- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 msgr2=0x7f76e8138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.967+0000 7f76ecaf0640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f76e000bbb0 tx=0x7f76e00097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e000cc80 con 0x7f76e8098140 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: dump: 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 5 monclient: authenticate success, global_id 5332 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e000cde0 con 0x7f76e8098140 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0016430 con 0x7f76e8098140 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 10 monclient: _renew_subs 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e81349c0 con 0x7f76e8098140 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e67fc640 4 set_mon_vals no callback set 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76e4ff9640 10 monclient: dump: 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:03.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 10 monclient: _renew_subs 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.968+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e8135680 con 0x7f76e8098140 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f76e0012020 con 0x7f76e8098140 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76eed54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76eed54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76eed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e8134c00 con 0x7f76e8098140 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76e4ff9640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 0x7f76c803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f76e004bf50 con 0x7f76e8098140 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76e7fff640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 0x7f76c803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.969+0000 7f76e7fff640 10 monclient: get_auth_request con 0x7f76c80386f0 auth_method 0 2024-09-15T01:55:03.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.971+0000 7f76e7fff640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 0x7f76c803aba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76e8138eb0 tx=0x7f76d8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:03.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.973+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f76e001d900 con 0x7f76e8098140 2024-09-15T01:55:03.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.973+0000 7f76e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:03.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:03.973+0000 7f76e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.142+0000 7f76eed54640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.142+0000 7f76eed54640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.142+0000 7f76eed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.142+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f76e8134f10 con 0x7f76e8098140 2024-09-15T01:55:04.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.165+0000 7f38f9b5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:04.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.165+0000 7f38f9b5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:04.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.165+0000 7f38f9b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:04.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.165+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f38f40689a0 con 0x7f38d4005c20 2024-09-15T01:55:04.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.166+0000 7f38ebfff640 1 -- 172.21.15.17:0/2954226210 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f38e4041020 con 0x7f38d4005c20 2024-09-15T01:55:04.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.166+0000 7f38ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:04.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.166+0000 7f38ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:04.168 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 msgr2=0x7f38d403c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 0x7f38d403c4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f38f409d6a0 tx=0x7f38dc009040 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 10 monclient: shutdown 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 msgr2=0x7f38f4145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 0x7f38f4145550 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38e4036000 tx=0x7f38e4039000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 shutdown_connections 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38d403a010 0x7f38d403c4c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4137e80 0x7f38f4135a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38d4009290 0x7f38f4145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 --2- 172.21.15.17:0/2954226210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4005c20 0x7f38f4145550 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 >> 172.21.15.17:0/2954226210 conn(0x7f38f40a2100 msgr2=0x7f38f409b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:04.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 shutdown_connections 2024-09-15T01:55:04.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.168+0000 7f38f9b5d640 1 -- 172.21.15.17:0/2954226210 wait complete. 2024-09-15T01:55:04.180 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:55:04.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:55:04.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:55:04.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.277+0000 7fac8e700640 10 monclient: get_monmap_and_config 2024-09-15T01:55:04.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.277+0000 7fac8e700640 10 monclient: build_initial_monmap 2024-09-15T01:55:04.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.277+0000 7fac8e700640 10 monclient: monmap: 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:04.278311+0000 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:04.278311+0000 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:04.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 Processor -- start 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 -- start start 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient: init 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 0x7fac88132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-b con 0x7fac88067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fac8809ce00 0x7fac88067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-a con 0x7fac8809ce00 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac880679a0 0x7fac8812e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-c con 0x7fac880679a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8809e4c0 con 0x7fac8809ce00 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8809cbf0 con 0x7fac880679a0 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8809e1e0 con 0x7fac88067090 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac87fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 msgr2=0x7fac88132140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac8e700640 10 monclient(hunting): authenticate will time out at 964.274802s 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac87fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 0x7fac88132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:04.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fac8809ce00 0x7fac88067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fac8809ce00 0x7fac88067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33128/0 (socket says 172.21.15.17:33128) 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 1 -- 172.21.15.17:0/3034413473 learned_addr learned my addr 172.21.15.17:0/3034413473 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 10 monclient(hunting): get_auth_request con 0x7fac8809ce00 auth_method 0 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.278+0000 7fac877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient(hunting): handle_auth_done global_id 44703 payload 274 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 -- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 msgr2=0x7fac88132140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 --2- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 0x7fac88132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac8cc9d640 1 --2- 172.21.15.17:0/3034413473 >> v2:172.21.15.67:3300/0 conn(0x7fac880679a0 0x7fac8812e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 -- 172.21.15.17:0/3034413473 >> v2:172.21.15.67:3300/0 conn(0x7fac880679a0 msgr2=0x7fac8812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 --2- 172.21.15.17:0/3034413473 >> v2:172.21.15.67:3300/0 conn(0x7fac880679a0 0x7fac8812e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.279+0000 7fac877fe640 1 -- 172.21.15.17:0/3034413473 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88067e00 con 0x7fac8809ce00 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac877fe640 1 -- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac8809ce00 msgr2=0x7fac88067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac877fe640 1 --2- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac8809ce00 0x7fac88067460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fac8809c970 tx=0x7fac7c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc26785812da867 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 1 -- 172.21.15.17:0/3034413473 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c032440 con 0x7fac8809ce00 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: dump: 2024-09-15T01:55:04.293 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 5 monclient: authenticate success, global_id 44703 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:04.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 1 -- 172.21.15.17:0/3034413473 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c0325a0 con 0x7fac8809ce00 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 1 -- 172.21.15.17:0/3034413473 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c032940 con 0x7fac8809ce00 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 4 set_mon_vals no callback set 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac86ffd640 10 monclient: dump: 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.295 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 10 monclient: shutdown 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 -- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac8809ce00 msgr2=0x7fac88067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 --2- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac8809ce00 0x7fac88067460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fac8809c970 tx=0x7fac7c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 -- 172.21.15.17:0/3034413473 shutdown_connections 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 --2- 172.21.15.17:0/3034413473 >> v2:172.21.15.67:3300/0 conn(0x7fac880679a0 0x7fac8812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 --2- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac8809ce00 0x7fac88067460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 --2- 172.21.15.17:0/3034413473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac88067090 0x7fac88132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 -- 172.21.15.17:0/3034413473 >> 172.21.15.17:0/3034413473 conn(0x7fac880a1ee0 msgr2=0x7fac880a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 -- 172.21.15.17:0/3034413473 shutdown_connections 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.282+0000 7fac8e700640 1 -- 172.21.15.17:0/3034413473 wait complete. 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.283+0000 7fac8e700640 10 monclient: build_initial_monmap 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.283+0000 7fac8e700640 10 monclient: monmap: 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:04.297 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.283+0000 7fac8e700640 1 Processor -- start 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.283+0000 7fac8e700640 1 -- start start 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.283+0000 7fac8e700640 10 monclient: init 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac88067090 0x7fac88130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-b con 0x7fac88067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 0x7fac8811a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-a con 0x7fac880679a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac8809ce00 0x7fac8811aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): picked mon.noname-c con 0x7fac8809ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:04.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8811dd70 con 0x7fac880679a0 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8809d5c0 con 0x7fac88067090 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac88134c60 con 0x7fac8809ce00 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8e700640 10 monclient(hunting): authenticate will time out at 964.280397s 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8cc9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac8809ce00 msgr2=0x7fac8811aae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac8cc9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac8809ce00 0x7fac8811aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 0x7fac8811a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 0x7fac8811a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33140/0 (socket says 172.21.15.17:33140) 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 -- 172.21.15.17:0/1347451960 learned_addr learned my addr 172.21.15.17:0/1347451960 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:04.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac88067090 0x7fac88130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): get_auth_request con 0x7fac880679a0 auth_method 0 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): get_auth_request con 0x7fac88067090 auth_method 0 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient(hunting): handle_auth_done global_id 44706 payload 824 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac8809ce00 msgr2=0x7fac8811aae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac8809ce00 0x7fac8811aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac88067090 msgr2=0x7fac88130970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac88067090 0x7fac88130970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.284+0000 7fac877fe640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c009450 con 0x7fac880679a0 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac877fe640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 msgr2=0x7fac8811a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac877fe640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 0x7fac8811a5a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009210 tx=0x7fac7c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c031a90 con 0x7fac880679a0 2024-09-15T01:55:04.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: dump: 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:04.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 5 monclient: authenticate success, global_id 44706 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c031bf0 con 0x7fac880679a0 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 10 monclient: _renew_subs 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c03b400 con 0x7fac880679a0 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac88141f60 con 0x7fac880679a0 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac6ffff640 10 monclient: dump: 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.303 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac867fc640 4 set_mon_vals no callback set 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 10 monclient: _renew_subs 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.287+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8811cb30 con 0x7fac880679a0 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8e700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8e700640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 0x7fac880298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8811dc30 con 0x7fac881364f0 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fac7c039040 con 0x7fac880679a0 2024-09-15T01:55:04.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 0x7fac880298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient: get_auth_request con 0x7fac881364f0 auth_method 0 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac6ffff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac740386f0 0x7fac7403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fac7c06be50 con 0x7fac880679a0 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 10 monclient(hunting): handle_auth_done global_id 44709 payload 824 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac87fff640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac880724a0 con 0x7fac881364f0 2024-09-15T01:55:04.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8cc9d640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac740386f0 0x7fac7403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.288+0000 7fac8cc9d640 10 monclient: get_auth_request con 0x7fac740386f0 auth_method 0 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac87fff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 msgr2=0x7fac880298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac87fff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 0x7fac880298c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7800e9f0 con 0x7fac881364f0 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.290+0000 7fac6ffff640 10 monclient: dump: 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fac7800eb50 con 0x7fac881364f0 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 msgr2=0x7fac880298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 0x7fac880298c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac780090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 msgr2=0x7fac880298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac6ffff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac881364f0 0x7fac880298c0 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac780090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.291+0000 7fac8cc9d640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac740386f0 0x7fac7403aba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac800012a0 tx=0x7fac800083a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac8e700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac8e700640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 0x7fac88029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac8e700640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88063870 con 0x7fac88130eb0 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac87fff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 0x7fac88029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac87fff640 10 monclient: get_auth_request con 0x7fac88130eb0 auth_method 0 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.296+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.297+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.297+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.297+0000 7fac87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.297+0000 7fac87fff640 10 monclient(hunting): handle_auth_done global_id 44712 payload 824 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.297+0000 7fac87fff640 1 -- 172.21.15.17:0/1347451960 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fac8809a990 con 0x7fac88130eb0 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac87fff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 msgr2=0x7fac88029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac87fff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 0x7fac88029e00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac78015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78025070 con 0x7fac88130eb0 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 10 monclient: dump: 2024-09-15T01:55:04.310 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4556 (secure 0 0 0) 0x7fac7801b660 con 0x7fac88130eb0 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 msgr2=0x7fac88029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 0x7fac88029e00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac78015000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 msgr2=0x7fac88029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.299+0000 7fac6ffff640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac88130eb0 0x7fac88029e00 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac780095e0 tx=0x7fac78015000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 46, 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:04.312 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.313 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:04.315 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.316 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:04.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.318 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.320 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.323 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:04.324 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:04.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac740386f0 msgr2=0x7fac7403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac740386f0 0x7fac7403aba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac800012a0 tx=0x7fac800083a0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 10 monclient: shutdown 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 msgr2=0x7fac8811a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 --2- 172.21.15.17:0/1347451960 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac880679a0 0x7fac8811a5a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009210 tx=0x7fac7c032000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8cc9d640 1 -- 172.21.15.17:0/1347451960 reap_dead start 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8cc9d640 1 -- 172.21.15.17:0/1347451960 reap_dead start 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 shutdown_connections 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 >> 172.21.15.17:0/1347451960 conn(0x7fac880a1ee0 msgr2=0x7fac880a0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 shutdown_connections 2024-09-15T01:55:04.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.300+0000 7fac8e700640 1 -- 172.21.15.17:0/1347451960 wait complete. 2024-09-15T01:55:04.326 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:55:04.326 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:55:04.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.421+0000 7f2b00ea8640 10 monclient: get_monmap_and_config 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.421+0000 7f2b00ea8640 10 monclient: build_initial_monmap 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.422+0000 7f2b00ea8640 10 monclient: monmap: 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:04.422837+0000 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:04.422837+0000 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:04.424 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:04.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.422+0000 7f2b00ea8640 1 Processor -- start 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.422+0000 7f2b00ea8640 1 -- start start 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.422+0000 7f2b00ea8640 10 monclient: init 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:55:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2afc09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f2afc09c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2afc09c580 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2afc09c990 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc09f790 con 0x7f2afc09c580 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc068b60 con 0x7f2afc09c990 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc069d40 con 0x7f2afc09c1b0 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2b00ea8640 10 monclient(hunting): authenticate will time out at 964.419339s 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2afa59c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 msgr2=0x7f2afc09a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2afa59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2afc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2af9d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2afc09c580 0x7f2afc06eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42754/0 (socket says 172.21.15.17:42754) 2024-09-15T01:55:04.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.423+0000 7f2af9d9b640 1 -- 172.21.15.17:0/1118127141 learned_addr learned my addr 172.21.15.17:0/1118127141 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:04.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af9d9b640 1 -- 172.21.15.17:0/1118127141 >> v1:172.21.15.17:6789/0 conn(0x7f2afc09c580 legacy=0x7f2afc06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 >> v1:172.21.15.67:6789/0 conn(0x7f2afc09c990 legacy=0x7f2afc139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102617592 0 0) 0x7f2afc09f790 con 0x7f2afc09c580 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af959a640 10 monclient(hunting): my global_id is 44715 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.425+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae40034e0 con 0x7f2afc09c580 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719797720 0 0) 0x7f2afc068b60 con 0x7f2afc09c990 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): my global_id is 5338 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc09f790 con 0x7f2afc09c990 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767617007 0 0) 0x7f2ae40034e0 con 0x7f2afc09c580 2024-09-15T01:55:04.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 msgr2=0x7f2afc09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2afc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 >> v1:172.21.15.67:6789/0 conn(0x7f2afc09c990 legacy=0x7f2afc139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: _finish_hunting 0 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc069aa0 con 0x7f2afc09c580 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: _finish_auth 0 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f2af0001720 con 0x7f2afc09c580 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2b00ea8640 5 monclient: authenticate success, global_id 44715 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2b00ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: dump: 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:55:04.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 >> v1:172.21.15.17:6789/0 conn(0x7f2afc09c580 legacy=0x7f2afc06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2ae4003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): picked mon.c con 0x7f2afc09c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004110 0x7f2ae4004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): picked mon.b con 0x7f2ae4004110 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 0x7f2ae4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): picked mon.a con 0x7f2ae4004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afa59c640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 msgr2=0x7f2ae4003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc069d40 con 0x7f2ae4004de0 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae400bd70 con 0x7f2ae4004110 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afa59c640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2ae4003b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae400bf30 con 0x7f2afc09c1b0 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2b00ea8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 0x7f2ae4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): get_auth_request con 0x7f2ae4004de0 auth_method 0 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004110 0x7f2ae4004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): get_auth_request con 0x7f2ae4004110 auth_method 0 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2af9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient(hunting): handle_auth_done global_id 44715 payload 293 2024-09-15T01:55:04.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 msgr2=0x7f2ae4003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc09c1b0 0x7f2ae4003b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004110 msgr2=0x7f2ae4004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004110 0x7f2ae4004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 monclient: found mon.a 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae40034e0 con 0x7f2ae4004de0 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 10 monclient: _finish_auth 0 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.426+0000 7f2afad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 msgr2=0x7f2ae4005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2afad9d640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 0x7f2ae4005250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007da0 tx=0x7f2af4001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a74b1287c88083 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af402ade0 con 0x7f2ae4004de0 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: dump: 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.440 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2b00ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af40070f0 con 0x7f2ae4004de0 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2b00ea8640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 1 -- 172.21.15.17:0/1118127141 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40074b0 con 0x7f2ae4004de0 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2b00ea8640 4 set_mon_vals no callback set 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.429+0000 7f2af959a640 10 monclient: dump: 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.441 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 10 monclient: shutdown 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 -- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 msgr2=0x7f2ae4005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 --2- 172.21.15.17:0/1118127141 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ae4004de0 0x7f2ae4005250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007da0 tx=0x7f2af4001b70 comp rx=0 tx=0).stop 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2afad9d640 1 -- 172.21.15.17:0/1118127141 reap_dead start 2024-09-15T01:55:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 -- 172.21.15.17:0/1118127141 shutdown_connections 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 -- 172.21.15.17:0/1118127141 >> 172.21.15.17:0/1118127141 conn(0x7f2afc071300 msgr2=0x7f2afc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 -- 172.21.15.17:0/1118127141 shutdown_connections 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 1 -- 172.21.15.17:0/1118127141 wait complete. 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 10 monclient: build_initial_monmap 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.430+0000 7f2b00ea8640 10 monclient: monmap: 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:04.443 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:04.444 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:04.444 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 Processor -- start 2024-09-15T01:55:04.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 -- start start 2024-09-15T01:55:04.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient: init 2024-09-15T01:55:04.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:04.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:04.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004de0 0x7f2afc141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae4004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc141420 0x7f2afc131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f2afc141420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 0x7f2afc097950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f2afc11e060 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc09cdb0 con 0x7f2afc11e060 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc11dc50 con 0x7f2ae4004de0 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc06eb10 con 0x7f2afc141420 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2af9d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc141420 msgr2=0x7f2afc131e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2b00ea8640 10 monclient(hunting): authenticate will time out at 964.427741s 2024-09-15T01:55:04.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2af9d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc141420 0x7f2afc131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 0x7f2afc097950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 0x7f2afc097950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33182/0 (socket says 172.21.15.17:33182) 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 learned_addr learned my addr 172.21.15.17:0/3929612578 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): get_auth_request con 0x7f2afc11e060 auth_method 0 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afa59c640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004de0 0x7f2afc141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afa59c640 10 monclient(hunting): get_auth_request con 0x7f2ae4004de0 auth_method 0 2024-09-15T01:55:04.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.431+0000 7f2afa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 10 monclient(hunting): handle_auth_done global_id 44718 payload 824 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc141420 msgr2=0x7f2afc131e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2afc141420 0x7f2afc131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004de0 msgr2=0x7f2afc141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae4004de0 0x7f2afc141cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 monclient: found mon.noname-a 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:04.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.432+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc11ef50 con 0x7f2afc11e060 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 msgr2=0x7f2afc097950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2afad9d640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 0x7f2afc097950 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007e00 tx=0x7f2af4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4033470 con 0x7f2afc11e060 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: dump: 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.454 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4038020 con 0x7f2afc11e060 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 5 monclient: authenticate success, global_id 44718 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 10 monclient: _renew_subs 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc099530 con 0x7f2afc11e060 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:04.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40337e0 con 0x7f2afc11e060 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2aea7fc640 10 monclient: dump: 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2af8d99640 4 set_mon_vals no callback set 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 10 monclient: _renew_subs 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.434+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc132510 con 0x7f2afc11e060 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2b00ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2b00ea8640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 0x7f2afc098260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc063bb0 con 0x7f2afc097e90 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2af4030070 con 0x7f2afc11e060 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 0x7f2afc098260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2aea7fc640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2adc0386f0 0x7f2adc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient: get_auth_request con 0x7f2afc097e90 auth_method 0 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f2af406b890 con 0x7f2afc11e060 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2af9d9b640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2adc0386f0 0x7f2adc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2af9d9b640 10 monclient: get_auth_request con 0x7f2adc0386f0 auth_method 0 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 10 monclient(hunting): handle_auth_done global_id 5344 payload 824 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.435+0000 7f2afa59c640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2afc0987a0 con 0x7f2afc097e90 2024-09-15T01:55:04.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2afa59c640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 msgr2=0x7f2afc098260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2afa59c640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 0x7f2afc098260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2af9d9b640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2adc0386f0 0x7f2adc03aba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2af0001900 tx=0x7f2af0007a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00eee0 con 0x7f2afc097e90 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 10 monclient: dump: 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.459 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2aec00f040 con 0x7f2afc097e90 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 msgr2=0x7f2afc098260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 0x7f2afc098260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec009760 comp rx=0 tx=0).stop 2024-09-15T01:55:04.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 msgr2=0x7f2afc098260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.438+0000 7f2aea7fc640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc097e90 0x7f2afc098260 secure :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec009760 comp rx=0 tx=0).stop 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2b00ea8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2b00ea8640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 0x7f2afc1330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2b00ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0724a0 con 0x7f2afc132ca0 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 0x7f2afc1330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient: get_auth_request con 0x7f2afc132ca0 auth_method 0 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:04.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.444+0000 7f2afa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.445+0000 7f2afa59c640 10 monclient(hunting): handle_auth_done global_id 5347 payload 824 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.445+0000 7f2afa59c640 1 -- 172.21.15.17:0/3929612578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2afc067220 con 0x7f2afc132ca0 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2afa59c640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 msgr2=0x7f2afc1330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2afa59c640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 0x7f2afc1330a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec01c620 con 0x7f2afc132ca0 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 10 monclient: dump: 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:04.462 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:04.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4724 (secure 0 0 0) 0x7f2aec01c780 con 0x7f2afc132ca0 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 msgr2=0x7f2afc1330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 0x7f2afc1330a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec018000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 msgr2=0x7f2afc1330a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:04.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.447+0000 7f2aea7fc640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc132ca0 0x7f2afc1330a0 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f90 tx=0x7f2aec018000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 46, 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:04.478 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2adc0386f0 msgr2=0x7f2adc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2adc0386f0 0x7f2adc03aba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2af0001900 tx=0x7f2af0007a30 comp rx=0 tx=0).stop 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 10 monclient: shutdown 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 msgr2=0x7f2afc097950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 --2- 172.21.15.17:0/3929612578 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2afc11e060 0x7f2afc097950 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007e00 tx=0x7f2af4030000 comp rx=0 tx=0).stop 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 reap_dead start 2024-09-15T01:55:04.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2afad9d640 1 -- 172.21.15.17:0/3929612578 reap_dead start 2024-09-15T01:55:04.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 shutdown_connections 2024-09-15T01:55:04.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 >> 172.21.15.17:0/3929612578 conn(0x7f2afc071300 msgr2=0x7f2afc06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:04.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 shutdown_connections 2024-09-15T01:55:04.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:04.448+0000 7f2b00ea8640 1 -- 172.21.15.17:0/3929612578 wait complete. 2024-09-15T01:55:04.480 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:55:04.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:55:04.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.511+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v79) v1 ==== 128+0+0 (secure 0 0 0) 0x7f76e0014340 con 0x7f76e8098140 2024-09-15T01:55:04.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.511+0000 7f76e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:04.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.511+0000 7f76e4ff9640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T01:55:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.614+0000 7f76eed54640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.614+0000 7f76eed54640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:04.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.614+0000 7f76eed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:04.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:04.614+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f76e804bf60 con 0x7f76e8098140 2024-09-15T01:55:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.520+0000 7f76e4ff9640 1 -- 172.21.15.67:0/3889014091 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v80) v1 ==== 128+0+0 (secure 0 0 0) 0x7f76e001d300 con 0x7f76e8098140 2024-09-15T01:55:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.520+0000 7f76e4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T01:55:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.520+0000 7f76e4ff9640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T01:55:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:removed item id -16 name 'host2' from crush map 2024-09-15T01:55:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.522+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 msgr2=0x7f76c803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.522+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 0x7f76c803aba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76e8138eb0 tx=0x7f76d8007f00 comp rx=0 tx=0).stop 2024-09-15T01:55:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.522+0000 7f76eed54640 10 monclient: shutdown 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.522+0000 7f76eed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.522+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 msgr2=0x7f76e8138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f76e000bbb0 tx=0x7f76e00097f0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 shutdown_connections 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c80386f0 0x7f76c803aba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76e8098c20 0x7f76e8130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76e80a0370 0x7f76e8131250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 --2- 172.21.15.67:0/3889014091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e8098140 0x7f76e8138890 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 >> 172.21.15.67:0/3889014091 conn(0x7f76e80a5780 msgr2=0x7f76e80a37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 shutdown_connections 2024-09-15T01:55:05.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.523+0000 7f76eed54640 1 -- 172.21.15.67:0/3889014091 wait complete. 2024-09-15T01:55:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.619+0000 7f8a69aef640 10 monclient: get_monmap_and_config 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.619+0000 7f8a69aef640 10 monclient: build_initial_monmap 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.619+0000 7f8a69aef640 10 monclient: monmap: 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:05.619920+0000 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:05.619920+0000 2024-09-15T01:55:05.621 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:05.622 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.620+0000 7f8a69aef640 1 Processor -- start 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.620+0000 7f8a69aef640 1 -- start start 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.620+0000 7f8a69aef640 10 monclient: init 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T01:55:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a640a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-b con 0x7f8a6409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8a64136e40 0x7f8a64139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-a con 0x7f8a64136e40 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640a61a0 con 0x7f8a640a0b50 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a640a1610 con 0x7f8a64136e40 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6409f7d0 con 0x7f8a6409eea0 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a69aef640 10 monclient(hunting): authenticate will time out at 938.089257s 2024-09-15T01:55:05.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a5affd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 msgr2=0x7f8a6409f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a637fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a640a0b50 0x7f8a64069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49864/0 (socket says 172.21.15.67:49864) 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a637fe640 1 -- 172.21.15.67:0/328428735 learned_addr learned my addr 172.21.15.67:0/328428735 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 --2- 172.21.15.67:0/328428735 >> v2:172.21.15.17:3300/0 conn(0x7f8a64136e40 0x7f8a64139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): get_auth_request con 0x7f8a64136e40 auth_method 0 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient(hunting): handle_auth_done global_id 44721 payload 274 2024-09-15T01:55:05.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 -- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 msgr2=0x7f8a6409f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 --2- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 -- 172.21.15.67:0/328428735 >> v1:172.21.15.67:6789/0 conn(0x7f8a640a0b50 legacy=0x7f8a64069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 monclient: found mon.noname-a 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:05.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.621+0000 7f8a63fff640 1 -- 172.21.15.67:0/328428735 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a640a04c0 con 0x7f8a64136e40 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.624+0000 7f8a63fff640 1 -- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 msgr2=0x7f8a64139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.624+0000 7f8a63fff640 1 --2- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a64139220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009e60 tx=0x7f8a5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450f94966f8b4471 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 1 -- 172.21.15.67:0/328428735 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c007840 con 0x7f8a64136e40 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: dump: 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:05.635 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 1 -- 172.21.15.67:0/328428735 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c02d050 con 0x7f8a64136e40 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 5 monclient: authenticate success, global_id 44721 2024-09-15T01:55:05.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 4 set_mon_vals no callback set 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 1 -- 172.21.15.67:0/328428735 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c007bd0 con 0x7f8a64136e40 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a62ffd640 10 monclient: dump: 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.637 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 10 monclient: shutdown 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 -- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 msgr2=0x7f8a64139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 --2- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a64139220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009e60 tx=0x7f8a5c009780 comp rx=0 tx=0).stop 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 -- 172.21.15.67:0/328428735 shutdown_connections 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 --2- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a64139220 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 --2- 172.21.15.67:0/328428735 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 -- 172.21.15.67:0/328428735 >> 172.21.15.67:0/328428735 conn(0x7f8a64071330 msgr2=0x7f8a640a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 -- 172.21.15.67:0/328428735 shutdown_connections 2024-09-15T01:55:05.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.625+0000 7f8a69aef640 1 -- 172.21.15.67:0/328428735 wait complete. 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.626+0000 7f8a69aef640 10 monclient: build_initial_monmap 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.626+0000 7f8a69aef640 10 monclient: monmap: 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.626+0000 7f8a69aef640 1 Processor -- start 2024-09-15T01:55:05.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.626+0000 7f8a69aef640 1 -- start start 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.626+0000 7f8a69aef640 10 monclient: init 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-c con 0x7f8a6409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-b con 0x7f8a640a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a641417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): picked mon.noname-a con 0x7f8a64136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6411dcb0 con 0x7f8a64136e40 2024-09-15T01:55:05.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a640a3480 con 0x7f8a640a0b50 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6409b7e0 con 0x7f8a6409eea0 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a69aef640 10 monclient(hunting): authenticate will time out at 938.095340s 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a637fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 msgr2=0x7f8a6409a510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48474/0 (socket says 172.21.15.67:48474) 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 -- 172.21.15.67:0/1179629255 learned_addr learned my addr 172.21.15.67:0/1179629255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): get_auth_request con 0x7f8a640a0b50 auth_method 0 2024-09-15T01:55:05.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a641417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): get_auth_request con 0x7f8a64136e40 auth_method 0 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient(hunting): handle_auth_done global_id 5350 payload 824 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 -- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 msgr2=0x7f8a6409a510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 -- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 msgr2=0x7f8a641417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a641417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 monclient: found mon.noname-b 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.627+0000 7f8a5affd640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c009a20 con 0x7f8a640a0b50 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5affd640 1 -- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 msgr2=0x7f8a641412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5affd640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00b630 tx=0x7f8a4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:05.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c015070 con 0x7f8a640a0b50 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: dump: 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:05.644 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: _finish_auth 0 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c003d90 con 0x7f8a640a0b50 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 5 monclient: authenticate success, global_id 5350 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 10 monclient: _renew_subs 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6411ed00 con 0x7f8a640a0b50 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c006380 con 0x7f8a640a0b50 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a5bfff640 10 monclient: dump: 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:05.645 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a627fc640 4 set_mon_vals no callback set 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 10 monclient: _renew_subs 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.630+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6411cae0 con 0x7f8a640a0b50 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8a4c010440 con 0x7f8a640a0b50 2024-09-15T01:55:05.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a69aef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a69aef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a69aef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6411dae0 con 0x7f8a640a0b50 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a5bfff640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 0x7f8a4403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f8a4c04b9d0 con 0x7f8a640a0b50 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a637fe640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 0x7f8a4403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.631+0000 7f8a637fe640 10 monclient: get_auth_request con 0x7f8a440386f0 auth_method 0 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.634+0000 7f8a637fe640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 0x7f8a4403aba0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8a540047d0 tx=0x7f8a54007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:05.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.635+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8a4c018a70 con 0x7f8a640a0b50 2024-09-15T01:55:05.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.635+0000 7f8a5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:05.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.635+0000 7f8a5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.803+0000 7f8a69aef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.803+0000 7f8a69aef640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.803+0000 7f8a69aef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:05.803+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f8a6404bf60 con 0x7f8a640a0b50 2024-09-15T01:55:06.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.532+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v81) v1 ==== 251+0+0 (secure 0 0 0) 0x7f8a4c0067b0 con 0x7f8a640a0b50 2024-09-15T01:55:06.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.532+0000 7f8a5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:06.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.532+0000 7f8a5bfff640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-15T01:55:06.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.636+0000 7f8a69aef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:06.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.636+0000 7f8a69aef640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:06.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.636+0000 7f8a69aef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:06.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.636+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f8a6406f000 con 0x7f8a640a0b50 2024-09-15T01:55:06.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.637+0000 7f8a5bfff640 1 -- 172.21.15.67:0/1179629255 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v81) v1 ==== 200+0+0 (secure 0 0 0) 0x7f8a4c0280e0 con 0x7f8a640a0b50 2024-09-15T01:55:06.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.637+0000 7f8a5bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T01:55:06.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.637+0000 7f8a5bfff640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-15T01:55:06.638 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'host-for-test' already exists 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 msgr2=0x7f8a4403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 0x7f8a4403aba0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8a540047d0 tx=0x7f8a54007b80 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 10 monclient: shutdown 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 msgr2=0x7f8a641412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00b630 tx=0x7f8a4c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 shutdown_connections 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8a440386f0 0x7f8a4403aba0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8a64136e40 0x7f8a641417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a640a0b50 0x7f8a641412b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 --2- 172.21.15.67:0/1179629255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8a6409eea0 0x7f8a6409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 >> 172.21.15.67:0/1179629255 conn(0x7f8a64071330 msgr2=0x7f8a6409dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:06.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 shutdown_connections 2024-09-15T01:55:06.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.639+0000 7f8a69aef640 1 -- 172.21.15.67:0/1179629255 wait complete. 2024-09-15T01:55:06.650 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:06.650 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep host-for-test 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.723+0000 7fd6cca99640 10 monclient: get_monmap_and_config 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.723+0000 7fd6cca99640 10 monclient: build_initial_monmap 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.723+0000 7fd6cca99640 10 monclient: monmap: 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:06.724113+0000 2024-09-15T01:55:06.725 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:06.724113+0000 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:06.726 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.724+0000 7fd6cca99640 1 Processor -- start 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.724+0000 7fd6cca99640 1 -- start start 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.724+0000 7fd6cca99640 10 monclient: init 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 0x7fd6c81322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-b con 0x7fd6c80981b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd6c8098580 0x7fd6c8098950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-a con 0x7fd6c8098580 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6c8068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:06.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c80a07b0 con 0x7fd6c8068dc0 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c80a16e0 con 0x7fd6c8098580 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c80a1400 con 0x7fd6c80981b0 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6cca99640 10 monclient(hunting): authenticate will time out at 939.193390s 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 msgr2=0x7fd6c81322c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 0x7fd6c81322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c6d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6c8068dc0 0x7fd6c8069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49872/0 (socket says 172.21.15.67:49872) 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c6d9d640 1 -- 172.21.15.67:0/3621451340 learned_addr learned my addr 172.21.15.67:0/3621451340 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/3621451340 >> v2:172.21.15.17:3300/0 conn(0x7fd6c8098580 0x7fd6c8098950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): get_auth_request con 0x7fd6c8098580 auth_method 0 2024-09-15T01:55:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_done global_id 44727 payload 274 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 msgr2=0x7fd6c81322c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 0x7fd6c81322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/3621451340 >> v1:172.21.15.67:6789/0 conn(0x7fd6c8068dc0 legacy=0x7fd6c8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 monclient: found mon.noname-a 2024-09-15T01:55:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:06.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:06.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.725+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/3621451340 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c80a19c0 con 0x7fd6c8098580 2024-09-15T01:55:06.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.728+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8098580 msgr2=0x7fd6c8098950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.728+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8098580 0x7fd6c8098950 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009850 tx=0x7fd6bc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb55f037c0d96cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 1 -- 172.21.15.67:0/3621451340 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc032440 con 0x7fd6c8098580 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: dump: 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: _finish_auth 0 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 1 -- 172.21.15.67:0/3621451340 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc0325a0 con 0x7fd6c8098580 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 1 -- 172.21.15.67:0/3621451340 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc032940 con 0x7fd6c8098580 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 5 monclient: authenticate success, global_id 44727 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 4 set_mon_vals no callback set 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6c559a640 10 monclient: dump: 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 10 monclient: shutdown 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 -- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8098580 msgr2=0x7fd6c8098950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 --2- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8098580 0x7fd6c8098950 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009850 tx=0x7fd6bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 -- 172.21.15.67:0/3621451340 shutdown_connections 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 --2- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8098580 0x7fd6c8098950 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 --2- 172.21.15.67:0/3621451340 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c80981b0 0x7fd6c81322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 -- 172.21.15.67:0/3621451340 >> 172.21.15.67:0/3621451340 conn(0x7fd6c8071150 msgr2=0x7fd6c80a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 -- 172.21.15.67:0/3621451340 shutdown_connections 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.729+0000 7fd6cca99640 1 -- 172.21.15.67:0/3621451340 wait complete. 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.730+0000 7fd6cca99640 10 monclient: build_initial_monmap 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.730+0000 7fd6cca99640 10 monclient: monmap: 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:06.740 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.730+0000 7fd6cca99640 1 Processor -- start 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.730+0000 7fd6cca99640 1 -- start start 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.730+0000 7fd6cca99640 10 monclient: init 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8068dc0 0x7fd6c8131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-a con 0x7fd6c8068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-b con 0x7fd6c80981b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 0x7fd6c811e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): picked mon.noname-c con 0x7fd6c8098580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8098ed0 con 0x7fd6c8068dc0 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c809c200 con 0x7fd6c80981b0 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8122040 con 0x7fd6c8098580 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6cca99640 10 monclient(hunting): authenticate will time out at 939.199292s 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:06.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48482/0 (socket says 172.21.15.67:48482) 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c6d9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 msgr2=0x7fd6c811e300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/2283041797 learned_addr learned my addr 172.21.15.67:0/2283041797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): get_auth_request con 0x7fd6c80981b0 auth_method 0 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c6d9d640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 0x7fd6c811e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c659c640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8068dc0 0x7fd6c8131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient(hunting): handle_auth_done global_id 5353 payload 824 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 msgr2=0x7fd6c811e300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:06.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 0x7fd6c811e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8068dc0 msgr2=0x7fd6c8131410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8068dc0 0x7fd6c8131410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 monclient: found mon.noname-b 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.731+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009390 con 0x7fd6c80981b0 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6c5d9b640 1 -- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 msgr2=0x7fd6c811ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009360 tx=0x7fd6bc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc031c70 con 0x7fd6c80981b0 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:06.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 10 monclient: dump: 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6c5d9b640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009360 tx=0x7fd6bc032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fd6bc031dd0 config(0 keys) v1 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 5 monclient: authenticate success, global_id 5353 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 10 monclient: _renew_subs 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc031dd0 con 0x7fd6c80981b0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c8130900 con 0x7fd6c80981b0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6c4d99640 4 set_mon_vals no callback set 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 10 monclient: _renew_subs 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.734+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c81201e0 con 0x7fd6c80981b0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc03d650 con 0x7fd6c80981b0 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 10 monclient: dump: 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd6bc039040 con 0x7fd6c80981b0 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6cca99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6cca99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6cca99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c804bf60 con 0x7fd6c80981b0 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6b67fc640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 0x7fd6a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6c659c640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 0x7fd6a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.735+0000 7fd6c659c640 10 monclient: get_auth_request con 0x7fd6a80386f0 auth_method 0 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.737+0000 7fd6c659c640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 0x7fd6a803aba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8006fd0 tx=0x7fd6b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:06.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:06.739+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7fd6bc06c070 con 0x7fd6c80981b0 2024-09-15T01:55:07.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.357+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd6bc03b3f0 con 0x7fd6c80981b0 2024-09-15T01:55:07.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.357+0000 7fd6b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:07.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.357+0000 7fd6b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:07.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.522+0000 7fd6cca99640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:07.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.522+0000 7fd6cca99640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.522+0000 7fd6cca99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.522+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd6c80a5c50 con 0x7fd6c80981b0 2024-09-15T01:55:07.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.525+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fd6bc0425f0 con 0x7fd6c80981b0 2024-09-15T01:55:07.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.525+0000 7fd6b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.525+0000 7fd6b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.622+0000 7fd6cca99640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.622+0000 7fd6cca99640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.622+0000 7fd6cca99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.622+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd6c8096a70 con 0x7fd6c80981b0 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.623+0000 7fd6b67fc640 1 -- 172.21.15.67:0/2283041797 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fd6bc042030 con 0x7fd6c80981b0 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.623+0000 7fd6b67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.623+0000 7fd6b67fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:07.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 msgr2=0x7fd6a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 0x7fd6a803aba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8006fd0 tx=0x7fd6b8008040 comp rx=0 tx=0).stop 2024-09-15T01:55:07.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 10 monclient: shutdown 2024-09-15T01:55:07.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 msgr2=0x7fd6c811ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.624+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009360 tx=0x7fd6bc032000 comp rx=0 tx=0).stop 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 shutdown_connections 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd6a80386f0 0x7fd6a803aba0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd6c8098580 0x7fd6c811e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6c80981b0 0x7fd6c811ddc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 --2- 172.21.15.67:0/2283041797 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd6c8068dc0 0x7fd6c8131410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 >> 172.21.15.67:0/2283041797 conn(0x7fd6c8071150 msgr2=0x7fd6c809b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 shutdown_connections 2024-09-15T01:55:07.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.625+0000 7fd6cca99640 1 -- 172.21.15.67:0/2283041797 wait complete. 2024-09-15T01:55:07.634 INFO:tasks.workunit.client.0.smithi067.stdout:-11 0 host host-for-test 2024-09-15T01:55:07.635 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:07.635 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep rack-for-test 2024-09-15T01:55:07.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.707+0000 7fe06959d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.707+0000 7fe06959d640 10 monclient: build_initial_monmap 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.707+0000 7fe06959d640 10 monclient: monmap: 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:07.707659+0000 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:07.707659+0000 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:07.709 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:07.710 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:07.710 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 Processor -- start 2024-09-15T01:55:07.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 -- start start 2024-09-15T01:55:07.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient: init 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe064099b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-b con 0x7fe064096a70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe064096e40 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe064133db0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064098fb0 con 0x7fe064133db0 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064097cc0 con 0x7fe064096e40 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe064097920 con 0x7fe064096a70 2024-09-15T01:55:07.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:07.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe06959d640 10 monclient(hunting): authenticate will time out at 940.176627s 2024-09-15T01:55:07.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe0627fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe064096e40 0x7fe064097210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49882/0 (socket says 172.21.15.67:49882) 2024-09-15T01:55:07.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe0627fc640 1 -- 172.21.15.67:0/3555711296 learned_addr learned my addr 172.21.15.67:0/3555711296 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:07.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe062ffd640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 msgr2=0x7fe064099b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:07.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.708+0000 7fe062ffd640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe064099b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:07.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.710+0000 7fe0627fc640 1 -- 172.21.15.67:0/3555711296 >> v1:172.21.15.67:6789/0 conn(0x7fe064096e40 legacy=0x7fe064097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:07.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 >> v1:172.21.15.17:6789/0 conn(0x7fe064133db0 legacy=0x7fe064136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045409961 0 0) 0x7fe064097cc0 con 0x7fe064096e40 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): my global_id is 5356 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c0034e0 con 0x7fe064096e40 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450766338 0 0) 0x7fe064098fb0 con 0x7fe064133db0 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): my global_id is 44730 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064097cc0 con 0x7fe064133db0 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975962660 0 0) 0x7fe04c0034e0 con 0x7fe064096e40 2024-09-15T01:55:07.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 msgr2=0x7fe064099b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe064099b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 >> v1:172.21.15.17:6789/0 conn(0x7fe064133db0 legacy=0x7fe064136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: _finish_hunting 0 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064098cd0 con 0x7fe064096e40 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe0580017e0 con 0x7fe064096e40 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe06959d640 5 monclient: authenticate success, global_id 5356 2024-09-15T01:55:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe06959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: dump: 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:07.720 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 >> v1:172.21.15.67:6789/0 conn(0x7fe064096e40 legacy=0x7fe064097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe04c003b60 0x7fe04c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): picked mon.a con 0x7fe04c003b60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe04c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): picked mon.c con 0x7fe064096a70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 0x7fe04c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): picked mon.b con 0x7fe04c004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064097920 con 0x7fe04c003b60 2024-09-15T01:55:07.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c00bcd0 con 0x7fe04c004de0 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c00be90 con 0x7fe064096a70 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe061ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe06959d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0627fc640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 msgr2=0x7fe04c004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0627fc640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe04c004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 0x7fe04c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): get_auth_request con 0x7fe04c004de0 auth_method 0 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:07.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe062ffd640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe04c003b60 0x7fe04c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe062ffd640 10 monclient(hunting): get_auth_request con 0x7fe04c003b60 auth_method 0 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe062ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.711+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 10 monclient(hunting): handle_auth_done global_id 5356 payload 293 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 msgr2=0x7fe04c004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe064096a70 0x7fe04c004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe04c003b60 msgr2=0x7fe04c003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe04c003b60 0x7fe04c003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 monclient: found mon.b 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0034e0 con 0x7fe04c004de0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.712+0000 7fe0637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.714+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 msgr2=0x7fe04c005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.714+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 0x7fe04c005250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe05c001860 tx=0x7fe05c001a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1016463e8c629bb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c005b00 con 0x7fe04c004de0 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:07.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: dump: 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:07.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c02d070 con 0x7fe04c004de0 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 4 set_mon_vals no callback set 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 1 -- 172.21.15.67:0/3555711296 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c005ce0 con 0x7fe04c004de0 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe061ffb640 10 monclient: dump: 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:07.726 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 10 monclient: shutdown 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 -- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 msgr2=0x7fe04c005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 --2- 172.21.15.67:0/3555711296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe04c004de0 0x7fe04c005250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe05c001860 tx=0x7fe05c001a70 comp rx=0 tx=0).stop 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe0637fe640 1 -- 172.21.15.67:0/3555711296 reap_dead start 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 -- 172.21.15.67:0/3555711296 shutdown_connections 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 -- 172.21.15.67:0/3555711296 >> 172.21.15.67:0/3555711296 conn(0x7fe0640a1020 msgr2=0x7fe0640a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 -- 172.21.15.67:0/3555711296 shutdown_connections 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.715+0000 7fe06959d640 1 -- 172.21.15.67:0/3555711296 wait complete. 2024-09-15T01:55:07.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.716+0000 7fe06959d640 10 monclient: build_initial_monmap 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.716+0000 7fe06959d640 10 monclient: monmap: 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.716+0000 7fe06959d640 1 Processor -- start 2024-09-15T01:55:07.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.716+0000 7fe06959d640 1 -- start start 2024-09-15T01:55:07.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.716+0000 7fe06959d640 10 monclient: init 2024-09-15T01:55:07.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:07.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:07.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 0x7fe06406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-c con 0x7fe04c004de0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06409ba10 0x7fe06409bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-a con 0x7fe06409ba10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): picked mon.noname-b con 0x7fe06409c320 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064098520 con 0x7fe06409ba10 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064133330 con 0x7fe06409c320 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06411bd00 con 0x7fe04c004de0 2024-09-15T01:55:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe06959d640 10 monclient(hunting): authenticate will time out at 940.185181s 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe062ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 msgr2=0x7fe06406e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe062ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 0x7fe06406e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48500/0 (socket says 172.21.15.67:48500) 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 -- 172.21.15.67:0/3058398702 learned_addr learned my addr 172.21.15.67:0/3058398702 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): get_auth_request con 0x7fe06409c320 auth_method 0 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:07.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0627fc640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06409ba10 0x7fe06409bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0627fc640 10 monclient(hunting): get_auth_request con 0x7fe06409ba10 auth_method 0 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient(hunting): handle_auth_done global_id 5359 payload 824 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 -- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 msgr2=0x7fe06406e190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 0x7fe06406e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 -- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06409ba10 msgr2=0x7fe06409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06409ba10 0x7fe06409bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:07.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.717+0000 7fe0637fe640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06411b460 con 0x7fe06409c320 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe0637fe640 1 -- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 msgr2=0x7fe064145260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe0637fe640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe05c02acf0 tx=0x7fe05c033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0077f0 con 0x7fe06409c320 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: dump: 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.736 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: _finish_auth 0 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c007970 con 0x7fe06409c320 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe06959d640 5 monclient: authenticate success, global_id 5359 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c032680 con 0x7fe06409c320 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe06959d640 10 monclient: _renew_subs 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe06959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0641199a0 con 0x7fe06409c320 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe04affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe04affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe04affd640 10 monclient: dump: 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:07.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.719+0000 7fe0617fa640 4 set_mon_vals no callback set 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe06959d640 10 monclient: _renew_subs 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe06959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0641457a0 con 0x7fe06409c320 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe05c02f030 con 0x7fe06409c320 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe06959d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.720+0000 7fe06959d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe06959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06404bf60 con 0x7fe06409c320 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe04affd640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 0x7fe05003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7fe05c06bf20 con 0x7fe06409c320 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe062ffd640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 0x7fe05003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.721+0000 7fe062ffd640 10 monclient: get_auth_request con 0x7fe050038720 auth_method 0 2024-09-15T01:55:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.723+0000 7fe062ffd640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 0x7fe05003abd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0540038f0 tx=0x7fe054009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:07.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.725+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe05c0455a0 con 0x7fe06409c320 2024-09-15T01:55:07.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.725+0000 7fe04affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:07.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.725+0000 7fe04affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.892+0000 7fe06959d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.892+0000 7fe06959d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.892+0000 7fe06959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.892+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe06411b820 con 0x7fe06409c320 2024-09-15T01:55:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.893+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fe05c044b00 con 0x7fe06409c320 2024-09-15T01:55:07.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.893+0000 7fe04affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.893+0000 7fe04affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe06959d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe06959d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe06959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe06411b820 con 0x7fe06409c320 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe04affd640 1 -- 172.21.15.67:0/3058398702 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fe05c044ce0 con 0x7fe06409c320 2024-09-15T01:55:07.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe04affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:07.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.994+0000 7fe04affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 msgr2=0x7fe05003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 0x7fe05003abd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0540038f0 tx=0x7fe054009060 comp rx=0 tx=0).stop 2024-09-15T01:55:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 10 monclient: shutdown 2024-09-15T01:55:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 msgr2=0x7fe064145260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.996+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe05c02acf0 tx=0x7fe05c033000 comp rx=0 tx=0).stop 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 shutdown_connections 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe050038720 0x7fe05003abd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06409ba10 0x7fe06409bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06409c320 0x7fe064145260 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 --2- 172.21.15.67:0/3058398702 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe04c004de0 0x7fe06406e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 >> 172.21.15.67:0/3058398702 conn(0x7fe0640a1020 msgr2=0x7fe06409fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 shutdown_connections 2024-09-15T01:55:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:07.997+0000 7fe06959d640 1 -- 172.21.15.67:0/3058398702 wait complete. 2024-09-15T01:55:08.007 INFO:tasks.workunit.client.0.smithi067.stdout:-12 0 rack rack-for-test 2024-09-15T01:55:08.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:08.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep root-for-test 2024-09-15T01:55:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.079+0000 7ff80295d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.079+0000 7ff80295d640 10 monclient: build_initial_monmap 2024-09-15T01:55:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.079+0000 7ff80295d640 10 monclient: monmap: 2024-09-15T01:55:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:08.080106+0000 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:08.080106+0000 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:08.082 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.080+0000 7ff80295d640 1 Processor -- start 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.080+0000 7ff80295d640 1 -- start start 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.080+0000 7ff80295d640 10 monclient: init 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff7fc0a3770 0x7ff7fc06b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc0a3770 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 0x7ff7fc06e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc06e130 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7fc09da30 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:08.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc069bf0 con 0x7ff7fc09da30 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7fc09f750 con 0x7ff7fc0a3770 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc068b20 con 0x7ff7fc06e130 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff80295d640 10 monclient(hunting): authenticate will time out at 940.549368s 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fb7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 msgr2=0x7ff7fc06e500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 0x7ff7fc06e500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff800efa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7fc09da30 0x7ff7fc09de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49892/0 (socket says 172.21.15.67:49892) 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff800efa640 1 -- 172.21.15.67:0/2951055190 learned_addr learned my addr 172.21.15.67:0/2951055190 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/2951055190 >> v2:172.21.15.17:3300/0 conn(0x7ff7fc0a3770 0x7ff7fc06b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request con 0x7ff7fc0a3770 auth_method 0 2024-09-15T01:55:08.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_done global_id 44736 payload 274 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 -- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 msgr2=0x7ff7fc06e500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 0x7ff7fc06e500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 -- 172.21.15.67:0/2951055190 >> v1:172.21.15.67:6789/0 conn(0x7ff7fc09da30 legacy=0x7ff7fc09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 monclient: found mon.noname-a 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:08.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.081+0000 7ff7fbfff640 1 -- 172.21.15.67:0/2951055190 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0a3b40 con 0x7ff7fc0a3770 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7fbfff640 1 -- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 msgr2=0x7ff7fc06b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc06b9b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009850 tx=0x7ff7ec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1496862c7a2be522 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 1 -- 172.21.15.67:0/2951055190 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec032440 con 0x7ff7fc0a3770 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: dump: 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 1 -- 172.21.15.67:0/2951055190 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec0325a0 con 0x7ff7fc0a3770 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 5 monclient: authenticate success, global_id 44736 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 4 set_mon_vals no callback set 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 1 -- 172.21.15.67:0/2951055190 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec032940 con 0x7ff7fc0a3770 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff7faffd640 10 monclient: dump: 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 10 monclient: shutdown 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 -- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 msgr2=0x7ff7fc06b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 --2- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc06b9b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009850 tx=0x7ff7ec0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 -- 172.21.15.67:0/2951055190 shutdown_connections 2024-09-15T01:55:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 --2- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc06e130 0x7ff7fc06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 --2- 172.21.15.67:0/2951055190 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc06b9b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 -- 172.21.15.67:0/2951055190 >> 172.21.15.67:0/2951055190 conn(0x7ff7fc070fc0 msgr2=0x7ff7fc0a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.085+0000 7ff80295d640 1 -- 172.21.15.67:0/2951055190 shutdown_connections 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.086+0000 7ff80295d640 1 -- 172.21.15.67:0/2951055190 wait complete. 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.086+0000 7ff80295d640 10 monclient: build_initial_monmap 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.086+0000 7ff80295d640 10 monclient: monmap: 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.097 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.086+0000 7ff80295d640 1 Processor -- start 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 -- start start 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient: init 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc06e130 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 0x7ff7fc133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc09da30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc11dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc0a3770 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc06ea90 con 0x7ff7fc0a3770 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc134c20 con 0x7ff7fc06e130 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc142010 con 0x7ff7fc09da30 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff80295d640 10 monclient(hunting): authenticate will time out at 940.555454s 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48516/0 (socket says 172.21.15.67:48516) 2024-09-15T01:55:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fb7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 msgr2=0x7ff7fc133de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 -- 172.21.15.67:0/689761373 learned_addr learned my addr 172.21.15.67:0/689761373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request con 0x7ff7fc06e130 auth_method 0 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fb7fe640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 0x7ff7fc133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc11dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): get_auth_request con 0x7ff7fc0a3770 auth_method 0 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff800efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_done global_id 5362 payload 824 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 -- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 msgr2=0x7ff7fc133de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 0x7ff7fc133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 -- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 msgr2=0x7ff7fc11dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc11dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 monclient: found mon.noname-b 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.087+0000 7ff7fbfff640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec009390 con 0x7ff7fc06e130 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7fbfff640 1 -- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 msgr2=0x7ff7fc141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7fbfff640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009360 tx=0x7ff7ec032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec031990 con 0x7ff7fc06e130 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: dump: 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.102 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec031af0 con 0x7ff7fc06e130 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 5 monclient: authenticate success, global_id 5362 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 10 monclient: _renew_subs 2024-09-15T01:55:08.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc120f60 con 0x7ff7fc06e130 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec031e70 con 0x7ff7fc06e130 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7e3fff640 10 monclient: dump: 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.104 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 10 monclient: _renew_subs 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc1416b0 con 0x7ff7fc06e130 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.090+0000 7ff7f9ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff7ec039040 con 0x7ff7fc06e130 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff80295d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff80295d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff80295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc04bf60 con 0x7ff7fc06e130 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff7e3fff640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 0x7ff7e803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7ff7ec06beb0 con 0x7ff7fc06e130 2024-09-15T01:55:08.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff7fb7fe640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 0x7ff7e803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.091+0000 7ff7fb7fe640 10 monclient: get_auth_request con 0x7ff7e80386f0 auth_method 0 2024-09-15T01:55:08.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.093+0000 7ff7fb7fe640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 0x7ff7e803aba0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc06dd20 tx=0x7ff7f0006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.093+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:55:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.096+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff7ec039940 con 0x7ff7fc06e130 2024-09-15T01:55:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.096+0000 7ff7e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.096+0000 7ff7e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff80295d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff80295d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:08.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff80295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff7fc067140 con 0x7ff7fc06e130 2024-09-15T01:55:08.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff7ec03f070 con 0x7ff7fc06e130 2024-09-15T01:55:08.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff7e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:08.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.261+0000 7ff7e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:08.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff80295d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:08.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff80295d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:08.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff80295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff7fc1210f0 con 0x7ff7fc06e130 2024-09-15T01:55:08.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff7e3fff640 1 -- 172.21.15.67:0/689761373 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff7ec037020 con 0x7ff7fc06e130 2024-09-15T01:55:08.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff7e3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:08.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.360+0000 7ff7e3fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:08.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 msgr2=0x7ff7e803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 0x7ff7e803aba0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc06dd20 tx=0x7ff7f0006d80 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 10 monclient: shutdown 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 msgr2=0x7ff7fc141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009360 tx=0x7ff7ec032000 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 shutdown_connections 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e80386f0 0x7ff7e803aba0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc0a3770 0x7ff7fc11dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc09da30 0x7ff7fc133de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 --2- 172.21.15.67:0/689761373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc06e130 0x7ff7fc141170 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 >> 172.21.15.67:0/689761373 conn(0x7ff7fc070fc0 msgr2=0x7ff7fc134e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:08.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 shutdown_connections 2024-09-15T01:55:08.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.362+0000 7ff80295d640 1 -- 172.21.15.67:0/689761373 wait complete. 2024-09-15T01:55:08.372 INFO:tasks.workunit.client.0.smithi067.stdout:-14 0 root root-for-test 2024-09-15T01:55:08.372 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm host-for-test 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.446+0000 7f085e0ef640 10 monclient: get_monmap_and_config 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.446+0000 7f085e0ef640 10 monclient: build_initial_monmap 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.446+0000 7f085e0ef640 10 monclient: monmap: 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:08.446874+0000 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:08.446874+0000 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:08.449 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 1 Processor -- start 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 1 -- start start 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 10 monclient: init 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:55:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.447+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08580a1a00 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f085809b1f0 0x7f0858133b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-c con 0x7f085809b1f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f085809d5a0 0x7f0858136070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-a con 0x7f085809d5a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085809cc40 con 0x7f08580a1a00 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085809cf80 con 0x7f085809d5a0 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08580a1dd0 con 0x7f085809b1f0 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f085e0ef640 10 monclient(hunting): authenticate will time out at 940.916130s 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f08577fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08580a1a00 0x7f085809aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49900/0 (socket says 172.21.15.67:49900) 2024-09-15T01:55:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f08577fe640 1 -- 172.21.15.67:0/1250534643 learned_addr learned my addr 172.21.15.67:0/1250534643 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 --2- 172.21.15.67:0/1250534643 >> v2:172.21.15.67:3300/0 conn(0x7f085809b1f0 0x7f0858133b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): get_auth_request con 0x7f085809b1f0 auth_method 0 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 1 --2- 172.21.15.67:0/1250534643 >> v2:172.21.15.17:3300/0 conn(0x7f085809d5a0 0x7f0858136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): get_auth_request con 0x7f085809d5a0 auth_method 0 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient(hunting): handle_auth_done global_id 5365 payload 274 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 -- 172.21.15.67:0/1250534643 >> v2:172.21.15.17:3300/0 conn(0x7f085809d5a0 msgr2=0x7f0858136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 --2- 172.21.15.67:0/1250534643 >> v2:172.21.15.17:3300/0 conn(0x7f085809d5a0 0x7f0858136070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 -- 172.21.15.67:0/1250534643 >> v1:172.21.15.67:6789/0 conn(0x7f08580a1a00 legacy=0x7f085809aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 monclient: found mon.noname-c 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.448+0000 7f084effd640 1 -- 172.21.15.67:0/1250534643 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085812efa0 con 0x7f085809b1f0 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f084effd640 1 -- 172.21.15.67:0/1250534643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085809b1f0 msgr2=0x7f0858133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f084effd640 1 --2- 172.21.15.67:0/1250534643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085809b1f0 0x7f0858133b20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f08580a2080 tx=0x7f08400090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c19ef9dc51e30af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 1 -- 172.21.15.67:0/1250534643 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084002ae00 con 0x7f085809b1f0 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:08.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: dump: 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:08.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 1 -- 172.21.15.67:0/1250534643 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840032440 con 0x7f085809b1f0 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f085e0ef640 5 monclient: authenticate success, global_id 5365 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f085e0ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f085e0ef640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 1 -- 172.21.15.67:0/1250534643 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08400327e0 con 0x7f085809b1f0 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f085e0ef640 4 set_mon_vals no callback set 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.451+0000 7f0856ffd640 10 monclient: dump: 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.461 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 10 monclient: shutdown 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 -- 172.21.15.67:0/1250534643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085809b1f0 msgr2=0x7f0858133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 --2- 172.21.15.67:0/1250534643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085809b1f0 0x7f0858133b20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f08580a2080 tx=0x7f08400090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 -- 172.21.15.67:0/1250534643 shutdown_connections 2024-09-15T01:55:08.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 --2- 172.21.15.67:0/1250534643 >> v2:172.21.15.17:3300/0 conn(0x7f085809d5a0 0x7f0858136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 --2- 172.21.15.67:0/1250534643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085809b1f0 0x7f0858133b20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 -- 172.21.15.67:0/1250534643 >> 172.21.15.67:0/1250534643 conn(0x7f0858071340 msgr2=0x7f08580a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 -- 172.21.15.67:0/1250534643 shutdown_connections 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 1 -- 172.21.15.67:0/1250534643 wait complete. 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 10 monclient: build_initial_monmap 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.452+0000 7f085e0ef640 10 monclient: monmap: 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:08.463 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 Processor -- start 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 -- start start 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient: init 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 0x7f085809b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-c con 0x7f085809b1f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 0x7f085812dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-a con 0x7f085809d5a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): picked mon.noname-b con 0x7f08580a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858118b50 con 0x7f085809d5a0 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085811de10 con 0x7f08580a1a00 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085811e3d0 con 0x7f085809b1f0 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f085e0ef640 10 monclient(hunting): authenticate will time out at 940.921756s 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f08577fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 msgr2=0x7f085809b9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f08577fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 0x7f085809b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48528/0 (socket says 172.21.15.67:48528) 2024-09-15T01:55:08.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 1 -- 172.21.15.67:0/658061846 learned_addr learned my addr 172.21.15.67:0/658061846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): get_auth_request con 0x7f08580a1a00 auth_method 0 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.453+0000 7f0857fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f084effd640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 0x7f085812dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 10 monclient(hunting): handle_auth_done global_id 5368 payload 824 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 -- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 msgr2=0x7f085809b9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 0x7f085809b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 -- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 msgr2=0x7f085812dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 0x7f085812dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 monclient: found mon.noname-b 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f0857fff640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0840009390 con 0x7f08580a1a00 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f084effd640 10 monclient: get_auth_request con 0x7f085809d5a0 auth_method 0 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.454+0000 7f084effd640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 0x7f085812dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f0857fff640 1 -- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 msgr2=0x7f0858117610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f0857fff640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0850009ef0 tx=0x7f085000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085000ce10 con 0x7f08580a1a00 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: dump: 2024-09-15T01:55:08.467 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085000f040 con 0x7f08580a1a00 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 5 monclient: authenticate success, global_id 5368 2024-09-15T01:55:08.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 10 monclient: _renew_subs 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085811e6b0 con 0x7f08580a1a00 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500148e0 con 0x7f08580a1a00 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f084ffff640 10 monclient: dump: 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:08.469 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f08567fc640 4 set_mon_vals no callback set 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 10 monclient: _renew_subs 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.456+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08580985b0 con 0x7f08580a1a00 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0850014c90 con 0x7f08580a1a00 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f085e0ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f085e0ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f085e0ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085809c410 con 0x7f08580a1a00 2024-09-15T01:55:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f084ffff640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 0x7f083803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7f085004d0e0 con 0x7f08580a1a00 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f08577fe640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 0x7f083803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.457+0000 7f08577fe640 10 monclient: get_auth_request con 0x7f08380386f0 auth_method 0 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.460+0000 7f08577fe640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 0x7f083803aba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f08480047d0 tx=0x7f0848007af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.461+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0850024900 con 0x7f08580a1a00 2024-09-15T01:55:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.461+0000 7f084ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:08.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.461+0000 7f084ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.631+0000 7f085e0ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.631+0000 7f085e0ef640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.631+0000 7f085e0ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:08.631+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f085804bf60 con 0x7f08580a1a00 2024-09-15T01:55:09.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.529+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v82) v1 ==== 140+0+0 (secure 0 0 0) 0x7f0850018020 con 0x7f08580a1a00 2024-09-15T01:55:09.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.530+0000 7f084ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:09.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.530+0000 7f084ffff640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-15T01:55:09.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.637+0000 7f085e0ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:09.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.637+0000 7f085e0ef640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:09.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.637+0000 7f085e0ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.637+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f085812ecc0 con 0x7f08580a1a00 2024-09-15T01:55:09.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.638+0000 7f084ffff640 1 -- 172.21.15.67:0/658061846 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v82) v1 ==== 140+0+0 (secure 0 0 0) 0x7f085003e030 con 0x7f08580a1a00 2024-09-15T01:55:09.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.638+0000 7f084ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T01:55:09.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.638+0000 7f084ffff640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-15T01:55:09.640 INFO:tasks.workunit.client.0.smithi067.stderr:device 'host-for-test' does not appear in the crush map 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 msgr2=0x7f083803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 0x7f083803aba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f08480047d0 tx=0x7f0848007af0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 10 monclient: shutdown 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 msgr2=0x7f0858117610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0850009ef0 tx=0x7f085000b040 comp rx=0 tx=0).stop 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 shutdown_connections 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f08380386f0 0x7f083803aba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.643 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f085809d5a0 0x7f085812dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08580a1a00 0x7f0858117610 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 --2- 172.21.15.67:0/658061846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f085809b1f0 0x7f085809b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 >> 172.21.15.67:0/658061846 conn(0x7f0858071340 msgr2=0x7f0858131b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:09.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 shutdown_connections 2024-09-15T01:55:09.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.640+0000 7f085e0ef640 1 -- 172.21.15.67:0/658061846 wait complete. 2024-09-15T01:55:09.652 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm rack-for-test 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.726+0000 7fd49471d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.726+0000 7fd49471d640 10 monclient: build_initial_monmap 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.726+0000 7fd49471d640 10 monclient: monmap: 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:09.727913+0000 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:09.727913+0000 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:09.729 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:09.730 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 Processor -- start 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 -- start start 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient: init 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd48c0a1a00 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd48c09b1f0 0x7fd48c133b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:09.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-c con 0x7fd48c09b1f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd48c09d5a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c09cc40 con 0x7fd48c09d5a0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0a1dd0 con 0x7fd48c0a1a00 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48c09cf80 con 0x7fd48c09b1f0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.727+0000 7fd49471d640 10 monclient(hunting): authenticate will time out at 942.196995s 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd48c09b1f0 0x7fd48c133b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd4924b9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd48c0a1a00 0x7fd48c09aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49914/0 (socket says 172.21.15.67:49914) 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd4924b9640 1 -- 172.21.15.67:0/1118731692 learned_addr learned my addr 172.21.15.67:0/1118731692 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): get_auth_request con 0x7fd48c09b1f0 auth_method 0 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:09.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient(hunting): handle_auth_done global_id 5371 payload 274 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 1 -- 172.21.15.67:0/1118731692 >> v1:172.21.15.67:6789/0 conn(0x7fd48c0a1a00 legacy=0x7fd48c09aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 1 -- 172.21.15.67:0/1118731692 >> v1:172.21.15.17:6789/0 conn(0x7fd48c09d5a0 legacy=0x7fd48c136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient: _finish_hunting 0 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 1 monclient: found mon.noname-c 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.728+0000 7fd491cb8640 1 -- 172.21.15.67:0/1118731692 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c12efa0 con 0x7fd48c09b1f0 2024-09-15T01:55:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd491cb8640 1 -- 172.21.15.67:0/1118731692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 msgr2=0x7fd48c133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd491cb8640 1 --2- 172.21.15.67:0/1118731692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c133b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009850 tx=0x7fd47c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d22de6c43e5eaa62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 1 -- 172.21.15.67:0/1118731692 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c032440 con 0x7fd48c09b1f0 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: dump: 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:09.740 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: _finish_auth 0 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 1 -- 172.21.15.67:0/1118731692 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c0325a0 con 0x7fd48c09b1f0 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 5 monclient: authenticate success, global_id 5371 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 1 -- 172.21.15.67:0/1118731692 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c032940 con 0x7fd48c09b1f0 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 4 set_mon_vals no callback set 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd4914b7640 10 monclient: dump: 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 10 monclient: shutdown 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 -- 172.21.15.67:0/1118731692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 msgr2=0x7fd48c133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 --2- 172.21.15.67:0/1118731692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c133b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009850 tx=0x7fd47c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 -- 172.21.15.67:0/1118731692 shutdown_connections 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 --2- 172.21.15.67:0/1118731692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c133b20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 -- 172.21.15.67:0/1118731692 >> 172.21.15.67:0/1118731692 conn(0x7fd48c071340 msgr2=0x7fd48c0a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.731+0000 7fd49471d640 1 -- 172.21.15.67:0/1118731692 shutdown_connections 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 1 -- 172.21.15.67:0/1118731692 wait complete. 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 10 monclient: build_initial_monmap 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 10 monclient: monmap: 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:09.743 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 1 Processor -- start 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 1 -- start start 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.732+0000 7fd49471d640 10 monclient: init 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-b con 0x7fd48c09b1f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 0x7fd48c11d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-c con 0x7fd48c09d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd48c0a1a00 0x7fd48c11da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): picked mon.noname-a con 0x7fd48c0a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:09.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0985b0 con 0x7fd48c0a1a00 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c12e9b0 con 0x7fd48c09b1f0 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c11e9a0 con 0x7fd48c09d5a0 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd49471d640 10 monclient(hunting): authenticate will time out at 942.202219s 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48554/0 (socket says 172.21.15.67:48554) 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 -- 172.21.15.67:0/679976452 learned_addr learned my addr 172.21.15.67:0/679976452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): get_auth_request con 0x7fd48c09b1f0 auth_method 0 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd491cb8640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 msgr2=0x7fd48c11d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd491cb8640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 0x7fd48c11d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient(hunting): handle_auth_done global_id 5374 payload 824 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 msgr2=0x7fd48c11d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 0x7fd48c11d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd48c0a1a00 msgr2=0x7fd48c11da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd48c0a1a00 0x7fd48c11da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 monclient: found mon.noname-b 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.733+0000 7fd4924b9640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c009390 con 0x7fd48c09b1f0 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4924b9640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 msgr2=0x7fd48c11cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4924b9640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd48800c5e0 tx=0x7fd488004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488005cb0 con 0x7fd48c09b1f0 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: dump: 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd488005e30 con 0x7fd48c09b1f0 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488007340 con 0x7fd48c09b1f0 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd49471d640 5 monclient: authenticate success, global_id 5374 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd49471d640 10 monclient: _renew_subs 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd49471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c120460 con 0x7fd48c09b1f0 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd490cb6640 4 set_mon_vals no callback set 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.735+0000 7fd4827fc640 10 monclient: dump: 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 10 monclient: _renew_subs 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c11f710 con 0x7fd48c09b1f0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd48801b400 con 0x7fd48c09b1f0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.736+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c120650 con 0x7fd48c09b1f0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.737+0000 7fd4827fc640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 0x7fd47003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.737+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6311+0+0 (secure 0 0 0) 0x7fd48804bde0 con 0x7fd48c09b1f0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.737+0000 7fd491cb8640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 0x7fd47003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.737+0000 7fd491cb8640 10 monclient: get_auth_request con 0x7fd4700386f0 auth_method 0 2024-09-15T01:55:09.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.739+0000 7fd491cb8640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 0x7fd47003aba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd48c098030 tx=0x7fd47c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:09.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.741+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd48801a720 con 0x7fd48c09b1f0 2024-09-15T01:55:09.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.741+0000 7fd4827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:09.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.741+0000 7fd4827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:09.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.907+0000 7fd49471d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:09.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.907+0000 7fd49471d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:09.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.907+0000 7fd49471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:09.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:09.907+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7fd48c04bf60 con 0x7fd48c09b1f0 2024-09-15T01:55:10.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.531+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v83) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd48801a360 con 0x7fd48c09b1f0 2024-09-15T01:55:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.531+0000 7fd4827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:10.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.531+0000 7fd4827fc640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-15T01:55:10.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.633+0000 7fd49471d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:10.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.633+0000 7fd49471d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:10.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.633+0000 7fd49471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.633+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7fd48c1208c0 con 0x7fd48c09b1f0 2024-09-15T01:55:10.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.635+0000 7fd4827fc640 1 -- 172.21.15.67:0/679976452 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v83) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd48801a540 con 0x7fd48c09b1f0 2024-09-15T01:55:10.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.635+0000 7fd4827fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T01:55:10.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.635+0000 7fd4827fc640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-15T01:55:10.637 INFO:tasks.workunit.client.0.smithi067.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-15T01:55:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 msgr2=0x7fd47003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 0x7fd47003aba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd48c098030 tx=0x7fd47c032000 comp rx=0 tx=0).stop 2024-09-15T01:55:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 10 monclient: shutdown 2024-09-15T01:55:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 msgr2=0x7fd48c11cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.636+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd48800c5e0 tx=0x7fd488004960 comp rx=0 tx=0).stop 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 shutdown_connections 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4700386f0 0x7fd47003aba0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd48c0a1a00 0x7fd48c11da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd48c09d5a0 0x7fd48c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 --2- 172.21.15.67:0/679976452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd48c09b1f0 0x7fd48c11cff0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 >> 172.21.15.67:0/679976452 conn(0x7fd48c071340 msgr2=0x7fd48c12dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 shutdown_connections 2024-09-15T01:55:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.637+0000 7fd49471d640 1 -- 172.21.15.67:0/679976452 wait complete. 2024-09-15T01:55:10.648 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm root-for-test 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.721+0000 7f57a524f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.721+0000 7f57a524f640 10 monclient: build_initial_monmap 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.721+0000 7f57a524f640 10 monclient: monmap: 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:10.722902+0000 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:10.722902+0000 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:10.724 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:10.725 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:10.725 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:10.725 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:10.725 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 1 Processor -- start 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 1 -- start start 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient: init 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57a00a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:10.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f57a009eea0 0x7f57a009f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-a con 0x7f57a009eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a0139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-b con 0x7f57a0136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.722+0000 7f57a524f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a006a270 con 0x7f57a00a0b50 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a006f290 con 0x7f57a009eea0 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a009f7e0 con 0x7f57a0136e40 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f57a524f640 10 monclient(hunting): authenticate will time out at 943.192044s 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57a00a0b50 0x7f57a0069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60544/0 (socket says 172.21.15.67:60544) 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579ed9d640 1 -- 172.21.15.67:0/1701784671 learned_addr learned my addr 172.21.15.67:0/1701784671 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:10.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579f59e640 1 -- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 msgr2=0x7f57a0139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579f59e640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a0139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 --2- 172.21.15.67:0/1701784671 >> v2:172.21.15.17:3300/0 conn(0x7f57a009eea0 0x7f57a009f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): get_auth_request con 0x7f57a009eea0 auth_method 0 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient(hunting): handle_auth_done global_id 44745 payload 274 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 -- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 msgr2=0x7f57a0139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a0139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 -- 172.21.15.67:0/1701784671 >> v1:172.21.15.67:6789/0 conn(0x7f57a00a0b50 legacy=0x7f57a0069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 monclient: found mon.noname-a 2024-09-15T01:55:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.723+0000 7f579e59c640 1 -- 172.21.15.67:0/1701784671 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00a0f20 con 0x7f57a009eea0 2024-09-15T01:55:10.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.726+0000 7f579e59c640 1 -- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 msgr2=0x7f57a009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.726+0000 7f579e59c640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a009f2a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5794009880 tx=0x7f57940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf520bf4e9edbce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 1 -- 172.21.15.67:0/1701784671 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794032440 con 0x7f57a009eea0 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: dump: 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:10.735 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: _finish_auth 0 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 5 monclient: authenticate success, global_id 44745 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 1 -- 172.21.15.67:0/1701784671 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57940325a0 con 0x7f57a009eea0 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 4 set_mon_vals no callback set 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 1 -- 172.21.15.67:0/1701784671 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794032940 con 0x7f57a009eea0 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:10.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f579dd9b640 10 monclient: dump: 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 10 monclient: shutdown 2024-09-15T01:55:10.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 -- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 msgr2=0x7f57a009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a009f2a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5794009880 tx=0x7f57940090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 -- 172.21.15.67:0/1701784671 shutdown_connections 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a0139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 --2- 172.21.15.67:0/1701784671 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a009f2a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 -- 172.21.15.67:0/1701784671 >> 172.21.15.67:0/1701784671 conn(0x7f57a0071290 msgr2=0x7f57a00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 -- 172.21.15.67:0/1701784671 shutdown_connections 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.727+0000 7f57a524f640 1 -- 172.21.15.67:0/1701784671 wait complete. 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.728+0000 7f57a524f640 10 monclient: build_initial_monmap 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.728+0000 7f57a524f640 10 monclient: monmap: 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:10.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.728+0000 7f57a524f640 1 Processor -- start 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.728+0000 7f57a524f640 1 -- start start 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.728+0000 7f57a524f640 10 monclient: init 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a0134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-a con 0x7f57a009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-b con 0x7f57a00a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a011dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): picked mon.noname-c con 0x7f57a0136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0137720 con 0x7f57a009eea0 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00a12d0 con 0x7f57a00a0b50 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00a3450 con 0x7f57a0136e40 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f57a524f640 10 monclient(hunting): authenticate will time out at 943.198194s 2024-09-15T01:55:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579f59e640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 msgr2=0x7f57a011dbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35102/0 (socket says 172.21.15.67:35102) 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579f59e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a011dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 -- 172.21.15.67:0/2799124220 learned_addr learned my addr 172.21.15.67:0/2799124220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): get_auth_request con 0x7f57a00a0b50 auth_method 0 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a0134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): get_auth_request con 0x7f57a009eea0 auth_method 0 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient(hunting): handle_auth_done global_id 5377 payload 824 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 -- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 msgr2=0x7f57a011dbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a011dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 -- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 msgr2=0x7f57a0134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a0134be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 monclient: found mon.noname-b 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.729+0000 7f579e59c640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794009390 con 0x7f57a00a0b50 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.731+0000 7f579e59c640 1 -- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 msgr2=0x7f57a011d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.731+0000 7f579e59c640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5794009360 tx=0x7f5794032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57940318c0 con 0x7f57a00a0b50 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: dump: 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 5 monclient: authenticate success, global_id 5377 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794031a20 con 0x7f57a00a0b50 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794031de0 con 0x7f57a00a0b50 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f5786ffd640 10 monclient: dump: 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 10 monclient: _renew_subs 2024-09-15T01:55:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f579d59a640 4 set_mon_vals no callback set 2024-09-15T01:55:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0120fd0 con 0x7f57a00a0b50 2024-09-15T01:55:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 10 monclient: _renew_subs 2024-09-15T01:55:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.732+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a0120cf0 con 0x7f57a00a0b50 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f57a524f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f57a524f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f57a524f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a004bf60 con 0x7f57a00a0b50 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5794039050 con 0x7f57a00a0b50 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f5786ffd640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 0x7f578c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6128+0+0 (secure 0 0 0) 0x7f579406bfc0 con 0x7f57a00a0b50 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f579ed9d640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 0x7f578c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.733+0000 7f579ed9d640 10 monclient: get_auth_request con 0x7f578c0386f0 auth_method 0 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.735+0000 7f579ed9d640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 0x7f578c03aba0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5790007fd0 tx=0x7f5790007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.737+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f579403a900 con 0x7f57a00a0b50 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.737+0000 7f5786ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.737+0000 7f5786ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:10.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.904+0000 7f57a524f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:10.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.904+0000 7f57a524f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:10.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.904+0000 7f57a524f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:10.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:10.904+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f57a0142070 con 0x7f57a00a0b50 2024-09-15T01:55:11.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.545+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v84) v1 ==== 140+0+0 (secure 0 0 0) 0x7f5794040020 con 0x7f57a00a0b50 2024-09-15T01:55:11.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.545+0000 7f5786ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:11.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.545+0000 7f5786ffd640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-15T01:55:11.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.649+0000 7f57a524f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:11.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.649+0000 7f57a524f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.649+0000 7f57a524f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.649+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f57a0142070 con 0x7f57a00a0b50 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.650+0000 7f5786ffd640 1 -- 172.21.15.67:0/2799124220 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v84) v1 ==== 140+0+0 (secure 0 0 0) 0x7f5794040030 con 0x7f57a00a0b50 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.650+0000 7f5786ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.650+0000 7f5786ffd640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-15T01:55:11.653 INFO:tasks.workunit.client.0.smithi067.stderr:device 'root-for-test' does not appear in the crush map 2024-09-15T01:55:11.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 msgr2=0x7f578c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:11.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 0x7f578c03aba0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5790007fd0 tx=0x7f5790007f00 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 10 monclient: shutdown 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 msgr2=0x7f57a011d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5794009360 tx=0x7f5794032000 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 shutdown_connections 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f578c0386f0 0x7f578c03aba0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f57a0136e40 0x7f57a011dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a0b50 0x7f57a011d6a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 --2- 172.21.15.67:0/2799124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f57a009eea0 0x7f57a0134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 >> 172.21.15.67:0/2799124220 conn(0x7f57a0071290 msgr2=0x7f57a0135e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:11.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 shutdown_connections 2024-09-15T01:55:11.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.652+0000 7f57a524f640 1 -- 172.21.15.67:0/2799124220 wait complete. 2024-09-15T01:55:11.663 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-15T01:55:11.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.740+0000 7f5943699640 10 monclient: get_monmap_and_config 2024-09-15T01:55:11.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.740+0000 7f5943699640 10 monclient: build_initial_monmap 2024-09-15T01:55:11.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.740+0000 7f5943699640 10 monclient: monmap: 2024-09-15T01:55:11.742 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:11.741617+0000 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:11.741617+0000 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:11.743 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 Processor -- start 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 -- start start 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient: init 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T01:55:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f593c0a1a00 0x7f593c09aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): picked mon.noname-a con 0x7f593c0a1a00 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f593c09d5a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f593c09b020 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c09cc40 con 0x7f593c09b020 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c09b450 con 0x7f593c09d5a0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c09cf80 con 0x7f593c0a1a00 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5943699640 10 monclient(hunting): authenticate will time out at 944.210785s 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5940c34640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f593c09d5a0 0x7f593c096a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60550/0 (socket says 172.21.15.67:60550) 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.741+0000 7f5940c34640 1 -- 172.21.15.67:0/1251684988 learned_addr learned my addr 172.21.15.67:0/1251684988 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 1 --2- 172.21.15.67:0/1251684988 >> v2:172.21.15.17:3300/0 conn(0x7f593c0a1a00 0x7f593c09aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): get_auth_request con 0x7f593c0a1a00 auth_method 0 2024-09-15T01:55:11.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient(hunting): handle_auth_done global_id 44751 payload 274 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 1 -- 172.21.15.67:0/1251684988 >> v1:172.21.15.67:6789/0 conn(0x7f593c09d5a0 legacy=0x7f593c096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 1 -- 172.21.15.67:0/1251684988 >> v1:172.21.15.17:6789/0 conn(0x7f593c09b020 legacy=0x7f593c097180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient: _finish_hunting 0 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 1 monclient: found mon.noname-a 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:11.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.742+0000 7f5941435640 1 -- 172.21.15.67:0/1251684988 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0a1dd0 con 0x7f593c0a1a00 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5941435640 1 -- 172.21.15.67:0/1251684988 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c0a1a00 msgr2=0x7f593c09aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5941435640 1 --2- 172.21.15.67:0/1251684988 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c0a1a00 0x7f593c09aae0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5938009850 tx=0x7f59380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20818c73305216e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 1 -- 172.21.15.67:0/1251684988 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938032440 con 0x7f593c0a1a00 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: dump: 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:11.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 1 -- 172.21.15.67:0/1251684988 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59380325a0 con 0x7f593c0a1a00 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 1 -- 172.21.15.67:0/1251684988 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938032940 con 0x7f593c0a1a00 2024-09-15T01:55:11.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5943699640 5 monclient: authenticate success, global_id 44751 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5943699640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5943699640 4 set_mon_vals no callback set 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.745+0000 7f5933fff640 10 monclient: dump: 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:11.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 10 monclient: shutdown 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 -- 172.21.15.67:0/1251684988 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c0a1a00 msgr2=0x7f593c09aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 --2- 172.21.15.67:0/1251684988 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c0a1a00 0x7f593c09aae0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5938009850 tx=0x7f59380090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 -- 172.21.15.67:0/1251684988 shutdown_connections 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 --2- 172.21.15.67:0/1251684988 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c0a1a00 0x7f593c09aae0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 -- 172.21.15.67:0/1251684988 >> 172.21.15.67:0/1251684988 conn(0x7f593c071340 msgr2=0x7f593c0a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 -- 172.21.15.67:0/1251684988 shutdown_connections 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 1 -- 172.21.15.67:0/1251684988 wait complete. 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 10 monclient: build_initial_monmap 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.746+0000 7f5943699640 10 monclient: monmap: 2024-09-15T01:55:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 Processor -- start 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 -- start start 2024-09-15T01:55:11.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient: init 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 0x7f593c09b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): picked mon.noname-a con 0x7f593c09b020 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 0x7f593c12df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): picked mon.noname-c con 0x7f593c09d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:11.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): picked mon.noname-b con 0x7f593c0a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c12eab0 con 0x7f593c09b020 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c11c0a0 con 0x7f593c0a1a00 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c11c380 con 0x7f593c09d5a0 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5943699640 10 monclient(hunting): authenticate will time out at 944.216856s 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5940c34640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 msgr2=0x7f593c12df50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5940c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 0x7f593c12df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5941c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5941435640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 0x7f593c09b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:11.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5941c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35114/0 (socket says 172.21.15.67:35114) 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5941c36640 1 -- 172.21.15.67:0/714613313 learned_addr learned my addr 172.21.15.67:0/714613313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.747+0000 7f5941c36640 10 monclient(hunting): get_auth_request con 0x7f593c0a1a00 auth_method 0 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): get_auth_request con 0x7f593c09b020 auth_method 0 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:11.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient(hunting): handle_auth_done global_id 5380 payload 824 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 -- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 msgr2=0x7f593c12df50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 0x7f593c12df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 -- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 msgr2=0x7f593c09b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 0x7f593c09b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient: _finish_hunting 0 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 monclient: found mon.noname-b 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941c36640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009390 con 0x7f593c0a1a00 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.748+0000 7f5941435640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 0x7f593c09b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5941c36640 1 -- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 msgr2=0x7f593c13f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:11.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5941c36640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f593400b8e0 tx=0x7f5934009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400cad0 con 0x7f593c0a1a00 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: dump: 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:11.761 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593400c040 con 0x7f593c0a1a00 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934014410 con 0x7f593c0a1a00 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 5 monclient: authenticate success, global_id 5380 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59317fa640 10 monclient: dump: 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:11.762 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 10 monclient: _renew_subs 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f59337fe640 4 set_mon_vals no callback set 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c11c660 con 0x7f593c0a1a00 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 10 monclient: _renew_subs 2024-09-15T01:55:11.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.750+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c11aba0 con 0x7f593c0a1a00 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.751+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5934014880 con 0x7f593c0a1a00 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.751+0000 7f5943699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.751+0000 7f5943699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.751+0000 7f5943699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.751+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c04bf60 con 0x7f593c0a1a00 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.752+0000 7f59317fa640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 0x7f592003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.752+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f593404ba00 con 0x7f593c0a1a00 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.752+0000 7f5941435640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 0x7f592003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.752+0000 7f5941435640 10 monclient: get_auth_request con 0x7f59200386f0 auth_method 0 2024-09-15T01:55:11.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.754+0000 7f5941435640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 0x7f592003aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5938031000 tx=0x7f5938032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:11.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.756+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f59340133c0 con 0x7f593c0a1a00 2024-09-15T01:55:11.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.756+0000 7f59317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:11.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.756+0000 7f59317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:11.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.920+0000 7f5943699640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:11.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.920+0000 7f5943699640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:11.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.920+0000 7f5943699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:11.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:11.920+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f593c09f230 con 0x7f593c0a1a00 2024-09-15T01:55:12.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.200+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v85) v1 ==== 145+0+0 (secure 0 0 0) 0x7f5934012e60 con 0x7f593c0a1a00 2024-09-15T01:55:12.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.200+0000 7f59317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:12.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.200+0000 7f59317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:12.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.300+0000 7f5943699640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.300+0000 7f5943699640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.300+0000 7f5943699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.300+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f593c067180 con 0x7f593c0a1a00 2024-09-15T01:55:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.301+0000 7f59317fa640 1 -- 172.21.15.67:0/714613313 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v85) v1 ==== 173+0+0 (secure 0 0 0) 0x7f59340116e0 con 0x7f593c0a1a00 2024-09-15T01:55:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.301+0000 7f59317fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T01:55:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.301+0000 7f59317fa640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-15T01:55:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo-rule already exists 2024-09-15T01:55:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 msgr2=0x7f592003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 0x7f592003aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5938031000 tx=0x7f5938032000 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 10 monclient: shutdown 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 msgr2=0x7f593c13f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f593400b8e0 tx=0x7f5934009400 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 shutdown_connections 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f59200386f0 0x7f592003aba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f593c0a1a00 0x7f593c13f270 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f593c09d5a0 0x7f593c12df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 --2- 172.21.15.67:0/714613313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f593c09b020 0x7f593c09b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 >> 172.21.15.67:0/714613313 conn(0x7f593c071340 msgr2=0x7f593c131a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:12.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 shutdown_connections 2024-09-15T01:55:12.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.303+0000 7f5943699640 1 -- 172.21.15.67:0/714613313 wait complete. 2024-09-15T01:55:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush rm foo 2024-09-15T01:55:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:55:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm foo 2024-09-15T01:55:12.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.388+0000 7fc3aff9d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.388+0000 7fc3aff9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.388+0000 7fc3aff9d640 10 monclient: monmap: 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:12.389677+0000 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:12.389677+0000 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:12.392 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 Processor -- start 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 -- start start 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient: init 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a809eea0 0x7fc3a809f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a809eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3a8131b10 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a809f7b0 con 0x7fc3a8131b10 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a806f290 con 0x7fc3a809eea0 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a806a270 con 0x7fc3a80a0b50 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3aff9d640 10 monclient(hunting): authenticate will time out at 944.858835s 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3add39640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 msgr2=0x7fc3a8069340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3add39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:12.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3ad538640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a809eea0 0x7fc3a809f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.389+0000 7fc3ae53a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3a8131b10 0x7fc3a8131f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60564/0 (socket says 172.21.15.67:60564) 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ae53a640 1 -- 172.21.15.67:0/390254680 learned_addr learned my addr 172.21.15.67:0/390254680 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): get_auth_request con 0x7fc3a809eea0 auth_method 0 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient(hunting): handle_auth_done global_id 5383 payload 274 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 1 -- 172.21.15.67:0/390254680 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 msgr2=0x7fc3a8069340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 1 --2- 172.21.15.67:0/390254680 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 1 -- 172.21.15.67:0/390254680 >> v1:172.21.15.67:6789/0 conn(0x7fc3a8131b10 legacy=0x7fc3a8131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:12.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient: _finish_hunting 0 2024-09-15T01:55:12.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 1 monclient: found mon.noname-c 2024-09-15T01:55:12.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:12.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.390+0000 7fc3ad538640 1 -- 172.21.15.67:0/390254680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80a0f20 con 0x7fc3a809eea0 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3ad538640 1 -- 172.21.15.67:0/390254680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a809eea0 msgr2=0x7fc3a809f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3ad538640 1 --2- 172.21.15.67:0/390254680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a809eea0 0x7fc3a809f270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc398009850 tx=0x7fc3980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d2d038a57d541e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 1 -- 172.21.15.67:0/390254680 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398032440 con 0x7fc3a809eea0 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: dump: 2024-09-15T01:55:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: _finish_auth 0 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 1 -- 172.21.15.67:0/390254680 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3980325a0 con 0x7fc3a809eea0 2024-09-15T01:55:12.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 5 monclient: authenticate success, global_id 5383 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 1 -- 172.21.15.67:0/390254680 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398032940 con 0x7fc3a809eea0 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 4 set_mon_vals no callback set 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3acd37640 10 monclient: dump: 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 10 monclient: shutdown 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 -- 172.21.15.67:0/390254680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a809eea0 msgr2=0x7fc3a809f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 --2- 172.21.15.67:0/390254680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a809eea0 0x7fc3a809f270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc398009850 tx=0x7fc3980090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 -- 172.21.15.67:0/390254680 shutdown_connections 2024-09-15T01:55:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 --2- 172.21.15.67:0/390254680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a809eea0 0x7fc3a809f270 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 --2- 172.21.15.67:0/390254680 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.393+0000 7fc3aff9d640 1 -- 172.21.15.67:0/390254680 >> 172.21.15.67:0/390254680 conn(0x7fc3a8071290 msgr2=0x7fc3a80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 1 -- 172.21.15.67:0/390254680 shutdown_connections 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 1 -- 172.21.15.67:0/390254680 wait complete. 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 10 monclient: monmap: 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 1 Processor -- start 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 1 -- start start 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.394+0000 7fc3aff9d640 10 monclient: init 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 0x7fc3a809aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:12.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8130e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a8131b10 0x7fc3a811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a8131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a811ca70 con 0x7fc3a809eea0 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81421e0 con 0x7fc3a8131b10 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80a1260 con 0x7fc3a80a0b50 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3aff9d640 10 monclient(hunting): authenticate will time out at 944.864434s 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a8131b10 0x7fc3a811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ad538640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 msgr2=0x7fc3a8130e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ad538640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8130e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a8131b10 0x7fc3a811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35122/0 (socket says 172.21.15.67:35122) 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 -- 172.21.15.67:0/996248318 learned_addr learned my addr 172.21.15.67:0/996248318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): get_auth_request con 0x7fc3a8131b10 auth_method 0 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 0x7fc3a809aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 10 monclient(hunting): get_auth_request con 0x7fc3a809eea0 auth_method 0 2024-09-15T01:55:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 1 -- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 msgr2=0x7fc3a809aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 1 -- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 msgr2=0x7fc3a809aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient(hunting): handle_auth_done global_id 5386 payload 824 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 0x7fc3a809aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 -- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 msgr2=0x7fc3a8130e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a80a0b50 0x7fc3a8130e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3add39640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 0x7fc3a809aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 -- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 msgr2=0x7fc3a809aae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a809eea0 0x7fc3a809aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient: _finish_hunting 0 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 monclient: found mon.noname-b 2024-09-15T01:55:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.395+0000 7fc3ae53a640 1 -- 172.21.15.67:0/996248318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398009390 con 0x7fc3a8131b10 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.397+0000 7fc3ae53a640 1 -- 172.21.15.67:0/996248318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a8131b10 msgr2=0x7fc3a811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.397+0000 7fc3ae53a640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a8131b10 0x7fc3a811b940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000ccf0 tx=0x7fc3a0005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a000aa60 con 0x7fc3a8131b10 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: dump: 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a000abe0 con 0x7fc3a8131b10 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00070f0 con 0x7fc3a8131b10 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 5 monclient: authenticate success, global_id 5386 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:12.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39dffb640 10 monclient: dump: 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 10 monclient: _renew_subs 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc39ffff640 4 set_mon_vals no callback set 2024-09-15T01:55:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 1 -- 172.21.15.67:0/996248318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8134bb0 con 0x7fc3a8131b10 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 10 monclient: _renew_subs 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.398+0000 7fc3aff9d640 1 -- 172.21.15.67:0/996248318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a811dfb0 con 0x7fc3a8131b10 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc3a00074e0 con 0x7fc3a8131b10 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc39dffb640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c0386f0 0x7fc38c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3aff9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3aff9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3aff9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc3a004c1c0 con 0x7fc3a8131b10 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3aff9d640 1 -- 172.21.15.67:0/996248318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a804bf60 con 0x7fc3a8131b10 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3add39640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c0386f0 0x7fc38c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.399+0000 7fc3add39640 10 monclient: get_auth_request con 0x7fc38c0386f0 auth_method 0 2024-09-15T01:55:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.401+0000 7fc3add39640 1 --2- 172.21.15.67:0/996248318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c0386f0 0x7fc38c03aba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40038f0 tx=0x7fc3a4006ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.403+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc3a0007790 con 0x7fc3a8131b10 2024-09-15T01:55:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.403+0000 7fc39dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.403+0000 7fc39dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:12.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.572+0000 7fc3aff9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:12.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.573+0000 7fc3aff9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:12.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.573+0000 7fc3aff9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.573+0000 7fc3aff9d640 1 -- 172.21.15.67:0/996248318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fc3a81327a0 con 0x7fc3a8131b10 2024-09-15T01:55:12.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.574+0000 7fc39dffb640 1 -- 172.21.15.67:0/996248318 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v85) v1 ==== 103+0+0 (secure 0 0 0) 0x7fc3a0029090 con 0x7fc3a8131b10 2024-09-15T01:55:12.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.574+0000 7fc39dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:12.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.574+0000 7fc39dffb640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-15T01:55:12.576 INFO:tasks.workunit.client.0.smithi067.stderr:Error EBUSY: (16) Device or resource busy 2024-09-15T01:55:12.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:55:12.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rule rm foo-rule 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.655+0000 7f23c1e4f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.655+0000 7f23c1e4f640 10 monclient: build_initial_monmap 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.655+0000 7f23c1e4f640 10 monclient: monmap: 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:12.656870+0000 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:12.656870+0000 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:12.658 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:12.659 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:12.659 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:12.659 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.656+0000 7f23c1e4f640 1 Processor -- start 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.656+0000 7f23c1e4f640 1 -- start start 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.656+0000 7f23c1e4f640 10 monclient: init 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 0x7f23bc09eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-b con 0x7f23bc131c80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23bc09f3e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:12.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23bc136fb0 0x7f23bc139390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f23bc136fb0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc0a0290 con 0x7f23bc09f3e0 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0a0030 con 0x7f23bc136fb0 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0a0470 con 0x7f23bc131c80 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23c1e4f640 10 monclient(hunting): authenticate will time out at 945.126096s 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23bc136fb0 0x7f23bc139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bb7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 msgr2=0x7f23bc09eea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 0x7f23bc09eea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23bc136fb0 0x7f23bc139390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35132/0 (socket says 172.21.15.67:35132) 2024-09-15T01:55:12.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 -- 172.21.15.67:0/240755108 learned_addr learned my addr 172.21.15.67:0/240755108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): get_auth_request con 0x7f23bc136fb0 auth_method 0 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23baffd640 1 -- 172.21.15.67:0/240755108 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient(hunting): handle_auth_done global_id 5389 payload 274 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 -- 172.21.15.67:0/240755108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 msgr2=0x7f23bc09eea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 --2- 172.21.15.67:0/240755108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 0x7f23bc09eea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 -- 172.21.15.67:0/240755108 >> v1:172.21.15.17:6789/0 conn(0x7f23bc09f3e0 legacy=0x7f23bc132050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:12.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 monclient: found mon.noname-c 2024-09-15T01:55:12.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:12.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.657+0000 7f23bbfff640 1 -- 172.21.15.67:0/240755108 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc09f7b0 con 0x7f23bc136fb0 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23bbfff640 1 -- 172.21.15.67:0/240755108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc136fb0 msgr2=0x7f23bc139390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23bbfff640 1 --2- 172.21.15.67:0/240755108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc136fb0 0x7f23bc139390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009f20 tx=0x7f23b4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c63740fd9258476 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 1 -- 172.21.15.67:0/240755108 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4007610 con 0x7f23bc136fb0 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: dump: 2024-09-15T01:55:12.667 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23c1e4f640 5 monclient: authenticate success, global_id 5389 2024-09-15T01:55:12.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 1 -- 172.21.15.67:0/240755108 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b402d050 con 0x7f23bc136fb0 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23c1e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23c1e4f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23c1e4f640 4 set_mon_vals no callback set 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 1 -- 172.21.15.67:0/240755108 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b40079a0 con 0x7f23bc136fb0 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.660+0000 7f23baffd640 10 monclient: dump: 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.669 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 10 monclient: shutdown 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 -- 172.21.15.67:0/240755108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc136fb0 msgr2=0x7f23bc139390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/240755108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc136fb0 0x7f23bc139390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009f20 tx=0x7f23b4009840 comp rx=0 tx=0).stop 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 -- 172.21.15.67:0/240755108 shutdown_connections 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/240755108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc136fb0 0x7f23bc139390 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/240755108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc131c80 0x7f23bc09eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 -- 172.21.15.67:0/240755108 >> 172.21.15.67:0/240755108 conn(0x7f23bc0a52e0 msgr2=0x7f23bc0a56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 -- 172.21.15.67:0/240755108 shutdown_connections 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 1 -- 172.21.15.67:0/240755108 wait complete. 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 10 monclient: build_initial_monmap 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.661+0000 7f23c1e4f640 10 monclient: monmap: 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:12.671 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 Processor -- start 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 -- start start 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient: init 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 0x7f23bc130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f23bc09f3e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-b con 0x7f23bc131c80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:12.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc136fb0 0x7f23bc099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f23bc136fb0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc11bad0 con 0x7f23bc136fb0 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.662+0000 7f23c1e4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc11d040 con 0x7f23bc131c80 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23c1e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23c1e4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc09c1c0 con 0x7f23bc09f3e0 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23c1e4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23c1e4f640 10 monclient(hunting): authenticate will time out at 945.132018s 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23bb7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 msgr2=0x7f23bc130c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23bb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 0x7f23bc130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35138/0 (socket says 172.21.15.67:35138) 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 -- 172.21.15.67:0/895088564 learned_addr learned my addr 172.21.15.67:0/895088564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request con 0x7f23bc131c80 auth_method 0 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_done global_id 5392 payload 824 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23bbfff640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc136fb0 0x7f23bc099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 -- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 msgr2=0x7f23bc130c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 0x7f23bc130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 -- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc136fb0 msgr2=0x7f23bc099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc136fb0 0x7f23bc099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 monclient: found mon.noname-b 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.663+0000 7f23b2ffd640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b4009ae0 con 0x7f23bc131c80 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b2ffd640 1 -- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 msgr2=0x7f23bc099960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b2ffd640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c5e0 tx=0x7f23a4007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4012920 con 0x7f23bc131c80 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: dump: 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.677 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 5 monclient: authenticate success, global_id 5392 2024-09-15T01:55:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 10 monclient: _renew_subs 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4012a80 con 0x7f23bc131c80 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc11d600 con 0x7f23bc131c80 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4012e80 con 0x7f23bc131c80 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23b3fff640 10 monclient: dump: 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:12.679 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23ba7fc640 4 set_mon_vals no callback set 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 10 monclient: _renew_subs 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.665+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc11ce10 con 0x7f23bc131c80 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f23a4011590 con 0x7f23bc131c80 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23b3fff640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 0x7f239c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23c1e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23c1e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f23a404c300 con 0x7f23bc131c80 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23c1e4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc11d240 con 0x7f23bc131c80 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.667+0000 7f23bb7fe640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 0x7f239c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.667+0000 7f23bb7fe640 10 monclient: get_auth_request con 0x7f239c0386f0 auth_method 0 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.669+0000 7f23bb7fe640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 0x7f239c03aba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0038f0 tx=0x7f23ac006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.671+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f23a4016030 con 0x7f23bc131c80 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.671+0000 7f23b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:12.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.671+0000 7f23b3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:12.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.838+0000 7f23c1e4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:12.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.838+0000 7f23c1e4f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:12.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.838+0000 7f23c1e4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:12.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:12.838+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f23bc04bf60 con 0x7f23bc131c80 2024-09-15T01:55:13.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.558+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v86) v1 ==== 85+0+0 (secure 0 0 0) 0x7f23a40104c0 con 0x7f23bc131c80 2024-09-15T01:55:13.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.558+0000 7f23b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:13.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.558+0000 7f23b3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:13.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.663+0000 7f23c1e4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:13.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.663+0000 7f23c1e4f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:13.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.663+0000 7f23c1e4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.663+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f23bc09bf20 con 0x7f23bc131c80 2024-09-15T01:55:13.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.664+0000 7f23b3fff640 1 -- 172.21.15.67:0/895088564 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v86) v1 ==== 113+0+0 (secure 0 0 0) 0x7f23a40106a0 con 0x7f23bc131c80 2024-09-15T01:55:13.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.664+0000 7f23b3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T01:55:13.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.664+0000 7f23b3fff640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-15T01:55:13.667 INFO:tasks.workunit.client.0.smithi067.stderr:rule foo-rule does not exist 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 msgr2=0x7f239c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 0x7f239c03aba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0038f0 tx=0x7f23ac006d80 comp rx=0 tx=0).stop 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 10 monclient: shutdown 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 msgr2=0x7f23bc099960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c5e0 tx=0x7f23a4007ed0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 shutdown_connections 2024-09-15T01:55:13.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f239c0386f0 0x7f239c03aba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc136fb0 0x7f23bc099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc131c80 0x7f23bc099960 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 --2- 172.21.15.67:0/895088564 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23bc09f3e0 0x7f23bc130c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 >> 172.21.15.67:0/895088564 conn(0x7f23bc0a52e0 msgr2=0x7f23bc09deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 shutdown_connections 2024-09-15T01:55:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.666+0000 7f23c1e4f640 1 -- 172.21.15.67:0/895088564 wait complete. 2024-09-15T01:55:13.678 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm bar 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.753+0000 7f286fca8640 10 monclient: get_monmap_and_config 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.753+0000 7f286fca8640 10 monclient: build_initial_monmap 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.753+0000 7f286fca8640 10 monclient: monmap: 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:13.755083+0000 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:13.755083+0000 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:13.756 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:13.757 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.754+0000 7f286fca8640 1 Processor -- start 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.754+0000 7f286fca8640 1 -- start start 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.754+0000 7f286fca8640 10 monclient: init 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2868097ba0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28680672c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f286813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-b con 0x7f2868138010 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286809e430 con 0x7f2868097ba0 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680977a0 con 0x7f28680672c0 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2868098680 con 0x7f2868138010 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286fca8640 10 monclient(hunting): authenticate will time out at 946.224379s 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286d243640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f28680672c0 0x7f2868097090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60572/0 (socket says 172.21.15.67:60572) 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 learned_addr learned my addr 172.21.15.67:0/1633334313 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286e245640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 msgr2=0x7f286813a3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:13.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.755+0000 7f286e245640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f286813a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:13.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 >> v1:172.21.15.67:6789/0 conn(0x7f28680672c0 legacy=0x7f2868097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286da44640 1 -- 172.21.15.67:0/1633334313 >> v1:172.21.15.17:6789/0 conn(0x7f2868097ba0 legacy=0x7f2868097f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495654564 0 0) 0x7f28680977a0 con 0x7f28680672c0 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286ca42640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286ca42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286ca42640 10 monclient(hunting): my global_id is 5395 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.757+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28540034e0 con 0x7f28680672c0 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976619462 0 0) 0x7f286809e430 con 0x7f2868097ba0 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): my global_id is 44757 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28680977a0 con 0x7f2868097ba0 2024-09-15T01:55:13.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2826853608 0 0) 0x7f28540034e0 con 0x7f28680672c0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 msgr2=0x7f286813a3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f286813a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 >> v1:172.21.15.17:6789/0 conn(0x7f2868097ba0 legacy=0x7f2868097f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: _finish_hunting 0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286809e150 con 0x7f28680672c0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: _finish_auth 0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286fca8640 5 monclient: authenticate success, global_id 5395 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f2858001720 con 0x7f28680672c0 2024-09-15T01:55:13.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286fca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: dump: 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:13.767 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 >> v1:172.21.15.67:6789/0 conn(0x7f28680672c0 legacy=0x7f2868097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f2854003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): picked mon.c con 0x7f2868138010 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 0x7f2854004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): picked mon.b con 0x7f28540040e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2854004de0 0x7f2854005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): picked mon.a con 0x7f2854004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285400bd00 con 0x7f2854004de0 2024-09-15T01:55:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285400be70 con 0x7f28540040e0 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285400c030 con 0x7f2868138010 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286ca42640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286fca8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286da44640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 msgr2=0x7f2854003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286da44640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f2854003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 0x7f2854004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): get_auth_request con 0x7f28540040e0 auth_method 0 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2854004de0 0x7f2854005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): get_auth_request con 0x7f2854004de0 auth_method 0 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient(hunting): handle_auth_done global_id 5395 payload 293 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 msgr2=0x7f2854003b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868138010 0x7f2854003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2854004de0 msgr2=0x7f2854005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2854004de0 0x7f2854005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient: _finish_hunting 0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 monclient: found mon.b 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540034e0 con 0x7f28540040e0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 10 monclient: _finish_auth 0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.758+0000 7f286d243640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.761+0000 7f286d243640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 msgr2=0x7f2854004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.761+0000 7f286d243640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 0x7f2854004780 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2858001490 tx=0x7f2858007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23c75c0bdbbb25d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858005890 con 0x7f28540040e0 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:13.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: dump: 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858005a10 con 0x7f28540040e0 2024-09-15T01:55:13.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 1 -- 172.21.15.67:0/1633334313 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858005dd0 con 0x7f28540040e0 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 4 set_mon_vals no callback set 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286ca42640 10 monclient: dump: 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 10 monclient: shutdown 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 -- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 msgr2=0x7f2854004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 --2- 172.21.15.67:0/1633334313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540040e0 0x7f2854004780 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2858001490 tx=0x7f2858007a50 comp rx=0 tx=0).stop 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286e245640 1 -- 172.21.15.67:0/1633334313 reap_dead start 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 -- 172.21.15.67:0/1633334313 shutdown_connections 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 -- 172.21.15.67:0/1633334313 >> 172.21.15.67:0/1633334313 conn(0x7f28680a2110 msgr2=0x7f28680a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 -- 172.21.15.67:0/1633334313 shutdown_connections 2024-09-15T01:55:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.762+0000 7f286fca8640 1 -- 172.21.15.67:0/1633334313 wait complete. 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient: build_initial_monmap 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient: monmap: 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:13.779 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 Processor -- start 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 -- start start 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient: init 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 0x7f2868134b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-a con 0x7f28540040e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 0x7f2868135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-c con 0x7f2868135080 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): picked mon.noname-b con 0x7f2868135990 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:13.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2868135d80 con 0x7f28540040e0 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286811d120 con 0x7f2868135990 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2868067ad0 con 0x7f2868135080 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.763+0000 7f286fca8640 10 monclient(hunting): authenticate will time out at 946.232924s 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286d243640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 msgr2=0x7f2868135450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286d243640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 0x7f2868135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35150/0 (socket says 172.21.15.67:35150) 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 -- 172.21.15.67:0/677225307 learned_addr learned my addr 172.21.15.67:0/677225307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:13.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): get_auth_request con 0x7f2868135990 auth_method 0 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286da44640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 0x7f2868134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient(hunting): handle_auth_done global_id 5398 payload 824 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 -- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 msgr2=0x7f2868135450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 0x7f2868135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 -- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 msgr2=0x7f2868134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 0x7f2868134b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:13.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient: _finish_hunting 0 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 monclient: found mon.noname-b 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286e245640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28680a08c0 con 0x7f2868135990 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286da44640 10 monclient: get_auth_request con 0x7f28540040e0 auth_method 0 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.764+0000 7f286da44640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 0x7f2868134b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f286e245640 1 -- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 msgr2=0x7f28681475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f286e245640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f286000cf50 tx=0x7f28600095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860011070 con 0x7f2868135990 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 10 monclient: dump: 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:13.783 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286000bbe0 con 0x7f2868135990 2024-09-15T01:55:13.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860016420 con 0x7f2868135990 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.766+0000 7f286fca8640 5 monclient: authenticate success, global_id 5398 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 10 monclient: _renew_subs 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2868136400 con 0x7f2868135990 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f285dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f285dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f285dffb640 10 monclient: dump: 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:13.785 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f285ffff640 4 set_mon_vals no callback set 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 10 monclient: _renew_subs 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2868136fd0 con 0x7f2868135990 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.767+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f28600167d0 con 0x7f2868135990 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f285dffb640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 0x7f284c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286fca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286fca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:13.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286fca8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f2860006e40 con 0x7f2868135990 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286804bf60 con 0x7f2868135990 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286da44640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 0x7f284c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.768+0000 7f286da44640 10 monclient: get_auth_request con 0x7f284c0386f0 auth_method 0 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.770+0000 7f286da44640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 0x7f284c03aba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28640014e0 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.772+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2860013500 con 0x7f2868135990 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.772+0000 7f285dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.772+0000 7f285dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:13.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.942+0000 7f286fca8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:13.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.942+0000 7f286fca8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:13.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.942+0000 7f286fca8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:13.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:13.942+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f2868067840 con 0x7f2868135990 2024-09-15T01:55:14.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.562+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f286001db30 con 0x7f2868135990 2024-09-15T01:55:14.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.562+0000 7f285dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:14.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.562+0000 7f285dffb640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-15T01:55:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.666+0000 7f286fca8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.666+0000 7f286fca8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.666+0000 7f286fca8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.666+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f2868137bd0 con 0x7f2868135990 2024-09-15T01:55:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.668+0000 7f285dffb640 1 -- 172.21.15.67:0/677225307 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f286001dd10 con 0x7f2868135990 2024-09-15T01:55:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.668+0000 7f285dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T01:55:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.668+0000 7f285dffb640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-15T01:55:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:device 'bar' does not appear in the crush map 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 msgr2=0x7f284c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 0x7f284c03aba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28640014e0 tx=0x7f2864009040 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 10 monclient: shutdown 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 msgr2=0x7f28681475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f286000cf50 tx=0x7f28600095f0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 shutdown_connections 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f284c0386f0 0x7f284c03aba0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2868135990 0x7f28681475b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2868135080 0x7f2868135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 --2- 172.21.15.67:0/677225307 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28540040e0 0x7f2868134b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 >> 172.21.15.67:0/677225307 conn(0x7f28680a2110 msgr2=0x7f28680a10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:14.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 shutdown_connections 2024-09-15T01:55:14.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.670+0000 7f286fca8640 1 -- 172.21.15.67:0/677225307 wait complete. 2024-09-15T01:55:14.681 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm foo 2024-09-15T01:55:14.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 10 monclient: get_monmap_and_config 2024-09-15T01:55:14.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 10 monclient: build_initial_monmap 2024-09-15T01:55:14.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 10 monclient: monmap: 2024-09-15T01:55:14.757 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:14.757 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:14.756266+0000 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:14.756266+0000 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:14.758 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 1 Processor -- start 2024-09-15T01:55:14.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 1 -- start start 2024-09-15T01:55:14.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.755+0000 7f888e100640 10 monclient: init 2024-09-15T01:55:14.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:14.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 0x7f88881321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): picked mon.noname-b con 0x7f8888098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88880a0370 0x7f8888098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): picked mon.noname-c con 0x7f88880a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 0x7f88880695c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): picked mon.noname-a con 0x7f8888098a50 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88880a1730 con 0x7f8888098a50 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8888098ef0 con 0x7f88880a0370 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88880a0b80 con 0x7f8888098140 2024-09-15T01:55:14.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f888e100640 10 monclient(hunting): authenticate will time out at 947.225382s 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f88877fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 msgr2=0x7f88881321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88880a0370 0x7f8888098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f88877fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 0x7f88881321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88880a0370 0x7f8888098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35160/0 (socket says 172.21.15.67:35160) 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 -- 172.21.15.67:0/4042751642 learned_addr learned my addr 172.21.15.67:0/4042751642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): get_auth_request con 0x7f88880a0370 auth_method 0 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:14.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8887fff640 1 --2- 172.21.15.67:0/4042751642 >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 0x7f88880695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient(hunting): handle_auth_done global_id 5401 payload 274 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 -- 172.21.15.67:0/4042751642 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 msgr2=0x7f88881321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 --2- 172.21.15.67:0/4042751642 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 0x7f88881321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 -- 172.21.15.67:0/4042751642 >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 msgr2=0x7f88880695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 --2- 172.21.15.67:0/4042751642 >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 0x7f88880695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 monclient: found mon.noname-c 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8886ffd640 1 -- 172.21.15.67:0/4042751642 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880a00b0 con 0x7f88880a0370 2024-09-15T01:55:14.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8887fff640 10 monclient: get_auth_request con 0x7f8888098a50 auth_method 0 2024-09-15T01:55:14.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.756+0000 7f8887fff640 1 --2- 172.21.15.67:0/4042751642 >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 0x7f88880695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:14.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f8886ffd640 1 -- 172.21.15.67:0/4042751642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88880a0370 msgr2=0x7f8888098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:14.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f8886ffd640 1 --2- 172.21.15.67:0/4042751642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88880a0370 0x7f8888098510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f88880a52f0 tx=0x7f887c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9ef962d591e8429 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:14.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 1 -- 172.21.15.67:0/4042751642 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c032440 con 0x7f88880a0370 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: dump: 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:14.770 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f888e100640 5 monclient: authenticate success, global_id 5401 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f888e100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 1 -- 172.21.15.67:0/4042751642 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c0325a0 con 0x7f88880a0370 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f888e100640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f888e100640 4 set_mon_vals no callback set 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 1 -- 172.21.15.67:0/4042751642 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c032940 con 0x7f88880a0370 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.759+0000 7f88867fc640 10 monclient: dump: 2024-09-15T01:55:14.771 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 10 monclient: shutdown 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 -- 172.21.15.67:0/4042751642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88880a0370 msgr2=0x7f8888098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:14.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 --2- 172.21.15.67:0/4042751642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88880a0370 0x7f8888098510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f88880a52f0 tx=0x7f887c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 -- 172.21.15.67:0/4042751642 shutdown_connections 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 --2- 172.21.15.67:0/4042751642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88880a0370 0x7f8888098510 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 --2- 172.21.15.67:0/4042751642 >> v2:172.21.15.17:3300/0 conn(0x7f8888098a50 0x7f88880695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 --2- 172.21.15.67:0/4042751642 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098140 0x7f88881321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 -- 172.21.15.67:0/4042751642 >> 172.21.15.67:0/4042751642 conn(0x7f88880a5780 msgr2=0x7f88880a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 -- 172.21.15.67:0/4042751642 shutdown_connections 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 1 -- 172.21.15.67:0/4042751642 wait complete. 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 10 monclient: build_initial_monmap 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.760+0000 7f888e100640 10 monclient: monmap: 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:14.773 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 Processor -- start 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 -- start start 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient: init 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): picked mon.noname-b con 0x7f8888098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 0x7f8888130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): picked mon.noname-c con 0x7f8888098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88880a0370 0x7f888811d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): picked mon.noname-a con 0x7f88880a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8888141ee0 con 0x7f88880a0370 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888811f410 con 0x7f8888098140 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8888121270 con 0x7f8888098a50 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:14.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f888e100640 10 monclient(hunting): authenticate will time out at 947.230910s 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f8886ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 msgr2=0x7f8888130df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f8886ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 0x7f8888130df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.761+0000 7f88877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35162/0 (socket says 172.21.15.67:35162) 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 -- 172.21.15.67:0/2020021384 learned_addr learned my addr 172.21.15.67:0/2020021384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): get_auth_request con 0x7f8888098140 auth_method 0 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:14.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88880a0370 0x7f888811d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): get_auth_request con 0x7f88880a0370 auth_method 0 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f8887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient(hunting): handle_auth_done global_id 5404 payload 824 2024-09-15T01:55:14.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 -- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 msgr2=0x7f8888130df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 0x7f8888130df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 -- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88880a0370 msgr2=0x7f888811d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88880a0370 0x7f888811d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.762+0000 7f88877fe640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009450 con 0x7f8888098140 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.764+0000 7f88877fe640 1 -- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 msgr2=0x7f8888140fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.764+0000 7f88877fe640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f887800bbb0 tx=0x7f88780097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.764+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887800cc80 con 0x7f8888098140 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: dump: 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:14.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887800cde0 con 0x7f8888098140 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 5 monclient: authenticate success, global_id 5404 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _renew_subs 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8888131df0 con 0x7f8888098140 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878016430 con 0x7f8888098140 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 10 monclient: dump: 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:14.780 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f8885ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _renew_subs 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88881416a0 con 0x7f8888098140 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8878012020 con 0x7f8888098140 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:14.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.765+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888804bf60 con 0x7f8888098140 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.766+0000 7f886f7fe640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 0x7f887403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.766+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5840+0+0 (secure 0 0 0) 0x7f887804bdd0 con 0x7f8888098140 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.766+0000 7f8886ffd640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 0x7f887403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.766+0000 7f8886ffd640 10 monclient: get_auth_request con 0x7f8874038720 auth_method 0 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.768+0000 7f8886ffd640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 0x7f887403abd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f88880a1660 tx=0x7f887c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.770+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f887801d900 con 0x7f8888098140 2024-09-15T01:55:14.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.770+0000 7f886f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:14.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.770+0000 7f886f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.937+0000 7f888e100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:14.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.937+0000 7f888e100640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:14.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.937+0000 7f888e100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:14.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:14.937+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f8888071470 con 0x7f8888098140 2024-09-15T01:55:15.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.565+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v88) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8878014340 con 0x7f8888098140 2024-09-15T01:55:15.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.565+0000 7f886f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:15.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.565+0000 7f886f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-15T01:55:15.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.670+0000 7f888e100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:15.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.670+0000 7f888e100640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:15.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.670+0000 7f888e100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:15.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.670+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f88880008d0 con 0x7f8888098140 2024-09-15T01:55:15.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.671+0000 7f886f7fe640 1 -- 172.21.15.67:0/2020021384 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v88) v1 ==== 120+0+0 (secure 0 0 0) 0x7f887801d300 con 0x7f8888098140 2024-09-15T01:55:15.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.671+0000 7f886f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:15.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.671+0000 7f886f7fe640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-15T01:55:15.673 INFO:tasks.workunit.client.0.smithi067.stderr:device 'foo' does not appear in the crush map 2024-09-15T01:55:15.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 msgr2=0x7f887403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:15.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 0x7f887403abd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f88880a1660 tx=0x7f887c032000 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 10 monclient: shutdown 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 msgr2=0x7f8888140fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f887800bbb0 tx=0x7f88780097f0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 shutdown_connections 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8874038720 0x7f887403abd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88880a0370 0x7f888811d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8888098a50 0x7f8888130df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 --2- 172.21.15.67:0/2020021384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8888098140 0x7f8888140fc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 >> 172.21.15.67:0/2020021384 conn(0x7f88880a5780 msgr2=0x7f888809e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 shutdown_connections 2024-09-15T01:55:15.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.673+0000 7f888e100640 1 -- 172.21.15.67:0/2020021384 wait complete. 2024-09-15T01:55:15.684 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.9 host2 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.758+0000 7f66a179d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.758+0000 7f66a179d640 10 monclient: build_initial_monmap 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.758+0000 7f66a179d640 10 monclient: monmap: 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:15.760115+0000 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:15.760115+0000 2024-09-15T01:55:15.761 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:15.762 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.759+0000 7f66a179d640 1 Processor -- start 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.759+0000 7f66a179d640 1 -- start start 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.759+0000 7f66a179d640 10 monclient: init 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f669c098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-b con 0x7f669c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f669c098a50 0x7f669c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-a con 0x7f669c098a50 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669c098ef0 con 0x7f669c098140 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669c0a1730 con 0x7f669c098a50 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669c0a0b80 con 0x7f669c0a0370 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f66a179d640 10 monclient(hunting): authenticate will time out at 948.229300s 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f669c098140 0x7f669c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60580/0 (socket says 172.21.15.67:60580) 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669affd640 1 -- 172.21.15.67:0/1469428376 learned_addr learned my addr 172.21.15.67:0/1469428376 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669a7fc640 1 -- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 msgr2=0x7f669c098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669a7fc640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 --2- 172.21.15.67:0/1469428376 >> v2:172.21.15.17:3300/0 conn(0x7f669c098a50 0x7f669c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): get_auth_request con 0x7f669c098a50 auth_method 0 2024-09-15T01:55:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient(hunting): handle_auth_done global_id 44763 payload 274 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 -- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 msgr2=0x7f669c098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 -- 172.21.15.67:0/1469428376 >> v1:172.21.15.67:6789/0 conn(0x7f669c098140 legacy=0x7f669c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:15.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.760+0000 7f669b7fe640 1 -- 172.21.15.67:0/1469428376 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c1328f0 con 0x7f669c098a50 2024-09-15T01:55:15.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.763+0000 7f669b7fe640 1 -- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098a50 msgr2=0x7f669c1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.763+0000 7f669b7fe640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098a50 0x7f669c1390c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f669c0a1660 tx=0x7f66940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eac5a97b02f700f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.763+0000 7f6699ffb640 1 -- 172.21.15.67:0/1469428376 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694032440 con 0x7f669c098a50 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.763+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: dump: 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 5 monclient: authenticate success, global_id 44763 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 1 -- 172.21.15.67:0/1469428376 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66940325a0 con 0x7f669c098a50 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 4 set_mon_vals no callback set 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 1 -- 172.21.15.67:0/1469428376 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694032940 con 0x7f669c098a50 2024-09-15T01:55:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f6699ffb640 10 monclient: dump: 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 10 monclient: shutdown 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 -- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098a50 msgr2=0x7f669c1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098a50 0x7f669c1390c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f669c0a1660 tx=0x7f66940090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 -- 172.21.15.67:0/1469428376 shutdown_connections 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098a50 0x7f669c1390c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 --2- 172.21.15.67:0/1469428376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 -- 172.21.15.67:0/1469428376 >> 172.21.15.67:0/1469428376 conn(0x7f669c0a5780 msgr2=0x7f669c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 -- 172.21.15.67:0/1469428376 shutdown_connections 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.764+0000 7f66a179d640 1 -- 172.21.15.67:0/1469428376 wait complete. 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient: build_initial_monmap 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient: monmap: 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 1 Processor -- start 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 1 -- start start 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient: init 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098140 0x7f669c09b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-a con 0x7f669c098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.765+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-b con 0x7f669c098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c12f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): picked mon.noname-c con 0x7f669c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c099100 con 0x7f669c098140 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1318a0 con 0x7f669c098a50 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1315c0 con 0x7f669c0a0370 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:15.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f66a179d640 10 monclient(hunting): authenticate will time out at 948.235032s 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35176/0 (socket says 172.21.15.67:35176) 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 -- 172.21.15.67:0/4249427304 learned_addr learned my addr 172.21.15.67:0/4249427304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): get_auth_request con 0x7f669c098a50 auth_method 0 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669b7fe640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 msgr2=0x7f669c12f580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:15.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669b7fe640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c12f580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669affd640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098140 0x7f669c09b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient(hunting): handle_auth_done global_id 5407 payload 824 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 msgr2=0x7f669c12f580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c12f580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098140 msgr2=0x7f669c09b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098140 0x7f669c09b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 monclient: found mon.noname-b 2024-09-15T01:55:15.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.766+0000 7f669a7fc640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694009390 con 0x7f669c098a50 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f669a7fc640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 msgr2=0x7f669c12f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f669a7fc640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f669000b5e0 tx=0x7f6690006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690003d80 con 0x7f669c098a50 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 10 monclient: dump: 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:15.781 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f669a7fc640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f669000b5e0 tx=0x7f6690006f90 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f6690006150 config(0 keys) v1 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 5 monclient: authenticate success, global_id 5407 2024-09-15T01:55:15.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 10 monclient: _renew_subs 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c131b80 con 0x7f669c098a50 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 10 monclient: _renew_subs 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c138a90 con 0x7f669c098a50 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6690006150 con 0x7f669c098a50 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f6682ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.768+0000 7f66997fa640 4 set_mon_vals no callback set 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690006510 con 0x7f669c098a50 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f6682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f6682ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f6682ffd640 10 monclient: dump: 2024-09-15T01:55:15.783 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f66900103e0 con 0x7f669c098a50 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f66a179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f66a179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:15.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f66a179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.769+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c097f60 con 0x7f669c098a50 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.770+0000 7f6682ffd640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 0x7f668803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.770+0000 7f669affd640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 0x7f668803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.770+0000 7f669affd640 10 monclient: get_auth_request con 0x7f6688038720 auth_method 0 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.772+0000 7f669affd640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 0x7f668803abd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f668c005b40 tx=0x7f668c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:15.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:15.774+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5711+0+0 (secure 0 0 0) 0x7f669004b730 con 0x7f669c098a50 2024-09-15T01:55:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.391+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f669c097f60 con 0x7f669c098a50 2024-09-15T01:55:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.391+0000 7f6682ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:16.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.391+0000 7f6682ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:16.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.563+0000 7f66a179d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:16.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.563+0000 7f66a179d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:16.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.564+0000 7f66a179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:16.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:16.564+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f669c04bf60 con 0x7f669c098a50 2024-09-15T01:55:17.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.202+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v89) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6690017020 con 0x7f669c098a50 2024-09-15T01:55:17.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.202+0000 7f6682ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:17.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.202+0000 7f6682ffd640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-15T01:55:17.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.311+0000 7f66a179d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:17.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.311+0000 7f66a179d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:17.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.311+0000 7f66a179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.311+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f669c071390 con 0x7f669c098a50 2024-09-15T01:55:17.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.313+0000 7f6682ffd640 1 -- 172.21.15.67:0/4249427304 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v89) v1 ==== 145+0+0 (secure 0 0 0) 0x7f6690017070 con 0x7f669c098a50 2024-09-15T01:55:17.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.313+0000 7f6682ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T01:55:17.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.313+0000 7f6682ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T01:55:17.316 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.9' does not appear in the crush map 2024-09-15T01:55:17.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 msgr2=0x7f668803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 0x7f668803abd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f668c005b40 tx=0x7f668c009040 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 10 monclient: shutdown 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 msgr2=0x7f669c12f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f669000b5e0 tx=0x7f6690006f90 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 shutdown_connections 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6688038720 0x7f668803abd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f669c0a0370 0x7f669c12f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c098a50 0x7f669c12f040 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 --2- 172.21.15.67:0/4249427304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f669c098140 0x7f669c09b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 >> 172.21.15.67:0/4249427304 conn(0x7f669c0a5780 msgr2=0x7f669c0a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:17.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 shutdown_connections 2024-09-15T01:55:17.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.315+0000 7f66a179d640 1 -- 172.21.15.67:0/4249427304 wait complete. 2024-09-15T01:55:17.326 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm host2 2024-09-15T01:55:17.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.401+0000 7f97b015d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:17.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.401+0000 7f97b015d640 10 monclient: build_initial_monmap 2024-09-15T01:55:17.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.401+0000 7f97b015d640 10 monclient: monmap: 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:17.403019+0000 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:17.403019+0000 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:17.405 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:17.406 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.402+0000 7f97b015d640 1 Processor -- start 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.402+0000 7f97b015d640 1 -- start start 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.402+0000 7f97b015d640 10 monclient: init 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97a80a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f97a809eea0 0x7f97a809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-a con 0x7f97a809eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a8136e40 0x7f97a8139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-c con 0x7f97a8136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a806a270 con 0x7f97a80a0b50 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a806f290 con 0x7f97a809eea0 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:17.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a809f7e0 con 0x7f97a8136e40 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97b015d640 10 monclient(hunting): authenticate will time out at 949.872165s 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a8136e40 0x7f97a8139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a8136e40 0x7f97a8139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35188/0 (socket says 172.21.15.67:35188) 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 -- 172.21.15.67:0/145257875 learned_addr learned my addr 172.21.15.67:0/145257875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): get_auth_request con 0x7f97a8136e40 auth_method 0 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:17.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ad6f8640 1 --2- 172.21.15.67:0/145257875 >> v2:172.21.15.17:3300/0 conn(0x7f97a809eea0 0x7f97a809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ad6f8640 10 monclient(hunting): get_auth_request con 0x7f97a809eea0 auth_method 0 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ad6f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ad6f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ad6f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_done global_id 5410 payload 274 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 -- 172.21.15.67:0/145257875 >> v2:172.21.15.17:3300/0 conn(0x7f97a809eea0 msgr2=0x7f97a809f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 --2- 172.21.15.67:0/145257875 >> v2:172.21.15.17:3300/0 conn(0x7f97a809eea0 0x7f97a809f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 -- 172.21.15.67:0/145257875 >> v1:172.21.15.17:6789/0 conn(0x7f97a80a0b50 legacy=0x7f97a8069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient: _finish_hunting 0 2024-09-15T01:55:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 monclient: found mon.noname-c 2024-09-15T01:55:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.403+0000 7f97ae6fa640 1 -- 172.21.15.67:0/145257875 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80a0f20 con 0x7f97a8136e40 2024-09-15T01:55:17.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97ae6fa640 1 -- 172.21.15.67:0/145257875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 msgr2=0x7f97a8139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97ae6fa640 1 --2- 172.21.15.67:0/145257875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a8139220 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f97a809cdb0 tx=0x7f97a0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9edfeefe8409bf93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 1 -- 172.21.15.67:0/145257875 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0007610 con 0x7f97a8136e40 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: dump: 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.419 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: _finish_auth 0 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 1 -- 172.21.15.67:0/145257875 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a002d050 con 0x7f97a8136e40 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97b015d640 5 monclient: authenticate success, global_id 5410 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97b015d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97b015d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 1 -- 172.21.15.67:0/145257875 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a00079a0 con 0x7f97a8136e40 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97b015d640 4 set_mon_vals no callback set 2024-09-15T01:55:17.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.406+0000 7f97acef7640 10 monclient: dump: 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 10 monclient: shutdown 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 -- 172.21.15.67:0/145257875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 msgr2=0x7f97a8139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 --2- 172.21.15.67:0/145257875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a8139220 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f97a809cdb0 tx=0x7f97a0009840 comp rx=0 tx=0).stop 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 -- 172.21.15.67:0/145257875 shutdown_connections 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 --2- 172.21.15.67:0/145257875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a8139220 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 --2- 172.21.15.67:0/145257875 >> v2:172.21.15.17:3300/0 conn(0x7f97a809eea0 0x7f97a809f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 -- 172.21.15.67:0/145257875 >> 172.21.15.67:0/145257875 conn(0x7f97a8071290 msgr2=0x7f97a80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 -- 172.21.15.67:0/145257875 shutdown_connections 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 1 -- 172.21.15.67:0/145257875 wait complete. 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 10 monclient: build_initial_monmap 2024-09-15T01:55:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.407+0000 7f97b015d640 10 monclient: monmap: 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 Processor -- start 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 -- start start 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient: init 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:17.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97a809eea0 0x7f97a811e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-a con 0x7f97a809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 0x7f97a811ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-c con 0x7f97a80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): picked mon.noname-b con 0x7f97a8136e40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8121bb0 con 0x7f97a809eea0 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:17.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80a1260 con 0x7f97a8136e40 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a809e650 con 0x7f97a80a0b50 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97b015d640 10 monclient(hunting): authenticate will time out at 949.877527s 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ad6f8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 msgr2=0x7f97a811ed20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ad6f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 0x7f97a811ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35204/0 (socket says 172.21.15.67:35204) 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 -- 172.21.15.67:0/1899852247 learned_addr learned my addr 172.21.15.67:0/1899852247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): get_auth_request con 0x7f97a8136e40 auth_method 0 2024-09-15T01:55:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97a809eea0 0x7f97a811e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): get_auth_request con 0x7f97a809eea0 auth_method 0 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97adef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 10 monclient(hunting): handle_auth_done global_id 5413 payload 824 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 -- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 msgr2=0x7f97a811ed20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 0x7f97a811ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 -- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97a809eea0 msgr2=0x7f97a811e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.408+0000 7f97ae6fa640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97a809eea0 0x7f97a811e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.409+0000 7f97ae6fa640 10 monclient: _finish_hunting 0 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.409+0000 7f97ae6fa640 1 monclient: found mon.noname-b 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.409+0000 7f97ae6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.409+0000 7f97ae6fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.409+0000 7f97ae6fa640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0009ae0 con 0x7f97a8136e40 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97ae6fa640 1 -- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 msgr2=0x7f97a811f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97ae6fa640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009ab0 tx=0x7f97a0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0032c60 con 0x7f97a8136e40 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: dump: 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:17.428 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a0032dc0 con 0x7f97a8136e40 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 5 monclient: authenticate success, global_id 5413 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 10 monclient: _renew_subs 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81200f0 con 0x7f97a8136e40 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a003b640 con 0x7f97a8136e40 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:17.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979dffb640 10 monclient: dump: 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 10 monclient: _renew_subs 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a81218a0 con 0x7f97a8136e40 2024-09-15T01:55:17.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.411+0000 7f979ffff640 4 set_mon_vals no callback set 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97a0038050 con 0x7f97a8136e40 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97b015d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97b015d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97b015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a804bf60 con 0x7f97a8136e40 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f979dffb640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 0x7f978c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5690+0+0 (secure 0 0 0) 0x7f97a006caf0 con 0x7f97a8136e40 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97adef9640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 0x7f978c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.412+0000 7f97adef9640 10 monclient: get_auth_request con 0x7f978c038740 auth_method 0 2024-09-15T01:55:17.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.415+0000 7f97adef9640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 0x7f978c03abf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:17.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.417+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f97a003b9f0 con 0x7f97a8136e40 2024-09-15T01:55:17.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.417+0000 7f979dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:17.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.417+0000 7f979dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:17.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.586+0000 7f97b015d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:17.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.586+0000 7f97b015d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:17.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.586+0000 7f97b015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:17.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:17.586+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f97a8120d10 con 0x7f97a8136e40 2024-09-15T01:55:18.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.568+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v90) v1 ==== 124+0+0 (secure 0 0 0) 0x7f97a003f030 con 0x7f97a8136e40 2024-09-15T01:55:18.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.568+0000 7f979dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:18.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.568+0000 7f979dffb640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T01:55:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.677+0000 7f97b015d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.677+0000 7f97b015d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.677+0000 7f97b015d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.677+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f97a8120d10 con 0x7f97a8136e40 2024-09-15T01:55:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.679+0000 7f979dffb640 1 -- 172.21.15.67:0/1899852247 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v90) v1 ==== 124+0+0 (secure 0 0 0) 0x7f97a003f020 con 0x7f97a8136e40 2024-09-15T01:55:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.679+0000 7f979dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T01:55:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.679+0000 7f979dffb640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-15T01:55:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:device 'host2' does not appear in the crush map 2024-09-15T01:55:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 msgr2=0x7f978c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 0x7f978c03abf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4007e10 comp rx=0 tx=0).stop 2024-09-15T01:55:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 10 monclient: shutdown 2024-09-15T01:55:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 msgr2=0x7f97a811f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009ab0 tx=0x7f97a0036000 comp rx=0 tx=0).stop 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 shutdown_connections 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f978c038740 0x7f978c03abf0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a8136e40 0x7f97a811f260 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97a80a0b50 0x7f97a811ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 --2- 172.21.15.67:0/1899852247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97a809eea0 0x7f97a811e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 >> 172.21.15.67:0/1899852247 conn(0x7f97a8071290 msgr2=0x7f97a809ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 shutdown_connections 2024-09-15T01:55:18.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.681+0000 7f97b015d640 1 -- 172.21.15.67:0/1899852247 wait complete. 2024-09-15T01:55:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket foo host 2024-09-15T01:55:18.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 10 monclient: get_monmap_and_config 2024-09-15T01:55:18.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 10 monclient: build_initial_monmap 2024-09-15T01:55:18.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 10 monclient: monmap: 2024-09-15T01:55:18.770 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:18.770 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:18.769352+0000 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:18.769352+0000 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:18.771 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 1 Processor -- start 2024-09-15T01:55:18.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 1 -- start start 2024-09-15T01:55:18.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.768+0000 7f51bc0a8640 10 monclient: init 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51b40a3850 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51b406a6e0 0x7f51b4136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f51b406a6e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f51b4137240 0x7f51b4139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f51b4137240 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b409f9a0 con 0x7f51b40a3850 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b4069c20 con 0x7f51b4137240 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:18.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b40696c0 con 0x7f51b406a6e0 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51bc0a8640 10 monclient(hunting): authenticate will time out at 951.238485s 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51b406a6e0 0x7f51b4136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51b406a6e0 0x7f51b4136d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35208/0 (socket says 172.21.15.67:35208) 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 -- 172.21.15.67:0/961240320 learned_addr learned my addr 172.21.15.67:0/961240320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): get_auth_request con 0x7f51b406a6e0 auth_method 0 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:18.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51ba645640 1 --2- 172.21.15.67:0/961240320 >> v2:172.21.15.17:3300/0 conn(0x7f51b4137240 0x7f51b4139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient(hunting): handle_auth_done global_id 5416 payload 274 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 -- 172.21.15.67:0/961240320 >> v2:172.21.15.17:3300/0 conn(0x7f51b4137240 msgr2=0x7f51b4139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 --2- 172.21.15.67:0/961240320 >> v2:172.21.15.17:3300/0 conn(0x7f51b4137240 0x7f51b4139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 -- 172.21.15.67:0/961240320 >> v1:172.21.15.17:6789/0 conn(0x7f51b40a3850 legacy=0x7f51b406e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient: _finish_hunting 0 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 monclient: found mon.noname-c 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.769+0000 7f51b9643640 1 -- 172.21.15.67:0/961240320 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b40a3c20 con 0x7f51b406a6e0 2024-09-15T01:55:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b9643640 1 -- 172.21.15.67:0/961240320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b406a6e0 msgr2=0x7f51b4136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b9643640 1 --2- 172.21.15.67:0/961240320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b406a6e0 0x7f51b4136d00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f51b40a3f00 tx=0x7f51a40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8229c1057dd9ed26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 1 -- 172.21.15.67:0/961240320 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a402ae30 con 0x7f51b406a6e0 2024-09-15T01:55:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: dump: 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: _finish_auth 0 2024-09-15T01:55:18.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 1 -- 172.21.15.67:0/961240320 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a402f4b0 con 0x7f51b406a6e0 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51bc0a8640 5 monclient: authenticate success, global_id 5416 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51bc0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51bc0a8640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 1 -- 172.21.15.67:0/961240320 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a402f850 con 0x7f51b406a6e0 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51bc0a8640 4 set_mon_vals no callback set 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.772+0000 7f51b8e42640 10 monclient: dump: 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.783 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 10 monclient: shutdown 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 -- 172.21.15.67:0/961240320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b406a6e0 msgr2=0x7f51b4136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/961240320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b406a6e0 0x7f51b4136d00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f51b40a3f00 tx=0x7f51a40091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 -- 172.21.15.67:0/961240320 shutdown_connections 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/961240320 >> v2:172.21.15.17:3300/0 conn(0x7f51b4137240 0x7f51b4139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/961240320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b406a6e0 0x7f51b4136d00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 -- 172.21.15.67:0/961240320 >> 172.21.15.67:0/961240320 conn(0x7f51b4070ff0 msgr2=0x7f51b40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 -- 172.21.15.67:0/961240320 shutdown_connections 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 1 -- 172.21.15.67:0/961240320 wait complete. 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 10 monclient: build_initial_monmap 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.773+0000 7f51bc0a8640 10 monclient: monmap: 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 Processor -- start 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 -- start start 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient: init 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51b406a6e0 0x7f51b411ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f51b406a6e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 0x7f51b411f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f51b40a3850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f51b4137240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b4120420 con 0x7f51b406a6e0 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b406e9e0 con 0x7f51b4137240 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b40a27e0 con 0x7f51b40a3850 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51bc0a8640 10 monclient(hunting): authenticate will time out at 951.243804s 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51b9643640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 msgr2=0x7f51b411f140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51b9643640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 0x7f51b411f140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:18.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35218/0 (socket says 172.21.15.67:35218) 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 1 -- 172.21.15.67:0/2029053198 learned_addr learned my addr 172.21.15.67:0/2029053198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 10 monclient(hunting): get_auth_request con 0x7f51b4137240 auth_method 0 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51ba645640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.774+0000 7f51b9e44640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51b406a6e0 0x7f51b411ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): get_auth_request con 0x7f51b406a6e0 auth_method 0 2024-09-15T01:55:18.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51b9e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient(hunting): handle_auth_done global_id 5419 payload 824 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 -- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 msgr2=0x7f51b411f140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 0x7f51b411f140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 -- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51b406a6e0 msgr2=0x7f51b411ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51b406a6e0 0x7f51b411ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient: _finish_hunting 0 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 monclient: found mon.noname-b 2024-09-15T01:55:18.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.775+0000 7f51ba645640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a4009450 con 0x7f51b4137240 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51ba645640 1 -- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 msgr2=0x7f51b411f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51ba645640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00cee0 tx=0x7f51ac007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ac00e610 con 0x7f51b4137240 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: dump: 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.790 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ac00e770 con 0x7f51b4137240 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 5 monclient: authenticate success, global_id 5419 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 10 monclient: _renew_subs 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b4121ec0 con 0x7f51b4137240 2024-09-15T01:55:18.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ac00eb30 con 0x7f51b4137240 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51a9ffb640 10 monclient: dump: 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:18.792 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51abfff640 4 set_mon_vals no callback set 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 10 monclient: _renew_subs 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.777+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b4120f60 con 0x7f51b4137240 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f51ac009030 con 0x7f51b4137240 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51bc0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51bc0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51bc0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b4121150 con 0x7f51b4137240 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51a9ffb640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 0x7f519803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5555+0+0 (secure 0 0 0) 0x7f51ac012070 con 0x7f51b4137240 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51b9e44640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 0x7f519803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.778+0000 7f51b9e44640 10 monclient: get_auth_request con 0x7f51980386f0 auth_method 0 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.781+0000 7f51b9e44640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 0x7f519803aba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f51b0006fd0 tx=0x7f51b0009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:18.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.783+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f51ac01d910 con 0x7f51b4137240 2024-09-15T01:55:18.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.783+0000 7f51a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:18.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.783+0000 7f51a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:18.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.947+0000 7f51bc0a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:18.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.947+0000 7f51bc0a8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:18.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.947+0000 7f51bc0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:18.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:18.947+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7f51b41222c0 con 0x7f51b4137240 2024-09-15T01:55:19.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.577+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7f51ac014330 con 0x7f51b4137240 2024-09-15T01:55:19.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.577+0000 7f51a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:19.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.577+0000 7f51a9ffb640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-15T01:55:19.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.679+0000 7f51bc0a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:19.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.679+0000 7f51bc0a8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:19.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.679+0000 7f51bc0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.679+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7f51b404bf60 con 0x7f51b4137240 2024-09-15T01:55:19.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.680+0000 7f51a9ffb640 1 -- 172.21.15.67:0/2029053198 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v91) v1 ==== 126+0+0 (secure 0 0 0) 0x7f51ac01d310 con 0x7f51b4137240 2024-09-15T01:55:19.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.680+0000 7f51a9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T01:55:19.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.680+0000 7f51a9ffb640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-15T01:55:19.682 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'foo' already exists 2024-09-15T01:55:19.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 msgr2=0x7f519803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:19.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 0x7f519803aba0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f51b0006fd0 tx=0x7f51b0009060 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 10 monclient: shutdown 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 msgr2=0x7f51b411f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00cee0 tx=0x7f51ac007870 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 shutdown_connections 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51980386f0 0x7f519803aba0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51b4137240 0x7f51b411f680 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51b40a3850 0x7f51b411f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 --2- 172.21.15.67:0/2029053198 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51b406a6e0 0x7f51b411ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 >> 172.21.15.67:0/2029053198 conn(0x7f51b4070ff0 msgr2=0x7f51b40a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 shutdown_connections 2024-09-15T01:55:19.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.682+0000 7f51bc0a8640 1 -- 172.21.15.67:0/2029053198 wait complete. 2024-09-15T01:55:19.693 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.768+0000 7f04ba26f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.768+0000 7f04ba26f640 10 monclient: build_initial_monmap 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.768+0000 7f04ba26f640 10 monclient: monmap: 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:19.769990+0000 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:19.769990+0000 2024-09-15T01:55:19.771 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:19.772 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.769+0000 7f04ba26f640 1 Processor -- start 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.769+0000 7f04ba26f640 1 -- start start 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.769+0000 7f04ba26f640 10 monclient: init 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T01:55:19.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04b40a3830 0x7f04b406b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-c con 0x7f04b40a3830 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04b406e2b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04b409e220 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b4069c60 con 0x7f04b409e220 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40a3c00 con 0x7f04b406e2b0 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b4069650 con 0x7f04b40a3830 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04ba26f640 10 monclient(hunting): authenticate will time out at 952.239193s 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04b40a3830 0x7f04b406b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04b406e2b0 0x7f04b409db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60594/0 (socket says 172.21.15.67:60594) 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b37fe640 1 -- 172.21.15.67:0/3933436364 learned_addr learned my addr 172.21.15.67:0/3933436364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): get_auth_request con 0x7f04b40a3830 auth_method 0 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient(hunting): handle_auth_done global_id 5422 payload 274 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 1 -- 172.21.15.67:0/3933436364 >> v1:172.21.15.67:6789/0 conn(0x7f04b406e2b0 legacy=0x7f04b409db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 1 -- 172.21.15.67:0/3933436364 >> v1:172.21.15.17:6789/0 conn(0x7f04b409e220 legacy=0x7f04b4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 1 monclient: found mon.noname-c 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:19.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.770+0000 7f04b3fff640 1 -- 172.21.15.67:0/3933436364 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b409f500 con 0x7f04b40a3830 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b3fff640 1 -- 172.21.15.67:0/3933436364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 msgr2=0x7f04b406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b3fff640 1 --2- 172.21.15.67:0/3933436364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b406b9b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009850 tx=0x7f04a40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e91b871678195140 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 1 -- 172.21.15.67:0/3933436364 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4032440 con 0x7f04b40a3830 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: dump: 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 1 -- 172.21.15.67:0/3933436364 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a40325a0 con 0x7f04b40a3830 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04ba26f640 5 monclient: authenticate success, global_id 5422 2024-09-15T01:55:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04ba26f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04ba26f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 1 -- 172.21.15.67:0/3933436364 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4032940 con 0x7f04b40a3830 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04ba26f640 4 set_mon_vals no callback set 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.773+0000 7f04b2ffd640 10 monclient: dump: 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 10 monclient: shutdown 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 -- 172.21.15.67:0/3933436364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 msgr2=0x7f04b406b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 --2- 172.21.15.67:0/3933436364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b406b9b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009850 tx=0x7f04a40090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 -- 172.21.15.67:0/3933436364 shutdown_connections 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 --2- 172.21.15.67:0/3933436364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b406b9b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 -- 172.21.15.67:0/3933436364 >> 172.21.15.67:0/3933436364 conn(0x7f04b4070ff0 msgr2=0x7f04b40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 -- 172.21.15.67:0/3933436364 shutdown_connections 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 -- 172.21.15.67:0/3933436364 wait complete. 2024-09-15T01:55:19.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 10 monclient: build_initial_monmap 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 10 monclient: monmap: 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.774+0000 7f04ba26f640 1 Processor -- start 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 -- start start 2024-09-15T01:55:19.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient: init 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 0x7f04b413e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-c con 0x7f04b406e2b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04b409e220 0x7f04b413db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-a con 0x7f04b409e220 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): picked mon.noname-b con 0x7f04b40a3830 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b4122f10 con 0x7f04b409e220 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b4099e80 con 0x7f04b40a3830 2024-09-15T01:55:19.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40a2dc0 con 0x7f04b406e2b0 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04ba26f640 10 monclient(hunting): authenticate will time out at 952.244649s 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b3fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 msgr2=0x7f04b413e4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 0x7f04b413e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35246/0 (socket says 172.21.15.67:35246) 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 1 -- 172.21.15.67:0/1760507459 learned_addr learned my addr 172.21.15.67:0/1760507459 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): get_auth_request con 0x7f04b40a3830 auth_method 0 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:19.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b880c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b37fe640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04b409e220 0x7f04b413db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.775+0000 7f04b37fe640 10 monclient(hunting): get_auth_request con 0x7f04b409e220 auth_method 0 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 10 monclient(hunting): handle_auth_done global_id 5425 payload 824 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 -- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 msgr2=0x7f04b413e4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 0x7f04b413e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 -- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04b409e220 msgr2=0x7f04b413db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04b409e220 0x7f04b413db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 monclient: found mon.noname-b 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.776+0000 7f04b880c640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4009390 con 0x7f04b40a3830 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04b880c640 1 -- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 msgr2=0x7f04b4141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04b880c640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c890 tx=0x7f04ac002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00ce80 con 0x7f04b40a3830 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: dump: 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:19.790 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: _finish_auth 0 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac00aa00 con 0x7f04b40a3830 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 5 monclient: authenticate success, global_id 5425 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 10 monclient: _renew_subs 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b413f0b0 con 0x7f04b40a3830 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00adc0 con 0x7f04b40a3830 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04abfff640 10 monclient: dump: 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:19.792 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04b27fc640 4 set_mon_vals no callback set 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 10 monclient: _renew_subs 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.778+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b413e0d0 con 0x7f04b40a3830 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f04ac012630 con 0x7f04b40a3830 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04ba26f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04ba26f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04ba26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b404bf60 con 0x7f04b40a3830 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04abfff640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 0x7f049403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5582+0+0 (secure 0 0 0) 0x7f04ac04c160 con 0x7f04b40a3830 2024-09-15T01:55:19.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04b3fff640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 0x7f049403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:19.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.779+0000 7f04b3fff640 10 monclient: get_auth_request con 0x7f0494038740 auth_method 0 2024-09-15T01:55:19.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.782+0000 7f04b3fff640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 0x7f049403abf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f04a4031000 tx=0x7f04a4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:19.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.783+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f04ac022d70 con 0x7f04b40a3830 2024-09-15T01:55:19.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.784+0000 7f04abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:19.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.784+0000 7f04abfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:19.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.950+0000 7f04ba26f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.950+0000 7f04ba26f640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.950+0000 7f04ba26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:19.950+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f04b406c070 con 0x7f04b40a3830 2024-09-15T01:55:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.577+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 204+0+0 (secure 0 0 0) 0x7f04ac0160c0 con 0x7f04b40a3830 2024-09-15T01:55:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.577+0000 7f04abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.577+0000 7f04abfff640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T01:55:20.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.679+0000 7f04ba26f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:20.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.679+0000 7f04ba26f640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:20.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.679+0000 7f04ba26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.679+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f04b409f980 con 0x7f04b40a3830 2024-09-15T01:55:20.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.681+0000 7f04abfff640 1 -- 172.21.15.67:0/1760507459 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 214+0+0 (secure 0 0 0) 0x7f04ac01c030 con 0x7f04b40a3830 2024-09-15T01:55:20.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.681+0000 7f04abfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T01:55:20.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.681+0000 7f04abfff640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T01:55:20.683 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T01:55:20.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 msgr2=0x7f049403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:20.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 0x7f049403abf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f04a4031000 tx=0x7f04a4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:20.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 10 monclient: shutdown 2024-09-15T01:55:20.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:20.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 msgr2=0x7f04b4141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c890 tx=0x7f04ac002640 comp rx=0 tx=0).stop 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 shutdown_connections 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0494038740 0x7f049403abf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04b409e220 0x7f04b413db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b40a3830 0x7f04b4141a30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 --2- 172.21.15.67:0/1760507459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f04b406e2b0 0x7f04b413e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.682+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 >> 172.21.15.67:0/1760507459 conn(0x7f04b4070ff0 msgr2=0x7f04b406d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.683+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 shutdown_connections 2024-09-15T01:55:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.683+0000 7f04ba26f640 1 -- 172.21.15.67:0/1760507459 wait complete. 2024-09-15T01:55:20.694 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-15T01:55:20.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:20.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 10 monclient: build_initial_monmap 2024-09-15T01:55:20.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 10 monclient: monmap: 2024-09-15T01:55:20.773 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:20.772217+0000 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:20.772217+0000 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:20.774 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 1 Processor -- start 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 1 -- start start 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.771+0000 7f63c9f5d640 10 monclient: init 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63c4098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63c40a0370 0x7f63c4098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63c4098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4098ef0 con 0x7f63c4098a50 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40a0b80 con 0x7f63c4098140 2024-09-15T01:55:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c41328f0 con 0x7f63c40a0370 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c9f5d640 10 monclient(hunting): authenticate will time out at 953.241349s 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63c4098140 0x7f63c41321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48324/0 (socket says 172.21.15.67:48324) 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c37fe640 1 -- 172.21.15.67:0/2041027661 learned_addr learned my addr 172.21.15.67:0/2041027661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/2041027661 >> v2:172.21.15.17:3300/0 conn(0x7f63c40a0370 0x7f63c4098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): get_auth_request con 0x7f63c40a0370 auth_method 0 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:20.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.772+0000 7f63c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 10 monclient(hunting): handle_auth_done global_id 44778 payload 274 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 1 -- 172.21.15.67:0/2041027661 >> v1:172.21.15.67:6789/0 conn(0x7f63c4098140 legacy=0x7f63c41321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 1 -- 172.21.15.67:0/2041027661 >> v1:172.21.15.17:6789/0 conn(0x7f63c4098a50 legacy=0x7f63c41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 1 monclient: found mon.noname-a 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:20.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.773+0000 7f63c2ffd640 1 -- 172.21.15.67:0/2041027661 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40a00b0 con 0x7f63c40a0370 2024-09-15T01:55:20.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c2ffd640 1 -- 172.21.15.67:0/2041027661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c40a0370 msgr2=0x7f63c4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/2041027661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c40a0370 0x7f63c4098510 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009850 tx=0x7f63b80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f877ae7c90e5f67d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 1 -- 172.21.15.67:0/2041027661 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8032440 con 0x7f63c40a0370 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: dump: 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.785 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 1 -- 172.21.15.67:0/2041027661 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b80325a0 con 0x7f63c40a0370 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 5 monclient: authenticate success, global_id 44778 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 4 set_mon_vals no callback set 2024-09-15T01:55:20.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 1 -- 172.21.15.67:0/2041027661 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8032940 con 0x7f63c40a0370 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c27fc640 10 monclient: dump: 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 10 monclient: shutdown 2024-09-15T01:55:20.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 1 -- 172.21.15.67:0/2041027661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c40a0370 msgr2=0x7f63c4098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.776+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/2041027661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c40a0370 0x7f63c4098510 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009850 tx=0x7f63b80090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 -- 172.21.15.67:0/2041027661 shutdown_connections 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/2041027661 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c40a0370 0x7f63c4098510 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 -- 172.21.15.67:0/2041027661 >> 172.21.15.67:0/2041027661 conn(0x7f63c40a5780 msgr2=0x7f63c40a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 -- 172.21.15.67:0/2041027661 shutdown_connections 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 -- 172.21.15.67:0/2041027661 wait complete. 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 10 monclient: build_initial_monmap 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 10 monclient: monmap: 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.777+0000 7f63c9f5d640 1 Processor -- start 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 -- start start 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient: init 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:20.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f63c4098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 0x7f63c412e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f63c4098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 0x7f63c412e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f63c40a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4130a00 con 0x7f63c4098a50 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c409bda0 con 0x7f63c4098140 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4130440 con 0x7f63c40a0370 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c9f5d640 10 monclient(hunting): authenticate will time out at 953.247470s 2024-09-15T01:55:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53406/0 (socket says 172.21.15.67:53406) 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 -- 172.21.15.67:0/139157427 learned_addr learned my addr 172.21.15.67:0/139157427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): get_auth_request con 0x7f63c4098140 auth_method 0 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c3fff640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 msgr2=0x7f63c412e550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c3fff640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 0x7f63c412e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 0x7f63c412e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient(hunting): handle_auth_done global_id 5428 payload 824 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 msgr2=0x7f63c412e550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 0x7f63c412e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 msgr2=0x7f63c412e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 0x7f63c412e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c37fe640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8009390 con 0x7f63c4098140 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c2ffd640 10 monclient: get_auth_request con 0x7f63c4098a50 auth_method 0 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.778+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 0x7f63c412e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c37fe640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 msgr2=0x7f63c409b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:20.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c37fe640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cca0 tx=0x7f63b40053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400aa60 con 0x7f63c4098140 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: dump: 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:20.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 5 monclient: authenticate success, global_id 5428 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b400abe0 con 0x7f63c4098140 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 10 monclient: _renew_subs 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4130720 con 0x7f63c4098140 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b40070f0 con 0x7f63c4098140 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:20.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63ab7fe640 10 monclient: dump: 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 10 monclient: _renew_subs 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c1ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.781+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4131100 con 0x7f63c4098140 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f63b4007520 con 0x7f63c4098140 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c9f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c9f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c9f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4130c20 con 0x7f63c4098140 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63ab7fe640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 0x7f63b003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f63b404c3b0 con 0x7f63c4098140 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 0x7f63b003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.782+0000 7f63c2ffd640 10 monclient: get_auth_request con 0x7f63b00386f0 auth_method 0 2024-09-15T01:55:20.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.785+0000 7f63c2ffd640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 0x7f63b003aba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63b8031000 tx=0x7f63b8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:20.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.786+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f63c4130c20 con 0x7f63c4098140 2024-09-15T01:55:20.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.786+0000 7f63ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:20.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.786+0000 7f63ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:20.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.955+0000 7f63c9f5d640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:20.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.955+0000 7f63c9f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:20.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.955+0000 7f63c9f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:20.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:20.955+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f63c404bf60 con 0x7f63c4098140 2024-09-15T01:55:21.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.584+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 212+0+0 (secure 0 0 0) 0x7f63b401a360 con 0x7f63c4098140 2024-09-15T01:55:21.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.584+0000 7f63ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:21.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.584+0000 7f63ab7fe640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T01:55:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.690+0000 7f63c9f5d640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.690+0000 7f63c9f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.690+0000 7f63c9f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.690+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f63c40713d0 con 0x7f63c4098140 2024-09-15T01:55:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.692+0000 7f63ab7fe640 1 -- 172.21.15.67:0/139157427 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 211+0+0 (secure 0 0 0) 0x7f63b4010af0 con 0x7f63c4098140 2024-09-15T01:55:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.692+0000 7f63ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T01:55:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.692+0000 7f63ab7fe640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T01:55:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 msgr2=0x7f63b003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 0x7f63b003aba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63b8031000 tx=0x7f63b8032000 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 10 monclient: shutdown 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 msgr2=0x7f63c409b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cca0 tx=0x7f63b40053f0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 shutdown_connections 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b00386f0 0x7f63b003aba0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63c4098a50 0x7f63c412e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63c40a0370 0x7f63c412e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 --2- 172.21.15.67:0/139157427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c4098140 0x7f63c409b480 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 >> 172.21.15.67:0/139157427 conn(0x7f63c40a5780 msgr2=0x7f63c412ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:21.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 shutdown_connections 2024-09-15T01:55:21.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.694+0000 7f63c9f5d640 1 -- 172.21.15.67:0/139157427 wait complete. 2024-09-15T01:55:21.706 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 10 monclient: get_monmap_and_config 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 10 monclient: build_initial_monmap 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 10 monclient: monmap: 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:21.783 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:21.782185+0000 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:21.782185+0000 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:21.784 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 1 Processor -- start 2024-09-15T01:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 1 -- start start 2024-09-15T01:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.781+0000 7f9b47edd640 10 monclient: init 2024-09-15T01:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b40068dc0 0x7f9b40132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-c con 0x7f9b40068dc0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b40069190 0x7f9b40069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-a con 0x7f9b40069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b4012e960 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b40096c10 con 0x7f9b4012e960 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b400a14d0 con 0x7f9b40069190 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4009ff20 con 0x7f9b40068dc0 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b47edd640 10 monclient(hunting): authenticate will time out at 954.251341s 2024-09-15T01:55:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b40068dc0 0x7f9b40132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b4647a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b4012e960 0x7f9b4012ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48340/0 (socket says 172.21.15.67:48340) 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b4647a640 1 -- 172.21.15.67:0/1396839644 learned_addr learned my addr 172.21.15.67:0/1396839644 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): get_auth_request con 0x7f9b40068dc0 auth_method 0 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient(hunting): handle_auth_done global_id 5431 payload 274 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 -- 172.21.15.67:0/1396839644 >> v2:172.21.15.17:3300/0 conn(0x7f9b40069190 msgr2=0x7f9b40069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45478640 1 --2- 172.21.15.67:0/1396839644 >> v2:172.21.15.17:3300/0 conn(0x7f9b40069190 0x7f9b40069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 --2- 172.21.15.67:0/1396839644 >> v2:172.21.15.17:3300/0 conn(0x7f9b40069190 0x7f9b40069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 -- 172.21.15.67:0/1396839644 >> v1:172.21.15.67:6789/0 conn(0x7f9b4012e960 legacy=0x7f9b4012ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient: _finish_hunting 0 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 monclient: found mon.noname-c 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.782+0000 7f9b45c79640 1 -- 172.21.15.67:0/1396839644 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b400a0180 con 0x7f9b40068dc0 2024-09-15T01:55:21.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b45c79640 1 -- 172.21.15.67:0/1396839644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 msgr2=0x7f9b40132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:21.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b45c79640 1 --2- 172.21.15.67:0/1396839644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b40132250 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9b4006a3f0 tx=0x7f9b3c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eda04dcbb8342a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:21.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 1 -- 172.21.15.67:0/1396839644 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c032440 con 0x7f9b40068dc0 2024-09-15T01:55:21.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: dump: 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:21.794 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: _finish_auth 0 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 1 -- 172.21.15.67:0/1396839644 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c0325a0 con 0x7f9b40068dc0 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 1 -- 172.21.15.67:0/1396839644 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c032940 con 0x7f9b40068dc0 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b47edd640 5 monclient: authenticate success, global_id 5431 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b47edd640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b47edd640 4 set_mon_vals no callback set 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.785+0000 7f9b44c77640 10 monclient: dump: 2024-09-15T01:55:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 10 monclient: shutdown 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 -- 172.21.15.67:0/1396839644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 msgr2=0x7f9b40132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 --2- 172.21.15.67:0/1396839644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b40132250 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9b4006a3f0 tx=0x7f9b3c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 -- 172.21.15.67:0/1396839644 shutdown_connections 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 --2- 172.21.15.67:0/1396839644 >> v2:172.21.15.17:3300/0 conn(0x7f9b40069190 0x7f9b40069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 --2- 172.21.15.67:0/1396839644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b40132250 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 -- 172.21.15.67:0/1396839644 >> 172.21.15.67:0/1396839644 conn(0x7f9b400711b0 msgr2=0x7f9b400a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 -- 172.21.15.67:0/1396839644 shutdown_connections 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.786+0000 7f9b47edd640 1 -- 172.21.15.67:0/1396839644 wait complete. 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient: build_initial_monmap 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient: monmap: 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:21.797 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 Processor -- start 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 -- start start 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient: init 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-b con 0x7f9b40068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 0x7f9b40097ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-c con 0x7f9b40069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b4012e960 0x7f9b40098400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4012e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400a0900 con 0x7f9b4012e960 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4012e780 con 0x7f9b40068dc0 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40134d30 con 0x7f9b40069190 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b47edd640 10 monclient(hunting): authenticate will time out at 954.256847s 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45478640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 msgr2=0x7f9b40097ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53424/0 (socket says 172.21.15.67:53424) 2024-09-15T01:55:21.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 1 -- 172.21.15.67:0/759306017 learned_addr learned my addr 172.21.15.67:0/759306017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 10 monclient(hunting): get_auth_request con 0x7f9b40068dc0 auth_method 0 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.787+0000 7f9b45478640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 0x7f9b40097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b4647a640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b4012e960 0x7f9b40098400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b4647a640 10 monclient(hunting): get_auth_request con 0x7f9b4012e960 auth_method 0 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b4647a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b4647a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:21.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b4647a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient(hunting): handle_auth_done global_id 5434 payload 824 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 -- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 msgr2=0x7f9b40097ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 0x7f9b40097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 -- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b4012e960 msgr2=0x7f9b40098400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b4012e960 0x7f9b40098400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient: _finish_hunting 0 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 monclient: found mon.noname-b 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.788+0000 7f9b45c79640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c009390 con 0x7f9b40068dc0 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b45c79640 1 -- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 msgr2=0x7f9b4012d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b45c79640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02a630 tx=0x7f9b3c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c000d10 con 0x7f9b40068dc0 2024-09-15T01:55:21.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: dump: 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b47edd640 5 monclient: authenticate success, global_id 5434 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c031c70 con 0x7f9b40068dc0 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c03c400 con 0x7f9b40068dc0 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b47edd640 10 monclient: _renew_subs 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b47edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b400999c0 con 0x7f9b40068dc0 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b35ffb640 10 monclient: dump: 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.803 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.790+0000 7f9b37fff640 4 set_mon_vals no callback set 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.791+0000 7f9b47edd640 10 monclient: _renew_subs 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.791+0000 7f9b47edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.791+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b40141fc0 con 0x7f9b40068dc0 2024-09-15T01:55:21.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.791+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b3c03c870 con 0x7f9b40068dc0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b47edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b47edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b47edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4009bb60 con 0x7f9b40068dc0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b35ffb640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 0x7f9b2403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f9b3c06c8b0 con 0x7f9b40068dc0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b45478640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 0x7f9b2403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.792+0000 7f9b45478640 10 monclient: get_auth_request con 0x7f9b24038740 auth_method 0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.794+0000 7f9b45478640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 0x7f9b2403abf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9b4009a520 tx=0x7f9b30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.796+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9b3c043350 con 0x7f9b40068dc0 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.796+0000 7f9b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.796+0000 7f9b35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:21.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.963+0000 7f9b47edd640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:21.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.963+0000 7f9b47edd640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:21.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.963+0000 7f9b47edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:21.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:21.963+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f9b40069ad0 con 0x7f9b40068dc0 2024-09-15T01:55:22.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.203+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 165+0+0 (secure 0 0 0) 0x7f9b3c037020 con 0x7f9b40068dc0 2024-09-15T01:55:22.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.203+0000 7f9b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:22.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.203+0000 7f9b35ffb640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T01:55:22.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.306+0000 7f9b47edd640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:22.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.306+0000 7f9b47edd640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.306+0000 7f9b47edd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.306+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f9b40069ad0 con 0x7f9b40068dc0 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.307+0000 7f9b35ffb640 1 -- 172.21.15.67:0/759306017 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 175+0+0 (secure 0 0 0) 0x7f9b3c042300 con 0x7f9b40068dc0 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.307+0000 7f9b35ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.307+0000 7f9b35ffb640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T01:55:22.309 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 msgr2=0x7f9b2403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 0x7f9b2403abf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9b4009a520 tx=0x7f9b30008040 comp rx=0 tx=0).stop 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 10 monclient: shutdown 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 msgr2=0x7f9b4012d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02a630 tx=0x7f9b3c032000 comp rx=0 tx=0).stop 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 shutdown_connections 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b24038740 0x7f9b2403abf0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b4012e960 0x7f9b40098400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b40069190 0x7f9b40097ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 --2- 172.21.15.67:0/759306017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b40068dc0 0x7f9b4012d950 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 >> 172.21.15.67:0/759306017 conn(0x7f9b400711b0 msgr2=0x7f9b40130cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:22.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 shutdown_connections 2024-09-15T01:55:22.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.309+0000 7f9b47edd640 1 -- 172.21.15.67:0/759306017 wait complete. 2024-09-15T01:55:22.321 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd find osd.7 2024-09-15T01:55:22.321 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep host 2024-09-15T01:55:22.321 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep foo 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.393+0000 7f9adf4c0640 10 monclient: get_monmap_and_config 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.393+0000 7f9adf4c0640 10 monclient: build_initial_monmap 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.393+0000 7f9adf4c0640 10 monclient: monmap: 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:22.394740+0000 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:22.394740+0000 2024-09-15T01:55:22.396 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:22.397 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 1 Processor -- start 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 1 -- start start 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient: init 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9ad8130e00 0x7f9ad8132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad8130e00 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ad81311d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ad81315f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.394+0000 7f9adf4c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad809ff20 con 0x7f9ad81315f0 2024-09-15T01:55:22.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adf4c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad8069be0 con 0x7f9ad81311d0 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adf4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adf4c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad81320b0 con 0x7f9ad8130e00 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adf4c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adf4c0640 10 monclient(hunting): authenticate will time out at 954.864079s 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adca5b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9ad81311d0 0x7f9ad8068dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48352/0 (socket says 172.21.15.67:48352) 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9adca5b640 1 -- 172.21.15.67:0/3001735386 learned_addr learned my addr 172.21.15.67:0/3001735386 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 1 --2- 172.21.15.67:0/3001735386 >> v2:172.21.15.17:3300/0 conn(0x7f9ad8130e00 0x7f9ad8132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): get_auth_request con 0x7f9ad8130e00 auth_method 0 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient(hunting): handle_auth_done global_id 44784 payload 274 2024-09-15T01:55:22.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 1 -- 172.21.15.67:0/3001735386 >> v1:172.21.15.67:6789/0 conn(0x7f9ad81311d0 legacy=0x7f9ad8068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:22.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 1 -- 172.21.15.67:0/3001735386 >> v1:172.21.15.17:6789/0 conn(0x7f9ad81315f0 legacy=0x7f9ad80694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:22.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:22.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 1 monclient: found mon.noname-a 2024-09-15T01:55:22.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:22.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:22.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.395+0000 7f9add25c640 1 -- 172.21.15.67:0/3001735386 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad80a00f0 con 0x7f9ad8130e00 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9add25c640 1 -- 172.21.15.67:0/3001735386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 msgr2=0x7f9ad8132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9add25c640 1 --2- 172.21.15.67:0/3001735386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8132430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009850 tx=0x7f9ad40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f590b6a40a93f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 1 -- 172.21.15.67:0/3001735386 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4032440 con 0x7f9ad8130e00 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: dump: 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.410 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 1 -- 172.21.15.67:0/3001735386 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad40325a0 con 0x7f9ad8130e00 2024-09-15T01:55:22.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 1 -- 172.21.15.67:0/3001735386 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4032940 con 0x7f9ad8130e00 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.398+0000 7f9acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9acffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9acffff640 10 monclient: dump: 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 5 monclient: authenticate success, global_id 44784 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 4 set_mon_vals no callback set 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 10 monclient: shutdown 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 -- 172.21.15.67:0/3001735386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 msgr2=0x7f9ad8132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/3001735386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8132430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009850 tx=0x7f9ad40090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 -- 172.21.15.67:0/3001735386 shutdown_connections 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/3001735386 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8132430 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 -- 172.21.15.67:0/3001735386 >> 172.21.15.67:0/3001735386 conn(0x7f9ad80710f0 msgr2=0x7f9ad80a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 -- 172.21.15.67:0/3001735386 shutdown_connections 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.399+0000 7f9adf4c0640 1 -- 172.21.15.67:0/3001735386 wait complete. 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient: build_initial_monmap 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient: monmap: 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:22.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 1 Processor -- start 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 1 -- start start 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient: init 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:22.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8098e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad8130e00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 0x7f9ad80993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.400+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad81311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad81315f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:22.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad809e860 con 0x7f9ad8130e00 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad809af80 con 0x7f9ad81315f0 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80a0820 con 0x7f9ad81311d0 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adf4c0640 10 monclient(hunting): authenticate will time out at 954.870037s 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adca5b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 msgr2=0x7f9ad80993d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:22.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adca5b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 0x7f9ad80993d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53436/0 (socket says 172.21.15.67:53436) 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 -- 172.21.15.67:0/2536461239 learned_addr learned my addr 172.21.15.67:0/2536461239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): get_auth_request con 0x7f9ad81315f0 auth_method 0 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 10 monclient(hunting): get_auth_request con 0x7f9ad8130e00 auth_method 0 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient(hunting): handle_auth_done global_id 5437 payload 824 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 -- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 msgr2=0x7f9ad80993d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 0x7f9ad80993d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 -- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 msgr2=0x7f9ad8098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient: _finish_hunting 0 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 monclient: found mon.noname-b 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9adda5d640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4009390 con 0x7f9ad81315f0 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.401+0000 7f9add25c640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.403+0000 7f9adda5d640 1 -- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 msgr2=0x7f9ad8099910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.403+0000 7f9adda5d640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c890 tx=0x7f9ad0002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.403+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:55:22.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000ce80 con 0x7f9ad81315f0 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: dump: 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad000aa00 con 0x7f9ad81315f0 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000ae00 con 0x7f9ad81315f0 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 5 monclient: authenticate success, global_id 5437 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 10 monclient: _renew_subs 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad811c380 con 0x7f9ad81315f0 2024-09-15T01:55:22.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acf7fe640 4 set_mon_vals no callback set 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9acd7fa640 10 monclient: dump: 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.423 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 10 monclient: _renew_subs 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.405+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad812f240 con 0x7f9ad81315f0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.406+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ad0012630 con 0x7f9ad81315f0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.406+0000 7f9adf4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.406+0000 7f9adf4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.406+0000 7f9adf4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.406+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad809b6d0 con 0x7f9ad81315f0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.407+0000 7f9acd7fa640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 0x7f9abc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.407+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6074+0+0 (secure 0 0 0) 0x7f9ad004c340 con 0x7f9ad81315f0 2024-09-15T01:55:22.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.407+0000 7f9add25c640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 0x7f9abc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.407+0000 7f9add25c640 10 monclient: get_auth_request con 0x7f9abc038740 auth_method 0 2024-09-15T01:55:22.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.409+0000 7f9add25c640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 0x7f9abc03abf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80a30f0 tx=0x7f9ad4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.411+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ad0022d70 con 0x7f9ad81315f0 2024-09-15T01:55:22.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.411+0000 7f9acd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.411+0000 7f9acd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:22.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9adf4c0640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9adf4c0640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9adf4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f9ad804bf60 con 0x7f9ad81315f0 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (secure 0 0 0) 0x7f9ad0016030 con 0x7f9ad81315f0 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9acd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.576+0000 7f9acd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:22.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9adf4c0640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9adf4c0640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9adf4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f9ad80008d0 con 0x7f9ad81315f0 2024-09-15T01:55:22.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9acd7fa640 1 -- 172.21.15.67:0/2536461239 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (secure 0 0 0) 0x7f9ad0029100 con 0x7f9ad81315f0 2024-09-15T01:55:22.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9acd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-15T01:55:22.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.675+0000 7f9acd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:22.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 msgr2=0x7f9abc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 0x7f9abc03abf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80a30f0 tx=0x7f9ad4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 10 monclient: shutdown 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 msgr2=0x7f9ad8099910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c890 tx=0x7f9ad0002640 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 shutdown_connections 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9abc038740 0x7f9abc03abf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad81315f0 0x7f9ad8099910 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad81311d0 0x7f9ad80993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 --2- 172.21.15.67:0/2536461239 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad8130e00 0x7f9ad8098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 >> 172.21.15.67:0/2536461239 conn(0x7f9ad80710f0 msgr2=0x7f9ad809edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:22.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 shutdown_connections 2024-09-15T01:55:22.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.677+0000 7f9adf4c0640 1 -- 172.21.15.67:0/2536461239 wait complete. 2024-09-15T01:55:22.688 INFO:tasks.workunit.client.0.smithi067.stdout: "host": "foo", 2024-09-15T01:55:22.688 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.7 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.762+0000 7f85b939d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.762+0000 7f85b939d640 10 monclient: build_initial_monmap 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.762+0000 7f85b939d640 10 monclient: monmap: 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:22.763991+0000 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:22.763991+0000 2024-09-15T01:55:22.765 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:22.766 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 Processor -- start 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 -- start start 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient: init 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85b4068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85b4069190 0x7f85b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-c con 0x7f85b4069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 0x7f85b4139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-b con 0x7f85b4137150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b4096c10 con 0x7f85b4068dc0 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b40a14d0 con 0x7f85b4069190 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4132960 con 0x7f85b4137150 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:22.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.763+0000 7f85b939d640 10 monclient(hunting): authenticate will time out at 955.232960s 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85b4069190 0x7f85b4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85b4068dc0 0x7f85b4132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48368/0 (socket says 172.21.15.67:48368) 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b2ffd640 1 -- 172.21.15.67:0/1151635826 learned_addr learned my addr 172.21.15.67:0/1151635826 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b37fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 msgr2=0x7f85b4139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): get_auth_request con 0x7f85b4069190 auth_method 0 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b37fe640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 0x7f85b4139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient(hunting): handle_auth_done global_id 5440 payload 274 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 -- 172.21.15.67:0/1151635826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 msgr2=0x7f85b4139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 0x7f85b4139530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 -- 172.21.15.67:0/1151635826 >> v1:172.21.15.67:6789/0 conn(0x7f85b4068dc0 legacy=0x7f85b4132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 monclient: found mon.noname-c 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.764+0000 7f85b27fc640 1 -- 172.21.15.67:0/1151635826 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b409ff20 con 0x7f85b4069190 2024-09-15T01:55:22.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b27fc640 1 -- 172.21.15.67:0/1151635826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4069190 msgr2=0x7f85b4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:22.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b27fc640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4069190 0x7f85b4069590 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009850 tx=0x7f85a80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42a3f85d2b7f303 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 1 -- 172.21.15.67:0/1151635826 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8032440 con 0x7f85b4069190 2024-09-15T01:55:22.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: dump: 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 1 -- 172.21.15.67:0/1151635826 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a80325a0 con 0x7f85b4069190 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b939d640 5 monclient: authenticate success, global_id 5440 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 1 -- 172.21.15.67:0/1151635826 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8032980 con 0x7f85b4069190 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b939d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b939d640 4 set_mon_vals no callback set 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.767+0000 7f85b1ffb640 10 monclient: dump: 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.779 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 10 monclient: shutdown 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 -- 172.21.15.67:0/1151635826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4069190 msgr2=0x7f85b4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4069190 0x7f85b4069590 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009850 tx=0x7f85a80090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 -- 172.21.15.67:0/1151635826 shutdown_connections 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4137150 0x7f85b4139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 --2- 172.21.15.67:0/1151635826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4069190 0x7f85b4069590 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 -- 172.21.15.67:0/1151635826 >> 172.21.15.67:0/1151635826 conn(0x7f85b40711b0 msgr2=0x7f85b40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 -- 172.21.15.67:0/1151635826 shutdown_connections 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 1 -- 172.21.15.67:0/1151635826 wait complete. 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 10 monclient: build_initial_monmap 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.768+0000 7f85b939d640 10 monclient: monmap: 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:22.781 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 Processor -- start 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 -- start start 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient: init 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 0x7f85b41415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-c con 0x7f85b4068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b4069190 0x7f85b4141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-a con 0x7f85b4069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): picked mon.noname-b con 0x7f85b4137150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b409ba70 con 0x7f85b4069190 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40a0890 con 0x7f85b4137150 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b409a550 con 0x7f85b4068dc0 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b939d640 10 monclient(hunting): authenticate will time out at 955.238770s 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b2ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 msgr2=0x7f85b41415c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b2ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 0x7f85b41415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53460/0 (socket says 172.21.15.67:53460) 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 1 -- 172.21.15.67:0/3817364173 learned_addr learned my addr 172.21.15.67:0/3817364173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 10 monclient(hunting): get_auth_request con 0x7f85b4137150 auth_method 0 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b27fc640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b4069190 0x7f85b4141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b27fc640 10 monclient(hunting): get_auth_request con 0x7f85b4069190 auth_method 0 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:22.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.769+0000 7f85b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient(hunting): handle_auth_done global_id 5443 payload 824 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 -- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 msgr2=0x7f85b41415c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 0x7f85b41415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 -- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b4069190 msgr2=0x7f85b4141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b4069190 0x7f85b4141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:22.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.770+0000 7f85b37fe640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a8009390 con 0x7f85b4137150 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b37fe640 1 -- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 msgr2=0x7f85b4142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b37fe640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bb30 tx=0x7f85ac0097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac00cbc0 con 0x7f85b4137150 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: dump: 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: _finish_auth 0 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac00cd20 con 0x7f85b4137150 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 5 monclient: authenticate success, global_id 5443 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:22.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac006150 con 0x7f85b4137150 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 10 monclient: _renew_subs 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b409b790 con 0x7f85b4137150 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f859affd640 10 monclient: dump: 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:22.788 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b17fa640 4 set_mon_vals no callback set 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 10 monclient: _renew_subs 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.772+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b411bdc0 con 0x7f85b4137150 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85ac012020 con 0x7f85b4137150 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f85b939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f85b939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f85b939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b404bf60 con 0x7f85b4137150 2024-09-15T01:55:22.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f859affd640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 0x7f85a003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.773+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6074+0+0 (secure 0 0 0) 0x7f85ac04bea0 con 0x7f85b4137150 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.774+0000 7f85b2ffd640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 0x7f85a003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.774+0000 7f85b2ffd640 10 monclient: get_auth_request con 0x7f85a00386f0 auth_method 0 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.776+0000 7f85b2ffd640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 0x7f85a003aba0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f85a40047d0 tx=0x7f85a4007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.778+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85ac01dd50 con 0x7f85b4137150 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.778+0000 7f859affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.778+0000 7f859affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:22.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.949+0000 7f85b939d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:22.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.949+0000 7f85b939d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:22.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.949+0000 7f85b939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:22.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:22.949+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f85b406f000 con 0x7f85b4137150 2024-09-15T01:55:23.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.205+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v95) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85ac01d770 con 0x7f85b4137150 2024-09-15T01:55:23.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.205+0000 7f859affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:23.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.205+0000 7f859affd640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-15T01:55:23.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.311+0000 7f85b939d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:23.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.311+0000 7f85b939d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:23.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.311+0000 7f85b939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.311+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f85b4098f40 con 0x7f85b4137150 2024-09-15T01:55:23.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.312+0000 7f859affd640 1 -- 172.21.15.67:0/3817364173 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v95) v1 ==== 124+0+0 (secure 0 0 0) 0x7f85ac01d770 con 0x7f85b4137150 2024-09-15T01:55:23.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.312+0000 7f859affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T01:55:23.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.312+0000 7f859affd640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-15T01:55:23.315 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.7' does not appear in the crush map 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 msgr2=0x7f85a003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 0x7f85a003aba0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f85a40047d0 tx=0x7f85a4007b80 comp rx=0 tx=0).stop 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 10 monclient: shutdown 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 msgr2=0x7f85b4142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bb30 tx=0x7f85ac0097a0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 shutdown_connections 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a00386f0 0x7f85a003aba0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b4069190 0x7f85b4141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4137150 0x7f85b4142040 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 --2- 172.21.15.67:0/3817364173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b4068dc0 0x7f85b41415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 >> 172.21.15.67:0/3817364173 conn(0x7f85b40711b0 msgr2=0x7f85b409e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 shutdown_connections 2024-09-15T01:55:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.314+0000 7f85b939d640 1 -- 172.21.15.67:0/3817364173 wait complete. 2024-09-15T01:55:23.326 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.9 2024-09-15T01:55:23.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 10 monclient: get_monmap_and_config 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 10 monclient: build_initial_monmap 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 10 monclient: monmap: 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:23.401240+0000 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:23.401240+0000 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 1 Processor -- start 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 1 -- start start 2024-09-15T01:55:23.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.400+0000 7fc1c102b640 10 monclient: init 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1bc098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1bc0a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1bc098c20 0x7fc1bc1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1bc098c20 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:23.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc0a0b80 con 0x7fc1bc0a0370 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc0a1730 con 0x7fc1bc098c20 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0a00b0 con 0x7fc1bc098140 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1c102b640 10 monclient(hunting): authenticate will time out at 955.870214s 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bad9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 msgr2=0x7fc1bc1321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1bc098c20 0x7fc1bc1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1bc098c20 0x7fc1bc1390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47404/0 (socket says 172.21.15.67:47404) 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 1 -- 172.21.15.67:0/3191433320 learned_addr learned my addr 172.21.15.67:0/3191433320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): get_auth_request con 0x7fc1bc098c20 auth_method 0 2024-09-15T01:55:23.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1ba59c640 1 --1- 172.21.15.67:0/3191433320 >> v1:172.21.15.17:6789/0 conn(0x7fc1bc0a0370 0x7fc1bc098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:42346/0 (socket says 172.21.15.67:42346) 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_done global_id 44793 payload 274 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.401+0000 7fc1bb59e640 1 -- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 msgr2=0x7fc1bc1321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 1 --2- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 1 -- 172.21.15.67:0/3191433320 >> v1:172.21.15.17:6789/0 conn(0x7fc1bc0a0370 legacy=0x7fc1bc098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 10 monclient: _finish_hunting 0 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 1 monclient: found mon.noname-a 2024-09-15T01:55:23.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:23.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:23.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.402+0000 7fc1bb59e640 1 -- 172.21.15.67:0/3191433320 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc071390 con 0x7fc1bc098c20 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.404+0000 7fc1bb59e640 1 -- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 msgr2=0x7fc1bc1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.404+0000 7fc1bb59e640 1 --2- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc1390c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009910 tx=0x7fc1b40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2feecfe70f6c023d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 1 -- 172.21.15.67:0/3191433320 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4032440 con 0x7fc1bc098c20 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: dump: 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.413 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: _finish_auth 0 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 1 -- 172.21.15.67:0/3191433320 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b40325a0 con 0x7fc1bc098c20 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 5 monclient: authenticate success, global_id 44793 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:23.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 4 set_mon_vals no callback set 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 1 -- 172.21.15.67:0/3191433320 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4032940 con 0x7fc1bc098c20 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1b9d9b640 10 monclient: dump: 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 10 monclient: shutdown 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 -- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 msgr2=0x7fc1bc1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 --2- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc1390c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009910 tx=0x7fc1b40091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 -- 172.21.15.67:0/3191433320 shutdown_connections 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 --2- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc1390c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 --2- 172.21.15.67:0/3191433320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 -- 172.21.15.67:0/3191433320 >> 172.21.15.67:0/3191433320 conn(0x7fc1bc0a5780 msgr2=0x7fc1bc0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:23.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 -- 172.21.15.67:0/3191433320 shutdown_connections 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.405+0000 7fc1c102b640 1 -- 172.21.15.67:0/3191433320 wait complete. 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.406+0000 7fc1c102b640 10 monclient: build_initial_monmap 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.406+0000 7fc1c102b640 10 monclient: monmap: 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:23.417 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.406+0000 7fc1c102b640 1 Processor -- start 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.406+0000 7fc1c102b640 1 -- start start 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.406+0000 7fc1c102b640 10 monclient: init 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1bc098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1bc098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1bc0a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc133440 con 0x7fc1bc098c20 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc099090 con 0x7fc1bc0a0370 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc09f1f0 con 0x7fc1bc098140 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1c102b640 10 monclient(hunting): authenticate will time out at 955.876383s 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bad9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 msgr2=0x7fc1bc138890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc138890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53474/0 (socket says 172.21.15.67:53474) 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 -- 172.21.15.67:0/1098124220 learned_addr learned my addr 172.21.15.67:0/1098124220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:23.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): get_auth_request con 0x7fc1bc0a0370 auth_method 0 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): get_auth_request con 0x7fc1bc098c20 auth_method 0 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:23.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient(hunting): handle_auth_done global_id 5446 payload 824 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 -- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 msgr2=0x7fc1bc138890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc138890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 -- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 msgr2=0x7fc1bc130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient: _finish_hunting 0 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 monclient: found mon.noname-b 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.407+0000 7fc1bb59e640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4009450 con 0x7fc1bc0a0370 2024-09-15T01:55:23.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1bb59e640 1 -- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 msgr2=0x7fc1bc131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1bb59e640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009970 tx=0x7fc1b4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4031c70 con 0x7fc1bc0a0370 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: dump: 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.422 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b4031dd0 con 0x7fc1bc0a0370 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b403c640 con 0x7fc1bc0a0370 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 5 monclient: authenticate success, global_id 5446 2024-09-15T01:55:23.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b2ffd640 10 monclient: dump: 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1b959a640 4 set_mon_vals no callback set 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 10 monclient: _renew_subs 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1bc134ca0 con 0x7fc1bc0a0370 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 10 monclient: _renew_subs 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.410+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc135680 con 0x7fc1bc0a0370 2024-09-15T01:55:23.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc1b4039070 con 0x7fc1bc0a0370 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1c102b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1c102b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1c102b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1bc134a50 con 0x7fc1bc0a0370 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1b2ffd640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 0x7fc19c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5966+0+0 (secure 0 0 0) 0x7fc1b406c580 con 0x7fc1bc0a0370 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1bad9d640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 0x7fc19c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.411+0000 7fc1bad9d640 10 monclient: get_auth_request con 0x7fc19c0386f0 auth_method 0 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.414+0000 7fc1bad9d640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 0x7fc19c03aba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac003c70 tx=0x7fc1ac009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.416+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc1b403aae0 con 0x7fc1bc0a0370 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.416+0000 7fc1b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.416+0000 7fc1b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:23.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.583+0000 7fc1c102b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.583+0000 7fc1c102b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.583+0000 7fc1c102b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.583+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7fc1bc04bf60 con 0x7fc1bc0a0370 2024-09-15T01:55:23.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.586+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc1b4040020 con 0x7fc1bc0a0370 2024-09-15T01:55:23.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.586+0000 7fc1b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.586+0000 7fc1b2ffd640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T01:55:23.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.688+0000 7fc1c102b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.688+0000 7fc1c102b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.688+0000 7fc1c102b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.688+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7fc1bc04bf60 con 0x7fc1bc0a0370 2024-09-15T01:55:23.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.689+0000 7fc1b2ffd640 1 -- 172.21.15.67:0/1098124220 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc1b4037070 con 0x7fc1bc0a0370 2024-09-15T01:55:23.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.689+0000 7fc1b2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T01:55:23.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.689+0000 7fc1b2ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T01:55:23.692 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.9' does not appear in the crush map 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 msgr2=0x7fc19c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 0x7fc19c03aba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac003c70 tx=0x7fc1ac009310 comp rx=0 tx=0).stop 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 10 monclient: shutdown 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 msgr2=0x7fc1bc131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009970 tx=0x7fc1b4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 shutdown_connections 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc19c0386f0 0x7fc19c03aba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1bc098c20 0x7fc1bc130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1bc0a0370 0x7fc1bc131250 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 --2- 172.21.15.67:0/1098124220 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1bc098140 0x7fc1bc138890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 >> 172.21.15.67:0/1098124220 conn(0x7fc1bc0a5780 msgr2=0x7fc1bc0a37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 shutdown_connections 2024-09-15T01:55:23.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.691+0000 7fc1c102b640 1 -- 172.21.15.67:0/1098124220 wait complete. 2024-09-15T01:55:23.702 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm foo 2024-09-15T01:55:23.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.776+0000 7fe81579d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:23.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.776+0000 7fe81579d640 10 monclient: build_initial_monmap 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.776+0000 7fe81579d640 10 monclient: monmap: 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:23.777589+0000 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:23.777589+0000 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:23.780 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.776+0000 7fe81579d640 1 Processor -- start 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 -- start start 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient: init 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 0x7fe810069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8100a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe81009eea0 0x7fe81009f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-a con 0x7fe81009eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:23.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe810131b10 0x7fe810131f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-c con 0x7fe810131b10 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8100a1610 con 0x7fe81009eea0 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81009f7d0 con 0x7fe810131b10 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100a61a0 con 0x7fe8100a0b50 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe81579d640 10 monclient(hunting): authenticate will time out at 956.246555s 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe810131b10 0x7fe810131f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80effd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 msgr2=0x7fe810069380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 0x7fe810069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:23.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe810131b10 0x7fe810131f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53484/0 (socket says 172.21.15.67:53484) 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 1 -- 172.21.15.67:0/855057924 learned_addr learned my addr 172.21.15.67:0/855057924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): get_auth_request con 0x7fe810131b10 auth_method 0 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80e7fc640 1 --2- 172.21.15.67:0/855057924 >> v2:172.21.15.17:3300/0 conn(0x7fe81009eea0 0x7fe81009f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80e7fc640 10 monclient(hunting): get_auth_request con 0x7fe81009eea0 auth_method 0 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.777+0000 7fe80e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 10 monclient(hunting): handle_auth_done global_id 5449 payload 274 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 -- 172.21.15.67:0/855057924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 msgr2=0x7fe810069380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 --2- 172.21.15.67:0/855057924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 0x7fe810069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 -- 172.21.15.67:0/855057924 >> v2:172.21.15.17:3300/0 conn(0x7fe81009eea0 msgr2=0x7fe81009f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 --2- 172.21.15.67:0/855057924 >> v2:172.21.15.17:3300/0 conn(0x7fe81009eea0 0x7fe81009f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 monclient: found mon.noname-c 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.778+0000 7fe80f7fe640 1 -- 172.21.15.67:0/855057924 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8100a04c0 con 0x7fe810131b10 2024-09-15T01:55:23.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80f7fe640 1 -- 172.21.15.67:0/855057924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe810131b10 msgr2=0x7fe810131f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80f7fe640 1 --2- 172.21.15.67:0/855057924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe810131b10 0x7fe810131f60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe81009ffe0 tx=0x7fe808009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2fd5e6be4ce1ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80dffb640 1 -- 172.21.15.67:0/855057924 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808007610 con 0x7fe810131b10 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.780+0000 7fe80dffb640 10 monclient: dump: 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 1 -- 172.21.15.67:0/855057924 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80802d050 con 0x7fe810131b10 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 1 -- 172.21.15.67:0/855057924 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8080079a0 con 0x7fe810131b10 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 5 monclient: authenticate success, global_id 5449 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 4 set_mon_vals no callback set 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe80dffb640 10 monclient: dump: 2024-09-15T01:55:23.792 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 10 monclient: shutdown 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 -- 172.21.15.67:0/855057924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe810131b10 msgr2=0x7fe810131f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 --2- 172.21.15.67:0/855057924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe810131b10 0x7fe810131f60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe81009ffe0 tx=0x7fe808009840 comp rx=0 tx=0).stop 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 -- 172.21.15.67:0/855057924 shutdown_connections 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 --2- 172.21.15.67:0/855057924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe810131b10 0x7fe810131f60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 --2- 172.21.15.67:0/855057924 >> v2:172.21.15.17:3300/0 conn(0x7fe81009eea0 0x7fe81009f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 --2- 172.21.15.67:0/855057924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8100a0b50 0x7fe810069380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 -- 172.21.15.67:0/855057924 >> 172.21.15.67:0/855057924 conn(0x7fe810071330 msgr2=0x7fe8100a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 -- 172.21.15.67:0/855057924 shutdown_connections 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.781+0000 7fe81579d640 1 -- 172.21.15.67:0/855057924 wait complete. 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.782+0000 7fe81579d640 10 monclient: build_initial_monmap 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.782+0000 7fe81579d640 10 monclient: monmap: 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:23.794 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.782+0000 7fe81579d640 1 Processor -- start 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.782+0000 7fe81579d640 1 -- start start 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.782+0000 7fe81579d640 10 monclient: init 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe81009eea0 0x7fe810141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-a con 0x7fe81009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8100a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 0x7fe81011e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): picked mon.noname-c con 0x7fe810131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100a1260 con 0x7fe81009eea0 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe810120940 con 0x7fe8100a0b50 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe810121ab0 con 0x7fe810131b10 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe81579d640 10 monclient(hunting): authenticate will time out at 956.252132s 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80f7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 msgr2=0x7fe81011e4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:23.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53492/0 (socket says 172.21.15.67:53492) 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 -- 172.21.15.67:0/1247605810 learned_addr learned my addr 172.21.15.67:0/1247605810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): get_auth_request con 0x7fe8100a0b50 auth_method 0 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80f7fe640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 0x7fe81011e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80effd640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe81009eea0 0x7fe810141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient(hunting): handle_auth_done global_id 5452 payload 824 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 -- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 msgr2=0x7fe81011e4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:23.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 0x7fe81011e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 -- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe81009eea0 msgr2=0x7fe810141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe81009eea0 0x7fe810141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 monclient: found mon.noname-b 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.783+0000 7fe80e7fc640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808009ae0 con 0x7fe8100a0b50 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe80e7fc640 1 -- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 msgr2=0x7fe810141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe80e7fc640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe80400d890 tx=0x7fe804008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400de20 con 0x7fe8100a0b50 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: dump: 2024-09-15T01:55:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80400bae0 con 0x7fe8100a0b50 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400be80 con 0x7fe8100a0b50 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 5 monclient: authenticate success, global_id 5452 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe7f6ffd640 10 monclient: dump: 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 10 monclient: _renew_subs 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe810121d90 con 0x7fe8100a0b50 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe80d7fa640 4 set_mon_vals no callback set 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 10 monclient: _renew_subs 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.785+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe810120bc0 con 0x7fe8100a0b50 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.786+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe804014640 con 0x7fe8100a0b50 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.786+0000 7fe81579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.786+0000 7fe81579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.786+0000 7fe81579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.786+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81004bf60 con 0x7fe8100a0b50 2024-09-15T01:55:23.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.787+0000 7fe7f6ffd640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 0x7fe7fc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.787+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5966+0+0 (secure 0 0 0) 0x7fe80404c630 con 0x7fe8100a0b50 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.787+0000 7fe80effd640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 0x7fe7fc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.787+0000 7fe80effd640 10 monclient: get_auth_request con 0x7fe7fc0386f0 auth_method 0 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.789+0000 7fe80effd640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 0x7fe7fc03aba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe800006fd0 tx=0x7fe800008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.791+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe804011570 con 0x7fe8100a0b50 2024-09-15T01:55:23.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.791+0000 7fe7f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:23.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.791+0000 7fe7f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:23.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.957+0000 7fe81579d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:23.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.957+0000 7fe81579d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:23.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.957+0000 7fe81579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:23.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:23.957+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fe8100a4be0 con 0x7fe8100a0b50 2024-09-15T01:55:24.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.208+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v96) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe80401dbb0 con 0x7fe8100a0b50 2024-09-15T01:55:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.208+0000 7fe7f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.208+0000 7fe7f6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-15T01:55:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.311+0000 7fe81579d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.311+0000 7fe81579d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:24.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.311+0000 7fe81579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.311+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fe8100008d0 con 0x7fe8100a0b50 2024-09-15T01:55:24.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.312+0000 7fe7f6ffd640 1 -- 172.21.15.67:0/1247605810 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v96) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe804011340 con 0x7fe8100a0b50 2024-09-15T01:55:24.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.312+0000 7fe7f6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T01:55:24.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.312+0000 7fe7f6ffd640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-15T01:55:24.315 INFO:tasks.workunit.client.0.smithi067.stderr:device 'foo' does not appear in the crush map 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 msgr2=0x7fe7fc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 0x7fe7fc03aba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe800006fd0 tx=0x7fe800008040 comp rx=0 tx=0).stop 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 10 monclient: shutdown 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 msgr2=0x7fe810141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe80400d890 tx=0x7fe804008040 comp rx=0 tx=0).stop 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 shutdown_connections 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe7fc0386f0 0x7fe7fc03aba0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe810131b10 0x7fe81011e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8100a0b50 0x7fe810141740 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 --2- 172.21.15.67:0/1247605810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe81009eea0 0x7fe810141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 >> 172.21.15.67:0/1247605810 conn(0x7fe810071330 msgr2=0x7fe8100a31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 shutdown_connections 2024-09-15T01:55:24.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.314+0000 7fe81579d640 1 -- 172.21.15.67:0/1247605810 wait complete. 2024-09-15T01:55:24.326 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 2024-09-15T01:55:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 10 monclient: build_initial_monmap 2024-09-15T01:55:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 10 monclient: monmap: 2024-09-15T01:55:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:24.399302+0000 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:24.399302+0000 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:24.401 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 1 Processor -- start 2024-09-15T01:55:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 1 -- start start 2024-09-15T01:55:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.398+0000 7f8befc6f640 10 monclient: init 2024-09-15T01:55:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8132360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-b con 0x7f8be8098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 0x7f8be8098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-a con 0x7f8be8098740 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8be8068e70 0x7f8be80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-c con 0x7f8be8068e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be80a17b0 con 0x7f8be8098740 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be809feb0 con 0x7f8be8068e70 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80a00f0 con 0x7f8be8098370 2024-09-15T01:55:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8befc6f640 10 monclient(hunting): authenticate will time out at 956.868424s 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8be8068e70 0x7f8be80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8beda0b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 msgr2=0x7f8be8132360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8beda0b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8132360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8be8068e70 0x7f8be80692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53502/0 (socket says 172.21.15.67:53502) 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 -- 172.21.15.67:0/3411929250 learned_addr learned my addr 172.21.15.67:0/3411929250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): get_auth_request con 0x7f8be8068e70 auth_method 0 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bed20a640 1 --2- 172.21.15.67:0/3411929250 >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 0x7f8be8098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient(hunting): handle_auth_done global_id 5455 payload 274 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 -- 172.21.15.67:0/3411929250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 msgr2=0x7f8be8132360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 --2- 172.21.15.67:0/3411929250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8132360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 -- 172.21.15.67:0/3411929250 >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 msgr2=0x7f8be8098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 --2- 172.21.15.67:0/3411929250 >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 0x7f8be8098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 monclient: found mon.noname-c 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bee20c640 1 -- 172.21.15.67:0/3411929250 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80a14d0 con 0x7f8be8068e70 2024-09-15T01:55:24.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bed20a640 10 monclient: get_auth_request con 0x7f8be8098740 auth_method 0 2024-09-15T01:55:24.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.399+0000 7f8bed20a640 1 --2- 172.21.15.67:0/3411929250 >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 0x7f8be8098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.402+0000 7f8bee20c640 1 -- 172.21.15.67:0/3411929250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8068e70 msgr2=0x7f8be80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.402+0000 7f8bee20c640 1 --2- 172.21.15.67:0/3411929250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8068e70 0x7f8be80692b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8be809fe40 tx=0x7f8bd8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125fc728d299e15a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 1 -- 172.21.15.67:0/3411929250 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8007610 con 0x7f8be8068e70 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: dump: 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: _finish_auth 0 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 1 -- 172.21.15.67:0/3411929250 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd802d050 con 0x7f8be8068e70 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 5 monclient: authenticate success, global_id 5455 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:24.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 1 -- 172.21.15.67:0/3411929250 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80079e0 con 0x7f8be8068e70 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 4 set_mon_vals no callback set 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8beca09640 10 monclient: dump: 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 10 monclient: shutdown 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 -- 172.21.15.67:0/3411929250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8068e70 msgr2=0x7f8be80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 --2- 172.21.15.67:0/3411929250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8068e70 0x7f8be80692b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8be809fe40 tx=0x7f8bd8009840 comp rx=0 tx=0).stop 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 -- 172.21.15.67:0/3411929250 shutdown_connections 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 --2- 172.21.15.67:0/3411929250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8068e70 0x7f8be80692b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 --2- 172.21.15.67:0/3411929250 >> v2:172.21.15.17:3300/0 conn(0x7f8be8098740 0x7f8be8098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 --2- 172.21.15.67:0/3411929250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8132360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 -- 172.21.15.67:0/3411929250 >> 172.21.15.67:0/3411929250 conn(0x7f8be80a59a0 msgr2=0x7f8be80a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 -- 172.21.15.67:0/3411929250 shutdown_connections 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.403+0000 7f8befc6f640 1 -- 172.21.15.67:0/3411929250 wait complete. 2024-09-15T01:55:24.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient: build_initial_monmap 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient: monmap: 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 Processor -- start 2024-09-15T01:55:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 -- start start 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient: init 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8be8068e70 0x7f8be8140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-a con 0x7f8be8068e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-c con 0x7f8be8098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): picked mon.noname-b con 0x7f8be8098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8141ec0 con 0x7f8be8068e70 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8121290 con 0x7f8be8098740 2024-09-15T01:55:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80a3520 con 0x7f8be8098370 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.404+0000 7f8befc6f640 10 monclient(hunting): authenticate will time out at 956.873943s 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bed20a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 msgr2=0x7f8be8130e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bed20a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8130e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53504/0 (socket says 172.21.15.67:53504) 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 -- 172.21.15.67:0/350518551 learned_addr learned my addr 172.21.15.67:0/350518551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): get_auth_request con 0x7f8be8098740 auth_method 0 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8beda0b640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8be8068e70 0x7f8be8140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8beda0b640 10 monclient(hunting): get_auth_request con 0x7f8be8068e70 auth_method 0 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8beda0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8beda0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8beda0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient(hunting): handle_auth_done global_id 5458 payload 824 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 -- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 msgr2=0x7f8be8130e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8130e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 -- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8be8068e70 msgr2=0x7f8be8140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8be8068e70 0x7f8be8140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 monclient: found mon.noname-b 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.405+0000 7f8bee20c640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd8009ae0 con 0x7f8be8098740 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8bee20c640 1 -- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 msgr2=0x7f8be811d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8bee20c640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009880 tx=0x7f8bd8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8bee20c640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009880 tx=0x7f8bd8036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8bd8032cc0 mon_map magic: 0 v1 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8032cc0 con 0x7f8be8098740 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 10 monclient: dump: 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:24.420 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd8032e20 con 0x7f8be8098740 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8befc6f640 5 monclient: authenticate success, global_id 5458 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:24.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8befc6f640 10 monclient: _renew_subs 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8befc6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be8120fb0 con 0x7f8be8098740 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.407+0000 7f8be7fff640 4 set_mon_vals no callback set 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 10 monclient: _renew_subs 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be811f4f0 con 0x7f8be8098740 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd803b640 con 0x7f8be8098740 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8be5ffb640 10 monclient: dump: 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.422 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8bd8038050 con 0x7f8be8098740 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.408+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be80a0890 con 0x7f8be8098740 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.409+0000 7f8be5ffb640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 0x7f8bd403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.409+0000 7f8beda0b640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 0x7f8bd403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.409+0000 7f8beda0b640 10 monclient: get_auth_request con 0x7f8bd4038770 auth_method 0 2024-09-15T01:55:24.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.411+0000 7f8beda0b640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 0x7f8bd403ac20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8be00038f0 tx=0x7f8be0006ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:24.413+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f8bd806cc50 con 0x7f8be8098740 2024-09-15T01:55:24.481 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:55:24.481 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T01:55:24.481 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:55:24.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:55:24.484 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:55:24.484 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:55:24.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:24.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.591+0000 7ff22d19d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:24.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.591+0000 7ff22d19d640 10 monclient: build_initial_monmap 2024-09-15T01:55:24.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.591+0000 7ff22d19d640 10 monclient: monmap: 2024-09-15T01:55:24.593 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:24.593 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:24.592436+0000 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:24.592436+0000 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:24.594 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 Processor -- start 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 -- start start 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient: init 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T01:55:24.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2280a58b0 0x7ff228132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2280a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2280980a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff228136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280a01e0 con 0x7ff2280980a0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280a1580 con 0x7ff228136dd0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280a5c80 con 0x7ff2280a58b0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22d19d640 10 monclient(hunting): authenticate will time out at 984.588793s 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22659c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff2280980a0 0x7ff2280984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55220/0 (socket says 172.21.15.17:55220) 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff22659c640 1 -- 172.21.15.17:0/674494917 learned_addr learned my addr 172.21.15.17:0/674494917 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.592+0000 7ff226d9d640 1 --2- 172.21.15.17:0/674494917 >> v2:172.21.15.67:3300/0 conn(0x7ff2280a58b0 0x7ff228132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): get_auth_request con 0x7ff2280a58b0 auth_method 0 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient(hunting): handle_auth_done global_id 5461 payload 274 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 1 -- 172.21.15.17:0/674494917 >> v1:172.21.15.67:6789/0 conn(0x7ff228136dd0 legacy=0x7ff2281391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 1 -- 172.21.15.17:0/674494917 >> v1:172.21.15.17:6789/0 conn(0x7ff2280980a0 legacy=0x7ff2280984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 1 monclient: found mon.noname-c 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:24.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.593+0000 7ff226d9d640 1 -- 172.21.15.17:0/674494917 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228098bb0 con 0x7ff2280a58b0 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff226d9d640 1 -- 172.21.15.17:0/674494917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 msgr2=0x7ff228132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff226d9d640 1 --2- 172.21.15.17:0/674494917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228132040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff218009850 tx=0x7ff2180090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d082c31f319b2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 1 -- 172.21.15.17:0/674494917 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218032440 con 0x7ff2280a58b0 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: dump: 2024-09-15T01:55:24.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: _finish_auth 0 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 1 -- 172.21.15.17:0/674494917 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2180325a0 con 0x7ff2280a58b0 2024-09-15T01:55:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 5 monclient: authenticate success, global_id 5461 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 1 -- 172.21.15.17:0/674494917 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218032940 con 0x7ff2280a58b0 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 4 set_mon_vals no callback set 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff225d9b640 10 monclient: dump: 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.604 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 10 monclient: shutdown 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 -- 172.21.15.17:0/674494917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 msgr2=0x7ff228132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 --2- 172.21.15.17:0/674494917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228132040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff218009850 tx=0x7ff2180090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 -- 172.21.15.17:0/674494917 shutdown_connections 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 --2- 172.21.15.17:0/674494917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228132040 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 -- 172.21.15.17:0/674494917 >> 172.21.15.17:0/674494917 conn(0x7ff228070ed0 msgr2=0x7ff2280a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 -- 172.21.15.17:0/674494917 shutdown_connections 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.596+0000 7ff22d19d640 1 -- 172.21.15.17:0/674494917 wait complete. 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.597+0000 7ff22d19d640 10 monclient: build_initial_monmap 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.597+0000 7ff22d19d640 10 monclient: monmap: 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:24.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.597+0000 7ff22d19d640 1 Processor -- start 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.597+0000 7ff22d19d640 1 -- start start 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.597+0000 7ff22d19d640 10 monclient: init 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2280980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-b con 0x7ff2280a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 0x7ff2281335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): picked mon.noname-a con 0x7ff228136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228135300 con 0x7ff228136dd0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228135020 con 0x7ff2280a58b0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff228135eb0 con 0x7ff2280980a0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22d19d640 10 monclient(hunting): authenticate will time out at 984.594432s 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 0x7ff2281335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38492/0 (socket says 172.21.15.17:38492) 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 -- 172.21.15.17:0/693834522 learned_addr learned my addr 172.21.15.17:0/693834522 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 10 monclient(hunting): get_auth_request con 0x7ff2280980a0 auth_method 0 2024-09-15T01:55:24.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): get_auth_request con 0x7ff2280a58b0 auth_method 0 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): get_auth_request con 0x7ff228136dd0 auth_method 0 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 msgr2=0x7ff228132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 msgr2=0x7ff228132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:24.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff226d9d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient(hunting): handle_auth_done global_id 5464 payload 824 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 msgr2=0x7ff228132b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 msgr2=0x7ff2281335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:24.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 0x7ff2281335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 monclient: found mon.noname-b 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22659c640 1 -- 172.21.15.17:0/693834522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218009390 con 0x7ff2280a58b0 2024-09-15T01:55:24.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.598+0000 7ff22759e640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 0x7ff2281335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22659c640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 msgr2=0x7ff228133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22659c640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228133060 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b490 tx=0x7ff21c006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011840 con 0x7ff2280a58b0 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: dump: 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:24.613 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: _finish_auth 0 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c0119a0 con 0x7ff2280a58b0 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011d60 con 0x7ff2280a58b0 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff20effd640 10 monclient: dump: 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:24.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 5 monclient: authenticate success, global_id 5464 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 10 monclient: _renew_subs 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff228133e50 con 0x7ff2280a58b0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22559a640 4 set_mon_vals no callback set 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 10 monclient: _renew_subs 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.601+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22809e500 con 0x7ff2280a58b0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff22d19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff22d19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff22d19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2280724a0 con 0x7ff2280a58b0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff21c010590 con 0x7ff2280a58b0 2024-09-15T01:55:24.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff20effd640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 0x7ff21403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.602+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5726+0+0 (secure 0 0 0) 0x7ff21c04be60 con 0x7ff2280a58b0 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.603+0000 7ff226d9d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 0x7ff21403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.603+0000 7ff226d9d640 10 monclient: get_auth_request con 0x7ff214038790 auth_method 0 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.605+0000 7ff226d9d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 0x7ff21403ac40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff218009330 tx=0x7ff218032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.607+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff21c04f030 con 0x7ff2280a58b0 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.607+0000 7ff20effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:24.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.607+0000 7ff20effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:24.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.836+0000 7ff22d19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:24.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.836+0000 7ff22d19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.836+0000 7ff22d19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:24.836+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff228135540 con 0x7ff2280a58b0 2024-09-15T01:55:25.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.030+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8be80a0890 con 0x7f8be8098740 2024-09-15T01:55:25.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.030+0000 7f8be5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:25.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.030+0000 7f8be5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:25.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.209+0000 7f8befc6f640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:25.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.209+0000 7f8befc6f640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T01:55:25.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.209+0000 7f8befc6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:25.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:25.209+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f8be804bf60 con 0x7f8be8098740 2024-09-15T01:55:27.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:27.405+0000 7f8be4ff9640 10 monclient: tick 2024-09-15T01:55:27.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:27.405+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:27.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:27.405+0000 7f8be4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:27.598+0000 7ff20dffb640 10 monclient: tick 2024-09-15T01:55:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:27.598+0000 7ff20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:27.598+0000 7ff20dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:28.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:28.405+0000 7f8be4ff9640 10 monclient: tick 2024-09-15T01:55:28.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:28.405+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:28.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:28.599+0000 7ff20dffb640 10 monclient: tick 2024-09-15T01:55:28.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:28.599+0000 7ff20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:29.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.405+0000 7f8be4ff9640 10 monclient: tick 2024-09-15T01:55:29.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.405+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.599+0000 7ff20dffb640 10 monclient: tick 2024-09-15T01:55:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.599+0000 7ff20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:29.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.801+0000 7ff20effd640 1 -- 172.21.15.17:0/693834522 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7ff21c015070 con 0x7ff2280a58b0 2024-09-15T01:55:29.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.801+0000 7ff20effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:29.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.801+0000 7ff20effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:29.803 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 msgr2=0x7ff21403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 0x7ff21403ac40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff218009330 tx=0x7ff218032000 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 10 monclient: shutdown 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 msgr2=0x7ff228133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228133060 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b490 tx=0x7ff21c006f50 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 shutdown_connections 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff214038790 0x7ff21403ac40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228136dd0 0x7ff2281335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280a58b0 0x7ff228133060 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 --2- 172.21.15.17:0/693834522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2280980a0 0x7ff228132b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 >> 172.21.15.17:0/693834522 conn(0x7ff228070ed0 msgr2=0x7ff2281391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 shutdown_connections 2024-09-15T01:55:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:29.804+0000 7ff22d19d640 1 -- 172.21.15.17:0/693834522 wait complete. 2024-09-15T01:55:29.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.871+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v97) v1 ==== 58+0+3 (secure 0 0 0) 0x7f8bd803f030 con 0x7f8be8098740 2024-09-15T01:55:29.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.871+0000 7f8be5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T01:55:29.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.871+0000 7f8be5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:29.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.983+0000 7f8befc6f640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:29.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.983+0000 7f8befc6f640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T01:55:29.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.983+0000 7f8befc6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:29.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:29.983+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f8be81328a0 con 0x7f8be8098740 2024-09-15T01:55:30.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.405+0000 7f8be4ff9640 10 monclient: tick 2024-09-15T01:55:30.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.405+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:30.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.872+0000 7f8be5ffb640 1 -- 172.21.15.67:0/350518551 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v98) v1 ==== 58+0+3 (secure 0 0 0) 0x7f8bd803f020 con 0x7f8be8098740 2024-09-15T01:55:30.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.872+0000 7f8be5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T01:55:30.875 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.872+0000 7f8be5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:30.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 msgr2=0x7f8bd403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 0x7f8bd403ac20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8be00038f0 tx=0x7f8be0006ef0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 10 monclient: shutdown 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 msgr2=0x7f8be811d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009880 tx=0x7f8bd8036000 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 shutdown_connections 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bd4038770 0x7f8bd403ac20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8098740 0x7f8be811d8f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8098370 0x7f8be8130e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 --2- 172.21.15.67:0/350518551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8be8068e70 0x7f8be8140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 >> 172.21.15.67:0/350518551 conn(0x7f8be80a59a0 msgr2=0x7f8be809b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 shutdown_connections 2024-09-15T01:55:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.875+0000 7f8befc6f640 1 -- 172.21.15.67:0/350518551 wait complete. 2024-09-15T01:55:30.888 INFO:tasks.workunit.client.0.smithi067.stderr:+ o3=11 2024-09-15T01:55:30.888 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 11 123 root=default 2024-09-15T01:55:30.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:30.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 10 monclient: build_initial_monmap 2024-09-15T01:55:30.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 10 monclient: monmap: 2024-09-15T01:55:30.972 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:30.972 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:30.971439+0000 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:30.971439+0000 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:30.973 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 1 Processor -- start 2024-09-15T01:55:30.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 1 -- start start 2024-09-15T01:55:30.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.970+0000 7fa27195d640 10 monclient: init 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa26c098140 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa26c098a50 0x7fa26c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-a con 0x7fa26c098a50 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26c098ef0 con 0x7fa26c098140 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26c0a1730 con 0x7fa26c098a50 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:30.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0a0b80 con 0x7fa26c0a0370 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa27195d640 10 monclient(hunting): authenticate will time out at 963.440443s 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa26c098a50 0x7fa26c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa26c098a50 0x7fa26c1390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:56452/0 (socket says 172.21.15.67:56452) 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26affd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa26c098140 0x7fa26c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:36398/0 (socket says 172.21.15.67:36398) 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:40924/0 (socket says 172.21.15.67:40924) 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 1 -- 172.21.15.67:0/2226914038 learned_addr learned my addr 172.21.15.67:0/2226914038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): get_auth_request con 0x7fa26c098a50 auth_method 0 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 10 monclient(hunting): get_auth_request con 0x7fa26c0a0370 auth_method 0 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 -- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 msgr2=0x7fa26c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 -- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 msgr2=0x7fa26c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.971+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 10 monclient(hunting): handle_auth_done global_id 44808 payload 274 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 1 -- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 msgr2=0x7fa26c098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 1 -- 172.21.15.67:0/2226914038 >> v1:172.21.15.17:6789/0 conn(0x7fa26c098140 legacy=0x7fa26c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:30.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:30.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:30.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:30.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:30.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.972+0000 7fa26b7fe640 1 -- 172.21.15.67:0/2226914038 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c1328f0 con 0x7fa26c098a50 2024-09-15T01:55:30.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.974+0000 7fa26b7fe640 1 -- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 msgr2=0x7fa26c1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:30.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.974+0000 7fa26b7fe640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c1390c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0a1660 tx=0x7fa2640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82e37f984043872 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 1 -- 172.21.15.67:0/2226914038 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26402ae80 con 0x7fa26c098a50 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: dump: 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:30.984 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 1 -- 172.21.15.67:0/2226914038 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa264032440 con 0x7fa26c098a50 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 1 -- 172.21.15.67:0/2226914038 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640327c0 con 0x7fa26c098a50 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 5 monclient: authenticate success, global_id 44808 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 4 set_mon_vals no callback set 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa269ffb640 10 monclient: dump: 2024-09-15T01:55:30.985 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 10 monclient: shutdown 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 -- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 msgr2=0x7fa26c1390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c1390c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0a1660 tx=0x7fa2640091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 -- 172.21.15.67:0/2226914038 shutdown_connections 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c1390c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 --2- 172.21.15.67:0/2226914038 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 -- 172.21.15.67:0/2226914038 >> 172.21.15.67:0/2226914038 conn(0x7fa26c0a5780 msgr2=0x7fa26c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 -- 172.21.15.67:0/2226914038 shutdown_connections 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.975+0000 7fa27195d640 1 -- 172.21.15.67:0/2226914038 wait complete. 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.976+0000 7fa27195d640 10 monclient: build_initial_monmap 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.976+0000 7fa27195d640 10 monclient: monmap: 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.987 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.976+0000 7fa27195d640 1 Processor -- start 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.976+0000 7fa27195d640 1 -- start start 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.976+0000 7fa27195d640 10 monclient: init 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-a con 0x7fa26c098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c131230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): picked mon.noname-c con 0x7fa26c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c09f380 con 0x7fa26c098a50 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1347f0 con 0x7fa26c098140 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c134db0 con 0x7fa26c0a0370 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa27195d640 10 monclient(hunting): authenticate will time out at 963.446371s 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46374/0 (socket says 172.21.15.67:46374) 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 -- 172.21.15.67:0/4020033392 learned_addr learned my addr 172.21.15.67:0/4020033392 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): get_auth_request con 0x7fa26c098140 auth_method 0 2024-09-15T01:55:30.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient(hunting): handle_auth_done global_id 5467 payload 824 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26b7fe640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c131230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 -- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 msgr2=0x7fa26c131230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c131230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 -- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 msgr2=0x7fa26c130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 monclient: found mon.noname-b 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26affd640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264009450 con 0x7fa26c098140 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26b7fe640 10 monclient: get_auth_request con 0x7fa26c0a0370 auth_method 0 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26a7fc640 10 monclient: get_auth_request con 0x7fa26c098a50 auth_method 0 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26b7fe640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c131230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.977+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa26affd640 1 -- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 msgr2=0x7fa26c1387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa26affd640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c960 tx=0x7fa25c004f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c00a5e0 con 0x7fa26c098140 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:30.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: dump: 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:30.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c00a760 con 0x7fa26c098140 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 5 monclient: authenticate success, global_id 5467 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c00ab20 con 0x7fa26c098140 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 10 monclient: _renew_subs 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c133480 con 0x7fa26c098140 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa2697fa640 4 set_mon_vals no callback set 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa262ffd640 10 monclient: dump: 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.993 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 10 monclient: _renew_subs 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.980+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c099090 con 0x7fa26c098140 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa25c0070f0 con 0x7fa26c098140 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa27195d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa27195d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa27195d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:30.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c134a30 con 0x7fa26c098140 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa262ffd640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 0x7fa24c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5956+0+0 (secure 0 0 0) 0x7fa25c04bed0 con 0x7fa26c098140 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 0x7fa24c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.981+0000 7fa26a7fc640 10 monclient: get_auth_request con 0x7fa24c038740 auth_method 0 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.983+0000 7fa26a7fc640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 0x7fa24c03abf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0a0760 tx=0x7fa254008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.985+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa25c010d70 con 0x7fa26c098140 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.985+0000 7fa262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:30.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:30.985+0000 7fa262ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:31.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.151+0000 7fa27195d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.151+0000 7fa27195d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.151+0000 7fa27195d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:31.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.151+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7fa26c04bf60 con 0x7fa26c098140 2024-09-15T01:55:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.881+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v99) v1 ==== 193+0+0 (secure 0 0 0) 0x7fa25c010770 con 0x7fa26c098140 2024-09-15T01:55:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.881+0000 7fa262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.881+0000 7fa262ffd640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-15T01:55:31.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.984+0000 7fa27195d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.984+0000 7fa27195d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.984+0000 7fa27195d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:31.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.984+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7fa26c071390 con 0x7fa26c098140 2024-09-15T01:55:31.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.985+0000 7fa262ffd640 1 -- 172.21.15.67:0/4020033392 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v99) v1 ==== 191+0+0 (secure 0 0 0) 0x7fa25c02a0b0 con 0x7fa26c098140 2024-09-15T01:55:31.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.986+0000 7fa262ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T01:55:31.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.986+0000 7fa262ffd640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-15T01:55:31.988 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-15T01:55:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 msgr2=0x7fa24c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 0x7fa24c03abf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0a0760 tx=0x7fa254008040 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 10 monclient: shutdown 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 msgr2=0x7fa26c1387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c960 tx=0x7fa25c004f70 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 shutdown_connections 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa24c038740 0x7fa24c03abf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa26c098a50 0x7fa26c130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa26c0a0370 0x7fa26c131230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 --2- 172.21.15.67:0/4020033392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa26c098140 0x7fa26c1387e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 >> 172.21.15.67:0/4020033392 conn(0x7fa26c0a5780 msgr2=0x7fa26c0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 shutdown_connections 2024-09-15T01:55:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:31.988+0000 7fa27195d640 1 -- 172.21.15.67:0/4020033392 wait complete. 2024-09-15T01:55:32.000 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:32.000 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.11 2024-09-15T01:55:32.000 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 123 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.071+0000 7fe47671d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.071+0000 7fe47671d640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.071+0000 7fe47671d640 10 monclient: monmap: 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:32.073094+0000 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:32.073094+0000 2024-09-15T01:55:32.074 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:32.075 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.072+0000 7fe47671d640 1 Processor -- start 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.072+0000 7fe47671d640 1 -- start start 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.072+0000 7fe47671d640 10 monclient: init 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T01:55:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe47006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4700a36e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe47006bdc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe470136bb0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47009f640 con 0x7fe47006bdc0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe470069f00 con 0x7fe470136bb0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47006e8c0 con 0x7fe4700a36e0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe47671d640 10 monclient(hunting): authenticate will time out at 964.542229s 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe474cba640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe470136bb0 0x7fe470138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49898/0 (socket says 172.21.15.67:49898) 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe474cba640 1 -- 172.21.15.67:0/2857031712 learned_addr learned my addr 172.21.15.67:0/2857031712 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe47006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 10 monclient(hunting): get_auth_request con 0x7fe4700a36e0 auth_method 0 2024-09-15T01:55:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 msgr2=0x7fe47006a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 msgr2=0x7fe47006a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe47006a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:32.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.073+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe47006a600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:32.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.075+0000 7fe474cba640 1 -- 172.21.15.67:0/2857031712 >> v1:172.21.15.67:6789/0 conn(0x7fe470136bb0 legacy=0x7fe470138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.075+0000 7fe46f7fe640 1 -- 172.21.15.67:0/2857031712 >> v1:172.21.15.17:6789/0 conn(0x7fe47006bdc0 legacy=0x7fe47006e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.075+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265364173 0 0) 0x7fe470069f00 con 0x7fe470136bb0 2024-09-15T01:55:32.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.075+0000 7fe46effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.075+0000 7fe46effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): my global_id is 5470 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4580034e0 con 0x7fe470136bb0 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661540599 0 0) 0x7fe47009f640 con 0x7fe47006bdc0 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): my global_id is 44811 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe470069f00 con 0x7fe47006bdc0 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924411664 0 0) 0x7fe4580034e0 con 0x7fe470136bb0 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 msgr2=0x7fe47006a600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe47006a600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 >> v1:172.21.15.17:6789/0 conn(0x7fe47006bdc0 legacy=0x7fe47006e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470069c20 con 0x7fe470136bb0 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe468001aa0 con 0x7fe470136bb0 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: dump: 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.090 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 >> v1:172.21.15.67:6789/0 conn(0x7fe470136bb0 legacy=0x7fe470138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe458003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): picked mon.c con 0x7fe4700a36e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 0x7fe458004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): picked mon.b con 0x7fe458004190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004de0 0x7fe458005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): picked mon.a con 0x7fe458004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47006e8c0 con 0x7fe458004de0 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45800bd00 con 0x7fe458004190 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45800bec0 con 0x7fe4700a36e0 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 0x7fe458004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): get_auth_request con 0x7fe458004190 auth_method 0 2024-09-15T01:55:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe458003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): get_auth_request con 0x7fe4700a36e0 auth_method 0 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe474cba640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004de0 0x7fe458005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe474cba640 10 monclient(hunting): get_auth_request con 0x7fe458004de0 auth_method 0 2024-09-15T01:55:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe474cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe474cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe474cba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient(hunting): handle_auth_done global_id 5470 payload 293 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 msgr2=0x7fe458003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4700a36e0 0x7fe458003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004de0 msgr2=0x7fe458005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004de0 0x7fe458005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 monclient: found mon.b 2024-09-15T01:55:32.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 1 -- 172.21.15.67:0/2857031712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4580034e0 con 0x7fe458004190 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe46f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe47671d640 5 monclient: authenticate success, global_id 5470 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.076+0000 7fe47671d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46f7fe640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 msgr2=0x7fe458004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46f7fe640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 0x7fe458004780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe464007d80 tx=0x7fe464001b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81807feeb09e56d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46402ade0 con 0x7fe458004190 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: dump: 2024-09-15T01:55:32.095 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4640070f0 con 0x7fe458004190 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe47671d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe47671d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 1 -- 172.21.15.67:0/2857031712 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640074b0 con 0x7fe458004190 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe47671d640 4 set_mon_vals no callback set 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.079+0000 7fe46effd640 10 monclient: dump: 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.097 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 10 monclient: shutdown 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 -- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 msgr2=0x7fe458004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 --2- 172.21.15.67:0/2857031712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458004190 0x7fe458004780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe464007d80 tx=0x7fe464001b50 comp rx=0 tx=0).stop 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe474cba640 1 -- 172.21.15.67:0/2857031712 reap_dead start 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 -- 172.21.15.67:0/2857031712 shutdown_connections 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 -- 172.21.15.67:0/2857031712 >> 172.21.15.67:0/2857031712 conn(0x7fe470071020 msgr2=0x7fe4700a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 -- 172.21.15.67:0/2857031712 shutdown_connections 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 1 -- 172.21.15.67:0/2857031712 wait complete. 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.080+0000 7fe47671d640 10 monclient: monmap: 2024-09-15T01:55:32.098 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 Processor -- start 2024-09-15T01:55:32.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 -- start start 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient: init 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 0x7fe470135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-a con 0x7fe458004190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe470136040 0x7fe47011e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-c con 0x7fe470136040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): picked mon.noname-b con 0x7fe47011eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470122260 con 0x7fe458004190 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47006ea30 con 0x7fe47011eb30 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47006dd30 con 0x7fe470136040 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe47671d640 10 monclient(hunting): authenticate will time out at 964.550695s 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46378/0 (socket says 172.21.15.67:46378) 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 1 -- 172.21.15.67:0/2528666782 learned_addr learned my addr 172.21.15.67:0/2528666782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): get_auth_request con 0x7fe47011eb30 auth_method 0 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe474cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46f7fe640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe470136040 0x7fe47011e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 0x7fe470135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46f7fe640 10 monclient(hunting): get_auth_request con 0x7fe470136040 auth_method 0 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.081+0000 7fe46f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe46ffff640 10 monclient(hunting): get_auth_request con 0x7fe458004190 auth_method 0 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe46ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe46ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe46ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 10 monclient(hunting): handle_auth_done global_id 5473 payload 824 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 -- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe470136040 msgr2=0x7fe47011e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe470136040 0x7fe47011e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 -- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 msgr2=0x7fe470135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 0x7fe470135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 monclient: found mon.noname-b 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe474cba640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4701205f0 con 0x7fe47011eb30 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.082+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 0x7fe470135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe474cba640 1 -- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 msgr2=0x7fe47011ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe474cba640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe468001810 tx=0x7fe4680077d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468010660 con 0x7fe47011eb30 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: dump: 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468010800 con 0x7fe47011eb30 2024-09-15T01:55:32.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 5 monclient: authenticate success, global_id 5473 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 10 monclient: _renew_subs 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4701200e0 con 0x7fe47011eb30 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468010bc0 con 0x7fe47011eb30 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe457fff640 10 monclient: dump: 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 10 monclient: _renew_subs 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe46e7fc640 4 set_mon_vals no callback set 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.084+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe470137460 con 0x7fe47011eb30 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe47671d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe468014430 con 0x7fe47011eb30 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe47671d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe47671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47004bf60 con 0x7fe47011eb30 2024-09-15T01:55:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe457fff640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 0x7fe45c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5978+0+0 (secure 0 0 0) 0x7fe46804bd00 con 0x7fe47011eb30 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 0x7fe45c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.085+0000 7fe46ffff640 10 monclient: get_auth_request con 0x7fe45c038770 auth_method 0 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.088+0000 7fe46ffff640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 0x7fe45c03ac20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe460006a70 tx=0x7fe460008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.090+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe46801bb10 con 0x7fe47011eb30 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.090+0000 7fe457fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.090+0000 7fe457fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.260+0000 7fe47671d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.260+0000 7fe47671d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.260+0000 7fe47671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.260+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe470067120 con 0x7fe47011eb30 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.261+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fe46801b770 con 0x7fe47011eb30 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.261+0000 7fe457fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:32.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.261+0000 7fe457fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:32.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe47671d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:32.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe47671d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:32.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe47671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe4700008d0 con 0x7fe47011eb30 2024-09-15T01:55:32.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe457fff640 1 -- 172.21.15.67:0/2528666782 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fe46801b950 con 0x7fe47011eb30 2024-09-15T01:55:32.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe457fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:32.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.360+0000 7fe457fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 msgr2=0x7fe45c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 0x7fe45c03ac20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe460006a70 tx=0x7fe460008040 comp rx=0 tx=0).stop 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 10 monclient: shutdown 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 msgr2=0x7fe47011ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe468001810 tx=0x7fe4680077d0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 shutdown_connections 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe45c038770 0x7fe45c03ac20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47011eb30 0x7fe47011ef80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe470136040 0x7fe47011e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 --2- 172.21.15.67:0/2528666782 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe458004190 0x7fe470135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 >> 172.21.15.67:0/2528666782 conn(0x7fe470071020 msgr2=0x7fe4700a1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:32.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 shutdown_connections 2024-09-15T01:55:32.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.362+0000 7fe47671d640 1 -- 172.21.15.67:0/2528666782 wait complete. 2024-09-15T01:55:32.373 INFO:tasks.workunit.client.0.smithi067.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-15T01:55:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush reweight osd.11 113 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 10 monclient: get_monmap_and_config 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 10 monclient: monmap: 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.451 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:32.450207+0000 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:32.450207+0000 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:32.452 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 1 Processor -- start 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 1 -- start start 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.449+0000 7fa166c34640 10 monclient: init 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa160098140 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1600a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa160098c20 0x7fa16013a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-c con 0x7fa160098c20 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1600a0b80 con 0x7fa160098140 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1601328f0 con 0x7fa1600a0370 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1600a00b0 con 0x7fa160098c20 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa166c34640 10 monclient(hunting): authenticate will time out at 964.919455s 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa160098c20 0x7fa16013a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa15ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa1600a0370 0x7fa160098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49914/0 (socket says 172.21.15.67:49914) 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa15ffff640 1 -- 172.21.15.67:0/3297625940 learned_addr learned my addr 172.21.15.67:0/3297625940 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): get_auth_request con 0x7fa160098c20 auth_method 0 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient(hunting): handle_auth_done global_id 5476 payload 274 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 1 -- 172.21.15.67:0/3297625940 >> v1:172.21.15.67:6789/0 conn(0x7fa1600a0370 legacy=0x7fa160098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 1 -- 172.21.15.67:0/3297625940 >> v1:172.21.15.17:6789/0 conn(0x7fa160098140 legacy=0x7fa1601321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 1 monclient: found mon.noname-c 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.450+0000 7fa1651d1640 1 -- 172.21.15.67:0/3297625940 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160071390 con 0x7fa160098c20 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa1651d1640 1 -- 172.21.15.67:0/3297625940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa160098c20 msgr2=0x7fa16013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa1651d1640 1 --2- 172.21.15.67:0/3297625940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa160098c20 0x7fa16013a0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa158009e60 tx=0x7fa158009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b21c64b8055e64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 1 -- 172.21.15.67:0/3297625940 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158007630 con 0x7fa160098c20 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 10 monclient: dump: 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.460 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa166c34640 5 monclient: authenticate success, global_id 5476 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.453+0000 7fa15f7fe640 1 -- 172.21.15.67:0/3297625940 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15802d050 con 0x7fa160098c20 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:32.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa15f7fe640 1 -- 172.21.15.67:0/3297625940 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1580079c0 con 0x7fa160098c20 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 4 set_mon_vals no callback set 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa15f7fe640 10 monclient: dump: 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 10 monclient: shutdown 2024-09-15T01:55:32.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 -- 172.21.15.67:0/3297625940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa160098c20 msgr2=0x7fa16013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 --2- 172.21.15.67:0/3297625940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa160098c20 0x7fa16013a0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa158009e60 tx=0x7fa158009780 comp rx=0 tx=0).stop 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 -- 172.21.15.67:0/3297625940 shutdown_connections 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 --2- 172.21.15.67:0/3297625940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa160098c20 0x7fa16013a0d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 -- 172.21.15.67:0/3297625940 >> 172.21.15.67:0/3297625940 conn(0x7fa1600a5780 msgr2=0x7fa1600a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 -- 172.21.15.67:0/3297625940 shutdown_connections 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.454+0000 7fa166c34640 1 -- 172.21.15.67:0/3297625940 wait complete. 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 10 monclient: monmap: 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:32.463 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 1 Processor -- start 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 1 -- start start 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 10 monclient: init 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.455+0000 7fa166c34640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa160098140 0x7fa16013da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-c con 0x7fa160098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa160098c20 0x7fa1601395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-a con 0x7fa160098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:32.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): picked mon.noname-b con 0x7fa1600a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa160133d90 con 0x7fa160098c20 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1600a3c60 con 0x7fa1600a0370 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa160131f40 con 0x7fa160098140 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa166c34640 10 monclient(hunting): authenticate will time out at 964.925099s 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46406/0 (socket says 172.21.15.67:46406) 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 -- 172.21.15.67:0/1204181629 learned_addr learned my addr 172.21.15.67:0/1204181629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): get_auth_request con 0x7fa1600a0370 auth_method 0 2024-09-15T01:55:32.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa160098c20 0x7fa1601395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1649d0640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa160098140 0x7fa16013da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): get_auth_request con 0x7fa160098c20 auth_method 0 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1649d0640 10 monclient(hunting): get_auth_request con 0x7fa160098140 auth_method 0 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1649d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1649d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1649d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa15ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient(hunting): handle_auth_done global_id 5479 payload 824 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 -- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa160098140 msgr2=0x7fa16013da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa160098140 0x7fa16013da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 -- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa160098c20 msgr2=0x7fa1601395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa160098c20 0x7fa1601395e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 monclient: found mon.noname-b 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.456+0000 7fa1651d1640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158009a20 con 0x7fa1600a0370 2024-09-15T01:55:32.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa1651d1640 1 -- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 msgr2=0x7fa160140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa1651d1640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa1580099f0 tx=0x7fa158037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158007ec0 con 0x7fa1600a0370 2024-09-15T01:55:32.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: dump: 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa158040070 con 0x7fa1600a0370 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158031cf0 con 0x7fa1600a0370 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15cff9640 10 monclient: dump: 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:32.471 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa166c34640 5 monclient: authenticate success, global_id 5479 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa166c34640 10 monclient: _renew_subs 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa15effd640 4 set_mon_vals no callback set 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa166c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.458+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1601337d0 con 0x7fa1600a0370 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.459+0000 7fa166c34640 10 monclient: _renew_subs 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.459+0000 7fa166c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.459+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16013e4f0 con 0x7fa1600a0370 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.459+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa158042030 con 0x7fa1600a0370 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa166c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa166c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa166c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1601340a0 con 0x7fa1600a0370 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa15cff9640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 0x7fa14403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6036+0+0 (secure 0 0 0) 0x7fa15806d830 con 0x7fa1600a0370 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa1649d0640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 0x7fa14403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.460+0000 7fa1649d0640 10 monclient: get_auth_request con 0x7fa144038740 auth_method 0 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.462+0000 7fa1649d0640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 0x7fa14403abf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa1540037d0 tx=0x7fa154009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.464+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa15803b420 con 0x7fa1600a0370 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.464+0000 7fa15cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.464+0000 7fa15cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:32.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.633+0000 7fa166c34640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:32.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.633+0000 7fa166c34640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:32.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.633+0000 7fa166c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:32.633+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fa1601340a0 con 0x7fa1600a0370 2024-09-15T01:55:32.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:32.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.927+0000 7f0a9271d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:32.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.927+0000 7f0a9271d640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.927+0000 7f0a9271d640 10 monclient: monmap: 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:32.928608+0000 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:32.928608+0000 2024-09-15T01:55:32.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:32.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.928+0000 7f0a9271d640 1 Processor -- start 2024-09-15T01:55:32.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.928+0000 7f0a9271d640 1 -- start start 2024-09-15T01:55:32.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.928+0000 7f0a9271d640 10 monclient: init 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a8c09c2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a8c09c690 0x7f0a8c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a8c09c690 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a8c06f070 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:32.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8c06a5c0 con 0x7f0a8c06f070 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c099f50 con 0x7f0a8c09c690 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c06a300 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a9271d640 10 monclient(hunting): authenticate will time out at 992.925167s 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54638/0 (socket says 172.21.15.17:54638) 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 -- 172.21.15.17:0/2166076473 learned_addr learned my addr 172.21.15.17:0/2166076473 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 10 monclient(hunting): get_auth_request con 0x7f0a8c09c2c0 auth_method 0 2024-09-15T01:55:32.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/2166076473 >> v2:172.21.15.67:3300/0 conn(0x7f0a8c09c690 0x7f0a8c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a8c09c690 auth_method 0 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 -- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 msgr2=0x7f0a8c09a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:32.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 -- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 msgr2=0x7f0a8c09a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8bfff640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_done global_id 5482 payload 274 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 msgr2=0x7f0a8c09a920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/2166076473 >> v1:172.21.15.67:6789/0 conn(0x7f0a8c06f070 legacy=0x7f0a8c09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:55:32.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:32.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.929+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/2166076473 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c068ac0 con 0x7f0a8c09c690 2024-09-15T01:55:32.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/2166076473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c09c690 msgr2=0x7f0a8c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c09c690 0x7f0a8c06eb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009850 tx=0x7f0a800090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6db672d013e6b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 1 -- 172.21.15.17:0/2166076473 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8002adf0 con 0x7f0a8c09c690 2024-09-15T01:55:32.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: dump: 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 1 -- 172.21.15.17:0/2166076473 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80032440 con 0x7f0a8c09c690 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 1 -- 172.21.15.17:0/2166076473 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a800327e0 con 0x7f0a8c09c690 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a9271d640 5 monclient: authenticate success, global_id 5482 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a9271d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a9271d640 4 set_mon_vals no callback set 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.932+0000 7f0a8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:32.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a8affd640 10 monclient: dump: 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 10 monclient: shutdown 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 -- 172.21.15.17:0/2166076473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c09c690 msgr2=0x7f0a8c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c09c690 0x7f0a8c06eb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009850 tx=0x7f0a800090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 -- 172.21.15.17:0/2166076473 shutdown_connections 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c09c690 0x7f0a8c06eb30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 --2- 172.21.15.17:0/2166076473 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c2c0 0x7f0a8c09a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 -- 172.21.15.17:0/2166076473 >> 172.21.15.17:0/2166076473 conn(0x7f0a8c0a2f70 msgr2=0x7f0a8c0a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 -- 172.21.15.17:0/2166076473 shutdown_connections 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.933+0000 7f0a9271d640 1 -- 172.21.15.17:0/2166076473 wait complete. 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.934+0000 7f0a9271d640 10 monclient: build_initial_monmap 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.934+0000 7f0a9271d640 10 monclient: monmap: 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:32.947 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.934+0000 7f0a9271d640 1 Processor -- start 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.934+0000 7f0a9271d640 1 -- start start 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.934+0000 7f0a9271d640 10 monclient: init 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c06f070 0x7f0a8c142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a8c06f070 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a8c09c2c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a8c09c690 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c127270 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c11cf30 con 0x7f0a8c06f070 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:32.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c09efd0 con 0x7f0a8c09c690 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a9271d640 10 monclient(hunting): authenticate will time out at 992.931121s 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53800/0 (socket says 172.21.15.17:53800) 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/1555192560 learned_addr learned my addr 172.21.15.17:0/1555192560 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a8c09c2c0 auth_method 0 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 10 monclient(hunting): get_auth_request con 0x7f0a8c09c690 auth_method 0 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 msgr2=0x7f0a8c145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 msgr2=0x7f0a8c145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a90cba640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8bfff640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c06f070 0x7f0a8c142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8bfff640 10 monclient(hunting): get_auth_request con 0x7f0a8c06f070 auth_method 0 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient(hunting): handle_auth_done global_id 44820 payload 824 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 msgr2=0x7f0a8c145a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c06f070 msgr2=0x7f0a8c142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c06f070 0x7f0a8c142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.935+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/1555192560 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a80009390 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.937+0000 7f0a8b7fe640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 msgr2=0x7f0a8c141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:32.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.937+0000 7f0a8b7fe640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098b0 tx=0x7f0a80032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80031a40 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: dump: 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.953 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 5 monclient: authenticate success, global_id 44820 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 10 monclient: _renew_subs 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80031ba0 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c11c0b0 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8003c400 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a73fff640 10 monclient: dump: 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:32.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 10 monclient: _renew_subs 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c142fd0 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.938+0000 7f0a8a7fc640 4 set_mon_vals no callback set 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a8003c830 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a9271d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a9271d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a9271d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c126dc0 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a73fff640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 0x7f0a7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f0a8006c930 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a8bfff640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 0x7f0a7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:32.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.939+0000 7f0a8bfff640 10 monclient: get_auth_request con 0x7f0a78038740 auth_method 0 2024-09-15T01:55:32.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.942+0000 7f0a8bfff640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 0x7f0a7803abf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c09cfd0 tx=0x7f0a7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:32.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.944+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0a80042ae0 con 0x7f0a8c09c2c0 2024-09-15T01:55:32.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.944+0000 7f0a73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:32.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:32.944+0000 7f0a73fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.171+0000 7f0a9271d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.171+0000 7f0a9271d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.171+0000 7f0a9271d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.171+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0a8c0724a0 con 0x7f0a8c09c2c0 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.172+0000 7f0a73fff640 1 -- 172.21.15.17:0/1555192560 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f0a80037070 con 0x7f0a8c09c2c0 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.172+0000 7f0a73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.172+0000 7f0a73fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:33.175 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:33.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 msgr2=0x7f0a7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 0x7f0a7803abf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c09cfd0 tx=0x7f0a7c008040 comp rx=0 tx=0).stop 2024-09-15T01:55:33.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 10 monclient: shutdown 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 msgr2=0x7f0a8c141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098b0 tx=0x7f0a80032000 comp rx=0 tx=0).stop 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 shutdown_connections 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a78038740 0x7f0a7803abf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a8c09c2c0 0x7f0a8c141c10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a8c09c690 0x7f0a8c145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 --2- 172.21.15.17:0/1555192560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c06f070 0x7f0a8c142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 >> 172.21.15.17:0/1555192560 conn(0x7f0a8c0a2f70 msgr2=0x7f0a8c06daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 shutdown_connections 2024-09-15T01:55:33.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.174+0000 7f0a9271d640 1 -- 172.21.15.17:0/1555192560 wait complete. 2024-09-15T01:55:33.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:55:33.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:55:33.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:55:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.209+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v101) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa1580330c0 con 0x7fa1600a0370 2024-09-15T01:55:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.209+0000 7fa15cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.209+0000 7fa15cff9640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T01:55:33.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.283+0000 7f35e3ec9640 10 monclient: get_monmap_and_config 2024-09-15T01:55:33.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.283+0000 7f35e3ec9640 10 monclient: build_initial_monmap 2024-09-15T01:55:33.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.283+0000 7f35e3ec9640 10 monclient: monmap: 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:33.284763+0000 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:33.284763+0000 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:33.287 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:33.288 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:33.288 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.284+0000 7f35e3ec9640 1 Processor -- start 2024-09-15T01:55:33.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.284+0000 7f35e3ec9640 1 -- start start 2024-09-15T01:55:33.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.284+0000 7f35e3ec9640 10 monclient: init 2024-09-15T01:55:33.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:33.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T01:55:33.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35dc1319a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:33.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35dc09eac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:33.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f35dc137ce0 0x7f35dc13a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f35dc137ce0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc0a1530 con 0x7f35dc09eac0 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc09fca0 con 0x7f35dc1319a0 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc0a00e0 con 0x7f35dc137ce0 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e3ec9640 10 monclient(hunting): authenticate will time out at 993.281292s 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e1464640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f35dc09eac0 0x7f35dc09eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35988/0 (socket says 172.21.15.17:35988) 2024-09-15T01:55:33.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e1464640 1 -- 172.21.15.17:0/2818593192 learned_addr learned my addr 172.21.15.17:0/2818593192 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f35dc137ce0 0x7f35dc13a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): get_auth_request con 0x7f35dc137ce0 auth_method 0 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient(hunting): handle_auth_done global_id 44823 payload 274 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 1 -- 172.21.15.17:0/2818593192 >> v1:172.21.15.67:6789/0 conn(0x7f35dc1319a0 legacy=0x7f35dc131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 1 -- 172.21.15.17:0/2818593192 >> v1:172.21.15.17:6789/0 conn(0x7f35dc09eac0 legacy=0x7f35dc09eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient: _finish_hunting 0 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 1 monclient: found mon.noname-a 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.285+0000 7f35e2466640 1 -- 172.21.15.17:0/2818593192 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0a1250 con 0x7f35dc137ce0 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.288+0000 7f35e2466640 1 -- 172.21.15.17:0/2818593192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc137ce0 msgr2=0x7f35dc13a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.288+0000 7f35e2466640 1 --2- 172.21.15.17:0/2818593192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc137ce0 0x7f35dc13a0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009850 tx=0x7f35d40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa5ac672d1d9101 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.288+0000 7f35e0c63640 1 -- 172.21.15.17:0/2818593192 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4032440 con 0x7f35dc137ce0 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: dump: 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.301 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: _finish_auth 0 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 1 -- 172.21.15.17:0/2818593192 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d40325a0 con 0x7f35dc137ce0 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 5 monclient: authenticate success, global_id 44823 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 4 set_mon_vals no callback set 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 1 -- 172.21.15.17:0/2818593192 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4032940 con 0x7f35dc137ce0 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e0c63640 10 monclient: dump: 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 10 monclient: shutdown 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 -- 172.21.15.17:0/2818593192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc137ce0 msgr2=0x7f35dc13a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/2818593192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc137ce0 0x7f35dc13a0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009850 tx=0x7f35d40090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 -- 172.21.15.17:0/2818593192 shutdown_connections 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/2818593192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc137ce0 0x7f35dc13a0c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 -- 172.21.15.17:0/2818593192 >> 172.21.15.17:0/2818593192 conn(0x7f35dc0a5000 msgr2=0x7f35dc0a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 -- 172.21.15.17:0/2818593192 shutdown_connections 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.289+0000 7f35e3ec9640 1 -- 172.21.15.17:0/2818593192 wait complete. 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 10 monclient: build_initial_monmap 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 10 monclient: monmap: 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 1 Processor -- start 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 1 -- start start 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 10 monclient: init 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.290+0000 7f35e3ec9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 0x7f35dc09b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f35dc09eac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 0x7f35dc096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f35dc1319a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35dc137ce0 0x7f35dc096f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f35dc137ce0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc099610 con 0x7f35dc09eac0 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc09bf50 con 0x7f35dc137ce0 2024-09-15T01:55:33.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc098060 con 0x7f35dc1319a0 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e3ec9640 10 monclient(hunting): authenticate will time out at 993.287082s 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 0x7f35dc09b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 0x7f35dc096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 0x7f35dc09b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53826/0 (socket says 172.21.15.17:53826) 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 -- 172.21.15.17:0/3005215770 learned_addr learned my addr 172.21.15.17:0/3005215770 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): get_auth_request con 0x7f35dc09eac0 auth_method 0 2024-09-15T01:55:33.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 10 monclient(hunting): get_auth_request con 0x7f35dc1319a0 auth_method 0 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35dc137ce0 0x7f35dc096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 msgr2=0x7f35dc096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:55:33.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 msgr2=0x7f35dc096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 0x7f35dc096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): get_auth_request con 0x7f35dc137ce0 auth_method 0 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 0x7f35dc096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e2466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient(hunting): handle_auth_done global_id 44826 payload 824 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 msgr2=0x7f35dc096a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f35dc1319a0 0x7f35dc096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35dc137ce0 msgr2=0x7f35dc096f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35dc137ce0 0x7f35dc096f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient: _finish_hunting 0 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 monclient: found mon.noname-a 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.291+0000 7f35e1c65640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d4009390 con 0x7f35dc09eac0 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e1c65640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 msgr2=0x7f35dc09b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e1c65640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 0x7f35dc09b440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00c8b0 tx=0x7f35cc0026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc00cea0 con 0x7f35dc09eac0 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: dump: 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.312 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc00aa60 con 0x7f35dc09eac0 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e3ec9640 5 monclient: authenticate success, global_id 44826 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e3ec9640 10 monclient: _renew_subs 2024-09-15T01:55:33.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e3ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc13a0d0 con 0x7f35dc09eac0 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc00ae20 con 0x7f35dc09eac0 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35d9ffb640 10 monclient: dump: 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.314 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.293+0000 7f35dbfff640 4 set_mon_vals no callback set 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 10 monclient: _renew_subs 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc09aa40 con 0x7f35dc09eac0 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:33.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 0x7f35dc097490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f35cc012630 con 0x7f35dc09eac0 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.294+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc063b00 con 0x7f35dc098720 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 0x7f35dc097490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient: get_auth_request con 0x7f35dc098720 auth_method 0 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35d9ffb640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35c8038740 0x7f35c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f35cc04c400 con 0x7f35dc09eac0 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 10 monclient(hunting): handle_auth_done global_id 44829 payload 824 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35dc099a80 con 0x7f35dc098720 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e2466640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35c8038740 0x7f35c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.295+0000 7f35e2466640 10 monclient: get_auth_request con 0x7f35c8038740 auth_method 0 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 msgr2=0x7f35dc097490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 0x7f35dc097490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35dc137a40 tx=0x7f35d0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0004e60 con 0x7f35dc098720 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 10 monclient: dump: 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.318 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35e2466640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35c8038740 0x7f35c803abf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f35dc09fb80 tx=0x7f35d4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f35d0007170 con 0x7f35dc098720 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 msgr2=0x7f35dc097490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 0x7f35dc097490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35dc137a40 tx=0x7f35d0009040 comp rx=0 tx=0).stop 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 msgr2=0x7f35dc097490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:33.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.297+0000 7f35d9ffb640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc098720 0x7f35dc097490 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35dc137a40 tx=0x7f35d0009040 comp rx=0 tx=0).stop 2024-09-15T01:55:33.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.314+0000 7fa166c34640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:33.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.314+0000 7fa166c34640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:33.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.314+0000 7fa166c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:33.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:33.314+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fa1601340a0 con 0x7fa1600a0370 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e3ec9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 0x7f35dc132e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e3ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc0724a0 con 0x7f35dc132a20 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 0x7f35dc132e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient: get_auth_request con 0x7f35dc132a20 auth_method 0 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.303+0000 7f35e1464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.304+0000 7f35e1464640 10 monclient(hunting): handle_auth_done global_id 44832 payload 824 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.304+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f35dc099830 con 0x7f35dc132a20 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35e1464640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 msgr2=0x7f35dc132e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35e1464640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 0x7f35dc132e20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35d000cfd0 tx=0x7f35d001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0020600 con 0x7f35dc132a20 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 10 monclient: dump: 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.323 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f35d0020760 con 0x7f35dc132a20 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 msgr2=0x7f35dc132e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 0x7f35dc132e20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35d000cfd0 tx=0x7f35d001a000 comp rx=0 tx=0).stop 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 msgr2=0x7f35dc132e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:33.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.306+0000 7f35d9ffb640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc132a20 0x7f35dc132e20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:33.325 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:33.326 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:33.327 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:33.327 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:33.327 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:33.328 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:33.328 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:33.329 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:33.329 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:33.329 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:33.330 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:33.331 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:33.332 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:33.333 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:33.334 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:33.336 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:33.337 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:33.338 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:33.338 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:33.339 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.341 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:33.343 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:33.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35c8038740 msgr2=0x7f35c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35c8038740 0x7f35c803abf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f35dc09fb80 tx=0x7f35d4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 10 monclient: shutdown 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 msgr2=0x7f35dc09b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 --2- 172.21.15.17:0/3005215770 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f35dc09eac0 0x7f35dc09b440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00c8b0 tx=0x7f35cc0026c0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e2466640 1 -- 172.21.15.17:0/3005215770 reap_dead start 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e2466640 1 -- 172.21.15.17:0/3005215770 reap_dead start 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 shutdown_connections 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 >> 172.21.15.17:0/3005215770 conn(0x7f35dc0a5000 msgr2=0x7f35dc0a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 shutdown_connections 2024-09-15T01:55:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.307+0000 7f35e3ec9640 1 -- 172.21.15.17:0/3005215770 wait complete. 2024-09-15T01:55:33.346 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:33.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:55:33.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:55:33.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.441+0000 7f4fe52af640 10 monclient: get_monmap_and_config 2024-09-15T01:55:33.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.441+0000 7f4fe52af640 10 monclient: build_initial_monmap 2024-09-15T01:55:33.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.441+0000 7f4fe52af640 10 monclient: monmap: 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:33.442460+0000 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:33.442460+0000 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:33.444 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 Processor -- start 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 -- start start 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient: init 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fe006eb30 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fe009bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fe009c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe009f6a0 con 0x7f4fe006eb30 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe0069f40 con 0x7f4fe009c4d0 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe009ed40 con 0x7f4fe009e970 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fe52af640 10 monclient(hunting): authenticate will time out at 993.438776s 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fde7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fe009bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fdeffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4fe006eb30 0x7f4fe006ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35998/0 (socket says 172.21.15.17:35998) 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 learned_addr learned my addr 172.21.15.17:0/2624447863 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request con 0x7f4fe009e970 auth_method 0 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.442+0000 7f4fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.443+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 msgr2=0x7f4fe009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:33.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.443+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 msgr2=0x7f4fe009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:33.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.443+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fe009bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:33.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.443+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fe009bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:33.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 >> v1:172.21.15.17:6789/0 conn(0x7f4fe006eb30 legacy=0x7f4fe006ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651757967 0 0) 0x7f4fe009f6a0 con 0x7f4fe006eb30 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): my global_id is 44835 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc80034e0 con 0x7f4fe006eb30 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 1 -- 172.21.15.17:0/2624447863 >> v1:172.21.15.67:6789/0 conn(0x7f4fe009c4d0 legacy=0x7f4fe0139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966525775 0 0) 0x7f4fe0069f40 con 0x7f4fe009c4d0 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): my global_id is 5491 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe009f6a0 con 0x7f4fe009c4d0 2024-09-15T01:55:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294132014 0 0) 0x7f4fc80034e0 con 0x7f4fe006eb30 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 msgr2=0x7f4fe009bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fe009bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 >> v1:172.21.15.67:6789/0 conn(0x7f4fe009c4d0 legacy=0x7f4fe0139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: _finish_hunting 0 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0069950 con 0x7f4fe006eb30 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4fd0001720 con 0x7f4fe006eb30 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: dump: 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:55:33.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 >> v1:172.21.15.17:6789/0 conn(0x7f4fe006eb30 legacy=0x7f4fe006ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 0x7f4fc8004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): picked mon.a con 0x7f4fc8003c10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc8004780 0x7f4fc8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): picked mon.b con 0x7f4fc8004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fc800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): picked mon.c con 0x7f4fe009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe009ed40 con 0x7f4fc8003c10 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8005550 con 0x7f4fc8004780 2024-09-15T01:55:33.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc800bf10 con 0x7f4fe009e970 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fddffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 0x7f4fc8004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fc800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request con 0x7f4fc8003c10 auth_method 0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc8004780 0x7f4fc8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4fe009e970 auth_method 0 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request con 0x7f4fc8004780 auth_method 0 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_done global_id 44835 payload 293 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 msgr2=0x7f4fc800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe009e970 0x7f4fc800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc8004780 msgr2=0x7f4fc8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc8004780 0x7f4fc8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 monclient: found mon.a 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80034e0 con 0x7f4fc8003c10 2024-09-15T01:55:33.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fe52af640 5 monclient: authenticate success, global_id 44835 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.445+0000 7f4fe52af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 msgr2=0x7f4fc8004000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 0x7f4fc8004000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0001490 tx=0x7f4fd0007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99afd9b86940c205 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0005890 con 0x7f4fc8003c10 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: dump: 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.462 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fe52af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0005a10 con 0x7f4fc8003c10 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 1 -- 172.21.15.17:0/2624447863 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0005dd0 con 0x7f4fc8003c10 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fe52af640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fe52af640 4 set_mon_vals no callback set 2024-09-15T01:55:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.448+0000 7f4fddffb640 10 monclient: dump: 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 10 monclient: shutdown 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 -- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 msgr2=0x7f4fc8004000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 --2- 172.21.15.17:0/2624447863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fc8003c10 0x7f4fc8004000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0001490 tx=0x7f4fd0007a50 comp rx=0 tx=0).stop 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fdf7fe640 1 -- 172.21.15.17:0/2624447863 reap_dead start 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 -- 172.21.15.17:0/2624447863 shutdown_connections 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 -- 172.21.15.17:0/2624447863 >> 172.21.15.17:0/2624447863 conn(0x7f4fe00712f0 msgr2=0x7f4fe00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 -- 172.21.15.17:0/2624447863 shutdown_connections 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.449+0000 7f4fe52af640 1 -- 172.21.15.17:0/2624447863 wait complete. 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient: build_initial_monmap 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient: monmap: 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:33.465 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 1 Processor -- start 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 1 -- start start 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient: init 2024-09-15T01:55:33.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-c con 0x7f4fc8003c10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.450+0000 7f4fe52af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 0x7f4fe0133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe01413d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe0134200 0x7f4fe011e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe0134200 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe009cd10 con 0x7f4fe01413d0 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe01223b0 con 0x7f4fe0134200 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe01220d0 con 0x7f4fc8003c10 2024-09-15T01:55:33.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fe52af640 10 monclient(hunting): authenticate will time out at 993.447071s 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 0x7f4fe0133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54692/0 (socket says 172.21.15.17:54692) 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 learned_addr learned my addr 172.21.15.17:0/2182909114 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request con 0x7f4fc8003c10 auth_method 0 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request con 0x7f4fe01413d0 auth_method 0 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 msgr2=0x7f4fe0140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 msgr2=0x7f4fe0140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe0134200 0x7f4fe011e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4fe0134200 auth_method 0 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient(hunting): handle_auth_done global_id 44838 payload 824 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 msgr2=0x7f4fe0140e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fc8003c10 0x7f4fe0140e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe0134200 msgr2=0x7f4fe011e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe0134200 0x7f4fe011e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 monclient: found mon.noname-a 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:33.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.451+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0141cf0 con 0x7f4fe01413d0 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fde7fc640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 msgr2=0x7f4fe0133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fde7fc640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 0x7f4fe0133cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4006fd0 tx=0x7f4fd4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400ec70 con 0x7f4fe01413d0 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: dump: 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.473 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd400edd0 con 0x7f4fe01413d0 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 5 monclient: authenticate success, global_id 44838 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 10 monclient: _renew_subs 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe0121df0 con 0x7f4fe01413d0 2024-09-15T01:55:33.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40166b0 con 0x7f4fe01413d0 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fc6ffd640 10 monclient: dump: 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:33.475 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 10 monclient: _renew_subs 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe01210a0 con 0x7f4fe01413d0 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.453+0000 7f4fdd7fa640 4 set_mon_vals no callback set 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4fd4016a20 con 0x7f4fe01413d0 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fe52af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fe52af640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe0063b90 con 0x7f4fe011ece0 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fc6ffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fcc0387c0 0x7f4fcc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:33.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fdeffd640 10 monclient: get_auth_request con 0x7f4fe011ece0 auth_method 0 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.454+0000 7f4fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f4fd404cbe0 con 0x7f4fe01413d0 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdf7fe640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fcc0387c0 0x7f4fcc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.455+0000 7f4fdf7fe640 10 monclient: get_auth_request con 0x7f4fcc0387c0 auth_method 0 2024-09-15T01:55:33.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.457+0000 7f4fdf7fe640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fcc0387c0 0x7f4fcc03ac70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4fe009c8a0 tx=0x7f4fd8001a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:33.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 10 monclient: get_auth_request con 0x7f4fe011ece0 auth_method 0 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:33.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:33.656+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.057+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:55:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.057+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.057+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:34.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.057+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.207+0000 7fa15cff9640 1 -- 172.21.15.67:0/1204181629 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v102) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa1580443d0 con 0x7fa1600a0370 2024-09-15T01:55:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.207+0000 7fa15cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T01:55:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.207+0000 7fa15cff9640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T01:55:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 msgr2=0x7fa14403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 0x7fa14403abf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa1540037d0 tx=0x7fa154009040 comp rx=0 tx=0).stop 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 10 monclient: shutdown 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 msgr2=0x7fa160140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa1580099f0 tx=0x7fa158037000 comp rx=0 tx=0).stop 2024-09-15T01:55:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 shutdown_connections 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa144038740 0x7fa14403abf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa160098c20 0x7fa1601395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1600a0370 0x7fa160140f80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 --2- 172.21.15.67:0/1204181629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa160098140 0x7fa16013da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 >> 172.21.15.67:0/1204181629 conn(0x7fa1600a5780 msgr2=0x7fa16009eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 shutdown_connections 2024-09-15T01:55:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.210+0000 7fa166c34640 1 -- 172.21.15.67:0/1204181629 wait complete. 2024-09-15T01:55:34.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-15T01:55:34.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:55:34.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-15T01:55:34.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.304+0000 7f3d4f15d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:34.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.304+0000 7f3d4f15d640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.304+0000 7f3d4f15d640 10 monclient: monmap: 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:34.305432+0000 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:34.305432+0000 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:34.307 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 Processor -- start 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 -- start start 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient: init 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d480a22f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d48136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d4806e160 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d4806e5d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d480710b0 con 0x7f3d480a22f0 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4812f550 con 0x7f3d4806e5d0 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812ee80 con 0x7f3d4806e160 2024-09-15T01:55:34.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4f15d640 10 monclient(hunting): authenticate will time out at 966.774630s 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4d6fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d4806e5d0 0x7f3d48139480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49918/0 (socket says 172.21.15.67:49918) 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 learned_addr learned my addr 172.21.15.67:0/1794984997 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d47fff640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d48136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d47fff640 10 monclient(hunting): get_auth_request con 0x7f3d4806e160 auth_method 0 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.305+0000 7f3d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.306+0000 7f3d47fff640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 msgr2=0x7f3d48136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.306+0000 7f3d47fff640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 msgr2=0x7f3d48136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.306+0000 7f3d47fff640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d48136f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.306+0000 7f3d47fff640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d48136f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:34.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.307+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 >> v1:172.21.15.67:6789/0 conn(0x7f3d4806e5d0 legacy=0x7f3d48139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310749105 0 0) 0x7f3d400017d0 con 0x7f3d4806e5d0 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): my global_id is 5497 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 1 -- 172.21.15.67:0/1794984997 >> v1:172.21.15.17:6789/0 conn(0x7f3d480a22f0 legacy=0x7f3d4812e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d380034e0 con 0x7f3d4806e5d0 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415382 0 0) 0x7f3d480710b0 con 0x7f3d480a22f0 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): my global_id is 44841 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d400017d0 con 0x7f3d480a22f0 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814702957 0 0) 0x7f3d380034e0 con 0x7f3d4806e5d0 2024-09-15T01:55:34.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 msgr2=0x7f3d48136f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d48136f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 >> v1:172.21.15.17:6789/0 conn(0x7f3d480a22f0 legacy=0x7f3d4812e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d480a26c0 con 0x7f3d4806e5d0 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f3d40001c30 con 0x7f3d4806e5d0 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: dump: 2024-09-15T01:55:34.316 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 >> v1:172.21.15.67:6789/0 conn(0x7f3d4806e5d0 legacy=0x7f3d48139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d38003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): picked mon.c con 0x7f3d4806e160 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d38004190 0x7f3d38004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): picked mon.a con 0x7f3d38004190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d38005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): picked mon.b con 0x7f3d38004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812ee80 con 0x7f3d38004190 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3800bd00 con 0x7f3d38004de0 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3800bec0 con 0x7f3d4806e160 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d477fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d38005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): get_auth_request con 0x7f3d38004de0 auth_method 0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d38003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d38004190 0x7f3d38004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): get_auth_request con 0x7f3d4806e160 auth_method 0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): get_auth_request con 0x7f3d38004190 auth_method 0 2024-09-15T01:55:34.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient(hunting): handle_auth_done global_id 5497 payload 293 2024-09-15T01:55:34.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 msgr2=0x7f3d38003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806e160 0x7f3d38003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d38004190 msgr2=0x7f3d38004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d38004190 0x7f3d38004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 1 monclient: found mon.b 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.308+0000 7f3d4d6fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.309+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380034e0 con 0x7f3d38004de0 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.309+0000 7f3d4d6fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.309+0000 7f3d4d6fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.309+0000 7f3d4f15d640 5 monclient: authenticate success, global_id 5497 2024-09-15T01:55:34.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.309+0000 7f3d4f15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:34.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.311+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 msgr2=0x7f3d38005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.311+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d38005250 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3d400019a0 tx=0x7f3d4806f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30ac6ac6cc7b67c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4002bb30 con 0x7f3d38004de0 2024-09-15T01:55:34.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: dump: 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4002bcb0 con 0x7f3d38004de0 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 1 -- 172.21.15.67:0/1794984997 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40007320 con 0x7f3d38004de0 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 4 set_mon_vals no callback set 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d477fe640 10 monclient: dump: 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 10 monclient: shutdown 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 -- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 msgr2=0x7f3d38005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 --2- 172.21.15.67:0/1794984997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d38005250 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3d400019a0 tx=0x7f3d4806f720 comp rx=0 tx=0).stop 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/1794984997 reap_dead start 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 -- 172.21.15.67:0/1794984997 shutdown_connections 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 -- 172.21.15.67:0/1794984997 >> 172.21.15.67:0/1794984997 conn(0x7f3d48067ae0 msgr2=0x7f3d480a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 -- 172.21.15.67:0/1794984997 shutdown_connections 2024-09-15T01:55:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.312+0000 7f3d4f15d640 1 -- 172.21.15.67:0/1794984997 wait complete. 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.313+0000 7f3d4f15d640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.313+0000 7f3d4f15d640 10 monclient: monmap: 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.313+0000 7f3d4f15d640 1 Processor -- start 2024-09-15T01:55:34.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.313+0000 7f3d4f15d640 1 -- start start 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.313+0000 7f3d4f15d640 10 monclient: init 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d481415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d38004de0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d48141b00 0x7f3d4806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d48141b00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 0x7f3d4806b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d4806b500 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4809cf20 con 0x7f3d48141b00 2024-09-15T01:55:34.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812e250 con 0x7f3d38004de0 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48118680 con 0x7f3d4806b500 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4f15d640 10 monclient(hunting): authenticate will time out at 966.783285s 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d481415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d481415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46422/0 (socket says 172.21.15.67:46422) 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 -- 172.21.15.67:0/3659738403 learned_addr learned my addr 172.21.15.67:0/3659738403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): get_auth_request con 0x7f3d38004de0 auth_method 0 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 0x7f3d4806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d48141b00 0x7f3d4806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 10 monclient(hunting): get_auth_request con 0x7f3d4806b500 auth_method 0 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): get_auth_request con 0x7f3d48141b00 auth_method 0 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 msgr2=0x7f3d4806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 1 -- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 msgr2=0x7f3d4806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient(hunting): handle_auth_done global_id 5500 payload 824 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 0x7f3d4806b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4d6fa640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 0x7f3d4806b930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 -- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 msgr2=0x7f3d4806b930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d4806b500 0x7f3d4806b930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 -- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d48141b00 msgr2=0x7f3d4806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d48141b00 0x7f3d4806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 monclient: found mon.noname-b 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.314+0000 7f3d4cef9640 1 -- 172.21.15.67:0/3659738403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4806ca10 con 0x7f3d38004de0 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.316+0000 7f3d4cef9640 1 -- 172.21.15.67:0/3659738403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 msgr2=0x7f3d481415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.316+0000 7f3d4cef9640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004de0 0x7f3d481415c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00f860 tx=0x7f3d3c001b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00fe50 con 0x7f3d38004de0 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: dump: 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.336 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c005ba0 con 0x7f3d38004de0 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 5 monclient: authenticate success, global_id 5500 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00d730 con 0x7f3d38004de0 2024-09-15T01:55:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d44ff9640 10 monclient: dump: 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 10 monclient: _renew_subs 2024-09-15T01:55:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d46ffd640 4 set_mon_vals no callback set 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 1 -- 172.21.15.67:0/3659738403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d481183a0 con 0x7f3d38004de0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 10 monclient: _renew_subs 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.317+0000 7f3d4f15d640 1 -- 172.21.15.67:0/3659738403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4812b8a0 con 0x7f3d38004de0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3d3c00daa0 con 0x7f3d38004de0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d4f15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d4f15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d4f15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d4f15d640 1 -- 172.21.15.67:0/3659738403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d481187f0 con 0x7f3d38004de0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d44ff9640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d2c038740 0x7f3d2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f3d3c04bff0 con 0x7f3d38004de0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d47fff640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d2c038740 0x7f3d2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.318+0000 7f3d47fff640 10 monclient: get_auth_request con 0x7f3d2c038740 auth_method 0 2024-09-15T01:55:34.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.320+0000 7f3d47fff640 1 --2- 172.21.15.67:0/3659738403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d2c038740 0x7f3d2c03abf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3d34006a70 tx=0x7f3d34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.322+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3d3c012dc0 con 0x7f3d38004de0 2024-09-15T01:55:34.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.322+0000 7f3d44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.322+0000 7f3d44ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:34.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.491+0000 7f3d4f15d640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T01:55:34.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.491+0000 7f3d4f15d640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T01:55:34.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.491+0000 7f3d4f15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.491+0000 7f3d4f15d640 1 -- 172.21.15.67:0/3659738403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) v1 -- 0x7f3d4804bf60 con 0x7f3d38004de0 2024-09-15T01:55:34.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.492+0000 7f3d44ff9640 1 -- 172.21.15.67:0/3659738403 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v102) v1 ==== 146+0+0 (secure 0 0 0) 0x7f3d3c01ba60 con 0x7f3d38004de0 2024-09-15T01:55:34.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.492+0000 7f3d44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T01:55:34.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.492+0000 7f3d44ff9640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-15T01:55:34.495 INFO:tasks.workunit.client.0.smithi067.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-15T01:55:34.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:55:34.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:34.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.11 2024-09-15T01:55:34.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 113 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 10 monclient: monmap: 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.572 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:34.571306+0000 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:34.571306+0000 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:34.573 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 1 Processor -- start 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 1 -- start start 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.570+0000 7f4ef9a0f640 10 monclient: init 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ef4098370 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef4098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4ef4136ea0 0x7f4ef4139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef4136ea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40a17b0 con 0x7f4ef4098370 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40a0160 con 0x7f4ef4136ea0 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4132a60 con 0x7f4ef4098740 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef9a0f640 10 monclient(hunting): authenticate will time out at 967.040285s 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ef4098370 0x7f4ef4132350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49932/0 (socket says 172.21.15.67:49932) 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/1016879399 learned_addr learned my addr 172.21.15.67:0/1016879399 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 1 --2- 172.21.15.67:0/1016879399 >> v2:172.21.15.17:3300/0 conn(0x7f4ef4136ea0 0x7f4ef4139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 10 monclient(hunting): get_auth_request con 0x7f4ef4098740 auth_method 0 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): get_auth_request con 0x7f4ef4136ea0 auth_method 0 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 1 -- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 msgr2=0x7f4ef4098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 1 -- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 msgr2=0x7f4ef4098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.571+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_done global_id 44847 payload 274 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 1 -- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 msgr2=0x7f4ef4098b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 1 -- 172.21.15.67:0/1016879399 >> v1:172.21.15.67:6789/0 conn(0x7f4ef4098370 legacy=0x7f4ef4132350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 1 monclient: found mon.noname-a 2024-09-15T01:55:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.572+0000 7f4ef37fe640 1 -- 172.21.15.67:0/1016879399 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef409fef0 con 0x7f4ef4136ea0 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.574+0000 7f4ef37fe640 1 -- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 msgr2=0x7f4ef4139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.574+0000 7f4ef37fe640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4139280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4096d60 tx=0x7f4eec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5ba8170d85bfcf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 1 -- 172.21.15.67:0/1016879399 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec007760 con 0x7f4ef4136ea0 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: dump: 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 5 monclient: authenticate success, global_id 44847 2024-09-15T01:55:34.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 1 -- 172.21.15.67:0/1016879399 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec02d050 con 0x7f4ef4136ea0 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 4 set_mon_vals no callback set 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 1 -- 172.21.15.67:0/1016879399 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec007af0 con 0x7f4ef4136ea0 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef27fc640 10 monclient: dump: 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.586 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 10 monclient: shutdown 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 msgr2=0x7f4ef4139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4139280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4096d60 tx=0x7f4eec009780 comp rx=0 tx=0).stop 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/1016879399 shutdown_connections 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4139280 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/1016879399 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098740 0x7f4ef4098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/1016879399 >> 172.21.15.67:0/1016879399 conn(0x7f4ef4071150 msgr2=0x7f4ef40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/1016879399 shutdown_connections 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.575+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/1016879399 wait complete. 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.576+0000 7f4ef9a0f640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.576+0000 7f4ef9a0f640 10 monclient: monmap: 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:34.588 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.576+0000 7f4ef9a0f640 1 Processor -- start 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.576+0000 7f4ef9a0f640 1 -- start start 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.576+0000 7f4ef9a0f640 10 monclient: init 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098370 0x7f4ef4133810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef4098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef4098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef4136ea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40a0890 con 0x7f4ef4136ea0 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40a3320 con 0x7f4ef4098740 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef411bd40 con 0x7f4ef4098370 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef9a0f640 10 monclient(hunting): authenticate will time out at 967.046202s 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46426/0 (socket says 172.21.15.67:46426) 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 -- 172.21.15.67:0/2125951965 learned_addr learned my addr 172.21.15.67:0/2125951965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): get_auth_request con 0x7f4ef4098740 auth_method 0 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098370 0x7f4ef4133810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): get_auth_request con 0x7f4ef4136ea0 auth_method 0 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f4ef4098370 auth_method 0 2024-09-15T01:55:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient(hunting): handle_auth_done global_id 5503 payload 824 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 -- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098370 msgr2=0x7f4ef4133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098370 0x7f4ef4133810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 -- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 msgr2=0x7f4ef4134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4134290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 monclient: found mon.noname-b 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.577+0000 7f4eea7fc640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec009a20 con 0x7f4ef4098740 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.579+0000 7f4eea7fc640 1 -- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 msgr2=0x7f4ef4133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.579+0000 7f4eea7fc640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00d890 tx=0x7f4edc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00de20 con 0x7f4ef4098740 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: dump: 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.593 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc00bae0 con 0x7f4ef4098740 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 5 monclient: authenticate success, global_id 5503 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _renew_subs 2024-09-15T01:55:34.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef411b9d0 con 0x7f4ef4098740 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00bea0 con 0x7f4ef4098740 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 10 monclient: dump: 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef1ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _renew_subs 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef4135bf0 con 0x7f4ef4098740 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4edc014640 con 0x7f4ef4098740 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.580+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef4136870 con 0x7f4ef4098740 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.581+0000 7f4eeb7fe640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 0x7f4ed403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.581+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f4edc04c5e0 con 0x7f4ef4098740 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.581+0000 7f4ef2ffd640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 0x7f4ed403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.581+0000 7f4ef2ffd640 10 monclient: get_auth_request con 0x7f4ed4038770 auth_method 0 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.583+0000 7f4ef2ffd640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 0x7f4ed403ac20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4007fd0 tx=0x7f4ee4007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.585+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4edc011570 con 0x7f4ef4098740 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.585+0000 7f4eeb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.585+0000 7f4eeb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.750+0000 7f4ef9a0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.750+0000 7f4ef9a0f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.750+0000 7f4ef9a0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.750+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4ef404bf60 con 0x7f4ef4098740 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.751+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v102) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f4edc01db60 con 0x7f4ef4098740 2024-09-15T01:55:34.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.751+0000 7f4eeb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:34.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.751+0000 7f4eeb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:34.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.850+0000 7f4ef9a0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.850+0000 7f4ef9a0f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.850+0000 7f4ef9a0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.850+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4ef40a14d0 con 0x7f4ef4098740 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.851+0000 7f4eeb7fe640 1 -- 172.21.15.67:0/2125951965 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v102) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f4edc01dd40 con 0x7f4ef4098740 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.851+0000 7f4eeb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.851+0000 7f4eeb7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 msgr2=0x7f4ed403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 0x7f4ed403ac20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4007fd0 tx=0x7f4ee4007f00 comp rx=0 tx=0).stop 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 10 monclient: shutdown 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 msgr2=0x7f4ef4133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00d890 tx=0x7f4edc008040 comp rx=0 tx=0).stop 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 shutdown_connections 2024-09-15T01:55:34.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ed4038770 0x7f4ed403ac20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ef4136ea0 0x7f4ef4134290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4098740 0x7f4ef4133d50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 --2- 172.21.15.67:0/2125951965 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ef4098370 0x7f4ef4133810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.852+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 >> 172.21.15.67:0/2125951965 conn(0x7f4ef4071150 msgr2=0x7f4ef40a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.853+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 shutdown_connections 2024-09-15T01:55:34.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.853+0000 7f4ef9a0f640 1 -- 172.21.15.67:0/2125951965 wait complete. 2024-09-15T01:55:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient: get_auth_request con 0x7f4fe011ece0 auth_method 0 2024-09-15T01:55:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_done global_id 74102 payload 824 2024-09-15T01:55:34.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.858+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fe011f5f0 con 0x7f4fe011ece0 2024-09-15T01:55:34.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0007a90 tx=0x7f4fd0034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0005e60 con 0x7f4fe011ece0 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 10 monclient: dump: 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.864 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.864 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:34.864 INFO:tasks.workunit.client.0.smithi067.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-15T01:55:34.864 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.11 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4fd0007510 con 0x7f4fe011ece0 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0007a90 tx=0x7f4fd0034000 comp rx=0 tx=0).stop 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 msgr2=0x7f4fe011f0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:34.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.861+0000 7f4fc6ffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe011ece0 0x7f4fe011f0b0 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0007a90 tx=0x7f4fd0034000 comp rx=0 tx=0).stop 2024-09-15T01:55:34.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fe52af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:34.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fe52af640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 0x7f4fe0096e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fe52af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00724a0 con 0x7f4fe0096a10 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 0x7f4fe0096e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient: get_auth_request con 0x7f4fe0096a10 auth_method 0 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 10 monclient(hunting): handle_auth_done global_id 74105 payload 824 2024-09-15T01:55:34.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.868+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4fe0067220 con 0x7f4fe0096a10 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fdeffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 msgr2=0x7f4fe0096e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fdeffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 0x7f4fe0096e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c3d0 tx=0x7f4fd0043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0005870 con 0x7f4fe0096a10 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 10 monclient: dump: 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f4fd0046410 con 0x7f4fe0096a10 2024-09-15T01:55:34.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 msgr2=0x7f4fe0096e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 0x7f4fe0096e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c3d0 tx=0x7f4fd0043000 comp rx=0 tx=0).stop 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 msgr2=0x7f4fe0096e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.871+0000 7f4fc6ffd640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe0096a10 0x7f4fe0096e10 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c3d0 tx=0x7f4fd0043000 comp rx=0 tx=0).stop 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:34.875 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:34.876 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:34.877 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:34.878 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:34.879 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:34.880 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:34.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:34.884 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:34.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:34.887 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fcc0387c0 msgr2=0x7f4fcc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fcc0387c0 0x7f4fcc03ac70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4fe009c8a0 tx=0x7f4fd8001a10 comp rx=0 tx=0).stop 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 10 monclient: shutdown 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 msgr2=0x7f4fe0133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 --2- 172.21.15.17:0/2182909114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe01413d0 0x7f4fe0133cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4006fd0 tx=0x7f4fd4008040 comp rx=0 tx=0).stop 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fdf7fe640 1 -- 172.21.15.17:0/2182909114 reap_dead start 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fdf7fe640 1 -- 172.21.15.17:0/2182909114 reap_dead start 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 shutdown_connections 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 >> 172.21.15.17:0/2182909114 conn(0x7f4fe00712f0 msgr2=0x7f4fe00a29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 shutdown_connections 2024-09-15T01:55:34.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.872+0000 7f4fe52af640 1 -- 172.21.15.17:0/2182909114 wait complete. 2024-09-15T01:55:34.890 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:34.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:55:34.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.937+0000 7f29e619d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.937+0000 7f29e619d640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.937+0000 7f29e619d640 10 monclient: monmap: 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:34.938802+0000 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:34.938802+0000 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:34.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 1 Processor -- start 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 1 -- start start 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient: init 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-b con 0x7f29e00a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29e009eea0 0x7f29e009f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-c con 0x7f29e009eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.938+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29e0131b10 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00a61a0 con 0x7f29e0131b10 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e00a1610 con 0x7f29e009eea0 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e009f7d0 con 0x7f29e00a0b50 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29e619d640 10 monclient(hunting): authenticate will time out at 967.408043s 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29e009eea0 0x7f29e009f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29dffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f29e0131b10 0x7f29e0131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49940/0 (socket says 172.21.15.67:49940) 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29dffff640 1 -- 172.21.15.67:0/3848115622 learned_addr learned my addr 172.21.15.67:0/3848115622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:34.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): get_auth_request con 0x7f29e009eea0 auth_method 0 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): get_auth_request con 0x7f29e00a0b50 auth_method 0 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient(hunting): handle_auth_done global_id 5506 payload 274 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 -- 172.21.15.67:0/3848115622 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 msgr2=0x7f29e0069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0069380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 -- 172.21.15.67:0/3848115622 >> v1:172.21.15.67:6789/0 conn(0x7f29e0131b10 legacy=0x7f29e0131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 monclient: found mon.noname-c 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.939+0000 7f29deffd640 1 -- 172.21.15.67:0/3848115622 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00a04c0 con 0x7f29e009eea0 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29deffd640 1 -- 172.21.15.67:0/3848115622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 msgr2=0x7f29e009f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29deffd640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009f290 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009850 tx=0x7f29d40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dda07eab315f864 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 1 -- 172.21.15.67:0/3848115622 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d402adf0 con 0x7f29e009eea0 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: dump: 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.951 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 1 -- 172.21.15.67:0/3848115622 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4032440 con 0x7f29e009eea0 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 1 -- 172.21.15.67:0/3848115622 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40327e0 con 0x7f29e009eea0 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 5 monclient: authenticate success, global_id 5506 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 4 set_mon_vals no callback set 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29de7fc640 10 monclient: dump: 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 10 monclient: shutdown 2024-09-15T01:55:34.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 1 -- 172.21.15.67:0/3848115622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 msgr2=0x7f29e009f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.942+0000 7f29e619d640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009f290 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009850 tx=0x7f29d40090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 -- 172.21.15.67:0/3848115622 shutdown_connections 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009f290 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 --2- 172.21.15.67:0/3848115622 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0069380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 -- 172.21.15.67:0/3848115622 >> 172.21.15.67:0/3848115622 conn(0x7f29e0071330 msgr2=0x7f29e00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 -- 172.21.15.67:0/3848115622 shutdown_connections 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 1 -- 172.21.15.67:0/3848115622 wait complete. 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.943+0000 7f29e619d640 10 monclient: monmap: 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.954 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 Processor -- start 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 -- start start 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient: init 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-b con 0x7f29e009eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-c con 0x7f29e00a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f29e0131b10 0x7f29e011b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): picked mon.noname-a con 0x7f29e0131b10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e009b1e0 con 0x7f29e0131b10 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e009acf0 con 0x7f29e009eea0 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00a1260 con 0x7f29e00a0b50 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29e619d640 10 monclient(hunting): authenticate will time out at 967.413629s 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46442/0 (socket says 172.21.15.67:46442) 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 -- 172.21.15.67:0/821690388 learned_addr learned my addr 172.21.15.67:0/821690388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): get_auth_request con 0x7f29e009eea0 auth_method 0 2024-09-15T01:55:34.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): get_auth_request con 0x7f29e00a0b50 auth_method 0 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f29e0131b10 0x7f29e011b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): get_auth_request con 0x7f29e0131b10 auth_method 0 2024-09-15T01:55:34.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient(hunting): handle_auth_done global_id 5509 payload 824 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 -- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 msgr2=0x7f29e0130b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0130b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 -- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f29e0131b10 msgr2=0x7f29e011b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f29e0131b10 0x7f29e011b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.944+0000 7f29df7fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.945+0000 7f29df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.945+0000 7f29df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.945+0000 7f29df7fe640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d4009390 con 0x7f29e009eea0 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29df7fe640 1 -- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 msgr2=0x7f29e009a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29df7fe640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f29d000ea50 tx=0x7f29d00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0009600 con 0x7f29e009eea0 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: dump: 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.959 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d000cc20 con 0x7f29e009eea0 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 5 monclient: authenticate success, global_id 5509 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 10 monclient: _renew_subs 2024-09-15T01:55:34.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e011dc80 con 0x7f29e009eea0 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0013470 con 0x7f29e009eea0 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29c77fe640 10 monclient: dump: 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:34.961 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29ddffb640 4 set_mon_vals no callback set 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 10 monclient: _renew_subs 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.947+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e011cd90 con 0x7f29e009eea0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f29d0013820 con 0x7f29e009eea0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29c77fe640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 0x7f29cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f29d004c6e0 con 0x7f29e009eea0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29e619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29e619d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29e619d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e004bf60 con 0x7f29e009eea0 2024-09-15T01:55:34.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29deffd640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 0x7f29cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:34.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.948+0000 7f29deffd640 10 monclient: get_auth_request con 0x7f29cc038740 auth_method 0 2024-09-15T01:55:34.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.951+0000 7f29deffd640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 0x7f29cc03abf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f29e009ffe0 tx=0x7f29d4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:34.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.953+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f29d001b9f0 con 0x7f29e009eea0 2024-09-15T01:55:34.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.953+0000 7f29c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:34.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:34.953+0000 7f29c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:34.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.994+0000 7f30432d4640 10 monclient: get_monmap_and_config 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.994+0000 7f30432d4640 10 monclient: build_initial_monmap 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.995+0000 7f30432d4640 10 monclient: monmap: 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:34.995826+0000 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:34.995826+0000 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:34.997 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:34.998 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:34.998 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:34.998 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.995+0000 7f30432d4640 1 Processor -- start 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.995+0000 7f30432d4640 1 -- start start 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.995+0000 7f30432d4640 10 monclient: init 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T01:55:35.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-b con 0x7f303c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f303c09ccf0 0x7f303c069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-c con 0x7f303c09ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f303c069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303c0a5b50 con 0x7f303c069ac0 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c0a18b0 con 0x7f303c09ccf0 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c09c5f0 con 0x7f303c0a5780 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f30432d4640 10 monclient(hunting): authenticate will time out at 994.992414s 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54754/0 (socket says 172.21.15.17:54754) 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 -- 172.21.15.17:0/3013083765 learned_addr learned my addr 172.21.15.17:0/3013083765 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): get_auth_request con 0x7f303c0a5780 auth_method 0 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041871640 1 --1- 172.21.15.17:0/3013083765 >> v1:172.21.15.17:6789/0 conn(0x7f303c069ac0 0x7f303c138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36004/0 (socket says 172.21.15.17:36004) 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 1 --2- 172.21.15.17:0/3013083765 >> v2:172.21.15.67:3300/0 conn(0x7f303c09ccf0 0x7f303c069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): get_auth_request con 0x7f303c09ccf0 auth_method 0 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient(hunting): handle_auth_done global_id 74114 payload 274 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 -- 172.21.15.17:0/3013083765 >> v2:172.21.15.67:3300/0 conn(0x7f303c09ccf0 msgr2=0x7f303c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 --2- 172.21.15.17:0/3013083765 >> v2:172.21.15.67:3300/0 conn(0x7f303c09ccf0 0x7f303c069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 -- 172.21.15.17:0/3013083765 >> v1:172.21.15.17:6789/0 conn(0x7f303c069ac0 legacy=0x7f303c138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 monclient: found mon.noname-b 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.996+0000 7f3041070640 1 -- 172.21.15.17:0/3013083765 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c09d210 con 0x7f303c0a5780 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3041070640 1 -- 172.21.15.17:0/3013083765 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 msgr2=0x7f303c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3041070640 1 --2- 172.21.15.17:0/3013083765 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3038009910 tx=0x7f30380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=990aee91553fd316 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 1 -- 172.21.15.17:0/3013083765 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303802aea0 con 0x7f303c0a5780 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: dump: 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.009 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 1 -- 172.21.15.17:0/3013083765 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038032440 con 0x7f303c0a5780 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f30432d4640 5 monclient: authenticate success, global_id 74114 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f30432d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 1 -- 172.21.15.17:0/3013083765 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30380327e0 con 0x7f303c0a5780 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f30432d4640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f30432d4640 4 set_mon_vals no callback set 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:34.999+0000 7f3033fff640 10 monclient: dump: 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.011 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 10 monclient: shutdown 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 -- 172.21.15.17:0/3013083765 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 msgr2=0x7f303c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 --2- 172.21.15.17:0/3013083765 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3038009910 tx=0x7f30380091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 -- 172.21.15.17:0/3013083765 shutdown_connections 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 --2- 172.21.15.17:0/3013083765 >> v2:172.21.15.67:3300/0 conn(0x7f303c09ccf0 0x7f303c069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 --2- 172.21.15.17:0/3013083765 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c0a5780 0x7f303c09bc90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 -- 172.21.15.17:0/3013083765 >> 172.21.15.17:0/3013083765 conn(0x7f303c070e10 msgr2=0x7f303c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 -- 172.21.15.17:0/3013083765 shutdown_connections 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.000+0000 7f30432d4640 1 -- 172.21.15.17:0/3013083765 wait complete. 2024-09-15T01:55:35.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.001+0000 7f30432d4640 10 monclient: build_initial_monmap 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.001+0000 7f30432d4640 10 monclient: monmap: 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.001+0000 7f30432d4640 1 Processor -- start 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.001+0000 7f30432d4640 1 -- start start 2024-09-15T01:55:35.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.001+0000 7f30432d4640 10 monclient: init 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 0x7f303c13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-a con 0x7f303c069ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 0x7f303c13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-c con 0x7f303c09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0a5780 0x7f303c141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): picked mon.noname-b con 0x7f303c0a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c121620 con 0x7f303c069ac0 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c1225d0 con 0x7f303c0a5780 2024-09-15T01:55:35.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c1227d0 con 0x7f303c09ccf0 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f30432d4640 10 monclient(hunting): authenticate will time out at 994.998115s 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 0x7f303c13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 0x7f303c13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 0x7f303c13dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53870/0 (socket says 172.21.15.17:53870) 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 0x7f303c13d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54770/0 (socket says 172.21.15.17:54770) 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 -- 172.21.15.17:0/1116298181 learned_addr learned my addr 172.21.15.17:0/1116298181 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): get_auth_request con 0x7f303c069ac0 auth_method 0 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): get_auth_request con 0x7f303c09ccf0 auth_method 0 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0a5780 0x7f303c141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): get_auth_request con 0x7f303c0a5780 auth_method 0 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient(hunting): handle_auth_done global_id 44856 payload 824 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 msgr2=0x7f303c13d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 0x7f303c13d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0a5780 msgr2=0x7f303c141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0a5780 0x7f303c141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 monclient: found mon.noname-a 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041070640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038009450 con 0x7f303c069ac0 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f304086f640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f303c09ccf0 0x7f303c13d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.002+0000 7f3041871640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0a5780 0x7f303c141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.004+0000 7f3041070640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 msgr2=0x7f303c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.004+0000 7f3041070640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 0x7f303c13dda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3038009910 tx=0x7f3038032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303802aea0 con 0x7f303c069ac0 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: dump: 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.018 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038000d10 con 0x7f303c069ac0 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 5 monclient: authenticate success, global_id 44856 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 10 monclient: _renew_subs 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c122d90 con 0x7f303c069ac0 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038031ea0 con 0x7f303c069ac0 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30317fa640 10 monclient: dump: 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.020 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 10 monclient: _renew_subs 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30337fe640 4 set_mon_vals no callback set 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.005+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c122b60 con 0x7f303c069ac0 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30432d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30432d4640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 0x7f303c141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3038039040 con 0x7f303c069ac0 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c063b20 con 0x7f303c120030 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 0x7f303c141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient: get_auth_request con 0x7f303c120030 auth_method 0 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30317fa640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3020038740 0x7f302003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6036+0+0 (secure 0 0 0) 0x7f303806bd70 con 0x7f303c069ac0 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 10 monclient(hunting): handle_auth_done global_id 5518 payload 824 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f304086f640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f303c122990 con 0x7f303c120030 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f3041871640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3020038740 0x7f302003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.006+0000 7f3041871640 10 monclient: get_auth_request con 0x7f3020038740 auth_method 0 2024-09-15T01:55:35.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f304086f640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 msgr2=0x7f303c141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f304086f640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 0x7f303c141850 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f30 tx=0x7f302c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c014860 con 0x7f303c120030 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f30317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f30317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.008+0000 7f30317fa640 10 monclient: dump: 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f3041871640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3020038740 0x7f302003abf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f303c121440 tx=0x7f303400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f302c0149c0 con 0x7f303c120030 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 msgr2=0x7f303c141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 0x7f303c141850 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f30 tx=0x7f302c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 msgr2=0x7f303c141850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.009+0000 7f30317fa640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c120030 0x7f303c141850 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.014+0000 7f30432d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f30432d4640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 0x7f303c11b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f30432d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c142190 con 0x7f303c141d90 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 0x7f303c11b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient: get_auth_request con 0x7f303c141d90 auth_method 0 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 10 monclient(hunting): handle_auth_done global_id 5521 payload 824 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.015+0000 7f304086f640 1 -- 172.21.15.17:0/1116298181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f303c0724a0 con 0x7f303c141d90 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.017+0000 7f304086f640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 msgr2=0x7f303c11b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.017+0000 7f304086f640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 0x7f303c11b6d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f30 tx=0x7f302c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c024040 con 0x7f303c141d90 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 10 monclient: dump: 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f302c014650 con 0x7f303c141d90 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 msgr2=0x7f303c11b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 0x7f303c11b6d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f30 tx=0x7f302c01c000 comp rx=0 tx=0).stop 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 msgr2=0x7f303c11b6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:35.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30317fa640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c141d90 0x7f303c11b6d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:35.030 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:35.031 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:35.032 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:35.033 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:35.034 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:35.035 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:35.037 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:35.038 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:35.039 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:55:35.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:35.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:35.043 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:35.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:35.043 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:35.043 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3020038740 msgr2=0x7f302003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3020038740 0x7f302003abf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f303c121440 tx=0x7f303400d040 comp rx=0 tx=0).stop 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 10 monclient: shutdown 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 msgr2=0x7f303c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 --2- 172.21.15.17:0/1116298181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f303c069ac0 0x7f303c13dda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3038009910 tx=0x7f3038032000 comp rx=0 tx=0).stop 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f3041871640 1 -- 172.21.15.17:0/1116298181 reap_dead start 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f3041871640 1 -- 172.21.15.17:0/1116298181 reap_dead start 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 shutdown_connections 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 >> 172.21.15.17:0/1116298181 conn(0x7f303c070e10 msgr2=0x7f303c09af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 shutdown_connections 2024-09-15T01:55:35.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.018+0000 7f30432d4640 1 -- 172.21.15.17:0/1116298181 wait complete. 2024-09-15T01:55:35.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:35.045 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:55:35.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:55:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.121+0000 7f29e619d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.121+0000 7f29e619d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.121+0000 7f29e619d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.121+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f29e011e360 con 0x7f29e009eea0 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.140+0000 7fc87e909640 10 monclient: get_monmap_and_config 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.140+0000 7fc87e909640 10 monclient: build_initial_monmap 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.140+0000 7fc87e909640 10 monclient: monmap: 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:35.141109+0000 2024-09-15T01:55:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:35.141109+0000 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:35.143 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.140+0000 7fc87e909640 1 Processor -- start 2024-09-15T01:55:35.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 -- start start 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient: init 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-b con 0x7fc878131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-a con 0x7fc8780a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc87809f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8780a0040 con 0x7fc87809f1e0 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87809bff0 con 0x7fc8780a0280 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780a14a0 con 0x7fc878131b20 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc87e909640 10 monclient(hunting): authenticate will time out at 995.137534s 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53874/0 (socket says 172.21.15.17:53874) 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54776/0 (socket says 172.21.15.17:54776) 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 1 -- 172.21.15.17:0/412427637 learned_addr learned my addr 172.21.15.17:0/412427637 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): get_auth_request con 0x7fc8780a0280 auth_method 0 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): get_auth_request con 0x7fc878131b20 auth_method 0 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient(hunting): handle_auth_done global_id 74120 payload 274 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 -- 172.21.15.17:0/412427637 >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 msgr2=0x7fc878131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 --2- 172.21.15.17:0/412427637 >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 -- 172.21.15.17:0/412427637 >> v1:172.21.15.67:6789/0 conn(0x7fc87809f1e0 legacy=0x7fc878068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 monclient: found mon.noname-b 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc877fff640 1 -- 172.21.15.17:0/412427637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878132430 con 0x7fc878131b20 2024-09-15T01:55:35.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.141+0000 7fc8777fe640 1 --2- 172.21.15.17:0/412427637 >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc877fff640 1 -- 172.21.15.17:0/412427637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 msgr2=0x7fc87809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc877fff640 1 --2- 172.21.15.17:0/412427637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc868009850 tx=0x7fc8680090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6457efb701179f12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 1 -- 172.21.15.17:0/412427637 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86802adf0 con 0x7fc878131b20 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 10 monclient: dump: 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.144+0000 7fc876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 5 monclient: authenticate success, global_id 74120 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 1 -- 172.21.15.17:0/412427637 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc868032440 con 0x7fc878131b20 2024-09-15T01:55:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 4 set_mon_vals no callback set 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 1 -- 172.21.15.17:0/412427637 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8680327e0 con 0x7fc878131b20 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc876ffd640 10 monclient: dump: 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 10 monclient: shutdown 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 -- 172.21.15.17:0/412427637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 msgr2=0x7fc87809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 --2- 172.21.15.17:0/412427637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc868009850 tx=0x7fc8680090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 -- 172.21.15.17:0/412427637 shutdown_connections 2024-09-15T01:55:35.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 --2- 172.21.15.17:0/412427637 >> v2:172.21.15.17:3300/0 conn(0x7fc8780a0280 0x7fc878131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 --2- 172.21.15.17:0/412427637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87809eca0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 -- 172.21.15.17:0/412427637 >> 172.21.15.17:0/412427637 conn(0x7fc878071270 msgr2=0x7fc8780a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 -- 172.21.15.17:0/412427637 shutdown_connections 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.145+0000 7fc87e909640 1 -- 172.21.15.17:0/412427637 wait complete. 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient: build_initial_monmap 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient: monmap: 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:35.159 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 Processor -- start 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 -- start start 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient: init 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:35.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc87809f1e0 0x7fc8781411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-b con 0x7fc87809f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 0x7fc87812eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-a con 0x7fc8780a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): picked mon.noname-c con 0x7fc878131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780a09b0 con 0x7fc8780a0280 2024-09-15T01:55:35.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780a5580 con 0x7fc87809f1e0 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87811c480 con 0x7fc878131b20 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.146+0000 7fc87e909640 10 monclient(hunting): authenticate will time out at 995.142994s 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 0x7fc87812eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 0x7fc87812eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53876/0 (socket says 172.21.15.17:53876) 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 1 -- 172.21.15.17:0/2107686462 learned_addr learned my addr 172.21.15.17:0/2107686462 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): get_auth_request con 0x7fc8780a0280 auth_method 0 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54784/0 (socket says 172.21.15.17:54784) 2024-09-15T01:55:35.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): get_auth_request con 0x7fc878131b20 auth_method 0 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc87809f1e0 0x7fc8781411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): get_auth_request con 0x7fc87809f1e0 auth_method 0 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc8777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient(hunting): handle_auth_done global_id 74123 payload 824 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 -- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc87809f1e0 msgr2=0x7fc8781411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc87809f1e0 0x7fc8781411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 -- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 msgr2=0x7fc87812eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 0x7fc87812eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 monclient: found mon.noname-c 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.147+0000 7fc87cea6640 1 -- 172.21.15.17:0/2107686462 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c009450 con 0x7fc878131b20 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87cea6640 1 -- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 msgr2=0x7fc87812f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87cea6640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c7e0 tx=0x7fc8700025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87000a850 con 0x7fc878131b20 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: dump: 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87000a9d0 con 0x7fc878131b20 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 5 monclient: authenticate success, global_id 74123 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 10 monclient: _renew_subs 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc868009390 con 0x7fc878131b20 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87000ad90 con 0x7fc878131b20 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc85ffff640 10 monclient: dump: 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.167 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 10 monclient: _renew_subs 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87809b580 con 0x7fc878131b20 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.149+0000 7fc8767fc640 4 set_mon_vals no callback set 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.150+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc870012630 con 0x7fc878131b20 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.150+0000 7fc87e909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.150+0000 7fc87e909640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.150+0000 7fc87e909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.150+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87811c610 con 0x7fc878131b20 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.151+0000 7fc85ffff640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 0x7fc86403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.151+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6036+0+0 (secure 0 0 0) 0x7fc87004c1e0 con 0x7fc878131b20 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.151+0000 7fc877fff640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 0x7fc86403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.151+0000 7fc877fff640 10 monclient: get_auth_request con 0x7fc864038790 auth_method 0 2024-09-15T01:55:35.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.154+0000 7fc877fff640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 0x7fc86403ac40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8780a2f90 tx=0x7fc868032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.155+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc870022ae0 con 0x7fc878131b20 2024-09-15T01:55:35.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.155+0000 7fc85ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.155+0000 7fc85ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:35.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.383+0000 7fc87e909640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:55:35.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.383+0000 7fc87e909640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:55:35.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.383+0000 7fc87e909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:35.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.383+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc8780724a0 con 0x7fc878131b20 2024-09-15T01:55:35.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=3085074147,config=3140514009,health=1361790972,kv=1822525652,logm=3425281864}) 2024-09-15T01:55:35.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.384+0000 7fc85ffff640 1 -- 172.21.15.17:0/2107686462 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc870016020 con 0x7fc878131b20 2024-09-15T01:55:35.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.384+0000 7fc85ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:55:35.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.384+0000 7fc85ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=2034600192,config=3140514009,health=1361790972,kv=1529054990,logm=1438664780}) 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=3085074147,config=3140514009,health=1361790972,kv=1822525652,logm=3425281864}) 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.384+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=2034600192,config=3140514009,health=86774458,kv=2364430354,logm=351986077}) 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328138080}) 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268467120}) 2024-09-15T01:55:35.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328138080}) 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.385+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633122059}) 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584032464}) 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=317565163}) 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584032464}) 2024-09-15T01:55:35.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.386+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=577358407}) 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2636071287}) 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212477075}) 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2636071287}) 2024-09-15T01:55:35.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919184337}) 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 msgr2=0x7fc86403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 0x7fc86403ac40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8780a2f90 tx=0x7fc868032000 comp rx=0 tx=0).stop 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 10 monclient: shutdown 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 msgr2=0x7fc87812f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c7e0 tx=0x7fc8700025b0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 shutdown_connections 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc864038790 0x7fc86403ac40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8780a0280 0x7fc87812eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc878131b20 0x7fc87812f530 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 --2- 172.21.15.17:0/2107686462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc87809f1e0 0x7fc8781411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 >> 172.21.15.17:0/2107686462 conn(0x7fc878071270 msgr2=0x7fc87809d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 shutdown_connections 2024-09-15T01:55:35.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:35.387+0000 7fc87e909640 1 -- 172.21.15.17:0/2107686462 wait complete. 2024-09-15T01:55:35.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479997495}) 2024-09-15T01:55:35.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=89210544}) 2024-09-15T01:55:35.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479997495}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.387+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2011846483}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896762921}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511763227}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896762921}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.388+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107842217}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473021528}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327771613}) 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473021528}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.389+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130059582}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504264695}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118459884}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504264695}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.390+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354370729}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451664023}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487542980}) 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451664023}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.391+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778170030}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002799400}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388927168}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002799400}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213937739}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753177145}) 2024-09-15T01:55:35.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1807680328}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753177145}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.392+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749172363}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755373404}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674974489}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755373404}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.393+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1532267520}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24201701}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=928199753}) 2024-09-15T01:55:35.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24201701}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.394+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152354103}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530585505}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875857119}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530585505}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.395+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727579769}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719331047}) 2024-09-15T01:55:35.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832202427}) 2024-09-15T01:55:35.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719331047}) 2024-09-15T01:55:35.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.396+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637944498}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333097195}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1095617675}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333097195}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.397+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3057269416}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=3410113667,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgrstat=771924799,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2103415550,osd_snap=1063953654,osdmap=521279177}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=1871767813,mdsmap=1531151339,mgr=3226792663,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=806668162,mon_config_key=199500175,monmap=3178194974,osd_metadata=1079644522,osd_pg_creating=2103415550,osd_snap=1063953654,osdmap=3185407561}) 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=3410113667,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgrstat=771924799,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2103415550,osd_snap=1063953654,osdmap=521279177}) 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.398+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=21} crc {logm=1799444620,mdsmap=1228700967,mgr=3226792663,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3878954933,mon_config_key=1383706935,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=2103415550,osd_snap=1063953654,osdmap=2044319534}) 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3688244864}) 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3410840924}) 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3688244864}) 2024-09-15T01:55:35.407 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.399+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=502598185}) 2024-09-15T01:55:35.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:55:35.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3755535615}) 2024-09-15T01:55:35.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=43} crc {osdmap=3529210938}) 2024-09-15T01:55:35.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:55:35.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3755535615}) 2024-09-15T01:55:35.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.402+0000 7f73a1ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=3295222166}) 2024-09-15T01:55:35.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.581+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v103) v1 ==== 125+0+0 (secure 0 0 0) 0x7f29d001b3f0 con 0x7f29e009eea0 2024-09-15T01:55:35.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.581+0000 7f29c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.581+0000 7f29c77fe640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-15T01:55:35.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.686+0000 7f29e619d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.686+0000 7f29e619d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.686+0000 7f29e619d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.686+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f29e00008d0 con 0x7f29e009eea0 2024-09-15T01:55:35.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.687+0000 7f29c77fe640 1 -- 172.21.15.67:0/821690388 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v103) v1 ==== 126+0+0 (secure 0 0 0) 0x7f29d001b5d0 con 0x7f29e009eea0 2024-09-15T01:55:35.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.687+0000 7f29c77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T01:55:35.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.687+0000 7f29c77fe640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-15T01:55:35.689 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.11' does not appear in the crush map 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 msgr2=0x7f29cc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 0x7f29cc03abf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f29e009ffe0 tx=0x7f29d4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 10 monclient: shutdown 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 msgr2=0x7f29e009a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f29d000ea50 tx=0x7f29d00090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 shutdown_connections 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f29cc038740 0x7f29cc03abf0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f29e0131b10 0x7f29e011b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f29e00a0b50 0x7f29e0130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 --2- 172.21.15.67:0/821690388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e009eea0 0x7f29e009a0f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 >> 172.21.15.67:0/821690388 conn(0x7f29e0071330 msgr2=0x7f29e009ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 shutdown_connections 2024-09-15T01:55:35.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.689+0000 7f29e619d640 1 -- 172.21.15.67:0/821690388 wait complete. 2024-09-15T01:55:35.700 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm osd.11 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.772+0000 7f0e3bd9d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.772+0000 7f0e3bd9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.772+0000 7f0e3bd9d640 10 monclient: monmap: 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:35.773824+0000 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:35.773824+0000 2024-09-15T01:55:35.775 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:35.776 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 Processor -- start 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 -- start start 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient: init 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e34132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e34131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e34098260 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e34098d40 0x7f0e34139130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e34098d40 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e34096a70 con 0x7f0e34098260 2024-09-15T01:55:35.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3406a6b0 con 0x7f0e34098d40 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340a16e0 con 0x7f0e34131f90 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.773+0000 7f0e3bd9d640 10 monclient(hunting): authenticate will time out at 968.242935s 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e34098d40 0x7f0e34139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e34098d40 0x7f0e34139130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46444/0 (socket says 172.21.15.67:46444) 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 -- 172.21.15.67:0/1346176835 learned_addr learned my addr 172.21.15.67:0/1346176835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): get_auth_request con 0x7f0e34098d40 auth_method 0 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e34132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): get_auth_request con 0x7f0e34131f90 auth_method 0 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e39b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient(hunting): handle_auth_done global_id 5527 payload 274 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 -- 172.21.15.67:0/1346176835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 msgr2=0x7f0e34132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e34132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 -- 172.21.15.67:0/1346176835 >> v1:172.21.15.17:6789/0 conn(0x7f0e34098260 legacy=0x7f0e34098630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 monclient: found mon.noname-c 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.774+0000 7f0e3a33a640 1 -- 172.21.15.67:0/1346176835 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e340a19c0 con 0x7f0e34098d40 2024-09-15T01:55:35.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3a33a640 1 -- 172.21.15.67:0/1346176835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 msgr2=0x7f0e34139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3a33a640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34139130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009910 tx=0x7f0e2c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=168febdeab4da442 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 1 -- 172.21.15.67:0/1346176835 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c02aea0 con 0x7f0e34098d40 2024-09-15T01:55:35.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: dump: 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 1 -- 172.21.15.67:0/1346176835 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c032440 con 0x7f0e34098d40 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 5 monclient: authenticate success, global_id 5527 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 4 set_mon_vals no callback set 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 1 -- 172.21.15.67:0/1346176835 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0327e0 con 0x7f0e34098d40 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e38b37640 10 monclient: dump: 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:35.789 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 10 monclient: shutdown 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/1346176835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 msgr2=0x7f0e34139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34139130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009910 tx=0x7f0e2c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/1346176835 shutdown_connections 2024-09-15T01:55:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34139130 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/1346176835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e34132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.777+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/1346176835 >> 172.21.15.67:0/1346176835 conn(0x7f0e34071690 msgr2=0x7f0e340a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/1346176835 shutdown_connections 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/1346176835 wait complete. 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 10 monclient: monmap: 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 1 Processor -- start 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 1 -- start start 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.778+0000 7f0e3bd9d640 10 monclient: init 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e34098260 0x7f0e34134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e34098260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e34098d40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e341350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e34131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340a31e0 con 0x7f0e34098260 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e34130e40 con 0x7f0e34098d40 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e34137c80 con 0x7f0e34131f90 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3bd9d640 10 monclient(hunting): authenticate will time out at 968.248321s 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46460/0 (socket says 172.21.15.67:46460) 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 -- 172.21.15.67:0/3356926197 learned_addr learned my addr 172.21.15.67:0/3356926197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): get_auth_request con 0x7f0e34098d40 auth_method 0 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39b39640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e34098260 0x7f0e34134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3a33a640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e341350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39b39640 10 monclient(hunting): get_auth_request con 0x7f0e34098260 auth_method 0 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3a33a640 10 monclient(hunting): get_auth_request con 0x7f0e34131f90 auth_method 0 2024-09-15T01:55:35.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3a33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e3a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient(hunting): handle_auth_done global_id 5530 payload 824 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 -- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 msgr2=0x7f0e341350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e341350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 -- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e34098260 msgr2=0x7f0e34134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e34098260 0x7f0e34134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient: _finish_hunting 0 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 monclient: found mon.noname-b 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.779+0000 7f0e39338640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c009450 con 0x7f0e34098d40 2024-09-15T01:55:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.781+0000 7f0e39338640 1 -- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 msgr2=0x7f0e34134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.781+0000 7f0e39338640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400c5e0 tx=0x7f0e24007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24012840 con 0x7f0e34098d40 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: dump: 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e240129a0 con 0x7f0e34098d40 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 5 monclient: authenticate success, global_id 5530 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 10 monclient: _renew_subs 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e34137f60 con 0x7f0e34098d40 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24012da0 con 0x7f0e34098d40 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:35.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e29ffb640 10 monclient: dump: 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 10 monclient: _renew_subs 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e2bfff640 4 set_mon_vals no callback set 2024-09-15T01:55:35.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.782+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e34136d90 con 0x7f0e34098d40 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e3bd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e3bd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e3bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0e24011590 con 0x7f0e34098d40 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e340639e0 con 0x7f0e34098d40 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e29ffb640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 0x7f0e1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6014+0+0 (secure 0 0 0) 0x7f0e2404c180 con 0x7f0e34098d40 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e39b39640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 0x7f0e1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:35.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.783+0000 7f0e39b39640 10 monclient: get_auth_request con 0x7f0e18038740 auth_method 0 2024-09-15T01:55:35.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.786+0000 7f0e39b39640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 0x7f0e1803abf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0e300076a0 tx=0x7f0e3000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:35.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.787+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0e24010ac0 con 0x7f0e34098d40 2024-09-15T01:55:35.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.787+0000 7f0e29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:35.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.787+0000 7f0e29ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.963+0000 7f0e3bd9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.963+0000 7f0e3bd9d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.963+0000 7f0e3bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:35.964+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f0e34132990 con 0x7f0e34098d40 2024-09-15T01:55:35.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:35.965+0000 7f73a1ff1640 -1 mon.a@0(leader).osd e103 definitely_dead 0 2024-09-15T01:55:36.409 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:55:36.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.515+0000 7ffb5f64f640 10 monclient: get_monmap_and_config 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.515+0000 7ffb5f64f640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.515+0000 7ffb5f64f640 10 monclient: monmap: 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:36.516653+0000 2024-09-15T01:55:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:36.516653+0000 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:36.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.516+0000 7ffb5f64f640 1 Processor -- start 2024-09-15T01:55:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.516+0000 7ffb5f64f640 1 -- start start 2024-09-15T01:55:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.516+0000 7ffb5f64f640 10 monclient: init 2024-09-15T01:55:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb5809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-b con 0x7ffb580a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb5809ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb58069c90 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb580a5b50 con 0x7ffb5809ccf0 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5809c5f0 con 0x7ffb58069c90 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb580a18b0 con 0x7ffb580a5780 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5f64f640 10 monclient(hunting): authenticate will time out at 996.513444s 2024-09-15T01:55:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb5809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5cbea640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ffb5809ccf0 0x7ffb58069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36010/0 (socket says 172.21.15.17:36010) 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5cbea640 1 -- 172.21.15.17:0/459510016 learned_addr learned my addr 172.21.15.17:0/459510016 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): get_auth_request con 0x7ffb580a5780 auth_method 0 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_done global_id 74132 payload 274 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 1 -- 172.21.15.17:0/459510016 >> v1:172.21.15.67:6789/0 conn(0x7ffb58069c90 legacy=0x7ffb58139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 1 -- 172.21.15.17:0/459510016 >> v1:172.21.15.17:6789/0 conn(0x7ffb5809ccf0 legacy=0x7ffb58069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 1 monclient: found mon.noname-b 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.517+0000 7ffb5d3eb640 1 -- 172.21.15.17:0/459510016 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5809d210 con 0x7ffb580a5780 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5d3eb640 1 -- 172.21.15.17:0/459510016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 msgr2=0x7ffb5809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5d3eb640 1 --2- 172.21.15.17:0/459510016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb5809bc90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009880 tx=0x7ffb540090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5486d1316df4d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 1 -- 172.21.15.17:0/459510016 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54032440 con 0x7ffb580a5780 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: dump: 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 1 -- 172.21.15.17:0/459510016 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb540325a0 con 0x7ffb580a5780 2024-09-15T01:55:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5f64f640 5 monclient: authenticate success, global_id 74132 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5f64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5f64f640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 1 -- 172.21.15.17:0/459510016 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54032940 con 0x7ffb580a5780 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb5f64f640 4 set_mon_vals no callback set 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.520+0000 7ffb4bfff640 10 monclient: dump: 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 10 monclient: shutdown 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 -- 172.21.15.17:0/459510016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 msgr2=0x7ffb5809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/459510016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb5809bc90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009880 tx=0x7ffb540090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 -- 172.21.15.17:0/459510016 shutdown_connections 2024-09-15T01:55:36.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/459510016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb5809bc90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 -- 172.21.15.17:0/459510016 >> 172.21.15.17:0/459510016 conn(0x7ffb58070e10 msgr2=0x7ffb580a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 -- 172.21.15.17:0/459510016 shutdown_connections 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.521+0000 7ffb5f64f640 1 -- 172.21.15.17:0/459510016 wait complete. 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.522+0000 7ffb5f64f640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.522+0000 7ffb5f64f640 10 monclient: monmap: 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:36.531 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.522+0000 7ffb5f64f640 1 Processor -- start 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.522+0000 7ffb5f64f640 1 -- start start 2024-09-15T01:55:36.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.522+0000 7ffb5f64f640 10 monclient: init 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 0x7ffb5813dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-a con 0x7ffb58069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 0x7ffb5813d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-b con 0x7ffb5809ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb58141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): picked mon.noname-c con 0x7ffb580a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58121830 con 0x7ffb58069c90 2024-09-15T01:55:36.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5813eaf0 con 0x7ffb5809ccf0 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58122700 con 0x7ffb580a5780 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5f64f640 10 monclient(hunting): authenticate will time out at 996.519086s 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 0x7ffb5813dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb58141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 0x7ffb5813dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53882/0 (socket says 172.21.15.17:53882) 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 1 -- 172.21.15.17:0/339072516 learned_addr learned my addr 172.21.15.17:0/339072516 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): get_auth_request con 0x7ffb58069c90 auth_method 0 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): get_auth_request con 0x7ffb580a5780 auth_method 0 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 0x7ffb5813d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5d3eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 10 monclient(hunting): get_auth_request con 0x7ffb5809ccf0 auth_method 0 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient(hunting): handle_auth_done global_id 74135 payload 824 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 -- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 msgr2=0x7ffb5813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 0x7ffb5813d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 -- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 msgr2=0x7ffb5813dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 0x7ffb5813dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 monclient: found mon.noname-c 2024-09-15T01:55:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5dbec640 1 -- 172.21.15.17:0/339072516 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54009390 con 0x7ffb580a5780 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.523+0000 7ffb5cbea640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 0x7ffb5813d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb5dbec640 1 -- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 msgr2=0x7ffb58141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb5dbec640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb58141520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000b810 tx=0x7ffb500092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000c940 con 0x7ffb580a5780 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 10 monclient: dump: 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb5f64f640 5 monclient: authenticate success, global_id 74135 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5000cae0 con 0x7ffb580a5780 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb5f64f640 10 monclient: _renew_subs 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000cea0 con 0x7ffb580a5780 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb5f64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb581229e0 con 0x7ffb580a5780 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.525+0000 7ffb4b7fe640 4 set_mon_vals no callback set 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb497fa640 10 monclient: dump: 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb5f64f640 10 monclient: _renew_subs 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb5f64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb58121a20 con 0x7ffb580a5780 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.526+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffb50014620 con 0x7ffb580a5780 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5f64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5f64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5f64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5809b940 con 0x7ffb580a5780 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb497fa640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 0x7ffb3c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6014+0+0 (secure 0 0 0) 0x7ffb5004b940 con 0x7ffb580a5780 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5d3eb640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 0x7ffb3c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.527+0000 7ffb5d3eb640 10 monclient: get_auth_request con 0x7ffb3c038740 auth_method 0 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.530+0000 7ffb5d3eb640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 0x7ffb3c03abf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009360 tx=0x7ffb54032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.532+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffb5809b940 con 0x7ffb580a5780 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.532+0000 7ffb497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.532+0000 7ffb497fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:36.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.584+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v104) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0e24016070 con 0x7f0e34098d40 2024-09-15T01:55:36.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.584+0000 7f0e29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:36.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.584+0000 7f0e29ffb640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-15T01:55:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.698+0000 7f0e3bd9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.698+0000 7f0e3bd9d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.698+0000 7f0e3bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.698+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f0e34067a20 con 0x7f0e34098d40 2024-09-15T01:55:36.702 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:55:36.700+0000 7f73a1ff1640 -1 mon.a@0(leader).osd e104 definitely_dead 0 2024-09-15T01:55:36.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.700+0000 7f0e29ffb640 1 -- 172.21.15.67:0/3356926197 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v104) v1 ==== 96+0+0 (secure 0 0 0) 0x7f0e240102e0 con 0x7f0e34098d40 2024-09-15T01:55:36.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.700+0000 7f0e29ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T01:55:36.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.700+0000 7f0e29ffb640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-15T01:55:36.702 INFO:tasks.workunit.client.0.smithi067.stderr:osd.11 does not exist. 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 msgr2=0x7f0e1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 0x7f0e1803abf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0e300076a0 tx=0x7f0e3000b040 comp rx=0 tx=0).stop 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 10 monclient: shutdown 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 msgr2=0x7f0e34134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400c5e0 tx=0x7f0e24007ed0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 shutdown_connections 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e18038740 0x7f0e1803abf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e34131f90 0x7f0e341350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e34098d40 0x7f0e34134bb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 --2- 172.21.15.67:0/3356926197 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e34098260 0x7f0e34134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 >> 172.21.15.67:0/3356926197 conn(0x7f0e34071690 msgr2=0x7f0e340a33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 shutdown_connections 2024-09-15T01:55:36.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.702+0000 7f0e3bd9d640 1 -- 172.21.15.67:0/3356926197 wait complete. 2024-09-15T01:55:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.766+0000 7ffb5f64f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.766+0000 7ffb5f64f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.766+0000 7ffb5f64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.766+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffb5809b940 con 0x7ffb580a5780 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.767+0000 7ffb497fa640 1 -- 172.21.15.17:0/339072516 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7ffb5809b940 con 0x7ffb580a5780 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.767+0000 7ffb497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.767+0000 7ffb497fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:36.769 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 msgr2=0x7ffb3c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 0x7ffb3c03abf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009360 tx=0x7ffb54032000 comp rx=0 tx=0).stop 2024-09-15T01:55:36.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 10 monclient: shutdown 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 msgr2=0x7ffb58141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb58141520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000b810 tx=0x7ffb500092b0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 shutdown_connections 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb3c038740 0x7ffb3c03abf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb580a5780 0x7ffb58141520 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb5809ccf0 0x7ffb5813d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 --2- 172.21.15.17:0/339072516 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb58069c90 0x7ffb5813dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 >> 172.21.15.17:0/339072516 conn(0x7ffb58070e10 msgr2=0x7ffb5809b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 shutdown_connections 2024-09-15T01:55:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.769+0000 7ffb5f64f640 1 -- 172.21.15.17:0/339072516 wait complete. 2024-09-15T01:55:36.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:55:36.782 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:55:36.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:55:36.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:55:36.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.785+0000 7fa292b1d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.785+0000 7fa292b1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.785+0000 7fa292b1d640 10 monclient: monmap: 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:36.786685+0000 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:36.786685+0000 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:36.788 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:36.789 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 Processor -- start 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 -- start start 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient: init 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa28c130e00 0x7fa28c132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa28c130e00 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:36.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c1311d0 0x7fa28c1315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa28c1311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa28c068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c09ff20 con 0x7fa28c068dc0 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28c1320b0 con 0x7fa28c130e00 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a17b0 con 0x7fa28c1311d0 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa292b1d640 10 monclient(hunting): authenticate will time out at 969.255837s 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa2908b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa28c130e00 0x7fa28c132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.786+0000 7fa2910ba640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa28c068dc0 0x7fa28c069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49954/0 (socket says 172.21.15.67:49954) 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2910ba640 1 -- 172.21.15.67:0/2915210943 learned_addr learned my addr 172.21.15.67:0/2915210943 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): get_auth_request con 0x7fa28c130e00 auth_method 0 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c1311d0 0x7fa28c1315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): get_auth_request con 0x7fa28c1311d0 auth_method 0 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient(hunting): handle_auth_done global_id 5536 payload 274 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 1 -- 172.21.15.67:0/2915210943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c1311d0 msgr2=0x7fa28c1315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c1311d0 0x7fa28c1315c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 1 -- 172.21.15.67:0/2915210943 >> v1:172.21.15.67:6789/0 conn(0x7fa28c068dc0 legacy=0x7fa28c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 1 monclient: found mon.noname-c 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.787+0000 7fa2908b9640 1 -- 172.21.15.67:0/2915210943 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c0a00f0 con 0x7fa28c130e00 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa2908b9640 1 -- 172.21.15.67:0/2915210943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 msgr2=0x7fa28c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa2908b9640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c132430 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa280009850 tx=0x7fa2800090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8534ccdc061a1fc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 1 -- 172.21.15.67:0/2915210943 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28002adf0 con 0x7fa28c130e00 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: dump: 2024-09-15T01:55:36.800 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 5 monclient: authenticate success, global_id 5536 2024-09-15T01:55:36.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 1 -- 172.21.15.67:0/2915210943 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa280032440 con 0x7fa28c130e00 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 4 set_mon_vals no callback set 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 1 -- 172.21.15.67:0/2915210943 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800327e0 con 0x7fa28c130e00 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa28b7fe640 10 monclient: dump: 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.802 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 10 monclient: shutdown 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 -- 172.21.15.67:0/2915210943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 msgr2=0x7fa28c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c132430 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa280009850 tx=0x7fa2800090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 -- 172.21.15.67:0/2915210943 shutdown_connections 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c1311d0 0x7fa28c1315c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 --2- 172.21.15.67:0/2915210943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c132430 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 -- 172.21.15.67:0/2915210943 >> 172.21.15.67:0/2915210943 conn(0x7fa28c0710f0 msgr2=0x7fa28c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 -- 172.21.15.67:0/2915210943 shutdown_connections 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.790+0000 7fa292b1d640 1 -- 172.21.15.67:0/2915210943 wait complete. 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.791+0000 7fa292b1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.791+0000 7fa292b1d640 10 monclient: monmap: 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:36.804 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.791+0000 7fa292b1d640 1 Processor -- start 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.791+0000 7fa292b1d640 1 -- start start 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.791+0000 7fa292b1d640 10 monclient: init 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c068dc0 0x7fa28c12ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa28c068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa28c130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 0x7fa28c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa28c1311d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c09eb80 con 0x7fa28c1311d0 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a0820 con 0x7fa28c130e00 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a4a30 con 0x7fa28c068dc0 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa292b1d640 10 monclient(hunting): authenticate will time out at 969.261360s 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46482/0 (socket says 172.21.15.67:46482) 2024-09-15T01:55:36.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 -- 172.21.15.67:0/3307861626 learned_addr learned my addr 172.21.15.67:0/3307861626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c068dc0 0x7fa28c12ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): get_auth_request con 0x7fa28c130e00 auth_method 0 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): get_auth_request con 0x7fa28c068dc0 auth_method 0 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 0x7fa28c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): get_auth_request con 0x7fa28c1311d0 auth_method 0 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2908b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient(hunting): handle_auth_done global_id 5539 payload 824 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 -- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c068dc0 msgr2=0x7fa28c12ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c068dc0 0x7fa28c12ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 -- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 msgr2=0x7fa28c1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 0x7fa28c1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 monclient: found mon.noname-b 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa28bfff640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280009390 con 0x7fa28c130e00 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.792+0000 7fa2910ba640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 0x7fa28c1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa28bfff640 1 -- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 msgr2=0x7fa28c1413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa28bfff640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa27800ea80 tx=0x7fa2780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27800cc20 con 0x7fa28c130e00 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: dump: 2024-09-15T01:55:36.809 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27800cd80 con 0x7fa28c130e00 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278013680 con 0x7fa28c130e00 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa288ff9640 10 monclient: dump: 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:36.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 5 monclient: authenticate success, global_id 5539 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 10 monclient: _renew_subs 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c09e690 con 0x7fa28c130e00 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa28affd640 4 set_mon_vals no callback set 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 10 monclient: _renew_subs 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.795+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c09edd0 con 0x7fa28c130e00 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa278013a70 con 0x7fa28c130e00 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa292b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa292b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa292b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c04bf60 con 0x7fa28c130e00 2024-09-15T01:55:36.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa288ff9640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 0x7fa27003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6004+0+0 (secure 0 0 0) 0x7fa27804c690 con 0x7fa28c130e00 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa2908b9640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 0x7fa27003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.796+0000 7fa2908b9640 10 monclient: get_auth_request con 0x7fa270038740 auth_method 0 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.798+0000 7fa2908b9640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 0x7fa27003abf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa28c068d50 tx=0x7fa280032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.800+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa27801bde0 con 0x7fa28c130e00 2024-09-15T01:55:36.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.800+0000 7fa288ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.800+0000 7fa288ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.877+0000 7f9d20c68640 10 monclient: get_monmap_and_config 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.877+0000 7f9d20c68640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.877+0000 7f9d20c68640 10 monclient: monmap: 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:36.878217+0000 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:36.878217+0000 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:36.880 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 Processor -- start 2024-09-15T01:55:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 -- start start 2024-09-15T01:55:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient: init 2024-09-15T01:55:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d1c06eb30 0x7f9d1c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-c con 0x7f9d1c06eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-b con 0x7f9d1c06f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d1c09c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1c09f6a0 con 0x7f9d1c09c4d0 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d1c069f40 con 0x7f9d1c06eb30 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c069950 con 0x7f9d1c06f440 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d20c68640 10 monclient(hunting): authenticate will time out at 996.874717s 2024-09-15T01:55:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54812/0 (socket says 172.21.15.17:54812) 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 1 -- 172.21.15.17:0/3355093620 learned_addr learned my addr 172.21.15.17:0/3355093620 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): get_auth_request con 0x7f9d1c06f440 auth_method 0 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/3355093620 >> v2:172.21.15.67:3300/0 conn(0x7f9d1c06eb30 0x7f9d1c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request con 0x7f9d1c06eb30 auth_method 0 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.878+0000 7f9d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 10 monclient(hunting): handle_auth_done global_id 74144 payload 274 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 1 -- 172.21.15.17:0/3355093620 >> v2:172.21.15.67:3300/0 conn(0x7f9d1c06eb30 msgr2=0x7f9d1c06ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 1 --2- 172.21.15.17:0/3355093620 >> v2:172.21.15.67:3300/0 conn(0x7f9d1c06eb30 0x7f9d1c06ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 1 -- 172.21.15.17:0/3355093620 >> v1:172.21.15.67:6789/0 conn(0x7f9d1c09c4d0 legacy=0x7f9d1c09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 1 monclient: found mon.noname-b 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.879+0000 7f9d13fff640 1 -- 172.21.15.17:0/3355093620 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c09ad80 con 0x7f9d1c06f440 2024-09-15T01:55:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.881+0000 7f9d13fff640 1 -- 172.21.15.17:0/3355093620 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 msgr2=0x7f9d1c09bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d13fff640 1 --2- 172.21.15.17:0/3355093620 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911157a04edcd63e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 1 -- 172.21.15.17:0/3355093620 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0402aea0 con 0x7f9d1c06f440 2024-09-15T01:55:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: dump: 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 5 monclient: authenticate success, global_id 74144 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 1 -- 172.21.15.17:0/3355093620 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04032440 con 0x7f9d1c06f440 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 1 -- 172.21.15.17:0/3355093620 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04032820 con 0x7f9d1c06f440 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 4 set_mon_vals no callback set 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d19d9b640 10 monclient: dump: 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 10 monclient: shutdown 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 -- 172.21.15.17:0/3355093620 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 msgr2=0x7f9d1c09bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 --2- 172.21.15.17:0/3355093620 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d040091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 -- 172.21.15.17:0/3355093620 shutdown_connections 2024-09-15T01:55:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 --2- 172.21.15.17:0/3355093620 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c09bf90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 --2- 172.21.15.17:0/3355093620 >> v2:172.21.15.67:3300/0 conn(0x7f9d1c06eb30 0x7f9d1c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 -- 172.21.15.17:0/3355093620 >> 172.21.15.17:0/3355093620 conn(0x7f9d1c0712f0 msgr2=0x7f9d1c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 -- 172.21.15.17:0/3355093620 shutdown_connections 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.882+0000 7f9d20c68640 1 -- 172.21.15.17:0/3355093620 wait complete. 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 10 monclient: build_initial_monmap 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 10 monclient: monmap: 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 1 Processor -- start 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 1 -- start start 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 10 monclient: init 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.883+0000 7f9d20c68640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1c06eb30 0x7f9d1c140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-b con 0x7f9d1c06eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-c con 0x7f9d1c06f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c09c4d0 0x7f9d1c11d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): picked mon.noname-a con 0x7f9d1c09c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c09cd10 con 0x7f9d1c09c4d0 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1213f0 con 0x7f9d1c06eb30 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c121110 con 0x7f9d1c06f440 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d20c68640 10 monclient(hunting): authenticate will time out at 996.880068s 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54822/0 (socket says 172.21.15.17:54822) 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 -- 172.21.15.17:0/2579060106 learned_addr learned my addr 172.21.15.17:0/2579060106 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): get_auth_request con 0x7f9d1c06f440 auth_method 0 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c09c4d0 0x7f9d1c11d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): get_auth_request con 0x7f9d1c09c4d0 auth_method 0 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1c06eb30 0x7f9d1c140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request con 0x7f9d1c06eb30 auth_method 0 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient(hunting): handle_auth_done global_id 74147 payload 824 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1c06eb30 msgr2=0x7f9d1c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1c06eb30 0x7f9d1c140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c09c4d0 msgr2=0x7f9d1c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c09c4d0 0x7f9d1c11d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 monclient: found mon.noname-c 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:36.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.884+0000 7f9d13fff640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009450 con 0x7f9d1c06f440 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d13fff640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 msgr2=0x7f9d1c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d13fff640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c132d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009970 tx=0x7f9d04032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0402aea0 con 0x7f9d1c06f440 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: dump: 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d20c68640 5 monclient: authenticate success, global_id 74147 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04000c40 con 0x7f9d1c06f440 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04031ea0 con 0x7f9d1c06f440 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d20c68640 10 monclient: _renew_subs 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d20c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c141e50 con 0x7f9d1c06f440 2024-09-15T01:55:36.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d1959a640 4 set_mon_vals no callback set 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.886+0000 7f9d127fc640 10 monclient: dump: 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.903 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 10 monclient: _renew_subs 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c1200e0 con 0x7f9d1c06f440 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 0x7f9d1c11df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c063b90 con 0x7f9d1c141530 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d0403c610 con 0x7f9d1c06f440 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 0x7f9d1c11df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient: get_auth_request con 0x7f9d1c141530 auth_method 0 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.887+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d127fc640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cfc038740 0x7f9cfc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_done global_id 44877 payload 824 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f9d0406c610 con 0x7f9d1c06f440 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d1a59c640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d1c120fa0 con 0x7f9d1c141530 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d1ad9d640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cfc038740 0x7f9cfc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.888+0000 7f9d1ad9d640 10 monclient: get_auth_request con 0x7f9cfc038740 auth_method 0 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d1a59c640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 msgr2=0x7f9d1c11df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 0x7f9d1c11df20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c011c20 tx=0x7f9d0c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c011e60 con 0x7f9d1c141530 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 10 monclient: dump: 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.908 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9d0c009e00 con 0x7f9d1c141530 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 msgr2=0x7f9d1c11df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 0x7f9d1c11df20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c011c20 tx=0x7f9d0c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d1ad9d640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cfc038740 0x7f9cfc03abf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9d14007fd0 tx=0x7f9d140025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 msgr2=0x7f9d1c11df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.890+0000 7f9d127fc640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c141530 0x7f9d1c11df20 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c011c20 tx=0x7f9d0c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d20c68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d20c68640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 0x7f9d1c11e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d20c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c121250 con 0x7f9d1c133240 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 0x7f9d1c11e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient: get_auth_request con 0x7f9d1c133240 auth_method 0 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:36.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 10 monclient(hunting): handle_auth_done global_id 44880 payload 824 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.896+0000 7f9d1a59c640 1 -- 172.21.15.17:0/2579060106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d1c0724a0 con 0x7f9d1c133240 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d1a59c640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 msgr2=0x7f9d1c11e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d1a59c640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 0x7f9d1c11e460 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c01b000 tx=0x7f9d0c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c01de50 con 0x7f9d1c133240 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 10 monclient: dump: 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:36.911 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:36.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f9d0c01c3e0 con 0x7f9d1c133240 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 msgr2=0x7f9d1c11e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 0x7f9d1c11e460 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c01b000 tx=0x7f9d0c01c000 comp rx=0 tx=0).stop 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 msgr2=0x7f9d1c11e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d127fc640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d1c133240 0x7f9d1c11e460 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c01b000 tx=0x7f9d0c01c000 comp rx=0 tx=0).stop 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:36.926 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cfc038740 msgr2=0x7f9cfc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cfc038740 0x7f9cfc03abf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9d14007fd0 tx=0x7f9d140025b0 comp rx=0 tx=0).stop 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 10 monclient: shutdown 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 msgr2=0x7f9d1c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 --2- 172.21.15.17:0/2579060106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d1c06f440 0x7f9d1c132d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009970 tx=0x7f9d04032000 comp rx=0 tx=0).stop 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d1ad9d640 1 -- 172.21.15.17:0/2579060106 reap_dead start 2024-09-15T01:55:36.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d1ad9d640 1 -- 172.21.15.17:0/2579060106 reap_dead start 2024-09-15T01:55:36.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 shutdown_connections 2024-09-15T01:55:36.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 >> 172.21.15.17:0/2579060106 conn(0x7f9d1c0712f0 msgr2=0x7f9d1c09eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:36.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 shutdown_connections 2024-09-15T01:55:36.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:36.899+0000 7f9d20c68640 1 -- 172.21.15.17:0/2579060106 wait complete. 2024-09-15T01:55:36.929 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:36.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:55:36.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:55:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.975+0000 7fa292b1d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.975+0000 7fa292b1d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T01:55:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.975+0000 7fa292b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:36.975+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa28c0a14d0 con 0x7fa28c130e00 2024-09-15T01:55:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.025+0000 7f09ad59d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.025+0000 7f09ad59d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.025+0000 7f09ad59d640 10 monclient: monmap: 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:37.026333+0000 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:37.026333+0000 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:37.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 Processor -- start 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 -- start start 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient: init 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a8097320 0x7f09a80976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-a con 0x7f09a8097320 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-b con 0x7f09a809a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09a809a960 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a80985d0 con 0x7f09a809a960 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a809d200 con 0x7f09a8097320 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8097c30 con 0x7f09a809a590 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09ad59d640 10 monclient(hunting): authenticate will time out at 997.022784s 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a8097320 0x7f09a80976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54826/0 (socket says 172.21.15.17:54826) 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a8097320 0x7f09a80976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53902/0 (socket says 172.21.15.17:53902) 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 1 -- 172.21.15.17:0/4185959938 learned_addr learned my addr 172.21.15.17:0/4185959938 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 10 monclient(hunting): get_auth_request con 0x7f09a809a590 auth_method 0 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a77fe640 1 --1- 172.21.15.17:0/4185959938 >> v1:172.21.15.17:6789/0 conn(0x7f09a809a960 0x7f09a812ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36020/0 (socket says 172.21.15.17:36020) 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8097320 auth_method 0 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.026+0000 7f09a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_done global_id 44883 payload 274 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 1 -- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 msgr2=0x7f09a812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 1 -- 172.21.15.17:0/4185959938 >> v1:172.21.15.17:6789/0 conn(0x7f09a809a960 legacy=0x7f09a812ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 1 monclient: found mon.noname-a 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a6ffd640 1 -- 172.21.15.17:0/4185959938 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80992f0 con 0x7f09a8097320 2024-09-15T01:55:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.027+0000 7f09a67fc640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a6ffd640 1 -- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 msgr2=0x7f09a80976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a80976f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0998009850 tx=0x7f09980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4262c7343c7e7bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 1 -- 172.21.15.17:0/4185959938 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099802adf0 con 0x7f09a8097320 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: dump: 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 1 -- 172.21.15.17:0/4185959938 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998032440 con 0x7f09a8097320 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 1 -- 172.21.15.17:0/4185959938 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998032820 con 0x7f09a8097320 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 5 monclient: authenticate success, global_id 44883 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 4 set_mon_vals no callback set 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09a5ffb640 10 monclient: dump: 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.043 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 10 monclient: shutdown 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 -- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 msgr2=0x7f09a80976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a80976f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0998009850 tx=0x7f09980090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 -- 172.21.15.17:0/4185959938 shutdown_connections 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 --2- 172.21.15.17:0/4185959938 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a80976f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 -- 172.21.15.17:0/4185959938 >> 172.21.15.17:0/4185959938 conn(0x7f09a80a1000 msgr2=0x7f09a80a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 -- 172.21.15.17:0/4185959938 shutdown_connections 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.030+0000 7f09ad59d640 1 -- 172.21.15.17:0/4185959938 wait complete. 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.031+0000 7f09ad59d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.031+0000 7f09ad59d640 10 monclient: monmap: 2024-09-15T01:55:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.031+0000 7f09ad59d640 1 Processor -- start 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.031+0000 7f09ad59d640 1 -- start start 2024-09-15T01:55:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.031+0000 7f09ad59d640 10 monclient: init 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-a con 0x7f09a8097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a811a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-c con 0x7f09a809a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a809a960 0x7f09a811a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): picked mon.noname-b con 0x7f09a809a960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a811d6b0 con 0x7f09a8097320 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a809ad50 con 0x7f09a809a960 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a812e860 con 0x7f09a809a590 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09ad59d640 10 monclient(hunting): authenticate will time out at 997.028395s 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a811a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53916/0 (socket says 172.21.15.17:53916) 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 -- 172.21.15.17:0/2350993363 learned_addr learned my addr 172.21.15.17:0/2350993363 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8097320 auth_method 0 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): get_auth_request con 0x7f09a809a590 auth_method 0 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a809a960 0x7f09a811a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): get_auth_request con 0x7f09a809a960 auth_method 0 2024-09-15T01:55:37.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient(hunting): handle_auth_done global_id 44886 payload 824 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 msgr2=0x7f09a811a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a811a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a809a960 msgr2=0x7f09a811a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a809a960 0x7f09a811a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 monclient: found mon.noname-a 2024-09-15T01:55:37.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a6ffd640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c00b700 con 0x7f09a8097320 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.032+0000 7f09a67fc640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a809a590 0x7f09a811a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09a6ffd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 msgr2=0x7f09a812d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0998009360 tx=0x7f0998032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998031a10 con 0x7f09a8097320 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: dump: 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.051 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 5 monclient: authenticate success, global_id 44886 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998031b70 con 0x7f09a8097320 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 10 monclient: _renew_subs 2024-09-15T01:55:37.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0998009390 con 0x7f09a8097320 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099803c400 con 0x7f09a8097320 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f098effd640 10 monclient: dump: 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.053 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 10 monclient: _renew_subs 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09a57fa640 4 set_mon_vals no callback set 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.035+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a811c7c0 con 0x7f09a8097320 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09ad59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a6ffd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0998009360 tx=0x7f0998032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f099803c7f0 mgrmap(e 5) v1 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09ad59d640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 0x7f09a811ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8063c20 con 0x7f09a812daa0 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f099803c7f0 con 0x7f09a8097320 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 0x7f09a811ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient: get_auth_request con 0x7f09a812daa0 auth_method 0 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f098effd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0994038740 0x7f099403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 10 monclient(hunting): handle_auth_done global_id 74156 payload 824 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a67fc640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09a811db20 con 0x7f09a812daa0 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a77fe640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0994038740 0x7f099403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.036+0000 7f09a77fe640 10 monclient: get_auth_request con 0x7f0994038740 auth_method 0 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f09a67fc640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 msgr2=0x7f09a811ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f09a67fc640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 0x7f09a811ab70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c00b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c016070 con 0x7f09a812daa0 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f098effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.038+0000 7f098effd640 10 monclient: dump: 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.056 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f099c010e80 con 0x7f09a812daa0 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f09a77fe640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0994038740 0x7f099403abf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f09a811e440 tx=0x7f09a0004e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 msgr2=0x7f09a811ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 0x7f09a811ab70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c00b460 comp rx=0 tx=0).stop 2024-09-15T01:55:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 msgr2=0x7f09a811ab70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.039+0000 7f098effd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a812daa0 0x7f09a811ab70 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c00b460 comp rx=0 tx=0).stop 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09ad59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09ad59d640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 0x7f09a81174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09ad59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a812de70 con 0x7f09a811b0b0 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 0x7f09a81174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient: get_auth_request con 0x7f09a811b0b0 auth_method 0 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 10 monclient(hunting): handle_auth_done global_id 74159 payload 824 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.045+0000 7f09a67fc640 1 -- 172.21.15.17:0/2350993363 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09a811b4b0 con 0x7f09a811b0b0 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f09a67fc640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 msgr2=0x7f09a81174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f09a67fc640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 0x7f09a81174c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c01e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c01a8d0 con 0x7f09a811b0b0 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f098effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.047+0000 7f098effd640 10 monclient: dump: 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f099c01aa30 con 0x7f09a811b0b0 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 msgr2=0x7f09a81174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 0x7f09a81174c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c01e000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 msgr2=0x7f09a81174c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:37.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a811b0b0 0x7f09a81174c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f099c006c90 tx=0x7f099c01e000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.063 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:37.063 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.065 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:37.066 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:37.067 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.069 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:37.070 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:37.073 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:37.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:37.075 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0994038740 msgr2=0x7f099403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0994038740 0x7f099403abf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f09a811e440 tx=0x7f09a0004e20 comp rx=0 tx=0).stop 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 10 monclient: shutdown 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 msgr2=0x7f09a812d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0998009360 tx=0x7f0998032000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09a77fe640 1 -- 172.21.15.17:0/2350993363 reap_dead start 2024-09-15T01:55:37.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f098effd640 1 -- 172.21.15.17:0/2350993363 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6004+0+0 (unknown 0 0 0) 0x7f099806c7a0 con 0x7f09a8097320 2024-09-15T01:55:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 shutdown_connections 2024-09-15T01:55:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 --2- 172.21.15.17:0/2350993363 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a8097320 0x7f09a812d560 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 >> 172.21.15.17:0/2350993363 conn(0x7f09a80a1000 msgr2=0x7f09a806c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 shutdown_connections 2024-09-15T01:55:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.048+0000 7f09ad59d640 1 -- 172.21.15.17:0/2350993363 wait complete. 2024-09-15T01:55:37.078 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:37.078 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:55:37.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:55:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.173+0000 7fc4c5f1d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.173+0000 7fc4c5f1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.173+0000 7fc4c5f1d640 10 monclient: monmap: 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:37.174396+0000 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:37.174396+0000 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:37.176 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 Processor -- start 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 -- start start 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient: init 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T01:55:37.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4c006eb30 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc4c009e970 0x7fc4c009bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c009e970 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4c009c490 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c0069c60 con 0x7fc4c009c490 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c009f690 con 0x7fc4c006eb30 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c006a920 con 0x7fc4c009e970 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4c5f1d640 10 monclient(hunting): authenticate will time out at 997.170880s 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.174+0000 7fc4beffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc4c009e970 0x7fc4c009bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc4c009e970 0x7fc4c009bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53928/0 (socket says 172.21.15.17:53928) 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4bffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc4c009c490 0x7fc4c0139b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36034/0 (socket says 172.21.15.17:36034) 2024-09-15T01:55:37.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 -- 172.21.15.17:0/2076030116 learned_addr learned my addr 172.21.15.17:0/2076030116 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): get_auth_request con 0x7fc4c009e970 auth_method 0 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient(hunting): handle_auth_done global_id 44889 payload 274 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 -- 172.21.15.17:0/2076030116 >> v1:172.21.15.67:6789/0 conn(0x7fc4c006eb30 legacy=0x7fc4c006ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 -- 172.21.15.17:0/2076030116 >> v1:172.21.15.17:6789/0 conn(0x7fc4c009c490 legacy=0x7fc4c0139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 monclient: found mon.noname-a 2024-09-15T01:55:37.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.175+0000 7fc4beffd640 1 -- 172.21.15.17:0/2076030116 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c009ed40 con 0x7fc4c009e970 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4beffd640 1 -- 172.21.15.17:0/2076030116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 msgr2=0x7fc4c009bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4beffd640 1 --2- 172.21.15.17:0/2076030116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c009bf50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009910 tx=0x7fc4b40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2586ad60c57f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 1 -- 172.21.15.17:0/2076030116 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4032440 con 0x7fc4c009e970 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: dump: 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.190 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 5 monclient: authenticate success, global_id 44889 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 1 -- 172.21.15.17:0/2076030116 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b40325a0 con 0x7fc4c009e970 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 1 -- 172.21.15.17:0/2076030116 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4032940 con 0x7fc4c009e970 2024-09-15T01:55:37.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 4 set_mon_vals no callback set 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4be7fc640 10 monclient: dump: 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 10 monclient: shutdown 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/2076030116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 msgr2=0x7fc4c009bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.178+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/2076030116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c009bf50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009910 tx=0x7fc4b40091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/2076030116 shutdown_connections 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/2076030116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c009bf50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/2076030116 >> 172.21.15.17:0/2076030116 conn(0x7fc4c00712f0 msgr2=0x7fc4c00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/2076030116 shutdown_connections 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/2076030116 wait complete. 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 10 monclient: monmap: 2024-09-15T01:55:37.193 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.179+0000 7fc4c5f1d640 1 Processor -- start 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 -- start start 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient: init 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc4c006eb30 0x7fc4c0141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c009c490 0x7fc4c0133dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c009c490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c011d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c009e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c011f510 con 0x7fc4c009e970 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c01420f0 con 0x7fc4c009c490 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c01211d0 con 0x7fc4c006eb30 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4c5f1d640 10 monclient(hunting): authenticate will time out at 997.176500s 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c011d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc4c006eb30 0x7fc4c0141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c011d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53940/0 (socket says 172.21.15.17:53940) 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc4c006eb30 0x7fc4c0141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54870/0 (socket says 172.21.15.17:54870) 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 learned_addr learned my addr 172.21.15.17:0/377015588 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): get_auth_request con 0x7fc4c009e970 auth_method 0 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc4c006eb30 auth_method 0 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c009c490 0x7fc4c0133dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): get_auth_request con 0x7fc4c009c490 auth_method 0 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient(hunting): handle_auth_done global_id 44892 payload 824 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc4c006eb30 msgr2=0x7fc4c0141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc4c006eb30 0x7fc4c0141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c009c490 msgr2=0x7fc4c0133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c009c490 0x7fc4c0133dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 monclient: found mon.noname-a 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.180+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4009450 con 0x7fc4c009e970 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 msgr2=0x7fc4c011d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4bffff640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c011d9e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c880 tx=0x7fc4b8004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800ce70 con 0x7fc4c009e970 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: dump: 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.199 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b800a5e0 con 0x7fc4c009e970 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 5 monclient: authenticate success, global_id 44892 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 10 monclient: _renew_subs 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c0120ef0 con 0x7fc4c009e970 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800a9a0 con 0x7fc4c009e970 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4a77fe640 10 monclient: dump: 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.200 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 10 monclient: _renew_subs 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c01418f0 con 0x7fc4c009e970 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.183+0000 7fc4bdffb640 4 set_mon_vals no callback set 2024-09-15T01:55:37.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4c5f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 0x7fc4c011df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0063b90 con 0x7fc4c0134310 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc4b80070f0 con 0x7fc4c009e970 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4a77fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4ac038740 0x7fc4ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6004+0+0 (secure 0 0 0) 0x7fc4b804bd20 con 0x7fc4c009e970 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 0x7fc4c011df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient: get_auth_request con 0x7fc4c0134310 auth_method 0 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4beffd640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4ac038740 0x7fc4ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4beffd640 10 monclient: get_auth_request con 0x7fc4ac038740 auth_method 0 2024-09-15T01:55:37.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_done global_id 5554 payload 824 2024-09-15T01:55:37.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.184+0000 7fc4bf7fe640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4c00724a0 con 0x7fc4c0134310 2024-09-15T01:55:37.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.186+0000 7fc4bf7fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 msgr2=0x7fc4c011df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.186+0000 7fc4bf7fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 0x7fc4c011df20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e7b0 tx=0x7fc4b0006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4beffd640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4ac038740 0x7fc4ac03abf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4034000 tx=0x7fc4b4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000cac0 con 0x7fc4c0134310 2024-09-15T01:55:37.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 10 monclient: dump: 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc4b000cc20 con 0x7fc4c0134310 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 msgr2=0x7fc4c011df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 0x7fc4c011df20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e7b0 tx=0x7fc4b0006f50 comp rx=0 tx=0).stop 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 msgr2=0x7fc4c011df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.187+0000 7fc4a77fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c0134310 0x7fc4c011df20 secure :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e7b0 tx=0x7fc4b0006f50 comp rx=0 tx=0).stop 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4c5f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 0x7fc4c00a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4c5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c01346e0 con 0x7fc4c011e460 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 0x7fc4c00a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient: get_auth_request con 0x7fc4c011e460 auth_method 0 2024-09-15T01:55:37.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 10 monclient(hunting): handle_auth_done global_id 5557 payload 824 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.193+0000 7fc4bf7fe640 1 -- 172.21.15.17:0/377015588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4c0134bb0 con 0x7fc4c011e460 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4bf7fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 msgr2=0x7fc4c00a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4bf7fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 0x7fc4c00a13a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009fd0 tx=0x7fc4b0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b001b940 con 0x7fc4c011e460 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 10 monclient: dump: 2024-09-15T01:55:37.208 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7fc4b001baa0 con 0x7fc4c011e460 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 msgr2=0x7fc4c00a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 0x7fc4c00a13a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009fd0 tx=0x7fc4b0015000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 msgr2=0x7fc4c00a13a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4a77fe640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4c011e460 0x7fc4c00a13a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 52, 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:37.210 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.211 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.212 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:37.213 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:37.214 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:37.215 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:37.218 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:37.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:37.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:55:37.222 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:37.223 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:37.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.205+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v105) v1 ==== 58+0+3 (secure 0 0 0) 0x7fa278021080 con 0x7fa28c130e00 2024-09-15T01:55:37.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.205+0000 7fa288ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T01:55:37.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.205+0000 7fa288ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4ac038740 msgr2=0x7fc4ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc4ac038740 0x7fc4ac03abf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4034000 tx=0x7fc4b4032000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 10 monclient: shutdown 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 msgr2=0x7fc4c011d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 --2- 172.21.15.17:0/377015588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc4c009e970 0x7fc4c011d9e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c880 tx=0x7fc4b8004ef0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 reap_dead start 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4bffff640 1 -- 172.21.15.17:0/377015588 reap_dead start 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 shutdown_connections 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 >> 172.21.15.17:0/377015588 conn(0x7fc4c00712f0 msgr2=0x7fc4c0134de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.196+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 shutdown_connections 2024-09-15T01:55:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.197+0000 7fc4c5f1d640 1 -- 172.21.15.17:0/377015588 wait complete. 2024-09-15T01:55:37.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:55:37.227 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T01:55:37.227 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:55:37.227 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T01:55:37.227 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:55:37.227 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T01:55:37.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:55:37.238 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T01:55:37.238 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:55:37.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:37.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.318+0000 7fa292b1d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:37.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.318+0000 7fa292b1d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T01:55:37.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.318+0000 7fa292b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:37.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:37.318+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa28c0008d0 con 0x7fa28c130e00 2024-09-15T01:55:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.371+0000 7f0d42f1d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.371+0000 7f0d42f1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.372+0000 7f0d42f1d640 10 monclient: monmap: 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:37.372807+0000 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:37.372807+0000 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:37.375 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.372+0000 7f0d42f1d640 1 Processor -- start 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.372+0000 7f0d42f1d640 1 -- start start 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.372+0000 7f0d42f1d640 10 monclient: init 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d3c09c4f0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c09c8c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 0x7f0d3c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c06eb30 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:37.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c06a390 con 0x7f0d3c09c4f0 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c099f50 con 0x7f0d3c06eb30 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c09f790 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d42f1d640 10 monclient(hunting): authenticate will time out at 997.369242s 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d414ba640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 msgr2=0x7f0d3c06ef90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d414ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 0x7f0d3c06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54894/0 (socket says 172.21.15.17:54894) 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 -- 172.21.15.17:0/1653959743 learned_addr learned my addr 172.21.15.17:0/1653959743 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): get_auth_request con 0x7f0d3c09c8c0 auth_method 0 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient(hunting): handle_auth_done global_id 74165 payload 274 2024-09-15T01:55:37.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 -- 172.21.15.17:0/1653959743 >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 msgr2=0x7f0d3c06ef90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 --2- 172.21.15.17:0/1653959743 >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 0x7f0d3c06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 -- 172.21.15.17:0/1653959743 >> v1:172.21.15.67:6789/0 conn(0x7f0d3c09c4f0 legacy=0x7f0d3c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 monclient: found mon.noname-b 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.373+0000 7f0d3bfff640 1 -- 172.21.15.17:0/1653959743 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c09b0a0 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3bfff640 1 -- 172.21.15.17:0/1653959743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 msgr2=0x7f0d3c09e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3bfff640 1 --2- 172.21.15.17:0/1653959743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009910 tx=0x7f0d280091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff67be1b6a48de91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 1 -- 172.21.15.17:0/1653959743 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d28032440 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: dump: 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 1 -- 172.21.15.17:0/1653959743 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d280325a0 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 1 -- 172.21.15.17:0/1653959743 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d28032940 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d42f1d640 5 monclient: authenticate success, global_id 74165 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d42f1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d42f1d640 4 set_mon_vals no callback set 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.376+0000 7f0d3b7fe640 10 monclient: dump: 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.392 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 10 monclient: shutdown 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 -- 172.21.15.17:0/1653959743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 msgr2=0x7f0d3c09e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/1653959743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009910 tx=0x7f0d280091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 -- 172.21.15.17:0/1653959743 shutdown_connections 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/1653959743 >> v2:172.21.15.17:3300/0 conn(0x7f0d3c06eb30 0x7f0d3c06ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/1653959743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c09c8c0 0x7f0d3c09e9e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 -- 172.21.15.17:0/1653959743 >> 172.21.15.17:0/1653959743 conn(0x7f0d3c071300 msgr2=0x7f0d3c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 -- 172.21.15.17:0/1653959743 shutdown_connections 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 1 -- 172.21.15.17:0/1653959743 wait complete. 2024-09-15T01:55:37.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 10 monclient: build_initial_monmap 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.377+0000 7f0d42f1d640 10 monclient: monmap: 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:37.394 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 Processor -- start 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 -- start start 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient: init 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 0x7f0d3c133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c09c4f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c09c8c0 0x7f0d3c11da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c09c8c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:37.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c134cd0 con 0x7f0d3c09c4f0 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c1211a0 con 0x7f0d3c09c8c0 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c120ec0 con 0x7f0d3c06eb30 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d42f1d640 10 monclient(hunting): authenticate will time out at 997.374738s 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d3bfff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 msgr2=0x7f0d3c133da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d3bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 0x7f0d3c133da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54906/0 (socket says 172.21.15.17:54906) 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 1 -- 172.21.15.17:0/2663552725 learned_addr learned my addr 172.21.15.17:0/2663552725 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): get_auth_request con 0x7f0d3c06eb30 auth_method 0 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d40cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d414ba640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c09c8c0 0x7f0d3c11da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d414ba640 10 monclient(hunting): get_auth_request con 0x7f0d3c09c8c0 auth_method 0 2024-09-15T01:55:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d414ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d414ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.378+0000 7f0d414ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d414ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d414ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d414ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 10 monclient(hunting): handle_auth_done global_id 74168 payload 824 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 -- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c09c8c0 msgr2=0x7f0d3c11da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c09c8c0 0x7f0d3c11da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 -- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 msgr2=0x7f0d3c133da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 0x7f0d3c133da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 monclient: found mon.noname-c 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.379+0000 7f0d40cb9640 1 -- 172.21.15.17:0/2663552725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28009450 con 0x7f0d3c06eb30 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d40cb9640 1 -- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 msgr2=0x7f0d3c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d40cb9640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000c610 tx=0x7f0d30007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30012840 con 0x7f0d3c06eb30 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: dump: 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d300129c0 con 0x7f0d3c06eb30 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 5 monclient: authenticate success, global_id 74168 2024-09-15T01:55:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 10 monclient: _renew_subs 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c11f420 con 0x7f0d3c06eb30 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30012d80 con 0x7f0d3c06eb30 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d38ff9640 10 monclient: dump: 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d3a7fc640 4 set_mon_vals no callback set 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 10 monclient: _renew_subs 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.381+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c141fd0 con 0x7f0d3c06eb30 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0d30011590 con 0x7f0d3c06eb30 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d42f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d42f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d42f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c121310 con 0x7f0d3c06eb30 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d38ff9640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 0x7f0d2003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f0d3004c180 con 0x7f0d3c06eb30 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d3bfff640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 0x7f0d2003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.382+0000 7f0d3bfff640 10 monclient: get_auth_request con 0x7f0d200386f0 auth_method 0 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.385+0000 7f0d3bfff640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 0x7f0d2003aba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c04b6a0 tx=0x7f0d28032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.386+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0d30010ac0 con 0x7f0d3c06eb30 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.386+0000 7f0d38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.386+0000 7f0d38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:37.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d42f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d42f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d42f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d3c0724a0 con 0x7f0d3c06eb30 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d38ff9640 1 -- 172.21.15.17:0/2663552725 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f0d30016020 con 0x7f0d3c06eb30 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.614+0000 7f0d38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:37.617 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 msgr2=0x7f0d2003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 0x7f0d2003aba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c04b6a0 tx=0x7f0d28032000 comp rx=0 tx=0).stop 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 10 monclient: shutdown 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 msgr2=0x7f0d3c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000c610 tx=0x7f0d30007ed0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 shutdown_connections 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d200386f0 0x7f0d2003aba0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d3c09c4f0 0x7f0d3c133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c09c8c0 0x7f0d3c11da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.616+0000 7f0d42f1d640 1 --2- 172.21.15.17:0/2663552725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d3c06eb30 0x7f0d3c141220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:37.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.617+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 >> 172.21.15.17:0/2663552725 conn(0x7f0d3c071300 msgr2=0x7f0d3c134ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:37.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.617+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 shutdown_connections 2024-09-15T01:55:37.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:37.617+0000 7f0d42f1d640 1 -- 172.21.15.17:0/2663552725 wait complete. 2024-09-15T01:55:39.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:39.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:39.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:39.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:39.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:39.792+0000 7fa2777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:40.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.737+0000 7fb85b35d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.737+0000 7fb85b35d640 10 monclient: build_initial_monmap 2024-09-15T01:55:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.737+0000 7fb85b35d640 10 monclient: monmap: 2024-09-15T01:55:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:40.738094+0000 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:40.738094+0000 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.737+0000 7fb85b35d640 1 Processor -- start 2024-09-15T01:55:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 -- start start 2024-09-15T01:55:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient: init 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb854131b20 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8540a0280 0x7fb854131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8540a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb854132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8540a0040 con 0x7fb854131b20 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85409bff0 con 0x7fb8540a0280 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540a14a0 con 0x7fb854132430 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb85b35d640 10 monclient(hunting): authenticate will time out at 1000.734647s 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54908/0 (socket says 172.21.15.17:54908) 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 1 -- 172.21.15.17:0/3667983072 learned_addr learned my addr 172.21.15.17:0/3667983072 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): get_auth_request con 0x7fb854132430 auth_method 0 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8598fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.738+0000 7fb8588f8640 1 --2- 172.21.15.17:0/3667983072 >> v2:172.21.15.67:3300/0 conn(0x7fb8540a0280 0x7fb854131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8588f8640 10 monclient(hunting): get_auth_request con 0x7fb8540a0280 auth_method 0 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8588f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8588f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8588f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 10 monclient(hunting): handle_auth_done global_id 74171 payload 274 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 1 -- 172.21.15.17:0/3667983072 >> v2:172.21.15.67:3300/0 conn(0x7fb8540a0280 msgr2=0x7fb854131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 1 --2- 172.21.15.17:0/3667983072 >> v2:172.21.15.67:3300/0 conn(0x7fb8540a0280 0x7fb854131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 1 -- 172.21.15.17:0/3667983072 >> v1:172.21.15.67:6789/0 conn(0x7fb854131b20 legacy=0x7fb85409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 10 monclient: _finish_hunting 0 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 1 monclient: found mon.noname-b 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.739+0000 7fb8598fa640 1 -- 172.21.15.17:0/3667983072 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85409f3b0 con 0x7fb854132430 2024-09-15T01:55:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.741+0000 7fb8598fa640 1 -- 172.21.15.17:0/3667983072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 msgr2=0x7fb854138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.741+0000 7fb8598fa640 1 --2- 172.21.15.17:0/3667983072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009f20 tx=0x7fb84c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb50d0417ff084c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 1 -- 172.21.15.17:0/3667983072 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c007610 con 0x7fb854132430 2024-09-15T01:55:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: dump: 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: _finish_auth 0 2024-09-15T01:55:40.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 1 -- 172.21.15.17:0/3667983072 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c02d050 con 0x7fb854132430 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 5 monclient: authenticate success, global_id 74171 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 1 -- 172.21.15.17:0/3667983072 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c0079a0 con 0x7fb854132430 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 4 set_mon_vals no callback set 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb847fff640 10 monclient: dump: 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:40.752 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 10 monclient: shutdown 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 -- 172.21.15.17:0/3667983072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 msgr2=0x7fb854138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 --2- 172.21.15.17:0/3667983072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009f20 tx=0x7fb84c009840 comp rx=0 tx=0).stop 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 -- 172.21.15.17:0/3667983072 shutdown_connections 2024-09-15T01:55:40.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 --2- 172.21.15.17:0/3667983072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb854132430 0x7fb854138e60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 --2- 172.21.15.17:0/3667983072 >> v2:172.21.15.67:3300/0 conn(0x7fb8540a0280 0x7fb854131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 -- 172.21.15.17:0/3667983072 >> 172.21.15.17:0/3667983072 conn(0x7fb854071270 msgr2=0x7fb8540a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 -- 172.21.15.17:0/3667983072 shutdown_connections 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.742+0000 7fb85b35d640 1 -- 172.21.15.17:0/3667983072 wait complete. 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.743+0000 7fb85b35d640 10 monclient: build_initial_monmap 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.743+0000 7fb85b35d640 10 monclient: monmap: 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.754 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.743+0000 7fb85b35d640 1 Processor -- start 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.743+0000 7fb85b35d640 1 -- start start 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.743+0000 7fb85b35d640 10 monclient: init 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8540a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb854131b20 0x7fb8541338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb854131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:40.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 0x7fb854133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb854132430 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8541375b0 con 0x7fb854132430 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540a09b0 con 0x7fb854131b20 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85409edd0 con 0x7fb8540a0280 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb85b35d640 10 monclient(hunting): authenticate will time out at 1000.740427s 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8598fa640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 msgr2=0x7fb854133e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8598fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 0x7fb854133e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:40.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54912/0 (socket says 172.21.15.17:54912) 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 -- 172.21.15.17:0/791207156 learned_addr learned my addr 172.21.15.17:0/791207156 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): get_auth_request con 0x7fb8540a0280 auth_method 0 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb854131b20 0x7fb8541338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): get_auth_request con 0x7fb854131b20 auth_method 0 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:40.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8588f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient(hunting): handle_auth_done global_id 74174 payload 824 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 -- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb854131b20 msgr2=0x7fb8541338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb854131b20 0x7fb8541338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 -- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 msgr2=0x7fb854133e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 0x7fb854133e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 monclient: found mon.noname-c 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.744+0000 7fb8590f9640 1 -- 172.21.15.17:0/791207156 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c009ae0 con 0x7fb8540a0280 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8590f9640 1 -- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 msgr2=0x7fb854138c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:40.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8590f9640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c640 tx=0x7fb850007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850012840 con 0x7fb8540a0280 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: dump: 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:40.759 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8500129c0 con 0x7fb8540a0280 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 5 monclient: authenticate success, global_id 74174 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850012d80 con 0x7fb8540a0280 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8457fa640 10 monclient: dump: 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:40.760 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 10 monclient: _renew_subs 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb854135940 con 0x7fb8540a0280 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb8477fe640 4 set_mon_vals no callback set 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 10 monclient: _renew_subs 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.747+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb854136f60 con 0x7fb8540a0280 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb850011590 con 0x7fb8540a0280 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb85b35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb85b35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb85b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8540724a0 con 0x7fb8540a0280 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb8457fa640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 0x7fb83803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7fb85004c170 con 0x7fb8540a0280 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb8588f8640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 0x7fb83803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:40.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.748+0000 7fb8588f8640 10 monclient: get_auth_request con 0x7fb8380386f0 auth_method 0 2024-09-15T01:55:40.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.751+0000 7fb8588f8640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 0x7fb83803aba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb8540a2eb0 tx=0x7fb848008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:40.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.753+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb8500108e0 con 0x7fb8540a0280 2024-09-15T01:55:40.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.753+0000 7fb8457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:40.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.753+0000 7fb8457fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:40.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:40.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:40.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:40.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb85b35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb85b35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb85b35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8541372d0 con 0x7fb8540a0280 2024-09-15T01:55:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb8457fa640 1 -- 172.21.15.17:0/791207156 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fb850016020 con 0x7fb8540a0280 2024-09-15T01:55:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb8457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.978+0000 7fb8457fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:40.981 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 msgr2=0x7fb83803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 0x7fb83803aba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb8540a2eb0 tx=0x7fb848008040 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 10 monclient: shutdown 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 msgr2=0x7fb854138c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c640 tx=0x7fb850007ed0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 shutdown_connections 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb8380386f0 0x7fb83803aba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb854132430 0x7fb854133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb854131b20 0x7fb8541338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 --2- 172.21.15.17:0/791207156 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb8540a0280 0x7fb854138c70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 >> 172.21.15.17:0/791207156 conn(0x7fb854071270 msgr2=0x7fb8540a4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 shutdown_connections 2024-09-15T01:55:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:40.980+0000 7fb85b35d640 1 -- 172.21.15.17:0/791207156 wait complete. 2024-09-15T01:55:41.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:41.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:41.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:41.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:42.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:42.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:42.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:42.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:43.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:43.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:43.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:43.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:43.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:44.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.099+0000 7f8e4f59d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:44.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.099+0000 7f8e4f59d640 10 monclient: build_initial_monmap 2024-09-15T01:55:44.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.099+0000 7f8e4f59d640 10 monclient: monmap: 2024-09-15T01:55:44.102 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:44.102 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:44.101170+0000 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:44.101170+0000 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:44.103 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:44.104 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.099+0000 7f8e4f59d640 1 Processor -- start 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 1 -- start start 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient: init 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e4809c620 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4806f450 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e481347d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4809bbd0 con 0x7f8e481347d0 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4809b570 con 0x7f8e4809c620 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4809f750 con 0x7f8e4806f450 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4f59d640 10 monclient(hunting): authenticate will time out at 1004.097861s 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4db3a640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f8e481347d0 legacy=0x7f8e48136bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.100+0000 7f8e4cb38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:51946/0 (socket says 172.21.15.17:51946) 2024-09-15T01:55:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 -- 172.21.15.17:0/2953714030 learned_addr learned my addr 172.21.15.17:0/2953714030 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): get_auth_request con 0x7f8e4806f450 auth_method 0 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_done global_id 74177 payload 274 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 -- 172.21.15.17:0/2953714030 >> v1:172.21.15.67:6789/0 conn(0x7f8e4809c620 legacy=0x7f8e4806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 -- 172.21.15.17:0/2953714030 >> v1:172.21.15.17:6789/0 conn(0x7f8e481347d0 legacy=0x7f8e48136bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient: _finish_hunting 0 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 monclient: found mon.noname-b 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.101+0000 7f8e4cb38640 1 -- 172.21.15.17:0/2953714030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4809a680 con 0x7f8e4806f450 2024-09-15T01:55:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4cb38640 1 -- 172.21.15.17:0/2953714030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 msgr2=0x7f8e4809c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4cb38640 1 --2- 172.21.15.17:0/2953714030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009940 tx=0x7f8e380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6a56bcdf2121b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 1 -- 172.21.15.17:0/2953714030 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38032440 con 0x7f8e4806f450 2024-09-15T01:55:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: dump: 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4f59d640 5 monclient: authenticate success, global_id 74177 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 1 -- 172.21.15.17:0/2953714030 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e380325a0 con 0x7f8e4806f450 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4f59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4f59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e4f59d640 4 set_mon_vals no callback set 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 1 -- 172.21.15.17:0/2953714030 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38032940 con 0x7f8e4806f450 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.104+0000 7f8e3ffff640 10 monclient: dump: 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:44.118 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 10 monclient: shutdown 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 -- 172.21.15.17:0/2953714030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 msgr2=0x7f8e4809c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/2953714030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009940 tx=0x7f8e380091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 -- 172.21.15.17:0/2953714030 shutdown_connections 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/2953714030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4806f450 0x7f8e4809c9f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 -- 172.21.15.17:0/2953714030 >> 172.21.15.17:0/2953714030 conn(0x7f8e48071300 msgr2=0x7f8e480a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 -- 172.21.15.17:0/2953714030 shutdown_connections 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 1 -- 172.21.15.17:0/2953714030 wait complete. 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 10 monclient: build_initial_monmap 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.105+0000 7f8e4f59d640 10 monclient: monmap: 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 Processor -- start 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 -- start start 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient: init 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e4806f450 0x7f8e48136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4806f450 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4809c620 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 0x7f8e4806c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e481347d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e481300b0 con 0x7f8e481347d0 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806f1c0 con 0x7f8e4806f450 2024-09-15T01:55:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806e480 con 0x7f8e4809c620 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4f59d640 10 monclient(hunting): authenticate will time out at 1004.103698s 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4db3a640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 msgr2=0x7f8e4806c040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4db3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 0x7f8e4806c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:51950/0 (socket says 172.21.15.17:51950) 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 1 -- 172.21.15.17:0/1619389123 learned_addr learned my addr 172.21.15.17:0/1619389123 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): get_auth_request con 0x7f8e4809c620 auth_method 0 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.106+0000 7f8e4d339640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e4806f450 0x7f8e48136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): get_auth_request con 0x7f8e4806f450 auth_method 0 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 10 monclient(hunting): handle_auth_done global_id 74180 payload 824 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 -- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e4806f450 msgr2=0x7f8e48136c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e4806f450 0x7f8e48136c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 -- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 msgr2=0x7f8e4806c040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 0x7f8e4806c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 10 monclient: _finish_hunting 0 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 monclient: found mon.noname-c 2024-09-15T01:55:44.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.107+0000 7f8e4cb38640 1 -- 172.21.15.17:0/1619389123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38009450 con 0x7f8e4809c620 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4cb38640 1 -- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 msgr2=0x7f8e4806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4cb38640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e3802a700 tx=0x7f8e38032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4cb38640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e3802a700 tx=0x7f8e38032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8e38031990 mon_map magic: 0 v1 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38031990 con 0x7f8e4809c620 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 10 monclient: dump: 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.125 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38031b10 con 0x7f8e4809c620 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 5 monclient: authenticate success, global_id 74180 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 10 monclient: _renew_subs 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4806d350 con 0x7f8e4809c620 2024-09-15T01:55:44.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:44.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e3f7fe640 4 set_mon_vals no callback set 2024-09-15T01:55:44.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 10 monclient: _renew_subs 2024-09-15T01:55:44.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.109+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e481377a0 con 0x7f8e4809c620 2024-09-15T01:55:44.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3803c410 con 0x7f8e4809c620 2024-09-15T01:55:44.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:44.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e4f59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:44.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:44.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 10 monclient: dump: 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8e38039050 con 0x7f8e4809c620 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e4f59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:44.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e4f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e480724a0 con 0x7f8e4809c620 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.110+0000 7f8e3d7fa640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 0x7f8e2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.111+0000 7f8e4d339640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 0x7f8e2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.111+0000 7f8e4d339640 10 monclient: get_auth_request con 0x7f8e2c038740 auth_method 0 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.114+0000 7f8e4d339640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 0x7f8e2c03abf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8e44007fd0 tx=0x7f8e44007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:44.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.115+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f8e3806c070 con 0x7f8e4809c620 2024-09-15T01:55:44.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.732+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8e38039ae0 con 0x7f8e4809c620 2024-09-15T01:55:44.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.732+0000 7f8e3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:44.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.732+0000 7f8e3d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:44.792+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:44.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:44.792+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:44.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e4f59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:44.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e4f59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e4f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e480724a0 con 0x7f8e4809c620 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e3d7fa640 1 -- 172.21.15.17:0/1619389123 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f8e38037020 con 0x7f8e4809c620 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.966+0000 7f8e3d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:44.970 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 msgr2=0x7f8e2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 0x7f8e2c03abf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8e44007fd0 tx=0x7f8e44007e10 comp rx=0 tx=0).stop 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 10 monclient: shutdown 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 msgr2=0x7f8e4806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.968+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e3802a700 tx=0x7f8e38032000 comp rx=0 tx=0).stop 2024-09-15T01:55:44.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 shutdown_connections 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8e2c038740 0x7f8e2c03abf0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e481347d0 0x7f8e4806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8e4809c620 0x7f8e4806bb00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 --2- 172.21.15.17:0/1619389123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e4806f450 0x7f8e48136c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 >> 172.21.15.17:0/1619389123 conn(0x7f8e48071300 msgr2=0x7f8e4812eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 shutdown_connections 2024-09-15T01:55:44.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:44.969+0000 7f8e4f59d640 1 -- 172.21.15.17:0/1619389123 wait complete. 2024-09-15T01:55:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:45.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:45.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:46.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:46.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:46.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:46.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:47.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:47.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:47.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:47.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:47.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:55:48.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.088+0000 7ff44195d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.088+0000 7ff44195d640 10 monclient: build_initial_monmap 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.088+0000 7ff44195d640 10 monclient: monmap: 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:48.090576+0000 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:48.090576+0000 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:48.092 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:48.093 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:48.093 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:48.093 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:48.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 1 Processor -- start 2024-09-15T01:55:48.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 1 -- start start 2024-09-15T01:55:48.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 10 monclient: init 2024-09-15T01:55:48.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff43c0708a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.089+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff43c097870 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-a con 0x7ff43c136120 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c06afe0 con 0x7ff43c0708a0 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff43affd640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7ff43c0708a0 legacy=0x7ff43c06a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c06d7d0 con 0x7ff43c097870 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c096b40 con 0x7ff43c136120 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff44195d640 10 monclient(hunting): authenticate will time out at 1008.087088s 2024-09-15T01:55:48.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff43b7fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 msgr2=0x7ff43c138500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:48.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff43b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:48.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff43a7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff43c097870 0x7ff43c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:38898/0 (socket says 172.21.15.17:38898) 2024-09-15T01:55:48.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.090+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 learned_addr learned my addr 172.21.15.17:0/1421819724 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:48.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.092+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.67:6789/0 conn(0x7ff43c097870 legacy=0x7ff43c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:48.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.291+0000 7ff43b7fe640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 msgr2=0x7ff43c138500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:48.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.291+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.17:6789/0 conn(0x7ff43c0708a0 legacy=0x7ff43c06a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:48.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.291+0000 7ff43b7fe640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:48.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.692+0000 7ff43b7fe640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 msgr2=0x7ff43c138500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:48.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.692+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.17:6789/0 conn(0x7ff43c0708a0 legacy=0x7ff43c06a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:48.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:48.692+0000 7ff43b7fe640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:48.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:48.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:48.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:48.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:49.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:49.493+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.17:6789/0 conn(0x7ff43c0708a0 legacy=0x7ff43c06a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:49.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:49.493+0000 7ff43b7fe640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 msgr2=0x7ff43c138500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:49.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:49.493+0000 7ff43b7fe640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:55:49.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:49.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:49.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:49.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:49.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:49.794+0000 7fa2777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:50.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:50.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:50.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:50.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:51.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): tick 2024-09-15T01:55:51.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 monclient(hunting): continuing hunt 2024-09-15T01:55:51.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:55:51.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 msgr2=0x7ff43c138500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:51.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.17:3300/0 conn(0x7ff43c136120 0x7ff43c138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.67:6789/0 conn(0x7ff43c097870 legacy=0x7ff43c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 -- 172.21.15.17:0/1421819724 >> v1:172.21.15.17:6789/0 conn(0x7ff43c0708a0 legacy=0x7ff43c06a1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient: _add_conns ranks=[4,2] 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff41c003700 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff41c006200 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 -- 172.21.15.17:0/1421819724 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c096b40 con 0x7ff41c003700 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 1 -- 172.21.15.17:0/1421819724 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c00d1f0 con 0x7ff41c006200 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff438ff9640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request con 0x7ff41c006200 auth_method 0 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 10 monclient(hunting): get_auth_request con 0x7ff41c003700 auth_method 0 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:51.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:51.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.090+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request con 0x7ff41c006200 auth_method 0 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 10 monclient(hunting): get_auth_request con 0x7ff41c003700 auth_method 0 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.291+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:55:51.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request con 0x7ff41c006200 auth_method 0 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 10 monclient(hunting): get_auth_request con 0x7ff41c003700 auth_method 0 2024-09-15T01:55:51.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:51.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:51.692+0000 7ff43affd640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:55:51.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:51.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:51.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:51.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request con 0x7ff41c006200 auth_method 0 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient(hunting): handle_auth_done global_id 74183 payload 274 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 msgr2=0x7ff41c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> v2:172.21.15.67:3300/0 conn(0x7ff41c003700 0x7ff41c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 1 monclient: found mon.noname-b 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.494+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c097c40 con 0x7ff41c006200 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.496+0000 7ff43a7fc640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.496+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff424007c10 tx=0x7ff424009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd0306066f02019 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 1 -- 172.21.15.17:0/1421819724 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240313f0 con 0x7ff41c006200 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: dump: 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.500 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 1 -- 172.21.15.17:0/1421819724 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff424031550 con 0x7ff41c006200 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 1 -- 172.21.15.17:0/1421819724 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240318f0 con 0x7ff41c006200 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff44195d640 5 monclient: authenticate success, global_id 74183 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff44195d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff44195d640 4 set_mon_vals no callback set 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.497+0000 7ff439ffb640 10 monclient: dump: 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.502 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.503 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.503 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.503 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 10 monclient: shutdown 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 -- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff41c0086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 --2- 172.21.15.17:0/1421819724 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff41c0086b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff424007c10 tx=0x7ff424009040 comp rx=0 tx=0).stop 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff43b7fe640 1 -- 172.21.15.17:0/1421819724 reap_dead start 2024-09-15T01:55:52.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 -- 172.21.15.17:0/1421819724 shutdown_connections 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 -- 172.21.15.17:0/1421819724 >> 172.21.15.17:0/1421819724 conn(0x7ff43c067ae0 msgr2=0x7ff43c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 -- 172.21.15.17:0/1421819724 shutdown_connections 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.498+0000 7ff44195d640 1 -- 172.21.15.17:0/1421819724 wait complete. 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.499+0000 7ff44195d640 10 monclient: build_initial_monmap 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.499+0000 7ff44195d640 10 monclient: monmap: 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:52.509 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.499+0000 7ff44195d640 1 Processor -- start 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.499+0000 7ff44195d640 1 -- start start 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.499+0000 7ff44195d640 10 monclient: init 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-c con 0x7ff41c006200 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 0x7ff43c09ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-a con 0x7ff43c13cef0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c09cf40 0x7ff43c097e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): picked mon.noname-b con 0x7ff43c09cf40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c09b1f0 con 0x7ff43c13cef0 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c06cb60 con 0x7ff43c09cf40 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c09ac30 con 0x7ff41c006200 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff44195d640 10 monclient(hunting): authenticate will time out at 1012.497356s 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43a7fc640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 msgr2=0x7ff43c09ca00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43a7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 0x7ff43c09ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:51964/0 (socket says 172.21.15.17:51964) 2024-09-15T01:55:52.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 -- 172.21.15.17:0/215220810 learned_addr learned my addr 172.21.15.17:0/215220810 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): get_auth_request con 0x7ff41c006200 auth_method 0 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43b7fe640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c09cf40 0x7ff43c097e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43b7fe640 10 monclient(hunting): get_auth_request con 0x7ff43c09cf40 auth_method 0 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient(hunting): handle_auth_done global_id 74186 payload 824 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 -- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c09cf40 msgr2=0x7ff43c097e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c09cf40 0x7ff43c097e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 -- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 msgr2=0x7ff43c09ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 0x7ff43c09ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 monclient: found mon.noname-c 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.500+0000 7ff43affd640 1 -- 172.21.15.17:0/215220810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42400f040 con 0x7ff41c006200 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff43affd640 1 -- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff43c135820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff43affd640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff42c007e20 tx=0x7ff42c006f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c011e50 con 0x7ff41c006200 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:52.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: dump: 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c00f070 con 0x7ff41c006200 2024-09-15T01:55:52.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 5 monclient: authenticate success, global_id 74186 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 10 monclient: _renew_subs 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c09af10 con 0x7ff41c006200 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c015810 con 0x7ff41c006200 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff432ffd640 10 monclient: dump: 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff4397fa640 4 set_mon_vals no callback set 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 10 monclient: _renew_subs 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.503+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c099d40 con 0x7ff41c006200 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff42c015c40 con 0x7ff41c006200 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff44195d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff44195d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff44195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c0724a0 con 0x7ff41c006200 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff432ffd640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 0x7ff41c03ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.504+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7ff42c04cb70 con 0x7ff41c006200 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.505+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 0x7ff41c03ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.505+0000 7ff43a7fc640 10 monclient: get_auth_request con 0x7ff41c03da90 auth_method 0 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.507+0000 7ff43a7fc640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 0x7ff41c03ff40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff424033000 tx=0x7ff424039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.509+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff42c01d3b0 con 0x7ff41c006200 2024-09-15T01:55:52.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.509+0000 7ff432ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.509+0000 7ff432ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:52.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff44195d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:55:52.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff44195d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:52.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff44195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff43c0711e0 con 0x7ff41c006200 2024-09-15T01:55:52.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff432ffd640 1 -- 172.21.15.17:0/215220810 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7ff42c019070 con 0x7ff41c006200 2024-09-15T01:55:52.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff432ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:55:52.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.743+0000 7ff432ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:52.747 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 msgr2=0x7ff41c03ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 0x7ff41c03ff40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff424033000 tx=0x7ff424039000 comp rx=0 tx=0).stop 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 10 monclient: shutdown 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 msgr2=0x7ff43c135820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff42c007e20 tx=0x7ff42c006f90 comp rx=0 tx=0).stop 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 shutdown_connections 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff41c03da90 0x7ff41c03ff40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff43c13cef0 0x7ff43c09ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c09cf40 0x7ff43c097e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 --2- 172.21.15.17:0/215220810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff41c006200 0x7ff43c135820 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 >> 172.21.15.17:0/215220810 conn(0x7ff43c067ae0 msgr2=0x7ff43c06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 shutdown_connections 2024-09-15T01:55:52.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.745+0000 7ff44195d640 1 -- 172.21.15.17:0/215220810 wait complete. 2024-09-15T01:55:52.759 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:55:52.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T01:55:52.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:55:52.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.794+0000 7fa2777fe640 10 monclient: tick 2024-09-15T01:55:52.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.794+0000 7fa2777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.796+0000 7fa288ff9640 1 -- 172.21.15.67:0/3307861626 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v106) v1 ==== 58+0+3 (secure 0 0 0) 0x7fa278005970 con 0x7fa28c130e00 2024-09-15T01:55:52.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.796+0000 7fa288ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T01:55:52.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.796+0000 7fa288ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:52.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 msgr2=0x7fa27003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 0x7fa27003abf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa28c068d50 tx=0x7fa280032000 comp rx=0 tx=0).stop 2024-09-15T01:55:52.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 10 monclient: shutdown 2024-09-15T01:55:52.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 msgr2=0x7fa28c1413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa27800ea80 tx=0x7fa2780090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 shutdown_connections 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa270038740 0x7fa27003abf0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa28c1311d0 0x7fa28c1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa28c130e00 0x7fa28c1413b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 --2- 172.21.15.67:0/3307861626 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa28c068dc0 0x7fa28c12ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 >> 172.21.15.67:0/3307861626 conn(0x7fa28c0710f0 msgr2=0x7fa28c0a4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.799+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 shutdown_connections 2024-09-15T01:55:52.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.800+0000 7fa292b1d640 1 -- 172.21.15.67:0/3307861626 wait complete. 2024-09-15T01:55:52.812 INFO:tasks.workunit.client.0.smithi067.stderr:+ o4=12 2024-09-15T01:55:52.812 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 2024-09-15T01:55:52.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.854+0000 7f21d0d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:52.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.854+0000 7f21d0d9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.854+0000 7f21d0d9d640 10 monclient: monmap: 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:52.856561+0000 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:52.856561+0000 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:52.858 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 1 Processor -- start 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 1 -- start start 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 10 monclient: init 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 0x7f21cc09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f21cc09c1b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21cc09c580 0x7f21cc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.855+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f21cc09c580 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21cc09c9c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc09f790 con 0x7f21cc09c9c0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc06f780 con 0x7f21cc09c1b0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc069d40 con 0x7f21cc09c580 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21d0d9d640 10 monclient(hunting): authenticate will time out at 1012.853052s 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21ca59c640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 msgr2=0x7f21cc09a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:52.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21ca59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 0x7f21cc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21cc09c580 0x7f21cc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21cad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21cc09c9c0 0x7f21cc06f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:33938/0 (socket says 172.21.15.17:33938) 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21cad9d640 1 -- 172.21.15.17:0/1660624457 learned_addr learned my addr 172.21.15.17:0/1660624457 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): get_auth_request con 0x7f21cc09c580 auth_method 0 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_done global_id 5575 payload 274 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 -- 172.21.15.17:0/1660624457 >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 msgr2=0x7f21cc09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:52.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 --2- 172.21.15.17:0/1660624457 >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 0x7f21cc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 -- 172.21.15.17:0/1660624457 >> v1:172.21.15.67:6789/0 conn(0x7f21cc09c9c0 legacy=0x7f21cc06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 monclient: found mon.noname-c 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.856+0000 7f21c9d9b640 1 -- 172.21.15.17:0/1660624457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc068b60 con 0x7f21cc09c580 2024-09-15T01:55:52.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.859+0000 7f21c9d9b640 1 -- 172.21.15.17:0/1660624457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c580 msgr2=0x7f21cc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.859+0000 7f21c9d9b640 1 --2- 172.21.15.17:0/1660624457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c580 0x7f21cc06eb30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f21cc060cf0 tx=0x7f21c00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed8da76a27bd5c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 1 -- 172.21.15.17:0/1660624457 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0032440 con 0x7f21cc09c580 2024-09-15T01:55:52.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: dump: 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 1 -- 172.21.15.17:0/1660624457 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c00325a0 con 0x7f21cc09c580 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 1 -- 172.21.15.17:0/1660624457 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0032940 con 0x7f21cc09c580 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 5 monclient: authenticate success, global_id 5575 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 4 set_mon_vals no callback set 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21c959a640 10 monclient: dump: 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.871 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 10 monclient: shutdown 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 -- 172.21.15.17:0/1660624457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c580 msgr2=0x7f21cc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/1660624457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c580 0x7f21cc06eb30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f21cc060cf0 tx=0x7f21c00090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 -- 172.21.15.17:0/1660624457 shutdown_connections 2024-09-15T01:55:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/1660624457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c580 0x7f21cc06eb30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/1660624457 >> v2:172.21.15.17:3300/0 conn(0x7f21cc09c1b0 0x7f21cc09a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 -- 172.21.15.17:0/1660624457 >> 172.21.15.17:0/1660624457 conn(0x7f21cc071300 msgr2=0x7f21cc0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 -- 172.21.15.17:0/1660624457 shutdown_connections 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.860+0000 7f21d0d9d640 1 -- 172.21.15.17:0/1660624457 wait complete. 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient: monmap: 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 Processor -- start 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 -- start start 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient: init 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c1b0 0x7f21cc13df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f21cc09c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 0x7f21cc13d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f21cc09c580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21cc09c9c0 0x7f21cc1414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f21cc09c9c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc122f70 con 0x7f21cc09c9c0 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1229b0 con 0x7f21cc09c1b0 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc120490 con 0x7f21cc09c580 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21d0d9d640 10 monclient(hunting): authenticate will time out at 1012.858945s 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.861+0000 7f21cad9d640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21cc09c9c0 msgr2=0x7f21cc1414f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21cad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21cc09c9c0 0x7f21cc1414f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 0x7f21cc13d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 0x7f21cc13d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:43504/0 (socket says 172.21.15.17:43504) 2024-09-15T01:55:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 -- 172.21.15.17:0/688019151 learned_addr learned my addr 172.21.15.17:0/688019151 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): get_auth_request con 0x7f21cc09c580 auth_method 0 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c1b0 0x7f21cc13df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): get_auth_request con 0x7f21cc09c1b0 auth_method 0 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient(hunting): handle_auth_done global_id 74189 payload 824 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c1b0 msgr2=0x7f21cc13df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21cc09c1b0 0x7f21cc13df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21cc09c9c0 msgr2=0x7f21cc1414f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21cc09c9c0 0x7f21cc1414f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 monclient: found mon.noname-c 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.862+0000 7f21c9d9b640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0009390 con 0x7f21cc09c580 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.864+0000 7f21c9d9b640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 msgr2=0x7f21cc13d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.864+0000 7f21c9d9b640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 0x7f21cc13d630 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21c002a650 tx=0x7f21c0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0031940 con 0x7f21cc09c580 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: dump: 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.878 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 5 monclient: authenticate success, global_id 74189 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 monclient: _renew_subs 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c0031aa0 con 0x7f21cc09c580 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc1226d0 con 0x7f21cc09c580 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0031e40 con 0x7f21cc09c580 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ba7fc640 10 monclient: dump: 2024-09-15T01:55:52.879 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21c8d99640 4 set_mon_vals no callback set 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 monclient: _renew_subs 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc122d40 con 0x7f21cc09c580 2024-09-15T01:55:52.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 0x7f21cc141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc063bb0 con 0x7f21cc121800 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ca59c640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 0x7f21cc141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ca59c640 10 monclient: get_auth_request con 0x7f21cc121800 auth_method 0 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.865+0000 7f21ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f21c0039070 con 0x7f21cc09c580 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ca59c640 10 monclient(hunting): handle_auth_done global_id 74192 payload 824 2024-09-15T01:55:52.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ca59c640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21cc1230e0 con 0x7f21cc121800 2024-09-15T01:55:52.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ba7fc640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f21ac0386f0 0x7f21ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6123+0+0 (secure 0 0 0) 0x7f21c006c380 con 0x7f21cc09c580 2024-09-15T01:55:52.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21cad9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f21ac0386f0 0x7f21ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.866+0000 7f21cad9d640 10 monclient: get_auth_request con 0x7f21ac0386f0 auth_method 0 2024-09-15T01:55:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ca59c640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 msgr2=0x7f21cc141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ca59c640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 0x7f21cc141a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0128f0 con 0x7f21cc121800 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 10 monclient: dump: 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f21bc012a50 con 0x7f21cc121800 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 msgr2=0x7f21cc141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 0x7f21cc141a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 msgr2=0x7f21cc141a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21ba7fc640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc121800 0x7f21cc141a30 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.868+0000 7f21cad9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f21ac0386f0 0x7f21ac03aba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f21cc13e500 tx=0x7f21c4005080 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21d0d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 0x7f21cc141f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21d0d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0724a0 con 0x7f21cc13e8e0 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 0x7f21cc141f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient: get_auth_request con 0x7f21cc13e8e0 auth_method 0 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 10 monclient(hunting): handle_auth_done global_id 74195 payload 824 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.874+0000 7f21ca59c640 1 -- 172.21.15.17:0/688019151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21cc0a1af0 con 0x7f21cc13e8e0 2024-09-15T01:55:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ca59c640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 msgr2=0x7f21cc141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ca59c640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 0x7f21cc141f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc025070 con 0x7f21cc13e8e0 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 10 monclient: dump: 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4554 (secure 0 0 0) 0x7f21bc01b660 con 0x7f21cc13e8e0 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 msgr2=0x7f21cc141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 0x7f21cc141f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc015000 comp rx=0 tx=0).stop 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 msgr2=0x7f21cc141f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.877+0000 7f21ba7fc640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc13e8e0 0x7f21cc141f70 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003940 tx=0x7f21bc015000 comp rx=0 tx=0).stop 2024-09-15T01:55:52.889 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 54, 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:52.890 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:52.891 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:52.892 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:52.893 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:52.894 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:52.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:52.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:52.898 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:52.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:52.901 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:52.902 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f21ac0386f0 msgr2=0x7f21ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f21ac0386f0 0x7f21ac03aba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f21cc13e500 tx=0x7f21c4005080 comp rx=0 tx=0).stop 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 10 monclient: shutdown 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 msgr2=0x7f21cc13d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 --2- 172.21.15.17:0/688019151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f21cc09c580 0x7f21cc13d630 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21c002a650 tx=0x7f21c0032000 comp rx=0 tx=0).stop 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21cad9d640 1 -- 172.21.15.17:0/688019151 reap_dead start 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21cad9d640 1 -- 172.21.15.17:0/688019151 reap_dead start 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 shutdown_connections 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 >> 172.21.15.17:0/688019151 conn(0x7f21cc071300 msgr2=0x7f21cc06ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 shutdown_connections 2024-09-15T01:55:52.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:52.878+0000 7f21d0d9d640 1 -- 172.21.15.17:0/688019151 wait complete. 2024-09-15T01:55:52.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:55:52.906 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:55:52.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:55:52.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.891+0000 7f80af91d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:52.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.891+0000 7f80af91d640 10 monclient: build_initial_monmap 2024-09-15T01:55:52.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.891+0000 7f80af91d640 10 monclient: monmap: 2024-09-15T01:55:52.914 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:52.914 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:52.891829+0000 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:52.891829+0000 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.892+0000 7f80af91d640 1 Processor -- start 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.892+0000 7f80af91d640 1 -- start start 2024-09-15T01:55:52.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.892+0000 7f80af91d640 10 monclient: init 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80a8097300 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80a8097d30 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80a8133f30 0x7f80a8136310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-c con 0x7f80a8133f30 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a8098d80 con 0x7f80a8097d30 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a8097af0 con 0x7f80a8097300 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80976d0 con 0x7f80a8133f30 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80af91d640 10 monclient(hunting): authenticate will time out at 985.361192s 2024-09-15T01:55:52.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80ad6b9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80a8097300 0x7f80a8099130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39290/0 (socket says 172.21.15.67:39290) 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80ad6b9640 1 -- 172.21.15.67:0/2471185710 learned_addr learned my addr 172.21.15.67:0/2471185710 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80aceb8640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f80a8097d30 legacy=0x7f80a8132810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f809ffff640 1 -- 172.21.15.67:0/2471185710 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 1 --2- 172.21.15.67:0/2471185710 >> v2:172.21.15.67:3300/0 conn(0x7f80a8133f30 0x7f80a8136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): get_auth_request con 0x7f80a8133f30 auth_method 0 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient(hunting): handle_auth_done global_id 5581 payload 274 2024-09-15T01:55:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 1 -- 172.21.15.67:0/2471185710 >> v1:172.21.15.67:6789/0 conn(0x7f80a8097300 legacy=0x7f80a8099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 1 -- 172.21.15.67:0/2471185710 >> v1:172.21.15.17:6789/0 conn(0x7f80a8097d30 legacy=0x7f80a8132810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 1 monclient: found mon.noname-c 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.893+0000 7f80adeba640 1 -- 172.21.15.67:0/2471185710 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a806ed60 con 0x7f80a8133f30 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80adeba640 1 -- 172.21.15.67:0/2471185710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8133f30 msgr2=0x7f80a8136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80adeba640 1 --2- 172.21.15.67:0/2471185710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8133f30 0x7f80a8136310 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009e90 tx=0x7f80a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e06fb7780b772c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 1 -- 172.21.15.67:0/2471185710 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0007840 con 0x7f80a8133f30 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: dump: 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 1 -- 172.21.15.67:0/2471185710 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a002d050 con 0x7f80a8133f30 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80af91d640 5 monclient: authenticate success, global_id 5581 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80af91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80af91d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 1 -- 172.21.15.67:0/2471185710 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0007bd0 con 0x7f80a8133f30 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f80af91d640 4 set_mon_vals no callback set 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.896+0000 7f809ffff640 10 monclient: dump: 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.920 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 10 monclient: shutdown 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 -- 172.21.15.67:0/2471185710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8133f30 msgr2=0x7f80a8136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 --2- 172.21.15.67:0/2471185710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8133f30 0x7f80a8136310 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009e90 tx=0x7f80a0009780 comp rx=0 tx=0).stop 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 -- 172.21.15.67:0/2471185710 shutdown_connections 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 --2- 172.21.15.67:0/2471185710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8133f30 0x7f80a8136310 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 -- 172.21.15.67:0/2471185710 >> 172.21.15.67:0/2471185710 conn(0x7f80a80a1020 msgr2=0x7f80a80a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 -- 172.21.15.67:0/2471185710 shutdown_connections 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 1 -- 172.21.15.67:0/2471185710 wait complete. 2024-09-15T01:55:52.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 10 monclient: build_initial_monmap 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.897+0000 7f80af91d640 10 monclient: monmap: 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 1 Processor -- start 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 1 -- start start 2024-09-15T01:55:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 10 monclient: init 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-b con 0x7f80a8097300 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.898+0000 7f80af91d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 0x7f80a811aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-a con 0x7f80a8097d30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80a8133f30 0x7f80a811af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): picked mon.noname-c con 0x7f80a8133f30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a811de10 con 0x7f80a8097d30 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a811cc70 con 0x7f80a8097300 2024-09-15T01:55:52.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a809cd10 con 0x7f80a8133f30 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80af91d640 10 monclient(hunting): authenticate will time out at 985.367072s 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58388/0 (socket says 172.21.15.67:58388) 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 -- 172.21.15.67:0/1161891910 learned_addr learned my addr 172.21.15.67:0/1161891910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): get_auth_request con 0x7f80a8097300 auth_method 0 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80aceb8640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 msgr2=0x7f80a811aa50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80aceb8640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 0x7f80a811aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:52.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80a8133f30 0x7f80a811af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): get_auth_request con 0x7f80a8133f30 auth_method 0 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80adeba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient(hunting): handle_auth_done global_id 5584 payload 824 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80a8133f30 msgr2=0x7f80a811af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:52.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80a8133f30 0x7f80a811af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 msgr2=0x7f80a811aa50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 0x7f80a811aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient: _finish_hunting 0 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 monclient: found mon.noname-b 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.899+0000 7f80ad6b9640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0009a20 con 0x7f80a8097300 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80ad6b9640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 msgr2=0x7f80a811a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80ad6b9640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c560 tx=0x7f80a4004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4005cb0 con 0x7f80a8097300 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:52.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: dump: 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:52.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 5 monclient: authenticate success, global_id 5584 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4005e30 con 0x7f80a8097300 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 10 monclient: _renew_subs 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a809c1d0 con 0x7f80a8097300 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4007320 con 0x7f80a8097300 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809d7fa640 10 monclient: dump: 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.928 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f809f7fe640 4 set_mon_vals no callback set 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 10 monclient: _renew_subs 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.901+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a8133bb0 con 0x7f80a8097300 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.902+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f80a401b3b0 con 0x7f80a8097300 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.902+0000 7f80af91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.902+0000 7f80af91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.902+0000 7f80af91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.902+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a804bf60 con 0x7f80a8097300 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.903+0000 7f809d7fa640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 0x7f808c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.903+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6123+0+0 (secure 0 0 0) 0x7f80a404b5b0 con 0x7f80a8097300 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.903+0000 7f80aceb8640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 0x7f808c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.903+0000 7f80aceb8640 10 monclient: get_auth_request con 0x7f808c038740 auth_method 0 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.905+0000 7f80aceb8640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 0x7f808c03abf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80980038f0 tx=0x7f8098006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.907+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f80a4007710 con 0x7f80a8097300 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.907+0000 7f809d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:52.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:52.907+0000 7f809d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.002+0000 7f3f80aa8640 10 monclient: get_monmap_and_config 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.002+0000 7f3f80aa8640 10 monclient: build_initial_monmap 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.002+0000 7f3f80aa8640 10 monclient: monmap: 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:55:53.004674+0000 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:55:53.004674+0000 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:53.007 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.003+0000 7f3f80aa8640 1 Processor -- start 2024-09-15T01:55:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.003+0000 7f3f80aa8640 1 -- start start 2024-09-15T01:55:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.003+0000 7f3f80aa8640 10 monclient: init 2024-09-15T01:55:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 0x7f3f7c09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c09c1b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f7c09c580 0x7f3f7c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c09c580 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f7c09c9c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c09f790 con 0x7f3f7c09c9c0 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c06f780 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c069d40 con 0x7f3f7c09c580 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f80aa8640 10 monclient(hunting): authenticate will time out at 1013.001192s 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f7a59c640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 msgr2=0x7f3f7c09a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f7a59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 0x7f3f7c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f7c09c580 0x7f3f7c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f7ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3f7c09c9c0 0x7f3f7c06f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:33952/0 (socket says 172.21.15.17:33952) 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f7ad9d640 1 -- 172.21.15.17:0/3067234574 learned_addr learned my addr 172.21.15.17:0/3067234574 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): get_auth_request con 0x7f3f7c09c580 auth_method 0 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_done global_id 5587 payload 274 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 -- 172.21.15.17:0/3067234574 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 msgr2=0x7f3f7c09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/3067234574 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 0x7f3f7c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 -- 172.21.15.17:0/3067234574 >> v1:172.21.15.67:6789/0 conn(0x7f3f7c09c9c0 legacy=0x7f3f7c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 monclient: found mon.noname-c 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.004+0000 7f3f79d9b640 1 -- 172.21.15.17:0/3067234574 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c068b60 con 0x7f3f7c09c580 2024-09-15T01:55:53.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f79d9b640 1 -- 172.21.15.17:0/3067234574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c580 msgr2=0x7f3f7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:53.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/3067234574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c580 0x7f3f7c06eb30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c060cf0 tx=0x7f3f700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab733d7587d9608 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 1 -- 172.21.15.17:0/3067234574 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70032440 con 0x7f3f7c09c580 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: dump: 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: _finish_auth 0 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 1 -- 172.21.15.17:0/3067234574 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f700325a0 con 0x7f3f7c09c580 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 1 -- 172.21.15.17:0/3067234574 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70032940 con 0x7f3f7c09c580 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f7959a640 10 monclient: dump: 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f80aa8640 5 monclient: authenticate success, global_id 5587 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f80aa8640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.007+0000 7f3f80aa8640 4 set_mon_vals no callback set 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 10 monclient: shutdown 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 -- 172.21.15.17:0/3067234574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c580 msgr2=0x7f3f7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/3067234574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c580 0x7f3f7c06eb30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c060cf0 tx=0x7f3f700090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 -- 172.21.15.17:0/3067234574 shutdown_connections 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/3067234574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c580 0x7f3f7c06eb30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/3067234574 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c09c1b0 0x7f3f7c09a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 -- 172.21.15.17:0/3067234574 >> 172.21.15.17:0/3067234574 conn(0x7f3f7c071300 msgr2=0x7f3f7c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 -- 172.21.15.17:0/3067234574 shutdown_connections 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 1 -- 172.21.15.17:0/3067234574 wait complete. 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 10 monclient: build_initial_monmap 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.008+0000 7f3f80aa8640 10 monclient: monmap: 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:55:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 Processor -- start 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 -- start start 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient: init 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 0x7f3f7c13df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c09c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09c580 0x7f3f7c13d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c09c580 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c9c0 0x7f3f7c1414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c09c9c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c122f70 con 0x7f3f7c09c580 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c1229b0 con 0x7f3f7c09c9c0 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c120490 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f80aa8640 10 monclient(hunting): authenticate will time out at 1013.006735s 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f79d9b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09c580 msgr2=0x7f3f7c13d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f79d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09c580 0x7f3f7c13d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 0x7f3f7c13df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 0x7f3f7c13df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:43528/0 (socket says 172.21.15.17:43528) 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 1 -- 172.21.15.17:0/4201639760 learned_addr learned my addr 172.21.15.17:0/4201639760 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): get_auth_request con 0x7f3f7c09c1b0 auth_method 0 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7ad9d640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c9c0 0x7f3f7c1414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7ad9d640 10 monclient(hunting): get_auth_request con 0x7f3f7c09c9c0 auth_method 0 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.009+0000 7f3f7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 10 monclient(hunting): handle_auth_done global_id 74201 payload 824 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c9c0 msgr2=0x7f3f7c1414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09c9c0 0x7f3f7c1414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09c580 msgr2=0x7f3f7c13d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09c580 0x7f3f7c13d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 monclient: found mon.noname-c 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:53.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.010+0000 7f3f7a59c640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70009390 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f7a59c640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 msgr2=0x7f3f7c13df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f7a59c640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 0x7f3f7c13df80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00b630 tx=0x7f3f6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c015070 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: dump: 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 5 monclient: authenticate success, global_id 74201 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c003d90 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 10 monclient: _renew_subs 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c1226d0 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c006380 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f6a7fc640 10 monclient: dump: 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.030 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 10 monclient: _renew_subs 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f78d99640 4 set_mon_vals no callback set 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.012+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c13dd10 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f80aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f6c010440 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 0x7f3f7c141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c063bb0 con 0x7f3f7c121800 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 0x7f3f7c141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient: get_auth_request con 0x7f3f7c121800 auth_method 0 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f6a7fc640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f5c0386f0 0x7f3f5c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6123+0+0 (secure 0 0 0) 0x7f3f6c04ba60 con 0x7f3f7c09c1b0 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_done global_id 5593 payload 824 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f79d9b640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f7c122e00 con 0x7f3f7c121800 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f7ad9d640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f5c0386f0 0x7f3f5c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.013+0000 7f3f7ad9d640 10 monclient: get_auth_request con 0x7f3f5c0386f0 auth_method 0 2024-09-15T01:55:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f79d9b640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 msgr2=0x7f3f7c141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 0x7f3f7c141a30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70032440 con 0x7f3f7c121800 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 10 monclient: dump: 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.033 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f7ad9d640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f5c0386f0 0x7f3f5c03aba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c13e500 tx=0x7f3f740052b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3f70032710 con 0x7f3f7c121800 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 msgr2=0x7f3f7c141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 0x7f3f7c141a30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70032000 comp rx=0 tx=0).stop 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 msgr2=0x7f3f7c141a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:53.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.016+0000 7f3f6a7fc640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c121800 0x7f3f7c141a30 secure :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70032000 comp rx=0 tx=0).stop 2024-09-15T01:55:53.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f80aa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:55:53.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 0x7f3f7c141f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:53.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f80aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:53.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c122bf0 con 0x7f3f7c13e900 2024-09-15T01:55:53.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 0x7f3f7c141f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient: get_auth_request con 0x7f3f7c13e900 auth_method 0 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 10 monclient(hunting): handle_auth_done global_id 5596 payload 824 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.022+0000 7f3f79d9b640 1 -- 172.21.15.17:0/4201639760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f7c1230e0 con 0x7f3f7c13e900 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f79d9b640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 msgr2=0x7f3f7c141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f79d9b640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 0x7f3f7c141f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009130 tx=0x7f3f7003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:53.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7003de80 con 0x7f3f7c13e900 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 10 monclient: dump: 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:55:53.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4726 (secure 0 0 0) 0x7f3f70009210 con 0x7f3f7c13e900 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 msgr2=0x7f3f7c141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 0x7f3f7c141f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009130 tx=0x7f3f7003d000 comp rx=0 tx=0).stop 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 msgr2=0x7f3f7c141f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.025+0000 7f3f6a7fc640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c13e900 0x7f3f7c141f70 secure :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009130 tx=0x7f3f7003d000 comp rx=0 tx=0).stop 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:53.039 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 54, 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:55:53.043 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.044 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:53.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:53.047 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:55:53.048 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.049 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.050 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.051 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:55:53.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:55:53.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:55:53.053 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:55:53.053 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:55:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f5c0386f0 msgr2=0x7f3f5c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f5c0386f0 0x7f3f5c03aba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c13e500 tx=0x7f3f740052b0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 10 monclient: shutdown 2024-09-15T01:55:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:53.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 msgr2=0x7f3f7c13df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 --2- 172.21.15.17:0/4201639760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c09c1b0 0x7f3f7c13df80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00b630 tx=0x7f3f6c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f7ad9d640 1 -- 172.21.15.17:0/4201639760 reap_dead start 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f7ad9d640 1 -- 172.21.15.17:0/4201639760 reap_dead start 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 shutdown_connections 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 >> 172.21.15.17:0/4201639760 conn(0x7f3f7c071300 msgr2=0x7f3f7c06ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 shutdown_connections 2024-09-15T01:55:53.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:55:53.026+0000 7f3f80aa8640 1 -- 172.21.15.17:0/4201639760 wait complete. 2024-09-15T01:55:53.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:55:53.055 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:55:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.082+0000 7f80af91d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.082+0000 7f80af91d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T01:55:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.082+0000 7f80af91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.082+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f80a8096a70 con 0x7f80a8097300 2024-09-15T01:55:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.797+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) v1 ==== 58+0+3 (secure 0 0 0) 0x7f80a4017080 con 0x7f80a8097300 2024-09-15T01:55:53.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.797+0000 7f809d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T01:55:53.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.797+0000 7f809d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:53.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.911+0000 7f80af91d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T01:55:53.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.911+0000 7f80af91d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T01:55:53.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.911+0000 7f80af91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:53.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:53.911+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f80a80978d0 con 0x7f80a8097300 2024-09-15T01:55:54.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.798+0000 7f809d7fa640 1 -- 172.21.15.67:0/1161891910 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v108) v1 ==== 58+0+3 (secure 0 0 0) 0x7f80a400e290 con 0x7f80a8097300 2024-09-15T01:55:54.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.798+0000 7f809d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T01:55:54.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.798+0000 7f809d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:54.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.801+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 msgr2=0x7f808c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:54.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.801+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 0x7f808c03abf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80980038f0 tx=0x7f8098006d80 comp rx=0 tx=0).stop 2024-09-15T01:55:54.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 10 monclient: shutdown 2024-09-15T01:55:54.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 msgr2=0x7f80a811a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c560 tx=0x7f80a4004960 comp rx=0 tx=0).stop 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 shutdown_connections 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f808c038740 0x7f808c03abf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80a8097d30 0x7f80a811aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80a8133f30 0x7f80a811af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 --2- 172.21.15.67:0/1161891910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a8097300 0x7f80a811a510 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 >> 172.21.15.67:0/1161891910 conn(0x7f80a80a1020 msgr2=0x7f80a809da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 shutdown_connections 2024-09-15T01:55:54.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.802+0000 7f80af91d640 1 -- 172.21.15.67:0/1161891910 wait complete. 2024-09-15T01:55:54.815 INFO:tasks.workunit.client.0.smithi067.stderr:+ o5=14 2024-09-15T01:55:54.816 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.897+0000 7f0d77f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.897+0000 7f0d77f9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.897+0000 7f0d77f9d640 10 monclient: monmap: 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:54.898942+0000 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:54.898942+0000 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:54.901 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:54.901 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:54.901 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:54.901 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:54.901 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.898+0000 7f0d77f9d640 1 Processor -- start 2024-09-15T01:55:54.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.898+0000 7f0d77f9d640 1 -- start start 2024-09-15T01:55:54.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.898+0000 7f0d77f9d640 10 monclient: init 2024-09-15T01:55:54.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 0x7f0d7006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d7006e210 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 0x7f0d7009db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7006e5e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d7009e050 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7006aa60 con 0x7f0d7009e050 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7006a190 con 0x7f0d7006e210 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d70069bd0 con 0x7f0d7006e5e0 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:54.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d77f9d640 10 monclient(hunting): authenticate will time out at 987.368217s 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75d39640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 msgr2=0x7f0d7006b9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75d39640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 0x7f0d7006b9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d7653a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0d7009e050 0x7f0d70138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39296/0 (socket says 172.21.15.67:39296) 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d7653a640 1 -- 172.21.15.67:0/3724328926 learned_addr learned my addr 172.21.15.67:0/3724328926 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 --2- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 0x7f0d7009db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): get_auth_request con 0x7f0d7006e5e0 auth_method 0 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:54.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient(hunting): handle_auth_done global_id 74204 payload 274 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 -- 172.21.15.67:0/3724328926 >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 msgr2=0x7f0d7006b9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 --2- 172.21.15.67:0/3724328926 >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 0x7f0d7006b9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 -- 172.21.15.67:0/3724328926 >> v1:172.21.15.67:6789/0 conn(0x7f0d7009e050 legacy=0x7f0d70138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient: _finish_hunting 0 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 monclient: found mon.noname-b 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:55:54.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.899+0000 7f0d75538640 1 -- 172.21.15.67:0/3724328926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700698f0 con 0x7f0d7006e5e0 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.902+0000 7f0d75538640 1 -- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 msgr2=0x7f0d7009db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.902+0000 7f0d75538640 1 --2- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 0x7f0d7009db10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009850 tx=0x7f0d640090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f88322c4f41c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 1 -- 172.21.15.67:0/3724328926 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6402ae00 con 0x7f0d7006e5e0 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: dump: 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:54.911 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: _finish_auth 0 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 1 -- 172.21.15.67:0/3724328926 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6402f4b0 con 0x7f0d7006e5e0 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 5 monclient: authenticate success, global_id 74204 2024-09-15T01:55:54.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 1 -- 172.21.15.67:0/3724328926 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6402f850 con 0x7f0d7006e5e0 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 4 set_mon_vals no callback set 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d74d37640 10 monclient: dump: 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:54.913 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 10 monclient: shutdown 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 -- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 msgr2=0x7f0d7009db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 0x7f0d7009db10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009850 tx=0x7f0d640090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 -- 172.21.15.67:0/3724328926 shutdown_connections 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/3724328926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e5e0 0x7f0d7009db10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/3724328926 >> v2:172.21.15.17:3300/0 conn(0x7f0d7006e210 0x7f0d7006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 -- 172.21.15.67:0/3724328926 >> 172.21.15.67:0/3724328926 conn(0x7f0d70070ff0 msgr2=0x7f0d700a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 -- 172.21.15.67:0/3724328926 shutdown_connections 2024-09-15T01:55:54.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.903+0000 7f0d77f9d640 1 -- 172.21.15.67:0/3724328926 wait complete. 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.904+0000 7f0d77f9d640 10 monclient: build_initial_monmap 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.904+0000 7f0d77f9d640 10 monclient: monmap: 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.904+0000 7f0d77f9d640 1 Processor -- start 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.904+0000 7f0d77f9d640 1 -- start start 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.904+0000 7f0d77f9d640 10 monclient: init 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 0x7f0d701417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d7006e210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 0x7f0d701322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d7006e5e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7009e050 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d70099ee0 con 0x7f0d7006e5e0 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700a2c10 con 0x7f0d7009e050 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7009d440 con 0x7f0d7006e210 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d77f9d640 10 monclient(hunting): authenticate will time out at 987.374269s 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75538640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 msgr2=0x7f0d701322b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75538640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 0x7f0d701322b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 0x7f0d701417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58404/0 (socket says 172.21.15.67:58404) 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 -- 172.21.15.67:0/1720724553 learned_addr learned my addr 172.21.15.67:0/1720724553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): get_auth_request con 0x7f0d7009e050 auth_method 0 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): get_auth_request con 0x7f0d7006e210 auth_method 0 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient(hunting): handle_auth_done global_id 5599 payload 824 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 -- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 msgr2=0x7f0d701417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:54.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 0x7f0d701417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 -- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 msgr2=0x7f0d701322b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 0x7f0d701322b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient: _finish_hunting 0 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 monclient: found mon.noname-b 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d7653a640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64009390 con 0x7f0d7009e050 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.905+0000 7f0d75d39640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 0x7f0d701417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:55:54.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d7653a640 1 -- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 msgr2=0x7f0d7009b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d7653a640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c510 tx=0x7f0d68004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68005cb0 con 0x7f0d7009e050 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: dump: 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:54.920 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d68005e30 con 0x7f0d7009e050 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 5 monclient: authenticate success, global_id 5599 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 10 monclient: _renew_subs 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7011f3d0 con 0x7f0d7009e050 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68007320 con 0x7f0d7009e050 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:54.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6dffb640 10 monclient: dump: 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 10 monclient: _renew_subs 2024-09-15T01:55:54.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:54.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d6ffff640 4 set_mon_vals no callback set 2024-09-15T01:55:54.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.908+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7011e470 con 0x7f0d7009e050 2024-09-15T01:55:54.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0d6801b3b0 con 0x7f0d7009e050 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d6dffb640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 0x7f0d5c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f0d6804b5d0 con 0x7f0d7009e050 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d77f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d75d39640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 0x7f0d5c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d77f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d77f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7004bf60 con 0x7f0d7009e050 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.909+0000 7f0d75d39640 10 monclient: get_auth_request con 0x7f0d5c0386f0 auth_method 0 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.912+0000 7f0d75d39640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 0x7f0d5c03aba0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003c70 tx=0x7f0d60009760 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:54.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.914+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0d68017080 con 0x7f0d7009e050 2024-09-15T01:55:54.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.914+0000 7f0d6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:54.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:54.914+0000 7f0d6dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.083+0000 7f0d77f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.083+0000 7f0d77f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.083+0000 7f0d77f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.083+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f0d7009f980 con 0x7f0d7009e050 2024-09-15T01:55:55.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.800+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v109) v1 ==== 220+0+0 (secure 0 0 0) 0x7f0d68018830 con 0x7f0d7009e050 2024-09-15T01:55:55.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.800+0000 7f0d6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.800+0000 7f0d6dffb640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-15T01:55:55.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.907+0000 7f0d77f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.907+0000 7f0d77f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.907+0000 7f0d77f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:55.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.907+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f0d70068a20 con 0x7f0d7009e050 2024-09-15T01:55:55.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.908+0000 7f0d6dffb640 1 -- 172.21.15.67:0/1720724553 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v109) v1 ==== 218+0+0 (secure 0 0 0) 0x7f0d68018260 con 0x7f0d7009e050 2024-09-15T01:55:55.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.908+0000 7f0d6dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:55.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.908+0000 7f0d6dffb640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T01:55:55.911 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T01:55:55.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 msgr2=0x7f0d5c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 0x7f0d5c03aba0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003c70 tx=0x7f0d60009760 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 10 monclient: shutdown 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 msgr2=0x7f0d7009b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c510 tx=0x7f0d68004900 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 shutdown_connections 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0d5c0386f0 0x7f0d5c03aba0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0d7006e5e0 0x7f0d701322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d7009e050 0x7f0d7009b9a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 --2- 172.21.15.67:0/1720724553 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0d7006e210 0x7f0d701417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 >> 172.21.15.67:0/1720724553 conn(0x7f0d70070ff0 msgr2=0x7f0d700a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 shutdown_connections 2024-09-15T01:55:55.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.910+0000 7f0d77f9d640 1 -- 172.21.15.67:0/1720724553 wait complete. 2024-09-15T01:55:55.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.998+0000 7f12614b8640 10 monclient: get_monmap_and_config 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.998+0000 7f12614b8640 10 monclient: build_initial_monmap 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.998+0000 7f12614b8640 10 monclient: monmap: 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:55.999971+0000 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:55.999971+0000 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:56.002 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.999+0000 7f12614b8640 1 Processor -- start 2024-09-15T01:55:56.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.999+0000 7f12614b8640 1 -- start start 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:55.999+0000 7f12614b8640 10 monclient: init 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 0x7f125c069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-b con 0x7f125c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c09eea0 0x7f125c09f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-c con 0x7f125c09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 0x7f125c131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-a con 0x7f125c131b10 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c06f290 con 0x7f125c131b10 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c09f7b0 con 0x7f125c09eea0 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c06a270 con 0x7f125c0a0b50 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f12614b8640 10 monclient(hunting): authenticate will time out at 988.469194s 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c09eea0 0x7f125c09f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f125b7fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 msgr2=0x7f125c131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c09eea0 0x7f125c09f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58416/0 (socket says 172.21.15.67:58416) 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 -- 172.21.15.67:0/963760275 learned_addr learned my addr 172.21.15.67:0/963760275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): get_auth_request con 0x7f125c09eea0 auth_method 0 2024-09-15T01:55:56.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f125b7fe640 1 --2- 172.21.15.67:0/963760275 >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 0x7f125c131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f125affd640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 0x7f125c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient(hunting): handle_auth_done global_id 5602 payload 274 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 -- 172.21.15.67:0/963760275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 msgr2=0x7f125c069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 0x7f125c069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 -- 172.21.15.67:0/963760275 >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 msgr2=0x7f125c131f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 --2- 172.21.15.67:0/963760275 >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 0x7f125c131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient: _finish_hunting 0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 monclient: found mon.noname-c 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f1253fff640 1 -- 172.21.15.67:0/963760275 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0a0f20 con 0x7f125c09eea0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f125affd640 10 monclient: get_auth_request con 0x7f125c0a0b50 auth_method 0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.000+0000 7f125affd640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 0x7f125c069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f1253fff640 1 -- 172.21.15.67:0/963760275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 msgr2=0x7f125c09f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f1253fff640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c09f270 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f125c06a480 tx=0x7f12440091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1ccd4e1b64a0e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 1 -- 172.21.15.67:0/963760275 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244032440 con 0x7f125c09eea0 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: dump: 2024-09-15T01:55:56.017 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f12614b8640 5 monclient: authenticate success, global_id 5602 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f12614b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:56.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 1 -- 172.21.15.67:0/963760275 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12440325a0 con 0x7f125c09eea0 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f12614b8640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f12614b8640 4 set_mon_vals no callback set 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 1 -- 172.21.15.67:0/963760275 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244032940 con 0x7f125c09eea0 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.003+0000 7f125a7fc640 10 monclient: dump: 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:56.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 10 monclient: shutdown 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 -- 172.21.15.67:0/963760275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 msgr2=0x7f125c09f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c09f270 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f125c06a480 tx=0x7f12440091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 -- 172.21.15.67:0/963760275 shutdown_connections 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c09f270 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 --2- 172.21.15.67:0/963760275 >> v2:172.21.15.17:3300/0 conn(0x7f125c131b10 0x7f125c131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 --2- 172.21.15.67:0/963760275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c0a0b50 0x7f125c069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 -- 172.21.15.67:0/963760275 >> 172.21.15.67:0/963760275 conn(0x7f125c071290 msgr2=0x7f125c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 -- 172.21.15.67:0/963760275 shutdown_connections 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 -- 172.21.15.67:0/963760275 wait complete. 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 10 monclient: build_initial_monmap 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 10 monclient: monmap: 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.004+0000 7f12614b8640 1 Processor -- start 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 -- start start 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient: init 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-b con 0x7f125c09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 0x7f125c140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-a con 0x7f125c0a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c131b10 0x7f125c141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): picked mon.noname-c con 0x7f125c131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c141a60 con 0x7f125c0a0b50 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c11d2f0 con 0x7f125c09eea0 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c11f3e0 con 0x7f125c131b10 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f12614b8640 10 monclient(hunting): authenticate will time out at 988.474473s 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f1253fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 msgr2=0x7f125c140fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f1253fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 0x7f125c140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58418/0 (socket says 172.21.15.67:58418) 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 -- 172.21.15.67:0/3242639833 learned_addr learned my addr 172.21.15.67:0/3242639833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): get_auth_request con 0x7f125c09eea0 auth_method 0 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c131b10 0x7f125c141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): get_auth_request con 0x7f125c131b10 auth_method 0 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient(hunting): handle_auth_done global_id 5605 payload 824 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 -- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c131b10 msgr2=0x7f125c141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c131b10 0x7f125c141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 -- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 msgr2=0x7f125c140fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 0x7f125c140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 monclient: found mon.noname-b 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.005+0000 7f125affd640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009450 con 0x7f125c09eea0 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f125affd640 1 -- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 msgr2=0x7f125c130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f125affd640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f124c00c880 tx=0x7f124c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00ce10 con 0x7f125c09eea0 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: dump: 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.026 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c007aa0 con 0x7f125c09eea0 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 5 monclient: authenticate success, global_id 5605 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 10 monclient: _renew_subs 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c11f6c0 con 0x7f125c09eea0 2024-09-15T01:55:56.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c007e20 con 0x7f125c09eea0 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1252ffd640 10 monclient: dump: 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:56.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f1259ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 10 monclient: _renew_subs 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.008+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c131230 con 0x7f125c09eea0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f124c011600 con 0x7f125c09eea0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f12614b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f12614b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f12614b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125c04bf60 con 0x7f125c09eea0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f1252ffd640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 0x7f123c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f124c04c3f0 con 0x7f125c09eea0 2024-09-15T01:55:56.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f1253fff640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 0x7f123c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:56.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.009+0000 7f1253fff640 10 monclient: get_auth_request con 0x7f123c0386f0 auth_method 0 2024-09-15T01:55:56.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.012+0000 7f1253fff640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 0x7f123c03aba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f125c09ccd0 tx=0x7f1244032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:56.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.013+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f124c017ae0 con 0x7f125c09eea0 2024-09-15T01:55:56.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.013+0000 7f1252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:56.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.013+0000 7f1252ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:56.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.180+0000 7f12614b8640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.180+0000 7f12614b8640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.180+0000 7f12614b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.180+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f125c06f000 con 0x7f125c09eea0 2024-09-15T01:55:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.808+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v110) v1 ==== 220+0+0 (secure 0 0 0) 0x7f124c017300 con 0x7f125c09eea0 2024-09-15T01:55:56.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.808+0000 7f1252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.808+0000 7f1252ffd640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-15T01:55:56.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f12614b8640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f12614b8640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f12614b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:56.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f125c131450 con 0x7f125c09eea0 2024-09-15T01:55:56.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f1252ffd640 1 -- 172.21.15.67:0/3242639833 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v110) v1 ==== 218+0+0 (secure 0 0 0) 0x7f124c0174e0 con 0x7f125c09eea0 2024-09-15T01:55:56.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f1252ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T01:55:56.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.912+0000 7f1252ffd640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T01:55:56.915 INFO:tasks.workunit.client.0.smithi067.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 msgr2=0x7f123c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 0x7f123c03aba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f125c09ccd0 tx=0x7f1244032000 comp rx=0 tx=0).stop 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 10 monclient: shutdown 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 msgr2=0x7f125c130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f124c00c880 tx=0x7f124c008040 comp rx=0 tx=0).stop 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 shutdown_connections 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f123c0386f0 0x7f123c03aba0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f125c0a0b50 0x7f125c140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f125c131b10 0x7f125c141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 --2- 172.21.15.67:0/3242639833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c09eea0 0x7f125c130cf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:56.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 >> 172.21.15.67:0/3242639833 conn(0x7f125c071290 msgr2=0x7f125c131f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:56.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 shutdown_connections 2024-09-15T01:55:56.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.914+0000 7f12614b8640 1 -- 172.21.15.67:0/3242639833 wait complete. 2024-09-15T01:55:56.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:56.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.12 2024-09-15T01:55:56.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 123 2024-09-15T01:55:56.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.997+0000 7fb099f54640 10 monclient: get_monmap_and_config 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.997+0000 7fb099f54640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.997+0000 7fb099f54640 10 monclient: monmap: 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:56.998681+0000 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:56.998681+0000 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:57.001 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:57.001 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 Processor -- start 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 -- start start 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient: init 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T01:55:57.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0940981b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb094098580 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb094098980 0x7fb09413a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-a con 0x7fb094098980 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940a07b0 con 0x7fb094098580 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940a16e0 con 0x7fb0940981b0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb094098d90 con 0x7fb094098980 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb099f54640 10 monclient(hunting): authenticate will time out at 989.467932s 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb093fff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb094098980 msgr2=0x7fb09413a5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb092ffd640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fb094098580 legacy=0x7fb094136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.998+0000 7fb0937fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb0940981b0 0x7fb0941322c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39302/0 (socket says 172.21.15.67:39302) 2024-09-15T01:55:57.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.999+0000 7fb093fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb094098980 0x7fb09413a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:56.999+0000 7fb0937fe640 1 -- 172.21.15.67:0/859250856 learned_addr learned my addr 172.21.15.67:0/859250856 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:57.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0937fe640 1 -- 172.21.15.67:0/859250856 >> v1:172.21.15.67:6789/0 conn(0x7fb0940981b0 legacy=0x7fb0941322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306583525 0 0) 0x7fb0940a16e0 con 0x7fb0940981b0 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): my global_id is 5608 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c0034e0 con 0x7fb0940981b0 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 904436693 0 0) 0x7fb07c0034e0 con 0x7fb0940981b0 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 >> v2:172.21.15.17:3300/0 conn(0x7fb094098980 msgr2=0x7fb09413a5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 --2- 172.21.15.67:0/859250856 >> v2:172.21.15.17:3300/0 conn(0x7fb094098980 0x7fb09413a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 >> v1:172.21.15.17:6789/0 conn(0x7fb094098580 legacy=0x7fb094136ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:55:57.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0941329d0 con 0x7fb0940981b0 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fb084001720 con 0x7fb0940981b0 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb099f54640 5 monclient: authenticate success, global_id 5608 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb099f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: dump: 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.008 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 >> v1:172.21.15.67:6789/0 conn(0x7fb0940981b0 legacy=0x7fb0941322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb07c0039d0 0x7fb07c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): picked mon.a con 0x7fb07c0039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb07c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): picked mon.b con 0x7fb07c004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb07c005330 0x7fb07c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): picked mon.c con 0x7fb07c005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094098d90 con 0x7fb07c0039d0 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c00c290 con 0x7fb07c004780 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c00c450 con 0x7fb07c005330 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0927fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb099f54640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0937fe640 1 -- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb07c0039d0 msgr2=0x7fb07c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb0937fe640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb07c0039d0 0x7fb07c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb07c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): get_auth_request con 0x7fb07c004780 auth_method 0 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:55:57.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.001+0000 7fb092ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb093fff640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb07c005330 0x7fb07c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 10 monclient(hunting): handle_auth_done global_id 5608 payload 293 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 -- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb07c005330 msgr2=0x7fb07c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb07c005330 0x7fb07c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 -- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb07c0039d0 msgr2=0x7fb07c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb07c0039d0 0x7fb07c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 monclient: found mon.b 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 1 -- 172.21.15.67:0/859250856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0034e0 con 0x7fb07c004780 2024-09-15T01:55:57.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb093fff640 10 monclient: get_auth_request con 0x7fb07c005330 auth_method 0 2024-09-15T01:55:57.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb093fff640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb07c005330 0x7fb07c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:57.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.002+0000 7fb093fff640 1 -- 172.21.15.67:0/859250856 reap_dead start 2024-09-15T01:55:57.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.004+0000 7fb092ffd640 1 -- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 msgr2=0x7fb07c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.004+0000 7fb092ffd640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb07c004bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb0880099c0 tx=0x7fb088009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=740cea6927b13ca2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088003b80 con 0x7fb07c004780 2024-09-15T01:55:57.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: dump: 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb088003d20 con 0x7fb07c004780 2024-09-15T01:55:57.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 1 -- 172.21.15.67:0/859250856 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08802ab80 con 0x7fb07c004780 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 4 set_mon_vals no callback set 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb0927fc640 10 monclient: dump: 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.020 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 10 monclient: shutdown 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 -- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 msgr2=0x7fb07c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb07c004bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb0880099c0 tx=0x7fb088009060 comp rx=0 tx=0).stop 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 -- 172.21.15.67:0/859250856 shutdown_connections 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 --2- 172.21.15.67:0/859250856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb07c004bf0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 -- 172.21.15.67:0/859250856 >> 172.21.15.67:0/859250856 conn(0x7fb094071150 msgr2=0x7fb0940a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:57.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 -- 172.21.15.67:0/859250856 shutdown_connections 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.005+0000 7fb099f54640 1 -- 172.21.15.67:0/859250856 wait complete. 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.006+0000 7fb099f54640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.006+0000 7fb099f54640 10 monclient: monmap: 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:57.022 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.006+0000 7fb099f54640 1 Processor -- start 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.006+0000 7fb099f54640 1 -- start start 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.006+0000 7fb099f54640 10 monclient: init 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-b con 0x7fb07c004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 0x7fb094130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-a con 0x7fb09411cdc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb094131140 0x7fb09411b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): picked mon.noname-c con 0x7fb094131140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094098ed0 con 0x7fb09411cdc0 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940a45f0 con 0x7fb07c004780 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09411e000 con 0x7fb094131140 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb099f54640 10 monclient(hunting): authenticate will time out at 989.476413s 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58432/0 (socket says 172.21.15.67:58432) 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 -- 172.21.15.67:0/3151409176 learned_addr learned my addr 172.21.15.67:0/3151409176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): get_auth_request con 0x7fb07c004780 auth_method 0 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb092ffd640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 msgr2=0x7fb094130c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 0x7fb094130c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb093fff640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb094131140 0x7fb09411b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb093fff640 10 monclient(hunting): get_auth_request con 0x7fb094131140 auth_method 0 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb093fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient(hunting): handle_auth_done global_id 5611 payload 824 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb094131140 msgr2=0x7fb09411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb094131140 0x7fb09411b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 msgr2=0x7fb094130c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 0x7fb094130c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 monclient: found mon.noname-b 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.007+0000 7fb0937fe640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09409f120 con 0x7fb07c004780 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb0937fe640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 msgr2=0x7fb09409e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb0937fe640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb084001350 tx=0x7fb084001890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084007360 con 0x7fb07c004780 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: dump: 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 5 monclient: authenticate success, global_id 5611 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0840074c0 con 0x7fb07c004780 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 10 monclient: _renew_subs 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094141ff0 con 0x7fb07c004780 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0840047a0 con 0x7fb07c004780 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb07b7fe640 10 monclient: dump: 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 10 monclient: _renew_subs 2024-09-15T01:55:57.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb091ffb640 4 set_mon_vals no callback set 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.010+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09411bde0 con 0x7fb07c004780 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb084004b50 con 0x7fb07c004780 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb099f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb099f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb099f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09411e1c0 con 0x7fb07c004780 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb07b7fe640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fb08404ba30 con 0x7fb07c004780 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 10 monclient: get_auth_request con 0x7fb0800386f0 auth_method 0 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 msgr2=0x7fb08003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:55:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 msgr2=0x7fb08003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:55:57.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fb0880038f0 tx=0x7fb088030000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:55:57.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.011+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.015+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb08401c3b0 con 0x7fb07c004780 2024-09-15T01:55:57.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.015+0000 7fb07b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.015+0000 7fb07b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb099f54640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb099f54640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb099f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb09404bf60 con 0x7fb07c004780 2024-09-15T01:55:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fb084010300 con 0x7fb07c004780 2024-09-15T01:55:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb07b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.180+0000 7fb07b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.212+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.212+0000 7fb092ffd640 10 monclient: get_auth_request con 0x7fb0800386f0 auth_method 0 2024-09-15T01:55:57.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.214+0000 7fb092ffd640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb088009420 tx=0x7fb08803b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.278+0000 7fb099f54640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.278+0000 7fb099f54640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.278+0000 7fb099f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.278+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb0940a1400 con 0x7fb07c004780 2024-09-15T01:55:57.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.279+0000 7fb07b7fe640 1 -- 172.21.15.67:0/3151409176 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fb084019260 con 0x7fb07c004780 2024-09-15T01:55:57.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.279+0000 7fb07b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.279+0000 7fb07b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 msgr2=0x7fb08003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb088009420 tx=0x7fb08803b000 comp rx=0 tx=0).stop 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 10 monclient: shutdown 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 msgr2=0x7fb09409e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb084001350 tx=0x7fb084001890 comp rx=0 tx=0).stop 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 shutdown_connections 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0800386f0 0x7fb08003aba0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb09411cdc0 0x7fb094130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb094131140 0x7fb09411b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 --2- 172.21.15.67:0/3151409176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c004780 0x7fb09409e720 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 >> 172.21.15.67:0/3151409176 conn(0x7fb094071150 msgr2=0x7fb094096f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:57.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 shutdown_connections 2024-09-15T01:55:57.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.281+0000 7fb099f54640 1 -- 172.21.15.67:0/3151409176 wait complete. 2024-09-15T01:55:57.292 INFO:tasks.workunit.client.0.smithi067.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-15T01:55:57.293 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:55:57.293 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.14 2024-09-15T01:55:57.293 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 123 2024-09-15T01:55:57.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.367+0000 7fa38ac60640 10 monclient: get_monmap_and_config 2024-09-15T01:55:57.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.367+0000 7fa38ac60640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.367+0000 7fa38ac60640 10 monclient: monmap: 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:57.368368+0000 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:57.368368+0000 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:57.370 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.367+0000 7fa38ac60640 1 Processor -- start 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 -- start start 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient: init 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa384098370 0x7fa384132350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-c con 0x7fa384098370 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098740 0x7fa384098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-b con 0x7fa384098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa384068e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3840a17b0 con 0x7fa384068e70 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3840a0160 con 0x7fa384098370 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa384096fc0 con 0x7fa384098740 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa38ac60640 10 monclient(hunting): authenticate will time out at 989.837622s 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa384098370 0x7fa384132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3891fd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa384068e70 0x7fa3840692f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39304/0 (socket says 172.21.15.67:39304) 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3891fd640 1 -- 172.21.15.67:0/1902053465 learned_addr learned my addr 172.21.15.67:0/1902053465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): get_auth_request con 0x7fa384098370 auth_method 0 2024-09-15T01:55:57.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa383fff640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098740 0x7fa384098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa383fff640 10 monclient(hunting): get_auth_request con 0x7fa384098740 auth_method 0 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa383fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.368+0000 7fa3889fc640 10 monclient(hunting): handle_auth_done global_id 5614 payload 274 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 1 -- 172.21.15.67:0/1902053465 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098740 msgr2=0x7fa384098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:57.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098740 0x7fa384098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 1 -- 172.21.15.67:0/1902053465 >> v1:172.21.15.67:6789/0 conn(0x7fa384068e70 legacy=0x7fa3840692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 1 monclient: found mon.noname-c 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.369+0000 7fa3889fc640 1 -- 172.21.15.67:0/1902053465 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38409fef0 con 0x7fa384098370 2024-09-15T01:55:57.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.371+0000 7fa3889fc640 1 -- 172.21.15.67:0/1902053465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384098370 msgr2=0x7fa384132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.371+0000 7fa3889fc640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384098370 0x7fa384132350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa38406a170 tx=0x7fa3780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314089aa431ac09b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 1 -- 172.21.15.67:0/1902053465 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37802adb0 con 0x7fa384098370 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: dump: 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.382 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 1 -- 172.21.15.67:0/1902053465 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa378032440 con 0x7fa384098370 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 5 monclient: authenticate success, global_id 5614 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 1 -- 172.21.15.67:0/1902053465 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780327e0 con 0x7fa384098370 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 4 set_mon_vals no callback set 2024-09-15T01:55:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa3837fe640 10 monclient: dump: 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 10 monclient: shutdown 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 -- 172.21.15.67:0/1902053465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384098370 msgr2=0x7fa384132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384098370 0x7fa384132350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa38406a170 tx=0x7fa3780090f0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 -- 172.21.15.67:0/1902053465 shutdown_connections 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098740 0x7fa384098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1902053465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384098370 0x7fa384132350 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 -- 172.21.15.67:0/1902053465 >> 172.21.15.67:0/1902053465 conn(0x7fa384071150 msgr2=0x7fa3840a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 -- 172.21.15.67:0/1902053465 shutdown_connections 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.372+0000 7fa38ac60640 1 -- 172.21.15.67:0/1902053465 wait complete. 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.373+0000 7fa38ac60640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.373+0000 7fa38ac60640 10 monclient: monmap: 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.373+0000 7fa38ac60640 1 Processor -- start 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.373+0000 7fa38ac60640 1 -- start start 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.373+0000 7fa38ac60640 10 monclient: init 2024-09-15T01:55:57.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-b con 0x7fa384068e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098370 0x7fa384130d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-c con 0x7fa384098370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 0x7fa38411b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): picked mon.noname-a con 0x7fa384098740 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38411e180 con 0x7fa384098740 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840a4430 con 0x7fa384068e70 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38411cf40 con 0x7fa384098370 2024-09-15T01:55:57.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa38ac60640 10 monclient(hunting): authenticate will time out at 989.843170s 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3891fd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 msgr2=0x7fa38411b830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3891fd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 0x7fa38411b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58454/0 (socket says 172.21.15.67:58454) 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 -- 172.21.15.67:0/1545914257 learned_addr learned my addr 172.21.15.67:0/1545914257 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): get_auth_request con 0x7fa384068e70 auth_method 0 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098370 0x7fa384130d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): get_auth_request con 0x7fa384098370 auth_method 0 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa383fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient(hunting): handle_auth_done global_id 5617 payload 824 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 -- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098370 msgr2=0x7fa384130d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098370 0x7fa384130d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 -- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 msgr2=0x7fa38411b830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 0x7fa38411b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 monclient: found mon.noname-b 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.374+0000 7fa3889fc640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa378009390 con 0x7fa384068e70 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.376+0000 7fa3889fc640 1 -- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 msgr2=0x7fa38409e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.376+0000 7fa3889fc640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa378009360 tx=0x7fa378032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.376+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378031d40 con 0x7fa384068e70 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: dump: 2024-09-15T01:55:57.390 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37803d440 con 0x7fa384068e70 2024-09-15T01:55:57.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 5 monclient: authenticate success, global_id 5617 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 10 monclient: _renew_subs 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa384142310 con 0x7fa384068e70 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378031ea0 con 0x7fa384068e70 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa380ff9640 10 monclient: dump: 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa382ffd640 4 set_mon_vals no callback set 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 10 monclient: _renew_subs 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.377+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38409f250 con 0x7fa384068e70 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa378039040 con 0x7fa384068e70 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa38ac60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa38ac60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa38ac60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3840a0900 con 0x7fa384068e70 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa380ff9640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 0x7fa36803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fa37806c350 con 0x7fa384068e70 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa383fff640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 0x7fa36803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.378+0000 7fa383fff640 10 monclient: get_auth_request con 0x7fa368038740 auth_method 0 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.381+0000 7fa383fff640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 0x7fa36803abf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa38411ddd0 tx=0x7fa370008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.382+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa3840a0900 con 0x7fa384068e70 2024-09-15T01:55:57.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.382+0000 7fa380ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.382+0000 7fa380ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:57.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.546+0000 7fa38ac60640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:57.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.546+0000 7fa38ac60640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.546+0000 7fa38ac60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.546+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa3840a0900 con 0x7fa384068e70 2024-09-15T01:55:57.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.547+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fa3780375c0 con 0x7fa384068e70 2024-09-15T01:55:57.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.547+0000 7fa380ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.547+0000 7fa380ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:55:57.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa38ac60640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:55:57.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa38ac60640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa38ac60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa38411e5f0 con 0x7fa384068e70 2024-09-15T01:55:57.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa380ff9640 1 -- 172.21.15.67:0/1545914257 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fa378037070 con 0x7fa384068e70 2024-09-15T01:55:57.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa380ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:55:57.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.645+0000 7fa380ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:55:57.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 msgr2=0x7fa36803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 0x7fa36803abf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa38411ddd0 tx=0x7fa370008040 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 10 monclient: shutdown 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 msgr2=0x7fa38409e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa378009360 tx=0x7fa378032000 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 shutdown_connections 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa368038740 0x7fa36803abf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa384098740 0x7fa38411b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa384098370 0x7fa384130d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 --2- 172.21.15.67:0/1545914257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa384068e70 0x7fa38409e7f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 >> 172.21.15.67:0/1545914257 conn(0x7fa384071150 msgr2=0x7fa3840a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 shutdown_connections 2024-09-15T01:55:57.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.647+0000 7fa38ac60640 1 -- 172.21.15.67:0/1545914257 wait complete. 2024-09-15T01:55:57.660 INFO:tasks.workunit.client.0.smithi067.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-15T01:55:57.660 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-15T01:55:57.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.732+0000 7fdff0878640 10 monclient: get_monmap_and_config 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.732+0000 7fdff0878640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.732+0000 7fdff0878640 10 monclient: monmap: 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:57.733665+0000 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:57.733665+0000 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:57.735 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:57.736 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:57.736 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 Processor -- start 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 -- start start 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient: init 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfec0a3850 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfec06a6e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfec138420 0x7fdfec13a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec138420 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec09f9a0 con 0x7fdfec0a3850 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec0a3c20 con 0x7fdfec06a6e0 2024-09-15T01:55:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfec069c20 con 0x7fdfec138420 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdff0878640 10 monclient(hunting): authenticate will time out at 990.202789s 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfead9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfec138420 0x7fdfec13a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfea59c640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fdfec0a3850 legacy=0x7fdfec06e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfe9d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdfec06a6e0 0x7fdfec136d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39308/0 (socket says 172.21.15.67:39308) 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfe9d9b640 1 -- 172.21.15.67:0/3258605321 learned_addr learned my addr 172.21.15.67:0/3258605321 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfead9d640 10 monclient(hunting): get_auth_request con 0x7fdfec138420 auth_method 0 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfead9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.733+0000 7fdfead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient(hunting): handle_auth_done global_id 5620 payload 274 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 1 -- 172.21.15.67:0/3258605321 >> v1:172.21.15.67:6789/0 conn(0x7fdfec06a6e0 legacy=0x7fdfec136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 1 -- 172.21.15.67:0/3258605321 >> v1:172.21.15.17:6789/0 conn(0x7fdfec0a3850 legacy=0x7fdfec06e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 1 monclient: found mon.noname-c 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.734+0000 7fdfead9d640 1 -- 172.21.15.67:0/3258605321 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec09f4b0 con 0x7fdfec138420 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfead9d640 1 -- 172.21.15.67:0/3258605321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec138420 msgr2=0x7fdfec13a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfead9d640 1 --2- 172.21.15.67:0/3258605321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec138420 0x7fdfec13a800 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009e60 tx=0x7fdfe4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5107c48065cde85c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 1 -- 172.21.15.67:0/3258605321 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4007840 con 0x7fdfec138420 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: dump: 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.746 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 1 -- 172.21.15.67:0/3258605321 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe402d050 con 0x7fdfec138420 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 5 monclient: authenticate success, global_id 5620 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 1 -- 172.21.15.67:0/3258605321 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4007bd0 con 0x7fdfec138420 2024-09-15T01:55:57.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 4 set_mon_vals no callback set 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdfe959a640 10 monclient: dump: 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 10 monclient: shutdown 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 -- 172.21.15.67:0/3258605321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec138420 msgr2=0x7fdfec13a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 --2- 172.21.15.67:0/3258605321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec138420 0x7fdfec13a800 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009e60 tx=0x7fdfe4009780 comp rx=0 tx=0).stop 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 -- 172.21.15.67:0/3258605321 shutdown_connections 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 --2- 172.21.15.67:0/3258605321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec138420 0x7fdfec13a800 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 -- 172.21.15.67:0/3258605321 >> 172.21.15.67:0/3258605321 conn(0x7fdfec070ff0 msgr2=0x7fdfec0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 -- 172.21.15.67:0/3258605321 shutdown_connections 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.737+0000 7fdff0878640 1 -- 172.21.15.67:0/3258605321 wait complete. 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 10 monclient: build_initial_monmap 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 10 monclient: monmap: 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:57.749 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 1 Processor -- start 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 1 -- start start 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 10 monclient: init 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.738+0000 7fdff0878640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-b con 0x7fdfec06a6e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfec0a3850 0x7fdfec11e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec0a3850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 0x7fdfec11ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): picked mon.noname-a con 0x7fdfec138420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec1222c0 con 0x7fdfec138420 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec06e9e0 con 0x7fdfec06a6e0 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec0a2470 con 0x7fdfec0a3850 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdff0878640 10 monclient(hunting): authenticate will time out at 990.208094s 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58480/0 (socket says 172.21.15.67:58480) 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 -- 172.21.15.67:0/935899365 learned_addr learned my addr 172.21.15.67:0/935899365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): get_auth_request con 0x7fdfec06a6e0 auth_method 0 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfead9d640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 msgr2=0x7fdfec11ec10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfead9d640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 0x7fdfec11ec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfe9d9b640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfec0a3850 0x7fdfec11e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient(hunting): handle_auth_done global_id 5623 payload 824 2024-09-15T01:55:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfec0a3850 msgr2=0x7fdfec11e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfec0a3850 0x7fdfec11e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 msgr2=0x7fdfec11ec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 0x7fdfec11ec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient: _finish_hunting 0 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 monclient: found mon.noname-b 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.739+0000 7fdfea59c640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe4009a20 con 0x7fdfec06a6e0 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfea59c640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 msgr2=0x7fdfec137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfea59c640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00b5f0 tx=0x7fdfdc009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc015070 con 0x7fdfec06a6e0 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: dump: 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: _finish_auth 0 2024-09-15T01:55:57.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc003d40 con 0x7fdfec06a6e0 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 5 monclient: authenticate success, global_id 5623 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 10 monclient: _renew_subs 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec137290 con 0x7fdfec06a6e0 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc006380 con 0x7fdfec06a6e0 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfda7fc640 10 monclient: dump: 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.755 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdfe8d99640 4 set_mon_vals no callback set 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 10 monclient: _renew_subs 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.741+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec1206e0 con 0x7fdfec06a6e0 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdff0878640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdff0878640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdff0878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdfdc010440 con 0x7fdfec06a6e0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec121e50 con 0x7fdfec06a6e0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdfda7fc640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 0x7fdfcc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.742+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fdfdc04be40 con 0x7fdfec06a6e0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.743+0000 7fdfe9d9b640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 0x7fdfcc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.743+0000 7fdfe9d9b640 10 monclient: get_auth_request con 0x7fdfcc0386f0 auth_method 0 2024-09-15T01:55:57.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.745+0000 7fdfe9d9b640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 0x7fdfcc03aba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdfec11fe60 tx=0x7fdfe0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:57.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.747+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdfdc0067b0 con 0x7fdfec06a6e0 2024-09-15T01:55:57.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.747+0000 7fdfda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:57.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.747+0000 7fdfda7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:55:57.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.914+0000 7fdff0878640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:57.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.914+0000 7fdff0878640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:57.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.914+0000 7fdff0878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:57.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:57.914+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7fdfec04bf60 con 0x7fdfec06a6e0 2024-09-15T01:55:58.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.294+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v112) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdfdc018690 con 0x7fdfec06a6e0 2024-09-15T01:55:58.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.294+0000 7fdfda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:58.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.294+0000 7fdfda7fc640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T01:55:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.397+0000 7fdff0878640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.397+0000 7fdff0878640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.397+0000 7fdff0878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:58.397+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7fdfec0008d0 con 0x7fdfec06a6e0 2024-09-15T01:55:59.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.306+0000 7fdfda7fc640 1 -- 172.21.15.67:0/935899365 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v113) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdfdc018ab0 con 0x7fdfec06a6e0 2024-09-15T01:55:59.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.307+0000 7fdfda7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T01:55:59.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.307+0000 7fdfda7fc640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T01:55:59.310 INFO:tasks.workunit.client.0.smithi067.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T01:55:59.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 msgr2=0x7fdfcc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:59.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 0x7fdfcc03aba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdfec11fe60 tx=0x7fdfe0009040 comp rx=0 tx=0).stop 2024-09-15T01:55:59.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 10 monclient: shutdown 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 msgr2=0x7fdfec137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.309+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00b5f0 tx=0x7fdfdc009120 comp rx=0 tx=0).stop 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 shutdown_connections 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdfcc0386f0 0x7fdfcc03aba0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfec138420 0x7fdfec11ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfec0a3850 0x7fdfec11e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 --2- 172.21.15.67:0/935899365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfec06a6e0 0x7fdfec137d80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 >> 172.21.15.67:0/935899365 conn(0x7fdfec070ff0 msgr2=0x7fdfec09fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 shutdown_connections 2024-09-15T01:55:59.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.310+0000 7fdff0878640 1 -- 172.21.15.67:0/935899365 wait complete. 2024-09-15T01:55:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-15T01:55:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:55:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.403+0000 7f9753a80640 10 monclient: get_monmap_and_config 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.403+0000 7f9753a80640 10 monclient: build_initial_monmap 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.403+0000 7f9753a80640 10 monclient: monmap: 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:55:59.404947+0000 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:55:59.404947+0000 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.406 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:55:59.407 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:55:59.407 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:55:59.407 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:55:59.407 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:55:59.407 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.404+0000 7f9753a80640 1 Processor -- start 2024-09-15T01:55:59.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.404+0000 7f9753a80640 1 -- start start 2024-09-15T01:55:59.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.404+0000 7f9753a80640 10 monclient: init 2024-09-15T01:55:59.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:59.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 0x7f974c132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-a con 0x7f974c131f90 addr v2:172.21.15.17:3300/0 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f974c098260 0x7f974c098660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-c con 0x7f974c098260 addr v2:172.21.15.67:3300/0 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f974c098ba0 addr v1:172.21.15.17:6789/0 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c096a70 con 0x7f974c098ba0 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c06a6b0 con 0x7f974c131f90 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c098fb0 con 0x7f974c098260 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f9753a80640 10 monclient(hunting): authenticate will time out at 991.874138s 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f974c098260 0x7f974c098660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975181c640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 msgr2=0x7f974c132410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975201d640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f974c098ba0 legacy=0x7f974c068e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975181c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 0x7f974c132410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f974c098260 0x7f974c098660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58482/0 (socket says 172.21.15.67:58482) 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 -- 172.21.15.67:0/1028678584 learned_addr learned my addr 172.21.15.67:0/1028678584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): get_auth_request con 0x7f974c098260 auth_method 0 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient(hunting): handle_auth_done global_id 5626 payload 274 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 -- 172.21.15.67:0/1028678584 >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 msgr2=0x7f974c132410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 --2- 172.21.15.67:0/1028678584 >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 0x7f974c132410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 -- 172.21.15.67:0/1028678584 >> v1:172.21.15.17:6789/0 conn(0x7f974c098ba0 legacy=0x7f974c068e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 monclient: found mon.noname-c 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:55:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.405+0000 7f975101b640 1 -- 172.21.15.67:0/1028678584 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0a16e0 con 0x7f974c098260 2024-09-15T01:55:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975101b640 1 -- 172.21.15.67:0/1028678584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098260 msgr2=0x7f974c098660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975101b640 1 --2- 172.21.15.67:0/1028678584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098260 0x7f974c098660 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f974c0a2e00 tx=0x7f97480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2d5a6c24630f0a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 1 -- 172.21.15.67:0/1028678584 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748032440 con 0x7f974c098260 2024-09-15T01:55:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: dump: 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:59.418 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: _finish_auth 0 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f9753a80640 5 monclient: authenticate success, global_id 5626 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 1 -- 172.21.15.67:0/1028678584 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97480325a0 con 0x7f974c098260 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f9753a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f9753a80640 10 monclient: get_monmap_and_config success 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f9753a80640 4 set_mon_vals no callback set 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 1 -- 172.21.15.67:0/1028678584 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748032940 con 0x7f974c098260 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.408+0000 7f975081a640 10 monclient: dump: 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 10 monclient: shutdown 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 -- 172.21.15.67:0/1028678584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098260 msgr2=0x7f974c098660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 --2- 172.21.15.67:0/1028678584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098260 0x7f974c098660 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f974c0a2e00 tx=0x7f97480091d0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 -- 172.21.15.67:0/1028678584 shutdown_connections 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 --2- 172.21.15.67:0/1028678584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098260 0x7f974c098660 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 --2- 172.21.15.67:0/1028678584 >> v2:172.21.15.17:3300/0 conn(0x7f974c131f90 0x7f974c132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 -- 172.21.15.67:0/1028678584 >> 172.21.15.67:0/1028678584 conn(0x7f974c071690 msgr2=0x7f974c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 -- 172.21.15.67:0/1028678584 shutdown_connections 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 1 -- 172.21.15.67:0/1028678584 wait complete. 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 10 monclient: build_initial_monmap 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.409+0000 7f9753a80640 10 monclient: monmap: 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:55:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 Processor -- start 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 -- start start 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient: init 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient: _reopen_session rank -1 2024-09-15T01:55:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f974c098260 0x7f974c12f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-a con 0x7f974c098260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 0x7f974c11c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-b con 0x7f974c098ba0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974c131f90 0x7f974c11c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): picked mon.noname-c con 0x7f974c131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c11fee0 con 0x7f974c098260 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0a0d80 con 0x7f974c098ba0 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c130f30 con 0x7f974c131f90 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): _renew_subs 2024-09-15T01:55:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f9753a80640 10 monclient(hunting): authenticate will time out at 991.879713s 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975181c640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f974c098260 msgr2=0x7f974c12f940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 0x7f974c11c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975181c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f974c098260 0x7f974c12f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 0x7f974c11c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58496/0 (socket says 172.21.15.67:58496) 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 1 -- 172.21.15.67:0/258608454 learned_addr learned my addr 172.21.15.67:0/258608454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): get_auth_request con 0x7f974c098ba0 auth_method 0 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:55:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.410+0000 7f975101b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975201d640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974c131f90 0x7f974c11c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 10 monclient(hunting): handle_auth_done global_id 5629 payload 824 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 -- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974c131f90 msgr2=0x7f974c11c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974c131f90 0x7f974c11c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 -- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f974c098260 msgr2=0x7f974c12f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f974c098260 0x7f974c12f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 10 monclient: _finish_hunting 0 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 monclient: found mon.noname-b 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975101b640 1 -- 172.21.15.67:0/258608454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748009450 con 0x7f974c098ba0 2024-09-15T01:55:59.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975201d640 10 monclient: get_auth_request con 0x7f974c131f90 auth_method 0 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.411+0000 7f975201d640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974c131f90 0x7f974c11c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f975101b640 1 -- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 msgr2=0x7f974c11c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f975101b640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 0x7f974c11c210 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9748009210 tx=0x7f9748032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f975101b640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974c098ba0 0x7f974c11c210 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9748009210 tx=0x7f9748032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f97480313a0 mon_map magic: 0 v1 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974803ca40 con 0x7f974c098ba0 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 10 monclient: dump: 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 10 monclient: _finish_auth 0 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 5 monclient: authenticate success, global_id 5629 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974803cba0 con 0x7f974c098ba0 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 10 monclient: _renew_subs 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 1 -- 172.21.15.67:0/258608454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c120570 con 0x7f974c098ba0 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:55:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f973ffff640 4 set_mon_vals no callback set 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 10 monclient: _renew_subs 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.413+0000 7f9753a80640 1 -- 172.21.15.67:0/258608454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c12edb0 con 0x7f974c098ba0 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97480313a0 con 0x7f974c098ba0 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 10 monclient: dump: 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:55:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9748039040 con 0x7f974c098ba0 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f973dffb640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97300386f0 0x7f973003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f9753a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f9753a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f9753a80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.414+0000 7f9753a80640 1 -- 172.21.15.67:0/258608454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c11e7c0 con 0x7f974c098ba0 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.415+0000 7f975181c640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97300386f0 0x7f973003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:55:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.415+0000 7f975181c640 10 monclient: get_auth_request con 0x7f97300386f0 auth_method 0 2024-09-15T01:55:59.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.417+0000 7f975181c640 1 --2- 172.21.15.67:0/258608454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97300386f0 0x7f973003aba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f97400038f0 tx=0x7f9740006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:55:59.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:55:59.419+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6625+0+0 (secure 0 0 0) 0x7f974806c760 con 0x7f974c098ba0 2024-09-15T01:56:00.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.036+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9748040080 con 0x7f974c098ba0 2024-09-15T01:56:00.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.036+0000 7f973dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.036+0000 7f973dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:00.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f9753a80640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T01:56:00.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f9753a80640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T01:56:00.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f9753a80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f9753a80640 1 -- 172.21.15.67:0/258608454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7f974c0639e0 con 0x7f974c098ba0 2024-09-15T01:56:00.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f973dffb640 1 -- 172.21.15.67:0/258608454 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v113) v1 ==== 154+0+0 (secure 0 0 0) 0x7f9748049020 con 0x7f974c098ba0 2024-09-15T01:56:00.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f973dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T01:56:00.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.211+0000 7f973dffb640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-15T01:56:00.215 INFO:tasks.workunit.client.0.smithi067.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-15T01:56:00.217 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:56:00.218 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:56:00.218 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.12 2024-09-15T01:56:00.218 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 155 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.289+0000 7f0b0bd00640 10 monclient: get_monmap_and_config 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.289+0000 7f0b0bd00640 10 monclient: build_initial_monmap 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.289+0000 7f0b0bd00640 10 monclient: monmap: 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:00.290428+0000 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:00.290428+0000 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:00.292 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.289+0000 7f0b0bd00640 1 Processor -- start 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 -- start start 2024-09-15T01:56:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient: init 2024-09-15T01:56:00.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:00.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T01:56:00.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b04098370 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:00.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-b con 0x7f0b04098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b04136ea0 0x7f0b04139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-c con 0x7f0b04136ea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b040a17b0 con 0x7f0b04098370 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b04132a70 con 0x7f0b04136ea0 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0409feb0 con 0x7f0b04098740 2024-09-15T01:56:00.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0bd00640 10 monclient(hunting): authenticate will time out at 992.759562s 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b09a9c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0b04098370 0x7f0b04132360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55502/0 (socket says 172.21.15.67:55502) 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b09a9c640 1 -- 172.21.15.67:0/1724454960 learned_addr learned my addr 172.21.15.67:0/1724454960 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/1724454960 >> v2:172.21.15.67:3300/0 conn(0x7f0b04136ea0 0x7f0b04139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): get_auth_request con 0x7f0b04136ea0 auth_method 0 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:00.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0929b640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0929b640 10 monclient(hunting): get_auth_request con 0x7f0b04098740 auth_method 0 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0929b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0929b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0929b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_done global_id 5632 payload 274 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 1 -- 172.21.15.67:0/1724454960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 msgr2=0x7f0b04098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 1 -- 172.21.15.67:0/1724454960 >> v1:172.21.15.67:6789/0 conn(0x7f0b04098370 legacy=0x7f0b04132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.290+0000 7f0b0a29d640 1 monclient: found mon.noname-c 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.291+0000 7f0b0a29d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:00.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.291+0000 7f0b0a29d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.291+0000 7f0b0a29d640 1 -- 172.21.15.67:0/1724454960 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b040a00f0 con 0x7f0b04136ea0 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.293+0000 7f0b0a29d640 1 -- 172.21.15.67:0/1724454960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 msgr2=0x7f0b04139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.293+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04139280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0b0409fe40 tx=0x7f0afc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f65f53140069de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 1 -- 172.21.15.67:0/1724454960 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc007760 con 0x7f0b04136ea0 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: dump: 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.305 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: _finish_auth 0 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 1 -- 172.21.15.67:0/1724454960 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc02d050 con 0x7f0b04136ea0 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 5 monclient: authenticate success, global_id 5632 2024-09-15T01:56:00.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 1 -- 172.21.15.67:0/1724454960 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc007af0 con 0x7f0b04136ea0 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 4 set_mon_vals no callback set 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b08a9a640 10 monclient: dump: 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.307 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 10 monclient: shutdown 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 -- 172.21.15.67:0/1724454960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 msgr2=0x7f0b04139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04139280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0b0409fe40 tx=0x7f0afc009780 comp rx=0 tx=0).stop 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 -- 172.21.15.67:0/1724454960 shutdown_connections 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04139280 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/1724454960 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 -- 172.21.15.67:0/1724454960 >> 172.21.15.67:0/1724454960 conn(0x7f0b040a59a0 msgr2=0x7f0b040a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 -- 172.21.15.67:0/1724454960 shutdown_connections 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.294+0000 7f0b0bd00640 1 -- 172.21.15.67:0/1724454960 wait complete. 2024-09-15T01:56:00.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.295+0000 7f0b0bd00640 10 monclient: build_initial_monmap 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.295+0000 7f0b0bd00640 10 monclient: monmap: 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.309 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.295+0000 7f0b0bd00640 1 Processor -- start 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.295+0000 7f0b0bd00640 1 -- start start 2024-09-15T01:56:00.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.295+0000 7f0b0bd00640 10 monclient: init 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 0x7f0b04134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-a con 0x7f0b04098370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04135020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-c con 0x7f0b04098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): picked mon.noname-b con 0x7f0b04136ea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0411c920 con 0x7f0b04098370 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04069630 con 0x7f0b04136ea0 2024-09-15T01:56:00.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04136c30 con 0x7f0b04098740 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0bd00640 10 monclient(hunting): authenticate will time out at 992.765120s 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b09a9c640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 msgr2=0x7f0b04134ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b09a9c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 0x7f0b04134ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45740/0 (socket says 172.21.15.67:45740) 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 -- 172.21.15.67:0/3483586679 learned_addr learned my addr 172.21.15.67:0/3483586679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): get_auth_request con 0x7f0b04136ea0 auth_method 0 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04135020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): get_auth_request con 0x7f0b04098740 auth_method 0 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0929b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient(hunting): handle_auth_done global_id 5635 payload 824 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 -- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 msgr2=0x7f0b04135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04135020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 -- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 msgr2=0x7f0b04134ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 0x7f0b04134ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 monclient: found mon.noname-b 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.296+0000 7f0b0a29d640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc009a20 con 0x7f0b04136ea0 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0b0a29d640 1 -- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 msgr2=0x7f0b04135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0b0a29d640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0099f0 tx=0x7f0afc037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc031c10 con 0x7f0b04136ea0 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 10 monclient: dump: 2024-09-15T01:56:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc040070 con 0x7f0b04136ea0 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.298+0000 7f0b0bd00640 5 monclient: authenticate success, global_id 5635 2024-09-15T01:56:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 10 monclient: _renew_subs 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0411d1c0 con 0x7f0b04136ea0 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0af9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc045430 con 0x7f0b04136ea0 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0af9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0af9ffb640 10 monclient: dump: 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 10 monclient: _renew_subs 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0afbfff640 4 set_mon_vals no callback set 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.299+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0411bb40 con 0x7f0b04136ea0 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b0bd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b0bd00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b0bd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0afc003020 con 0x7f0b04136ea0 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0404bf60 con 0x7f0b04136ea0 2024-09-15T01:56:00.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0af9ffb640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 0x7f0ae803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6625+0+0 (secure 0 0 0) 0x7f0afc06dec0 con 0x7f0b04136ea0 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b09a9c640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 0x7f0ae803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.300+0000 7f0b09a9c640 10 monclient: get_auth_request con 0x7f0ae8038770 auth_method 0 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.302+0000 7f0b09a9c640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 0x7f0ae803ac20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b000047d0 tx=0x7f0b00007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.304+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0afc04e430 con 0x7f0b04136ea0 2024-09-15T01:56:00.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.304+0000 7f0af9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.304+0000 7f0af9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:00.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.469+0000 7f0b0bd00640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:56:00.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.469+0000 7f0b0bd00640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:56:00.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.469+0000 7f0b0bd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.469+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0b0411cfa0 con 0x7f0b04136ea0 2024-09-15T01:56:00.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.470+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f0afc03a3a0 con 0x7f0b04136ea0 2024-09-15T01:56:00.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.470+0000 7f0af9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:56:00.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.470+0000 7f0af9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:00.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0b0bd00640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0b0bd00640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0b0bd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0b040a0820 con 0x7f0b04136ea0 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0af9ffb640 1 -- 172.21.15.67:0/3483586679 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f0afc03a580 con 0x7f0b04136ea0 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0af9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:56:00.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.568+0000 7f0af9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:00.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 msgr2=0x7f0ae803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 0x7f0ae803ac20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b000047d0 tx=0x7f0b00007b80 comp rx=0 tx=0).stop 2024-09-15T01:56:00.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 10 monclient: shutdown 2024-09-15T01:56:00.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 msgr2=0x7f0b04135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0099f0 tx=0x7f0afc037000 comp rx=0 tx=0).stop 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 shutdown_connections 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ae8038770 0x7f0ae803ac20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b04136ea0 0x7f0b04135560 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b04098740 0x7f0b04135020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 --2- 172.21.15.67:0/3483586679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b04098370 0x7f0b04134ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 >> 172.21.15.67:0/3483586679 conn(0x7f0b040a59a0 msgr2=0x7f0b04097130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 shutdown_connections 2024-09-15T01:56:00.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.570+0000 7f0b0bd00640 1 -- 172.21.15.67:0/3483586679 wait complete. 2024-09-15T01:56:00.581 INFO:tasks.workunit.client.0.smithi067.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-15T01:56:00.582 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd tree 2024-09-15T01:56:00.582 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep osd.14 2024-09-15T01:56:00.582 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 155 2024-09-15T01:56:00.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.653+0000 7fe959f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:00.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.653+0000 7fe959f5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.653+0000 7fe959f5d640 10 monclient: monmap: 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:00.654568+0000 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:00.654568+0000 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:00.656 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:00.657 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:00.657 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 Processor -- start 2024-09-15T01:56:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 -- start start 2024-09-15T01:56:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient: init 2024-09-15T01:56:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9540a36e0 0x7fe95406a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe9540a36e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 0x7fe95406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe95406bdc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe95409d9c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95409f640 con 0x7fe95409d9c0 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe954099fa0 con 0x7fe95406bdc0 2024-09-15T01:56:00.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe954069f00 con 0x7fe9540a36e0 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe959f5d640 10 monclient(hunting): authenticate will time out at 993.123777s 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe952ffd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 msgr2=0x7fe95406e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe9537fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9540a36e0 0x7fe95406a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe952ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 0x7fe95406e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe953fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe95409d9c0 0x7fe95409de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55514/0 (socket says 172.21.15.67:55514) 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe953fff640 1 -- 172.21.15.67:0/880624951 learned_addr learned my addr 172.21.15.67:0/880624951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe9537fe640 10 monclient(hunting): get_auth_request con 0x7fe9540a36e0 auth_method 0 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe9537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe9537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.654+0000 7fe9537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient(hunting): handle_auth_done global_id 5638 payload 274 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 1 -- 172.21.15.67:0/880624951 >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 msgr2=0x7fe95406e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 1 --2- 172.21.15.67:0/880624951 >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 0x7fe95406e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 1 -- 172.21.15.67:0/880624951 >> v1:172.21.15.67:6789/0 conn(0x7fe95409d9c0 legacy=0x7fe95409de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 1 monclient: found mon.noname-c 2024-09-15T01:56:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:00.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.655+0000 7fe9537fe640 1 -- 172.21.15.67:0/880624951 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954069c20 con 0x7fe9540a36e0 2024-09-15T01:56:00.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9537fe640 1 -- 172.21.15.67:0/880624951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9540a36e0 msgr2=0x7fe95406a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9537fe640 1 --2- 172.21.15.67:0/880624951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9540a36e0 0x7fe95406a600 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe95409eae0 tx=0x7fe9440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a050cef8b147aef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 1 -- 172.21.15.67:0/880624951 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944032440 con 0x7fe9540a36e0 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: dump: 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 1 -- 172.21.15.67:0/880624951 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9440325a0 con 0x7fe9540a36e0 2024-09-15T01:56:00.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 5 monclient: authenticate success, global_id 5638 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 1 -- 172.21.15.67:0/880624951 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944032940 con 0x7fe9540a36e0 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 4 set_mon_vals no callback set 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe9527fc640 10 monclient: dump: 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 10 monclient: shutdown 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 1 -- 172.21.15.67:0/880624951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9540a36e0 msgr2=0x7fe95406a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.658+0000 7fe959f5d640 1 --2- 172.21.15.67:0/880624951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9540a36e0 0x7fe95406a600 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe95409eae0 tx=0x7fe9440090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 -- 172.21.15.67:0/880624951 shutdown_connections 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 --2- 172.21.15.67:0/880624951 >> v2:172.21.15.17:3300/0 conn(0x7fe95406bdc0 0x7fe95406e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 --2- 172.21.15.67:0/880624951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9540a36e0 0x7fe95406a600 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 -- 172.21.15.67:0/880624951 >> 172.21.15.67:0/880624951 conn(0x7fe954071020 msgr2=0x7fe9540a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 -- 172.21.15.67:0/880624951 shutdown_connections 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 1 -- 172.21.15.67:0/880624951 wait complete. 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.659+0000 7fe959f5d640 10 monclient: monmap: 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:00.674 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 Processor -- start 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 -- start start 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient: init 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:00.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe95406bdc0 0x7fe954131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe95406bdc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe95409d9c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 0x7fe95409b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe9540a36e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95406ea60 con 0x7fe9540a36e0 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540a27d0 con 0x7fe95409d9c0 2024-09-15T01:56:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95411f110 con 0x7fe95406bdc0 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe959f5d640 10 monclient(hunting): authenticate will time out at 993.129607s 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45744/0 (socket says 172.21.15.67:45744) 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 -- 172.21.15.67:0/118785356 learned_addr learned my addr 172.21.15.67:0/118785356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): get_auth_request con 0x7fe95409d9c0 auth_method 0 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe953fff640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 msgr2=0x7fe95409b7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe953fff640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 0x7fe95409b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient(hunting): handle_auth_done global_id 5641 payload 824 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe95406bdc0 msgr2=0x7fe954131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe95406bdc0 0x7fe954131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 msgr2=0x7fe95409b7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 0x7fe95409b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 monclient: found mon.noname-b 2024-09-15T01:56:00.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.660+0000 7fe952ffd640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944009390 con 0x7fe95409d9c0 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe952ffd640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 msgr2=0x7fe95409b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe952ffd640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00b600 tx=0x7fe93c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c015070 con 0x7fe95409d9c0 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe952ffd640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00b600 tx=0x7fe93c0091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fe93c006380 mon_map magic: 0 v1 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 10 monclient: dump: 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.679 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 5 monclient: authenticate success, global_id 5641 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 10 monclient: _renew_subs 2024-09-15T01:56:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c003d90 con 0x7fe95409d9c0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe954141fc0 con 0x7fe95409d9c0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe94b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe951ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 10 monclient: _renew_subs 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.663+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95411dec0 con 0x7fe95409d9c0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c006380 con 0x7fe95409d9c0 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 10 monclient: dump: 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:00.682 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe93c010440 con 0x7fe95409d9c0 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe959f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe959f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe959f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95404bf60 con 0x7fe95409d9c0 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe94b7fe640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 0x7fe93403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe9537fe640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 0x7fe93403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.664+0000 7fe9537fe640 10 monclient: get_auth_request con 0x7fe9340386f0 auth_method 0 2024-09-15T01:56:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.666+0000 7fe9537fe640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 0x7fe93403aba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe95409f270 tx=0x7fe944032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:00.684 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:00.668+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fe93c04bee0 con 0x7fe95409d9c0 2024-09-15T01:56:01.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.286+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe93c018d60 con 0x7fe95409d9c0 2024-09-15T01:56:01.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.286+0000 7fe94b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:01.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.286+0000 7fe94b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:01.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe959f5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe959f5d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe959f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe954069940 con 0x7fe95409d9c0 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fe93c04d020 con 0x7fe95409d9c0 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe94b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T01:56:01.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.454+0000 7fe94b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:01.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe959f5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T01:56:01.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe959f5d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T01:56:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe959f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe95411f250 con 0x7fe95409d9c0 2024-09-15T01:56:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe94b7fe640 1 -- 172.21.15.67:0/118785356 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fe93c04d5c0 con 0x7fe95409d9c0 2024-09-15T01:56:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe94b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T01:56:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.552+0000 7fe94b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:01.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 msgr2=0x7fe93403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:01.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 0x7fe93403aba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe95409f270 tx=0x7fe944032000 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 10 monclient: shutdown 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 msgr2=0x7fe95409b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00b600 tx=0x7fe93c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 shutdown_connections 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9340386f0 0x7fe93403aba0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9540a36e0 0x7fe95409b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe95409d9c0 0x7fe95409b290 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 --2- 172.21.15.67:0/118785356 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe95406bdc0 0x7fe954131c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 >> 172.21.15.67:0/118785356 conn(0x7fe954071020 msgr2=0x7fe954069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 shutdown_connections 2024-09-15T01:56:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.554+0000 7fe959f5d640 1 -- 172.21.15.67:0/118785356 wait complete. 2024-09-15T01:56:01.566 INFO:tasks.workunit.client.0.smithi067.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-15T01:56:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.12 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 10 monclient: build_initial_monmap 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 10 monclient: monmap: 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:01.641 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:01.640254+0000 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:01.640254+0000 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:01.642 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 1 Processor -- start 2024-09-15T01:56:01.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 1 -- start start 2024-09-15T01:56:01.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.639+0000 7f93f380f640 10 monclient: init 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 0x7f93ec132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-a con 0x7f93ec068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93ec069190 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93ec137150 0x7f93ec139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-c con 0x7f93ec137150 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec096c10 con 0x7f93ec069190 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec06a2a0 con 0x7f93ec068dc0 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:01.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0a14d0 con 0x7f93ec137150 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f380f640 10 monclient(hunting): authenticate will time out at 994.109457s 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f0daa640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f93ec069190 legacy=0x7f93ec069590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f15ab640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 msgr2=0x7f93ec132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93ec137150 0x7f93ec139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f15ab640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 0x7f93ec132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93ec137150 0x7f93ec139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45746/0 (socket says 172.21.15.67:45746) 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 -- 172.21.15.67:0/1078358240 learned_addr learned my addr 172.21.15.67:0/1078358240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): get_auth_request con 0x7f93ec137150 auth_method 0 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:01.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient(hunting): handle_auth_done global_id 5644 payload 274 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 -- 172.21.15.67:0/1078358240 >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 msgr2=0x7f93ec132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 --2- 172.21.15.67:0/1078358240 >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 0x7f93ec132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 -- 172.21.15.67:0/1078358240 >> v1:172.21.15.17:6789/0 conn(0x7f93ec069190 legacy=0x7f93ec069590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient: _finish_hunting 0 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 monclient: found mon.noname-c 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.640+0000 7f93f1dac640 1 -- 172.21.15.67:0/1078358240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec09ff20 con 0x7f93ec137150 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93f1dac640 1 -- 172.21.15.67:0/1078358240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec137150 msgr2=0x7f93ec139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93f1dac640 1 --2- 172.21.15.67:0/1078358240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec137150 0x7f93ec139530 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f93ec096f10 tx=0x7f93dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f79ac965c82ed4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 1 -- 172.21.15.67:0/1078358240 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc02aea0 con 0x7f93ec137150 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:01.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: dump: 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:01.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 1 -- 172.21.15.67:0/1078358240 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc02d050 con 0x7f93ec137150 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 1 -- 172.21.15.67:0/1078358240 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc007840 con 0x7f93ec137150 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93f380f640 5 monclient: authenticate success, global_id 5644 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93f380f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93f380f640 4 set_mon_vals no callback set 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.643+0000 7f93ebfff640 10 monclient: dump: 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.655 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 10 monclient: shutdown 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 -- 172.21.15.67:0/1078358240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec137150 msgr2=0x7f93ec139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 --2- 172.21.15.67:0/1078358240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec137150 0x7f93ec139530 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f93ec096f10 tx=0x7f93dc009840 comp rx=0 tx=0).stop 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 -- 172.21.15.67:0/1078358240 shutdown_connections 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 --2- 172.21.15.67:0/1078358240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec137150 0x7f93ec139530 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 --2- 172.21.15.67:0/1078358240 >> v2:172.21.15.17:3300/0 conn(0x7f93ec068dc0 0x7f93ec132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 -- 172.21.15.67:0/1078358240 >> 172.21.15.67:0/1078358240 conn(0x7f93ec0711b0 msgr2=0x7f93ec0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 -- 172.21.15.67:0/1078358240 shutdown_connections 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.644+0000 7f93f380f640 1 -- 172.21.15.67:0/1078358240 wait complete. 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 10 monclient: build_initial_monmap 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 10 monclient: monmap: 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:01.657 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 1 Processor -- start 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 1 -- start start 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 10 monclient: init 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.645+0000 7f93f380f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 0x7f93ec097ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-a con 0x7f93ec068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-b con 0x7f93ec069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f93ec137150 0x7f93ec098940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): picked mon.noname-c con 0x7f93ec137150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:01.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0a0890 con 0x7f93ec068dc0 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec131470 con 0x7f93ec069190 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec09b8c0 con 0x7f93ec137150 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f380f640 10 monclient(hunting): authenticate will time out at 994.115116s 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f15ab640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 msgr2=0x7f93ec097ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f15ab640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 0x7f93ec097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45762/0 (socket says 172.21.15.67:45762) 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 -- 172.21.15.67:0/3337189120 learned_addr learned my addr 172.21.15.67:0/3337189120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): get_auth_request con 0x7f93ec069190 auth_method 0 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f1dac640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f93ec137150 0x7f93ec098940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f1dac640 10 monclient(hunting): get_auth_request con 0x7f93ec137150 auth_method 0 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f1dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f1dac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f1dac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient(hunting): handle_auth_done global_id 5647 payload 824 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 -- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f93ec137150 msgr2=0x7f93ec098940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f93ec137150 0x7f93ec098940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 -- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 msgr2=0x7f93ec097ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 0x7f93ec097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 monclient: found mon.noname-b 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.646+0000 7f93f0daa640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc009ae0 con 0x7f93ec069190 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f0daa640 1 -- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 msgr2=0x7f93ec098400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f0daa640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b630 tx=0x7f93e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0015070 con 0x7f93ec069190 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: dump: 2024-09-15T01:56:01.661 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:01.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0003d90 con 0x7f93ec069190 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0006380 con 0x7f93ec069190 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 5 monclient: authenticate success, global_id 5647 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 10 monclient: _renew_subs 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec09b050 con 0x7f93ec069190 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93eb7fe640 4 set_mon_vals no callback set 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93e97fa640 10 monclient: dump: 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:01.663 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 10 monclient: _renew_subs 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.648+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec0997d0 con 0x7f93ec069190 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f93e0010440 con 0x7f93ec069190 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93f380f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:01.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93f380f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93f380f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.649+0000 7f93e97fa640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 0x7f93d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.650+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6625+0+0 (secure 0 0 0) 0x7f93e004bcd0 con 0x7f93ec069190 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.650+0000 7f93f15ab640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 0x7f93d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.650+0000 7f93f15ab640 10 monclient: get_auth_request con 0x7f93d80386f0 auth_method 0 2024-09-15T01:56:01.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.652+0000 7f93f15ab640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 0x7f93d803aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93e4006fd0 tx=0x7f93e4006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:01.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.654+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:01.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.654+0000 7f93e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:01.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.654+0000 7f93e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:01.823 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.821+0000 7f93f380f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:01.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.821+0000 7f93f380f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:01.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.821+0000 7f93f380f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:01.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:01.821+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.315+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v114) v1 ==== 125+0+0 (secure 0 0 0) 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.315+0000 7f93e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.315+0000 7f93e97fa640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-15T01:56:02.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.421+0000 7f93f380f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:02.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.421+0000 7f93f380f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:02.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.421+0000 7f93f380f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.421+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:02.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.422+0000 7f93e97fa640 1 -- 172.21.15.67:0/3337189120 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v114) v1 ==== 126+0+0 (secure 0 0 0) 0x7f93ec099d40 con 0x7f93ec069190 2024-09-15T01:56:02.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.422+0000 7f93e97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T01:56:02.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.422+0000 7f93e97fa640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-15T01:56:02.425 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.12' does not appear in the crush map 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 msgr2=0x7f93d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 0x7f93d803aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93e4006fd0 tx=0x7f93e4006d30 comp rx=0 tx=0).stop 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 10 monclient: shutdown 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 msgr2=0x7f93ec098400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b630 tx=0x7f93e00091a0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 shutdown_connections 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f93d80386f0 0x7f93d803aba0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f93ec137150 0x7f93ec098940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec069190 0x7f93ec098400 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 --2- 172.21.15.67:0/3337189120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93ec068dc0 0x7f93ec097ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 >> 172.21.15.67:0/3337189120 conn(0x7f93ec0711b0 msgr2=0x7f93ec09ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 shutdown_connections 2024-09-15T01:56:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.424+0000 7f93f380f640 1 -- 172.21.15.67:0/3337189120 wait complete. 2024-09-15T01:56:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm osd.14 2024-09-15T01:56:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 10 monclient: build_initial_monmap 2024-09-15T01:56:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 10 monclient: monmap: 2024-09-15T01:56:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:02.510321+0000 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:02.510321+0000 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:02.512 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 1 Processor -- start 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 1 -- start start 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.509+0000 7f744a79d640 10 monclient: init 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7444098140 0x7f74441321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-a con 0x7f7444098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74440a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7444098c20 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74440a0b80 con 0x7f7444098c20 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74440a00b0 con 0x7f74440a0370 2024-09-15T01:56:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7444071390 con 0x7f7444098140 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f744a79d640 10 monclient(hunting): authenticate will time out at 994.979473s 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f74437fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74440a0370 0x7f7444098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55526/0 (socket says 172.21.15.67:55526) 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f74437fe640 1 -- 172.21.15.67:0/3391201000 learned_addr learned my addr 172.21.15.67:0/3391201000 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f7443fff640 1 -- 172.21.15.67:0/3391201000 >> v2:172.21.15.17:3300/0 conn(0x7f7444098140 msgr2=0x7f74441321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 >> v1:172.21.15.17:6789/0 conn(0x7f7444098c20 legacy=0x7f74441390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:02.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.510+0000 7f7443fff640 1 --2- 172.21.15.67:0/3391201000 >> v2:172.21.15.17:3300/0 conn(0x7f7444098140 0x7f74441321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:02.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f74437fe640 1 -- 172.21.15.67:0/3391201000 >> v1:172.21.15.67:6789/0 conn(0x7f74440a0370 legacy=0x7f7444098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713751351 0 0) 0x7f74440a00b0 con 0x7f74440a0370 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f7442ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f7442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f7442ffd640 10 monclient(hunting): my global_id is 5650 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.512+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74300034e0 con 0x7f74440a0370 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384830163 0 0) 0x7f74300034e0 con 0x7f74440a0370 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 >> v2:172.21.15.17:3300/0 conn(0x7f7444098140 msgr2=0x7f74441321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 --2- 172.21.15.67:0/3391201000 >> v2:172.21.15.17:3300/0 conn(0x7f7444098140 0x7f74441321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 >> v1:172.21.15.17:6789/0 conn(0x7f7444098c20 legacy=0x7f74441390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:56:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440714d0 con 0x7f74440a0370 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: _finish_auth 0 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f742c001720 con 0x7f74440a0370 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: dump: 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 >> v1:172.21.15.67:6789/0 conn(0x7f74440a0370 legacy=0x7f7444098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7430003a50 0x7f7430003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): picked mon.c con 0x7f7430003a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7430004780 0x7f7430004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): picked mon.a con 0x7f7430004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 0x7f743000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): picked mon.b con 0x7f74300099c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444071390 con 0x7f7430004780 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743000c360 con 0x7f74300099c0 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743000c520 con 0x7f7430003a50 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7442ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f74437fe640 1 -- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7430004780 msgr2=0x7f7430004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 0x7f743000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f74437fe640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7430004780 0x7f7430004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): get_auth_request con 0x7f74300099c0 auth_method 0 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7443fff640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7430003a50 0x7f7430003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7443fff640 10 monclient(hunting): get_auth_request con 0x7f7430003a50 auth_method 0 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7443fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7443fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7443fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient(hunting): handle_auth_done global_id 5650 payload 293 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7430003a50 msgr2=0x7f7430003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7430003a50 0x7f7430003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7430004780 msgr2=0x7f7430004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7430004780 0x7f7430004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 monclient: found mon.b 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74300034e0 con 0x7f74300099c0 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 10 monclient: _finish_auth 0 2024-09-15T01:56:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f7448d3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f744a79d640 5 monclient: authenticate success, global_id 5650 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.513+0000 7f744a79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.514+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 reap_dead start 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7448d3a640 1 -- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 msgr2=0x7f743000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7448d3a640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 0x7f743000bda0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7444068d00 tx=0x7f743c0095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4162887fd255fd05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c02c6d0 con 0x7f74300099c0 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: dump: 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f744a79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c02c870 con 0x7f74300099c0 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 1 -- 172.21.15.67:0/3391201000 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c02cc30 con 0x7f74300099c0 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f7442ffd640 10 monclient: dump: 2024-09-15T01:56:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f744a79d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.516+0000 7f744a79d640 4 set_mon_vals no callback set 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 10 monclient: shutdown 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 -- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 msgr2=0x7f743000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 0x7f743000bda0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7444068d00 tx=0x7f743c0095f0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 -- 172.21.15.67:0/3391201000 shutdown_connections 2024-09-15T01:56:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 --2- 172.21.15.67:0/3391201000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300099c0 0x7f743000bda0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 -- 172.21.15.67:0/3391201000 >> 172.21.15.67:0/3391201000 conn(0x7f74440a5780 msgr2=0x7f74440a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 -- 172.21.15.67:0/3391201000 shutdown_connections 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 1 -- 172.21.15.67:0/3391201000 wait complete. 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 10 monclient: build_initial_monmap 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.517+0000 7f744a79d640 10 monclient: monmap: 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 Processor -- start 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 -- start start 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient: init 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 0x7f7444140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-a con 0x7f74300099c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-b con 0x7f7444120070 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:02.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74441313c0 0x7f744411d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): picked mon.noname-c con 0x7f74441313c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444099120 con 0x7f74300099c0 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74441212b0 con 0x7f7444120070 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744411f510 con 0x7f74441313c0 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f744a79d640 10 monclient(hunting): authenticate will time out at 994.987908s 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f7443fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 msgr2=0x7f7444140fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.518+0000 7f7443fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 0x7f7444140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45790/0 (socket says 172.21.15.67:45790) 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 -- 172.21.15.67:0/3769069758 learned_addr learned my addr 172.21.15.67:0/3769069758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:02.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): get_auth_request con 0x7f7444120070 auth_method 0 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f7448d3a640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74441313c0 0x7f744411d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f7448d3a640 10 monclient(hunting): get_auth_request con 0x7f74441313c0 auth_method 0 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f7448d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f7448d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f7448d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient(hunting): handle_auth_done global_id 5653 payload 824 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 -- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74441313c0 msgr2=0x7f744411d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:02.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74441313c0 0x7f744411d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 -- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 msgr2=0x7f7444140fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 0x7f7444140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 monclient: found mon.noname-b 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.519+0000 7f74437fe640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7444120fd0 con 0x7f7444120070 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f74437fe640 1 -- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 msgr2=0x7f7444130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f74437fe640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c810 tx=0x7f742c001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c007360 con 0x7f7444120070 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:02.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: dump: 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:02.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c007500 con 0x7f7444120070 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c0047a0 con 0x7f7444120070 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f743bfff640 10 monclient: dump: 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:02.539 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 5 monclient: authenticate success, global_id 5653 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 10 monclient: _renew_subs 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7444121590 con 0x7f7444120070 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f74427fc640 4 set_mon_vals no callback set 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 10 monclient: _renew_subs 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.521+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7444141500 con 0x7f7444120070 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f742c004b50 con 0x7f7444120070 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f744a79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f744a79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f744a79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744404bf60 con 0x7f7444120070 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.522+0000 7f743bfff640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 0x7f742403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.523+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6603+0+0 (secure 0 0 0) 0x7f742c04b940 con 0x7f7444120070 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.523+0000 7f7443fff640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 0x7f742403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.523+0000 7f7443fff640 10 monclient: get_auth_request con 0x7f74240386f0 auth_method 0 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.525+0000 7f7443fff640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 0x7f742403aba0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7434003800 tx=0x7f7434008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.527+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f742c004e00 con 0x7f7444120070 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.527+0000 7f743bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:02.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.527+0000 7f743bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:02.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.697+0000 7f744a79d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:02.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.697+0000 7f744a79d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:02.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.697+0000 7f744a79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:02.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:02.697+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7f7444098510 con 0x7f7444120070 2024-09-15T01:56:03.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.319+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v115) v1 ==== 125+0+0 (secure 0 0 0) 0x7f742c018020 con 0x7f7444120070 2024-09-15T01:56:03.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.319+0000 7f743bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:03.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.319+0000 7f743bfff640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-15T01:56:03.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.430+0000 7f744a79d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:03.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.430+0000 7f744a79d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:03.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.430+0000 7f744a79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.430+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7f744409f7f0 con 0x7f7444120070 2024-09-15T01:56:03.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.431+0000 7f743bfff640 1 -- 172.21.15.67:0/3769069758 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v115) v1 ==== 126+0+0 (secure 0 0 0) 0x7f742c010af0 con 0x7f7444120070 2024-09-15T01:56:03.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.431+0000 7f743bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T01:56:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.431+0000 7f743bfff640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-15T01:56:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:device 'osd.14' does not appear in the crush map 2024-09-15T01:56:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 msgr2=0x7f742403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 0x7f742403aba0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7434003800 tx=0x7f7434008040 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 10 monclient: shutdown 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 msgr2=0x7f7444130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c810 tx=0x7f742c001150 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 shutdown_connections 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f74240386f0 0x7f742403aba0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f74441313c0 0x7f744411d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7444120070 0x7f7444130e80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 --2- 172.21.15.67:0/3769069758 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f74300099c0 0x7f7444140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.432+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 >> 172.21.15.67:0/3769069758 conn(0x7f74440a5780 msgr2=0x7f74440a4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.433+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 shutdown_connections 2024-09-15T01:56:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.433+0000 7f744a79d640 1 -- 172.21.15.67:0/3769069758 wait complete. 2024-09-15T01:56:03.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm osd.12 2024-09-15T01:56:03.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.518+0000 7fe189f09640 10 monclient: get_monmap_and_config 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.518+0000 7fe189f09640 10 monclient: build_initial_monmap 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.518+0000 7fe189f09640 10 monclient: monmap: 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:03.519579+0000 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:03.519579+0000 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:03.521 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:03.522 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:03.522 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:03.522 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 Processor -- start 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 -- start start 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient: init 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 0x7fe18406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-b con 0x7fe1840a3800 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:03.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe18406be80 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe18406a670 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1840a3bd0 con 0x7fe18406be80 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18409e850 con 0x7fe18406a670 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18409a030 con 0x7fe1840a3800 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe189f09640 10 monclient(hunting): authenticate will time out at 995.988708s 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe183fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe18406a670 0x7fe18409e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55532/0 (socket says 172.21.15.67:55532) 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe183fff640 1 -- 172.21.15.67:0/78080776 learned_addr learned my addr 172.21.15.67:0/78080776 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe182ffd640 1 -- 172.21.15.67:0/78080776 >> v1:172.21.15.17:6789/0 conn(0x7fe18406be80 legacy=0x7fe18409da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.519+0000 7fe1837fe640 1 --2- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 0x7fe18406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): get_auth_request con 0x7fe1840a3800 auth_method 0 2024-09-15T01:56:03.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient(hunting): handle_auth_done global_id 74234 payload 274 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 1 -- 172.21.15.67:0/78080776 >> v1:172.21.15.67:6789/0 conn(0x7fe18406a670 legacy=0x7fe18409e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 1 -- 172.21.15.67:0/78080776 >> v1:172.21.15.17:6789/0 conn(0x7fe18406be80 legacy=0x7fe18409da30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 1 monclient: found mon.noname-b 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:03.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:03.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.520+0000 7fe1837fe640 1 -- 172.21.15.67:0/78080776 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18409f720 con 0x7fe1840a3800 2024-09-15T01:56:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1837fe640 1 -- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 msgr2=0x7fe18406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1837fe640 1 --2- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 0x7fe18406e310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009880 tx=0x7fe16c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a6c0374c936129 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 1 -- 172.21.15.67:0/78080776 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c032440 con 0x7fe1840a3800 2024-09-15T01:56:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: dump: 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 1 -- 172.21.15.67:0/78080776 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c0325a0 con 0x7fe1840a3800 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 1 -- 172.21.15.67:0/78080776 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c032940 con 0x7fe1840a3800 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe189f09640 5 monclient: authenticate success, global_id 74234 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe189f09640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe189f09640 4 set_mon_vals no callback set 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.523+0000 7fe1827fc640 10 monclient: dump: 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 10 monclient: shutdown 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 -- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 msgr2=0x7fe18406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 --2- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 0x7fe18406e310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009880 tx=0x7fe16c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 -- 172.21.15.67:0/78080776 shutdown_connections 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 --2- 172.21.15.67:0/78080776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1840a3800 0x7fe18406e310 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 -- 172.21.15.67:0/78080776 >> 172.21.15.67:0/78080776 conn(0x7fe184071090 msgr2=0x7fe1840a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 -- 172.21.15.67:0/78080776 shutdown_connections 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 1 -- 172.21.15.67:0/78080776 wait complete. 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 10 monclient: build_initial_monmap 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.524+0000 7fe189f09640 10 monclient: monmap: 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:03.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 Processor -- start 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 -- start start 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient: init 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 0x7fe1841414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-a con 0x7fe18406a670 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 0x7fe184132020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-c con 0x7fe18406be80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): picked mon.noname-b con 0x7fe1840a3800 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:03.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe184099810 con 0x7fe18406a670 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840a2260 con 0x7fe1840a3800 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18411f140 con 0x7fe18406be80 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe189f09640 10 monclient(hunting): authenticate will time out at 995.994939s 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.525+0000 7fe1837fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 msgr2=0x7fe1841414e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe1837fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 0x7fe1841414e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45816/0 (socket says 172.21.15.67:45816) 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 -- 172.21.15.67:0/180064425 learned_addr learned my addr 172.21.15.67:0/180064425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): get_auth_request con 0x7fe1840a3800 auth_method 0 2024-09-15T01:56:03.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 0x7fe184132020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 10 monclient(hunting): get_auth_request con 0x7fe18406be80 auth_method 0 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient(hunting): handle_auth_done global_id 5656 payload 824 2024-09-15T01:56:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 msgr2=0x7fe184132020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 0x7fe184132020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 msgr2=0x7fe1841414e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 0x7fe1841414e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 monclient: found mon.noname-b 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe183fff640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c009390 con 0x7fe1840a3800 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.526+0000 7fe182ffd640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 0x7fe184132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe183fff640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 msgr2=0x7fe184097ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe183fff640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00bbb0 tx=0x7fe17c0097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:03.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c00cc80 con 0x7fe1840a3800 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: dump: 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 5 monclient: authenticate success, global_id 5656 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c00cde0 con 0x7fe1840a3800 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 10 monclient: _renew_subs 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18411f420 con 0x7fe1840a3800 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c014690 con 0x7fe1840a3800 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe17b7fe640 10 monclient: dump: 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:03.543 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe181ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 10 monclient: _renew_subs 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.528+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe184141a20 con 0x7fe1840a3800 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe17c004b20 con 0x7fe1840a3800 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe189f09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe189f09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe189f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.529+0000 7fe17b7fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6581+0+0 (secure 0 0 0) 0x7fe17c04bd30 con 0x7fe1840a3800 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 10 monclient: get_auth_request con 0x7fe1640386f0 auth_method 0 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 msgr2=0x7fe16403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 msgr2=0x7fe16403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.530+0000 7fe1837fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.534+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.534+0000 7fe17b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.534+0000 7fe17b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:03.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.709+0000 7fe189f09640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.709+0000 7fe189f09640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.709+0000 7fe189f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:03.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.709+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:03.712 INFO:tasks.ceph.mon.b.smithi067.stderr:2024-09-15T01:56:03.709+0000 7ff7889f1640 -1 mon.b@1(leader).osd e115 definitely_dead 0 2024-09-15T01:56:03.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.731+0000 7fe1837fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:03.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.731+0000 7fe1837fe640 10 monclient: get_auth_request con 0x7fe1640386f0 auth_method 0 2024-09-15T01:56:03.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:03.733+0000 7fe1837fe640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe1840a3ec0 tx=0x7fe16c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:04.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.321+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v116) v1 ==== 87+0+0 (secure 0 0 0) 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:04.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.321+0000 7fe17b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:04.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.321+0000 7fe17b7fe640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-15T01:56:04.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe189f09640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:04.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe189f09640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:04.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe189f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:04.440 INFO:tasks.ceph.mon.b.smithi067.stderr:2024-09-15T01:56:04.437+0000 7ff7889f1640 -1 mon.b@1(leader).osd e116 definitely_dead 0 2024-09-15T01:56:04.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe17b7fe640 1 -- 172.21.15.67:0/180064425 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v116) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe18411f840 con 0x7fe1840a3800 2024-09-15T01:56:04.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe17b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T01:56:04.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.437+0000 7fe17b7fe640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T01:56:04.441 INFO:tasks.workunit.client.0.smithi067.stderr:osd.12 does not exist. 2024-09-15T01:56:04.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 msgr2=0x7fe16403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:04.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe1840a3ec0 tx=0x7fe16c032000 comp rx=0 tx=0).stop 2024-09-15T01:56:04.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 10 monclient: shutdown 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 msgr2=0x7fe184097ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00bbb0 tx=0x7fe17c0097e0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 shutdown_connections 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe1640386f0 0x7fe16403aba0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1840a3800 0x7fe184097ba0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe18406be80 0x7fe184132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 --2- 172.21.15.67:0/180064425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe18406a670 0x7fe1841414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.439+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 >> 172.21.15.67:0/180064425 conn(0x7fe184071090 msgr2=0x7fe184134e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:04.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.440+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 shutdown_connections 2024-09-15T01:56:04.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.440+0000 7fe189f09640 1 -- 172.21.15.67:0/180064425 wait complete. 2024-09-15T01:56:04.451 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm osd.14 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.524+0000 7fbf6d59d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.524+0000 7fbf6d59d640 10 monclient: build_initial_monmap 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.524+0000 7fbf6d59d640 10 monclient: monmap: 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:04.525896+0000 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:04.525896+0000 2024-09-15T01:56:04.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:04.528 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.525+0000 7fbf6d59d640 1 Processor -- start 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.525+0000 7fbf6d59d640 1 -- start start 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.525+0000 7fbf6d59d640 10 monclient: init 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf68131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf68098420 0x7fbf68098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf68098420 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:04.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf681371a0 0x7fbf68139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-b con 0x7fbf681371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680a13e0 con 0x7fbf68131e70 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf680a0440 con 0x7fbf68098420 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680a01e0 con 0x7fbf681371a0 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf6d59d640 10 monclient(hunting): authenticate will time out at 996.995219s 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf68098420 0x7fbf68098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf66ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf68131e70 0x7fbf68132240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55540/0 (socket says 172.21.15.67:55540) 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf66ffd640 1 -- 172.21.15.67:0/1458508043 learned_addr learned my addr 172.21.15.67:0/1458508043 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): get_auth_request con 0x7fbf68098420 auth_method 0 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:04.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf681371a0 0x7fbf68139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): get_auth_request con 0x7fbf681371a0 auth_method 0 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:04.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient(hunting): handle_auth_done global_id 5659 payload 274 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 -- 172.21.15.67:0/1458508043 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf681371a0 msgr2=0x7fbf68139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf681371a0 0x7fbf68139580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 -- 172.21.15.67:0/1458508043 >> v1:172.21.15.67:6789/0 conn(0x7fbf68131e70 legacy=0x7fbf68132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient: _finish_hunting 0 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 monclient: found mon.noname-c 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.526+0000 7fbf667fc640 1 -- 172.21.15.67:0/1458508043 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680a19b0 con 0x7fbf68098420 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf667fc640 1 -- 172.21.15.67:0/1458508043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf68098420 msgr2=0x7fbf68098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf667fc640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf68098420 0x7fbf68098850 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009850 tx=0x7fbf5c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d22182ce63a2e339 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 1 -- 172.21.15.67:0/1458508043 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c02adf0 con 0x7fbf68098420 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: dump: 2024-09-15T01:56:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 1 -- 172.21.15.67:0/1458508043 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c032440 con 0x7fbf68098420 2024-09-15T01:56:04.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 1 -- 172.21.15.67:0/1458508043 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c0327e0 con 0x7fbf68098420 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf6d59d640 5 monclient: authenticate success, global_id 5659 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf6d59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf6d59d640 4 set_mon_vals no callback set 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.529+0000 7fbf65ffb640 10 monclient: dump: 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:04.545 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 10 monclient: shutdown 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 -- 172.21.15.67:0/1458508043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf68098420 msgr2=0x7fbf68098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf68098420 0x7fbf68098850 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009850 tx=0x7fbf5c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 -- 172.21.15.67:0/1458508043 shutdown_connections 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf681371a0 0x7fbf68139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/1458508043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf68098420 0x7fbf68098850 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 -- 172.21.15.67:0/1458508043 >> 172.21.15.67:0/1458508043 conn(0x7fbf680a5c10 msgr2=0x7fbf680a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 -- 172.21.15.67:0/1458508043 shutdown_connections 2024-09-15T01:56:04.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.530+0000 7fbf6d59d640 1 -- 172.21.15.67:0/1458508043 wait complete. 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 10 monclient: build_initial_monmap 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 10 monclient: monmap: 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 1 Processor -- start 2024-09-15T01:56:04.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 1 -- start start 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 10 monclient: init 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.531+0000 7fbf6d59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf68141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf68098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf68142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-a con 0x7fbf68131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): picked mon.noname-b con 0x7fbf681371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68097da0 con 0x7fbf68131e70 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809b870 con 0x7fbf681371a0 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68133ee0 con 0x7fbf68098420 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf6d59d640 10 monclient(hunting): authenticate will time out at 997.001076s 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf667fc640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 msgr2=0x7fbf68142330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf667fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf68142330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45848/0 (socket says 172.21.15.67:45848) 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf68141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 learned_addr learned my addr 172.21.15.67:0/3910633766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): get_auth_request con 0x7fbf681371a0 auth_method 0 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request con 0x7fbf68098420 auth_method 0 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:04.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient(hunting): handle_auth_done global_id 5662 payload 824 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 msgr2=0x7fbf68141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf68141df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 msgr2=0x7fbf68142330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf68142330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 monclient: found mon.noname-b 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c009390 con 0x7fbf681371a0 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.532+0000 7fbf66ffd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf68141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 msgr2=0x7fbf68099c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:04.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009ec0 tx=0x7fbf6000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000ce10 con 0x7fbf681371a0 2024-09-15T01:56:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: dump: 2024-09-15T01:56:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:04.553 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: _finish_auth 0 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf6d59d640 5 monclient: authenticate success, global_id 5662 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000f040 con 0x7fbf681371a0 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf6d59d640 10 monclient: _renew_subs 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600148e0 con 0x7fbf681371a0 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf6d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf681341c0 con 0x7fbf681371a0 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf4effd640 10 monclient: dump: 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.534+0000 7fbf657fa640 4 set_mon_vals no callback set 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 10 monclient: _renew_subs 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68133ac0 con 0x7fbf681371a0 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbf60014c90 con 0x7fbf681371a0 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf680639e0 con 0x7fbf681371a0 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 msgr2=0x7fbf68099c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.535+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 msgr2=0x7fbf68099c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009ec0 tx=0x7fbf6000b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009ec0 tx=0x7fbf6000b040 comp rx=0 tx=0).stop 2024-09-15T01:56:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 0x7fbf5403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 msgr2=0x7fbf68099c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf681371a0 0x7fbf68099c90 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 0x7fbf5403d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): picked mon.b con 0x7fbf5403b300 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf540400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf66ffd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 0x7fbf5403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): picked mon.a con 0x7fbf68131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf54042a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): picked mon.c con 0x7fbf68098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540433f0 con 0x7fbf68131e70 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf54043530 con 0x7fbf5403b300 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf54043670 con 0x7fbf68098420 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf4effd640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request con 0x7fbf54038740 auth_method 0 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 0x7fbf5403d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf677fe640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 msgr2=0x7fbf540400c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): get_auth_request con 0x7fbf5403b300 auth_method 0 2024-09-15T01:56:04.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf677fe640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf540400c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf66ffd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf54042a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient(hunting): handle_auth_done global_id 5662 payload 293 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 msgr2=0x7fbf54042a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf54042a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 msgr2=0x7fbf540400c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf540400c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient: _finish_hunting 0 2024-09-15T01:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 monclient: found mon.b 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fbf540437b0 con 0x7fbf5403b300 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf680a19b0 con 0x7fbf5403b300 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.536+0000 7fbf667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.538+0000 7fbf66ffd640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 0x7fbf5403abf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009c30 tx=0x7fbf58006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.538+0000 7fbf667fc640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 msgr2=0x7fbf5403d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.538+0000 7fbf667fc640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 0x7fbf5403d6e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c000c70 tx=0x7fbf5c02f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c0325b0 con 0x7fbf5403b300 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 10 monclient: dump: 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:04.561 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:04.562 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:04.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c032730 con 0x7fbf5403b300 2024-09-15T01:56:04.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:04.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.540+0000 7fbf64ff9640 4 set_mon_vals no callback set 2024-09-15T01:56:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.541+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbf5c039050 con 0x7fbf5403b300 2024-09-15T01:56:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.541+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 4 ==== osd_map(116..116 src has 1..116) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fbf5c06a950 con 0x7fbf5403b300 2024-09-15T01:56:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.544+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbf5c03a300 con 0x7fbf5403b300 2024-09-15T01:56:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.544+0000 7fbf4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.544+0000 7fbf4effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.719+0000 7fbf6d59d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.719+0000 7fbf6d59d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.719+0000 7fbf6d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:04.719+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fbf68132fa0 con 0x7fbf5403b300 2024-09-15T01:56:04.722 INFO:tasks.ceph.mon.b.smithi067.stderr:2024-09-15T01:56:04.719+0000 7ff7889f1640 -1 mon.b@1(leader).osd e116 definitely_dead 0 2024-09-15T01:56:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.321+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v117) v1 ==== 87+0+0 (secure 0 0 0) 0x7fbf5c037090 con 0x7fbf5403b300 2024-09-15T01:56:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.321+0000 7fbf4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.321+0000 7fbf4effd640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-15T01:56:05.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf6d59d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:05.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf6d59d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:05.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf6d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fbf68098850 con 0x7fbf5403b300 2024-09-15T01:56:05.437 INFO:tasks.ceph.mon.b.smithi067.stderr:2024-09-15T01:56:05.434+0000 7ff7889f1640 -1 mon.b@1(leader).osd e117 definitely_dead 0 2024-09-15T01:56:05.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf4effd640 1 -- 172.21.15.67:0/3910633766 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v117) v1 ==== 96+0+0 (secure 0 0 0) 0x7fbf5c040020 con 0x7fbf5403b300 2024-09-15T01:56:05.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf4effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T01:56:05.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.434+0000 7fbf4effd640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-15T01:56:05.437 INFO:tasks.workunit.client.0.smithi067.stderr:osd.14 does not exist. 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 msgr2=0x7fbf5403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 0x7fbf5403abf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009c30 tx=0x7fbf58006f50 comp rx=0 tx=0).stop 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 10 monclient: shutdown 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 msgr2=0x7fbf5403d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 0x7fbf5403d6e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c000c70 tx=0x7fbf5c02f000 comp rx=0 tx=0).stop 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 shutdown_connections 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf68131e70 0x7fbf540400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf54038740 0x7fbf5403abf0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf68098420 0x7fbf54042a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 --2- 172.21.15.67:0/3910633766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf5403b300 0x7fbf5403d6e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 >> 172.21.15.67:0/3910633766 conn(0x7fbf680a5c10 msgr2=0x7fbf68138e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:05.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 shutdown_connections 2024-09-15T01:56:05.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.436+0000 7fbf6d59d640 1 -- 172.21.15.67:0/3910633766 wait complete. 2024-09-15T01:56:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-15T01:56:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.520+0000 7fa9a3a6f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.520+0000 7fa9a3a6f640 10 monclient: build_initial_monmap 2024-09-15T01:56:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.520+0000 7fa9a3a6f640 10 monclient: monmap: 2024-09-15T01:56:05.522 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:05.521423+0000 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:05.521423+0000 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:05.523 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.520+0000 7fa9a3a6f640 1 Processor -- start 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 -- start start 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient: init 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa99c098370 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c098740 0x7fa99c098b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa99c136ea0 0x7fa99c139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c136ea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0a17b0 con 0x7fa99c098370 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c132a70 con 0x7fa99c136ea0 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c09feb0 con 0x7fa99c098740 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a3a6f640 10 monclient(hunting): authenticate will time out at 997.990544s 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa99c136ea0 0x7fa99c139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa99c136ea0 0x7fa99c139280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45870/0 (socket says 172.21.15.67:45870) 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a180b640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fa99c098370 legacy=0x7fa99c132360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 1 -- 172.21.15.67:0/2431860808 learned_addr learned my addr 172.21.15.67:0/2431860808 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): get_auth_request con 0x7fa99c136ea0 auth_method 0 2024-09-15T01:56:05.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a0809640 1 -- 172.21.15.67:0/2431860808 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c098740 0x7fa99c098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): get_auth_request con 0x7fa99c098740 auth_method 0 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:05.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.521+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 10 monclient(hunting): handle_auth_done global_id 5665 payload 274 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 1 -- 172.21.15.67:0/2431860808 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c098740 msgr2=0x7fa99c098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c098740 0x7fa99c098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 1 -- 172.21.15.67:0/2431860808 >> v1:172.21.15.17:6789/0 conn(0x7fa99c098370 legacy=0x7fa99c132360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 1 monclient: found mon.noname-c 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.522+0000 7fa9a200c640 1 -- 172.21.15.67:0/2431860808 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0a00f0 con 0x7fa99c136ea0 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a200c640 1 -- 172.21.15.67:0/2431860808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c136ea0 msgr2=0x7fa99c139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a200c640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c136ea0 0x7fa99c139280 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa99c09fe40 tx=0x7fa994009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3acd94e4329eb72b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 1 -- 172.21.15.67:0/2431860808 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994007610 con 0x7fa99c136ea0 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 10 monclient: dump: 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:05.535 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 10 monclient: _finish_auth 0 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 5 monclient: authenticate success, global_id 5665 2024-09-15T01:56:05.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.524+0000 7fa9a0809640 1 -- 172.21.15.67:0/2431860808 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99402d050 con 0x7fa99c136ea0 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a0809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 4 set_mon_vals no callback set 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a0809640 1 -- 172.21.15.67:0/2431860808 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9940079a0 con 0x7fa99c136ea0 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a0809640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a0809640 10 monclient: dump: 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:05.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 10 monclient: shutdown 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2431860808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c136ea0 msgr2=0x7fa99c139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c136ea0 0x7fa99c139280 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa99c09fe40 tx=0x7fa994009840 comp rx=0 tx=0).stop 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2431860808 shutdown_connections 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c136ea0 0x7fa99c139280 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2431860808 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c098740 0x7fa99c098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2431860808 >> 172.21.15.67:0/2431860808 conn(0x7fa99c0a59a0 msgr2=0x7fa99c0a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:05.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2431860808 shutdown_connections 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.525+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2431860808 wait complete. 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient: build_initial_monmap 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient: monmap: 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:05.539 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 Processor -- start 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 -- start start 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient: init 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 0x7fa99c141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa99c098370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c136ea0 0x7fa99c132bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c136ea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:05.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c069eb0 con 0x7fa99c098370 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1359e0 con 0x7fa99c098740 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c135700 con 0x7fa99c136ea0 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a3a6f640 10 monclient(hunting): authenticate will time out at 997.995869s 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a180b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 msgr2=0x7fa99c141170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a180b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 0x7fa99c141170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:05.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45874/0 (socket says 172.21.15.67:45874) 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 1 -- 172.21.15.67:0/2352563778 learned_addr learned my addr 172.21.15.67:0/2352563778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 10 monclient(hunting): get_auth_request con 0x7fa99c098740 auth_method 0 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.526+0000 7fa9a100a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c136ea0 0x7fa99c132bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:05.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): get_auth_request con 0x7fa99c136ea0 auth_method 0 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a200c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient(hunting): handle_auth_done global_id 5668 payload 824 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 -- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c136ea0 msgr2=0x7fa99c132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c136ea0 0x7fa99c132bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 -- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 msgr2=0x7fa99c141170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 0x7fa99c141170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient: _finish_hunting 0 2024-09-15T01:56:05.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 monclient: found mon.noname-b 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.527+0000 7fa9a100a640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994009ae0 con 0x7fa99c098740 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a100a640 1 -- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 msgr2=0x7fa99c1416b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a100a640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00ea20 tx=0x7fa98c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00cc20 con 0x7fa99c098740 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: dump: 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.544 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c00cd80 con 0x7fa99c098740 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 5 monclient: authenticate success, global_id 5668 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 10 monclient: _renew_subs 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c133c40 con 0x7fa99c098740 2024-09-15T01:56:05.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c013680 con 0x7fa99c098740 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa991ffb640 10 monclient: dump: 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:05.546 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 10 monclient: _renew_subs 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa993fff640 4 set_mon_vals no callback set 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.529+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c09ee00 con 0x7fa99c098740 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.530+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa98c013a70 con 0x7fa99c098740 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.530+0000 7fa9a3a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.530+0000 7fa9a3a6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.530+0000 7fa9a3a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.530+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c04bf60 con 0x7fa99c098740 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.531+0000 7fa991ffb640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 0x7fa98003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.531+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fa98c04c5b0 con 0x7fa99c098740 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.531+0000 7fa9a180b640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 0x7fa98003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:05.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.531+0000 7fa9a180b640 10 monclient: get_auth_request con 0x7fa9800386f0 auth_method 0 2024-09-15T01:56:05.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.533+0000 7fa9a180b640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 0x7fa98003aba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9980038f0 tx=0x7fa998006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:05.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.535+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa98c023d40 con 0x7fa99c098740 2024-09-15T01:56:05.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.535+0000 7fa991ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:05.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.535+0000 7fa991ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:05.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.705+0000 7fa9a3a6f640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:05.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.705+0000 7fa9a3a6f640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:05.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.705+0000 7fa9a3a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:05.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:05.705+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fa99c135ba0 con 0x7fa99c098740 2024-09-15T01:56:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.345+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v118) v1 ==== 145+0+0 (secure 0 0 0) 0x7fa98c023760 con 0x7fa99c098740 2024-09-15T01:56:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.345+0000 7fa991ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.345+0000 7fa991ffb640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-15T01:56:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.454+0000 7fa9a3a6f640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.454+0000 7fa9a3a6f640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.454+0000 7fa9a3a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:06.454+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fa99c0008d0 con 0x7fa99c098740 2024-09-15T01:56:07.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.297+0000 7fa991ffb640 1 -- 172.21.15.67:0/2352563778 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v119) v1 ==== 145+0+0 (secure 0 0 0) 0x7fa98c023d40 con 0x7fa99c098740 2024-09-15T01:56:07.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.297+0000 7fa991ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T01:56:07.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.297+0000 7fa991ffb640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-15T01:56:07.300 INFO:tasks.workunit.client.0.smithi067.stderr:set require_min_compat_client to luminous 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 msgr2=0x7fa98003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 0x7fa98003aba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9980038f0 tx=0x7fa998006d80 comp rx=0 tx=0).stop 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 10 monclient: shutdown 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 msgr2=0x7fa99c1416b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00ea20 tx=0x7fa98c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 shutdown_connections 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa9800386f0 0x7fa98003aba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa99c136ea0 0x7fa99c132bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c098740 0x7fa99c1416b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.300+0000 7fa9a3a6f640 1 --2- 172.21.15.67:0/2352563778 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa99c098370 0x7fa99c141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.301+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 >> 172.21.15.67:0/2352563778 conn(0x7fa99c0a59a0 msgr2=0x7fa99c138940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.301+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 shutdown_connections 2024-09-15T01:56:07.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.301+0000 7fa9a3a6f640 1 -- 172.21.15.67:0/2352563778 wait complete. 2024-09-15T01:56:07.316 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set dump 2024-09-15T01:56:07.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.396+0000 7ff8c9d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:07.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.396+0000 7ff8c9d9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.396+0000 7ff8c9d9d640 10 monclient: monmap: 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:07.397712+0000 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:07.397712+0000 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:07.399 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:07.400 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 Processor -- start 2024-09-15T01:56:07.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 -- start start 2024-09-15T01:56:07.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient: init 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8c406e5f0 0x7ff8c406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c406e5f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 0x7ff8c409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c406ef00 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c409eef0 0x7ff8c409e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8c409eef0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c4068a90 con 0x7ff8c406ef00 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c4068f80 con 0x7ff8c406e5f0 2024-09-15T01:56:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c40a4080 con 0x7ff8c409eef0 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.397+0000 7ff8c9d9d640 10 monclient(hunting): authenticate will time out at 999.866964s 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8c406e5f0 0x7ff8c406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c2ffd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 msgr2=0x7ff8c409dba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8c406e5f0 0x7ff8c406e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45876/0 (socket says 172.21.15.67:45876) 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 0x7ff8c409dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 -- 172.21.15.67:0/1413517908 learned_addr learned my addr 172.21.15.67:0/1413517908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request con 0x7ff8c406e5f0 auth_method 0 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c27fc640 1 -- 172.21.15.67:0/1413517908 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c3fff640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c409eef0 0x7ff8c409e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c3fff640 10 monclient(hunting): get_auth_request con 0x7ff8c409eef0 auth_method 0 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_done global_id 5671 payload 274 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 -- 172.21.15.67:0/1413517908 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c409eef0 msgr2=0x7ff8c409e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c409eef0 0x7ff8c409e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 -- 172.21.15.67:0/1413517908 >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 msgr2=0x7ff8c409dba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1413517908 >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 0x7ff8c409dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 monclient: found mon.noname-c 2024-09-15T01:56:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.398+0000 7ff8c37fe640 1 -- 172.21.15.67:0/1413517908 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c4068ca0 con 0x7ff8c406e5f0 2024-09-15T01:56:07.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c37fe640 1 -- 172.21.15.67:0/1413517908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c406e5f0 msgr2=0x7ff8c406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c406e5f0 0x7ff8c406e9c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40970c0 tx=0x7ff8b40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637f26d7d3a32f18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 1 -- 172.21.15.67:0/1413517908 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4032440 con 0x7ff8c406e5f0 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: dump: 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.412 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 5 monclient: authenticate success, global_id 5671 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 1 -- 172.21.15.67:0/1413517908 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b40325a0 con 0x7ff8c406e5f0 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 4 set_mon_vals no callback set 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 1 -- 172.21.15.67:0/1413517908 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4032940 con 0x7ff8c406e5f0 2024-09-15T01:56:07.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c27fc640 10 monclient: dump: 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 10 monclient: shutdown 2024-09-15T01:56:07.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1413517908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c406e5f0 msgr2=0x7ff8c406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c406e5f0 0x7ff8c406e9c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40970c0 tx=0x7ff8b40091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1413517908 shutdown_connections 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c409eef0 0x7ff8c409e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1413517908 >> v2:172.21.15.17:3300/0 conn(0x7ff8c406ef00 0x7ff8c409dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1413517908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c406e5f0 0x7ff8c406e9c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1413517908 >> 172.21.15.67:0/1413517908 conn(0x7ff8c4071230 msgr2=0x7ff8c40a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.401+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1413517908 shutdown_connections 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.402+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1413517908 wait complete. 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.402+0000 7ff8c9d9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.402+0000 7ff8c9d9d640 10 monclient: monmap: 2024-09-15T01:56:07.415 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.402+0000 7ff8c9d9d640 1 Processor -- start 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 -- start start 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient: init 2024-09-15T01:56:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c406e5f0 0x7ff8c41412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c406e5f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 0x7ff8c4135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c409eef0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8c41355c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c409f3a0 con 0x7ff8c409eef0 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c406d400 con 0x7ff8c41355c0 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4141800 con 0x7ff8c406e5f0 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c9d9d640 10 monclient(hunting): authenticate will time out at 999.872489s 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c2ffd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 msgr2=0x7ff8c4135080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45884/0 (socket says 172.21.15.67:45884) 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 -- 172.21.15.67:0/1181365980 learned_addr learned my addr 172.21.15.67:0/1181365980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): get_auth_request con 0x7ff8c41355c0 auth_method 0 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c2ffd640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 0x7ff8c4135080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c406e5f0 0x7ff8c41412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request con 0x7ff8c406e5f0 auth_method 0 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient(hunting): handle_auth_done global_id 5674 payload 824 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 -- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c406e5f0 msgr2=0x7ff8c41412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c406e5f0 0x7ff8c41412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 -- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 msgr2=0x7ff8c4135080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 0x7ff8c4135080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 monclient: found mon.noname-b 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.403+0000 7ff8c3fff640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4009450 con 0x7ff8c41355c0 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c3fff640 1 -- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 msgr2=0x7ff8c411ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c3fff640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009fa0 tx=0x7ff8bc009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00ee60 con 0x7ff8c41355c0 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: dump: 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc013590 con 0x7ff8c41355c0 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 5 monclient: authenticate success, global_id 5674 2024-09-15T01:56:07.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:07.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc013950 con 0x7ff8c41355c0 2024-09-15T01:56:07.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 10 monclient: _renew_subs 2024-09-15T01:56:07.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c4121240 con 0x7ff8c41355c0 2024-09-15T01:56:07.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8ab7fe640 10 monclient: dump: 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.424 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c1ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 10 monclient: _renew_subs 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.406+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c41221a0 con 0x7ff8c41355c0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff8bc013cc0 con 0x7ff8c41355c0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c9d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c9d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c9d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c40639e0 con 0x7ff8c41355c0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8ab7fe640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 0x7ff8b003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6523+0+0 (secure 0 0 0) 0x7ff8bc04c9b0 con 0x7ff8c41355c0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 0x7ff8b003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.407+0000 7ff8c37fe640 10 monclient: get_auth_request con 0x7ff8b0038770 auth_method 0 2024-09-15T01:56:07.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.410+0000 7ff8c37fe640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 0x7ff8b003ac20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40a3160 tx=0x7ff8b4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.411+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff8bc014420 con 0x7ff8c41355c0 2024-09-15T01:56:07.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.411+0000 7ff8ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.411+0000 7ff8ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:07.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8c9d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8c9d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8c9d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7ff8c4069d00 con 0x7ff8c41355c0 2024-09-15T01:56:07.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v119) v1 ==== 73+0+3 (secure 0 0 0) 0x7ff8bc01b310 con 0x7ff8c41355c0 2024-09-15T01:56:07.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.582+0000 7ff8ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:07.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8c9d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8c9d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8c9d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7ff8c409ecd0 con 0x7ff8c41355c0 2024-09-15T01:56:07.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8ab7fe640 1 -- 172.21.15.67:0/1181365980 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v119) v1 ==== 73+0+3 (secure 0 0 0) 0x7ff8bc0122e0 con 0x7ff8c41355c0 2024-09-15T01:56:07.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:07.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.686+0000 7ff8ab7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:07.689 INFO:tasks.workunit.client.0.smithi067.stdout:{} 2024-09-15T01:56:07.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 msgr2=0x7ff8b003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 0x7ff8b003ac20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40a3160 tx=0x7ff8b4032000 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 10 monclient: shutdown 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 msgr2=0x7ff8c411ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009fa0 tx=0x7ff8bc009900 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 shutdown_connections 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8b0038770 0x7ff8b003ac20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff8c409eef0 0x7ff8c4135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c41355c0 0x7ff8c411ec10 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 --2- 172.21.15.67:0/1181365980 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff8c406e5f0 0x7ff8c41412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 >> 172.21.15.67:0/1181365980 conn(0x7ff8c4071230 msgr2=0x7ff8c406d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 shutdown_connections 2024-09-15T01:56:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.688+0000 7ff8c9d9d640 1 -- 172.21.15.67:0/1181365980 wait complete. 2024-09-15T01:56:07.699 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 10 monclient: get_monmap_and_config 2024-09-15T01:56:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 10 monclient: build_initial_monmap 2024-09-15T01:56:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 10 monclient: monmap: 2024-09-15T01:56:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:07.774224+0000 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:07.774224+0000 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:07.776 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 1 Processor -- start 2024-09-15T01:56:07.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 1 -- start start 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.773+0000 7f9cffcb8640 10 monclient: init 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 0x7f9cf8132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-a con 0x7f9cf8131e70 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-b con 0x7f9cf8098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cf8068e70 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf80a13e0 con 0x7f9cf8068e70 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf80a0440 con 0x7f9cf8131e70 2024-09-15T01:56:07.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80a01e0 con 0x7f9cf8098420 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cffcb8640 10 monclient(hunting): authenticate will time out at 1000.243368s 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfda54640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 msgr2=0x7f9cf8132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfe255640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f9cf8068e70 legacy=0x7f9cf80692f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfda54640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 0x7f9cf8132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:40596/0 (socket says 172.21.15.67:40596) 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 1 -- 172.21.15.67:0/3523867210 learned_addr learned my addr 172.21.15.67:0/3523867210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): get_auth_request con 0x7f9cf8098420 auth_method 0 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.774+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 10 monclient(hunting): handle_auth_done global_id 74258 payload 274 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 1 -- 172.21.15.67:0/3523867210 >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 msgr2=0x7f9cf8132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 1 --2- 172.21.15.67:0/3523867210 >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 0x7f9cf8132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 1 -- 172.21.15.67:0/3523867210 >> v1:172.21.15.17:6789/0 conn(0x7f9cf8068e70 legacy=0x7f9cf80692f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 10 monclient: _finish_hunting 0 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 1 monclient: found mon.noname-b 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:07.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.775+0000 7f9cfd253640 1 -- 172.21.15.67:0/3523867210 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80a19b0 con 0x7f9cf8098420 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfd253640 1 -- 172.21.15.67:0/3523867210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 msgr2=0x7f9cf8098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfd253640 1 --2- 172.21.15.67:0/3523867210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009910 tx=0x7f9ce80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf9a3a234e6875b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 1 -- 172.21.15.67:0/3523867210 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8032440 con 0x7f9cf8098420 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: dump: 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: _finish_auth 0 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 1 -- 172.21.15.67:0/3523867210 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce80325a0 con 0x7f9cf8098420 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 1 -- 172.21.15.67:0/3523867210 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8032980 con 0x7f9cf8098420 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cfca52640 10 monclient: dump: 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 5 monclient: authenticate success, global_id 74258 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 4 set_mon_vals no callback set 2024-09-15T01:56:07.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 10 monclient: shutdown 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 1 -- 172.21.15.67:0/3523867210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 msgr2=0x7f9cf8098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.778+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/3523867210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009910 tx=0x7f9ce80091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 -- 172.21.15.67:0/3523867210 shutdown_connections 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/3523867210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8098420 0x7f9cf8098850 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/3523867210 >> v2:172.21.15.17:3300/0 conn(0x7f9cf8131e70 0x7f9cf8132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 -- 172.21.15.67:0/3523867210 >> 172.21.15.67:0/3523867210 conn(0x7f9cf80a5c10 msgr2=0x7f9cf80a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 -- 172.21.15.67:0/3523867210 shutdown_connections 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 -- 172.21.15.67:0/3523867210 wait complete. 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 10 monclient: build_initial_monmap 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 10 monclient: monmap: 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 Processor -- start 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 1 -- start start 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.779+0000 7f9cffcb8640 10 monclient: init 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:07.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 0x7f9cf81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-a con 0x7f9cf8068e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-b con 0x7f9cf8098420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8131e70 0x7f9cf8099010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): picked mon.noname-c con 0x7f9cf8131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf811c090 con 0x7f9cf8068e70 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80a0d80 con 0x7f9cf8098420 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf809ab80 con 0x7f9cf8131e70 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cffcb8640 10 monclient(hunting): authenticate will time out at 1000.249366s 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfda54640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 msgr2=0x7f9cf81414b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45888/0 (socket says 172.21.15.67:45888) 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 -- 172.21.15.67:0/586057463 learned_addr learned my addr 172.21.15.67:0/586057463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfda54640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 0x7f9cf81414b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): get_auth_request con 0x7f9cf8098420 auth_method 0 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:07.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfe255640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8131e70 0x7f9cf8099010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfe255640 10 monclient(hunting): get_auth_request con 0x7f9cf8131e70 auth_method 0 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfe255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfe255640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfe255640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient(hunting): handle_auth_done global_id 5677 payload 824 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 -- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8131e70 msgr2=0x7f9cf8099010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8131e70 0x7f9cf8099010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 -- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 msgr2=0x7f9cf81414b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 0x7f9cf81414b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient: _finish_hunting 0 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 monclient: found mon.noname-b 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.794 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.780+0000 7f9cfd253640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8009450 con 0x7f9cf8098420 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cfd253640 1 -- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 msgr2=0x7f9cf812ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cfd253640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009970 tx=0x7f9ce8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8000d10 con 0x7f9cf8098420 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:07.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: dump: 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:07.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 5 monclient: authenticate success, global_id 5677 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8031c70 con 0x7f9cf8098420 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce803c400 con 0x7f9cf8098420 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 10 monclient: _renew_subs 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf809b070 con 0x7f9cf8098420 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9ceffff640 4 set_mon_vals no callback set 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cedffb640 10 monclient: dump: 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 10 monclient: _renew_subs 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.783+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf8142380 con 0x7f9cf8098420 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ce803c870 con 0x7f9cf8098420 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cffcb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cffcb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cffcb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf80639e0 con 0x7f9cf8098420 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cedffb640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 0x7f9cdc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f9ce806c7c0 con 0x7f9cf8098420 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cfda54640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 0x7f9cdc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.784+0000 7f9cfda54640 10 monclient: get_auth_request con 0x7f9cdc0386f0 auth_method 0 2024-09-15T01:56:07.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.787+0000 7f9cfda54640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 0x7f9cdc03aba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40038f0 tx=0x7f9cf4006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:07.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.788+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ce804a2a0 con 0x7f9cf8098420 2024-09-15T01:56:07.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.788+0000 7f9cedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:07.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.788+0000 7f9cedffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.960+0000 7f9cffcb8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.960+0000 7f9cffcb8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.960+0000 7f9cffcb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.960+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f9cf8067a50 con 0x7f9cf8098420 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.961+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v119) v1 ==== 71+0+0 (secure 0 0 0) 0x7f9ce80388c0 con 0x7f9cf8098420 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.961+0000 7f9cedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:07.961+0000 7f9cedffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:08.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cffcb8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cffcb8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cffcb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f9cf8067a50 con 0x7f9cf8098420 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cedffb640 1 -- 172.21.15.67:0/586057463 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v119) v1 ==== 71+0+0 (secure 0 0 0) 0x7f9ce8037030 con 0x7f9cf8098420 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cedffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:08.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.064+0000 7f9cedffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:08.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 msgr2=0x7f9cdc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 0x7f9cdc03aba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40038f0 tx=0x7f9cf4006d80 comp rx=0 tx=0).stop 2024-09-15T01:56:08.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 10 monclient: shutdown 2024-09-15T01:56:08.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 msgr2=0x7f9cf812ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009970 tx=0x7f9ce8032000 comp rx=0 tx=0).stop 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 shutdown_connections 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9cdc0386f0 0x7f9cdc03aba0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9cf8131e70 0x7f9cf8099010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8098420 0x7f9cf812ec90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 --2- 172.21.15.67:0/586057463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cf8068e70 0x7f9cf81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.066+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 >> 172.21.15.67:0/586057463 conn(0x7f9cf80a5c10 msgr2=0x7f9cf8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.067+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 shutdown_connections 2024-09-15T01:56:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.067+0000 7f9cffcb8640 1 -- 172.21.15.67:0/586057463 wait complete. 2024-09-15T01:56:08.077 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T01:56:08.078 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:56:08.078 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 10 monclient: build_initial_monmap 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 10 monclient: monmap: 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:08.155309+0000 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:08.155309+0000 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.157 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 1 Processor -- start 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 1 -- start start 2024-09-15T01:56:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.154+0000 7f451571d640 10 monclient: init 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 0x7f45101321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): picked mon.noname-a con 0x7f4510098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4510098a50 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510098ef0 con 0x7f4510098a50 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45100a1730 con 0x7f4510098140 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45100a0b80 con 0x7f45100a0370 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f451571d640 10 monclient(hunting): authenticate will time out at 1000.624280s 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450f7fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f4510098a50 legacy=0x7f45100695c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450effd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 msgr2=0x7f45101321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 0x7f45101321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:40608/0 (socket says 172.21.15.67:40608) 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 -- 172.21.15.67:0/2873888212 learned_addr learned my addr 172.21.15.67:0/2873888212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): get_auth_request con 0x7f45100a0370 auth_method 0 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient(hunting): handle_auth_done global_id 74264 payload 274 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 -- 172.21.15.67:0/2873888212 >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 msgr2=0x7f45101321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 --2- 172.21.15.67:0/2873888212 >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 0x7f45101321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 -- 172.21.15.67:0/2873888212 >> v1:172.21.15.17:6789/0 conn(0x7f4510098a50 legacy=0x7f45100695c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 monclient: found mon.noname-b 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:08.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.155+0000 7f450e7fc640 1 -- 172.21.15.67:0/2873888212 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100a00b0 con 0x7f45100a0370 2024-09-15T01:56:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450e7fc640 1 -- 172.21.15.67:0/2873888212 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 msgr2=0x7f4510098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450e7fc640 1 --2- 172.21.15.67:0/2873888212 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f45100a5340 tx=0x7f44f80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac853249073ea241 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 1 -- 172.21.15.67:0/2873888212 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8032440 con 0x7f45100a0370 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: dump: 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.168 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 1 -- 172.21.15.67:0/2873888212 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f80325a0 con 0x7f45100a0370 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 5 monclient: authenticate success, global_id 74264 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 1 -- 172.21.15.67:0/2873888212 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8032940 con 0x7f45100a0370 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f450dffb640 10 monclient: dump: 2024-09-15T01:56:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 4 set_mon_vals no callback set 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 10 monclient: shutdown 2024-09-15T01:56:08.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 -- 172.21.15.67:0/2873888212 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 msgr2=0x7f4510098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 --2- 172.21.15.67:0/2873888212 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f45100a5340 tx=0x7f44f80091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 -- 172.21.15.67:0/2873888212 shutdown_connections 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 --2- 172.21.15.67:0/2873888212 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f45100a0370 0x7f4510098510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 --2- 172.21.15.67:0/2873888212 >> v2:172.21.15.17:3300/0 conn(0x7f4510098140 0x7f45101321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.159+0000 7f451571d640 1 -- 172.21.15.67:0/2873888212 >> 172.21.15.67:0/2873888212 conn(0x7f45100a5780 msgr2=0x7f45100a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.160+0000 7f451571d640 1 -- 172.21.15.67:0/2873888212 shutdown_connections 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.160+0000 7f451571d640 1 -- 172.21.15.67:0/2873888212 wait complete. 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.160+0000 7f451571d640 10 monclient: build_initial_monmap 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.160+0000 7f451571d640 10 monclient: monmap: 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:08.171 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.160+0000 7f451571d640 1 Processor -- start 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 -- start start 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient: init 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4510098140 0x7f451009b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): picked mon.noname-a con 0x7f4510098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510098a50 0x7f45101417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): picked mon.noname-c con 0x7f4510098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45100a0370 0x7f4510130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510131c40 con 0x7f4510098140 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451011e600 con 0x7f45100a0370 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4510099090 con 0x7f4510098a50 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f451571d640 10 monclient(hunting): authenticate will time out at 1000.630525s 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450effd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4510098140 msgr2=0x7f451009b340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4510098140 0x7f451009b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45100a0370 0x7f4510130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45100a0370 0x7f4510130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45890/0 (socket says 172.21.15.67:45890) 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 -- 172.21.15.67:0/3966694416 learned_addr learned my addr 172.21.15.67:0/3966694416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): get_auth_request con 0x7f45100a0370 auth_method 0 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450e7fc640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510098a50 0x7f45101417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450e7fc640 10 monclient(hunting): get_auth_request con 0x7f4510098a50 auth_method 0 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient(hunting): handle_auth_done global_id 5680 payload 824 2024-09-15T01:56:08.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 -- 172.21.15.67:0/3966694416 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510098a50 msgr2=0x7f45101417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4510098a50 0x7f45101417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 -- 172.21.15.67:0/3966694416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4510098140 msgr2=0x7f451009b340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4510098140 0x7f451009b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 monclient: found mon.noname-b 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.161+0000 7f450f7fe640 1 -- 172.21.15.67:0/3966694416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8009450 con 0x7f45100a0370 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f450f7fe640 1 -- 172.21.15.67:0/3966694416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45100a0370 msgr2=0x7f4510130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f450f7fe640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45100a0370 0x7f4510130f20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f450000bb40 tx=0x7f45000097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450000cbf0 con 0x7f45100a0370 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: dump: 2024-09-15T01:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: _finish_auth 0 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 5 monclient: authenticate success, global_id 5680 2024-09-15T01:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 10 monclient: _renew_subs 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 1 -- 172.21.15.67:0/3966694416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451011e040 con 0x7f45100a0370 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450000cd70 con 0x7f45100a0370 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500006150 con 0x7f45100a0370 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f4506ffd640 10 monclient: dump: 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 10 monclient: _renew_subs 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f450d7fa640 4 set_mon_vals no callback set 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.164+0000 7f451571d640 1 -- 172.21.15.67:0/3966694416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4510131460 con 0x7f45100a0370 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f451571d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f451571d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f451571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4500012020 con 0x7f45100a0370 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f451571d640 1 -- 172.21.15.67:0/3966694416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451004bf60 con 0x7f45100a0370 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f4506ffd640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f0038720 0x7f44f003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f450004c050 con 0x7f45100a0370 2024-09-15T01:56:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f450effd640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f0038720 0x7f44f003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.165+0000 7f450effd640 10 monclient: get_auth_request con 0x7f44f0038720 auth_method 0 2024-09-15T01:56:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.168+0000 7f450effd640 1 --2- 172.21.15.67:0/3966694416 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44f0038720 0x7f44f003abd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f45080038f0 tx=0x7f4508006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.169+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f450007de40 con 0x7f45100a0370 2024-09-15T01:56:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.169+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.169+0000 7f4506ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:08.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.339+0000 7f451571d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.339+0000 7f451571d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.339+0000 7f451571d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.339+0000 7f451571d640 1 -- 172.21.15.67:0/3966694416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f451011dcb0 con 0x7f45100a0370 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.340+0000 7f4506ffd640 1 -- 172.21.15.67:0/3966694416 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v119) v1 ==== 152+0+0 (secure 0 0 0) 0x7f450001daa0 con 0x7f45100a0370 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.340+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:08.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.340+0000 7f4506ffd640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-15T01:56:08.343 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOENT: pool 'fooset' not found 2024-09-15T01:56:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:56:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create fooset 8 2024-09-15T01:56:08.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.420+0000 7fb7f8ea8640 10 monclient: get_monmap_and_config 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.420+0000 7fb7f8ea8640 10 monclient: build_initial_monmap 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.421+0000 7fb7f8ea8640 10 monclient: monmap: 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:08.422168+0000 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:08.422168+0000 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:08.424 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:08.425 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:08.425 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.421+0000 7fb7f8ea8640 1 Processor -- start 2024-09-15T01:56:08.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.421+0000 7fb7f8ea8640 1 -- start start 2024-09-15T01:56:08.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.421+0000 7fb7f8ea8640 10 monclient: init 2024-09-15T01:56:08.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:08.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:56:08.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7f40a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:08.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 0x7fb7f409f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f409eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7f4136e40 0x7fb7f4139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f4136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f406a270 con 0x7fb7f40a0b50 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f406f290 con 0x7fb7f409eea0 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f409f7e0 con 0x7fb7f4136e40 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f8ea8640 10 monclient(hunting): authenticate will time out at 1000.891309s 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f259c640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fb7f40a0b50 legacy=0x7fb7f4069340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7f4136e40 0x7fb7f4139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f1d9b640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 msgr2=0x7fb7f409f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f1d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 0x7fb7f409f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7f4136e40 0x7fb7f4139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45906/0 (socket says 172.21.15.67:45906) 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 -- 172.21.15.67:0/708949143 learned_addr learned my addr 172.21.15.67:0/708949143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): get_auth_request con 0x7fb7f4136e40 auth_method 0 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient(hunting): handle_auth_done global_id 5683 payload 274 2024-09-15T01:56:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 -- 172.21.15.67:0/708949143 >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 msgr2=0x7fb7f409f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 --2- 172.21.15.67:0/708949143 >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 0x7fb7f409f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 -- 172.21.15.67:0/708949143 >> v1:172.21.15.17:6789/0 conn(0x7fb7f40a0b50 legacy=0x7fb7f4069340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 monclient: found mon.noname-c 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.422+0000 7fb7f2d9d640 1 -- 172.21.15.67:0/708949143 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f40a0f20 con 0x7fb7f4136e40 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f2d9d640 1 -- 172.21.15.67:0/708949143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f4136e40 msgr2=0x7fb7f4139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f2d9d640 1 --2- 172.21.15.67:0/708949143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f4136e40 0x7fb7f4139220 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409cdb0 tx=0x7fb7ec0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a21fc1fb1d74370b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 1 -- 172.21.15.67:0/708949143 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec032440 con 0x7fb7f4136e40 2024-09-15T01:56:08.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: dump: 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.433 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: _finish_auth 0 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 1 -- 172.21.15.67:0/708949143 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0325a0 con 0x7fb7f4136e40 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f8ea8640 5 monclient: authenticate success, global_id 5683 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f8ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 1 -- 172.21.15.67:0/708949143 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec032940 con 0x7fb7f4136e40 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f8ea8640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:08.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f8ea8640 4 set_mon_vals no callback set 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.425+0000 7fb7f159a640 10 monclient: dump: 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.435 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 10 monclient: shutdown 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/708949143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f4136e40 msgr2=0x7fb7f4139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/708949143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f4136e40 0x7fb7f4139220 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409cdb0 tx=0x7fb7ec0091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/708949143 shutdown_connections 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/708949143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f4136e40 0x7fb7f4139220 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/708949143 >> v2:172.21.15.17:3300/0 conn(0x7fb7f409eea0 0x7fb7f409f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/708949143 >> 172.21.15.67:0/708949143 conn(0x7fb7f4071290 msgr2=0x7fb7f40a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/708949143 shutdown_connections 2024-09-15T01:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/708949143 wait complete. 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 10 monclient: build_initial_monmap 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.426+0000 7fb7f8ea8640 10 monclient: monmap: 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:08.438 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 Processor -- start 2024-09-15T01:56:08.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 -- start start 2024-09-15T01:56:08.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient: init 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f409eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 0x7fb7f41417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f40a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7f4136e40 0x7fb7f4141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f4136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41204e0 con 0x7fb7f40a0b50 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f411ff60 con 0x7fb7f409eea0 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40a1260 con 0x7fb7f4136e40 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f8ea8640 10 monclient(hunting): authenticate will time out at 1000.896812s 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45916/0 (socket says 172.21.15.67:45916) 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 1 -- 172.21.15.67:0/81460882 learned_addr learned my addr 172.21.15.67:0/81460882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): get_auth_request con 0x7fb7f409eea0 auth_method 0 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f1d9b640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 msgr2=0x7fb7f41417f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:08.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f1d9b640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 0x7fb7f41417f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.427+0000 7fb7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f2d9d640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7f4136e40 0x7fb7f4141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f2d9d640 10 monclient(hunting): get_auth_request con 0x7fb7f4136e40 auth_method 0 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 10 monclient(hunting): handle_auth_done global_id 5686 payload 824 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7f4136e40 msgr2=0x7fb7f4141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7f4136e40 0x7fb7f4141d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 msgr2=0x7fb7f41417f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:08.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 0x7fb7f41417f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 monclient: found mon.noname-b 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.428+0000 7fb7f259c640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec009450 con 0x7fb7f409eea0 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f259c640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 msgr2=0x7fb7f41412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f259c640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400bb80 tx=0x7fb7e40097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400cc80 con 0x7fb7f409eea0 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: dump: 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.445 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 5 monclient: authenticate success, global_id 5686 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e400cde0 con 0x7fb7f409eea0 2024-09-15T01:56:08.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4016430 con 0x7fb7f409eea0 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 10 monclient: _renew_subs 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f40a33e0 con 0x7fb7f409eea0 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f0d99640 4 set_mon_vals no callback set 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7ea7fc640 10 monclient: dump: 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:08.447 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 10 monclient: _renew_subs 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.430+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f4120b30 con 0x7fb7f409eea0 2024-09-15T01:56:08.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb7e4012020 con 0x7fb7f409eea0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f8ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f8ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f8ea8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f404bf60 con 0x7fb7f409eea0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7ea7fc640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 0x7fb7d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fb7e404c0e0 con 0x7fb7f409eea0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f1d9b640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 0x7fb7d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.431+0000 7fb7f1d9b640 10 monclient: get_auth_request con 0x7fb7d40386f0 auth_method 0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.434+0000 7fb7f1d9b640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 0x7fb7d403aba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0038f0 tx=0x7fb7dc006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.436+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb7e401d900 con 0x7fb7f409eea0 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.436+0000 7fb7ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:08.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.436+0000 7fb7ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:08.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.603+0000 7fb7f8ea8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:08.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.603+0000 7fb7f8ea8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:08.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.603+0000 7fb7f8ea8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:08.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:08.603+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fb7f4121a80 con 0x7fb7f409eea0 2024-09-15T01:56:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.354+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v120) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb7e401d300 con 0x7fb7f409eea0 2024-09-15T01:56:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.354+0000 7fb7ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.354+0000 7fb7ea7fc640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-15T01:56:09.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.457+0000 7fb7f8ea8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:09.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.457+0000 7fb7f8ea8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.457+0000 7fb7f8ea8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.457+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fb7f40008d0 con 0x7fb7f409eea0 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.458+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/81460882 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v120) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb7e401d4e0 con 0x7fb7f409eea0 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.458+0000 7fb7ea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.458+0000 7fb7ea7fc640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-15T01:56:09.461 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'fooset' already exists 2024-09-15T01:56:09.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 msgr2=0x7fb7d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:09.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 0x7fb7d403aba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0038f0 tx=0x7fb7dc006d80 comp rx=0 tx=0).stop 2024-09-15T01:56:09.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 10 monclient: shutdown 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 msgr2=0x7fb7f41412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400bb80 tx=0x7fb7e40097f0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 shutdown_connections 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb7d40386f0 0x7fb7d403aba0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb7f40a0b50 0x7fb7f41417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7f4136e40 0x7fb7f4141d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 --2- 172.21.15.67:0/81460882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f409eea0 0x7fb7f41412b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 >> 172.21.15.67:0/81460882 conn(0x7fb7f4071290 msgr2=0x7fb7f409ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 shutdown_connections 2024-09-15T01:56:09.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.460+0000 7fb7f8ea8640 1 -- 172.21.15.67:0/81460882 wait complete. 2024-09-15T01:56:09.471 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create barset 8 2024-09-15T01:56:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.546+0000 7f193c699640 10 monclient: get_monmap_and_config 2024-09-15T01:56:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.546+0000 7f193c699640 10 monclient: build_initial_monmap 2024-09-15T01:56:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.546+0000 7f193c699640 10 monclient: monmap: 2024-09-15T01:56:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:09.547846+0000 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:09.547846+0000 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 Processor -- start 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 -- start start 2024-09-15T01:56:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient: init 2024-09-15T01:56:09.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:09.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:56:09.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19340a1a00 0x7f193409aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): picked mon.noname-c con 0x7f19340a1a00 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f193409d5a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 0x7f1934097180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): picked mon.noname-a con 0x7f193409b020 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193409cc40 con 0x7f193409d5a0 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193409cf80 con 0x7f193409b020 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193409b450 con 0x7f19340a1a00 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.547+0000 7f193c699640 10 monclient(hunting): authenticate will time out at 1002.016942s 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19340a1a00 0x7f193409aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193ac36640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 msgr2=0x7f1934097180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f1939c34640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f193409d5a0 legacy=0x7f1934096a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193ac36640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 0x7f1934097180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19340a1a00 0x7f193409aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45920/0 (socket says 172.21.15.67:45920) 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 -- 172.21.15.67:0/535719480 learned_addr learned my addr 172.21.15.67:0/535719480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): get_auth_request con 0x7f19340a1a00 auth_method 0 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient(hunting): handle_auth_done global_id 5689 payload 274 2024-09-15T01:56:09.554 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 -- 172.21.15.67:0/535719480 >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 msgr2=0x7f1934097180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 --2- 172.21.15.67:0/535719480 >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 0x7f1934097180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 -- 172.21.15.67:0/535719480 >> v1:172.21.15.17:6789/0 conn(0x7f193409d5a0 legacy=0x7f1934096a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient: _finish_hunting 0 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 monclient: found mon.noname-c 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.548+0000 7f193a435640 1 -- 172.21.15.67:0/535719480 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340a1dd0 con 0x7f19340a1a00 2024-09-15T01:56:09.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193a435640 1 -- 172.21.15.67:0/535719480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340a1a00 msgr2=0x7f193409aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193a435640 1 --2- 172.21.15.67:0/535719480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340a1a00 0x7f193409aae0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1934068b40 tx=0x7f19300091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b9c611b80e68904 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 1 -- 172.21.15.67:0/535719480 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930032440 con 0x7f19340a1a00 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: dump: 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:09.561 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: _finish_auth 0 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 1 -- 172.21.15.67:0/535719480 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19300325a0 con 0x7f19340a1a00 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 5 monclient: authenticate success, global_id 5689 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 4 set_mon_vals no callback set 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 1 -- 172.21.15.67:0/535719480 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930032980 con 0x7f19340a1a00 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:09.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f1939433640 10 monclient: dump: 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 10 monclient: shutdown 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:09.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 -- 172.21.15.67:0/535719480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340a1a00 msgr2=0x7f193409aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 --2- 172.21.15.67:0/535719480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340a1a00 0x7f193409aae0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1934068b40 tx=0x7f19300091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 -- 172.21.15.67:0/535719480 shutdown_connections 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 --2- 172.21.15.67:0/535719480 >> v2:172.21.15.17:3300/0 conn(0x7f193409b020 0x7f1934097180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 --2- 172.21.15.67:0/535719480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340a1a00 0x7f193409aae0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 -- 172.21.15.67:0/535719480 >> 172.21.15.67:0/535719480 conn(0x7f1934071340 msgr2=0x7f19340a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 -- 172.21.15.67:0/535719480 shutdown_connections 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.551+0000 7f193c699640 1 -- 172.21.15.67:0/535719480 wait complete. 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.552+0000 7f193c699640 10 monclient: build_initial_monmap 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.552+0000 7f193c699640 10 monclient: monmap: 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.552+0000 7f193c699640 1 Processor -- start 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.552+0000 7f193c699640 1 -- start start 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.552+0000 7f193c699640 10 monclient: init 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): picked mon.noname-b con 0x7f193409b020 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:09.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f193409d5a0 0x7f19341293e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): picked mon.noname-c con 0x7f193409d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 0x7f193412e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): picked mon.noname-a con 0x7f19340a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934129aa0 con 0x7f19340a1a00 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193412b7f0 con 0x7f193409b020 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193412bad0 con 0x7f193409d5a0 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193c699640 10 monclient(hunting): authenticate will time out at 1002.022204s 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:09.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45922/0 (socket says 172.21.15.67:45922) 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 -- 172.21.15.67:0/3823261821 learned_addr learned my addr 172.21.15.67:0/3823261821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): get_auth_request con 0x7f193409b020 auth_method 0 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193ac36640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 msgr2=0x7f193412e010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193ac36640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 0x7f193412e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f1939c34640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f193409d5a0 0x7f19341293e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f1939c34640 10 monclient(hunting): get_auth_request con 0x7f193409d5a0 auth_method 0 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f1939c34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:09.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f1939c34640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f1939c34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient(hunting): handle_auth_done global_id 5692 payload 824 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f193409d5a0 msgr2=0x7f19341293e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f193409d5a0 0x7f19341293e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 msgr2=0x7f193412e010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 0x7f193412e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient: _finish_hunting 0 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 monclient: found mon.noname-b 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.553+0000 7f193a435640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930009450 con 0x7f193409b020 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.555+0000 7f193a435640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 msgr2=0x7f193406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.555+0000 7f193a435640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1930009420 tx=0x7f1930032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193003c960 con 0x7f193409b020 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: dump: 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:09.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193003cac0 con 0x7f193409b020 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 5 monclient: authenticate success, global_id 5692 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 10 monclient: _renew_subs 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193406e680 con 0x7f193409b020 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193003ce40 con 0x7f193409b020 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:09.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f192a7fc640 10 monclient: dump: 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 10 monclient: _renew_subs 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f1938c32640 4 set_mon_vals no callback set 2024-09-15T01:56:09.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.556+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193412e6f0 con 0x7f193409b020 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f193c699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f193c699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f193c699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193404bf60 con 0x7f193409b020 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1930039040 con 0x7f193409b020 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f192a7fc640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 0x7f191803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f193006c9c0 con 0x7f193409b020 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f1939c34640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 0x7f191803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.557+0000 7f1939c34640 10 monclient: get_auth_request con 0x7f1918038740 auth_method 0 2024-09-15T01:56:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.560+0000 7f1939c34640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 0x7f191803abf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.561+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f19300445d0 con 0x7f193409b020 2024-09-15T01:56:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.561+0000 7f192a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.561+0000 7f192a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.726+0000 7f193c699640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.726+0000 7f193c699640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.726+0000 7f193c699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:09.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:09.726+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f193412efa0 con 0x7f193409b020 2024-09-15T01:56:10.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.357+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v121) v1 ==== 115+0+0 (secure 0 0 0) 0x7f19300370a0 con 0x7f193409b020 2024-09-15T01:56:10.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.357+0000 7f192a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:10.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.357+0000 7f192a7fc640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-15T01:56:10.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.459+0000 7f193c699640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:10.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.459+0000 7f193c699640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:10.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.459+0000 7f193c699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.459+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f193412aea0 con 0x7f193409b020 2024-09-15T01:56:10.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.460+0000 7f192a7fc640 1 -- 172.21.15.67:0/3823261821 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v121) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1930040070 con 0x7f193409b020 2024-09-15T01:56:10.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.460+0000 7f192a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T01:56:10.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.460+0000 7f192a7fc640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-15T01:56:10.463 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'barset' already exists 2024-09-15T01:56:10.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 msgr2=0x7f191803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 0x7f191803abf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924008040 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 10 monclient: shutdown 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 msgr2=0x7f193406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1930009420 tx=0x7f1930032000 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 shutdown_connections 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1918038740 0x7f191803abf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19340a1a00 0x7f193412e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f193409d5a0 0x7f19341293e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 --2- 172.21.15.67:0/3823261821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193409b020 0x7f193406db30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 >> 172.21.15.67:0/3823261821 conn(0x7f1934071340 msgr2=0x7f1934099ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:10.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 shutdown_connections 2024-09-15T01:56:10.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.462+0000 7f193c699640 1 -- 172.21.15.67:0/3823261821 wait complete. 2024-09-15T01:56:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set barset size 3 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.548+0000 7fb6ca19d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.548+0000 7fb6ca19d640 10 monclient: build_initial_monmap 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.548+0000 7fb6ca19d640 10 monclient: monmap: 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:10.551 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:10.549867+0000 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:10.549867+0000 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:10.552 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 1 Processor -- start 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 1 -- start start 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 10 monclient: init 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c4098140 0x7fb6c41321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.549+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c4098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 0x7fb6c4098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c40a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6c4098a50 0x7fb6c40695c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.557 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c4098a50 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c40a1730 con 0x7fb6c40a0370 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c4098ef0 con 0x7fb6c4098a50 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c40a0b80 con 0x7fb6c4098140 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6ca19d640 10 monclient(hunting): authenticate will time out at 1003.019051s 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c2ffd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 msgr2=0x7fb6c4098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 0x7fb6c4098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:10.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6c4098a50 0x7fb6c40695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6c4098a50 0x7fb6c40695c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55776/0 (socket says 172.21.15.67:55776) 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 -- 172.21.15.67:0/3261348784 learned_addr learned my addr 172.21.15.67:0/3261348784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request con 0x7fb6c4098a50 auth_method 0 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c4098140 0x7fb6c41321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:10.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): get_auth_request con 0x7fb6c4098140 auth_method 0 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_done global_id 5695 payload 274 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 -- 172.21.15.67:0/3261348784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c4098140 msgr2=0x7fb6c41321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:10.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c4098140 0x7fb6c41321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 -- 172.21.15.67:0/3261348784 >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 msgr2=0x7fb6c4098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 --2- 172.21.15.67:0/3261348784 >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 0x7fb6c4098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 monclient: found mon.noname-c 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.550+0000 7fb6c3fff640 1 -- 172.21.15.67:0/3261348784 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40a00b0 con 0x7fb6c4098a50 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c3fff640 1 -- 172.21.15.67:0/3261348784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 msgr2=0x7fb6c40695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c3fff640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c40695c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40a1660 tx=0x7fb6bc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65eea7da96e30576 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 1 -- 172.21.15.67:0/3261348784 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc007610 con 0x7fb6c4098a50 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:10.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: dump: 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:10.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6ca19d640 5 monclient: authenticate success, global_id 5695 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6ca19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 1 -- 172.21.15.67:0/3261348784 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc02d050 con 0x7fb6c4098a50 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6ca19d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6ca19d640 4 set_mon_vals no callback set 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 1 -- 172.21.15.67:0/3261348784 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0079e0 con 0x7fb6c4098a50 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.553+0000 7fb6c27fc640 10 monclient: dump: 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 10 monclient: shutdown 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3261348784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 msgr2=0x7fb6c40695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c40695c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40a1660 tx=0x7fb6bc009840 comp rx=0 tx=0).stop 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3261348784 shutdown_connections 2024-09-15T01:56:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c40695c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3261348784 >> v2:172.21.15.17:3300/0 conn(0x7fb6c40a0370 0x7fb6c4098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3261348784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c4098140 0x7fb6c41321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3261348784 >> 172.21.15.67:0/3261348784 conn(0x7fb6c40a5780 msgr2=0x7fb6c40a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3261348784 shutdown_connections 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3261348784 wait complete. 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 10 monclient: build_initial_monmap 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.554+0000 7fb6ca19d640 10 monclient: monmap: 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.565 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:10.566 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:10.566 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:10.566 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 Processor -- start 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 -- start start 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient: init 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:10.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 0x7fb6c4130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c4098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c4098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a0370 0x7fb6c411dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c40a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c41213f0 con 0x7fb6c4098140 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4121080 con 0x7fb6c4098a50 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4099090 con 0x7fb6c40a0370 2024-09-15T01:56:10.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6ca19d640 10 monclient(hunting): authenticate will time out at 1003.024726s 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c37fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 msgr2=0x7fb6c4130c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c37fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 0x7fb6c4130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55778/0 (socket says 172.21.15.67:55778) 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 1 -- 172.21.15.67:0/3607465975 learned_addr learned my addr 172.21.15.67:0/3607465975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): get_auth_request con 0x7fb6c4098a50 auth_method 0 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c3fff640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a0370 0x7fb6c411dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request con 0x7fb6c40a0370 auth_method 0 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.555+0000 7fb6c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 10 monclient(hunting): handle_auth_done global_id 5698 payload 824 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 -- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a0370 msgr2=0x7fb6c411dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a0370 0x7fb6c411dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 -- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 msgr2=0x7fb6c4130c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 0x7fb6c4130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 monclient: found mon.noname-b 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.556+0000 7fb6c2ffd640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc009ae0 con 0x7fb6c4098a50 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6c2ffd640 1 -- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 msgr2=0x7fb6c411d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6c2ffd640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800b5e0 tx=0x7fb6b8006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8003d80 con 0x7fb6c4098a50 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: dump: 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:10.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b8006150 con 0x7fb6c4098a50 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 5 monclient: authenticate success, global_id 5698 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 10 monclient: _renew_subs 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c4141cb0 con 0x7fb6c4098a50 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8006510 con 0x7fb6c4098a50 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ab7fe640 10 monclient: dump: 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 10 monclient: _renew_subs 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c411feb0 con 0x7fb6c4098a50 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.558+0000 7fb6c1ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb6b80103e0 con 0x7fb6c4098a50 2024-09-15T01:56:10.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ca19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ca19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ca19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c404bf60 con 0x7fb6c4098a50 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ab7fe640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 0x7fb6b003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fb6b804bc70 con 0x7fb6c4098a50 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6c37fe640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 0x7fb6b003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.559+0000 7fb6c37fe640 10 monclient: get_auth_request con 0x7fb6b00386f0 auth_method 0 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.562+0000 7fb6c37fe640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 0x7fb6b003aba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009430 tx=0x7fb6b40090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.564+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6b800eac0 con 0x7fb6c4098a50 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.564+0000 7fb6ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.564+0000 7fb6ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.728+0000 7fb6ca19d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:10.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.728+0000 7fb6ca19d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:10.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.728+0000 7fb6ca19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:10.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:10.728+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fb6c4071470 con 0x7fb6c4098a50 2024-09-15T01:56:11.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.360+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v122) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb6b8017070 con 0x7fb6c4098a50 2024-09-15T01:56:11.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.360+0000 7fb6ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:11.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.360+0000 7fb6ab7fe640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-15T01:56:11.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.465+0000 7fb6ca19d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:11.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.465+0000 7fb6ca19d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:11.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.465+0000 7fb6ca19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:11.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:11.465+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fb6c406f000 con 0x7fb6c4098a50 2024-09-15T01:56:12.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.296+0000 7fb6ab7fe640 1 -- 172.21.15.67:0/3607465975 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v123) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb6b800e2e0 con 0x7fb6c4098a50 2024-09-15T01:56:12.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.296+0000 7fb6ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T01:56:12.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.296+0000 7fb6ab7fe640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-15T01:56:12.300 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 10 size to 3 2024-09-15T01:56:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 msgr2=0x7fb6b003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 0x7fb6b003aba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009430 tx=0x7fb6b40090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 10 monclient: shutdown 2024-09-15T01:56:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 msgr2=0x7fb6c411d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.301+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800b5e0 tx=0x7fb6b8006f90 comp rx=0 tx=0).stop 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 shutdown_connections 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6b00386f0 0x7fb6b003aba0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb6c40a0370 0x7fb6c411dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4098a50 0x7fb6c411d7e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 --2- 172.21.15.67:0/3607465975 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb6c4098140 0x7fb6c4130c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 >> 172.21.15.67:0/3607465975 conn(0x7fb6c40a5780 msgr2=0x7fb6c40a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 shutdown_connections 2024-09-15T01:56:12.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.302+0000 7fb6ca19d640 1 -- 172.21.15.67:0/3607465975 wait complete. 2024-09-15T01:56:12.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T01:56:12.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:56:12.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T01:56:12.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.399+0000 7f5aeb8f8640 10 monclient: get_monmap_and_config 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.399+0000 7f5aeb8f8640 10 monclient: build_initial_monmap 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.399+0000 7f5aeb8f8640 10 monclient: monmap: 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:12.400662+0000 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:12.400662+0000 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:12.402 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:12.403 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:12.403 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 Processor -- start 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 -- start start 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient: init 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ae40a0b50 0x7f5ae4069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae40a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ae409eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:12.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ae409f2a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae406a270 con 0x7f5ae409eea0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae40a0f20 con 0x7f5ae409f2a0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae406f290 con 0x7f5ae40a0b50 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.400+0000 7f5aeb8f8640 10 monclient(hunting): authenticate will time out at 1004.869915s 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ae40a0b50 0x7f5ae4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae8e93640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f5ae409eea0 legacy=0x7f5ae4131b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9e95640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ae409f2a0 0x7f5ae4132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59820/0 (socket says 172.21.15.67:59820) 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9e95640 1 -- 172.21.15.67:0/4159273412 learned_addr learned my addr 172.21.15.67:0/4159273412 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): get_auth_request con 0x7f5ae40a0b50 auth_method 0 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient(hunting): handle_auth_done global_id 5701 payload 274 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 1 -- 172.21.15.67:0/4159273412 >> v1:172.21.15.67:6789/0 conn(0x7f5ae409f2a0 legacy=0x7f5ae4132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 1 -- 172.21.15.67:0/4159273412 >> v1:172.21.15.17:6789/0 conn(0x7f5ae409eea0 legacy=0x7f5ae4131b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient: _finish_hunting 0 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 1 monclient: found mon.noname-c 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:12.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.401+0000 7f5ae9694640 1 -- 172.21.15.67:0/4159273412 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4132930 con 0x7f5ae40a0b50 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5ae9694640 1 -- 172.21.15.67:0/4159273412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40a0b50 msgr2=0x7f5ae4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5ae9694640 1 --2- 172.21.15.67:0/4159273412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4069340 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009850 tx=0x7f5ae00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=278295f33e7b5d11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 1 -- 172.21.15.67:0/4159273412 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0032440 con 0x7f5ae40a0b50 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: dump: 2024-09-15T01:56:12.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 5 monclient: authenticate success, global_id 5701 2024-09-15T01:56:12.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 1 -- 172.21.15.67:0/4159273412 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae00325a0 con 0x7f5ae40a0b50 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 4 set_mon_vals no callback set 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 1 -- 172.21.15.67:0/4159273412 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0032940 con 0x7f5ae40a0b50 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5adbfff640 10 monclient: dump: 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.416 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 10 monclient: shutdown 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/4159273412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40a0b50 msgr2=0x7f5ae4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 1 --2- 172.21.15.67:0/4159273412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4069340 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009850 tx=0x7f5ae00090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/4159273412 shutdown_connections 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 1 --2- 172.21.15.67:0/4159273412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4069340 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.404+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/4159273412 >> 172.21.15.67:0/4159273412 conn(0x7f5ae4071290 msgr2=0x7f5ae40a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:12.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/4159273412 shutdown_connections 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/4159273412 wait complete. 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 10 monclient: build_initial_monmap 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 10 monclient: monmap: 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:12.418 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 1 Processor -- start 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 1 -- start start 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.405+0000 7f5aeb8f8640 10 monclient: init 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 0x7f5ae409aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae409eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ae409f2a0 0x7f5ae4130da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae409f2a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae40a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:12.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae409b5e0 con 0x7f5ae40a0b50 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae4131a80 con 0x7f5ae409eea0 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae411dec0 con 0x7f5ae409f2a0 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5aeb8f8640 10 monclient(hunting): authenticate will time out at 1004.875575s 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 0x7f5ae409aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 0x7f5ae409aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55794/0 (socket says 172.21.15.67:55794) 2024-09-15T01:56:12.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 1 -- 172.21.15.67:0/2753627935 learned_addr learned my addr 172.21.15.67:0/2753627935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): get_auth_request con 0x7f5ae409eea0 auth_method 0 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9e95640 1 -- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ae40a0b50 msgr2=0x7f5ae4141250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9e95640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae8e93640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ae409f2a0 0x7f5ae4130da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae8e93640 10 monclient(hunting): get_auth_request con 0x7f5ae409f2a0 auth_method 0 2024-09-15T01:56:12.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae8e93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae8e93640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae8e93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 10 monclient(hunting): handle_auth_done global_id 5704 payload 824 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 1 -- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ae409f2a0 msgr2=0x7f5ae4130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.406+0000 7f5ae9694640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ae409f2a0 0x7f5ae4130da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 1 -- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ae40a0b50 msgr2=0x7f5ae4141250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ae40a0b50 0x7f5ae4141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 10 monclient: _finish_hunting 0 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 1 monclient: found mon.noname-b 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:12.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae9694640 1 -- 172.21.15.67:0/2753627935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0009390 con 0x7f5ae409eea0 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.407+0000 7f5ae8e93640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ae409f2a0 0x7f5ae4130da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ae9694640 1 -- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 msgr2=0x7f5ae409aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ae9694640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 0x7f5ae409aa90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009850 tx=0x7f5ae0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ae9694640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae409eea0 0x7f5ae409aa90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009850 tx=0x7f5ae0032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f5ae00318c0 mon_map magic: 0 v1 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00318c0 con 0x7f5ae409eea0 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:12.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 10 monclient: dump: 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:12.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae0031a20 con 0x7f5ae409eea0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5ad97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 5 monclient: authenticate success, global_id 5704 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 10 monclient: _renew_subs 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/2753627935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae411e1a0 con 0x7f5ae409eea0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5adb7fe640 4 set_mon_vals no callback set 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 10 monclient: _renew_subs 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.409+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/2753627935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae411c9f0 con 0x7f5ae409eea0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0031dc0 con 0x7f5ae409eea0 2024-09-15T01:56:12.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:12.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:12.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 10 monclient: dump: 2024-09-15T01:56:12.426 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:12.426 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:12.426 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ae0039040 con 0x7f5ae409eea0 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5aeb8f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5aeb8f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5aeb8f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/2753627935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae404bf60 con 0x7f5ae409eea0 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ad97fa640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ac8038740 0x7f5ac803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ae8e93640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ac8038740 0x7f5ac803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.410+0000 7f5ae8e93640 10 monclient: get_auth_request con 0x7f5ac8038740 auth_method 0 2024-09-15T01:56:12.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.413+0000 7f5ae8e93640 1 --2- 172.21.15.67:0/2753627935 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ac8038740 0x7f5ac803abf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4006fd0 tx=0x7f5ad4006f20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:12.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:12.415+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f5ae006c750 con 0x7f5ae409eea0 2024-09-15T01:56:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.032+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ae004b500 con 0x7f5ae409eea0 2024-09-15T01:56:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.032+0000 7f5ad97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.032+0000 7f5ad97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:13.056 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:56:13.056 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T01:56:13.056 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T01:56:13.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T01:56:13.060 INFO:tasks.ceph.mon.a:Started 2024-09-15T01:56:13.060 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:56:13.060 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:13.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.166+0000 7f7e7ab9d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:13.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.166+0000 7f7e7ab9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:13.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.166+0000 7f7e7ab9d640 10 monclient: monmap: 2024-09-15T01:56:13.168 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:13.168 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:13.167158+0000 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:13.167158+0000 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:13.169 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 Processor -- start 2024-09-15T01:56:13.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 -- start start 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient: init 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e74131b20 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e740a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e74132430 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740a0040 con 0x7f7e74131b20 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7409bff0 con 0x7f7e74132430 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740a14a0 con 0x7f7e740a0280 2024-09-15T01:56:13.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7ab9d640 10 monclient(hunting): authenticate will time out at 1033.163683s 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47400/0 (socket says 172.21.15.17:47400) 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47790/0 (socket says 172.21.15.17:47790) 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 -- 172.21.15.17:0/2444170080 learned_addr learned my addr 172.21.15.17:0/2444170080 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 10 monclient(hunting): get_auth_request con 0x7f7e74132430 auth_method 0 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e78939640 1 --1- 172.21.15.17:0/2444170080 >> v1:172.21.15.17:6789/0 conn(0x7f7e74131b20 0x7f7e7409eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39924/0 (socket says 172.21.15.17:39924) 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): get_auth_request con 0x7f7e740a0280 auth_method 0 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 -- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 msgr2=0x7f7e74138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 -- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 msgr2=0x7f7e74138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 --2- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.167+0000 7f7e7913a640 1 --2- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 10 monclient(hunting): handle_auth_done global_id 74285 payload 274 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 1 -- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 msgr2=0x7f7e74138e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 1 --2- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 1 -- 172.21.15.17:0/2444170080 >> v1:172.21.15.17:6789/0 conn(0x7f7e74131b20 legacy=0x7f7e7409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:13.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:13.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 1 monclient: found mon.noname-b 2024-09-15T01:56:13.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:13.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.168+0000 7f7e73fff640 1 -- 172.21.15.17:0/2444170080 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7409f3b0 con 0x7f7e740a0280 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e73fff640 1 -- 172.21.15.17:0/2444170080 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 msgr2=0x7f7e74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e73fff640 1 --2- 172.21.15.17:0/2444170080 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009850 tx=0x7f7e600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad397d827de1baa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 1 -- 172.21.15.17:0/2444170080 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6002adf0 con 0x7f7e740a0280 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: dump: 2024-09-15T01:56:13.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 1 -- 172.21.15.17:0/2444170080 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60032440 con 0x7f7e740a0280 2024-09-15T01:56:13.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 5 monclient: authenticate success, global_id 74285 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 1 -- 172.21.15.17:0/2444170080 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600327e0 con 0x7f7e740a0280 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 4 set_mon_vals no callback set 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e737fe640 10 monclient: dump: 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 10 monclient: shutdown 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/2444170080 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 msgr2=0x7f7e74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/2444170080 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009850 tx=0x7f7e600090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/2444170080 shutdown_connections 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/2444170080 >> v2:172.21.15.17:3300/0 conn(0x7f7e74132430 0x7f7e74138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/2444170080 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e740a0280 0x7f7e74131ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/2444170080 >> 172.21.15.17:0/2444170080 conn(0x7f7e74071270 msgr2=0x7f7e740a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:13.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/2444170080 shutdown_connections 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.171+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/2444170080 wait complete. 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.172+0000 7f7e7ab9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.172+0000 7f7e7ab9d640 10 monclient: monmap: 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:13.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.172+0000 7f7e7ab9d640 1 Processor -- start 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.172+0000 7f7e7ab9d640 1 -- start start 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.172+0000 7f7e7ab9d640 10 monclient: init 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e740a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e74131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e74132430 0x7f7e74133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e74132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74137b10 con 0x7f7e740a0280 2024-09-15T01:56:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74135360 con 0x7f7e74132430 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741359b0 con 0x7f7e74131b20 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7ab9d640 10 monclient(hunting): authenticate will time out at 1033.169499s 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47416/0 (socket says 172.21.15.17:47416) 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47794/0 (socket says 172.21.15.17:47794) 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 -- 172.21.15.17:0/519276404 learned_addr learned my addr 172.21.15.17:0/519276404 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 10 monclient(hunting): get_auth_request con 0x7f7e740a0280 auth_method 0 2024-09-15T01:56:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): get_auth_request con 0x7f7e74131b20 auth_method 0 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e74132430 0x7f7e74133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): get_auth_request con 0x7f7e74132430 auth_method 0 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 msgr2=0x7f7e741375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 msgr2=0x7f7e741375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e78939640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e7913a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient(hunting): handle_auth_done global_id 74288 payload 824 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e74132430 msgr2=0x7f7e74133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e74132430 0x7f7e74133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 msgr2=0x7f7e741375d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:13.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 monclient: found mon.noname-c 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.173+0000 7f7e73fff640 1 -- 172.21.15.17:0/519276404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e60009390 con 0x7f7e74131b20 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e73fff640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 msgr2=0x7f7e741333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e73fff640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e60034000 tx=0x7f7e60032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60031b10 con 0x7f7e74131b20 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: dump: 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: _finish_auth 0 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 5 monclient: authenticate success, global_id 74288 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 10 monclient: _renew_subs 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60031cb0 con 0x7f7e74131b20 2024-09-15T01:56:13.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e740a09b0 con 0x7f7e74131b20 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6003c650 con 0x7f7e74131b20 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e70ff9640 10 monclient: dump: 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e72ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 10 monclient: _renew_subs 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.176+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e74136470 con 0x7f7e74131b20 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7e6003ca40 con 0x7f7e74131b20 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e7ab9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e7ab9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e7ab9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e740724a0 con 0x7f7e74131b20 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e70ff9640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 0x7f7e5803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f7e6006d140 con 0x7f7e74131b20 2024-09-15T01:56:13.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e78939640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 0x7f7e5803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.177+0000 7f7e78939640 10 monclient: get_auth_request con 0x7f7e58038740 auth_method 0 2024-09-15T01:56:13.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.180+0000 7f7e78939640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 0x7f7e5803abf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7e68007d10 tx=0x7f7e68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.182+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7e60030b20 con 0x7f7e74131b20 2024-09-15T01:56:13.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.182+0000 7f7e70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.182+0000 7f7e70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:13.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5aeb8f8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:13.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5aeb8f8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:13.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5aeb8f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:13.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5aeb8f8640 1 -- 172.21.15.67:0/2753627935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f5ae404bf60 con 0x7f5ae409eea0 2024-09-15T01:56:13.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5ad97fa640 1 -- 172.21.15.67:0/2753627935 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v123) v1 ==== 160+0+0 (secure 0 0 0) 0x7f5ae003f020 con 0x7f5ae409eea0 2024-09-15T01:56:13.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5ad97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:13.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.208+0000 7f5ad97fa640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-15T01:56:13.211 INFO:tasks.workunit.client.0.smithi067.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-15T01:56:13.214 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:56:13.215 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 10 monclient: build_initial_monmap 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 10 monclient: monmap: 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:13.291242+0000 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:13.291242+0000 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:13.293 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 1 Processor -- start 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 1 -- start start 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.290+0000 7f934a24f640 10 monclient: init 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9344098140 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93440a0370 0x7f9344098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-c con 0x7f93440a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098a50 0x7f93441390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-b con 0x7f9344098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9344098ef0 con 0x7f9344098140 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93440a1730 con 0x7f93440a0370 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93440a0b80 con 0x7f9344098a50 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f934a24f640 10 monclient(hunting): authenticate will time out at 1005.760438s 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93440a0370 0x7f9344098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93440a0370 0x7f9344098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55810/0 (socket says 172.21.15.67:55810) 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 -- 172.21.15.67:0/4287012455 learned_addr learned my addr 172.21.15.67:0/4287012455 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): get_auth_request con 0x7f93440a0370 auth_method 0 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9343fff640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098a50 0x7f93441390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9343fff640 10 monclient(hunting): get_auth_request con 0x7f9344098a50 auth_method 0 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9343fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9343fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient(hunting): handle_auth_done global_id 5710 payload 274 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 -- 172.21.15.67:0/4287012455 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098a50 msgr2=0x7f93441390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098a50 0x7f93441390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 -- 172.21.15.67:0/4287012455 >> v1:172.21.15.17:6789/0 conn(0x7f9344098140 legacy=0x7f93441321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:13.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 monclient: found mon.noname-c 2024-09-15T01:56:13.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:13.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:13.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.291+0000 7f9342ffd640 1 -- 172.21.15.67:0/4287012455 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93441328f0 con 0x7f93440a0370 2024-09-15T01:56:13.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f9342ffd640 1 -- 172.21.15.67:0/4287012455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93440a0370 msgr2=0x7f9344098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:13.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f9342ffd640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93440a0370 0x7f9344098510 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9338009910 tx=0x7f93380091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9713464b088d1930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 1 -- 172.21.15.67:0/4287012455 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933802aea0 con 0x7f93440a0370 2024-09-15T01:56:13.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: dump: 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:13.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 1 -- 172.21.15.67:0/4287012455 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9338032440 con 0x7f93440a0370 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f934a24f640 5 monclient: authenticate success, global_id 5710 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f934a24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f934a24f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 1 -- 172.21.15.67:0/4287012455 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380327e0 con 0x7f93440a0370 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f934a24f640 4 set_mon_vals no callback set 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.294+0000 7f93427fc640 10 monclient: dump: 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.303 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 10 monclient: shutdown 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 -- 172.21.15.67:0/4287012455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93440a0370 msgr2=0x7f9344098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93440a0370 0x7f9344098510 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9338009910 tx=0x7f93380091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 -- 172.21.15.67:0/4287012455 shutdown_connections 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098a50 0x7f93441390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 --2- 172.21.15.67:0/4287012455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93440a0370 0x7f9344098510 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 -- 172.21.15.67:0/4287012455 >> 172.21.15.67:0/4287012455 conn(0x7f93440a5780 msgr2=0x7f93440a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 -- 172.21.15.67:0/4287012455 shutdown_connections 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.295+0000 7f934a24f640 1 -- 172.21.15.67:0/4287012455 wait complete. 2024-09-15T01:56:13.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.296+0000 7f934a24f640 10 monclient: build_initial_monmap 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.296+0000 7f934a24f640 10 monclient: monmap: 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.296+0000 7f934a24f640 1 Processor -- start 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.296+0000 7f934a24f640 1 -- start start 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.296+0000 7f934a24f640 10 monclient: init 2024-09-15T01:56:13.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-c con 0x7f9344098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9344098a50 0x7f934412f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-b con 0x7f9344098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): picked mon.noname-a con 0x7f93440a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9344132780 con 0x7f93440a0370 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9344130690 con 0x7f9344098a50 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93441392c0 con 0x7f9344098140 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:13.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f934a24f640 10 monclient(hunting): authenticate will time out at 1005.766621s 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9344098a50 0x7f934412f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:59614/0 (socket says 172.21.15.67:59614) 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 1 -- 172.21.15.67:0/881333436 learned_addr learned my addr 172.21.15.67:0/881333436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): get_auth_request con 0x7f9344098a50 auth_method 0 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): get_auth_request con 0x7f9344098140 auth_method 0 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 10 monclient(hunting): get_auth_request con 0x7f93440a0370 auth_method 0 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f93437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 msgr2=0x7f934412f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 msgr2=0x7f934412f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9342ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.297+0000 7f9343fff640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 10 monclient(hunting): handle_auth_done global_id 74294 payload 824 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9344098a50 msgr2=0x7f934412f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9344098a50 0x7f934412f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 msgr2=0x7f934412f690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 monclient: found mon.noname-c 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.298+0000 7f93437fe640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338009450 con 0x7f9344098140 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f93437fe640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 msgr2=0x7f9344132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f93437fe640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f933400c5e0 tx=0x7f9334007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334012840 con 0x7f9344098140 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: dump: 2024-09-15T01:56:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 5 monclient: authenticate success, global_id 74294 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93340129c0 con 0x7f9344098140 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 10 monclient: _renew_subs 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334012d80 con 0x7f9344098140 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9344131860 con 0x7f9344098140 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f932b7fe640 10 monclient: dump: 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:13.313 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f9341ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 10 monclient: _renew_subs 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.300+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9344099090 con 0x7f9344098140 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.301+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9334011590 con 0x7f9344098140 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.301+0000 7f934a24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.301+0000 7f934a24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.301+0000 7f934a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.301+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934404bf60 con 0x7f9344098140 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.302+0000 7f932b7fe640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 0x7f933003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.302+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f933404c630 con 0x7f9344098140 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.302+0000 7f9342ffd640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 0x7f933003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.302+0000 7f9342ffd640 10 monclient: get_auth_request con 0x7f9330038740 auth_method 0 2024-09-15T01:56:13.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.304+0000 7f9342ffd640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 0x7f933003abf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f93440a5340 tx=0x7f9338032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:13.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.310+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9334010920 con 0x7f9344098140 2024-09-15T01:56:13.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.310+0000 7f932b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:13.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.310+0000 7f932b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:13.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e7ab9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:13.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e7ab9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:13.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e7ab9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7e740990f0 con 0x7f7e74131b20 2024-09-15T01:56:13.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e70ff9640 1 -- 172.21.15.17:0/519276404 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f7e60030340 con 0x7f7e74131b20 2024-09-15T01:56:13.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:13.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.414+0000 7f7e70ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:13.419 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:13.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 msgr2=0x7f7e5803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 0x7f7e5803abf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7e68007d10 tx=0x7f7e68009040 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 10 monclient: shutdown 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 msgr2=0x7f7e741333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e60034000 tx=0x7f7e60032000 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 shutdown_connections 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7e58038740 0x7f7e5803abf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e74132430 0x7f7e74133930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7e74131b20 0x7f7e741333f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 --2- 172.21.15.17:0/519276404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7e740a0280 0x7f7e741375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 >> 172.21.15.17:0/519276404 conn(0x7f7e74071270 msgr2=0x7f7e740a4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 shutdown_connections 2024-09-15T01:56:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:13.416+0000 7f7e7ab9d640 1 -- 172.21.15.17:0/519276404 wait complete. 2024-09-15T01:56:13.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.480+0000 7f934a24f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:13.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.480+0000 7f934a24f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:13.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.480+0000 7f934a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:13.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:13.480+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f934409bd60 con 0x7f9344098140 2024-09-15T01:56:14.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.372+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v125) v1 ==== 109+0+0 (secure 0 0 0) 0x7f9334016020 con 0x7f9344098140 2024-09-15T01:56:14.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.372+0000 7f932b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.372+0000 7f932b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:14.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.480+0000 7f934a24f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:14.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.480+0000 7f934a24f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:14.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.480+0000 7f934a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:14.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.480+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f93441319f0 con 0x7f9344098140 2024-09-15T01:56:14.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.481+0000 7f932b7fe640 1 -- 172.21.15.67:0/881333436 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v125) v1 ==== 153+0+0 (secure 0 0 0) 0x7f9334010320 con 0x7f9344098140 2024-09-15T01:56:14.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.481+0000 7f932b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T01:56:14.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.481+0000 7f932b7fe640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-15T01:56:14.484 INFO:tasks.workunit.client.0.smithi067.stderr:weight-set for pool 'fooset' already created 2024-09-15T01:56:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 msgr2=0x7f933003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 0x7f933003abf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f93440a5340 tx=0x7f9338032000 comp rx=0 tx=0).stop 2024-09-15T01:56:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 10 monclient: shutdown 2024-09-15T01:56:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 msgr2=0x7f9344132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f933400c5e0 tx=0x7f9334007ed0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 shutdown_connections 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9330038740 0x7f933003abf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f93440a0370 0x7f934412f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9344098a50 0x7f934412f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 --2- 172.21.15.67:0/881333436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9344098140 0x7f9344132240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.483+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 >> 172.21.15.67:0/881333436 conn(0x7f93440a5780 msgr2=0x7f93440a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.484+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 shutdown_connections 2024-09-15T01:56:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.484+0000 7f934a24f640 1 -- 172.21.15.67:0/881333436 wait complete. 2024-09-15T01:56:14.496 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create barset positional 2024-09-15T01:56:14.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.572+0000 7f91bb20f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:14.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.572+0000 7f91bb20f640 10 monclient: build_initial_monmap 2024-09-15T01:56:14.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.572+0000 7f91bb20f640 10 monclient: monmap: 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:14.573552+0000 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:14.573552+0000 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:14.575 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 Processor -- start 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 -- start start 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient: init 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f91b4068dc0 0x7f91b4132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-a con 0x7f91b4068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b4069190 0x7f91b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-c con 0x7f91b4069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b412e960 0x7f91b412ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-b con 0x7f91b412e960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b40a14d0 con 0x7f91b4068dc0 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b4096c10 con 0x7f91b4069190 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b409ff20 con 0x7f91b412e960 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91bb20f640 10 monclient(hunting): authenticate will time out at 1007.042661s 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b4069190 0x7f91b4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b4069190 0x7f91b4069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55828/0 (socket says 172.21.15.67:55828) 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 1 -- 172.21.15.67:0/1982196762 learned_addr learned my addr 172.21.15.67:0/1982196762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): get_auth_request con 0x7f91b4069190 auth_method 0 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:14.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.573+0000 7f91b97ac640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b412e960 0x7f91b412ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 10 monclient(hunting): handle_auth_done global_id 5716 payload 274 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b8fab640 1 --2- 172.21.15.67:0/1982196762 >> v2:172.21.15.17:3300/0 conn(0x7f91b4068dc0 0x7f91b4132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 -- 172.21.15.67:0/1982196762 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b412e960 msgr2=0x7f91b412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b412e960 0x7f91b412ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 -- 172.21.15.67:0/1982196762 >> v2:172.21.15.17:3300/0 conn(0x7f91b4068dc0 msgr2=0x7f91b4132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 --2- 172.21.15.67:0/1982196762 >> v2:172.21.15.17:3300/0 conn(0x7f91b4068dc0 0x7f91b4132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 monclient: found mon.noname-c 2024-09-15T01:56:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.574+0000 7f91b3fff640 1 -- 172.21.15.67:0/1982196762 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b40a0180 con 0x7f91b4069190 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.576+0000 7f91b3fff640 1 -- 172.21.15.67:0/1982196762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b4069190 msgr2=0x7f91b4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.576+0000 7f91b3fff640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b4069190 0x7f91b4069590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f91b406a680 tx=0x7f91a00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f40e871f7b5bfa15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 1 -- 172.21.15.67:0/1982196762 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0032440 con 0x7f91b4069190 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: dump: 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.586 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 1 -- 172.21.15.67:0/1982196762 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a00325a0 con 0x7f91b4069190 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 5 monclient: authenticate success, global_id 5716 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:14.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 4 set_mon_vals no callback set 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 1 -- 172.21.15.67:0/1982196762 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0032940 con 0x7f91b4069190 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91b37fe640 10 monclient: dump: 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:14.588 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 10 monclient: shutdown 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 -- 172.21.15.67:0/1982196762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b4069190 msgr2=0x7f91b4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b4069190 0x7f91b4069590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f91b406a680 tx=0x7f91a00091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 -- 172.21.15.67:0/1982196762 shutdown_connections 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b4069190 0x7f91b4069590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 --2- 172.21.15.67:0/1982196762 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b412e960 0x7f91b412ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 --2- 172.21.15.67:0/1982196762 >> v2:172.21.15.17:3300/0 conn(0x7f91b4068dc0 0x7f91b4132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 -- 172.21.15.67:0/1982196762 >> 172.21.15.67:0/1982196762 conn(0x7f91b40711b0 msgr2=0x7f91b40a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 -- 172.21.15.67:0/1982196762 shutdown_connections 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.577+0000 7f91bb20f640 1 -- 172.21.15.67:0/1982196762 wait complete. 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.578+0000 7f91bb20f640 10 monclient: build_initial_monmap 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.578+0000 7f91bb20f640 10 monclient: monmap: 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:14.590 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.578+0000 7f91bb20f640 1 Processor -- start 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.578+0000 7f91bb20f640 1 -- start start 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.578+0000 7f91bb20f640 10 monclient: init 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b4068dc0 0x7f91b4141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-c con 0x7f91b4068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-a con 0x7f91b4069190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): picked mon.noname-b con 0x7f91b412e960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40a0890 con 0x7f91b4069190 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b409b670 con 0x7f91b412e960 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b411bbf0 con 0x7f91b4068dc0 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91bb20f640 10 monclient(hunting): authenticate will time out at 1007.048192s 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55840/0 (socket says 172.21.15.67:55840) 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b4068dc0 0x7f91b4141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 -- 172.21.15.67:0/3372439920 learned_addr learned my addr 172.21.15.67:0/3372439920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): get_auth_request con 0x7f91b412e960 auth_method 0 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): get_auth_request con 0x7f91b4068dc0 auth_method 0 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:14.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 10 monclient(hunting): get_auth_request con 0x7f91b4069190 auth_method 0 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b8fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient(hunting): handle_auth_done global_id 5719 payload 824 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 msgr2=0x7f91b412dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 msgr2=0x7f91b412dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b4068dc0 msgr2=0x7f91b4141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b4068dc0 0x7f91b4141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 msgr2=0x7f91b412dac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:14.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b3fff640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient: _finish_hunting 0 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 monclient: found mon.noname-b 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.579+0000 7f91b97ac640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0009450 con 0x7f91b412e960 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.581+0000 7f91b97ac640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 msgr2=0x7f91b4098f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.581+0000 7f91b97ac640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00ceb0 tx=0x7f91ac007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac00e610 con 0x7f91b412e960 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: dump: 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:14.595 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac00e770 con 0x7f91b412e960 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 5 monclient: authenticate success, global_id 5719 2024-09-15T01:56:14.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 10 monclient: _renew_subs 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b411bed0 con 0x7f91b412e960 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac00eb30 con 0x7f91b412e960 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b0ff9640 10 monclient: dump: 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:14.597 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91b2ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 10 monclient: _renew_subs 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.582+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b4141db0 con 0x7f91b412e960 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91bb20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91bb20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91bb20f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f91ac009030 con 0x7f91b412e960 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b404bf60 con 0x7f91b412e960 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91b0ff9640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 0x7f919803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f91ac012070 con 0x7f91b412e960 2024-09-15T01:56:14.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91b8fab640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 0x7f919803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:14.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.583+0000 7f91b8fab640 10 monclient: get_auth_request con 0x7f9198038740 auth_method 0 2024-09-15T01:56:14.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.586+0000 7f91b8fab640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 0x7f919803abf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f91a800ac20 tx=0x7f91a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:14.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.587+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f91ac052020 con 0x7f91b412e960 2024-09-15T01:56:14.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.587+0000 7f91b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:14.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.587+0000 7f91b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.756+0000 7f91bb20f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.756+0000 7f91bb20f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.756+0000 7f91bb20f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:14.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:14.756+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f91b4069ad0 con 0x7f91b412e960 2024-09-15T01:56:15.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.380+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v126) v1 ==== 115+0+0 (secure 0 0 0) 0x7f91ac01d910 con 0x7f91b412e960 2024-09-15T01:56:15.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.380+0000 7f91b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:15.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.380+0000 7f91b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.485+0000 7f91bb20f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.485+0000 7f91bb20f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.485+0000 7f91bb20f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.485+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f91b406f000 con 0x7f91b412e960 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.486+0000 7f91b0ff9640 1 -- 172.21.15.67:0/3372439920 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v126) v1 ==== 159+0+0 (secure 0 0 0) 0x7f91ac014330 con 0x7f91b412e960 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.486+0000 7f91b0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T01:56:15.488 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.486+0000 7f91b0ff9640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-15T01:56:15.489 INFO:tasks.workunit.client.0.smithi067.stderr:weight-set for pool 'barset' already created 2024-09-15T01:56:15.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 msgr2=0x7f919803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 0x7f919803abf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f91a800ac20 tx=0x7f91a8008040 comp rx=0 tx=0).stop 2024-09-15T01:56:15.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 10 monclient: shutdown 2024-09-15T01:56:15.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:15.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 msgr2=0x7f91b4098f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00ceb0 tx=0x7f91ac007870 comp rx=0 tx=0).stop 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 shutdown_connections 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9198038740 0x7f919803abf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f91b4069190 0x7f91b412dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.487+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b412e960 0x7f91b4098f60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.488+0000 7f91bb20f640 1 --2- 172.21.15.67:0/3372439920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f91b4068dc0 0x7f91b4141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.488+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 >> 172.21.15.67:0/3372439920 conn(0x7f91b40711b0 msgr2=0x7f91b4130240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.488+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 shutdown_connections 2024-09-15T01:56:15.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.488+0000 7f91bb20f640 1 -- 172.21.15.67:0/3372439920 wait complete. 2024-09-15T01:56:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep fooset 2024-09-15T01:56:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 10 monclient: monmap: 2024-09-15T01:56:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:15.574277+0000 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:15.574277+0000 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:15.576 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 1 Processor -- start 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 1 -- start start 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.573+0000 7f4493d5d640 10 monclient: init 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T01:56:15.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 0x7f448c132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f448c131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f448c098420 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f448c068e70 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0a13e0 con 0x7f448c068e70 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0a0440 con 0x7f448c098420 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0a01e0 con 0x7f448c131e70 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:15.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4493d5d640 10 monclient(hunting): authenticate will time out at 1008.043404s 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55850/0 (socket says 172.21.15.67:55850) 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 -- 172.21.15.67:0/875782851 learned_addr learned my addr 172.21.15.67:0/875782851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 10 monclient(hunting): get_auth_request con 0x7f448c098420 auth_method 0 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 msgr2=0x7f448c098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 msgr2=0x7f448c098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44912f8640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:15.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 1 --2- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 0x7f448c132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): get_auth_request con 0x7f448c131e70 auth_method 0 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44922fa640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44922fa640 10 monclient(hunting): get_auth_request con 0x7f448c068e70 auth_method 0 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44922fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44922fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f44922fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.574+0000 7f4491af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f44922fa640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 msgr2=0x7f448c0692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f44922fa640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 msgr2=0x7f448c0692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f44922fa640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f44922fa640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 10 monclient(hunting): handle_auth_done global_id 74300 payload 274 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 msgr2=0x7f448c098850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 -- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 msgr2=0x7f448c0692f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 10 monclient: _finish_hunting 0 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 monclient: found mon.noname-b 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.575+0000 7f4491af9640 1 -- 172.21.15.67:0/875782851 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0a19b0 con 0x7f448c131e70 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4491af9640 1 -- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 msgr2=0x7f448c132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4491af9640 1 --2- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 0x7f448c132240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f448c071b00 tx=0x7f44800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21210a4afbd1fff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 1 -- 172.21.15.67:0/875782851 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448002adf0 con 0x7f448c131e70 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: dump: 2024-09-15T01:56:15.588 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: _finish_auth 0 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 5 monclient: authenticate success, global_id 74300 2024-09-15T01:56:15.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 1 -- 172.21.15.67:0/875782851 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4480032440 con 0x7f448c131e70 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 4 set_mon_vals no callback set 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 1 -- 172.21.15.67:0/875782851 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480032820 con 0x7f448c131e70 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4490af7640 10 monclient: dump: 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.590 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 10 monclient: shutdown 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 1 -- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 msgr2=0x7f448c132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.578+0000 7f4493d5d640 1 --2- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 0x7f448c132240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f448c071b00 tx=0x7f44800090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 -- 172.21.15.67:0/875782851 shutdown_connections 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.67:3300/0 conn(0x7f448c098420 0x7f448c098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 --2- 172.21.15.67:0/875782851 >> v2:172.21.15.17:3300/0 conn(0x7f448c068e70 0x7f448c0692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 --2- 172.21.15.67:0/875782851 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c131e70 0x7f448c132240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 -- 172.21.15.67:0/875782851 >> 172.21.15.67:0/875782851 conn(0x7f448c0a5c10 msgr2=0x7f448c0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 -- 172.21.15.67:0/875782851 shutdown_connections 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 -- 172.21.15.67:0/875782851 wait complete. 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 10 monclient: monmap: 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:15.592 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 Processor -- start 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 1 -- start start 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.579+0000 7f4493d5d640 10 monclient: init 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c068e70 0x7f448c12e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f448c068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f448c098420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f448c131e70 0x7f448c099550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f448c131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0a0d80 con 0x7f448c131e70 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0a3760 con 0x7f448c098420 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c11bdd0 con 0x7f448c068e70 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4493d5d640 10 monclient(hunting): authenticate will time out at 1008.049376s 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55856/0 (socket says 172.21.15.67:55856) 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 -- 172.21.15.67:0/3702831472 learned_addr learned my addr 172.21.15.67:0/3702831472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): get_auth_request con 0x7f448c098420 auth_method 0 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient(hunting): handle_auth_done global_id 5725 payload 824 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f4491af9640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c068e70 0x7f448c12e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 -- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c068e70 msgr2=0x7f448c12e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c068e70 0x7f448c12e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 -- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f448c131e70 msgr2=0x7f448c099550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f448c131e70 0x7f448c099550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient: _finish_hunting 0 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 monclient: found mon.noname-b 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:15.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.580+0000 7f44912f8640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4480009390 con 0x7f448c098420 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f44912f8640 1 -- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 msgr2=0x7f448c099010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f44912f8640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f448800e870 tx=0x7f4488008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448800c6d0 con 0x7f448c098420 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: dump: 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.596 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448800c830 con 0x7f448c098420 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 5 monclient: authenticate success, global_id 5725 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 10 monclient: _renew_subs 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c11bb50 con 0x7f448c098420 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:15.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448800cbf0 con 0x7f448c098420 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447dffb640 10 monclient: dump: 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.598 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 10 monclient: _renew_subs 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f447ffff640 4 set_mon_vals no callback set 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.583+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c141f50 con 0x7f448c098420 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4488013410 con 0x7f448c098420 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4493d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4493d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4493d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448c11bf60 con 0x7f448c098420 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f447dffb640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 0x7f447003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f448804c880 con 0x7f448c098420 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4491af9640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 0x7f447003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.584+0000 7f4491af9640 10 monclient: get_auth_request con 0x7f4470038740 auth_method 0 2024-09-15T01:56:15.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.587+0000 7f4491af9640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 0x7f447003abf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4480034000 tx=0x7f4480032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.588+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f44880136c0 con 0x7f448c098420 2024-09-15T01:56:15.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.588+0000 7f447dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.588+0000 7f447dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:15.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f4493d5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f4493d5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f4493d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f448c0639e0 con 0x7f448c098420 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v126) v1 ==== 71+0+14 (secure 0 0 0) 0x7f4488024350 con 0x7f448c098420 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f447dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.763+0000 7f447dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:15.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f4493d5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f4493d5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f4493d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f448c067a50 con 0x7f448c098420 2024-09-15T01:56:15.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f447dffb640 1 -- 172.21.15.67:0/3702831472 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v126) v1 ==== 71+0+14 (secure 0 0 0) 0x7f4488023390 con 0x7f448c098420 2024-09-15T01:56:15.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f447dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:15.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.869+0000 7f447dffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:15.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 msgr2=0x7f447003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 0x7f447003abf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4480034000 tx=0x7f4480032000 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 10 monclient: shutdown 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 msgr2=0x7f448c099010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f448800e870 tx=0x7f4488008040 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 shutdown_connections 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4470038740 0x7f447003abf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f448c131e70 0x7f448c099550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f448c098420 0x7f448c099010 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 --2- 172.21.15.67:0/3702831472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f448c068e70 0x7f448c12e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 >> 172.21.15.67:0/3702831472 conn(0x7f448c0a5c10 msgr2=0x7f448c097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 shutdown_connections 2024-09-15T01:56:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.871+0000 7f4493d5d640 1 -- 172.21.15.67:0/3702831472 wait complete. 2024-09-15T01:56:15.882 INFO:tasks.workunit.client.0.smithi067.stdout:fooset 2024-09-15T01:56:15.882 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:15.882 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep barset 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.955+0000 7f9c6849f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.955+0000 7f9c6849f640 10 monclient: build_initial_monmap 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.955+0000 7f9c6849f640 10 monclient: monmap: 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:15.956901+0000 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:15.956901+0000 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.958 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:15.959 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:15.959 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:15.959 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:15.959 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:15.959 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.956+0000 7f9c6849f640 1 Processor -- start 2024-09-15T01:56:15.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.956+0000 7f9c6849f640 1 -- start start 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.956+0000 7f9c6849f640 10 monclient: init 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.956+0000 7f9c6849f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.956+0000 7f9c6849f640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e220 0x7f9c6009e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6006e220 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9c6006e5f0 0x7f9c6006e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c6006e5f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6012f9e0 0x7f9c6012fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6012f9e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6009eee0 con 0x7f9c6006e5f0 2024-09-15T01:56:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6006c500 con 0x7f9c6012f9e0 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600a4080 con 0x7f9c6006e220 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6849f640 10 monclient(hunting): authenticate will time out at 1008.426099s 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6012f9e0 0x7f9c6012fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6012f9e0 0x7f9c6012fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55868/0 (socket says 172.21.15.67:55868) 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 -- 172.21.15.67:0/4275810984 learned_addr learned my addr 172.21.15.67:0/4275810984 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): get_auth_request con 0x7f9c6012f9e0 auth_method 0 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e220 0x7f9c6009e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): get_auth_request con 0x7f9c6006e220 auth_method 0 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c65a3a640 1 --2- 172.21.15.67:0/4275810984 >> v2:172.21.15.17:3300/0 conn(0x7f9c6006e5f0 0x7f9c6006e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c65a3a640 10 monclient(hunting): get_auth_request con 0x7f9c6006e5f0 auth_method 0 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c65a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c65a3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c65a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c6623b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_done global_id 5728 payload 274 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 -- 172.21.15.67:0/4275810984 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e220 msgr2=0x7f9c6009e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e220 0x7f9c6009e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 -- 172.21.15.67:0/4275810984 >> v2:172.21.15.17:3300/0 conn(0x7f9c6006e5f0 msgr2=0x7f9c6006e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/4275810984 >> v2:172.21.15.17:3300/0 conn(0x7f9c6006e5f0 0x7f9c6006e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 monclient: found mon.noname-c 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.957+0000 7f9c66a3c640 1 -- 172.21.15.67:0/4275810984 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6006c220 con 0x7f9c6012f9e0 2024-09-15T01:56:15.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c66a3c640 1 -- 172.21.15.67:0/4275810984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 msgr2=0x7f9c6012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6012fe40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9c6009d7c0 tx=0x7f9c58009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0ae8be1a40c3a3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 1 -- 172.21.15.67:0/4275810984 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58007610 con 0x7f9c6012f9e0 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: dump: 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.972 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: _finish_auth 0 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 1 -- 172.21.15.67:0/4275810984 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5802d050 con 0x7f9c6012f9e0 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 5 monclient: authenticate success, global_id 5728 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 4 set_mon_vals no callback set 2024-09-15T01:56:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 1 -- 172.21.15.67:0/4275810984 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c580079e0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c65239640 10 monclient: dump: 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 10 monclient: shutdown 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 1 -- 172.21.15.67:0/4275810984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 msgr2=0x7f9c6012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.960+0000 7f9c6849f640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6012fe40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9c6009d7c0 tx=0x7f9c58009840 comp rx=0 tx=0).stop 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 -- 172.21.15.67:0/4275810984 shutdown_connections 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6012fe40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 --2- 172.21.15.67:0/4275810984 >> v2:172.21.15.17:3300/0 conn(0x7f9c6006e5f0 0x7f9c6006e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 --2- 172.21.15.67:0/4275810984 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e220 0x7f9c6009e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 -- 172.21.15.67:0/4275810984 >> 172.21.15.67:0/4275810984 conn(0x7f9c60071230 msgr2=0x7f9c600a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 -- 172.21.15.67:0/4275810984 shutdown_connections 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 -- 172.21.15.67:0/4275810984 wait complete. 2024-09-15T01:56:15.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 10 monclient: build_initial_monmap 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 10 monclient: monmap: 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 Processor -- start 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 1 -- start start 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.961+0000 7f9c6849f640 10 monclient: init 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c6006e220 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e5f0 0x7f9c6011b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6006e5f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6012f9e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6006ef40 con 0x7f9c6006e220 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600a11e0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6012f7d0 con 0x7f9c6006e5f0 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6849f640 10 monclient(hunting): authenticate will time out at 1008.431420s 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55874/0 (socket says 172.21.15.67:55874) 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 -- 172.21.15.67:0/2406678286 learned_addr learned my addr 172.21.15.67:0/2406678286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): get_auth_request con 0x7f9c6012f9e0 auth_method 0 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e5f0 0x7f9c6011b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): get_auth_request con 0x7f9c6006e5f0 auth_method 0 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 10 monclient(hunting): get_auth_request con 0x7f9c6006e220 auth_method 0 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c65a3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 msgr2=0x7f9c6012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 msgr2=0x7f9c6012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c6623b640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient(hunting): handle_auth_done global_id 5731 payload 824 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e5f0 msgr2=0x7f9c6011b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e5f0 0x7f9c6011b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 msgr2=0x7f9c6012ea30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:15.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 monclient: found mon.noname-b 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.962+0000 7f9c66a3c640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009ae0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c66a3c640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 msgr2=0x7f9c6011ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c66a3c640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9c5802abb0 tx=0x7f9c58036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58031cd0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: dump: 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c58031e30 con 0x7f9c6012f9e0 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 5 monclient: authenticate success, global_id 5731 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 10 monclient: _renew_subs 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c60130e70 con 0x7f9c6012f9e0 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5803c650 con 0x7f9c6012f9e0 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c567fc640 10 monclient: dump: 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:15.983 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 10 monclient: _renew_subs 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6011dc20 con 0x7f9c6012f9e0 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.965+0000 7f9c64a38640 4 set_mon_vals no callback set 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9c5803c9c0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6849f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6849f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6849f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:15.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c600639e0 con 0x7f9c6012f9e0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c567fc640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 0x7f9c4403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f9c5806d920 con 0x7f9c6012f9e0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6623b640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 0x7f9c4403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.966+0000 7f9c6623b640 10 monclient: get_auth_request con 0x7f9c44038740 auth_method 0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.968+0000 7f9c6623b640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 0x7f9c4403abf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009490 tx=0x7f9c5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.970+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9c5803cc70 con 0x7f9c6012f9e0 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.970+0000 7f9c567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:15.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:15.970+0000 7f9c567fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.143+0000 7f9c6849f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.143+0000 7f9c6849f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.143+0000 7f9c6849f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.143+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f9c6009e680 con 0x7f9c6012f9e0 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.144+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v126) v1 ==== 71+0+14 (secure 0 0 0) 0x7f9c58040030 con 0x7f9c6012f9e0 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.144+0000 7f9c567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.144+0000 7f9c567fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c6849f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c6849f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c6849f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f9c6011d050 con 0x7f9c6012f9e0 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c567fc640 1 -- 172.21.15.67:0/2406678286 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v126) v1 ==== 71+0+14 (secure 0 0 0) 0x7f9c58040070 con 0x7f9c6012f9e0 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c567fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:16.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.247+0000 7f9c567fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 msgr2=0x7f9c4403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 0x7f9c4403abf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009490 tx=0x7f9c5c008040 comp rx=0 tx=0).stop 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 10 monclient: shutdown 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 msgr2=0x7f9c6011ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9c5802abb0 tx=0x7f9c58036000 comp rx=0 tx=0).stop 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 shutdown_connections 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9c44038740 0x7f9c4403abf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6012f9e0 0x7f9c6011ba50 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c6006e5f0 0x7f9c6011b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 --2- 172.21.15.67:0/2406678286 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c6006e220 0x7f9c6012ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 >> 172.21.15.67:0/2406678286 conn(0x7f9c60071230 msgr2=0x7f9c6009c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 shutdown_connections 2024-09-15T01:56:16.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.249+0000 7f9c6849f640 1 -- 172.21.15.67:0/2406678286 wait complete. 2024-09-15T01:56:16.260 INFO:tasks.workunit.client.0.smithi067.stdout:barset 2024-09-15T01:56:16.260 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set dump 2024-09-15T01:56:16.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 10 monclient: get_monmap_and_config 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 10 monclient: monmap: 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:16.335219+0000 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:16.335219+0000 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:16.337 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:16.338 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:16.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 1 Processor -- start 2024-09-15T01:56:16.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 1 -- start start 2024-09-15T01:56:16.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.334+0000 7fedf86c0640 10 monclient: init 2024-09-15T01:56:16.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T01:56:16.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedf0131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fedf0098420 0x7fedf0098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-a con 0x7fedf0098420 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedf01371a0 0x7fedf0139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-c con 0x7fedf01371a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf00a13e0 con 0x7fedf0131e70 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf00a0440 con 0x7fedf0098420 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf00a01e0 con 0x7fedf01371a0 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf86c0640 10 monclient(hunting): authenticate will time out at 1008.804350s 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedf01371a0 0x7fedf0139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf645c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fedf0131e70 0x7fedf0132240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59828/0 (socket says 172.21.15.67:59828) 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf645c640 1 -- 172.21.15.67:0/2369066368 learned_addr learned my addr 172.21.15.67:0/2369066368 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): get_auth_request con 0x7fedf01371a0 auth_method 0 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_done global_id 5734 payload 274 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 -- 172.21.15.67:0/2369066368 >> v2:172.21.15.17:3300/0 conn(0x7fedf0098420 msgr2=0x7fedf0098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 --2- 172.21.15.67:0/2369066368 >> v2:172.21.15.17:3300/0 conn(0x7fedf0098420 0x7fedf0098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 -- 172.21.15.67:0/2369066368 >> v1:172.21.15.67:6789/0 conn(0x7fedf0131e70 legacy=0x7fedf0132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 monclient: found mon.noname-c 2024-09-15T01:56:16.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.335+0000 7fedf6c5d640 1 -- 172.21.15.67:0/2369066368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00a19b0 con 0x7fedf01371a0 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.338+0000 7fedf6c5d640 1 -- 172.21.15.67:0/2369066368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 msgr2=0x7fedf0139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.338+0000 7fedf6c5d640 1 --2- 172.21.15.67:0/2369066368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0139580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fedf0132a10 tx=0x7fede8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5281cbc644f8b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.338+0000 7fedf545a640 1 -- 172.21.15.67:0/2369066368 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8007840 con 0x7fedf01371a0 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: dump: 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.346 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 1 -- 172.21.15.67:0/2369066368 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede802d050 con 0x7fedf01371a0 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 5 monclient: authenticate success, global_id 5734 2024-09-15T01:56:16.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 1 -- 172.21.15.67:0/2369066368 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8007c10 con 0x7fedf01371a0 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 4 set_mon_vals no callback set 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf545a640 10 monclient: dump: 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.348 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 10 monclient: shutdown 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 -- 172.21.15.67:0/2369066368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 msgr2=0x7fedf0139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 --2- 172.21.15.67:0/2369066368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0139580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fedf0132a10 tx=0x7fede8009780 comp rx=0 tx=0).stop 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 -- 172.21.15.67:0/2369066368 shutdown_connections 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 --2- 172.21.15.67:0/2369066368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0139580 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 --2- 172.21.15.67:0/2369066368 >> v2:172.21.15.17:3300/0 conn(0x7fedf0098420 0x7fedf0098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 -- 172.21.15.67:0/2369066368 >> 172.21.15.67:0/2369066368 conn(0x7fedf00a5c10 msgr2=0x7fedf00a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 -- 172.21.15.67:0/2369066368 shutdown_connections 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.339+0000 7fedf86c0640 1 -- 172.21.15.67:0/2369066368 wait complete. 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient: monmap: 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 Processor -- start 2024-09-15T01:56:16.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 -- start start 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient: init 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-a con 0x7fedf0098420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fedf0131e70 0x7fedf0134620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-c con 0x7fedf0131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): picked mon.noname-b con 0x7fedf01371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00a0d80 con 0x7fedf0098420 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0069dd0 con 0x7fedf01371a0 2024-09-15T01:56:16.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0135a60 con 0x7fedf0131e70 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf86c0640 10 monclient(hunting): authenticate will time out at 1008.809873s 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf6c5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.340+0000 7fedf6c5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55898/0 (socket says 172.21.15.67:55898) 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 -- 172.21.15.67:0/128649273 learned_addr learned my addr 172.21.15.67:0/128649273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): get_auth_request con 0x7fedf01371a0 auth_method 0 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fedf0131e70 0x7fedf0134620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 10 monclient(hunting): get_auth_request con 0x7fedf0098420 auth_method 0 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): get_auth_request con 0x7fedf0131e70 auth_method 0 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 msgr2=0x7fedf01340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 msgr2=0x7fedf01340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf5c5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient(hunting): handle_auth_done global_id 5737 payload 824 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf645c640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fedf0131e70 msgr2=0x7fedf0134620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fedf0131e70 0x7fedf0134620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 msgr2=0x7fedf01340e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 monclient: found mon.noname-b 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.341+0000 7fedf6c5d640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede8009a20 con 0x7fedf01371a0 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf6c5d640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 msgr2=0x7fedf0134b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf6c5d640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fede80097c0 tx=0x7fede8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8031cd0 con 0x7fedf01371a0 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: dump: 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.355 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 5 monclient: authenticate success, global_id 5737 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 10 monclient: _renew_subs 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8040070 con 0x7fedf01371a0 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf011bf40 con 0x7fedf01371a0 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede80456b0 con 0x7fedf01371a0 2024-09-15T01:56:16.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fede67fc640 10 monclient: dump: 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf4c59640 4 set_mon_vals no callback set 2024-09-15T01:56:16.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 10 monclient: _renew_subs 2024-09-15T01:56:16.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.343+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf01365c0 con 0x7fedf01371a0 2024-09-15T01:56:16.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fede8042030 con 0x7fedf01371a0 2024-09-15T01:56:16.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fedf86c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fedf86c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fedf86c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf01367b0 con 0x7fedf01371a0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fede67fc640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 0x7fedd403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.344+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fede806dc00 con 0x7fedf01371a0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.345+0000 7fedf645c640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 0x7fedd403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.345+0000 7fedf645c640 10 monclient: get_auth_request con 0x7fedd4038740 auth_method 0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.347+0000 7fedf645c640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 0x7fedd403abf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fedec007d10 tx=0x7fedec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.349+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fede8031e30 con 0x7fedf01371a0 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.349+0000 7fede67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.349+0000 7fede67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:16.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fedf86c0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fedf86c0640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fedf86c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fedf00639e0 con 0x7fedf01371a0 2024-09-15T01:56:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v126) v1 ==== 73+0+6748 (secure 0 0 0) 0x7fede804a030 con 0x7fedf01371a0 2024-09-15T01:56:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fede67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.522+0000 7fede67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.535+0000 7f4d0395d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.535+0000 7f4d0395d640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.535+0000 7f4d0395d640 10 monclient: monmap: 2024-09-15T01:56:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:16.536669+0000 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:16.536669+0000 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:16.539 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.536+0000 7f4d0395d640 1 Processor -- start 2024-09-15T01:56:16.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.536+0000 7f4d0395d640 1 -- start start 2024-09-15T01:56:16.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.536+0000 7f4d0395d640 10 monclient: init 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc0a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cfc0980a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cfc136dd0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0a01e0 con 0x7f4cfc136dd0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0a1580 con 0x7f4cfc0980a0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cfc0a5c80 con 0x7f4cfc0a58b0 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d0395d640 10 monclient(hunting): authenticate will time out at 1036.533404s 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d01efa640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4cfc136dd0 0x7f4cfc1391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39926/0 (socket says 172.21.15.17:39926) 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47432/0 (socket says 172.21.15.17:47432) 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d01efa640 1 -- 172.21.15.17:0/1161840902 learned_addr learned my addr 172.21.15.17:0/1161840902 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 10 monclient(hunting): get_auth_request con 0x7f4cfc0a58b0 auth_method 0 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 msgr2=0x7f4cfc132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 msgr2=0x7f4cfc132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.537+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:16.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4d01efa640 1 -- 172.21.15.17:0/1161840902 >> v1:172.21.15.17:6789/0 conn(0x7f4cfc136dd0 legacy=0x7f4cfc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4d00ef8640 1 -- 172.21.15.17:0/1161840902 >> v1:172.21.15.67:6789/0 conn(0x7f4cfc0980a0 legacy=0x7f4cfc0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893816642 0 0) 0x7f4cfc0a1580 con 0x7f4cfc0980a0 2024-09-15T01:56:16.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4cfbfff640 10 monclient(hunting): my global_id is 5740 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.539+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce40034e0 con 0x7f4cfc0980a0 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075074243 0 0) 0x7f4ce40034e0 con 0x7f4cfc0980a0 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 msgr2=0x7f4cfc132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 --2- 172.21.15.17:0/1161840902 >> v2:172.21.15.17:3300/0 conn(0x7f4cfc0a58b0 0x7f4cfc132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 >> v1:172.21.15.17:6789/0 conn(0x7f4cfc136dd0 legacy=0x7f4cfc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc098bb0 con 0x7f4cfc0980a0 2024-09-15T01:56:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4cf0001830 con 0x7f4cfc0980a0 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: dump: 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.549 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 >> v1:172.21.15.67:6789/0 conn(0x7f4cfc0980a0 legacy=0x7f4cfc0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 0x7f4ce4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): picked mon.b con 0x7f4ce4003a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ce4004780 0x7f4ce4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): picked mon.c con 0x7f4ce4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4005250 0x7f4ce400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): picked mon.a con 0x7f4ce4005250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0a5c80 con 0x7f4ce4005250 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce400c210 con 0x7f4ce4003a50 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce400c3d0 con 0x7f4ce4004780 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4cfbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4005250 0x7f4ce400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 0x7f4ce4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): get_auth_request con 0x7f4ce4005250 auth_method 0 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ce4004780 0x7f4ce4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): get_auth_request con 0x7f4ce4003a50 auth_method 0 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): get_auth_request con 0x7f4ce4004780 auth_method 0 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient(hunting): handle_auth_done global_id 5740 payload 293 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ce4004780 msgr2=0x7f4ce4004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ce4004780 0x7f4ce4004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4005250 msgr2=0x7f4ce400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4005250 0x7f4ce400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 monclient: found mon.b 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40034e0 con 0x7f4ce4003a50 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d016f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d0395d640 5 monclient: authenticate success, global_id 5740 2024-09-15T01:56:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d0395d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d00ef8640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ce4004780 0x7f4ce4004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4005250 0x7f4ce400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.540+0000 7f4d01efa640 1 -- 172.21.15.17:0/1161840902 reap_dead start 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4d016f9640 1 -- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 msgr2=0x7f4ce4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4d016f9640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 0x7f4ce4003e60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8f7c9c923e4e99c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf402add0 con 0x7f4ce4003a50 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: dump: 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf402e070 con 0x7f4ce4003a50 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 1 -- 172.21.15.17:0/1161840902 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4034660 con 0x7f4ce4003a50 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.543+0000 7f4cfbfff640 10 monclient: dump: 2024-09-15T01:56:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.558 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 4 set_mon_vals no callback set 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 10 monclient: shutdown 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 -- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 msgr2=0x7f4ce4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 0x7f4ce4003e60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4008040 comp rx=0 tx=0).stop 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 -- 172.21.15.17:0/1161840902 shutdown_connections 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 --2- 172.21.15.17:0/1161840902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce4003a50 0x7f4ce4003e60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 -- 172.21.15.17:0/1161840902 >> 172.21.15.17:0/1161840902 conn(0x7f4cfc070ed0 msgr2=0x7f4cfc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 -- 172.21.15.17:0/1161840902 shutdown_connections 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.544+0000 7f4d0395d640 1 -- 172.21.15.17:0/1161840902 wait complete. 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.545+0000 7f4d0395d640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.545+0000 7f4d0395d640 10 monclient: monmap: 2024-09-15T01:56:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.545+0000 7f4d0395d640 1 Processor -- start 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.545+0000 7f4d0395d640 1 -- start start 2024-09-15T01:56:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.545+0000 7f4d0395d640 10 monclient: init 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce4003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc135020 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cfc12f040 0x7f4cfc132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc12f040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc11b860 con 0x7f4ce4003a50 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc1344e0 con 0x7f4cfc12f040 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc098dc0 con 0x7f4cfc135020 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d0395d640 10 monclient(hunting): authenticate will time out at 1036.542372s 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47858/0 (socket says 172.21.15.17:47858) 2024-09-15T01:56:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 -- 172.21.15.17:0/3256881249 learned_addr learned my addr 172.21.15.17:0/3256881249 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 10 monclient(hunting): get_auth_request con 0x7f4ce4003a50 auth_method 0 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): get_auth_request con 0x7f4cfc135020 auth_method 0 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cfc12f040 0x7f4cfc132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): get_auth_request con 0x7f4cfc12f040 auth_method 0 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 msgr2=0x7f4cfc1412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 msgr2=0x7f4cfc1412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d016f9640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d01efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient(hunting): handle_auth_done global_id 74312 payload 824 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cfc12f040 msgr2=0x7f4cfc132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cfc12f040 0x7f4cfc132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 msgr2=0x7f4cfc1412b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 monclient: found mon.noname-c 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.546+0000 7f4d00ef8640 1 -- 172.21.15.17:0/3256881249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0a38e0 con 0x7f4cfc135020 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d00ef8640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 msgr2=0x7f4cfc12eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d00ef8640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000c00 tx=0x7f4cf0009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0001210 con 0x7f4cfc135020 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: dump: 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 5 monclient: authenticate success, global_id 74312 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf00072e0 con 0x7f4cfc135020 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0005780 con 0x7f4cfc135020 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 10 monclient: _renew_subs 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc142140 con 0x7f4cfc135020 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cf97fa640 10 monclient: dump: 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4cfb7fe640 4 set_mon_vals no callback set 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 10 monclient: _renew_subs 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.549+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc1417f0 con 0x7f4cfc135020 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4cf0005af0 con 0x7f4cfc135020 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d0395d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d0395d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d0395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4cf97fa640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 0x7f4ce803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc0724a0 con 0x7f4cfc135020 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f4cf004c900 con 0x7f4cfc135020 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d016f9640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 0x7f4ce803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.550+0000 7f4d016f9640 10 monclient: get_auth_request con 0x7f4ce8038770 auth_method 0 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.553+0000 7f4d016f9640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 0x7f4ce803ac20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4032000 tx=0x7f4cf4035000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.555+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4cf0012950 con 0x7f4cfc135020 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.555+0000 7f4cf97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.555+0000 7f4cf97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:16.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.626+0000 7fedf86c0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.626+0000 7fedf86c0640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.626+0000 7fedf86c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.626+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fedf0067a50 con 0x7fedf01371a0 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.627+0000 7fede67fc640 1 -- 172.21.15.67:0/128649273 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v126) v1 ==== 73+0+6748 (secure 0 0 0) 0x7fede804a070 con 0x7fedf01371a0 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.627+0000 7fede67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T01:56:16.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.627+0000 7fede67fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: "9": [ 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -1, 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.630 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -2, 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -3, 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.631 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -4, 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.632 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -5, 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.633 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -6, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -7, 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.634 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -8, 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -9, 2024-09-15T01:56:16.635 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -10, 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.636 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -11, 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -12, 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -13, 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -14, 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -15, 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: "10": [ 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -1, 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0.263671875, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.640 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -2, 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.641 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -3, 2024-09-15T01:56:16.642 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.643 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -4, 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -5, 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.644 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.645 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -6, 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.646 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.647 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -7, 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625 2024-09-15T01:56:16.648 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -8, 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -9, 2024-09-15T01:56:16.649 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: [ 2024-09-15T01:56:16.650 INFO:tasks.workunit.client.0.smithi067.stdout: 0.087890625, 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: 0, 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -10, 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.651 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -11, 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -12, 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.652 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -13, 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.653 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -14, 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: "bucket_id": -15, 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: "weight_set": [ 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [], 2024-09-15T01:56:16.654 INFO:tasks.workunit.client.0.smithi067.stdout: [] 2024-09-15T01:56:16.655 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.655 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-09-15T01:56:16.655 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-09-15T01:56:16.655 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 msgr2=0x7fedd403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 0x7fedd403abf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fedec007d10 tx=0x7fedec009040 comp rx=0 tx=0).stop 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 10 monclient: shutdown 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 msgr2=0x7fedf0134b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fede80097c0 tx=0x7fede8037000 comp rx=0 tx=0).stop 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 shutdown_connections 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedd4038740 0x7fedd403abf0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf01371a0 0x7fedf0134b60 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fedf0131e70 0x7fedf0134620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 --2- 172.21.15.67:0/128649273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedf0098420 0x7fedf01340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 >> 172.21.15.67:0/128649273 conn(0x7fedf00a5c10 msgr2=0x7fedf00a2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 shutdown_connections 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.629+0000 7fedf86c0640 1 -- 172.21.15.67:0/128649273 wait complete. 2024-09-15T01:56:16.657 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T01:56:16.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.716+0000 7f273e11d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:16.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.716+0000 7f273e11d640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.716+0000 7f273e11d640 10 monclient: monmap: 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:16.717511+0000 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:16.717511+0000 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:16.719 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 Processor -- start 2024-09-15T01:56:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 -- start start 2024-09-15T01:56:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient: init 2024-09-15T01:56:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f273806e210 0x7f273806b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-c con 0x7f273806e210 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f273806e5e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f273809e220 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273806aa60 con 0x7f273809e220 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2738069bd0 con 0x7f273806e5e0 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273806a190 con 0x7f273806e210 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f273e11d640 10 monclient(hunting): authenticate will time out at 1009.186650s 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f273806e210 0x7f273806b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f2736ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f273806e5e0 0x7f273809db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59832/0 (socket says 172.21.15.67:59832) 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f2736ffd640 1 -- 172.21.15.67:0/953187914 learned_addr learned my addr 172.21.15.67:0/953187914 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): get_auth_request con 0x7f273806e210 auth_method 0 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.717+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 10 monclient(hunting): handle_auth_done global_id 5746 payload 274 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 1 -- 172.21.15.67:0/953187914 >> v1:172.21.15.67:6789/0 conn(0x7f273806e5e0 legacy=0x7f273809db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 1 -- 172.21.15.67:0/953187914 >> v1:172.21.15.17:6789/0 conn(0x7f273809e220 legacy=0x7f2738139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 1 monclient: found mon.noname-c 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.718+0000 7f27377fe640 1 -- 172.21.15.67:0/953187914 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380698f0 con 0x7f273806e210 2024-09-15T01:56:16.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.720+0000 7f27377fe640 1 -- 172.21.15.67:0/953187914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273806e210 msgr2=0x7f273806b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.720+0000 7f27377fe640 1 --2- 172.21.15.67:0/953187914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273806e210 0x7f273806b9b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2728009880 tx=0x7f27280090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8656b3e133473d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 1 -- 172.21.15.67:0/953187914 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728032440 con 0x7f273806e210 2024-09-15T01:56:16.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: dump: 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 1 -- 172.21.15.67:0/953187914 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27280325a0 con 0x7f273806e210 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 5 monclient: authenticate success, global_id 5746 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 1 -- 172.21.15.67:0/953187914 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728032940 con 0x7f273806e210 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 4 set_mon_vals no callback set 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f27367fc640 10 monclient: dump: 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.729 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 10 monclient: shutdown 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 -- 172.21.15.67:0/953187914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273806e210 msgr2=0x7f273806b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 --2- 172.21.15.67:0/953187914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273806e210 0x7f273806b9b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2728009880 tx=0x7f27280090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 -- 172.21.15.67:0/953187914 shutdown_connections 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 --2- 172.21.15.67:0/953187914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273806e210 0x7f273806b9b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 -- 172.21.15.67:0/953187914 >> 172.21.15.67:0/953187914 conn(0x7f2738070ff0 msgr2=0x7f27380a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 -- 172.21.15.67:0/953187914 shutdown_connections 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.721+0000 7f273e11d640 1 -- 172.21.15.67:0/953187914 wait complete. 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.722+0000 7f273e11d640 10 monclient: build_initial_monmap 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.722+0000 7f273e11d640 10 monclient: monmap: 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:16.731 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.722+0000 7f273e11d640 1 Processor -- start 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.722+0000 7f273e11d640 1 -- start start 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.722+0000 7f273e11d640 10 monclient: init 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f273806e210 0x7f273813e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-c con 0x7f273806e210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-a con 0x7f273806e5e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): picked mon.noname-b con 0x7f273809e220 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738123200 con 0x7f273806e5e0 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738122c40 con 0x7f273809e220 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2738120720 con 0x7f273806e210 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f273e11d640 10 monclient(hunting): authenticate will time out at 1009.192178s 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55924/0 (socket says 172.21.15.67:55924) 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 -- 172.21.15.67:0/2345814527 learned_addr learned my addr 172.21.15.67:0/2345814527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): get_auth_request con 0x7f273809e220 auth_method 0 2024-09-15T01:56:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f273806e210 0x7f273813e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): get_auth_request con 0x7f273806e210 auth_method 0 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 10 monclient(hunting): get_auth_request con 0x7f273806e5e0 auth_method 0 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 msgr2=0x7f273813d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 msgr2=0x7f273813d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f27377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2736ffd640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient(hunting): handle_auth_done global_id 5749 payload 824 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f273806e210 msgr2=0x7f273813e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f273806e210 0x7f273813e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 msgr2=0x7f273813d8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 monclient: found mon.noname-b 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.723+0000 7f2737fff640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728009390 con 0x7f273809e220 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.725+0000 7f2737fff640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 msgr2=0x7f2738141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.725+0000 7f2737fff640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f273000c890 tx=0x7f2730002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273000ce80 con 0x7f273809e220 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: dump: 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:16.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273000aa00 con 0x7f273809e220 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 5 monclient: authenticate success, global_id 5749 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 10 monclient: _renew_subs 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2738122960 con 0x7f273809e220 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273000adc0 con 0x7f273809e220 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 10 monclient: dump: 2024-09-15T01:56:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f2735ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 10 monclient: _renew_subs 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273813dfa0 con 0x7f273809e220 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.726+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2730012630 con 0x7f273809e220 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f273e11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f273e11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f273e11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273804bf60 con 0x7f273809e220 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f272f7fe640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 0x7f271803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f273004cb10 con 0x7f273809e220 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f27377fe640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 0x7f271803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.727+0000 7f27377fe640 10 monclient: get_auth_request con 0x7f2718038740 auth_method 0 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.729+0000 7f27377fe640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 0x7f271803abf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2728009850 tx=0x7f2728032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.731+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2730022950 con 0x7f273809e220 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.731+0000 7f272f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:16.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.731+0000 7f272f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4d0395d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4d0395d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4d0395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4cfc09fc90 con 0x7f4cfc135020 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4cf97fa640 1 -- 172.21.15.17:0/3256881249 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f4cf001b7e0 con 0x7f4cfc135020 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4cf97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:16.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.782+0000 7f4cf97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:16.785 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 msgr2=0x7f4ce803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 0x7f4ce803ac20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4032000 tx=0x7f4cf4035000 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 10 monclient: shutdown 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 msgr2=0x7f4cfc12eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000c00 tx=0x7f4cf0009040 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 shutdown_connections 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4ce8038770 0x7f4ce803ac20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cfc12f040 0x7f4cfc132b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cfc135020 0x7f4cfc12eb00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 --2- 172.21.15.17:0/3256881249 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ce4003a50 0x7f4cfc1412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 >> 172.21.15.17:0/3256881249 conn(0x7f4cfc070ed0 msgr2=0x7f4cfc0a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 shutdown_connections 2024-09-15T01:56:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:16.784+0000 7f4d0395d640 1 -- 172.21.15.17:0/3256881249 wait complete. 2024-09-15T01:56:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.900+0000 7f273e11d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.900+0000 7f273e11d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.900+0000 7f273e11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:16.900+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f27380639f0 con 0x7f273809e220 2024-09-15T01:56:17.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.298+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v127) v1 ==== 129+0+0 (secure 0 0 0) 0x7f2730016070 con 0x7f273809e220 2024-09-15T01:56:17.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.299+0000 7f272f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:17.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.299+0000 7f272f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:17.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.406+0000 7f273e11d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.406+0000 7f273e11d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.406+0000 7f273e11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:17.406+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f2738068a20 con 0x7f273809e220 2024-09-15T01:56:18.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.300+0000 7f272f7fe640 1 -- 172.21.15.67:0/2345814527 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v128) v1 ==== 129+0+0 (secure 0 0 0) 0x7f2730022770 con 0x7f273809e220 2024-09-15T01:56:18.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.300+0000 7f272f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:18.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.300+0000 7f272f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.303+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 msgr2=0x7f271803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.303+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 0x7f271803abf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2728009850 tx=0x7f2728032000 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 10 monclient: shutdown 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 msgr2=0x7f2738141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f273000c890 tx=0x7f2730002640 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 shutdown_connections 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2718038740 0x7f271803abf0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273806e5e0 0x7f273813d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f273809e220 0x7f2738141780 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 --2- 172.21.15.67:0/2345814527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f273806e210 0x7f273813e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 >> 172.21.15.67:0/2345814527 conn(0x7f2738070ff0 msgr2=0x7f2738099310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 shutdown_connections 2024-09-15T01:56:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.304+0000 7f273e11d640 1 -- 172.21.15.67:0/2345814527 wait complete. 2024-09-15T01:56:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-15T01:56:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:56:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-15T01:56:18.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 10 monclient: get_monmap_and_config 2024-09-15T01:56:18.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 10 monclient: monmap: 2024-09-15T01:56:18.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:18.402223+0000 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:18.402223+0000 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:18.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 1 Processor -- start 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 1 -- start start 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.401+0000 7fd826e17640 10 monclient: init 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd820098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-a con 0x7fd8200a0370 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 0x7fd8201390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-b con 0x7fd820098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820098ef0 con 0x7fd820098140 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8200a1730 con 0x7fd8200a0370 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200a0b80 con 0x7fd820098a50 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd826e17640 10 monclient(hunting): authenticate will time out at 1010.871438s 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd824bb3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd820098140 0x7fd8201321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59848/0 (socket says 172.21.15.67:59848) 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd824bb3640 1 -- 172.21.15.67:0/3905636509 learned_addr learned my addr 172.21.15.67:0/3905636509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 1 --2- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 0x7fd8201390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): get_auth_request con 0x7fd820098a50 auth_method 0 2024-09-15T01:56:18.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 1 --2- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 10 monclient(hunting): get_auth_request con 0x7fd8200a0370 auth_method 0 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd8253b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 1 -- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 msgr2=0x7fd820098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 1 -- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 msgr2=0x7fd820098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.402+0000 7fd81ffff640 1 --2- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd81ffff640 1 --2- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 10 monclient(hunting): handle_auth_done global_id 74318 payload 274 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 1 -- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 msgr2=0x7fd820098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 1 --2- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 1 -- 172.21.15.67:0/3905636509 >> v1:172.21.15.67:6789/0 conn(0x7fd820098140 legacy=0x7fd8201321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 1 monclient: found mon.noname-b 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:18.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.403+0000 7fd8253b4640 1 -- 172.21.15.67:0/3905636509 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8201328f0 con 0x7fd820098a50 2024-09-15T01:56:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd8253b4640 1 -- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 msgr2=0x7fd8201390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd8253b4640 1 --2- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 0x7fd8201390c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd818009e60 tx=0x7fd818009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a5f82981f7036b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 1 -- 172.21.15.67:0/3905636509 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818007760 con 0x7fd820098a50 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: dump: 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 1 -- 172.21.15.67:0/3905636509 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81802d050 con 0x7fd820098a50 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd826e17640 5 monclient: authenticate success, global_id 74318 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd826e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd826e17640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd826e17640 4 set_mon_vals no callback set 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 1 -- 172.21.15.67:0/3905636509 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818007af0 con 0x7fd820098a50 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.406+0000 7fd81f7fe640 10 monclient: dump: 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.418 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 10 monclient: shutdown 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 -- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 msgr2=0x7fd8201390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 --2- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 0x7fd8201390c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd818009e60 tx=0x7fd818009780 comp rx=0 tx=0).stop 2024-09-15T01:56:18.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 -- 172.21.15.67:0/3905636509 shutdown_connections 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 --2- 172.21.15.67:0/3905636509 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd820098a50 0x7fd8201390c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 --2- 172.21.15.67:0/3905636509 >> v2:172.21.15.17:3300/0 conn(0x7fd8200a0370 0x7fd820098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 -- 172.21.15.67:0/3905636509 >> 172.21.15.67:0/3905636509 conn(0x7fd8200a5780 msgr2=0x7fd8200a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 -- 172.21.15.67:0/3905636509 shutdown_connections 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 1 -- 172.21.15.67:0/3905636509 wait complete. 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.407+0000 7fd826e17640 10 monclient: monmap: 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 1 Processor -- start 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 1 -- start start 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 10 monclient: init 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.408+0000 7fd826e17640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 0x7fd820130e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-a con 0x7fd820098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd820098a50 0x7fd820131340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-b con 0x7fd820098a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd8200a0370 0x7fd820131880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): picked mon.noname-c con 0x7fd8200a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82009f3f0 con 0x7fd820098140 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82006d920 con 0x7fd820098a50 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82006c2d0 con 0x7fd8200a0370 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd826e17640 10 monclient(hunting): authenticate will time out at 1010.878092s 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd820098a50 0x7fd820131340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd820098a50 0x7fd820131340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55932/0 (socket says 172.21.15.67:55932) 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 -- 172.21.15.67:0/525124672 learned_addr learned my addr 172.21.15.67:0/525124672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): get_auth_request con 0x7fd820098a50 auth_method 0 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 0x7fd820130e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd8253b4640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd8200a0370 0x7fd820131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 10 monclient(hunting): get_auth_request con 0x7fd820098140 auth_method 0 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd8253b4640 10 monclient(hunting): get_auth_request con 0x7fd8200a0370 auth_method 0 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd8253b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd8253b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd8253b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 1 -- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 msgr2=0x7fd820130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 1 -- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 msgr2=0x7fd820130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient(hunting): handle_auth_done global_id 5752 payload 824 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 0x7fd820130e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd824bb3640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 0x7fd820130e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 -- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd8200a0370 msgr2=0x7fd820131880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd8200a0370 0x7fd820131880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 -- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 msgr2=0x7fd820130e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd820098140 0x7fd820130e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 monclient: found mon.noname-b 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.409+0000 7fd81ffff640 1 -- 172.21.15.67:0/525124672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd818009a20 con 0x7fd820098a50 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.411+0000 7fd81ffff640 1 -- 172.21.15.67:0/525124672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd820098a50 msgr2=0x7fd820131340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.411+0000 7fd81ffff640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd820098a50 0x7fd820131340 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd81400d860 tx=0x7fd814008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400b8d0 con 0x7fd820098a50 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: dump: 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 5 monclient: authenticate success, global_id 5752 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 10 monclient: _renew_subs 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81400ba50 con 0x7fd820098a50 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 1 -- 172.21.15.67:0/525124672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82006abf0 con 0x7fd820098a50 2024-09-15T01:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400be10 con 0x7fd820098a50 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81cff9640 10 monclient: dump: 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.427 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 10 monclient: _renew_subs 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd81effd640 4 set_mon_vals no callback set 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.412+0000 7fd826e17640 1 -- 172.21.15.67:0/525124672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82006b7a0 con 0x7fd820098a50 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd814014640 con 0x7fd820098a50 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd826e17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd826e17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd826e17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd826e17640 1 -- 172.21.15.67:0/525124672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82004bf60 con 0x7fd820098a50 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd81cff9640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd80c038740 0x7fd80c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fd81404cfd0 con 0x7fd820098a50 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd824bb3640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd80c038740 0x7fd80c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.413+0000 7fd824bb3640 10 monclient: get_auth_request con 0x7fd80c038740 auth_method 0 2024-09-15T01:56:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.415+0000 7fd824bb3640 1 --2- 172.21.15.67:0/525124672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd80c038740 0x7fd80c03abf0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd810007d10 tx=0x7fd810009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.417+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd8140148f0 con 0x7fd820098a50 2024-09-15T01:56:18.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.417+0000 7fd81cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.417+0000 7fd81cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:18.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.591+0000 7fd826e17640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T01:56:18.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.591+0000 7fd826e17640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T01:56:18.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.591+0000 7fd826e17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.591+0000 7fd826e17640 1 -- 172.21.15.67:0/525124672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7fd82006f000 con 0x7fd820098a50 2024-09-15T01:56:18.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.592+0000 7fd81cff9640 1 -- 172.21.15.67:0/525124672 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v128) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd81401d3c0 con 0x7fd820098a50 2024-09-15T01:56:18.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.592+0000 7fd81cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T01:56:18.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.592+0000 7fd81cff9640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-15T01:56:18.595 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-15T01:56:18.598 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:56:18.598 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-15T01:56:18.598 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -x 2024-09-15T01:56:18.598 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.674+0000 7f26bbb00640 10 monclient: get_monmap_and_config 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.674+0000 7f26bbb00640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.674+0000 7f26bbb00640 10 monclient: monmap: 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:18.675728+0000 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:18.675728+0000 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:18.678 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:18.678 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:18.678 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:18.678 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:18.678 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 Processor -- start 2024-09-15T01:56:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 -- start start 2024-09-15T01:56:18.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient: init 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-b con 0x7f26b40a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26b409eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26b409f290 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40a61a0 con 0x7f26b409f290 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40a04c0 con 0x7f26b409eea0 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40a1610 con 0x7f26b40a0b50 2024-09-15T01:56:18.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.675+0000 7f26bbb00640 10 monclient(hunting): authenticate will time out at 1011.144974s 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b909b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26b409eea0 0x7f26b4131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59852/0 (socket says 172.21.15.67:59852) 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b909b640 1 -- 172.21.15.67:0/2801933330 learned_addr learned my addr 172.21.15.67:0/2801933330 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 1 --2- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): get_auth_request con 0x7f26b40a0b50 auth_method 0 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient(hunting): handle_auth_done global_id 74324 payload 274 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 1 -- 172.21.15.67:0/2801933330 >> v1:172.21.15.67:6789/0 conn(0x7f26b409eea0 legacy=0x7f26b4131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 1 -- 172.21.15.67:0/2801933330 >> v1:172.21.15.17:6789/0 conn(0x7f26b409f290 legacy=0x7f26b4132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 1 monclient: found mon.noname-b 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:18.683 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.676+0000 7f26b989c640 1 -- 172.21.15.67:0/2801933330 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40a0f20 con 0x7f26b40a0b50 2024-09-15T01:56:18.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b989c640 1 -- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 msgr2=0x7f26b4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b989c640 1 --2- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4069380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009880 tx=0x7f26b00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f82ec81badf55f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.689 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 1 -- 172.21.15.67:0/2801933330 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0032440 con 0x7f26b40a0b50 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: dump: 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.690 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 1 -- 172.21.15.67:0/2801933330 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b00325a0 con 0x7f26b40a0b50 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26bbb00640 5 monclient: authenticate success, global_id 74324 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26bbb00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26bbb00640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:18.691 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26bbb00640 4 set_mon_vals no callback set 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 1 -- 172.21.15.67:0/2801933330 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0032940 con 0x7f26b40a0b50 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.679+0000 7f26b889a640 10 monclient: dump: 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.692 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 10 monclient: shutdown 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 -- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 msgr2=0x7f26b4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 --2- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4069380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009880 tx=0x7f26b00090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 -- 172.21.15.67:0/2801933330 shutdown_connections 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 --2- 172.21.15.67:0/2801933330 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4069380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 -- 172.21.15.67:0/2801933330 >> 172.21.15.67:0/2801933330 conn(0x7f26b4071330 msgr2=0x7f26b40a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 -- 172.21.15.67:0/2801933330 shutdown_connections 2024-09-15T01:56:18.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 1 -- 172.21.15.67:0/2801933330 wait complete. 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.680+0000 7f26bbb00640 10 monclient: monmap: 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:18.694 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 Processor -- start 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 -- start start 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient: init 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 0x7f26b409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-a con 0x7f26b409eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26b409f290 0x7f26b4130e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-b con 0x7f26b409f290 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:18.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): picked mon.noname-c con 0x7f26b40a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40a1260 con 0x7f26b409eea0 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4131b40 con 0x7f26b409f290 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b411df40 con 0x7f26b40a0b50 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26bbb00640 10 monclient(hunting): authenticate will time out at 1011.150765s 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26b409f290 0x7f26b4130e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26b409f290 0x7f26b4130e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55944/0 (socket says 172.21.15.67:55944) 2024-09-15T01:56:18.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 1 -- 172.21.15.67:0/3375082832 learned_addr learned my addr 172.21.15.67:0/3375082832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 10 monclient(hunting): get_auth_request con 0x7f26b409f290 auth_method 0 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b909b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26ba09d640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b989c640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 0x7f26b409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26ba09d640 10 monclient(hunting): get_auth_request con 0x7f26b40a0b50 auth_method 0 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26ba09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26ba09d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26ba09d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.681+0000 7f26b989c640 10 monclient(hunting): get_auth_request con 0x7f26b409eea0 auth_method 0 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26ba09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26ba09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26ba09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 1 -- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 msgr2=0x7f26b409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 1 -- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 msgr2=0x7f26b409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 0x7f26b409a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b989c640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 0x7f26b409a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient(hunting): handle_auth_done global_id 5755 payload 824 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 -- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 msgr2=0x7f26b4141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f26b40a0b50 0x7f26b4141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 -- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 msgr2=0x7f26b409a9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26b409eea0 0x7f26b409a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 monclient: found mon.noname-b 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.682+0000 7f26b909b640 1 -- 172.21.15.67:0/3375082832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0009390 con 0x7f26b409f290 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26b909b640 1 -- 172.21.15.67:0/3375082832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26b409f290 msgr2=0x7f26b4130e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26b909b640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26b409f290 0x7f26b4130e20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f26a800c5e0 tx=0x7f26a8004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8005cb0 con 0x7f26b409f290 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: dump: 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.700 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 5 monclient: authenticate success, global_id 5755 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 10 monclient: _renew_subs 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a8005e30 con 0x7f26b409f290 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 1 -- 172.21.15.67:0/3375082832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b411e220 con 0x7f26b409f290 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8007340 con 0x7f26b409f290 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a5ffb640 10 monclient: dump: 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.702 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26a7fff640 4 set_mon_vals no callback set 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 10 monclient: _renew_subs 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.684+0000 7f26bbb00640 1 -- 172.21.15.67:0/3375082832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b411ca30 con 0x7f26b409f290 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.685+0000 7f26bbb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.685+0000 7f26bbb00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.685+0000 7f26bbb00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.685+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f26a801b400 con 0x7f26b409f290 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.685+0000 7f26bbb00640 1 -- 172.21.15.67:0/3375082832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b404bf60 con 0x7f26b409f290 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.686+0000 7f26a5ffb640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2698038740 0x7f269803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.686+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f26a804c3c0 con 0x7f26b409f290 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.686+0000 7f26b989c640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2698038740 0x7f269803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.686+0000 7f26b989c640 10 monclient: get_auth_request con 0x7f2698038740 auth_method 0 2024-09-15T01:56:18.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.688+0000 7f26b989c640 1 --2- 172.21.15.67:0/3375082832 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2698038740 0x7f269803abf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f26b409ffe0 tx=0x7f26b0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.690+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f26a801b6b0 con 0x7f26b409f290 2024-09-15T01:56:18.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.690+0000 7f26a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.690+0000 7f26a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:18.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26bbb00640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26bbb00640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26bbb00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26bbb00640 1 -- 172.21.15.67:0/3375082832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f26b406f000 con 0x7f26b409f290 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26a5ffb640 1 -- 172.21.15.67:0/3375082832 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v128) v1 ==== 163+0+0 (secure 0 0 0) 0x7f26a801a360 con 0x7f26b409f290 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.862+0000 7f26a5ffb640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-15T01:56:18.865 INFO:tasks.workunit.client.0.smithi067.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-15T01:56:18.868 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T01:56:18.868 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-15T01:56:18.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.946+0000 7f226795d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:18.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.946+0000 7f226795d640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.946+0000 7f226795d640 10 monclient: monmap: 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:18.947365+0000 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:18.947365+0000 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:18.949 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.946+0000 7f226795d640 1 Processor -- start 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 1 -- start start 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient: init 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22600a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f226009eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226009f2a0 0x7f226013a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): picked mon.noname-c con 0x7f226009f2a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226006a270 con 0x7f22600a0b50 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600a0f20 con 0x7f226009eea0 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226006f290 con 0x7f226009f2a0 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f226795d640 10 monclient(hunting): authenticate will time out at 1011.416502s 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226009f2a0 0x7f226013a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2264ef8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f226009eea0 0x7f2260136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59864/0 (socket says 172.21.15.67:59864) 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226009f2a0 0x7f226013a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55956/0 (socket says 172.21.15.67:55956) 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2264ef8640 1 -- 172.21.15.67:0/3210676371 learned_addr learned my addr 172.21.15.67:0/3210676371 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): get_auth_request con 0x7f226009f2a0 auth_method 0 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient(hunting): handle_auth_done global_id 5758 payload 274 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 -- 172.21.15.67:0/3210676371 >> v1:172.21.15.67:6789/0 conn(0x7f226009eea0 legacy=0x7f2260136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 -- 172.21.15.67:0/3210676371 >> v1:172.21.15.17:6789/0 conn(0x7f22600a0b50 legacy=0x7f2260069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:18.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 monclient: found mon.noname-c 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.947+0000 7f2265efa640 1 -- 172.21.15.67:0/3210676371 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600a6160 con 0x7f226009f2a0 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2265efa640 1 -- 172.21.15.67:0/3210676371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 msgr2=0x7f226013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2265efa640 1 --2- 172.21.15.67:0/3210676371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f226013a570 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e60 tx=0x7f2258009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6097072334884a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 1 -- 172.21.15.67:0/3210676371 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258007630 con 0x7f226009f2a0 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: dump: 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.963 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 1 -- 172.21.15.67:0/3210676371 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225802d050 con 0x7f226009f2a0 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 1 -- 172.21.15.67:0/3210676371 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580079c0 con 0x7f226009f2a0 2024-09-15T01:56:18.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f226795d640 5 monclient: authenticate success, global_id 5758 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f226795d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f226795d640 4 set_mon_vals no callback set 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.950+0000 7f2257fff640 10 monclient: dump: 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.965 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 10 monclient: shutdown 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 -- 172.21.15.67:0/3210676371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 msgr2=0x7f226013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 --2- 172.21.15.67:0/3210676371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f226013a570 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2258009e60 tx=0x7f2258009780 comp rx=0 tx=0).stop 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 -- 172.21.15.67:0/3210676371 shutdown_connections 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 --2- 172.21.15.67:0/3210676371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f226013a570 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 -- 172.21.15.67:0/3210676371 >> 172.21.15.67:0/3210676371 conn(0x7f2260071290 msgr2=0x7f22600a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 -- 172.21.15.67:0/3210676371 shutdown_connections 2024-09-15T01:56:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.951+0000 7f226795d640 1 -- 172.21.15.67:0/3210676371 wait complete. 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient: build_initial_monmap 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient: monmap: 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 Processor -- start 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 -- start start 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient: init 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f226009eea0 0x7f2260140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): picked mon.noname-c con 0x7f226009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): picked mon.noname-b con 0x7f226009f2a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): picked mon.noname-a con 0x7f22600a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600a1260 con 0x7f22600a0b50 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260141da0 con 0x7f226009f2a0 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.952+0000 7f226795d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2260121d90 con 0x7f226009eea0 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f226795d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:18.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f226795d640 10 monclient(hunting): authenticate will time out at 1011.422009s 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55960/0 (socket says 172.21.15.67:55960) 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 -- 172.21.15.67:0/943508116 learned_addr learned my addr 172.21.15.67:0/943508116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): get_auth_request con 0x7f226009f2a0 auth_method 0 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f226009eea0 0x7f2260140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 10 monclient(hunting): get_auth_request con 0x7f22600a0b50 auth_method 0 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): get_auth_request con 0x7f226009eea0 auth_method 0 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 msgr2=0x7f226011e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 msgr2=0x7f226011e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:18.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f22656f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2265efa640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient(hunting): handle_auth_done global_id 5761 payload 824 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f226009eea0 msgr2=0x7f2260140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f226009eea0 0x7f2260140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 msgr2=0x7f226011e6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient: _finish_hunting 0 2024-09-15T01:56:18.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 monclient: found mon.noname-b 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.953+0000 7f2264ef8640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2258009a20 con 0x7f226009f2a0 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f2264ef8640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 msgr2=0x7f2260133c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f2264ef8640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f225000c5e0 tx=0x7f2250004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250005cb0 con 0x7f226009f2a0 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: dump: 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.973 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f226795d640 5 monclient: authenticate success, global_id 5761 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2250005e30 con 0x7f226009f2a0 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:18.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f226795d640 10 monclient: _renew_subs 2024-09-15T01:56:18.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250007320 con 0x7f226009f2a0 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f226795d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22601220d0 con 0x7f226009f2a0 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22577fe640 4 set_mon_vals no callback set 2024-09-15T01:56:18.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.955+0000 7f22557fa640 10 monclient: dump: 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:18.977 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 10 monclient: _renew_subs 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2260121040 con 0x7f226009f2a0 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f225001b3b0 con 0x7f226009f2a0 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:18.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.956+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226004bf60 con 0x7f226009f2a0 2024-09-15T01:56:18.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.957+0000 7f22557fa640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 0x7f224403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:18.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.957+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f225004bd00 con 0x7f226009f2a0 2024-09-15T01:56:18.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.957+0000 7f22656f9640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 0x7f224403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:18.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.957+0000 7f22656f9640 10 monclient: get_auth_request con 0x7f2244038770 auth_method 0 2024-09-15T01:56:18.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.959+0000 7f22656f9640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 0x7f224403ac20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f225c007fd0 tx=0x7f225c007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:18.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.961+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2250018650 con 0x7f226009f2a0 2024-09-15T01:56:18.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.961+0000 7f22557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:18.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:18.961+0000 7f22557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:19.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.131+0000 7f226795d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:19.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.131+0000 7f226795d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:19.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.131+0000 7f226795d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:19.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.131+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f22600a04e0 con 0x7f226009f2a0 2024-09-15T01:56:19.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.392+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v129) v1 ==== 139+0+0 (secure 0 0 0) 0x7f2250017070 con 0x7f226009f2a0 2024-09-15T01:56:19.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.392+0000 7f22557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:19.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.392+0000 7f22557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:19.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.497+0000 7f226795d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:19.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.497+0000 7f226795d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:19.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.497+0000 7f226795d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:19.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:19.497+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f226006f000 con 0x7f226009f2a0 2024-09-15T01:56:19.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.904+0000 7f9d8099d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.904+0000 7f9d8099d640 10 monclient: build_initial_monmap 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.904+0000 7f9d8099d640 10 monclient: monmap: 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:19.905265+0000 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:19.905265+0000 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:19.907 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 Processor -- start 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 -- start start 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient: init 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 0x7f9d7c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7c0a0280 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d7c09f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c0a0040 con 0x7f9d7c09f1e0 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c09bff0 con 0x7f9d7c0a0280 2024-09-15T01:56:19.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0a14a0 con 0x7f9d7c131b20 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d8099d640 10 monclient(hunting): authenticate will time out at 1039.901905s 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.905+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47874/0 (socket says 172.21.15.17:47874) 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 -- 172.21.15.17:0/1003041431 learned_addr learned my addr 172.21.15.17:0/1003041431 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): get_auth_request con 0x7f9d7c131b20 auth_method 0 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7ad9d640 1 --1- 172.21.15.17:0/1003041431 >> v1:172.21.15.17:6789/0 conn(0x7f9d7c09f1e0 0x7f9d7c068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39930/0 (socket says 172.21.15.17:39930) 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:19.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 1 --2- 172.21.15.17:0/1003041431 >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 0x7f9d7c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 10 monclient(hunting): get_auth_request con 0x7f9d7c0a0280 auth_method 0 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient(hunting): handle_auth_done global_id 74333 payload 274 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 -- 172.21.15.17:0/1003041431 >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 msgr2=0x7f9d7c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/1003041431 >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 0x7f9d7c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 -- 172.21.15.17:0/1003041431 >> v1:172.21.15.17:6789/0 conn(0x7f9d7c09f1e0 legacy=0x7f9d7c068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 monclient: found mon.noname-b 2024-09-15T01:56:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d7a59c640 1 -- 172.21.15.17:0/1003041431 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c132430 con 0x7f9d7c131b20 2024-09-15T01:56:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.906+0000 7f9d79d9b640 1 --2- 172.21.15.17:0/1003041431 >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 0x7f9d7c131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7a59c640 1 -- 172.21.15.17:0/1003041431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 msgr2=0x7f9d7c09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/1003041431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009910 tx=0x7f9d6c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff26c6cc18f3f479 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 1 -- 172.21.15.17:0/1003041431 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c02aea0 con 0x7f9d7c131b20 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:19.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: dump: 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: _finish_auth 0 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 1 -- 172.21.15.17:0/1003041431 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c032440 con 0x7f9d7c131b20 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 5 monclient: authenticate success, global_id 74333 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 1 -- 172.21.15.17:0/1003041431 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0327e0 con 0x7f9d7c131b20 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 4 set_mon_vals no callback set 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d7959a640 10 monclient: dump: 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 10 monclient: shutdown 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 1 -- 172.21.15.17:0/1003041431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 msgr2=0x7f9d7c09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.909+0000 7f9d8099d640 1 --2- 172.21.15.17:0/1003041431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009910 tx=0x7f9d6c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 -- 172.21.15.17:0/1003041431 shutdown_connections 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 --2- 172.21.15.17:0/1003041431 >> v2:172.21.15.67:3300/0 conn(0x7f9d7c0a0280 0x7f9d7c131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 --2- 172.21.15.17:0/1003041431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c131b20 0x7f9d7c09eca0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 -- 172.21.15.17:0/1003041431 >> 172.21.15.17:0/1003041431 conn(0x7f9d7c071270 msgr2=0x7f9d7c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 -- 172.21.15.17:0/1003041431 shutdown_connections 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 -- 172.21.15.17:0/1003041431 wait complete. 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 10 monclient: build_initial_monmap 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 10 monclient: monmap: 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 Processor -- start 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 1 -- start start 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.910+0000 7f9d8099d640 10 monclient: init 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d7c09f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d7c131b20 0x7f9d7c098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7c131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0a5360 con 0x7f9d7c09f1e0 2024-09-15T01:56:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c11c320 con 0x7f9d7c131b20 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c11c040 con 0x7f9d7c0a0280 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d8099d640 10 monclient(hunting): authenticate will time out at 1039.907447s 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47886/0 (socket says 172.21.15.17:47886) 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 -- 172.21.15.17:0/2884885171 learned_addr learned my addr 172.21.15.17:0/2884885171 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): get_auth_request con 0x7f9d7c0a0280 auth_method 0 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47452/0 (socket says 172.21.15.17:47452) 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 10 monclient(hunting): get_auth_request con 0x7f9d7c09f1e0 auth_method 0 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d7c131b20 0x7f9d7c098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 msgr2=0x7f9d7c140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 msgr2=0x7f9d7c140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): get_auth_request con 0x7f9d7c131b20 auth_method 0 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient(hunting): handle_auth_done global_id 74336 payload 824 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d7c131b20 msgr2=0x7f9d7c098b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d7c131b20 0x7f9d7c098b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 msgr2=0x7f9d7c140fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 monclient: found mon.noname-c 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.911+0000 7f9d79d9b640 1 -- 172.21.15.17:0/2884885171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c009450 con 0x7f9d7c0a0280 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d79d9b640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 msgr2=0x7f9d7c12ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d79d9b640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000d8c0 tx=0x7f9d70008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000deb0 con 0x7f9d7c0a0280 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: dump: 2024-09-15T01:56:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7000bae0 con 0x7f9d7c0a0280 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 5 monclient: authenticate success, global_id 74336 2024-09-15T01:56:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000bea0 con 0x7f9d7c0a0280 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 10 monclient: _renew_subs 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c11bd60 con 0x7f9d7c0a0280 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d78d99640 4 set_mon_vals no callback set 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d6a7fc640 10 monclient: dump: 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 10 monclient: _renew_subs 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.914+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c12f180 con 0x7f9d7c0a0280 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d70014640 con 0x7f9d7c0a0280 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d8099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d8099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d8099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c0724a0 con 0x7f9d7c0a0280 2024-09-15T01:56:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d6a7fc640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 0x7f9d5c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f9d7004ce50 con 0x7f9d7c0a0280 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 0x7f9d5c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.915+0000 7f9d7a59c640 10 monclient: get_auth_request con 0x7f9d5c038790 auth_method 0 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.918+0000 7f9d7a59c640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 0x7f9d5c03ac40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0a2f90 tx=0x7f9d6c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.920+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d70011570 con 0x7f9d7c0a0280 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.920+0000 7f9d6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:19.920+0000 7f9d6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.144+0000 7f9d8099d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.144+0000 7f9d8099d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.144+0000 7f9d8099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.144+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d7c06efc0 con 0x7f9d7c0a0280 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.145+0000 7f9d6a7fc640 1 -- 172.21.15.17:0/2884885171 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f9d7001dbb0 con 0x7f9d7c0a0280 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.145+0000 7f9d6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.145+0000 7f9d6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:20.147 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 msgr2=0x7f9d5c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 0x7f9d5c03ac40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0a2f90 tx=0x7f9d6c032000 comp rx=0 tx=0).stop 2024-09-15T01:56:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 10 monclient: shutdown 2024-09-15T01:56:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 msgr2=0x7f9d7c12ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000d8c0 tx=0x7f9d70008040 comp rx=0 tx=0).stop 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 shutdown_connections 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d5c038790 0x7f9d5c03ac40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d7c131b20 0x7f9d7c098b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7c0a0280 0x7f9d7c12ec40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 --2- 172.21.15.17:0/2884885171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d7c09f1e0 0x7f9d7c140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 >> 172.21.15.17:0/2884885171 conn(0x7f9d7c071270 msgr2=0x7f9d7c09d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 shutdown_connections 2024-09-15T01:56:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:20.147+0000 7f9d8099d640 1 -- 172.21.15.17:0/2884885171 wait complete. 2024-09-15T01:56:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:21.953+0000 7f224bfff640 10 monclient: tick 2024-09-15T01:56:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:21.953+0000 7f224bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:21.953+0000 7f224bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:22.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:22.954+0000 7f224bfff640 10 monclient: tick 2024-09-15T01:56:22.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:22.954+0000 7f224bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:23.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 10 monclient: build_initial_monmap 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 10 monclient: monmap: 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:23.266955+0000 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:23.266955+0000 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:23.269 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:23.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 1 Processor -- start 2024-09-15T01:56:23.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 1 -- start start 2024-09-15T01:56:23.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.266+0000 7f442459d640 10 monclient: init 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): picked mon.noname-b con 0x7f441c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): picked mon.noname-a con 0x7f441c0a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f441c09f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0a0040 con 0x7f441c09f1e0 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c09bff0 con 0x7f441c0a0280 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:23.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0a14a0 con 0x7f441c131b20 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f442459d640 10 monclient(hunting): authenticate will time out at 1043.263498s 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422b3a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f441c09f1e0 0x7f441c068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55968/0 (socket says 172.21.15.17:55968) 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422b3a640 1 -- 172.21.15.17:0/3137834848 learned_addr learned my addr 172.21.15.17:0/3137834848 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c0a0280 auth_method 0 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.267+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.269+0000 7f4422b3a640 1 -- 172.21.15.17:0/3137834848 >> v1:172.21.15.17:6789/0 conn(0x7f441c09f1e0 legacy=0x7f441c068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:23.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c0a0280 auth_method 0 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:23.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:23.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.468+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:23.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:23.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c0a0280 auth_method 0 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:23.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:23.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:23.869+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:23.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:23.954+0000 7f224bfff640 10 monclient: tick 2024-09-15T01:56:23.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:23.954+0000 7f224bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c0a0280 auth_method 0 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:24.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:24.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:24.670+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:24.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:24.953+0000 7f224bfff640 10 monclient: tick 2024-09-15T01:56:24.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:24.953+0000 7f224bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.185+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152873637 0 0) 0x7f441c0a0040 con 0x7f441c09f1e0 2024-09-15T01:56:25.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.185+0000 7f4421337640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.185+0000 7f4421337640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.185+0000 7f4421337640 10 monclient(hunting): my global_id is 84099 2024-09-15T01:56:25.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.185+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44080034e0 con 0x7f441c09f1e0 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770382279 0 0) 0x7f44080034e0 con 0x7f441c09f1e0 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 msgr2=0x7f441c131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 --2- 172.21.15.17:0/3137834848 >> v2:172.21.15.17:3300/0 conn(0x7f441c0a0280 0x7f441c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:56:25.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c132430 con 0x7f441c09f1e0 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4414001aa0 con 0x7f441c09f1e0 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f442459d640 5 monclient: authenticate success, global_id 84099 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f442459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: dump: 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:25.192 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 >> v1:172.21.15.17:6789/0 conn(0x7f441c09f1e0 legacy=0x7f441c068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:25.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408003ad0 0x7f4408003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): picked mon.b con 0x7f4408003ad0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f4408004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): picked mon.c con 0x7f441c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4408004d30 0x7f44080051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): picked mon.a con 0x7f4408004d30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c09bff0 con 0x7f4408004d30 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0a14a0 con 0x7f4408003ad0 2024-09-15T01:56:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440800bcf0 con 0x7f441c131b20 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421337640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f442459d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f4408004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4408004d30 0x7f44080051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408003ad0 0x7f4408003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): get_auth_request con 0x7f4408004d30 auth_method 0 2024-09-15T01:56:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f4408003ad0 auth_method 0 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.186+0000 7f4422339640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 10 monclient(hunting): handle_auth_done global_id 84099 payload 293 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408003ad0 msgr2=0x7f4408003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408003ad0 0x7f4408003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4408004d30 msgr2=0x7f44080051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4408004d30 0x7f44080051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 monclient: found mon.c 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44080034e0 con 0x7f441c131b20 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4421b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.187+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4408004d30 0x7f44080051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421b38640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f4408004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421b38640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f4408004780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f440c006ea0 tx=0x7f440c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a020b4a9060cc8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c02b800 con 0x7f441c131b20 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: dump: 2024-09-15T01:56:25.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f442459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c02f070 con 0x7f441c131b20 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 1 -- 172.21.15.17:0/3137834848 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c02bbd0 con 0x7f441c131b20 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f442459d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f442459d640 4 set_mon_vals no callback set 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.190+0000 7f4421337640 10 monclient: dump: 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 10 monclient: shutdown 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 -- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f4408004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 --2- 172.21.15.17:0/3137834848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f4408004780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f440c006ea0 tx=0x7f440c008040 comp rx=0 tx=0).stop 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f4422b3a640 1 -- 172.21.15.17:0/3137834848 reap_dead start 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 -- 172.21.15.17:0/3137834848 shutdown_connections 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 -- 172.21.15.17:0/3137834848 >> 172.21.15.17:0/3137834848 conn(0x7f441c071270 msgr2=0x7f441c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 -- 172.21.15.17:0/3137834848 shutdown_connections 2024-09-15T01:56:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 1 -- 172.21.15.17:0/3137834848 wait complete. 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 10 monclient: build_initial_monmap 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.191+0000 7f442459d640 10 monclient: monmap: 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 Processor -- start 2024-09-15T01:56:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 -- start start 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient: init 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): picked mon.noname-c con 0x7f441c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 0x7f441c141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): picked mon.noname-a con 0x7f441c140e50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 0x7f441c099420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): picked mon.noname-b con 0x7f441c098ff0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0a09b0 con 0x7f441c140e50 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c130be0 con 0x7f441c098ff0 2024-09-15T01:56:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c09aa50 con 0x7f441c131b20 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.192+0000 7f442459d640 10 monclient(hunting): authenticate will time out at 1045.188976s 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54066/0 (socket says 172.21.15.17:54066) 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 0x7f441c141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 -- 172.21.15.17:0/3358995810 learned_addr learned my addr 172.21.15.17:0/3358995810 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 10 monclient(hunting): get_auth_request con 0x7f441c131b20 auth_method 0 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 0x7f441c099420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): get_auth_request con 0x7f441c140e50 auth_method 0 2024-09-15T01:56:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): get_auth_request con 0x7f441c098ff0 auth_method 0 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c12f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c12f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422339640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient(hunting): handle_auth_done global_id 5770 payload 824 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 msgr2=0x7f441c12f0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 msgr2=0x7f441c141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 0x7f441c141240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 monclient: found mon.noname-b 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4422b3a640 1 -- 172.21.15.17:0/3358995810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c11c1f0 con 0x7f441c098ff0 2024-09-15T01:56:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.193+0000 7f4421b38640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 0x7f441c141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.195+0000 7f4422b3a640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 msgr2=0x7f441c099420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.195+0000 7f4422b3a640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 0x7f441c099420 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f441c0a5010 tx=0x7f441400a240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44140020f0 con 0x7f441c098ff0 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: dump: 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4414007560 con 0x7f441c098ff0 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 5 monclient: authenticate success, global_id 5770 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 10 monclient: _renew_subs 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c11c4d0 con 0x7f441c098ff0 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414007940 con 0x7f441c098ff0 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f44127fc640 10 monclient: dump: 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 10 monclient: _renew_subs 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f4420b36640 4 set_mon_vals no callback set 2024-09-15T01:56:25.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.197+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c12f610 con 0x7f441c098ff0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f441401c410 con 0x7f441c098ff0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f442459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f442459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f442459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c0724a0 con 0x7f441c098ff0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f44127fc640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 0x7f440003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f441404c030 con 0x7f441c098ff0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f4422339640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 0x7f440003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.198+0000 7f4422339640 10 monclient: get_auth_request con 0x7f44000387c0 auth_method 0 2024-09-15T01:56:25.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.201+0000 7f4422339640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 0x7f440003ac70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4418006cf0 tx=0x7f4418008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.203+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f441407de40 con 0x7f441c098ff0 2024-09-15T01:56:25.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.203+0000 7f44127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.203+0000 7f44127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:25.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.245+0000 7f22557fa640 1 -- 172.21.15.67:0/943508116 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v130) v1 ==== 139+0+0 (secure 0 0 0) 0x7f225000e290 con 0x7f226009f2a0 2024-09-15T01:56:25.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.245+0000 7f22557fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T01:56:25.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.245+0000 7f22557fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:25.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 msgr2=0x7f224403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 0x7f224403ac20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f225c007fd0 tx=0x7f225c007e10 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 10 monclient: shutdown 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 msgr2=0x7f2260133c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f225000c5e0 tx=0x7f2250004960 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 shutdown_connections 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2244038770 0x7f224403ac20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f22600a0b50 0x7f226011e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226009f2a0 0x7f2260133c60 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 --2- 172.21.15.67:0/943508116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f226009eea0 0x7f2260140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 >> 172.21.15.67:0/943508116 conn(0x7f2260071290 msgr2=0x7f226009ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 shutdown_connections 2024-09-15T01:56:25.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.248+0000 7f226795d640 1 -- 172.21.15.67:0/943508116 wait complete. 2024-09-15T01:56:25.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:25.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c fooset 2024-09-15T01:56:25.263 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 1 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.341+0000 7f7b59034640 10 monclient: get_monmap_and_config 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.341+0000 7f7b59034640 10 monclient: build_initial_monmap 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.341+0000 7f7b59034640 10 monclient: monmap: 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:25.342559+0000 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:25.342559+0000 2024-09-15T01:56:25.344 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:25.345 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 Processor -- start 2024-09-15T01:56:25.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 -- start start 2024-09-15T01:56:25.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient: init 2024-09-15T01:56:25.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7b5409cb20 0x7f7b5409aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5409cb20 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b5409cef0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b5409d520 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b54099290 con 0x7f7b5409cef0 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b54098fb0 con 0x7f7b5409d520 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5412f810 con 0x7f7b5409cb20 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b59034640 10 monclient(hunting): authenticate will time out at 1017.811728s 2024-09-15T01:56:25.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b5359e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b5409d520 0x7f7b54097180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40268/0 (socket says 172.21.15.67:40268) 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b5359e640 1 -- 172.21.15.67:0/1659197392 learned_addr learned my addr 172.21.15.67:0/1659197392 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/1659197392 >> v2:172.21.15.17:3300/0 conn(0x7f7b5409cb20 0x7f7b5409aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b52d9d640 10 monclient(hunting): get_auth_request con 0x7f7b5409cb20 auth_method 0 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.342+0000 7f7b52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient(hunting): handle_auth_done global_id 84105 payload 274 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 1 -- 172.21.15.67:0/1659197392 >> v1:172.21.15.67:6789/0 conn(0x7f7b5409d520 legacy=0x7f7b54097180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 1 -- 172.21.15.67:0/1659197392 >> v1:172.21.15.17:6789/0 conn(0x7f7b5409cef0 legacy=0x7f7b54096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 1 monclient: found mon.noname-a 2024-09-15T01:56:25.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.343+0000 7f7b52d9d640 1 -- 172.21.15.67:0/1659197392 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b54097890 con 0x7f7b5409cb20 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b52d9d640 1 -- 172.21.15.67:0/1659197392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5409cb20 msgr2=0x7f7b5409aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/1659197392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5409cb20 0x7f7b5409aaa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009850 tx=0x7f7b440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1ab572a094c74c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 1 -- 172.21.15.67:0/1659197392 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44032440 con 0x7f7b5409cb20 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: dump: 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.355 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 1 -- 172.21.15.67:0/1659197392 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b440325a0 con 0x7f7b5409cb20 2024-09-15T01:56:25.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 1 -- 172.21.15.67:0/1659197392 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44032940 con 0x7f7b5409cb20 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 5 monclient: authenticate success, global_id 84105 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 4 set_mon_vals no callback set 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b51d9b640 10 monclient: dump: 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.357 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 10 monclient: shutdown 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 -- 172.21.15.67:0/1659197392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5409cb20 msgr2=0x7f7b5409aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 --2- 172.21.15.67:0/1659197392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5409cb20 0x7f7b5409aaa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009850 tx=0x7f7b440090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 -- 172.21.15.67:0/1659197392 shutdown_connections 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 --2- 172.21.15.67:0/1659197392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5409cb20 0x7f7b5409aaa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 -- 172.21.15.67:0/1659197392 >> 172.21.15.67:0/1659197392 conn(0x7f7b540a1020 msgr2=0x7f7b540a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 -- 172.21.15.67:0/1659197392 shutdown_connections 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.346+0000 7f7b59034640 1 -- 172.21.15.67:0/1659197392 wait complete. 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.347+0000 7f7b59034640 10 monclient: build_initial_monmap 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.347+0000 7f7b59034640 10 monclient: monmap: 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.347+0000 7f7b59034640 1 Processor -- start 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.347+0000 7f7b59034640 1 -- start start 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.347+0000 7f7b59034640 10 monclient: init 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5409cb20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5409d520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 0x7f7b5413f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5412e1b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5409c550 con 0x7f7b5412e1b0 2024-09-15T01:56:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54098620 con 0x7f7b5409d520 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54131a00 con 0x7f7b5409cb20 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b59034640 10 monclient(hunting): authenticate will time out at 1017.817503s 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51728/0 (socket says 172.21.15.67:51728) 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 0x7f7b5413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 1 -- 172.21.15.67:0/2553985629 learned_addr learned my addr 172.21.15.67:0/2553985629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): get_auth_request con 0x7f7b5409d520 auth_method 0 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): get_auth_request con 0x7f7b5412e1b0 auth_method 0 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 10 monclient(hunting): get_auth_request con 0x7f7b5409cb20 auth_method 0 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 msgr2=0x7f7b5409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 msgr2=0x7f7b5409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.348+0000 7f7b5259c640 10 monclient(hunting): handle_auth_done global_id 5773 payload 824 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 msgr2=0x7f7b5409ba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 msgr2=0x7f7b5413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 0x7f7b5413f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 monclient: found mon.noname-b 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5259c640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44009390 con 0x7f7b5409d520 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.349+0000 7f7b5359e640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 0x7f7b5413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b5259c640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 msgr2=0x7f7b5412dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b5259c640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c640 tx=0x7f7b48007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48012840 con 0x7f7b5409d520 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: dump: 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.364 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 5 monclient: authenticate success, global_id 5773 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b480129a0 con 0x7f7b5409d520 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 10 monclient: _renew_subs 2024-09-15T01:56:25.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5411a960 con 0x7f7b5409d520 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48012d60 con 0x7f7b5409d520 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b3affd640 10 monclient: dump: 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.366 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 10 monclient: _renew_subs 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b54119d60 con 0x7f7b5409d520 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.351+0000 7f7b5159a640 4 set_mon_vals no callback set 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7b48011590 con 0x7f7b5409d520 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b3affd640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 0x7f7b4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b59034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b59034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b59034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f7b4804c8e0 con 0x7f7b5409d520 2024-09-15T01:56:25.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5411bf40 con 0x7f7b5409d520 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 0x7f7b4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.352+0000 7f7b52d9d640 10 monclient: get_auth_request con 0x7f7b40038740 auth_method 0 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.355+0000 7f7b52d9d640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 0x7f7b4003abf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7b5409ee60 tx=0x7f7b44032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.357+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7b5411bf40 con 0x7f7b5409d520 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.357+0000 7f7b3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.357+0000 7f7b3affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:25.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.438+0000 7f442459d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:25.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.438+0000 7f442459d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:25.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.438+0000 7f442459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.438+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f441c11be60 con 0x7f441c098ff0 2024-09-15T01:56:25.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.439+0000 7f44127fc640 1 -- 172.21.15.17:0/3358995810 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f441401c6c0 con 0x7f441c098ff0 2024-09-15T01:56:25.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.439+0000 7f44127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:25.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.439+0000 7f44127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 msgr2=0x7f440003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 0x7f440003ac70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4418006cf0 tx=0x7f4418008040 comp rx=0 tx=0).stop 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 10 monclient: shutdown 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 msgr2=0x7f441c099420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.440+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 0x7f441c099420 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f441c0a5010 tx=0x7f441400a240 comp rx=0 tx=0).stop 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 shutdown_connections 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f44000387c0 0x7f440003ac70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f441c140e50 0x7f441c141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441c098ff0 0x7f441c099420 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 --2- 172.21.15.17:0/3358995810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f441c131b20 0x7f441c12f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 >> 172.21.15.17:0/3358995810 conn(0x7f441c071270 msgr2=0x7f441c130e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 shutdown_connections 2024-09-15T01:56:25.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:25.441+0000 7f442459d640 1 -- 172.21.15.17:0/3358995810 wait complete. 2024-09-15T01:56:25.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.527+0000 7f7b59034640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.527+0000 7f7b59034640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.527+0000 7f7b59034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.527+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f7b540639e0 con 0x7f7b5409d520 2024-09-15T01:56:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.528+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7f7b48016070 con 0x7f7b5409d520 2024-09-15T01:56:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.528+0000 7f7b3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.528+0000 7f7b3affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b59034640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b59034640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b59034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f7b54098cd0 con 0x7f7b5409d520 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b3affd640 1 -- 172.21.15.67:0/2553985629 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7f7b480102e0 con 0x7f7b5409d520 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b3affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:25.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.633+0000 7f7b3affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 msgr2=0x7f7b4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 0x7f7b4003abf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7b5409ee60 tx=0x7f7b44032000 comp rx=0 tx=0).stop 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 10 monclient: shutdown 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 msgr2=0x7f7b5412dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c640 tx=0x7f7b48007ed0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 shutdown_connections 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b40038740 0x7f7b4003abf0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b5412e1b0 0x7f7b5413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5409d520 0x7f7b5412dc70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 --2- 172.21.15.67:0/2553985629 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b5409cb20 0x7f7b5409ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 >> 172.21.15.67:0/2553985629 conn(0x7f7b540a1020 msgr2=0x7f7b5409d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.635+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 shutdown_connections 2024-09-15T01:56:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.636+0000 7f7b59034640 1 -- 172.21.15.67:0/2553985629 wait complete. 2024-09-15T01:56:25.647 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm fooset 2024-09-15T01:56:25.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.721+0000 7f9b05f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.721+0000 7f9b05f5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.721+0000 7f9b05f5d640 10 monclient: monmap: 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:25.722681+0000 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:25.722681+0000 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:25.724 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:25.725 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:25.725 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:25.725 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 Processor -- start 2024-09-15T01:56:25.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 -- start start 2024-09-15T01:56:25.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient: init 2024-09-15T01:56:25.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b000a0b50 0x7f9b00069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b000a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b0009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b00131b10 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000a61a0 con 0x7f9b00131b10 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b000a1610 con 0x7f9b000a0b50 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0009f7d0 con 0x7f9b0009eea0 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9b05f5d640 10 monclient(hunting): authenticate will time out at 1018.191785s 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9afffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b00131b10 0x7f9b00131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40274/0 (socket says 172.21.15.67:40274) 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.722+0000 7f9afffff640 1 -- 172.21.15.67:0/2109189517 learned_addr learned my addr 172.21.15.67:0/2109189517 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/2109189517 >> v2:172.21.15.17:3300/0 conn(0x7f9b000a0b50 0x7f9b00069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): get_auth_request con 0x7f9b000a0b50 auth_method 0 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 10 monclient(hunting): get_auth_request con 0x7f9b0009eea0 auth_method 0 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 1 -- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 msgr2=0x7f9b0009f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 1 -- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 msgr2=0x7f9b0009f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9afeffd640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_done global_id 84111 payload 274 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 -- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 msgr2=0x7f9b0009f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 -- 172.21.15.67:0/2109189517 >> v1:172.21.15.67:6789/0 conn(0x7f9b00131b10 legacy=0x7f9b00131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 monclient: found mon.noname-a 2024-09-15T01:56:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.723+0000 7f9aff7fe640 1 -- 172.21.15.67:0/2109189517 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000a04c0 con 0x7f9b000a0b50 2024-09-15T01:56:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9aff7fe640 1 -- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b000a0b50 msgr2=0x7f9b00069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b000a0b50 0x7f9b00069380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009850 tx=0x7f9af00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6888b61eacb7f3c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 1 -- 172.21.15.67:0/2109189517 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af002adf0 con 0x7f9b000a0b50 2024-09-15T01:56:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: dump: 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 1 -- 172.21.15.67:0/2109189517 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af0032440 con 0x7f9b000a0b50 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9b05f5d640 5 monclient: authenticate success, global_id 84111 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9b05f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 1 -- 172.21.15.67:0/2109189517 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af00327e0 con 0x7f9b000a0b50 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9b05f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9b05f5d640 4 set_mon_vals no callback set 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.726+0000 7f9afe7fc640 10 monclient: dump: 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 10 monclient: shutdown 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 -- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b000a0b50 msgr2=0x7f9b00069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b000a0b50 0x7f9b00069380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009850 tx=0x7f9af00090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 -- 172.21.15.67:0/2109189517 shutdown_connections 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b0009eea0 0x7f9b0009f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/2109189517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b000a0b50 0x7f9b00069380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 -- 172.21.15.67:0/2109189517 >> 172.21.15.67:0/2109189517 conn(0x7f9b00071330 msgr2=0x7f9b000a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 -- 172.21.15.67:0/2109189517 shutdown_connections 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 1 -- 172.21.15.67:0/2109189517 wait complete. 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.727+0000 7f9b05f5d640 10 monclient: monmap: 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 1 Processor -- start 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 1 -- start start 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient: init 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b0009eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b000a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 0x7f9b00141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.728+0000 7f9b05f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b00131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0011e110 con 0x7f9b0009eea0 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0009b5d0 con 0x7f9b00131b10 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00134bd0 con 0x7f9b000a0b50 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9b05f5d640 10 monclient(hunting): authenticate will time out at 1018.198027s 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 0x7f9b00141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:52782/0 (socket says 172.21.15.67:52782) 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 -- 172.21.15.67:0/15286426 learned_addr learned my addr 172.21.15.67:0/15286426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): get_auth_request con 0x7f9b0009eea0 auth_method 0 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 0x7f9b00141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51734/0 (socket says 172.21.15.67:51734) 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): get_auth_request con 0x7f9b00131b10 auth_method 0 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 10 monclient(hunting): get_auth_request con 0x7f9b000a0b50 auth_method 0 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 msgr2=0x7f9b00130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 msgr2=0x7f9b00130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9afeffd640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient(hunting): handle_auth_done global_id 84114 payload 824 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 msgr2=0x7f9b00130d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 msgr2=0x7f9b00141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:25.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 0x7f9b00141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 monclient: found mon.noname-a 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.729+0000 7f9aff7fe640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af0009390 con 0x7f9b0009eea0 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9aff7fe640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 msgr2=0x7f9b0009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9aff7fe640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009850 tx=0x7f9af0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af0031b10 con 0x7f9b0009eea0 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 10 monclient: dump: 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.745 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.731+0000 7f9ae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 5 monclient: authenticate success, global_id 84114 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af0031c70 con 0x7f9b0009eea0 2024-09-15T01:56:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _renew_subs 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0011cad0 con 0x7f9b0009eea0 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af003d650 con 0x7f9b0009eea0 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 10 monclient: dump: 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _renew_subs 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0011cf40 con 0x7f9b0009eea0 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9afdffb640 4 set_mon_vals no callback set 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9af0039040 con 0x7f9b0009eea0 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.732+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0004bf60 con 0x7f9b0009eea0 2024-09-15T01:56:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.733+0000 7f9ae77fe640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 0x7f9aec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.733+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f9af006c350 con 0x7f9b0009eea0 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.733+0000 7f9afeffd640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 0x7f9aec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.733+0000 7f9afeffd640 10 monclient: get_auth_request con 0x7f9aec038740 auth_method 0 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.735+0000 7f9afeffd640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 0x7f9aec03abf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9af4006fd0 tx=0x7f9af4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.737+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9af0044d60 con 0x7f9b0009eea0 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.737+0000 7f9ae77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.737+0000 7f9ae77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:25.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.906+0000 7f9b05f5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:25.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.906+0000 7f9b05f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:25.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.906+0000 7f9b05f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:25.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:25.906+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f9b000696a0 con 0x7f9b0009eea0 2024-09-15T01:56:28.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.560+0000 7f25b2699640 10 monclient: get_monmap_and_config 2024-09-15T01:56:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.560+0000 7f25b2699640 10 monclient: build_initial_monmap 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.560+0000 7f25b2699640 10 monclient: monmap: 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:28.561588+0000 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:28.561588+0000 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:28.564 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.561+0000 7f25b2699640 1 Processor -- start 2024-09-15T01:56:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.561+0000 7f25b2699640 1 -- start start 2024-09-15T01:56:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.561+0000 7f25b2699640 10 monclient: init 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-c con 0x7f25ac09c1b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25ac09c580 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac09c9c0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac09f790 con 0x7f25ac09c580 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac069d40 con 0x7f25ac09c9c0 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac068b60 con 0x7f25ac09c1b0 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b2699640 10 monclient(hunting): authenticate will time out at 1048.558132s 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51064/0 (socket says 172.21.15.17:51064) 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 learned_addr learned my addr 172.21.15.17:0/523541561 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25ac09c9c0 auth_method 0 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f25ac09c1b0 auth_method 0 2024-09-15T01:56:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.562+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.564+0000 7f25ab7fe640 1 -- 172.21.15.17:0/523541561 >> v1:172.21.15.67:6789/0 conn(0x7f25ac09c580 legacy=0x7f25ac06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:28.729+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:28.729+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:28.729+0000 7f9ae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25ac09c9c0 auth_method 0 2024-09-15T01:56:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f25ac09c1b0 auth_method 0 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:28.763+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25ac09c9c0 auth_method 0 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:29.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f25ac09c1b0 auth_method 0 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:29.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.164+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:29.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:29.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:29.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:29.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25ac09c9c0 auth_method 0 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:29.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f25ac09c1b0 auth_method 0 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:29.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:29.965+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:30.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:30.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:30.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:30.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:31.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598130281 0 0) 0x7f25ac09f790 con 0x7f25ac09c580 2024-09-15T01:56:31.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): my global_id is 5779 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25940034e0 con 0x7f25ac09c580 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865654326 0 0) 0x7f25940034e0 con 0x7f25ac09c580 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 msgr2=0x7f25ac09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.67:3300/0 conn(0x7f25ac09c1b0 0x7f25ac09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 msgr2=0x7f25ac09e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 --2- 172.21.15.17:0/523541561 >> v2:172.21.15.17:3300/0 conn(0x7f25ac09c9c0 0x7f25ac09e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac069aa0 con 0x7f25ac09c580 2024-09-15T01:56:31.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: _finish_auth 0 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f25a0001770 con 0x7f25ac09c580 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: dump: 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:31.180 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 >> v1:172.21.15.67:6789/0 conn(0x7f25ac09c580 legacy=0x7f25ac06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594003b20 0x7f2594003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): picked mon.c con 0x7f2594003b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 0x7f2594004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): picked mon.a con 0x7f2594004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25940052d0 0x7f259400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): picked mon.b con 0x7f25940052d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:31.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac069d40 con 0x7f2594004780 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac068b60 con 0x7f25940052d0 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259400c310 con 0x7f2594003b20 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.174+0000 7f25aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594003b20 0x7f2594003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 0x7f2594004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f2594003b20 auth_method 0 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request con 0x7f2594004780 auth_method 0 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25940052d0 0x7f259400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25940052d0 auth_method 0 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:31.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_done global_id 5779 payload 293 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 -- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594003b20 msgr2=0x7f2594003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594003b20 0x7f2594003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 -- 172.21.15.17:0/523541561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25940052d0 msgr2=0x7f259400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25940052d0 0x7f259400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:56:31.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 monclient: found mon.a 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b0c36640 1 -- 172.21.15.17:0/523541561 reap_dead start 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 1 -- 172.21.15.17:0/523541561 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940034e0 con 0x7f2594004780 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b2699640 5 monclient: authenticate success, global_id 5779 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25b2699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:31.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.175+0000 7f25abfff640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594003b20 0x7f2594003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:31.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25ab7fe640 1 -- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 msgr2=0x7f2594004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:31.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25ab7fe640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 0x7f2594004bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25a0001950 tx=0x7f25ac069c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d8b9c681b7a68f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:31.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00070f0 con 0x7f2594004780 2024-09-15T01:56:31.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: dump: 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25b2699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0007290 con 0x7f2594004780 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 1 -- 172.21.15.17:0/523541561 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0005780 con 0x7f2594004780 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.178+0000 7f25aaffd640 10 monclient: dump: 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:31.192 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 4 set_mon_vals no callback set 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 10 monclient: shutdown 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 -- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 msgr2=0x7f2594004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 0x7f2594004bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25a0001950 tx=0x7f25ac069c30 comp rx=0 tx=0).stop 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 -- 172.21.15.17:0/523541561 shutdown_connections 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 --2- 172.21.15.17:0/523541561 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2594004780 0x7f2594004bf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 -- 172.21.15.17:0/523541561 >> 172.21.15.17:0/523541561 conn(0x7f25ac071300 msgr2=0x7f25ac0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.179+0000 7f25b2699640 1 -- 172.21.15.17:0/523541561 shutdown_connections 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.180+0000 7f25b2699640 1 -- 172.21.15.17:0/523541561 wait complete. 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.180+0000 7f25b2699640 10 monclient: build_initial_monmap 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.180+0000 7f25b2699640 10 monclient: monmap: 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:31.194 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 1 Processor -- start 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 1 -- start start 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 10 monclient: init 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-c con 0x7f2594004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.181+0000 7f25b2699640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ac121310 0x7f25ac13d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-b con 0x7f25ac121310 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac13e410 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac122a80 con 0x7f25ac13e410 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1224c0 con 0x7f25ac121310 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1221e0 con 0x7f2594004780 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b2699640 10 monclient(hunting): authenticate will time out at 1051.178074s 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51110/0 (socket says 172.21.15.17:51110) 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54098/0 (socket says 172.21.15.17:54098) 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 -- 172.21.15.17:0/2418690048 learned_addr learned my addr 172.21.15.17:0/2418690048 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:31.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): get_auth_request con 0x7f25ac13e410 auth_method 0 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 10 monclient(hunting): get_auth_request con 0x7f2594004780 auth_method 0 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ac121310 0x7f25ac13d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request con 0x7f25ac121310 auth_method 0 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:31.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 msgr2=0x7f25ac13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 msgr2=0x7f25ac13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25abfff640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient(hunting): handle_auth_done global_id 84117 payload 824 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 msgr2=0x7f25ac13da90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:31.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ac121310 msgr2=0x7f25ac13d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ac121310 0x7f25ac13d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient: _finish_hunting 0 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 monclient: found mon.noname-a 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.182+0000 7f25b0c36640 1 -- 172.21.15.17:0/2418690048 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac1227a0 con 0x7f25ac13e410 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.184+0000 7f25b0c36640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 msgr2=0x7f25ac141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b0c36640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009450 tx=0x7f25a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4018870 con 0x7f25ac13e410 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: dump: 2024-09-15T01:56:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a400f040 con 0x7f25ac13e410 2024-09-15T01:56:31.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 5 monclient: authenticate success, global_id 84117 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 10 monclient: _renew_subs 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac11ffa0 con 0x7f25ac13e410 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4018c40 con 0x7f25ac13e410 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f2593fff640 10 monclient: dump: 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:31.201 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 10 monclient: _renew_subs 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25aa7fc640 4 set_mon_vals no callback set 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.185+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac13d820 con 0x7f25ac13e410 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f25a4021400 con 0x7f25ac13e410 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f2593fff640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 0x7f259803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f25a404d180 con 0x7f25ac13e410 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f25b2699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f25b2699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f25b2699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.186+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac120130 con 0x7f25ac13e410 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.187+0000 7f25abfff640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 0x7f259803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.187+0000 7f25abfff640 10 monclient: get_auth_request con 0x7f25980387c0 auth_method 0 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.189+0000 7f25abfff640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 0x7f259803ac70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25ac063740 tx=0x7f259c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.191+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f25ac120130 con 0x7f25ac13e410 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.191+0000 7f2593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:31.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.191+0000 7f2593fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:31.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.230+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v131) v1 ==== 89+0+0 (secure 0 0 0) 0x7f9af0042070 con 0x7f9b0009eea0 2024-09-15T01:56:31.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.230+0000 7f9ae77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:31.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.230+0000 7f9ae77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:31.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.342+0000 7f9b05f5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.342+0000 7f9b05f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.342+0000 7f9b05f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.342+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f9b000696a0 con 0x7f9b0009eea0 2024-09-15T01:56:31.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f25b2699640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f25b2699640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f25b2699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f25ac120130 con 0x7f25ac13e410 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f2593fff640 1 -- 172.21.15.17:0/2418690048 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f25ac120130 con 0x7f25ac13e410 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f2593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.428+0000 7f2593fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:31.431 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":62,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:31.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 msgr2=0x7f259803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:31.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 0x7f259803ac70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25ac063740 tx=0x7f259c008040 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 10 monclient: shutdown 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 msgr2=0x7f25ac141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009450 tx=0x7f25a400b040 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 shutdown_connections 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25980387c0 0x7f259803ac70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ac13e410 0x7f25ac141000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ac121310 0x7f25ac13d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 --2- 172.21.15.17:0/2418690048 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2594004780 0x7f25ac13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 >> 172.21.15.17:0/2418690048 conn(0x7f25ac071300 msgr2=0x7f25ac098b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 shutdown_connections 2024-09-15T01:56:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:31.430+0000 7f25b2699640 1 -- 172.21.15.17:0/2418690048 wait complete. 2024-09-15T01:56:31.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:31.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:31.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:32.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:32.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:32.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:32.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:33.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:33.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:33.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:33.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:34.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.549+0000 7f17f1ec0640 10 monclient: get_monmap_and_config 2024-09-15T01:56:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.549+0000 7f17f1ec0640 10 monclient: build_initial_monmap 2024-09-15T01:56:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.549+0000 7f17f1ec0640 10 monclient: monmap: 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:34.550442+0000 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:34.550442+0000 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:34.552 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:34.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.550+0000 7f17f1ec0640 1 Processor -- start 2024-09-15T01:56:34.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.550+0000 7f17f1ec0640 1 -- start start 2024-09-15T01:56:34.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.550+0000 7f17f1ec0640 10 monclient: init 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17ec0a5780 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f17ec09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f17ec069c90 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0a5b50 con 0x7f17ec0a5780 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec0a18b0 con 0x7f17ec069c90 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec09c5f0 con 0x7f17ec09ccf0 2024-09-15T01:56:34.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17f1ec0640 10 monclient(hunting): authenticate will time out at 1054.547120s 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38044/0 (socket says 172.21.15.17:38044) 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38944/0 (socket says 172.21.15.17:38944) 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 learned_addr learned my addr 172.21.15.17:0/3685037069 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17ec09ccf0 auth_method 0 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17ec069c90 auth_method 0 2024-09-15T01:56:34.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:34.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.551+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:34.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.553+0000 7f17eb7fe640 1 -- 172.21.15.17:0/3685037069 >> v1:172.21.15.67:6789/0 conn(0x7f17ec0a5780 legacy=0x7f17ec069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:34.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:34.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:34.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:34.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17ec09ccf0 auth_method 0 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17ec069c90 auth_method 0 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:34.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:34.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:34.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:34.752+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:35.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17ec09ccf0 auth_method 0 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17ec069c90 auth_method 0 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:35.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:35.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.153+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:35.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:35.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:35.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:35.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:35.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:35.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17ec069c90 auth_method 0 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17ec09ccf0 auth_method 0 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:35.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:35.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:35.954+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:36.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:36.730+0000 7f9ae67fc640 10 monclient: tick 2024-09-15T01:56:36.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:36.730+0000 7f9ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918032042 0 0) 0x7f17ec0a5b50 con 0x7f17ec0a5780 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 10 monclient(hunting): my global_id is 5785 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d40034e0 con 0x7f17ec0a5780 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026302757 0 0) 0x7f17d40034e0 con 0x7f17ec0a5780 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17ec09bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.220+0000 7f17ea7fc640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17ec09bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 msgr2=0x7f17ec139880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 --2- 172.21.15.17:0/3685037069 >> v2:172.21.15.17:3300/0 conn(0x7f17ec069c90 0x7f17ec139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec09d210 con 0x7f17ec0a5780 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f17dc001770 con 0x7f17ec0a5780 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17f1ec0640 5 monclient: authenticate success, global_id 5785 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17f1ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: dump: 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:37.226 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 >> v1:172.21.15.67:6789/0 conn(0x7f17ec0a5780 legacy=0x7f17ec069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:37.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17d4003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): picked mon.c con 0x7f17ec09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17d4004150 0x7f17d4004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): picked mon.a con 0x7f17d4004150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 0x7f17d40051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): picked mon.b con 0x7f17d4004d30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0a18b0 con 0x7f17d4004150 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec09c5f0 con 0x7f17d4004d30 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d400bd50 con 0x7f17ec09ccf0 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:37.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17f1ec0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17d4003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17d4004150 0x7f17d4004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 0x7f17d40051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request con 0x7f17ec09ccf0 auth_method 0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17d4004d30 auth_method 0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17d4004150 auth_method 0 2024-09-15T01:56:37.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient(hunting): handle_auth_done global_id 5785 payload 293 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 msgr2=0x7f17d4003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17ec09ccf0 0x7f17d4003ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17d4004150 msgr2=0x7f17d4004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17d4004150 0x7f17d4004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 monclient: found mon.b 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d40034e0 con 0x7f17d4004d30 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.221+0000 7f17ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.224+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 msgr2=0x7f17d40051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.224+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 0x7f17d40051a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007490 tx=0x7f17e400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51d55124d030be3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e4031480 con 0x7f17d4004d30 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: dump: 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e402e030 con 0x7f17d4004d30 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17f1ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17f1ec0640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 1 -- 172.21.15.17:0/3685037069 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e4031850 con 0x7f17d4004d30 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17f1ec0640 4 set_mon_vals no callback set 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.226+0000 7f17ea7fc640 10 monclient: dump: 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 10 monclient: shutdown 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 msgr2=0x7f17d40051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3685037069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d4004d30 0x7f17d40051a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007490 tx=0x7f17e400b040 comp rx=0 tx=0).stop 2024-09-15T01:56:37.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17ebfff640 1 -- 172.21.15.17:0/3685037069 reap_dead start 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3685037069 shutdown_connections 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3685037069 >> 172.21.15.17:0/3685037069 conn(0x7f17ec070e10 msgr2=0x7f17ec0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3685037069 shutdown_connections 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.227+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3685037069 wait complete. 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.228+0000 7f17f1ec0640 10 monclient: build_initial_monmap 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.228+0000 7f17f1ec0640 10 monclient: monmap: 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.228+0000 7f17f1ec0640 1 Processor -- start 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.228+0000 7f17f1ec0640 1 -- start start 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.228+0000 7f17f1ec0640 10 monclient: init 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f17d4004d30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 0x7f17ec134040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f17ec133c70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 0x7f17ec11dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f17ec11d770 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec134580 con 0x7f17ec11d770 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec121350 con 0x7f17ec133c70 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0a0920 con 0x7f17d4004d30 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17f1ec0640 10 monclient(hunting): authenticate will time out at 1057.225528s 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 0x7f17ec11dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38092/0 (socket says 172.21.15.17:38092) 2024-09-15T01:56:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 -- 172.21.15.17:0/3196112922 learned_addr learned my addr 172.21.15.17:0/3196112922 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request con 0x7f17d4004d30 auth_method 0 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 0x7f17ec134040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): get_auth_request con 0x7f17ec133c70 auth_method 0 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): get_auth_request con 0x7f17ec11d770 auth_method 0 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 msgr2=0x7f17ec141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 msgr2=0x7f17ec141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17eb7fe640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.229+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 10 monclient(hunting): handle_auth_done global_id 5788 payload 824 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 msgr2=0x7f17ec141080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:37.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 msgr2=0x7f17ec11dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 0x7f17ec11dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 monclient: found mon.noname-b 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17eaffd640 1 -- 172.21.15.17:0/3196112922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec11f260 con 0x7f17ec133c70 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.230+0000 7f17ebfff640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 0x7f17ec11dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17eaffd640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 msgr2=0x7f17ec134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17eaffd640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 0x7f17ec134040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006f60 tx=0x7f17e0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0015c20 con 0x7f17ec133c70 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: dump: 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e001c070 con 0x7f17ec133c70 2024-09-15T01:56:37.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0018450 con 0x7f17ec133c70 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17f1ec0640 5 monclient: authenticate success, global_id 5788 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17f1ec0640 10 monclient: _renew_subs 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17f1ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec11fe90 con 0x7f17ec133c70 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17d37fe640 10 monclient: dump: 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.232+0000 7f17e97fa640 4 set_mon_vals no callback set 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.233+0000 7f17f1ec0640 10 monclient: _renew_subs 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.233+0000 7f17f1ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.233+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec141e60 con 0x7f17ec133c70 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.233+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f17e001e030 con 0x7f17ec133c70 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17f1ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17f1ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17f1ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec0724a0 con 0x7f17ec133c70 2024-09-15T01:56:37.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17d37fe640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 0x7f17d803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f17e004d1c0 con 0x7f17ec133c70 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17eb7fe640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 0x7f17d803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.234+0000 7f17eb7fe640 10 monclient: get_auth_request con 0x7f17d8038790 auth_method 0 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.237+0000 7f17eb7fe640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 0x7f17d803ac40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001810 tx=0x7f17dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.239+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f17e0014da0 con 0x7f17ec133c70 2024-09-15T01:56:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.239+0000 7f17d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.239+0000 7f17d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:37.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.275+0000 7f9ae77fe640 1 -- 172.21.15.67:0/15286426 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v132) v1 ==== 89+0+0 (secure 0 0 0) 0x7f9af0037030 con 0x7f9b0009eea0 2024-09-15T01:56:37.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.275+0000 7f9ae77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T01:56:37.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.275+0000 7f9ae77fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 msgr2=0x7f9aec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 0x7f9aec03abf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9af4006fd0 tx=0x7f9af4008040 comp rx=0 tx=0).stop 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 10 monclient: shutdown 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 msgr2=0x7f9b0009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.279+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009850 tx=0x7f9af0032000 comp rx=0 tx=0).stop 2024-09-15T01:56:37.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 shutdown_connections 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9aec038740 0x7f9aec03abf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b00131b10 0x7f9b00141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b000a0b50 0x7f9b00130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 --2- 172.21.15.67:0/15286426 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b0009eea0 0x7f9b0009a990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 >> 172.21.15.67:0/15286426 conn(0x7f9b00071330 msgr2=0x7f9b0009d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 shutdown_connections 2024-09-15T01:56:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.280+0000 7f9b05f5d640 1 -- 172.21.15.67:0/15286426 wait complete. 2024-09-15T01:56:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c fooset 2024-09-15T01:56:37.297 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 0 2024-09-15T01:56:37.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:37.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:37.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 10 monclient: monmap: 2024-09-15T01:56:37.378 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:37.377305+0000 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:37.377305+0000 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:37.379 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 1 Processor -- start 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 1 -- start start 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.376+0000 7f50acd9d640 10 monclient: init 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f50a8131aa0 0x7f50a8131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f50a8131aa0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50a80a0a60 0x7f50a809ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50a80a0a60 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50a809f2c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a809f6c0 con 0x7f50a809f2c0 2024-09-15T01:56:37.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a8132930 con 0x7f50a8131aa0 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a80a1b00 con 0x7f50a80a0a60 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50acd9d640 10 monclient(hunting): authenticate will time out at 1029.846566s 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50a80a0a60 0x7f50a809ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50a80a0a60 0x7f50a809ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53116/0 (socket says 172.21.15.67:53116) 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a6d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50a809f2c0 0x7f50a8068e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51396/0 (socket says 172.21.15.67:51396) 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 1 -- 172.21.15.67:0/4147957338 learned_addr learned my addr 172.21.15.67:0/4147957338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): get_auth_request con 0x7f50a80a0a60 auth_method 0 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 1 --2- 172.21.15.67:0/4147957338 >> v2:172.21.15.17:3300/0 conn(0x7f50a8131aa0 0x7f50a8131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): get_auth_request con 0x7f50a8131aa0 auth_method 0 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 10 monclient(hunting): handle_auth_done global_id 5791 payload 274 2024-09-15T01:56:37.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 1 -- 172.21.15.67:0/4147957338 >> v2:172.21.15.17:3300/0 conn(0x7f50a8131aa0 msgr2=0x7f50a8131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.377+0000 7f50a5d9b640 1 --2- 172.21.15.67:0/4147957338 >> v2:172.21.15.17:3300/0 conn(0x7f50a8131aa0 0x7f50a8131e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 1 -- 172.21.15.67:0/4147957338 >> v1:172.21.15.67:6789/0 conn(0x7f50a809f2c0 legacy=0x7f50a8068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 1 monclient: found mon.noname-c 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.378+0000 7f50a5d9b640 1 -- 172.21.15.67:0/4147957338 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80a03d0 con 0x7f50a80a0a60 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a5d9b640 1 -- 172.21.15.67:0/4147957338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a80a0a60 msgr2=0x7f50a809ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a5d9b640 1 --2- 172.21.15.67:0/4147957338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a80a0a60 0x7f50a809ed80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50a80a3120 tx=0x7f509c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d81b4baa6f9f8cf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 1 -- 172.21.15.67:0/4147957338 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c032440 con 0x7f50a80a0a60 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: dump: 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 1 -- 172.21.15.67:0/4147957338 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c0325a0 con 0x7f50a80a0a60 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50acd9d640 5 monclient: authenticate success, global_id 5791 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50acd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50acd9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50acd9d640 4 set_mon_vals no callback set 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 1 -- 172.21.15.67:0/4147957338 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c032940 con 0x7f50a80a0a60 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.381+0000 7f50a559a640 10 monclient: dump: 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.394 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 10 monclient: shutdown 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 -- 172.21.15.67:0/4147957338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a80a0a60 msgr2=0x7f50a809ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 --2- 172.21.15.67:0/4147957338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a80a0a60 0x7f50a809ed80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50a80a3120 tx=0x7f509c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 -- 172.21.15.67:0/4147957338 shutdown_connections 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 --2- 172.21.15.67:0/4147957338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a80a0a60 0x7f50a809ed80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 --2- 172.21.15.67:0/4147957338 >> v2:172.21.15.17:3300/0 conn(0x7f50a8131aa0 0x7f50a8131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 -- 172.21.15.67:0/4147957338 >> 172.21.15.67:0/4147957338 conn(0x7f50a8071390 msgr2=0x7f50a80a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 -- 172.21.15.67:0/4147957338 shutdown_connections 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 1 -- 172.21.15.67:0/4147957338 wait complete. 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:37.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.382+0000 7f50acd9d640 10 monclient: monmap: 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 Processor -- start 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 -- start start 2024-09-15T01:56:37.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient: init 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50a809f2c0 0x7f50a809acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f50a809f2c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50a80a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f50a8131aa0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a8142220 con 0x7f50a809f2c0 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80a0f40 con 0x7f50a8131aa0 2024-09-15T01:56:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a8134b70 con 0x7f50a80a0a60 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50acd9d640 10 monclient(hunting): authenticate will time out at 1029.852732s 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53130/0 (socket says 172.21.15.67:53130) 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 1 -- 172.21.15.67:0/3883339897 learned_addr learned my addr 172.21.15.67:0/3883339897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): get_auth_request con 0x7f50a8131aa0 auth_method 0 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a659c640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50a809f2c0 0x7f50a809acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a5d9b640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a659c640 10 monclient(hunting): get_auth_request con 0x7f50a809f2c0 auth_method 0 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a5d9b640 10 monclient(hunting): get_auth_request con 0x7f50a80a0a60 auth_method 0 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.383+0000 7f50a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a5d9b640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 msgr2=0x7f50a8130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a5d9b640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 msgr2=0x7f50a8130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a5d9b640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a5d9b640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 10 monclient(hunting): handle_auth_done global_id 5794 payload 824 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 msgr2=0x7f50a8130e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50a809f2c0 msgr2=0x7f50a809acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50a809f2c0 0x7f50a809acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 monclient: found mon.noname-b 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.384+0000 7f50a6d9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c009450 con 0x7f50a8131aa0 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50a6d9d640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 msgr2=0x7f50a811b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50a6d9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c880 tx=0x7f50a0004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a000ce70 con 0x7f50a8131aa0 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: dump: 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 5 monclient: authenticate success, global_id 5794 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a000a640 con 0x7f50a8131aa0 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 10 monclient: _renew_subs 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a809b850 con 0x7f50a8131aa0 2024-09-15T01:56:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a000aa00 con 0x7f50a8131aa0 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50967fc640 10 monclient: dump: 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 10 monclient: _renew_subs 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50a4d99640 4 set_mon_vals no callback set 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.386+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a811d030 con 0x7f50a8131aa0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f50a00070f0 con 0x7f50a8131aa0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50acd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50acd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50967fc640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 0x7f508803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f50a004c3d0 con 0x7f50a8131aa0 2024-09-15T01:56:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50a659c640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 0x7f508803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.387+0000 7f50a659c640 10 monclient: get_auth_request con 0x7f5088038790 auth_method 0 2024-09-15T01:56:37.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.390+0000 7f50a659c640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 0x7f508803ac40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d20 tx=0x7f50980090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.392+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.392+0000 7f50967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.392+0000 7f50967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17f1ec0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17f1ec0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17f1ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f17ec09e380 con 0x7f17ec133c70 2024-09-15T01:56:37.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17d37fe640 1 -- 172.21.15.17:0/3196112922 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f17e00147d0 con 0x7f17ec133c70 2024-09-15T01:56:37.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:37.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.484+0000 7f17d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:37.487 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 msgr2=0x7f17d803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 0x7f17d803ac40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001810 tx=0x7f17dc009040 comp rx=0 tx=0).stop 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 10 monclient: shutdown 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 msgr2=0x7f17ec134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 0x7f17ec134040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006f60 tx=0x7f17e0008040 comp rx=0 tx=0).stop 2024-09-15T01:56:37.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 shutdown_connections 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f17d8038790 0x7f17d803ac40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17ec11d770 0x7f17ec11dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17ec133c70 0x7f17ec134040 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 --2- 172.21.15.17:0/3196112922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f17d4004d30 0x7f17ec141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 >> 172.21.15.17:0/3196112922 conn(0x7f17ec070e10 msgr2=0x7f17ec0a47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 shutdown_connections 2024-09-15T01:56:37.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:37.487+0000 7f17f1ec0640 1 -- 172.21.15.17:0/3196112922 wait complete. 2024-09-15T01:56:37.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50acd9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50acd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v132) v1 ==== 71+0+7 (secure 0 0 0) 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.564+0000 7f50967fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:37.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50acd9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50acd9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50967fc640 1 -- 172.21.15.67:0/3883339897 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v132) v1 ==== 71+0+7 (secure 0 0 0) 0x7f50a811e440 con 0x7f50a8131aa0 2024-09-15T01:56:37.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50967fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:37.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.670+0000 7f50967fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:37.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 msgr2=0x7f508803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 0x7f508803ac40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d20 tx=0x7f50980090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 10 monclient: shutdown 2024-09-15T01:56:37.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 msgr2=0x7f50a811b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c880 tx=0x7f50a0004ef0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 shutdown_connections 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5088038790 0x7f508803ac40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a8131aa0 0x7f50a811b8d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50a80a0a60 0x7f50a8130e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 --2- 172.21.15.67:0/3883339897 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50a809f2c0 0x7f50a809acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 >> 172.21.15.67:0/3883339897 conn(0x7f50a8071390 msgr2=0x7f50a8134d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.672+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 shutdown_connections 2024-09-15T01:56:37.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.673+0000 7f50acd9d640 1 -- 172.21.15.67:0/3883339897 wait complete. 2024-09-15T01:56:37.684 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:37.684 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep barset 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.757+0000 7f73b84c9640 10 monclient: get_monmap_and_config 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.757+0000 7f73b84c9640 10 monclient: build_initial_monmap 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.757+0000 7f73b84c9640 10 monclient: monmap: 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:37.758910+0000 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:37.758910+0000 2024-09-15T01:56:37.760 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:37.761 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 Processor -- start 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 -- start start 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient: init 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73b0131f90 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b0098260 0x7f73b0098660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-c con 0x7f73b0098260 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f73b0098ba0 0x7f73b0139130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f73b0098ba0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b0096a70 con 0x7f73b0131f90 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b006a6b0 con 0x7f73b0098ba0 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b0098fb0 con 0x7f73b0098260 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.758+0000 7f73b84c9640 10 monclient(hunting): authenticate will time out at 1030.227992s 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b0098260 0x7f73b0098660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b0098260 0x7f73b0098660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53146/0 (socket says 172.21.15.67:53146) 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 -- 172.21.15.67:0/646433897 learned_addr learned my addr 172.21.15.67:0/646433897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:37.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): get_auth_request con 0x7f73b0098260 auth_method 0 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b6a66640 1 --2- 172.21.15.67:0/646433897 >> v2:172.21.15.17:3300/0 conn(0x7f73b0098ba0 0x7f73b0139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient(hunting): handle_auth_done global_id 5797 payload 274 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 -- 172.21.15.67:0/646433897 >> v2:172.21.15.17:3300/0 conn(0x7f73b0098ba0 msgr2=0x7f73b0139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 --2- 172.21.15.67:0/646433897 >> v2:172.21.15.17:3300/0 conn(0x7f73b0098ba0 0x7f73b0139130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 -- 172.21.15.67:0/646433897 >> v1:172.21.15.17:6789/0 conn(0x7f73b0131f90 legacy=0x7f73b0132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:37.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 monclient: found mon.noname-c 2024-09-15T01:56:37.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.759+0000 7f73b5a64640 1 -- 172.21.15.67:0/646433897 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0132b20 con 0x7f73b0098260 2024-09-15T01:56:37.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5a64640 1 -- 172.21.15.67:0/646433897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0098260 msgr2=0x7f73b0098660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5a64640 1 --2- 172.21.15.67:0/646433897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0098260 0x7f73b0098660 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f73b00a2e00 tx=0x7f73a00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe52df55ec6489c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 1 -- 172.21.15.67:0/646433897 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0032440 con 0x7f73b0098260 2024-09-15T01:56:37.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: dump: 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 1 -- 172.21.15.67:0/646433897 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a00325a0 con 0x7f73b0098260 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 5 monclient: authenticate success, global_id 5797 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 1 -- 172.21.15.67:0/646433897 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0032940 con 0x7f73b0098260 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 4 set_mon_vals no callback set 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b5263640 10 monclient: dump: 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 10 monclient: shutdown 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:37.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 -- 172.21.15.67:0/646433897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0098260 msgr2=0x7f73b0098660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 --2- 172.21.15.67:0/646433897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0098260 0x7f73b0098660 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f73b00a2e00 tx=0x7f73a00091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 -- 172.21.15.67:0/646433897 shutdown_connections 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 --2- 172.21.15.67:0/646433897 >> v2:172.21.15.17:3300/0 conn(0x7f73b0098ba0 0x7f73b0139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 --2- 172.21.15.67:0/646433897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0098260 0x7f73b0098660 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 -- 172.21.15.67:0/646433897 >> 172.21.15.67:0/646433897 conn(0x7f73b0071690 msgr2=0x7f73b00a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.762+0000 7f73b84c9640 1 -- 172.21.15.67:0/646433897 shutdown_connections 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 1 -- 172.21.15.67:0/646433897 wait complete. 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 10 monclient: build_initial_monmap 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 10 monclient: monmap: 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:37.777 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 1 Processor -- start 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 1 -- start start 2024-09-15T01:56:37.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.763+0000 7f73b84c9640 10 monclient: init 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f73b0098260 0x7f73b0138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-c con 0x7f73b0098260 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f73b0098ba0 0x7f73b0133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f73b0098ba0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): picked mon.noname-b con 0x7f73b0131f90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00a3140 con 0x7f73b0098ba0 2024-09-15T01:56:37.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0137310 con 0x7f73b0131f90 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01375f0 con 0x7f73b0098260 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b84c9640 10 monclient(hunting): authenticate will time out at 1030.233246s 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53156/0 (socket says 172.21.15.67:53156) 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 -- 172.21.15.67:0/1045489521 learned_addr learned my addr 172.21.15.67:0/1045489521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:37.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): get_auth_request con 0x7f73b0131f90 auth_method 0 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient(hunting): handle_auth_done global_id 5800 payload 824 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b5a64640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f73b0098ba0 0x7f73b0133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6265640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f73b0098260 0x7f73b0138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 -- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f73b0098260 msgr2=0x7f73b0138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f73b0098260 0x7f73b0138270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 -- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f73b0098ba0 msgr2=0x7f73b0133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:37.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f73b0098ba0 0x7f73b0133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient: _finish_hunting 0 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 monclient: found mon.noname-b 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.764+0000 7f73b6a66640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0009450 con 0x7f73b0131f90 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.766+0000 7f73b6a66640 1 -- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 msgr2=0x7f73b0134240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.766+0000 7f73b6a66640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73a800ceb0 tx=0x7f73a8007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.766+0000 7f73b6a66640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73a800ceb0 tx=0x7f73a8007870 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f73a800e610 mon_map magic: 0 v1 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800e610 con 0x7f73b0131f90 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 10 monclient: dump: 2024-09-15T01:56:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a800e770 con 0x7f73b0131f90 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 5 monclient: authenticate success, global_id 5800 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _renew_subs 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b0136f70 con 0x7f73b0131f90 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b4a62640 4 set_mon_vals no callback set 2024-09-15T01:56:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _renew_subs 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b0134780 con 0x7f73b0131f90 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.767+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0135200 con 0x7f73b0131f90 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800eb30 con 0x7f73b0131f90 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 10 monclient: dump: 2024-09-15T01:56:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f73a8009030 con 0x7f73b0131f90 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73a67fc640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 0x7f739403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73b6265640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 0x7f739403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.768+0000 7f73b6265640 10 monclient: get_auth_request con 0x7f73940387c0 auth_method 0 2024-09-15T01:56:37.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.770+0000 7f73b6265640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 0x7f739403ac70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00a8a0 tx=0x7f73ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:37.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:37.772+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f73a8012070 con 0x7f73b0131f90 2024-09-15T01:56:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.389+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f73a801d910 con 0x7f73b0131f90 2024-09-15T01:56:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.389+0000 7f73a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.389+0000 7f73a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:38.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.561+0000 7f73b84c9640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.561+0000 7f73b84c9640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.561+0000 7f73b84c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.561+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f73b00a0df0 con 0x7f73b0131f90 2024-09-15T01:56:38.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.596+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (secure 0 0 0) 0x7f73b00a0df0 con 0x7f73b0131f90 2024-09-15T01:56:38.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.596+0000 7f73a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.596+0000 7f73a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73b84c9640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73b84c9640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73b84c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f73b00a0df0 con 0x7f73b0131f90 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73a67fc640 1 -- 172.21.15.67:0/1045489521 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (secure 0 0 0) 0x7f73b00a0df0 con 0x7f73b0131f90 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73a67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:38.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.701+0000 7f73a67fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 msgr2=0x7f739403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 0x7f739403ac70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00a8a0 tx=0x7f73ac008040 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 10 monclient: shutdown 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 msgr2=0x7f73b0134240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73a800ceb0 tx=0x7f73a8007870 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 shutdown_connections 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f73940387c0 0x7f739403ac70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f73b0098ba0 0x7f73b0133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0131f90 0x7f73b0134240 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 --2- 172.21.15.67:0/1045489521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f73b0098260 0x7f73b0138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 >> 172.21.15.67:0/1045489521 conn(0x7f73b0071690 msgr2=0x7f73b009b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:38.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 shutdown_connections 2024-09-15T01:56:38.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.704+0000 7f73b84c9640 1 -- 172.21.15.67:0/1045489521 wait complete. 2024-09-15T01:56:38.715 INFO:tasks.workunit.client.0.smithi067.stdout:barset 2024-09-15T01:56:38.715 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm barset 2024-09-15T01:56:38.791 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.789+0000 7f251799d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.789+0000 7f251799d640 10 monclient: build_initial_monmap 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.789+0000 7f251799d640 10 monclient: monmap: 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:38.790586+0000 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:38.790586+0000 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:38.792 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:38.793 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:38.793 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:38.793 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:38.793 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.789+0000 7f251799d640 1 Processor -- start 2024-09-15T01:56:38.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 -- start start 2024-09-15T01:56:38.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient: init 2024-09-15T01:56:38.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:38.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): picked mon.noname-b con 0x7f25100a1a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f251009d5a0 0x7f251009b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): picked mon.noname-a con 0x7f251009d5a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2510096a70 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251009cc40 con 0x7f2510096a70 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251009cf80 con 0x7f251009d5a0 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100a1dd0 con 0x7f25100a1a00 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:38.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f251799d640 10 monclient(hunting): authenticate will time out at 1031.259738s 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f2514f38640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f251009d5a0 0x7f251009b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.790+0000 7f2515739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:60264/0 (socket says 172.21.15.67:60264) 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 -- 172.21.15.67:0/3169707388 learned_addr learned my addr 172.21.15.67:0/3169707388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): get_auth_request con 0x7f25100a1a00 auth_method 0 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): get_auth_request con 0x7f251009d5a0 auth_method 0 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:38.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2514f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient(hunting): handle_auth_done global_id 74339 payload 274 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 -- 172.21.15.67:0/3169707388 >> v2:172.21.15.17:3300/0 conn(0x7f251009d5a0 msgr2=0x7f251009b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 --2- 172.21.15.67:0/3169707388 >> v2:172.21.15.17:3300/0 conn(0x7f251009d5a0 0x7f251009b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 -- 172.21.15.67:0/3169707388 >> v1:172.21.15.17:6789/0 conn(0x7f2510096a70 legacy=0x7f2510096ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient: _finish_hunting 0 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 monclient: found mon.noname-b 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:38.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.791+0000 7f2515739640 1 -- 172.21.15.67:0/3169707388 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251012efa0 con 0x7f25100a1a00 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2515739640 1 -- 172.21.15.67:0/3169707388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 msgr2=0x7f251009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2515739640 1 --2- 172.21.15.67:0/3169707388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f250c009910 tx=0x7f250c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a22da52fbcbbc1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 1 -- 172.21.15.67:0/3169707388 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c02aea0 con 0x7f25100a1a00 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: dump: 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:38.803 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 1 -- 172.21.15.67:0/3169707388 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c032440 con 0x7f25100a1a00 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 5 monclient: authenticate success, global_id 74339 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 1 -- 172.21.15.67:0/3169707388 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0327e0 con 0x7f25100a1a00 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 4 set_mon_vals no callback set 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f2503fff640 10 monclient: dump: 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:38.805 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 10 monclient: shutdown 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 -- 172.21.15.67:0/3169707388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 msgr2=0x7f251009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 --2- 172.21.15.67:0/3169707388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f250c009910 tx=0x7f250c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 -- 172.21.15.67:0/3169707388 shutdown_connections 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 --2- 172.21.15.67:0/3169707388 >> v2:172.21.15.17:3300/0 conn(0x7f251009d5a0 0x7f251009b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 --2- 172.21.15.67:0/3169707388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251009aae0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.794+0000 7f251799d640 1 -- 172.21.15.67:0/3169707388 >> 172.21.15.67:0/3169707388 conn(0x7f2510071340 msgr2=0x7f25100a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 1 -- 172.21.15.67:0/3169707388 shutdown_connections 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 1 -- 172.21.15.67:0/3169707388 wait complete. 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 10 monclient: build_initial_monmap 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 10 monclient: monmap: 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 1 Processor -- start 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 1 -- start start 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.795+0000 7f251799d640 10 monclient: init 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): picked mon.noname-b con 0x7f2510096a70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f251009d5a0 0x7f2510137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): picked mon.noname-a con 0x7f251009d5a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251012d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): picked mon.noname-c con 0x7f25100a1a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101305d0 con 0x7f251009d5a0 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100a0940 con 0x7f2510096a70 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25101308b0 con 0x7f25100a1a00 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f251799d640 10 monclient(hunting): authenticate will time out at 1031.265472s 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53168/0 (socket says 172.21.15.67:53168) 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 -- 172.21.15.67:0/2424239788 learned_addr learned my addr 172.21.15.67:0/2424239788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): get_auth_request con 0x7f2510096a70 auth_method 0 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2514f38640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f251009d5a0 0x7f2510137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515f3a640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251012d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2514f38640 10 monclient(hunting): get_auth_request con 0x7f251009d5a0 auth_method 0 2024-09-15T01:56:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2514f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2514f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2514f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515f3a640 10 monclient(hunting): get_auth_request con 0x7f25100a1a00 auth_method 0 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient(hunting): handle_auth_done global_id 5803 payload 824 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 -- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 msgr2=0x7f251012d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251012d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 -- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f251009d5a0 msgr2=0x7f2510137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f251009d5a0 0x7f2510137110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient: _finish_hunting 0 2024-09-15T01:56:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 monclient: found mon.noname-b 2024-09-15T01:56:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.796+0000 7f2515739640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009450 con 0x7f2510096a70 2024-09-15T01:56:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.799+0000 7f2515739640 1 -- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 msgr2=0x7f251006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.799+0000 7f2515739640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f250c035040 tx=0x7f250c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c02aea0 con 0x7f2510096a70 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: dump: 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.814 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c000d10 con 0x7f2510096a70 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 5 monclient: authenticate success, global_id 5803 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 10 monclient: _renew_subs 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2510130b90 con 0x7f2510096a70 2024-09-15T01:56:38.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c03d440 con 0x7f2510096a70 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f25017fa640 10 monclient: dump: 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:38.816 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 10 monclient: _renew_subs 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251006e6f0 con 0x7f2510096a70 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.801+0000 7f2502ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f250c039040 con 0x7f2510096a70 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f251799d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f251799d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f251799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251004bf60 con 0x7f2510096a70 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f25017fa640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 0x7f24f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f250c06c560 con 0x7f2510096a70 2024-09-15T01:56:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f2514f38640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 0x7f24f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:38.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.802+0000 7f2514f38640 10 monclient: get_auth_request con 0x7f24f4038740 auth_method 0 2024-09-15T01:56:38.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.805+0000 7f2514f38640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 0x7f24f403abf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f25101257c0 tx=0x7f2504008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:38.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.807+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f250c030b20 con 0x7f2510096a70 2024-09-15T01:56:38.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.807+0000 7f25017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:38.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.807+0000 7f25017fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.976+0000 7f251799d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.976+0000 7f251799d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.976+0000 7f251799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:38.976+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f2510067180 con 0x7f2510096a70 2024-09-15T01:56:39.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.607+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v134) v1 ==== 89+0+0 (secure 0 0 0) 0x7f250c0420a0 con 0x7f2510096a70 2024-09-15T01:56:39.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.607+0000 7f25017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:39.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.607+0000 7f25017fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.716+0000 7f251799d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.716+0000 7f251799d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.716+0000 7f251799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:39.716+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f2510067180 con 0x7f2510096a70 2024-09-15T01:56:40.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:40.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.606+0000 7f9e8815d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:40.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.606+0000 7f9e8815d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.606+0000 7f9e8815d640 10 monclient: monmap: 2024-09-15T01:56:40.608 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:40.608 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:40.607231+0000 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:40.607231+0000 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:40.609 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 1 Processor -- start 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 1 -- start start 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 10 monclient: init 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.607+0000 7f9e8815d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e80131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e800a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e8009f1e0 0x7f9e80138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e8009f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e800a0040 con 0x7f9e800a0280 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8009bff0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800a14a0 con 0x7f9e80131b20 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e8815d640 10 monclient(hunting): authenticate will time out at 1060.604166s 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38114/0 (socket says 172.21.15.17:38114) 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3379844477 learned_addr learned my addr 172.21.15.17:0/3379844477 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 10 monclient(hunting): get_auth_request con 0x7f9e80131b20 auth_method 0 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e85ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 --2- 172.21.15.17:0/3379844477 >> v2:172.21.15.67:3300/0 conn(0x7f9e8009f1e0 0x7f9e80138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): get_auth_request con 0x7f9e8009f1e0 auth_method 0 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient(hunting): handle_auth_done global_id 5806 payload 274 2024-09-15T01:56:40.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 -- 172.21.15.17:0/3379844477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 msgr2=0x7f9e8009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 -- 172.21.15.17:0/3379844477 >> v1:172.21.15.67:6789/0 conn(0x7f9e800a0280 legacy=0x7f9e80131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 monclient: found mon.noname-c 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.608+0000 7f9e866fa640 1 -- 172.21.15.17:0/3379844477 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e80132600 con 0x7f9e8009f1e0 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.609+0000 7f9e85ef9640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.612+0000 7f9e866fa640 1 -- 172.21.15.17:0/3379844477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8009f1e0 msgr2=0x7f9e80138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.612+0000 7f9e866fa640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8009f1e0 0x7f9e80138e60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009e60 tx=0x7f9e78009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=419c78b50900cd38 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.614+0000 7f25017fa640 1 -- 172.21.15.67:0/2424239788 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v135) v1 ==== 89+0+0 (secure 0 0 0) 0x7f250c037070 con 0x7f2510096a70 2024-09-15T01:56:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.614+0000 7f25017fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T01:56:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.614+0000 7f25017fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:40.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 1 -- 172.21.15.17:0/3379844477 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78007760 con 0x7f9e8009f1e0 2024-09-15T01:56:40.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:40.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: dump: 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e8815d640 5 monclient: authenticate success, global_id 5806 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e8815d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 1 -- 172.21.15.17:0/3379844477 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7802d050 con 0x7f9e8009f1e0 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 1 -- 172.21.15.17:0/3379844477 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78007af0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e84ef7640 10 monclient: dump: 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e8815d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.615+0000 7f9e8815d640 4 set_mon_vals no callback set 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 10 monclient: shutdown 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 -- 172.21.15.17:0/3379844477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8009f1e0 msgr2=0x7f9e80138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8009f1e0 0x7f9e80138e60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009e60 tx=0x7f9e78009780 comp rx=0 tx=0).stop 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 -- 172.21.15.17:0/3379844477 shutdown_connections 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8009f1e0 0x7f9e80138e60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3379844477 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.616+0000 7f9e8815d640 1 -- 172.21.15.17:0/3379844477 >> 172.21.15.17:0/3379844477 conn(0x7f9e80071270 msgr2=0x7f9e800a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:40.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.617+0000 7f9e8815d640 1 -- 172.21.15.17:0/3379844477 shutdown_connections 2024-09-15T01:56:40.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.617+0000 7f9e8815d640 1 -- 172.21.15.17:0/3379844477 wait complete. 2024-09-15T01:56:40.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 msgr2=0x7f24f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 0x7f24f403abf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f25101257c0 tx=0x7f2504008040 comp rx=0 tx=0).stop 2024-09-15T01:56:40.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 10 monclient: shutdown 2024-09-15T01:56:40.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:40.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 msgr2=0x7f251006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f250c035040 tx=0x7f250c032000 comp rx=0 tx=0).stop 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 shutdown_connections 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f4038740 0x7f24f403abf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f251009d5a0 0x7f2510137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25100a1a00 0x7f251012d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 --2- 172.21.15.67:0/2424239788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510096a70 0x7f251006dc40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 >> 172.21.15.67:0/2424239788 conn(0x7f2510071340 msgr2=0x7f251009d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 shutdown_connections 2024-09-15T01:56:40.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.619+0000 7f251799d640 1 -- 172.21.15.67:0/2424239788 wait complete. 2024-09-15T01:56:40.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.618+0000 7f9e8815d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.618+0000 7f9e8815d640 10 monclient: monmap: 2024-09-15T01:56:40.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.618+0000 7f9e8815d640 1 Processor -- start 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.618+0000 7f9e8815d640 1 -- start start 2024-09-15T01:56:40.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.618+0000 7f9e8815d640 10 monclient: init 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e8009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e800a0280 0x7f9e8012d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e800a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8012d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e80131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8012f0c0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e80097810 con 0x7f9e800a0280 2024-09-15T01:56:40.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800981f0 con 0x7f9e80131b20 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e8815d640 10 monclient(hunting): authenticate will time out at 1060.615836s 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38986/0 (socket says 172.21.15.17:38986) 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3224399297 learned_addr learned my addr 172.21.15.17:0/3224399297 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 10 monclient(hunting): get_auth_request con 0x7f9e8009f1e0 auth_method 0 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.619+0000 7f9e85ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8012d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): get_auth_request con 0x7f9e80131b20 auth_method 0 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e856f8640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e800a0280 0x7f9e8012d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e856f8640 10 monclient(hunting): get_auth_request con 0x7f9e800a0280 auth_method 0 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e856f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e856f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e856f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e866fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient(hunting): handle_auth_done global_id 84135 payload 824 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 msgr2=0x7f9e8012d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8012d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e800a0280 msgr2=0x7f9e8012d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e800a0280 0x7f9e8012d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 monclient: found mon.noname-a 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.620+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3224399297 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78009a20 con 0x7f9e8009f1e0 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e85ef9640 1 -- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 msgr2=0x7f9e8009b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e85ef9640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00e8a0 tx=0x7f9e7c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c00ee90 con 0x7f9e8009f1e0 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: dump: 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.635 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e8815d640 5 monclient: authenticate success, global_id 84135 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c00c880 con 0x7f9e8009f1e0 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e8815d640 10 monclient: _renew_subs 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e8815d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8009a990 con 0x7f9e8009f1e0 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c00cc00 con 0x7f9e8009f1e0 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e75ffb640 10 monclient: dump: 2024-09-15T01:56:40.636 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.637 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.622+0000 7f9e77fff640 4 set_mon_vals no callback set 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.623+0000 7f9e8815d640 10 monclient: _renew_subs 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.623+0000 7f9e8815d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.623+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8012ee40 con 0x7f9e8009f1e0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9e7c013410 con 0x7f9e8009f1e0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e8815d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e8815d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e8815d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e800724a0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e75ffb640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 0x7f9e6403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f9e7c04c530 con 0x7f9e8009f1e0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e856f8640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 0x7f9e6403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.624+0000 7f9e856f8640 10 monclient: get_auth_request con 0x7f9e64038740 auth_method 0 2024-09-15T01:56:40.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.627+0000 7f9e856f8640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 0x7f9e6403abf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9e8012e9b0 tx=0x7f9e70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.628+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e7c0136c0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.628+0000 7f9e75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.628+0000 7f9e75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:40.640 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:40.640 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -c barset 2024-09-15T01:56:40.640 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep -q 0 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.712+0000 7ff384d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.712+0000 7ff384d9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.712+0000 7ff384d9d640 10 monclient: monmap: 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:40.714124+0000 2024-09-15T01:56:40.715 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:40.714124+0000 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:40.716 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.713+0000 7ff384d9d640 1 Processor -- start 2024-09-15T01:56:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.713+0000 7ff384d9d640 1 -- start start 2024-09-15T01:56:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.713+0000 7ff384d9d640 10 monclient: init 2024-09-15T01:56:40.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff380131f90 0x7ff380132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff380131f90 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff380098260 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff380137120 0x7ff380139500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff380137120 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff380096a70 con 0x7ff380098260 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38006a6b0 con 0x7ff380131f90 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800a16e0 con 0x7ff380137120 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff384d9d640 10 monclient(hunting): authenticate will time out at 1033.183447s 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37dd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff380098260 0x7ff380098660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40148/0 (socket says 172.21.15.67:40148) 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37dd9b640 1 -- 172.21.15.67:0/2526558290 learned_addr learned my addr 172.21.15.67:0/2526558290 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 --2- 172.21.15.67:0/2526558290 >> v2:172.21.15.67:3300/0 conn(0x7ff380137120 0x7ff380139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 1 --2- 172.21.15.67:0/2526558290 >> v2:172.21.15.17:3300/0 conn(0x7ff380131f90 0x7ff380132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): get_auth_request con 0x7ff380137120 auth_method 0 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): get_auth_request con 0x7ff380131f90 auth_method 0 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_done global_id 5812 payload 274 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 -- 172.21.15.67:0/2526558290 >> v2:172.21.15.17:3300/0 conn(0x7ff380131f90 msgr2=0x7ff380132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 --2- 172.21.15.67:0/2526558290 >> v2:172.21.15.17:3300/0 conn(0x7ff380131f90 0x7ff380132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 -- 172.21.15.67:0/2526558290 >> v1:172.21.15.67:6789/0 conn(0x7ff380098260 legacy=0x7ff380098660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 monclient: found mon.noname-c 2024-09-15T01:56:40.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.714+0000 7ff37ed9d640 1 -- 172.21.15.67:0/2526558290 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800a19c0 con 0x7ff380137120 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37ed9d640 1 -- 172.21.15.67:0/2526558290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380137120 msgr2=0x7ff380139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37ed9d640 1 --2- 172.21.15.67:0/2526558290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380137120 0x7ff380139500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff38006aad0 tx=0x7ff378009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6b3238bdb3827b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 1 -- 172.21.15.67:0/2526558290 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378007760 con 0x7ff380137120 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: dump: 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.727 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 1 -- 172.21.15.67:0/2526558290 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37802d050 con 0x7ff380137120 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff384d9d640 5 monclient: authenticate success, global_id 5812 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff384d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 1 -- 172.21.15.67:0/2526558290 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378007af0 con 0x7ff380137120 2024-09-15T01:56:40.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff37d59a640 10 monclient: dump: 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff384d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.717+0000 7ff384d9d640 4 set_mon_vals no callback set 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 10 monclient: shutdown 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 -- 172.21.15.67:0/2526558290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380137120 msgr2=0x7ff380139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 --2- 172.21.15.67:0/2526558290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380137120 0x7ff380139500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff38006aad0 tx=0x7ff378009780 comp rx=0 tx=0).stop 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 -- 172.21.15.67:0/2526558290 shutdown_connections 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 --2- 172.21.15.67:0/2526558290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380137120 0x7ff380139500 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 --2- 172.21.15.67:0/2526558290 >> v2:172.21.15.17:3300/0 conn(0x7ff380131f90 0x7ff380132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 -- 172.21.15.67:0/2526558290 >> 172.21.15.67:0/2526558290 conn(0x7ff380071690 msgr2=0x7ff3800a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 -- 172.21.15.67:0/2526558290 shutdown_connections 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.718+0000 7ff384d9d640 1 -- 172.21.15.67:0/2526558290 wait complete. 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient: monmap: 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 1 Processor -- start 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 1 -- start start 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient: init 2024-09-15T01:56:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 0x7ff380132fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.719+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff380098260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff380131f90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff380137120 0x7ff380133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff380137120 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800a0d80 con 0x7ff380098260 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380136130 con 0x7ff380131f90 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff380136410 con 0x7ff380137120 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff384d9d640 10 monclient(hunting): authenticate will time out at 1033.189077s 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59644/0 (socket says 172.21.15.67:59644) 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 0x7ff380132fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 -- 172.21.15.67:0/1059066428 learned_addr learned my addr 172.21.15.67:0/1059066428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): get_auth_request con 0x7ff380131f90 auth_method 0 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): get_auth_request con 0x7ff380098260 auth_method 0 2024-09-15T01:56:40.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff380137120 0x7ff380133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): get_auth_request con 0x7ff380137120 auth_method 0 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient(hunting): handle_auth_done global_id 5815 payload 824 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 -- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff380137120 msgr2=0x7ff380133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff380137120 0x7ff380133a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 -- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 msgr2=0x7ff380132fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 0x7ff380132fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 monclient: found mon.noname-b 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37dd9b640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009a20 con 0x7ff380131f90 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.720+0000 7ff37e59c640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 0x7ff380132fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff37dd9b640 1 -- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 msgr2=0x7ff3801334e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff37dd9b640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c5e0 tx=0x7ff374004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374005cb0 con 0x7ff380131f90 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: dump: 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.736 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374005e30 con 0x7ff380131f90 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374007340 con 0x7ff380131f90 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff36e7fc640 10 monclient: dump: 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff384d9d640 5 monclient: authenticate success, global_id 5815 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff384d9d640 10 monclient: _renew_subs 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff384d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff380134400 con 0x7ff380131f90 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.722+0000 7ff37cd99640 4 set_mon_vals no callback set 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 10 monclient: _renew_subs 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380134f60 con 0x7ff380131f90 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff37401b400 con 0x7ff380131f90 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.723+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380135e50 con 0x7ff380131f90 2024-09-15T01:56:40.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.724+0000 7ff36e7fc640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 0x7ff36003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.724+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7ff37404c060 con 0x7ff380131f90 2024-09-15T01:56:40.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.724+0000 7ff37e59c640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 0x7ff36003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.724+0000 7ff37e59c640 10 monclient: get_auth_request con 0x7ff360038740 auth_method 0 2024-09-15T01:56:40.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.726+0000 7ff37e59c640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 0x7ff36003abf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff3700113e0 tx=0x7ff3700090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.728+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff37407de40 con 0x7ff380131f90 2024-09-15T01:56:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.728+0000 7ff36e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.728+0000 7ff36e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e8815d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e8815d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e8815d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e8006efc0 con 0x7f9e8009f1e0 2024-09-15T01:56:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e75ffb640 1 -- 172.21.15.17:0/3224399297 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f9e7c01ab50 con 0x7f9e8009f1e0 2024-09-15T01:56:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.855+0000 7f9e75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:40.858 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 msgr2=0x7f9e6403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 0x7f9e6403abf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9e8012e9b0 tx=0x7f9e70009040 comp rx=0 tx=0).stop 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 10 monclient: shutdown 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 msgr2=0x7f9e8009b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00e8a0 tx=0x7f9e7c008040 comp rx=0 tx=0).stop 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 shutdown_connections 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e64038740 0x7f9e6403abf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.857+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e80131b20 0x7f9e8012d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.858+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e800a0280 0x7f9e8012d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.858+0000 7f9e8815d640 1 --2- 172.21.15.17:0/3224399297 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e8009f1e0 0x7f9e8009b490 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.858+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 >> 172.21.15.17:0/3224399297 conn(0x7f9e80071270 msgr2=0x7f9e80130810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.858+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 shutdown_connections 2024-09-15T01:56:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.858+0000 7f9e8815d640 1 -- 172.21.15.17:0/3224399297 wait complete. 2024-09-15T01:56:40.870 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:56:40.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:56:40.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:56:40.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff384d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:40.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff384d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:40.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff384d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff3800639e0 con 0x7ff380131f90 2024-09-15T01:56:40.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) v1 ==== 71+0+0 (secure 0 0 0) 0x7ff37401b6b0 con 0x7ff380131f90 2024-09-15T01:56:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff36e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:40.898+0000 7ff36e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 10 monclient: monmap: 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:40.965991+0000 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:40.965991+0000 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.967 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:40.968 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 1 Processor -- start 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 1 -- start start 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.965+0000 7fc21df5d640 10 monclient: init 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2180a5a90 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc21809f240 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc218136c60 0x7fc218139040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc218136c60 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2180a5e60 con 0x7fc2180a5a90 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21809ffa0 con 0x7fc21809f240 2024-09-15T01:56:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2180a1860 con 0x7fc218136c60 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc21df5d640 10 monclient(hunting): authenticate will time out at 1060.962431s 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38990/0 (socket says 172.21.15.17:38990) 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 -- 172.21.15.17:0/2622834925 learned_addr learned my addr 172.21.15.17:0/2622834925 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 10 monclient(hunting): get_auth_request con 0x7fc21809f240 auth_method 0 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 1 --2- 172.21.15.17:0/2622834925 >> v2:172.21.15.67:3300/0 conn(0x7fc218136c60 0x7fc218139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 -- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 msgr2=0x7fc218131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:56:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): get_auth_request con 0x7fc218136c60 auth_method 0 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 -- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 msgr2=0x7fc218131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 --2- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc216ffd640 1 --2- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 10 monclient(hunting): handle_auth_done global_id 5818 payload 274 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 1 -- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 msgr2=0x7fc218131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.966+0000 7fc217fff640 1 --2- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 1 -- 172.21.15.17:0/2622834925 >> v1:172.21.15.67:6789/0 conn(0x7fc2180a5a90 legacy=0x7fc21809eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 1 monclient: found mon.noname-c 2024-09-15T01:56:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.967+0000 7fc217fff640 1 -- 172.21.15.17:0/2622834925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180a1580 con 0x7fc218136c60 2024-09-15T01:56:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.969+0000 7fc217fff640 1 -- 172.21.15.17:0/2622834925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc218136c60 msgr2=0x7fc218139040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.969+0000 7fc217fff640 1 --2- 172.21.15.17:0/2622834925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc218136c60 0x7fc218139040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc218132570 tx=0x7fc2100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb999ce5b5b825c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 1 -- 172.21.15.17:0/2622834925 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21002ae00 con 0x7fc218136c60 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: dump: 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 1 -- 172.21.15.17:0/2622834925 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210032440 con 0x7fc218136c60 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 1 -- 172.21.15.17:0/2622834925 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2100327e0 con 0x7fc218136c60 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 5 monclient: authenticate success, global_id 5818 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 4 set_mon_vals no callback set 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc2167fc640 10 monclient: dump: 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 10 monclient: shutdown 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 -- 172.21.15.17:0/2622834925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc218136c60 msgr2=0x7fc218139040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2622834925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc218136c60 0x7fc218139040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc218132570 tx=0x7fc2100090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 -- 172.21.15.17:0/2622834925 shutdown_connections 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2622834925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc218136c60 0x7fc218139040 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2622834925 >> v2:172.21.15.17:3300/0 conn(0x7fc21809f240 0x7fc218131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 -- 172.21.15.17:0/2622834925 >> 172.21.15.17:0/2622834925 conn(0x7fc218070fb0 msgr2=0x7fc2180a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 -- 172.21.15.17:0/2622834925 shutdown_connections 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.970+0000 7fc21df5d640 1 -- 172.21.15.17:0/2622834925 wait complete. 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.971+0000 7fc21df5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.971+0000 7fc21df5d640 10 monclient: monmap: 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.971+0000 7fc21df5d640 1 Processor -- start 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.971+0000 7fc21df5d640 1 -- start start 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.971+0000 7fc21df5d640 10 monclient: init 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21809f240 0x7fc218097380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc21809f240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2180a5a90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc218136c60 0x7fc218097e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc218136c60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21809bec0 con 0x7fc2180a5a90 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21809ee90 con 0x7fc21809f240 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc218098e30 con 0x7fc218136c60 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc21df5d640 10 monclient(hunting): authenticate will time out at 1060.968181s 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39002/0 (socket says 172.21.15.17:39002) 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 -- 172.21.15.17:0/2386419528 learned_addr learned my addr 172.21.15.17:0/2386419528 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): get_auth_request con 0x7fc2180a5a90 auth_method 0 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc218136c60 0x7fc218097e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): get_auth_request con 0x7fc218136c60 auth_method 0 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21809f240 0x7fc218097380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): get_auth_request con 0x7fc21809f240 auth_method 0 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient(hunting): handle_auth_done global_id 84147 payload 824 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc218136c60 msgr2=0x7fc218097e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc218136c60 0x7fc218097e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21809f240 msgr2=0x7fc218097380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21809f240 0x7fc218097380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 monclient: found mon.noname-a 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc216ffd640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc210009390 con 0x7fc2180a5a90 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.972+0000 7fc217fff640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc218136c60 0x7fc218097e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.974+0000 7fc216ffd640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 msgr2=0x7fc2180978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc216ffd640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00e870 tx=0x7fc20c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c00c670 con 0x7fc2180a5a90 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c00c7f0 con 0x7fc2180a5a90 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 5 monclient: authenticate success, global_id 84147 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 10 monclient: _renew_subs 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21809a960 con 0x7fc2180a5a90 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:40.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c00cbb0 con 0x7fc2180a5a90 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.993 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 10 monclient: _renew_subs 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc215ffb640 4 set_mon_vals no callback set 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.975+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2181314f0 con 0x7fc2180a5a90 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc20c013410 con 0x7fc2180a5a90 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc21df5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 0x7fc2181328b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc218063a20 con 0x7fc218098340 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc1ff7fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc204038770 0x7fc20403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fc20c04c3b0 con 0x7fc2180a5a90 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 0x7fc2181328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient: get_auth_request con 0x7fc218098340 auth_method 0 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc217fff640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc204038770 0x7fc20403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc217fff640 10 monclient: get_auth_request con 0x7fc204038770 auth_method 0 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 10 monclient(hunting): handle_auth_done global_id 84150 payload 824 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.976+0000 7fc2177fe640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc21809ae00 con 0x7fc218098340 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc2177fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 msgr2=0x7fc2181328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc2177fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 0x7fc2181328b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc20800fb00 tx=0x7fc208002630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208002dd0 con 0x7fc218098340 2024-09-15T01:56:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc217fff640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc204038770 0x7fc20403ac20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc210031000 tx=0x7fc210032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc20800d8b0 con 0x7fc218098340 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 msgr2=0x7fc2181328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 0x7fc2181328b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc20800fb00 tx=0x7fc208002630 comp rx=0 tx=0).stop 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 msgr2=0x7fc2181328b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.979+0000 7fc1ff7fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218098340 0x7fc2181328b0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc20800fb00 tx=0x7fc208002630 comp rx=0 tx=0).stop 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc21df5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 0x7fc2181331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc21df5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21809aaf0 con 0x7fc218132df0 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 0x7fc2181331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient: get_auth_request con 0x7fc218132df0 auth_method 0 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 10 monclient(hunting): handle_auth_done global_id 84153 payload 824 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.985+0000 7fc2177fe640 1 -- 172.21.15.17:0/2386419528 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc2180724a0 con 0x7fc218132df0 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc2177fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 msgr2=0x7fc2181331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc2177fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 0x7fc2181331f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc208018000 tx=0x7fc208019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208006b10 con 0x7fc218132df0 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.987+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7fc208006c70 con 0x7fc218132df0 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 msgr2=0x7fc2181331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 0x7fc2181331f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc208018000 tx=0x7fc208019000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 msgr2=0x7fc2181331f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc1ff7fe640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc218132df0 0x7fc2181331f0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc208018000 tx=0x7fc208019000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:41.002 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.003 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.004 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.005 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.006 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.008 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:41.009 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.010 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:41.011 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.012 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.013 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:41.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:41.015 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc204038770 msgr2=0x7fc20403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc204038770 0x7fc20403ac20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc210031000 tx=0x7fc210032000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 10 monclient: shutdown 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 msgr2=0x7fc2180978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00e870 tx=0x7fc20c008040 comp rx=0 tx=0).stop 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc217fff640 1 -- 172.21.15.17:0/2386419528 reap_dead start 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 shutdown_connections 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 --2- 172.21.15.17:0/2386419528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc2180a5a90 0x7fc2180978c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 >> 172.21.15.17:0/2386419528 conn(0x7fc218070fb0 msgr2=0x7fc218137ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 shutdown_connections 2024-09-15T01:56:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:40.988+0000 7fc21df5d640 1 -- 172.21.15.17:0/2386419528 wait complete. 2024-09-15T01:56:41.018 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:41.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:56:41.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.001+0000 7ff384d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.001+0000 7ff384d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.001+0000 7ff384d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.001+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff380067a20 con 0x7ff380131f90 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.002+0000 7ff36e7fc640 1 -- 172.21.15.67:0/1059066428 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) v1 ==== 71+0+0 (secure 0 0 0) 0x7ff37401a360 con 0x7ff380131f90 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.002+0000 7ff36e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.002+0000 7ff36e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 msgr2=0x7ff36003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 0x7ff36003abf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff3700113e0 tx=0x7ff3700090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 10 monclient: shutdown 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 msgr2=0x7ff3801334e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c5e0 tx=0x7ff374004960 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 shutdown_connections 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff360038740 0x7ff36003abf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff380137120 0x7ff380133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff380131f90 0x7ff3801334e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 --2- 172.21.15.67:0/1059066428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff380098260 0x7ff380132fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 >> 172.21.15.67:0/1059066428 conn(0x7ff380071690 msgr2=0x7ff3800a4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 shutdown_connections 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.004+0000 7ff384d9d640 1 -- 172.21.15.67:0/1059066428 wait complete. 2024-09-15T01:56:41.021 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.088+0000 7f5f10899640 10 monclient: get_monmap_and_config 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.088+0000 7f5f10899640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.088+0000 7f5f10899640 10 monclient: monmap: 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.091 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:41.090118+0000 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:41.090118+0000 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:41.093 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.089+0000 7f5f10899640 1 Processor -- start 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.089+0000 7f5f10899640 1 -- start start 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.089+0000 7f5f10899640 10 monclient: init 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f0c09cb00 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c0672c0 0x7f5f0c09ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0c0672c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0c09d410 0x7f5f0c1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-c con 0x7f5f0c09d410 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c067690 con 0x7f5f0c09cb00 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0c096a70 con 0x7f5f0c09d410 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c09e4e0 con 0x7f5f0c0672c0 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f10899640 10 monclient(hunting): authenticate will time out at 1033.559294s 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0c09d410 0x7f5f0c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5f0c09cb00 0x7f5f0c096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40164/0 (socket says 172.21.15.67:40164) 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0c09d410 0x7f5f0c1391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59654/0 (socket says 172.21.15.67:59654) 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0a59c640 1 -- 172.21.15.67:0/4146720251 learned_addr learned my addr 172.21.15.67:0/4146720251 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): get_auth_request con 0x7f5f0c09d410 auth_method 0 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c0672c0 0x7f5f0c09ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): get_auth_request con 0x7f5f0c0672c0 auth_method 0 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_done global_id 5824 payload 274 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 -- 172.21.15.67:0/4146720251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c0672c0 msgr2=0x7f5f0c09ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c0672c0 0x7f5f0c09ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 -- 172.21.15.67:0/4146720251 >> v1:172.21.15.67:6789/0 conn(0x7f5f0c09cb00 legacy=0x7f5f0c096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 monclient: found mon.noname-c 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.090+0000 7f5f0ad9d640 1 -- 172.21.15.67:0/4146720251 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c09e200 con 0x7f5f0c09d410 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0ad9d640 1 -- 172.21.15.67:0/4146720251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c09d410 msgr2=0x7f5f0c1391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0ad9d640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c09d410 0x7f5f0c1391b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009850 tx=0x7f5f040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f172874566577464 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 1 -- 172.21.15.67:0/4146720251 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0402adf0 con 0x7f5f0c09d410 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: dump: 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.104 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 1 -- 172.21.15.67:0/4146720251 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f04032440 con 0x7f5f0c09d410 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f10899640 5 monclient: authenticate success, global_id 5824 2024-09-15T01:56:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f10899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f10899640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 1 -- 172.21.15.67:0/4146720251 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f040327e0 con 0x7f5f0c09d410 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f10899640 4 set_mon_vals no callback set 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.093+0000 7f5f0959a640 10 monclient: dump: 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 10 monclient: shutdown 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 -- 172.21.15.67:0/4146720251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c09d410 msgr2=0x7f5f0c1391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c09d410 0x7f5f0c1391b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009850 tx=0x7f5f040090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 -- 172.21.15.67:0/4146720251 shutdown_connections 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c09d410 0x7f5f0c1391b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 --2- 172.21.15.67:0/4146720251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c0672c0 0x7f5f0c09ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 -- 172.21.15.67:0/4146720251 >> 172.21.15.67:0/4146720251 conn(0x7f5f0c0a2110 msgr2=0x7f5f0c0a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 -- 172.21.15.67:0/4146720251 shutdown_connections 2024-09-15T01:56:41.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 1 -- 172.21.15.67:0/4146720251 wait complete. 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.094+0000 7f5f10899640 10 monclient: monmap: 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 Processor -- start 2024-09-15T01:56:41.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 -- start start 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient: init 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0c0672c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c09cb00 0x7f5f0c1376a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-c con 0x7f5f0c09cb00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f0c09d410 0x7f5f0c137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0c09d410 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:41.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c1394e0 con 0x7f5f0c09d410 2024-09-15T01:56:41.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c06d260 con 0x7f5f0c0672c0 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c06b450 con 0x7f5f0c09cb00 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f10899640 10 monclient(hunting): authenticate will time out at 1033.564581s 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59656/0 (socket says 172.21.15.67:59656) 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 -- 172.21.15.67:0/2619584100 learned_addr learned my addr 172.21.15.67:0/2619584100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): get_auth_request con 0x7f5f0c0672c0 auth_method 0 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c09cb00 0x7f5f0c1376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): get_auth_request con 0x7f5f0c09cb00 auth_method 0 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f0c09d410 0x7f5f0c137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): get_auth_request con 0x7f5f0c09d410 auth_method 0 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient(hunting): handle_auth_done global_id 5827 payload 824 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 -- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c09cb00 msgr2=0x7f5f0c1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c09cb00 0x7f5f0c1376a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 -- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f0c09d410 msgr2=0x7f5f0c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f0c09d410 0x7f5f0c137be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 monclient: found mon.noname-b 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.095+0000 7f5f0a59c640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04009390 con 0x7f5f0c0672c0 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f0a59c640 1 -- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 msgr2=0x7f5f0c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f0a59c640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00ccd0 tx=0x7f5efc0053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00aa60 con 0x7f5f0c0672c0 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: dump: 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.114 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc00abe0 con 0x7f5f0c0672c0 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 5 monclient: authenticate success, global_id 5827 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 10 monclient: _renew_subs 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c06cf80 con 0x7f5f0c0672c0 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0070f0 con 0x7f5f0c0672c0 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f027fc640 10 monclient: dump: 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f08d99640 4 set_mon_vals no callback set 2024-09-15T01:56:41.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 10 monclient: _renew_subs 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.098+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c06e180 con 0x7f5f0c0672c0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5efc007520 con 0x7f5f0c0672c0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f10899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f10899640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f10899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f027fc640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 0x7f5eec03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f5efc04c6b0 con 0x7f5f0c0672c0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f09d9b640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 0x7f5eec03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.099+0000 7f5f09d9b640 10 monclient: get_auth_request con 0x7f5eec038790 auth_method 0 2024-09-15T01:56:41.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.102+0000 7f5f09d9b640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 0x7f5eec03ac40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c067ef0 tx=0x7f5ef40090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.103+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.103+0000 7f5f027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.103+0000 7f5f027fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.116+0000 7f7a7655d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.116+0000 7f7a7655d640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.116+0000 7f7a7655d640 10 monclient: monmap: 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:41.117288+0000 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:41.117288+0000 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:41.119 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:41.120 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:41.120 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 Processor -- start 2024-09-15T01:56:41.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 -- start start 2024-09-15T01:56:41.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient: init 2024-09-15T01:56:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a70068f20 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a700692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 0x7f7a7013a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a70137ff0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700a1950 con 0x7f7a700692f0 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700a1670 con 0x7f7a70068f20 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a701328d0 con 0x7f7a70137ff0 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a7655d640 10 monclient(hunting): authenticate will time out at 1061.113804s 2024-09-15T01:56:41.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a74afa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 0x7f7a7013a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a6f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7a700692f0 0x7f7a700696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47028/0 (socket says 172.21.15.17:47028) 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/1313333753 learned_addr learned my addr 172.21.15.17:0/1313333753 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a74afa640 10 monclient(hunting): get_auth_request con 0x7f7a70137ff0 auth_method 0 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a74afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a74afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.117+0000 7f7a74afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient(hunting): handle_auth_done global_id 74363 payload 274 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 1 -- 172.21.15.17:0/1313333753 >> v1:172.21.15.67:6789/0 conn(0x7f7a70068f20 legacy=0x7f7a701321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 1 -- 172.21.15.17:0/1313333753 >> v1:172.21.15.17:6789/0 conn(0x7f7a700692f0 legacy=0x7f7a700696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 1 monclient: found mon.noname-b 2024-09-15T01:56:41.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.118+0000 7f7a74afa640 1 -- 172.21.15.17:0/1313333753 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700a0020 con 0x7f7a70137ff0 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a74afa640 1 -- 172.21.15.17:0/1313333753 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 msgr2=0x7f7a7013a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a74afa640 1 --2- 172.21.15.17:0/1313333753 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 0x7f7a7013a3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009ec0 tx=0x7f7a68009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1c216bfb3eba56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 1 -- 172.21.15.17:0/1313333753 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68007840 con 0x7f7a70137ff0 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: dump: 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 1 -- 172.21.15.17:0/1313333753 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6802d050 con 0x7f7a70137ff0 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 5 monclient: authenticate success, global_id 74363 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:41.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 1 -- 172.21.15.17:0/1313333753 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68007bd0 con 0x7f7a70137ff0 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 4 set_mon_vals no callback set 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a6effd640 10 monclient: dump: 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.132 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 10 monclient: shutdown 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 -- 172.21.15.17:0/1313333753 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 msgr2=0x7f7a7013a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 --2- 172.21.15.17:0/1313333753 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 0x7f7a7013a3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009ec0 tx=0x7f7a68009780 comp rx=0 tx=0).stop 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 -- 172.21.15.17:0/1313333753 shutdown_connections 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 --2- 172.21.15.17:0/1313333753 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70137ff0 0x7f7a7013a3d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 -- 172.21.15.17:0/1313333753 >> 172.21.15.17:0/1313333753 conn(0x7f7a70070fb0 msgr2=0x7f7a700a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 -- 172.21.15.17:0/1313333753 shutdown_connections 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.121+0000 7f7a7655d640 1 -- 172.21.15.17:0/1313333753 wait complete. 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.122+0000 7f7a7655d640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.122+0000 7f7a7655d640 10 monclient: monmap: 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.122+0000 7f7a7655d640 1 Processor -- start 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.122+0000 7f7a7655d640 1 -- start start 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.122+0000 7f7a7655d640 10 monclient: init 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a70068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a700692f0 0x7f7a70141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a700692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70137ff0 0x7f7a70134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a70137ff0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a700a0750 con 0x7f7a700692f0 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7009c550 con 0x7f7a70137ff0 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7011bba0 con 0x7f7a70068f20 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a7655d640 10 monclient(hunting): authenticate will time out at 1061.119220s 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a700692f0 0x7f7a70141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a700692f0 0x7f7a70141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39020/0 (socket says 172.21.15.17:39020) 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38166/0 (socket says 172.21.15.17:38166) 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/2272668722 learned_addr learned my addr 172.21.15.17:0/2272668722 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f7a700692f0 auth_method 0 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): get_auth_request con 0x7f7a70068f20 auth_method 0 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70137ff0 0x7f7a70134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): get_auth_request con 0x7f7a70137ff0 auth_method 0 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a74afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient(hunting): handle_auth_done global_id 74366 payload 824 2024-09-15T01:56:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70137ff0 msgr2=0x7f7a70134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70137ff0 0x7f7a70134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a700692f0 msgr2=0x7f7a70141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a700692f0 0x7f7a70141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 monclient: found mon.noname-c 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.123+0000 7f7a6ffff640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68009a20 con 0x7f7a70068f20 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.125+0000 7f7a6ffff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 msgr2=0x7f7a70141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.125+0000 7f7a6ffff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c540 tx=0x7f7a60007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a600117a0 con 0x7f7a70068f20 2024-09-15T01:56:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: dump: 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 5 monclient: authenticate success, global_id 74366 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a60011900 con 0x7f7a70068f20 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 10 monclient: _renew_subs 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a701365b0 con 0x7f7a70068f20 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60011cc0 con 0x7f7a70068f20 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 10 monclient: dump: 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 10 monclient: _renew_subs 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a6e7fc640 4 set_mon_vals no callback set 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a701378c0 con 0x7f7a70068f20 2024-09-15T01:56:41.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a7655d640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 0x7f7a700298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.126+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a60012630 con 0x7f7a70068f20 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70063b90 con 0x7f7a70135260 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 0x7f7a700298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient: get_auth_request con 0x7f7a70135260 auth_method 0 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a57fff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f7a6004c0a0 con 0x7f7a70068f20 2024-09-15T01:56:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_done global_id 74369 payload 824 2024-09-15T01:56:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a700724a0 con 0x7f7a70135260 2024-09-15T01:56:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a74afa640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.127+0000 7f7a74afa640 10 monclient: get_auth_request con 0x7f7a5c038740 auth_method 0 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 msgr2=0x7f7a700298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a6f7fe640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 0x7f7a700298c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a64004eb0 tx=0x7f7a64004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64005cb0 con 0x7f7a70135260 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a57fff640 10 monclient: dump: 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.129+0000 7f7a74afa640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009e90 tx=0x7f7a68037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7a64005e10 con 0x7f7a70135260 2024-09-15T01:56:41.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 msgr2=0x7f7a700298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 0x7f7a700298c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a64004eb0 tx=0x7f7a64004960 comp rx=0 tx=0).stop 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 msgr2=0x7f7a700298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.130+0000 7f7a57fff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135260 0x7f7a700298c0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a64004eb0 tx=0x7f7a64004960 comp rx=0 tx=0).stop 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.135+0000 7f7a7655d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.135+0000 7f7a7655d640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 0x7f7a70029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.135+0000 7f7a7655d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.135+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7011bd10 con 0x7f7a70135630 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 0x7f7a70029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient: get_auth_request con 0x7f7a70135630 auth_method 0 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 10 monclient(hunting): handle_auth_done global_id 74372 payload 824 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.136+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/2272668722 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a7009a690 con 0x7f7a70135630 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a6f7fe640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 msgr2=0x7f7a70029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a6f7fe640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 0x7f7a70029e00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a64017000 tx=0x7f7a64018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64021620 con 0x7f7a70135630 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 10 monclient: dump: 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.150 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f7a64021780 con 0x7f7a70135630 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 msgr2=0x7f7a70029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 0x7f7a70029e00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a64017000 tx=0x7f7a64018000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 msgr2=0x7f7a70029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.138+0000 7f7a57fff640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70135630 0x7f7a70029e00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a64017000 tx=0x7f7a64018000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:41.152 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.153 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.154 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:41.155 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.159 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.160 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.163 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:41.164 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:41.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 msgr2=0x7f7a5c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009e90 tx=0x7f7a68037000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 10 monclient: shutdown 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 msgr2=0x7f7a70141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c540 tx=0x7f7a60007e10 comp rx=0 tx=0).stop 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a74afa640 1 -- 172.21.15.17:0/2272668722 reap_dead start 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 shutdown_connections 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 --2- 172.21.15.17:0/2272668722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a70068f20 0x7f7a70141680 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 >> 172.21.15.17:0/2272668722 conn(0x7f7a70070fb0 msgr2=0x7f7a7009def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 shutdown_connections 2024-09-15T01:56:41.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.139+0000 7f7a7655d640 1 -- 172.21.15.17:0/2272668722 wait complete. 2024-09-15T01:56:41.167 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:41.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:56:41.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:56:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.262+0000 7f34d0300640 10 monclient: get_monmap_and_config 2024-09-15T01:56:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.262+0000 7f34d0300640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.262+0000 7f34d0300640 10 monclient: monmap: 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:41.263449+0000 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:41.263449+0000 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:41.266 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 Processor -- start 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 -- start start 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient: init 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34c809c1b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c809e9e0 0x7f34c809c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-c con 0x7f34c809e9e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-b con 0x7f34c806eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c809f790 con 0x7f34c809c1b0 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c8069d40 con 0x7f34c809e9e0 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c809ee20 con 0x7f34c806eb30 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.263+0000 7f34d0300640 10 monclient(hunting): authenticate will time out at 1061.259889s 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38188/0 (socket says 172.21.15.17:38188) 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 -- 172.21.15.17:0/3769720767 learned_addr learned my addr 172.21.15.17:0/3769720767 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): get_auth_request con 0x7f34c806eb30 auth_method 0 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3769720767 >> v2:172.21.15.67:3300/0 conn(0x7f34c809e9e0 0x7f34c809c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): get_auth_request con 0x7f34c809e9e0 auth_method 0 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient(hunting): handle_auth_done global_id 74375 payload 274 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 -- 172.21.15.17:0/3769720767 >> v2:172.21.15.67:3300/0 conn(0x7f34c809e9e0 msgr2=0x7f34c809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 --2- 172.21.15.17:0/3769720767 >> v2:172.21.15.67:3300/0 conn(0x7f34c809e9e0 0x7f34c809c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 -- 172.21.15.17:0/3769720767 >> v1:172.21.15.67:6789/0 conn(0x7f34c809c1b0 legacy=0x7f34c809a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 monclient: found mon.noname-b 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.264+0000 7f34ce89d640 1 -- 172.21.15.17:0/3769720767 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8068b60 con 0x7f34c806eb30 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34ce89d640 1 -- 172.21.15.17:0/3769720767 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 msgr2=0x7f34c806ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34ce89d640 1 --2- 172.21.15.17:0/3769720767 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009f20 tx=0x7f34c0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe47e8eeccff790d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 1 -- 172.21.15.17:0/3769720767 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0007610 con 0x7f34c806eb30 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: dump: 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 1 -- 172.21.15.17:0/3769720767 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c002d050 con 0x7f34c806eb30 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 5 monclient: authenticate success, global_id 74375 2024-09-15T01:56:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 1 -- 172.21.15.17:0/3769720767 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c00079e0 con 0x7f34c806eb30 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 4 set_mon_vals no callback set 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34cd09a640 10 monclient: dump: 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 10 monclient: shutdown 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 1 -- 172.21.15.17:0/3769720767 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 msgr2=0x7f34c806ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.267+0000 7f34d0300640 1 --2- 172.21.15.17:0/3769720767 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009f20 tx=0x7f34c0009840 comp rx=0 tx=0).stop 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 -- 172.21.15.17:0/3769720767 shutdown_connections 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 --2- 172.21.15.17:0/3769720767 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c806ef90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 --2- 172.21.15.17:0/3769720767 >> v2:172.21.15.67:3300/0 conn(0x7f34c809e9e0 0x7f34c809c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 -- 172.21.15.17:0/3769720767 >> 172.21.15.17:0/3769720767 conn(0x7f34c8071300 msgr2=0x7f34c80a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 -- 172.21.15.17:0/3769720767 shutdown_connections 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 -- 172.21.15.17:0/3769720767 wait complete. 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 10 monclient: monmap: 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 Processor -- start 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 1 -- start start 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.268+0000 7f34d0300640 10 monclient: init 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c81364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-c con 0x7f34c806eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c809c1b0 0x7f34c8133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-b con 0x7f34c809c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34c809e9e0 0x7f34c8134180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): picked mon.noname-a con 0x7f34c809e9e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c809ce20 con 0x7f34c809e9e0 2024-09-15T01:56:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8134bc0 con 0x7f34c809c1b0 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8121060 con 0x7f34c806eb30 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34d0300640 10 monclient(hunting): authenticate will time out at 1061.265325s 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c81364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c81364b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38190/0 (socket says 172.21.15.17:38190) 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 -- 172.21.15.17:0/3531875364 learned_addr learned my addr 172.21.15.17:0/3531875364 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): get_auth_request con 0x7f34c806eb30 auth_method 0 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34c809e9e0 0x7f34c8134180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c809c1b0 0x7f34c8133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): get_auth_request con 0x7f34c809e9e0 auth_method 0 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): get_auth_request con 0x7f34c809c1b0 auth_method 0 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient(hunting): handle_auth_done global_id 74378 payload 824 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c809c1b0 msgr2=0x7f34c8133c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c809c1b0 0x7f34c8133c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34c809e9e0 msgr2=0x7f34c8134180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34c809e9e0 0x7f34c8134180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 monclient: found mon.noname-c 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.269+0000 7f34ce09c640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c0009ae0 con 0x7f34c806eb30 2024-09-15T01:56:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.271+0000 7f34ce09c640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 msgr2=0x7f34c81364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34ce09c640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c81364b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c5e0 tx=0x7f34c4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40117a0 con 0x7f34c806eb30 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: dump: 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 5 monclient: authenticate success, global_id 74378 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4011900 con 0x7f34c806eb30 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 monclient: _renew_subs 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4011d00 con 0x7f34c806eb30 2024-09-15T01:56:41.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c811f250 con 0x7f34c806eb30 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34be7fc640 10 monclient: dump: 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34cc899640 4 set_mon_vals no callback set 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 monclient: _renew_subs 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c811da50 con 0x7f34c806eb30 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 0x7f34c80a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.272+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8063bb0 con 0x7f34c80a1bf0 2024-09-15T01:56:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f34c4012630 con 0x7f34c806eb30 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 0x7f34c80a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient: get_auth_request con 0x7f34c80a1bf0 auth_method 0 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34be7fc640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34ac038740 0x7f34ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 10 monclient(hunting): handle_auth_done global_id 5839 payload 824 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f34c404c1a0 con 0x7f34c806eb30 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34cd89b640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34c811ee50 con 0x7f34c80a1bf0 2024-09-15T01:56:41.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34ce89d640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34ac038740 0x7f34ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.273+0000 7f34ce89d640 10 monclient: get_auth_request con 0x7f34ac038740 auth_method 0 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34cd89b640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 msgr2=0x7f34c80a1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 0x7f34c80a1fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34c8099a20 tx=0x7f34b80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8014440 con 0x7f34c80a1bf0 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.275+0000 7f34be7fc640 10 monclient: dump: 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34ce89d640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34ac038740 0x7f34ac03abf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f34c8068b30 tx=0x7f34c0036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f34b80145a0 con 0x7f34c80a1bf0 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 msgr2=0x7f34c80a1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 0x7f34c80a1fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34c8099a20 tx=0x7f34b80090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 msgr2=0x7f34c80a1fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:41.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.276+0000 7f34be7fc640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a1bf0 0x7f34c80a1fc0 secure :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34c8099a20 tx=0x7f34b80090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.272+0000 7f5f10899640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.272+0000 7f5f10899640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.272+0000 7f5f10899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.272+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34d0300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34d0300640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 0x7f34c811dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34d0300640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c806a3c0 con 0x7f34c80a2500 2024-09-15T01:56:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 0x7f34c811dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient: get_auth_request con 0x7f34c80a2500 auth_method 0 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 10 monclient(hunting): handle_auth_done global_id 5842 payload 824 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.282+0000 7f34cd89b640 1 -- 172.21.15.17:0/3531875364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f34c80724a0 con 0x7f34c80a2500 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.284+0000 7f34cd89b640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 msgr2=0x7f34c811dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.284+0000 7f34cd89b640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 0x7f34c811dcd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34c8099a20 tx=0x7f34b801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b801de80 con 0x7f34c80a2500 2024-09-15T01:56:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 10 monclient: dump: 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f34b801c3e0 con 0x7f34c80a2500 2024-09-15T01:56:41.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 msgr2=0x7f34c811dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 0x7f34c811dcd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34c8099a20 tx=0x7f34b801c000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 msgr2=0x7f34c811dcd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34be7fc640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c80a2500 0x7f34c811dcd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:41.301 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.302 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.303 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:41.304 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:41.305 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.306 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:41.308 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:41.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:41.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:41.312 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.313 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:41.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:41.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:41.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:41.315 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:41.315 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34ac038740 msgr2=0x7f34ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34ac038740 0x7f34ac03abf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f34c8068b30 tx=0x7f34c0036000 comp rx=0 tx=0).stop 2024-09-15T01:56:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 10 monclient: shutdown 2024-09-15T01:56:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 msgr2=0x7f34c81364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 --2- 172.21.15.17:0/3531875364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34c806eb30 0x7f34c81364b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c5e0 tx=0x7f34c4007ed0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34ce89d640 1 -- 172.21.15.17:0/3531875364 reap_dead start 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34ce89d640 1 -- 172.21.15.17:0/3531875364 reap_dead start 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 shutdown_connections 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 >> 172.21.15.17:0/3531875364 conn(0x7f34c8071300 msgr2=0x7f34c8134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 shutdown_connections 2024-09-15T01:56:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.285+0000 7f34d0300640 1 -- 172.21.15.17:0/3531875364 wait complete. 2024-09-15T01:56:41.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:41.317 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:56:41.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:56:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.411+0000 7fc7aac99640 10 monclient: get_monmap_and_config 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.411+0000 7fc7aac99640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.411+0000 7fc7aac99640 10 monclient: monmap: 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:41.412786+0000 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:41.412786+0000 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:41.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:41.415 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:41.415 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:41.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.412+0000 7fc7aac99640 1 Processor -- start 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.412+0000 7fc7aac99640 1 -- start start 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.412+0000 7fc7aac99640 10 monclient: init 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7a4068f20 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7a40692f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7a4137ff0 0x7fc7a413a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a4137ff0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40a1950 con 0x7fc7a40692f0 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40a1670 con 0x7fc7a4068f20 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a41328d0 con 0x7fc7a4137ff0 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7aac99640 10 monclient(hunting): authenticate will time out at 1061.409284s 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a3fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc7a40692f0 0x7fc7a40696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47032/0 (socket says 172.21.15.17:47032) 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a3fff640 1 -- 172.21.15.17:0/869281015 learned_addr learned my addr 172.21.15.17:0/869281015 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 1 --2- 172.21.15.17:0/869281015 >> v2:172.21.15.67:3300/0 conn(0x7fc7a4137ff0 0x7fc7a413a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): get_auth_request con 0x7fc7a4137ff0 auth_method 0 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient(hunting): handle_auth_done global_id 5845 payload 274 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 1 -- 172.21.15.17:0/869281015 >> v1:172.21.15.67:6789/0 conn(0x7fc7a4068f20 legacy=0x7fc7a41321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 1 -- 172.21.15.17:0/869281015 >> v1:172.21.15.17:6789/0 conn(0x7fc7a40692f0 legacy=0x7fc7a40696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 1 monclient: found mon.noname-c 2024-09-15T01:56:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.413+0000 7fc7a9236640 1 -- 172.21.15.17:0/869281015 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40a0020 con 0x7fc7a4137ff0 2024-09-15T01:56:41.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.416+0000 7fc7a9236640 1 -- 172.21.15.17:0/869281015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a4137ff0 msgr2=0x7fc7a413a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.416+0000 7fc7a9236640 1 --2- 172.21.15.17:0/869281015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a4137ff0 0x7fc7a413a3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009ec0 tx=0x7fc79c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0026e85df45ea97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 1 -- 172.21.15.17:0/869281015 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c007840 con 0x7fc7a4137ff0 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: dump: 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 1 -- 172.21.15.17:0/869281015 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c02d050 con 0x7fc7a4137ff0 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 1 -- 172.21.15.17:0/869281015 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c007bd0 con 0x7fc7a4137ff0 2024-09-15T01:56:41.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7aac99640 5 monclient: authenticate success, global_id 5845 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7aac99640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7aac99640 4 set_mon_vals no callback set 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.417+0000 7fc7a37fe640 10 monclient: dump: 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.426 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 10 monclient: shutdown 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 -- 172.21.15.17:0/869281015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a4137ff0 msgr2=0x7fc7a413a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 --2- 172.21.15.17:0/869281015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a4137ff0 0x7fc7a413a3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009ec0 tx=0x7fc79c009780 comp rx=0 tx=0).stop 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 -- 172.21.15.17:0/869281015 shutdown_connections 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 --2- 172.21.15.17:0/869281015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a4137ff0 0x7fc7a413a3d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 -- 172.21.15.17:0/869281015 >> 172.21.15.17:0/869281015 conn(0x7fc7a4070fb0 msgr2=0x7fc7a40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 -- 172.21.15.17:0/869281015 shutdown_connections 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 -- 172.21.15.17:0/869281015 wait complete. 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 10 monclient: monmap: 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.418+0000 7fc7aac99640 1 Processor -- start 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 -- start start 2024-09-15T01:56:41.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient: init 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a4068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40692f0 0x7fc7a4134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a40692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 0x7fc7a4135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a4137ff0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40a0750 con 0x7fc7a4068f20 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a409c5f0 con 0x7fc7a40692f0 2024-09-15T01:56:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a411bb50 con 0x7fc7a4137ff0 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7aac99640 10 monclient(hunting): authenticate will time out at 1061.415654s 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39034/0 (socket says 172.21.15.17:39034) 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 0x7fc7a4135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 1 -- 172.21.15.17:0/3996568636 learned_addr learned my addr 172.21.15.17:0/3996568636 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): get_auth_request con 0x7fc7a4068f20 auth_method 0 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): get_auth_request con 0x7fc7a4137ff0 auth_method 0 2024-09-15T01:56:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a8a35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40692f0 0x7fc7a4134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a9236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request con 0x7fc7a40692f0 auth_method 0 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.419+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 10 monclient(hunting): handle_auth_done global_id 84165 payload 824 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 -- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 msgr2=0x7fc7a4135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 0x7fc7a4135250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 -- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40692f0 msgr2=0x7fc7a4134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40692f0 0x7fc7a4134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 monclient: found mon.noname-a 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a8a35640 1 -- 172.21.15.17:0/3996568636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c009a20 con 0x7fc7a4068f20 2024-09-15T01:56:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.420+0000 7fc7a9236640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 0x7fc7a4135250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a8a35640 1 -- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 msgr2=0x7fc7a41347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a8a35640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c5e0 tx=0x7fc798007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798012840 con 0x7fc7a4068f20 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: dump: 2024-09-15T01:56:41.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 5 monclient: authenticate success, global_id 84165 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 10 monclient: _renew_subs 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7980129c0 con 0x7fc7a4068f20 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4137920 con 0x7fc7a4068f20 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798012d80 con 0x7fc7a4068f20 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a0ff9640 10 monclient: dump: 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:41.437 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7a2ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 10 monclient: _renew_subs 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.422+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a4136cb0 con 0x7fc7a4068f20 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc798011590 con 0x7fc7a4068f20 2024-09-15T01:56:41.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7aac99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7aac99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7aac99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a411bd90 con 0x7fc7a4068f20 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7a0ff9640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 0x7fc78803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fc79804c540 con 0x7fc7a4068f20 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 0x7fc78803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.423+0000 7fc7a3fff640 10 monclient: get_auth_request con 0x7fc788038740 auth_method 0 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.426+0000 7fc7a3fff640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 0x7fc78803abf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4131a70 tx=0x7fc790007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.428+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc7980108c0 con 0x7fc7a4068f20 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.428+0000 7fc7a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.428+0000 7fc7a0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:41.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.617+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v136) v1 ==== 82+0+0 (secure 0 0 0) 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.617+0000 7f5f027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.617+0000 7f5f027fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.654+0000 7fc7aac99640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.654+0000 7fc7aac99640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.654+0000 7fc7aac99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.654+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc7a40724a0 con 0x7fc7a4068f20 2024-09-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fc7a0ff9640 1 -- 172.21.15.17:0/3996568636 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc798016020 con 0x7fc7a4068f20 2024-09-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fc7a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:56:41.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fc7a0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:41.658 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.658 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=9,kv=8,logm=68} crc {auth=36912235,config=1228700967,health=1084682601,kv=1529054990,logm=1004344204}) 2024-09-15T01:56:41.659 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.655+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=4195919931,config=3140514009,health=3070137136,kv=1529054990,logm=1638102750}) 2024-09-15T01:56:41.659 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.656+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.659 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.656+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=9,kv=8,logm=68} crc {auth=36912235,config=1228700967,health=1084682601,kv=1529054990,logm=1004344204}) 2024-09-15T01:56:41.659 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.656+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=534062848,config=3140514009,health=3070137136,kv=1529054990,logm=1963007906}) 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334584722}) 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=713970924}) 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334584722}) 2024-09-15T01:56:41.660 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641851280}) 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 msgr2=0x7fc78803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 0x7fc78803abf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4131a70 tx=0x7fc790007f00 comp rx=0 tx=0).stop 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 10 monclient: shutdown 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 msgr2=0x7fc7a41347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c5e0 tx=0x7fc798007ed0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 shutdown_connections 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc788038740 0x7fc78803abf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7a4137ff0 0x7fc7a4135250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a40692f0 0x7fc7a4134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 --2- 172.21.15.17:0/3996568636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc7a4068f20 0x7fc7a41347d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.657+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 >> 172.21.15.17:0/3996568636 conn(0x7fc7a4070fb0 msgr2=0x7fc7a409ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 shutdown_connections 2024-09-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fc7aac99640 1 -- 172.21.15.17:0/3996568636 wait complete. 2024-09-15T01:56:41.664 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.664 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648539100}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2712026665}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648539100}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.658+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2314890438}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373259551}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3437167465}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373259551}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.659+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125415072}) 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.665 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877127009}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356507735}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877127009}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.660+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143444047}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926239966}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3973260857}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926239966}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.661+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=754069453}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976961960}) 2024-09-15T01:56:41.666 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2580479297}) 2024-09-15T01:56:41.667 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.667 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976961960}) 2024-09-15T01:56:41.667 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.662+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179876224}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786854295}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320868703}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786854295}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.663+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007163815}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488382377}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803834406}) 2024-09-15T01:56:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488382377}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.664+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370825865}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993883664}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=191385277}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993883664}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.665+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132310798}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547335650}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179861663}) 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547335650}) 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.666+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097153482}) 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525121736}) 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1650939955}) 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525121736}) 2024-09-15T01:56:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.667+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351072428}) 2024-09-15T01:56:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193387613}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119317326}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193387613}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.668+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335128365}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923735834}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359867236}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923735834}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.669+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1144487460}) 2024-09-15T01:56:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10623181}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372157965}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10623181}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.670+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443538284}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.671+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.671+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106297866}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.671+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780182651}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.671+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.671+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106297866}) 2024-09-15T01:56:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.672+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239694835}) 2024-09-15T01:56:41.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:56:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.672+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.672+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774755328}) 2024-09-15T01:56:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.673+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670594784}) 2024-09-15T01:56:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.673+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.673+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774755328}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.673+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783317164}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917623249}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628264205}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917623249}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.674+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335456058}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390042316}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4069057764}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390042316}) 2024-09-15T01:56:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.675+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118223293}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49371009}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903521512}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49371009}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.676+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542963293}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174310695}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1918882266}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174310695}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.677+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098416240}) 2024-09-15T01:56:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.683 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1351053552,mdsmap=1531151339}) 2024-09-15T01:56:41.683 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521964382}) 2024-09-15T01:56:41.683 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.683 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1351053552,mdsmap=1531151339}) 2024-09-15T01:56:41.683 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.678+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3514142321}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3205356370,mon_config_key=1282474241,monmap=3178194974,osd_metadata=1079644522,osd_pg_creating=224241314,osd_snap=1063953655,osdmap=4084490867}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=4173831893,mgr_command_descs=3545737493,mgr_metadata=2029235354,mgrstat=1260453028,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=224241314,osd_snap=1063953654,osdmap=3902019640}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3205356370,mon_config_key=1282474241,monmap=3178194974,osd_metadata=1079644522,osd_pg_creating=224241314,osd_snap=1063953655,osdmap=4084490867}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.679+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2472462377,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1149731095,osd_pg_creating=224241314,osd_snap=1063953654,osdmap=2733913828}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3954710038}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4271004238}) 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3954710038}) 2024-09-15T01:56:41.687 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.680+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=502490009}) 2024-09-15T01:56:41.687 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1308791695}) 2024-09-15T01:56:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=3519264576}) 2024-09-15T01:56:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:56:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1308791695}) 2024-09-15T01:56:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:56:41.685+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=860769703}) 2024-09-15T01:56:41.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.734+0000 7f5f10899640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.734+0000 7f5f10899640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.734+0000 7f5f10899640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.734+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.735+0000 7f5f027fc640 1 -- 172.21.15.67:0/2619584100 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v136) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5f0c06e460 con 0x7f5f0c0672c0 2024-09-15T01:56:41.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.735+0000 7f5f027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:56:41.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.735+0000 7f5f027fc640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T01:56:41.737 INFO:tasks.workunit.client.0.smithi067.stderr:compat weight-set already created 2024-09-15T01:56:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 msgr2=0x7f5eec03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 0x7f5eec03ac40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c067ef0 tx=0x7f5ef40090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 10 monclient: shutdown 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 msgr2=0x7f5f0c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00ccd0 tx=0x7f5efc0053f0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 shutdown_connections 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5eec038790 0x7f5eec03ac40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f0c09d410 0x7f5f0c137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f0c09cb00 0x7f5f0c1376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 --2- 172.21.15.67:0/2619584100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0c0672c0 0x7f5f0c06dc40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 >> 172.21.15.67:0/2619584100 conn(0x7f5f0c0a2110 msgr2=0x7f5f0c133f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 shutdown_connections 2024-09-15T01:56:41.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.737+0000 7f5f10899640 1 -- 172.21.15.67:0/2619584100 wait complete. 2024-09-15T01:56:41.749 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set ls 2024-09-15T01:56:41.749 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '(compat)' 2024-09-15T01:56:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 10 monclient: monmap: 2024-09-15T01:56:41.824 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:41.823247+0000 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:41.823247+0000 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:41.825 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 1 Processor -- start 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 1 -- start start 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.822+0000 7fc6a095d640 10 monclient: init 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c131e70 0x7fc69c132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc69c098420 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc69c1371a0 0x7fc69c139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c1371a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0a13e0 con 0x7fc69c098420 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69c0a0440 con 0x7fc69c1371a0 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c0a01e0 con 0x7fc69c131e70 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc6a095d640 10 monclient(hunting): authenticate will time out at 1034.292512s 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc699d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc69c098420 0x7fc69c098850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40178/0 (socket says 172.21.15.67:40178) 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc699d9b640 1 -- 172.21.15.67:0/2772075315 learned_addr learned my addr 172.21.15.67:0/2772075315 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:41.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/2772075315 >> v2:172.21.15.67:3300/0 conn(0x7fc69c1371a0 0x7fc69c139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): get_auth_request con 0x7fc69c1371a0 auth_method 0 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69a59c640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c131e70 0x7fc69c132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69a59c640 10 monclient(hunting): get_auth_request con 0x7fc69c131e70 auth_method 0 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_done global_id 5851 payload 274 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 -- 172.21.15.67:0/2772075315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c131e70 msgr2=0x7fc69c132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c131e70 0x7fc69c132240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 -- 172.21.15.67:0/2772075315 >> v1:172.21.15.67:6789/0 conn(0x7fc69c098420 legacy=0x7fc69c098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 monclient: found mon.noname-c 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.823+0000 7fc69ad9d640 1 -- 172.21.15.67:0/2772075315 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0a19b0 con 0x7fc69c1371a0 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69ad9d640 1 -- 172.21.15.67:0/2772075315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 msgr2=0x7fc69c139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c139580 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009e60 tx=0x7fc68c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d4aa15d4826f43b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 1 -- 172.21.15.67:0/2772075315 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c007760 con 0x7fc69c1371a0 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:41.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: dump: 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 1 -- 172.21.15.67:0/2772075315 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c02d050 con 0x7fc69c1371a0 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc6a095d640 5 monclient: authenticate success, global_id 5851 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc6a095d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 1 -- 172.21.15.67:0/2772075315 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c007b30 con 0x7fc69c1371a0 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc6a095d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc6a095d640 4 set_mon_vals no callback set 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.826+0000 7fc69959a640 10 monclient: dump: 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.839 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 10 monclient: shutdown 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 -- 172.21.15.67:0/2772075315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 msgr2=0x7fc69c139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c139580 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009e60 tx=0x7fc68c009780 comp rx=0 tx=0).stop 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 -- 172.21.15.67:0/2772075315 shutdown_connections 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c139580 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 --2- 172.21.15.67:0/2772075315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c131e70 0x7fc69c132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 -- 172.21.15.67:0/2772075315 >> 172.21.15.67:0/2772075315 conn(0x7fc69c0a5c10 msgr2=0x7fc69c0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 -- 172.21.15.67:0/2772075315 shutdown_connections 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.827+0000 7fc6a095d640 1 -- 172.21.15.67:0/2772075315 wait complete. 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient: build_initial_monmap 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient: monmap: 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:41.841 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 Processor -- start 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 -- start start 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient: init 2024-09-15T01:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c098420 0x7fc69c1340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc69c131e70 0x7fc69c1345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-a con 0x7fc69c131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c1371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c11beb0 con 0x7fc69c131e70 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c11c3e0 con 0x7fc69c1371a0 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:41.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c0a0d80 con 0x7fc69c098420 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc6a095d640 10 monclient(hunting): authenticate will time out at 1034.297836s 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59674/0 (socket says 172.21.15.67:59674) 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 1 -- 172.21.15.67:0/1592677205 learned_addr learned my addr 172.21.15.67:0/1592677205 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 10 monclient(hunting): get_auth_request con 0x7fc69c1371a0 auth_method 0 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c098420 0x7fc69c1340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69a59c640 10 monclient(hunting): get_auth_request con 0x7fc69c098420 auth_method 0 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.828+0000 7fc69a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc69c131e70 0x7fc69c1345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): get_auth_request con 0x7fc69c131e70 auth_method 0 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc699d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient(hunting): handle_auth_done global_id 5854 payload 824 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 -- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c098420 msgr2=0x7fc69c1340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c098420 0x7fc69c1340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 -- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc69c131e70 msgr2=0x7fc69c1345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc69c131e70 0x7fc69c1345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 monclient: found mon.noname-b 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:41.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.829+0000 7fc69ad9d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c009a20 con 0x7fc69c1371a0 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc69ad9d640 1 -- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 msgr2=0x7fc69c134b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc69ad9d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc68c007fd0 tx=0x7fc68c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c031c10 con 0x7fc69c1371a0 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: dump: 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.847 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c040070 con 0x7fc69c1371a0 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 5 monclient: authenticate success, global_id 5854 2024-09-15T01:56:41.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 10 monclient: _renew_subs 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c0a4c70 con 0x7fc69c1371a0 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c045430 con 0x7fc69c1371a0 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6927fc640 10 monclient: dump: 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:41.849 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:41.850 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc698d99640 4 set_mon_vals no callback set 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 10 monclient: _renew_subs 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.831+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c130e20 con 0x7fc69c1371a0 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc68c003020 con 0x7fc69c1371a0 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6a095d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6a095d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6a095d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c0639e0 con 0x7fc69c1371a0 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6927fc640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 0x7fc67c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc69a59c640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 0x7fc67c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 7545+0+0 (secure 0 0 0) 0x7fc68c06e270 con 0x7fc69c1371a0 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.832+0000 7fc69a59c640 10 monclient: get_auth_request con 0x7fc67c038790 auth_method 0 2024-09-15T01:56:41.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.835+0000 7fc69a59c640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 0x7fc67c03ac40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc694007ba0 tx=0x7fc694008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:41.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.837+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc68c04e430 con 0x7fc69c1371a0 2024-09-15T01:56:41.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.837+0000 7fc6927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:41.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:41.837+0000 7fc6927fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:42.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.008+0000 7fc6a095d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.008+0000 7fc6a095d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.008+0000 7fc6a095d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.008+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc69c11bff0 con 0x7fc69c1371a0 2024-09-15T01:56:42.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.009+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+9 (secure 0 0 0) 0x7fc68c0393c0 con 0x7fc69c1371a0 2024-09-15T01:56:42.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.009+0000 7fc6927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.009+0000 7fc6927fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6a095d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6a095d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6a095d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc69c11bff0 con 0x7fc69c1371a0 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6927fc640 1 -- 172.21.15.67:0/1592677205 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+9 (secure 0 0 0) 0x7fc68c0395a0 con 0x7fc69c1371a0 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T01:56:42.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.112+0000 7fc6927fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 msgr2=0x7fc67c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 0x7fc67c03ac40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc694007ba0 tx=0x7fc694008040 comp rx=0 tx=0).stop 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 10 monclient: shutdown 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 msgr2=0x7fc69c134b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc68c007fd0 tx=0x7fc68c037000 comp rx=0 tx=0).stop 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 shutdown_connections 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc67c038790 0x7fc67c03ac40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc69c131e70 0x7fc69c1345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc69c1371a0 0x7fc69c134b20 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 --2- 172.21.15.67:0/1592677205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc69c098420 0x7fc69c1340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 >> 172.21.15.67:0/1592677205 conn(0x7fc69c0a5c10 msgr2=0x7fc69c0a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 shutdown_connections 2024-09-15T01:56:42.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.114+0000 7fc6a095d640 1 -- 172.21.15.67:0/1592677205 wait complete. 2024-09-15T01:56:42.126 INFO:tasks.workunit.client.0.smithi067.stdout:(compat) 2024-09-15T01:56:42.126 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T01:56:42.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.199+0000 7fdd58c8b640 10 monclient: get_monmap_and_config 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.199+0000 7fdd58c8b640 10 monclient: build_initial_monmap 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.199+0000 7fdd58c8b640 10 monclient: monmap: 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:42.200627+0000 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:42.200627+0000 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:42.202 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 Processor -- start 2024-09-15T01:56:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 -- start start 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient: init 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd5406e860 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd540979c0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd54097df0 0x7fdd5409c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd54097df0 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5409dce0 con 0x7fdd540979c0 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5406ec30 con 0x7fdd54097df0 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541326a0 con 0x7fdd5406e860 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd58c8b640 10 monclient(hunting): authenticate will time out at 1034.669732s 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd54097df0 0x7fdd5409c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd54097df0 0x7fdd5409c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59676/0 (socket says 172.21.15.67:59676) 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 1 -- 172.21.15.67:0/2599213681 learned_addr learned my addr 172.21.15.67:0/2599213681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): get_auth_request con 0x7fdd54097df0 auth_method 0 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.200+0000 7fdd5259c640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): get_auth_request con 0x7fdd5406e860 auth_method 0 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 10 monclient(hunting): handle_auth_done global_id 5857 payload 274 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 1 -- 172.21.15.67:0/2599213681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 msgr2=0x7fdd54131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54131fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 1 -- 172.21.15.67:0/2599213681 >> v1:172.21.15.17:6789/0 conn(0x7fdd540979c0 legacy=0x7fdd5409bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 1 monclient: found mon.noname-c 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:42.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.201+0000 7fdd52d9d640 1 -- 172.21.15.67:0/2599213681 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5409d620 con 0x7fdd54097df0 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.203+0000 7fdd52d9d640 1 -- 172.21.15.67:0/2599213681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd54097df0 msgr2=0x7fdd5409c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.203+0000 7fdd52d9d640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd54097df0 0x7fdd5409c4a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009f20 tx=0x7fdd4c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c8638f98df4d597 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 1 -- 172.21.15.67:0/2599213681 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c007610 con 0x7fdd54097df0 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: dump: 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: _finish_auth 0 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 1 -- 172.21.15.67:0/2599213681 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c02d050 con 0x7fdd54097df0 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 5 monclient: authenticate success, global_id 5857 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 1 -- 172.21.15.67:0/2599213681 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0079a0 con 0x7fdd54097df0 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 4 set_mon_vals no callback set 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd5159a640 10 monclient: dump: 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.216 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 10 monclient: shutdown 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 -- 172.21.15.67:0/2599213681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd54097df0 msgr2=0x7fdd5409c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd54097df0 0x7fdd5409c4a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009f20 tx=0x7fdd4c009840 comp rx=0 tx=0).stop 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 -- 172.21.15.67:0/2599213681 shutdown_connections 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd54097df0 0x7fdd5409c4a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/2599213681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 -- 172.21.15.67:0/2599213681 >> 172.21.15.67:0/2599213681 conn(0x7fdd540a13a0 msgr2=0x7fdd540a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 -- 172.21.15.67:0/2599213681 shutdown_connections 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.204+0000 7fdd58c8b640 1 -- 172.21.15.67:0/2599213681 wait complete. 2024-09-15T01:56:42.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient: build_initial_monmap 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient: monmap: 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 Processor -- start 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 -- start start 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient: init 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54130e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd5406e860 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd540979c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 0x7fdd541269b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd54097df0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54129ae0 con 0x7fdd54097df0 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54128250 con 0x7fdd540979c0 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54097780 con 0x7fdd5406e860 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.205+0000 7fdd58c8b640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd58c8b640 10 monclient(hunting): authenticate will time out at 1034.675007s 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59680/0 (socket says 172.21.15.67:59680) 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 -- 172.21.15.67:0/3332171956 learned_addr learned my addr 172.21.15.67:0/3332171956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): get_auth_request con 0x7fdd540979c0 auth_method 0 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54130e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): get_auth_request con 0x7fdd5406e860 auth_method 0 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 0x7fdd541269b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 10 monclient(hunting): get_auth_request con 0x7fdd54097df0 auth_method 0 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient(hunting): handle_auth_done global_id 5860 payload 824 2024-09-15T01:56:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 -- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 msgr2=0x7fdd54130e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54130e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 -- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 msgr2=0x7fdd541269b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 0x7fdd541269b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 monclient: found mon.noname-b 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd51d9b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c009ae0 con 0x7fdd540979c0 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.206+0000 7fdd52d9d640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 0x7fdd541269b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd51d9b640 1 -- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 msgr2=0x7fdd54126470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd51d9b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800c610 tx=0x7fdd48007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48012840 con 0x7fdd540979c0 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: dump: 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:42.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd480129e0 con 0x7fdd540979c0 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 5 monclient: authenticate success, global_id 5860 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 10 monclient: _renew_subs 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd541291b0 con 0x7fdd540979c0 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48012da0 con 0x7fdd540979c0 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd427fc640 10 monclient: dump: 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 10 monclient: _renew_subs 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd50d99640 4 set_mon_vals no callback set 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.208+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd54127680 con 0x7fdd540979c0 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdd48011590 con 0x7fdd540979c0 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd58c8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd58c8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd58c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd541293d0 con 0x7fdd540979c0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd427fc640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 0x7fdd3403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.209+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 7545+0+0 (secure 0 0 0) 0x7fdd4804c770 con 0x7fdd540979c0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.210+0000 7fdd5259c640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 0x7fdd3403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.210+0000 7fdd5259c640 10 monclient: get_auth_request con 0x7fdd34038740 auth_method 0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.212+0000 7fdd5259c640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 0x7fdd3403abf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009c80 tx=0x7fdd440090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.214+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdd48010aa0 con 0x7fdd540979c0 2024-09-15T01:56:42.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.214+0000 7fdd427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.214+0000 7fdd427fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:42.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.387+0000 7fdd58c8b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:42.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.387+0000 7fdd58c8b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:42.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.387+0000 7fdd58c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.387+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdd5404bf60 con 0x7fdd540979c0 2024-09-15T01:56:42.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.623+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v137) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdd4801c030 con 0x7fdd540979c0 2024-09-15T01:56:42.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.623+0000 7fdd427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:42.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.623+0000 7fdd427fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:42.678 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:56:42.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:42.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.729+0000 7fdd58c8b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:42.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.729+0000 7fdd58c8b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:42.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.729+0000 7fdd58c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:42.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:42.729+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdd5409d900 con 0x7fdd540979c0 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.786+0000 7fa089cef640 10 monclient: get_monmap_and_config 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.786+0000 7fa089cef640 10 monclient: build_initial_monmap 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.786+0000 7fa089cef640 10 monclient: monmap: 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:42.787145+0000 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:42.787145+0000 2024-09-15T01:56:42.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:42.789 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 Processor -- start 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 -- start start 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient: init 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa084096c10 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa08409d3b0 0x7fa084096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-a con 0x7fa08409d3b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa084097520 0x7fa084135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-c con 0x7fa084097520 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840983b0 con 0x7fa084096c10 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa084099260 con 0x7fa08409d3b0 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa084097920 con 0x7fa084097520 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa089cef640 10 monclient(hunting): authenticate will time out at 1062.783706s 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa08409d3b0 0x7fa084096fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa08409d3b0 0x7fa084096fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37128/0 (socket says 172.21.15.17:37128) 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 1 -- 172.21.15.17:0/2205589282 learned_addr learned my addr 172.21.15.17:0/2205589282 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): get_auth_request con 0x7fa08409d3b0 auth_method 0 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa0837fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa084096c10 0x7fa084099f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38386/0 (socket says 172.21.15.17:38386) 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa083fff640 1 --2- 172.21.15.17:0/2205589282 >> v2:172.21.15.67:3300/0 conn(0x7fa084097520 0x7fa084135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa083fff640 10 monclient(hunting): get_auth_request con 0x7fa084097520 auth_method 0 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa083fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.787+0000 7fa083fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 10 monclient(hunting): handle_auth_done global_id 84174 payload 274 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 1 -- 172.21.15.17:0/2205589282 >> v2:172.21.15.67:3300/0 conn(0x7fa084097520 msgr2=0x7fa084135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 1 --2- 172.21.15.17:0/2205589282 >> v2:172.21.15.67:3300/0 conn(0x7fa084097520 0x7fa084135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 1 -- 172.21.15.17:0/2205589282 >> v1:172.21.15.17:6789/0 conn(0x7fa084096c10 legacy=0x7fa084099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 1 monclient: found mon.noname-a 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:42.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.788+0000 7fa07affd640 1 -- 172.21.15.17:0/2205589282 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa084097a60 con 0x7fa08409d3b0 2024-09-15T01:56:42.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa07affd640 1 -- 172.21.15.17:0/2205589282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 msgr2=0x7fa084096fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:42.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa07affd640 1 --2- 172.21.15.17:0/2205589282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa084096fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa08409a6f0 tx=0x7fa06c00b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fb38ff5ec9f200 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 1 -- 172.21.15.17:0/2205589282 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c032530 con 0x7fa08409d3b0 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: dump: 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.802 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: _finish_auth 0 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 1 -- 172.21.15.17:0/2205589282 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c032690 con 0x7fa08409d3b0 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 5 monclient: authenticate success, global_id 84174 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 1 -- 172.21.15.17:0/2205589282 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c032a30 con 0x7fa08409d3b0 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 4 set_mon_vals no callback set 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:42.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa082ffd640 10 monclient: dump: 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 10 monclient: shutdown 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 -- 172.21.15.17:0/2205589282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 msgr2=0x7fa084096fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 --2- 172.21.15.17:0/2205589282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa084096fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa08409a6f0 tx=0x7fa06c00b460 comp rx=0 tx=0).stop 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 -- 172.21.15.17:0/2205589282 shutdown_connections 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 --2- 172.21.15.17:0/2205589282 >> v2:172.21.15.67:3300/0 conn(0x7fa084097520 0x7fa084135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 --2- 172.21.15.17:0/2205589282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa084096fe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 -- 172.21.15.17:0/2205589282 >> 172.21.15.17:0/2205589282 conn(0x7fa0840a1000 msgr2=0x7fa0840a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 -- 172.21.15.17:0/2205589282 shutdown_connections 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.791+0000 7fa089cef640 1 -- 172.21.15.17:0/2205589282 wait complete. 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.792+0000 7fa089cef640 10 monclient: build_initial_monmap 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.792+0000 7fa089cef640 10 monclient: monmap: 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:42.805 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.792+0000 7fa089cef640 1 Processor -- start 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.792+0000 7fa089cef640 1 -- start start 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient: init 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 0x7fa084141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-b con 0x7fa084096c10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 0x7fa084141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-c con 0x7fa084097520 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): picked mon.noname-a con 0x7fa08409d3b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa084098e50 con 0x7fa08409d3b0 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841219d0 con 0x7fa084096c10 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841216f0 con 0x7fa084097520 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa089cef640 10 monclient(hunting): authenticate will time out at 1062.789457s 2024-09-15T01:56:42.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 0x7fa084141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37132/0 (socket says 172.21.15.17:37132) 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 -- 172.21.15.17:0/3316383397 learned_addr learned my addr 172.21.15.17:0/3316383397 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): get_auth_request con 0x7fa08409d3b0 auth_method 0 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): get_auth_request con 0x7fa084097520 auth_method 0 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 0x7fa084141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 10 monclient(hunting): get_auth_request con 0x7fa084096c10 auth_method 0 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient(hunting): handle_auth_done global_id 84177 payload 824 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 -- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 msgr2=0x7fa084141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 0x7fa084141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 -- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 msgr2=0x7fa084141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 0x7fa084141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 monclient: found mon.noname-a 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa083fff640 1 -- 172.21.15.17:0/3316383397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c00b700 con 0x7fa08409d3b0 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa07affd640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 0x7fa084141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.793+0000 7fa0837fe640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 0x7fa084141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa083fff640 1 -- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 msgr2=0x7fa08411dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa083fff640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009e90 tx=0x7fa07c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c014480 con 0x7fa08409d3b0 2024-09-15T01:56:42.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: dump: 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: _finish_auth 0 2024-09-15T01:56:42.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c00f040 con 0x7fa08409d3b0 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 5 monclient: authenticate success, global_id 84177 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 10 monclient: _renew_subs 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08411f9b0 con 0x7fa08409d3b0 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c014830 con 0x7fa08409d3b0 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa07bfff640 10 monclient: dump: 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:42.812 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 10 monclient: _renew_subs 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0841423e0 con 0x7fa08409d3b0 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.796+0000 7fa0827fc640 4 set_mon_vals no callback set 2024-09-15T01:56:42.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa07c014be0 con 0x7fa08409d3b0 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa089cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa089cef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa089cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0840724a0 con 0x7fa08409d3b0 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa07bfff640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 0x7fa06403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fa07c04d1a0 con 0x7fa08409d3b0 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa0837fe640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 0x7fa06403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:42.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.797+0000 7fa0837fe640 10 monclient: get_auth_request con 0x7fa064038790 auth_method 0 2024-09-15T01:56:42.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.800+0000 7fa0837fe640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 0x7fa06403ac40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa074005130 tx=0x7fa074009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:42.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.802+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa07c0118c0 con 0x7fa08409d3b0 2024-09-15T01:56:42.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.802+0000 7fa07bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:42.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:42.802+0000 7fa07bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:43.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.027+0000 7fa089cef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:43.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.027+0000 7fa089cef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:43.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.027+0000 7fa089cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.027+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa08406f720 con 0x7fa08409d3b0 2024-09-15T01:56:43.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.028+0000 7fa07bfff640 1 -- 172.21.15.17:0/3316383397 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fa07c018080 con 0x7fa08409d3b0 2024-09-15T01:56:43.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.028+0000 7fa07bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:43.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.028+0000 7fa07bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:43.030 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 msgr2=0x7fa06403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 0x7fa06403ac40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa074005130 tx=0x7fa074009040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 10 monclient: shutdown 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 msgr2=0x7fa08411dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.029+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009e90 tx=0x7fa07c00b040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 shutdown_connections 2024-09-15T01:56:43.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa064038790 0x7fa06403ac40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa08409d3b0 0x7fa08411dd90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa084097520 0x7fa084141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 --2- 172.21.15.17:0/3316383397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa084096c10 0x7fa084141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 >> 172.21.15.17:0/3316383397 conn(0x7fa0840a1000 msgr2=0x7fa08409ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 shutdown_connections 2024-09-15T01:56:43.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.030+0000 7fa089cef640 1 -- 172.21.15.17:0/3316383397 wait complete. 2024-09-15T01:56:43.042 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:56:43.042 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:56:43.042 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:56:43.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:56:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.137+0000 7f4713f09640 10 monclient: get_monmap_and_config 2024-09-15T01:56:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.137+0000 7f4713f09640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.137+0000 7f4713f09640 10 monclient: monmap: 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:43.138426+0000 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:43.138426+0000 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:43.140 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 Processor -- start 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 -- start start 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient: init 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 0x7f470c099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-c con 0x7f470c096c10 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f470c09a4d0 0x7f470c096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-a con 0x7f470c09a4d0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f470c097520 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0995b0 con 0x7f470c097520 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c09d190 con 0x7f470c09a4d0 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c0992d0 con 0x7f470c096c10 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.138+0000 7f4713f09640 10 monclient(hunting): authenticate will time out at 1063.134996s 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f470c09a4d0 0x7f470c096fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f470c09a4d0 0x7f470c096fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37138/0 (socket says 172.21.15.17:37138) 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 -- 172.21.15.17:0/3172533399 learned_addr learned my addr 172.21.15.17:0/3172533399 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): get_auth_request con 0x7f470c09a4d0 auth_method 0 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3172533399 >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 0x7f470c099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 10 monclient(hunting): get_auth_request con 0x7f470c096c10 auth_method 0 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient(hunting): handle_auth_done global_id 84180 payload 274 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 -- 172.21.15.17:0/3172533399 >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 msgr2=0x7f470c099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 --2- 172.21.15.17:0/3172533399 >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 0x7f470c099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 -- 172.21.15.17:0/3172533399 >> v1:172.21.15.67:6789/0 conn(0x7f470c097520 legacy=0x7f470c12e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f47114a4640 1 -- 172.21.15.17:0/3172533399 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c097960 con 0x7f470c09a4d0 2024-09-15T01:56:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.139+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3172533399 >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 0x7f470c099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f47114a4640 1 -- 172.21.15.17:0/3172533399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c09a4d0 msgr2=0x7f470c096fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f47114a4640 1 --2- 172.21.15.17:0/3172533399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c09a4d0 0x7f470c096fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f470c0769e0 tx=0x7f46fc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecc44884e1f0062 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 1 -- 172.21.15.17:0/3172533399 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc032440 con 0x7f470c09a4d0 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: dump: 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.152 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 1 -- 172.21.15.17:0/3172533399 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc0325a0 con 0x7f470c09a4d0 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 5 monclient: authenticate success, global_id 84180 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 1 -- 172.21.15.17:0/3172533399 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc032940 con 0x7f470c09a4d0 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 4 set_mon_vals no callback set 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4710ca3640 10 monclient: dump: 2024-09-15T01:56:43.153 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 10 monclient: shutdown 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 -- 172.21.15.17:0/3172533399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c09a4d0 msgr2=0x7f470c096fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 --2- 172.21.15.17:0/3172533399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c09a4d0 0x7f470c096fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f470c0769e0 tx=0x7f46fc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 -- 172.21.15.17:0/3172533399 shutdown_connections 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 --2- 172.21.15.17:0/3172533399 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c09a4d0 0x7f470c096fe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 --2- 172.21.15.17:0/3172533399 >> v2:172.21.15.67:3300/0 conn(0x7f470c096c10 0x7f470c099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 -- 172.21.15.17:0/3172533399 >> 172.21.15.17:0/3172533399 conn(0x7f470c0a1000 msgr2=0x7f470c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.142+0000 7f4713f09640 1 -- 172.21.15.17:0/3172533399 shutdown_connections 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 1 -- 172.21.15.17:0/3172533399 wait complete. 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 10 monclient: monmap: 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 1 Processor -- start 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 1 -- start start 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.143+0000 7f4713f09640 10 monclient: init 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c096c10 0x7f470c12e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-b con 0x7f470c096c10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 0x7f470c119a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-a con 0x7f470c097520 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c09a4d0 0x7f470c119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): picked mon.noname-c con 0x7f470c09a4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c11bce0 con 0x7f470c097520 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c098ec0 con 0x7f470c096c10 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c12e7d0 con 0x7f470c09a4d0 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4713f09640 10 monclient(hunting): authenticate will time out at 1063.140336s 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 0x7f470c119a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 0x7f470c119a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37146/0 (socket says 172.21.15.17:37146) 2024-09-15T01:56:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c09a4d0 0x7f470c119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 -- 172.21.15.17:0/3323068592 learned_addr learned my addr 172.21.15.17:0/3323068592 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): get_auth_request con 0x7f470c097520 auth_method 0 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): get_auth_request con 0x7f470c09a4d0 auth_method 0 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47124a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c096c10 0x7f470c12e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4711ca5640 10 monclient(hunting): get_auth_request con 0x7f470c096c10 auth_method 0 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4711ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4711ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f4711ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient(hunting): handle_auth_done global_id 84183 payload 824 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c09a4d0 msgr2=0x7f470c119f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c09a4d0 0x7f470c119f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c096c10 msgr2=0x7f470c12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c096c10 0x7f470c12e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.144+0000 7f47114a4640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc009450 con 0x7f470c097520 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.146+0000 7f47114a4640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 msgr2=0x7f470c119a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.146+0000 7f47114a4640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 0x7f470c119a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46fc035040 tx=0x7f46fc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc031940 con 0x7f470c097520 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: dump: 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 5 monclient: authenticate success, global_id 84183 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc031aa0 con 0x7f470c097520 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 10 monclient: _renew_subs 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c11d070 con 0x7f470c097520 2024-09-15T01:56:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc031e60 con 0x7f470c097520 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4701ffb640 10 monclient: dump: 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.162 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4703fff640 4 set_mon_vals no callback set 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 10 monclient: _renew_subs 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.147+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c11dcd0 con 0x7f470c097520 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4713f09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4713f09640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 0x7f470c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c12d6f0 con 0x7f470c11c0b0 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f46fc039040 con 0x7f470c097520 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 0x7f470c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient: get_auth_request con 0x7f470c11c0b0 auth_method 0 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4701ffb640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f0038790 0x7f46f003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f46fc06c200 con 0x7f470c097520 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 10 monclient(hunting): handle_auth_done global_id 84186 payload 824 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f4711ca5640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f470c11d530 con 0x7f470c11c0b0 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f47124a6640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f0038790 0x7f46f003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.148+0000 7f47124a6640 10 monclient: get_auth_request con 0x7f46f0038790 auth_method 0 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.150+0000 7f4711ca5640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 msgr2=0x7f470c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.150+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 0x7f470c02aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f470800f5b0 tx=0x7f4708008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f47124a6640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f0038790 0x7f46f003ac40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4704007fd0 tx=0x7f47040025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708014880 con 0x7f470c11c0b0 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 10 monclient: dump: 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f47080149e0 con 0x7f470c11c0b0 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 msgr2=0x7f470c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 0x7f470c02aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f470800f5b0 tx=0x7f4708008040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 msgr2=0x7f470c02aa40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.151+0000 7f4701ffb640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11c0b0 0x7f470c02aa40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4713f09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4713f09640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 0x7f470c05e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4713f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c063c20 con 0x7f470c11a4c0 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 0x7f470c05e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient: get_auth_request con 0x7f470c11a4c0 auth_method 0 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.157+0000 7f4711ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.158+0000 7f4711ca5640 10 monclient(hunting): handle_auth_done global_id 84189 payload 824 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.158+0000 7f4711ca5640 1 -- 172.21.15.17:0/3323068592 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f470c11a8c0 con 0x7f470c11a4c0 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4711ca5640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 msgr2=0x7f470c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4711ca5640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 0x7f470c05e310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f470801a000 tx=0x7f470801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470801ea70 con 0x7f470c11a4c0 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 10 monclient: dump: 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.170 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f4708026070 con 0x7f470c11a4c0 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 msgr2=0x7f470c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 0x7f470c05e310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f470801a000 tx=0x7f470801d000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 msgr2=0x7f470c05e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.160+0000 7f4701ffb640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c11a4c0 0x7f470c05e310 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f470801a000 tx=0x7f470801d000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:43.172 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.173 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:43.175 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.176 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:43.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.180 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.183 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:43.184 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f0038790 msgr2=0x7f46f003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f0038790 0x7f46f003ac40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4704007fd0 tx=0x7f47040025b0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 10 monclient: shutdown 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 msgr2=0x7f470c119a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 --2- 172.21.15.17:0/3323068592 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c097520 0x7f470c119a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46fc035040 tx=0x7f46fc032000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f47124a6640 1 -- 172.21.15.17:0/3323068592 reap_dead start 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f47124a6640 1 -- 172.21.15.17:0/3323068592 reap_dead start 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 shutdown_connections 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 >> 172.21.15.17:0/3323068592 conn(0x7f470c0a1000 msgr2=0x7f470c09fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 shutdown_connections 2024-09-15T01:56:43.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.161+0000 7f4713f09640 1 -- 172.21.15.17:0/3323068592 wait complete. 2024-09-15T01:56:43.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:43.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:56:43.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:56:43.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 10 monclient: monmap: 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:43.282981+0000 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:43.282981+0000 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:43.286 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 1 Processor -- start 2024-09-15T01:56:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 1 -- start start 2024-09-15T01:56:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.282+0000 7f97c879d640 10 monclient: init 2024-09-15T01:56:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 0x7f97c009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-c con 0x7f97c0131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97c00a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f97c009f1e0 0x7f97c0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-a con 0x7f97c009f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00a0040 con 0x7f97c00a0280 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c009bff0 con 0x7f97c009f1e0 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00a14a0 con 0x7f97c0131b20 2024-09-15T01:56:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c879d640 10 monclient(hunting): authenticate will time out at 1063.279430s 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f97c009f1e0 0x7f97c0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c5d38640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f97c00a0280 0x7f97c0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38400/0 (socket says 172.21.15.17:38400) 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): get_auth_request con 0x7f97c009f1e0 auth_method 0 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c5d38640 1 -- 172.21.15.17:0/2354418110 learned_addr learned my addr 172.21.15.17:0/2354418110 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_done global_id 84192 payload 274 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6539640 1 --2- 172.21.15.17:0/2354418110 >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 0x7f97c009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 -- 172.21.15.17:0/2354418110 >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 msgr2=0x7f97c009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 --2- 172.21.15.17:0/2354418110 >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 0x7f97c009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 -- 172.21.15.17:0/2354418110 >> v1:172.21.15.17:6789/0 conn(0x7f97c00a0280 legacy=0x7f97c0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6d3a640 1 -- 172.21.15.17:0/2354418110 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c0132600 con 0x7f97c009f1e0 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6539640 10 monclient: get_auth_request con 0x7f97c0131b20 auth_method 0 2024-09-15T01:56:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.283+0000 7f97c6539640 1 --2- 172.21.15.17:0/2354418110 >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 0x7f97c009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.286+0000 7f97c6d3a640 1 -- 172.21.15.17:0/2354418110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c009f1e0 msgr2=0x7f97c0138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.286+0000 7f97c6d3a640 1 --2- 172.21.15.17:0/2354418110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c009f1e0 0x7f97c0138e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97c009f710 tx=0x7f97b8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7a88a28d59b9b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 1 -- 172.21.15.17:0/2354418110 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8007840 con 0x7f97c009f1e0 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: dump: 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 1 -- 172.21.15.17:0/2354418110 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b802d050 con 0x7f97c009f1e0 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 5 monclient: authenticate success, global_id 84192 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 1 -- 172.21.15.17:0/2354418110 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8007c10 con 0x7f97c009f1e0 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c5537640 10 monclient: dump: 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 4 set_mon_vals no callback set 2024-09-15T01:56:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 10 monclient: shutdown 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 -- 172.21.15.17:0/2354418110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c009f1e0 msgr2=0x7f97c0138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 --2- 172.21.15.17:0/2354418110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c009f1e0 0x7f97c0138e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97c009f710 tx=0x7f97b8009780 comp rx=0 tx=0).stop 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 -- 172.21.15.17:0/2354418110 shutdown_connections 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 --2- 172.21.15.17:0/2354418110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c009f1e0 0x7f97c0138e60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 --2- 172.21.15.17:0/2354418110 >> v2:172.21.15.67:3300/0 conn(0x7f97c0131b20 0x7f97c009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 -- 172.21.15.17:0/2354418110 >> 172.21.15.17:0/2354418110 conn(0x7f97c0071270 msgr2=0x7f97c00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 -- 172.21.15.17:0/2354418110 shutdown_connections 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.287+0000 7f97c879d640 1 -- 172.21.15.17:0/2354418110 wait complete. 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.288+0000 7f97c879d640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.288+0000 7f97c879d640 10 monclient: monmap: 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.288+0000 7f97c879d640 1 Processor -- start 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.288+0000 7f97c879d640 1 -- start start 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.288+0000 7f97c879d640 10 monclient: init 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 0x7f97c012f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-c con 0x7f97c009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c00a0280 0x7f97c0096ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-b con 0x7f97c00a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 0x7f97c0097410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): picked mon.noname-a con 0x7f97c0131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c009b980 con 0x7f97c0131b20 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00a09b0 con 0x7f97c00a0280 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0139060 con 0x7f97c009f1e0 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c879d640 10 monclient(hunting): authenticate will time out at 1063.285248s 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 0x7f97c012f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 0x7f97c0097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 0x7f97c012f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:32822/0 (socket says 172.21.15.17:32822) 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 1 -- 172.21.15.17:0/594235874 learned_addr learned my addr 172.21.15.17:0/594235874 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): get_auth_request con 0x7f97c009f1e0 auth_method 0 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): get_auth_request con 0x7f97c0131b20 auth_method 0 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c5d38640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c00a0280 0x7f97c0096ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c5d38640 10 monclient(hunting): get_auth_request con 0x7f97c00a0280 auth_method 0 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient(hunting): handle_auth_done global_id 84195 payload 824 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 msgr2=0x7f97c012f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 0x7f97c012f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c00a0280 msgr2=0x7f97c0096ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c00a0280 0x7f97c0096ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b8009a20 con 0x7f97c0131b20 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.289+0000 7f97c6539640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c009f1e0 0x7f97c012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.291+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 msgr2=0x7f97c0097410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.291+0000 7f97c6d3a640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 0x7f97c0097410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f97c009ff60 tx=0x7f97b8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8045460 con 0x7f97c0131b20 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: dump: 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 5 monclient: authenticate success, global_id 84195 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b80455e0 con 0x7f97c0131b20 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 monclient: _renew_subs 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c0099480 con 0x7f97c0131b20 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b80459a0 con 0x7f97c0131b20 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 10 monclient: dump: 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 monclient: _renew_subs 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c4d36640 4 set_mon_vals no callback set 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c0097fc0 con 0x7f97c0131b20 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 0x7f97c0140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0063a20 con 0x7f97c0098490 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97b803c420 con 0x7f97c0131b20 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.292+0000 7f97c6539640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 0x7f97c0140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient: get_auth_request con 0x7f97c0098490 auth_method 0 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97b67fc640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97a4038740 0x7f97a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f97b806e220 con 0x7f97c0131b20 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 10 monclient(hunting): handle_auth_done global_id 74405 payload 824 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c6539640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97c0099610 con 0x7f97c0098490 2024-09-15T01:56:43.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c5d38640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97a4038740 0x7f97a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.293+0000 7f97c5d38640 10 monclient: get_auth_request con 0x7f97a4038740 auth_method 0 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97c6539640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 msgr2=0x7f97c0140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97c6539640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 0x7f97c0140d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc013650 con 0x7f97c0098490 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 10 monclient: dump: 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.312 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f97bc00c040 con 0x7f97c0098490 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 msgr2=0x7f97c0140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 0x7f97c0140d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc008040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 msgr2=0x7f97c0140d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97b67fc640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0098490 0x7f97c0140d20 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc008040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.295+0000 7f97c5d38640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97a4038740 0x7f97a403abf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007fd0 tx=0x7f97b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c879d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c879d640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 0x7f97c0141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0141ba0 con 0x7f97c0141260 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 0x7f97c0141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient: get_auth_request con 0x7f97c0141260 auth_method 0 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.301+0000 7f97c6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.302+0000 7f97c6539640 10 monclient(hunting): handle_auth_done global_id 74408 payload 824 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.302+0000 7f97c6539640 1 -- 172.21.15.17:0/594235874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97c00724a0 con 0x7f97c0141260 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97c6539640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 msgr2=0x7f97c0141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97c6539640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 0x7f97c0141660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc023040 con 0x7f97c0141260 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 10 monclient: dump: 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.315 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f97bc006250 con 0x7f97c0141260 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 msgr2=0x7f97c0141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 0x7f97c0141660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc01d000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 msgr2=0x7f97c0141660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.304+0000 7f97b67fc640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f97c0141260 0x7f97c0141660 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00af10 tx=0x7f97bc01d000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.323 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:43.326 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:43.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.329 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:43.329 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97a4038740 msgr2=0x7f97a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f97a4038740 0x7f97a403abf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007fd0 tx=0x7f97b0009040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 10 monclient: shutdown 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 msgr2=0x7f97c0097410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 --2- 172.21.15.17:0/594235874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f97c0131b20 0x7f97c0097410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f97c009ff60 tx=0x7f97b8037000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 reap_dead start 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c6d3a640 1 -- 172.21.15.17:0/594235874 reap_dead start 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 shutdown_connections 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 >> 172.21.15.17:0/594235874 conn(0x7f97c0071270 msgr2=0x7f97c00a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 shutdown_connections 2024-09-15T01:56:43.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.305+0000 7f97c879d640 1 -- 172.21.15.17:0/594235874 wait complete. 2024-09-15T01:56:43.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:43.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:56:43.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 10 monclient: monmap: 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:43.431009+0000 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:43.431009+0000 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:43.433 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 1 Processor -- start 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 1 -- start start 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.430+0000 7f25f391d640 10 monclient: init 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T01:56:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ec09c500 0x7f25ec06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-a con 0x7f25ec09c500 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25ec09c8d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25ec06f280 0x7f25ec09e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-c con 0x7f25ec06f280 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec06f680 con 0x7f25ec09c8d0 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec09ba20 con 0x7f25ec09c500 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec09f6c0 con 0x7f25ec06f280 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f391d640 10 monclient(hunting): authenticate will time out at 1063.427562s 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ec09c500 0x7f25ec06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f25ec09c500 0x7f25ec06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37180/0 (socket says 172.21.15.17:37180) 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 -- 172.21.15.17:0/2909492615 learned_addr learned my addr 172.21.15.17:0/2909492615 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): get_auth_request con 0x7f25ec09c500 auth_method 0 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f1eba640 1 --2- 172.21.15.17:0/2909492615 >> v2:172.21.15.67:3300/0 conn(0x7f25ec06f280 0x7f25ec09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient(hunting): handle_auth_done global_id 84198 payload 274 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 -- 172.21.15.17:0/2909492615 >> v2:172.21.15.67:3300/0 conn(0x7f25ec06f280 msgr2=0x7f25ec09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 --2- 172.21.15.17:0/2909492615 >> v2:172.21.15.67:3300/0 conn(0x7f25ec06f280 0x7f25ec09e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 -- 172.21.15.17:0/2909492615 >> v1:172.21.15.67:6789/0 conn(0x7f25ec09c8d0 legacy=0x7f25ec09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.431+0000 7f25f16b9640 1 -- 172.21.15.17:0/2909492615 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec09b740 con 0x7f25ec09c500 2024-09-15T01:56:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f16b9640 1 -- 172.21.15.17:0/2909492615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c500 msgr2=0x7f25ec06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f16b9640 1 --2- 172.21.15.17:0/2909492615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c500 0x7f25ec06ed40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25ec09aa60 tx=0x7f25e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fa203264caa06f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 1 -- 172.21.15.17:0/2909492615 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8007610 con 0x7f25ec09c500 2024-09-15T01:56:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: dump: 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 1 -- 172.21.15.17:0/2909492615 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e802d050 con 0x7f25ec09c500 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f391d640 5 monclient: authenticate success, global_id 84198 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f391d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f391d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 1 -- 172.21.15.17:0/2909492615 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e80079a0 con 0x7f25ec09c500 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25f391d640 4 set_mon_vals no callback set 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.434+0000 7f25e3fff640 10 monclient: dump: 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 10 monclient: shutdown 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 -- 172.21.15.17:0/2909492615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c500 msgr2=0x7f25ec06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 --2- 172.21.15.17:0/2909492615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c500 0x7f25ec06ed40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25ec09aa60 tx=0x7f25e8009840 comp rx=0 tx=0).stop 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 -- 172.21.15.17:0/2909492615 shutdown_connections 2024-09-15T01:56:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 --2- 172.21.15.17:0/2909492615 >> v2:172.21.15.67:3300/0 conn(0x7f25ec06f280 0x7f25ec09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 --2- 172.21.15.17:0/2909492615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c500 0x7f25ec06ed40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 -- 172.21.15.17:0/2909492615 >> 172.21.15.17:0/2909492615 conn(0x7f25ec0712f0 msgr2=0x7f25ec0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 -- 172.21.15.17:0/2909492615 shutdown_connections 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.435+0000 7f25f391d640 1 -- 172.21.15.17:0/2909492615 wait complete. 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient: monmap: 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 1 Processor -- start 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 1 -- start start 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient: init 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 0x7f25ec13bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-c con 0x7f25ec06f280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec09c500 0x7f25ec13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.436+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-b con 0x7f25ec09c500 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c8d0 0x7f25ec13f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): picked mon.noname-a con 0x7f25ec09c8d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec120960 con 0x7f25ec09c8d0 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec120580 con 0x7f25ec09c500 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec11e870 con 0x7f25ec06f280 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f391d640 10 monclient(hunting): authenticate will time out at 1063.433042s 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 0x7f25ec13bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c8d0 0x7f25ec13f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 0x7f25ec13bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:32840/0 (socket says 172.21.15.17:32840) 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 -- 172.21.15.17:0/3107548279 learned_addr learned my addr 172.21.15.17:0/3107548279 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): get_auth_request con 0x7f25ec06f280 auth_method 0 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): get_auth_request con 0x7f25ec09c8d0 auth_method 0 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f0eb8640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec09c500 0x7f25ec13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f0eb8640 10 monclient(hunting): get_auth_request con 0x7f25ec09c500 auth_method 0 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f0eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f0eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f0eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient(hunting): handle_auth_done global_id 74411 payload 824 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec09c500 msgr2=0x7f25ec13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec09c500 0x7f25ec13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c8d0 msgr2=0x7f25ec13f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c8d0 0x7f25ec13f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 monclient: found mon.noname-c 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f16b9640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e8009ae0 con 0x7f25ec06f280 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.437+0000 7f25f1eba640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25ec09c8d0 0x7f25ec13f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f16b9640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 msgr2=0x7f25ec13bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f16b9640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 0x7f25ec13bb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009ab0 tx=0x7f25e8036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8032a30 con 0x7f25ec06f280 2024-09-15T01:56:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: dump: 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8032b90 con 0x7f25ec06f280 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e803b410 con 0x7f25ec06f280 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 5 monclient: authenticate success, global_id 74411 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 10 monclient: _renew_subs 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec11e010 con 0x7f25ec06f280 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e17fa640 10 monclient: dump: 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25e2ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 10 monclient: _renew_subs 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.439+0000 7f25f391d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec13c560 con 0x7f25ec06f280 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f25e8038050 con 0x7f25ec06f280 2024-09-15T01:56:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25f391d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25f391d640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 0x7f25ec13f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec120440 con 0x7f25ec11f3d0 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25e17fa640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25d0038790 0x7f25d003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.440+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f25e806cfa0 con 0x7f25ec06f280 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 0x7f25ec13f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f1eba640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25d0038790 0x7f25d003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient: get_auth_request con 0x7f25ec11f3d0 auth_method 0 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f1eba640 10 monclient: get_auth_request con 0x7f25d0038790 auth_method 0 2024-09-15T01:56:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_done global_id 5881 payload 824 2024-09-15T01:56:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.441+0000 7f25f0eb8640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25ec120c40 con 0x7f25ec11f3d0 2024-09-15T01:56:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.443+0000 7f25f1eba640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25d0038790 0x7f25d003ac40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25ec120910 tx=0x7f25e4006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.443+0000 7f25f0eb8640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 msgr2=0x7f25ec13f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.443+0000 7f25f0eb8640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 0x7f25ec13f600 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc007450 con 0x7f25ec11f3d0 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 10 monclient: dump: 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.461 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f25dc0075b0 con 0x7f25ec11f3d0 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 msgr2=0x7f25ec13f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 0x7f25ec13f600 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc009040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 msgr2=0x7f25ec13f600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.444+0000 7f25e17fa640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec11f3d0 0x7f25ec13f600 secure :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc009040 comp rx=0 tx=0).stop 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f391d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f391d640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 0x7f25ec1195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f391d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec063b90 con 0x7f25ec13fb40 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 0x7f25ec1195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient: get_auth_request con 0x7f25ec13fb40 auth_method 0 2024-09-15T01:56:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 10 monclient(hunting): handle_auth_done global_id 5884 payload 824 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.450+0000 7f25f0eb8640 1 -- 172.21.15.17:0/3107548279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f25ec13ff40 con 0x7f25ec13fb40 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25f0eb8640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 msgr2=0x7f25ec1195d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25f0eb8640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 0x7f25ec1195d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc013c40 con 0x7f25ec13fb40 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 10 monclient: dump: 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f25dc013da0 con 0x7f25ec13fb40 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 msgr2=0x7f25ec1195d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 0x7f25ec1195d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc01b000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 msgr2=0x7f25ec1195d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.453+0000 7f25e17fa640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25ec13fb40 0x7f25ec1195d0 secure :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002ee0 tx=0x7f25dc01b000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:56:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 70, 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:43.478 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:43.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25d0038790 msgr2=0x7f25d003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f25d0038790 0x7f25d003ac40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25ec120910 tx=0x7f25e4006f90 comp rx=0 tx=0).stop 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 10 monclient: shutdown 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 msgr2=0x7f25ec13bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 --2- 172.21.15.17:0/3107548279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f25ec06f280 0x7f25ec13bb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009ab0 tx=0x7f25e8036000 comp rx=0 tx=0).stop 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f1eba640 1 -- 172.21.15.17:0/3107548279 reap_dead start 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f1eba640 1 -- 172.21.15.17:0/3107548279 reap_dead start 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 shutdown_connections 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 >> 172.21.15.17:0/3107548279 conn(0x7f25ec0712f0 msgr2=0x7f25ec0a1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 shutdown_connections 2024-09-15T01:56:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.454+0000 7f25f391d640 1 -- 172.21.15.17:0/3107548279 wait complete. 2024-09-15T01:56:43.481 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:56:43.481 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T01:56:43.481 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:56:43.481 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T01:56:43.481 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:56:43.481 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T01:56:43.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:56:43.492 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T01:56:43.492 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:56:43.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:43.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 10 monclient: get_monmap_and_config 2024-09-15T01:56:43.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 10 monclient: monmap: 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:43.626955+0000 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:43.626955+0000 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 1 Processor -- start 2024-09-15T01:56:43.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 1 -- start start 2024-09-15T01:56:43.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.626+0000 7fb40840f640 10 monclient: init 2024-09-15T01:56:43.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb400131b20 0x7fb40009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-c con 0x7fb400131b20 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4000a0280 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb40009f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4000a0040 con 0x7fb4000a0280 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40009bff0 con 0x7fb40009f1e0 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4000a14a0 con 0x7fb400131b20 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb40840f640 10 monclient(hunting): authenticate will time out at 1063.623464s 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4059aa640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb4000a0280 0x7fb400131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38416/0 (socket says 172.21.15.17:38416) 2024-09-15T01:56:43.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4059aa640 1 -- 172.21.15.17:0/3523694009 learned_addr learned my addr 172.21.15.17:0/3523694009 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 1 --2- 172.21.15.17:0/3523694009 >> v2:172.21.15.67:3300/0 conn(0x7fb400131b20 0x7fb40009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): get_auth_request con 0x7fb400131b20 auth_method 0 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.627+0000 7fb4061ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 10 monclient(hunting): handle_auth_done global_id 5887 payload 274 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 1 -- 172.21.15.17:0/3523694009 >> v1:172.21.15.67:6789/0 conn(0x7fb40009f1e0 legacy=0x7fb400138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 1 -- 172.21.15.17:0/3523694009 >> v1:172.21.15.17:6789/0 conn(0x7fb4000a0280 legacy=0x7fb400131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 1 monclient: found mon.noname-c 2024-09-15T01:56:43.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:43.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.628+0000 7fb4061ab640 1 -- 172.21.15.17:0/3523694009 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400132600 con 0x7fb400131b20 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.630+0000 7fb4061ab640 1 -- 172.21.15.17:0/3523694009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 msgr2=0x7fb40009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.630+0000 7fb4061ab640 1 --2- 172.21.15.17:0/3523694009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb40009eca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009850 tx=0x7fb3fc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94ec2e90e81c0d37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 1 -- 172.21.15.17:0/3523694009 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc032440 con 0x7fb400131b20 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: dump: 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 1 -- 172.21.15.17:0/3523694009 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc0325a0 con 0x7fb400131b20 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 1 -- 172.21.15.17:0/3523694009 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc032940 con 0x7fb400131b20 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 5 monclient: authenticate success, global_id 5887 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 4 set_mon_vals no callback set 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb4051a9640 10 monclient: dump: 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 10 monclient: shutdown 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 -- 172.21.15.17:0/3523694009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 msgr2=0x7fb40009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 --2- 172.21.15.17:0/3523694009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb40009eca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009850 tx=0x7fb3fc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 -- 172.21.15.17:0/3523694009 shutdown_connections 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 --2- 172.21.15.17:0/3523694009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb40009eca0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 -- 172.21.15.17:0/3523694009 >> 172.21.15.17:0/3523694009 conn(0x7fb400071270 msgr2=0x7fb4000a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 -- 172.21.15.17:0/3523694009 shutdown_connections 2024-09-15T01:56:43.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.631+0000 7fb40840f640 1 -- 172.21.15.17:0/3523694009 wait complete. 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.632+0000 7fb40840f640 10 monclient: build_initial_monmap 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.632+0000 7fb40840f640 10 monclient: monmap: 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.632+0000 7fb40840f640 1 Processor -- start 2024-09-15T01:56:43.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.632+0000 7fb40840f640 1 -- start start 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.632+0000 7fb40840f640 10 monclient: init 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 0x7fb40009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-c con 0x7fb40009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-a con 0x7fb4000a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb400096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): picked mon.noname-b con 0x7fb400131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40009b960 con 0x7fb4000a0280 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000982b0 con 0x7fb400131b20 2024-09-15T01:56:43.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40009a5d0 con 0x7fb40009f1e0 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb40840f640 10 monclient(hunting): authenticate will time out at 1063.629144s 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4061ab640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 msgr2=0x7fb40009aee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4061ab640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 0x7fb40009aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37210/0 (socket says 172.21.15.17:37210) 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 -- 172.21.15.17:0/1927298201 learned_addr learned my addr 172.21.15.17:0/1927298201 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): get_auth_request con 0x7fb4000a0280 auth_method 0 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:43.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient(hunting): handle_auth_done global_id 84204 payload 824 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4069ac640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb400096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 -- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 msgr2=0x7fb40009aee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 0x7fb40009aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 -- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 msgr2=0x7fb400096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb400096dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 monclient: found mon.noname-a 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.633+0000 7fb4059aa640 1 -- 172.21.15.17:0/1927298201 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc009390 con 0x7fb4000a0280 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.635+0000 7fb4059aa640 1 -- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 msgr2=0x7fb40009b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.635+0000 7fb4059aa640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c5e0 tx=0x7fb3f0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.635+0000 7fb4059aa640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c5e0 tx=0x7fb3f0004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb3f0005cb0 mon_map magic: 0 v1 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0005cb0 con 0x7fb4000a0280 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 10 monclient: dump: 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.646 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f0005e30 con 0x7fb4000a0280 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 5 monclient: authenticate success, global_id 84204 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb3f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 10 monclient: _renew_subs 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb400099460 con 0x7fb4000a0280 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb4049a8640 4 set_mon_vals no callback set 2024-09-15T01:56:43.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 10 monclient: _renew_subs 2024-09-15T01:56:43.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.636+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb400097310 con 0x7fb4000a0280 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb40840f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb40840f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb40840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000724a0 con 0x7fb4000a0280 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0007340 con 0x7fb4000a0280 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 10 monclient: dump: 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb3f001b400 con 0x7fb4000a0280 2024-09-15T01:56:43.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb3f67fc640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 0x7fb3e403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:43.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb4061ab640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 0x7fb3e403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:43.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.637+0000 7fb4061ab640 10 monclient: get_auth_request con 0x7fb3e4038740 auth_method 0 2024-09-15T01:56:43.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.640+0000 7fb4061ab640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 0x7fb3e403abf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc031000 tx=0x7fb3fc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:43.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:43.641+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fb3f004c1c0 con 0x7fb4000a0280 2024-09-15T01:56:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.258+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb3f001b6b0 con 0x7fb4000a0280 2024-09-15T01:56:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.258+0000 7fb3f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.258+0000 7fb3f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.490+0000 7fb40840f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.490+0000 7fb40840f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.490+0000 7fb40840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.490+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb40009a350 con 0x7fb4000a0280 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.491+0000 7fb3f67fc640 1 -- 172.21.15.17:0/1927298201 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7fb3f001a360 con 0x7fb4000a0280 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.491+0000 7fb3f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.491+0000 7fb3f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:44.494 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 msgr2=0x7fb3e403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 0x7fb3e403abf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc031000 tx=0x7fb3fc032000 comp rx=0 tx=0).stop 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 10 monclient: shutdown 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 msgr2=0x7fb40009b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:44.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c5e0 tx=0x7fb3f0004960 comp rx=0 tx=0).stop 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 shutdown_connections 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3e4038740 0x7fb3e403abf0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4000a0280 0x7fb40009b420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb400131b20 0x7fb400096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 --2- 172.21.15.17:0/1927298201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb40009f1e0 0x7fb40009aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 >> 172.21.15.17:0/1927298201 conn(0x7fb400071270 msgr2=0x7fb400129c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 shutdown_connections 2024-09-15T01:56:44.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:44.493+0000 7fb40840f640 1 -- 172.21.15.17:0/1927298201 wait complete. 2024-09-15T01:56:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:45.206+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:45.206+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:45.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:45.206+0000 7fdd417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:46.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:46.206+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:46.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:46.206+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:47.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:47.206+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:47.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:47.206+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:47.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 10 monclient: get_monmap_and_config 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 10 monclient: build_initial_monmap 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 10 monclient: monmap: 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:47.613087+0000 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:47.613087+0000 2024-09-15T01:56:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:47.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 1 Processor -- start 2024-09-15T01:56:47.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 1 -- start start 2024-09-15T01:56:47.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.612+0000 7f20208af640 10 monclient: init 2024-09-15T01:56:47.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f201c0a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f201c132040 0x7f201c0987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): picked mon.noname-a con 0x7f201c132040 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 0x7f201c1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): picked mon.noname-b con 0x7f201c136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201c0a01e0 con 0x7f201c0a58b0 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f201ad9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 msgr2=0x7f201c1391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201c0a1580 con 0x7f201c132040 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c0a5c80 con 0x7f201c136dd0 2024-09-15T01:56:47.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f201ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 0x7f201c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f20208af640 10 monclient(hunting): authenticate will time out at 1067.609770s 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f201a59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f201c0a58b0 0x7f201c0980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38420/0 (socket says 172.21.15.17:38420) 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f201a59c640 1 -- 172.21.15.17:0/2730617079 learned_addr learned my addr 172.21.15.17:0/2730617079 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 1 --2- 172.21.15.17:0/2730617079 >> v2:172.21.15.17:3300/0 conn(0x7f201c132040 0x7f201c0987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): get_auth_request con 0x7f201c132040 auth_method 0 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.613+0000 7f2019d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:47.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 10 monclient(hunting): handle_auth_done global_id 84207 payload 274 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 1 -- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 msgr2=0x7f201c1391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 1 --2- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 0x7f201c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 1 -- 172.21.15.17:0/2730617079 >> v1:172.21.15.17:6789/0 conn(0x7f201c0a58b0 legacy=0x7f201c0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 1 monclient: found mon.noname-a 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.614+0000 7f2019d9b640 1 -- 172.21.15.17:0/2730617079 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c132440 con 0x7f201c132040 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.616+0000 7f2019d9b640 1 -- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c132040 msgr2=0x7f201c0987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f2019d9b640 1 --2- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c132040 0x7f201c0987b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2010009850 tx=0x7f20100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282d6bacdb3b11df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 1 -- 172.21.15.17:0/2730617079 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010032440 con 0x7f201c132040 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: dump: 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:47.625 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: _finish_auth 0 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 1 -- 172.21.15.17:0/2730617079 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20100325a0 con 0x7f201c132040 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 5 monclient: authenticate success, global_id 84207 2024-09-15T01:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 1 -- 172.21.15.17:0/2730617079 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010032940 con 0x7f201c132040 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 4 set_mon_vals no callback set 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f201959a640 10 monclient: dump: 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:47.627 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 10 monclient: shutdown 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 -- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c132040 msgr2=0x7f201c0987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 --2- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c132040 0x7f201c0987b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2010009850 tx=0x7f20100090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 -- 172.21.15.17:0/2730617079 shutdown_connections 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 --2- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c136dd0 0x7f201c1391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 --2- 172.21.15.17:0/2730617079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c132040 0x7f201c0987b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 -- 172.21.15.17:0/2730617079 >> 172.21.15.17:0/2730617079 conn(0x7f201c070ed0 msgr2=0x7f201c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 -- 172.21.15.17:0/2730617079 shutdown_connections 2024-09-15T01:56:47.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.617+0000 7f20208af640 1 -- 172.21.15.17:0/2730617079 wait complete. 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.618+0000 7f20208af640 10 monclient: build_initial_monmap 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.618+0000 7f20208af640 10 monclient: monmap: 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.618+0000 7f20208af640 1 Processor -- start 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.618+0000 7f20208af640 1 -- start start 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.618+0000 7f20208af640 10 monclient: init 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:56:47.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): picked mon.noname-a con 0x7f201c0a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 0x7f201c133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): picked mon.noname-c con 0x7f201c132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 0x7f201c1337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): picked mon.noname-b con 0x7f201c136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c098dc0 con 0x7f201c0a58b0 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c0a3410 con 0x7f201c136dd0 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c135e90 con 0x7f201c132040 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:47.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f20208af640 10 monclient(hunting): authenticate will time out at 1067.615393s 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f2019d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 msgr2=0x7f201c133290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f2019d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 0x7f201c133290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37236/0 (socket says 172.21.15.17:37236) 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 -- 172.21.15.17:0/409269562 learned_addr learned my addr 172.21.15.17:0/409269562 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): get_auth_request con 0x7f201c0a58b0 auth_method 0 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:47.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient(hunting): handle_auth_done global_id 84210 payload 824 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201ad9d640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 0x7f201c1337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 -- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 msgr2=0x7f201c133290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 0x7f201c133290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 -- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 msgr2=0x7f201c1337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 0x7f201c1337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 monclient: found mon.noname-a 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201a59c640 1 -- 172.21.15.17:0/409269562 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010009390 con 0x7f201c0a58b0 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201ad9d640 10 monclient: get_auth_request con 0x7f201c136dd0 auth_method 0 2024-09-15T01:56:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.619+0000 7f201ad9d640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 0x7f201c1337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f201a59c640 1 -- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 msgr2=0x7f201c132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f201a59c640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f200c00b540 tx=0x7f200c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c011840 con 0x7f201c0a58b0 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: dump: 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.637 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c0119c0 con 0x7f201c0a58b0 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c011d80 con 0x7f201c0a58b0 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f200a7fc640 10 monclient: dump: 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:47.638 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 5 monclient: authenticate success, global_id 84210 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 10 monclient: _renew_subs 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201c1363c0 con 0x7f201c0a58b0 2024-09-15T01:56:47.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f2018d99640 4 set_mon_vals no callback set 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 10 monclient: _renew_subs 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.622+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201c133d10 con 0x7f201c0a58b0 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f200c010590 con 0x7f201c0a58b0 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f20208af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f20208af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f20208af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c134880 con 0x7f201c0a58b0 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f200a7fc640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 0x7f1ffc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:47.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f200c04bdd0 con 0x7f201c0a58b0 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f2019d9b640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 0x7f1ffc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.623+0000 7f2019d9b640 10 monclient: get_auth_request con 0x7f1ffc038740 auth_method 0 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.626+0000 7f2019d9b640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 0x7f1ffc03abf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f201c0a60e0 tx=0x7f2010032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.628+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f200c00d890 con 0x7f201c0a58b0 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.628+0000 7f200a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:47.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.628+0000 7f200a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:47.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.853+0000 7f20208af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.853+0000 7f20208af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.853+0000 7f20208af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.853+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f201c0724a0 con 0x7f201c0a58b0 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.853+0000 7f200a7fc640 1 -- 172.21.15.17:0/409269562 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f200c015020 con 0x7f201c0a58b0 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.854+0000 7f200a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.854+0000 7f200a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:47.856 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 msgr2=0x7f1ffc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 0x7f1ffc03abf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f201c0a60e0 tx=0x7f2010032000 comp rx=0 tx=0).stop 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 10 monclient: shutdown 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 msgr2=0x7f201c132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f200c00b540 tx=0x7f200c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 shutdown_connections 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ffc038740 0x7f1ffc03abf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201c136dd0 0x7f201c1337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f201c132040 0x7f201c133290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 --2- 172.21.15.17:0/409269562 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f201c0a58b0 0x7f201c132d50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:47.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 >> 172.21.15.17:0/409269562 conn(0x7f201c070ed0 msgr2=0x7f201c096e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:47.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 shutdown_connections 2024-09-15T01:56:47.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:47.856+0000 7f20208af640 1 -- 172.21.15.17:0/409269562 wait complete. 2024-09-15T01:56:48.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:48.206+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:48.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:48.206+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:49.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:49.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:49.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:49.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:50.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:50.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:50.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:50.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:50.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:50.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.975+0000 7f834ad5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.975+0000 7f834ad5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.975+0000 7f834ad5d640 10 monclient: monmap: 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:50.976688+0000 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:50.976688+0000 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:50.978 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:50.979 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:50.979 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:50.979 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.976+0000 7f834ad5d640 1 Processor -- start 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.976+0000 7f834ad5d640 1 -- start start 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.976+0000 7f834ad5d640 10 monclient: init 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f834406eb30 0x7f834406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-c con 0x7f834406eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f834406f440 0x7f834409bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-a con 0x7f834406f440 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:50.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f834409c4d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834409f6a0 con 0x7f834409c4d0 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8344069f40 con 0x7f834406f440 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8344069950 con 0x7f834406eb30 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f834ad5d640 10 monclient(hunting): authenticate will time out at 1070.973382s 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f834406f440 0x7f834409bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f83492fa640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f834409c4d0 0x7f834409d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38428/0 (socket says 172.21.15.17:38428) 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f83492fa640 1 -- 172.21.15.17:0/512799572 learned_addr learned my addr 172.21.15.17:0/512799572 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): get_auth_request con 0x7f834406f440 auth_method 0 2024-09-15T01:56:50.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient(hunting): handle_auth_done global_id 84213 payload 274 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 -- 172.21.15.17:0/512799572 >> v2:172.21.15.67:3300/0 conn(0x7f834406eb30 msgr2=0x7f834406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 --2- 172.21.15.17:0/512799572 >> v2:172.21.15.67:3300/0 conn(0x7f834406eb30 0x7f834406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 -- 172.21.15.17:0/512799572 >> v1:172.21.15.17:6789/0 conn(0x7f834409c4d0 legacy=0x7f834409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 monclient: found mon.noname-a 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:50.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.977+0000 7f8343fff640 1 -- 172.21.15.17:0/512799572 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834409ad80 con 0x7f834406f440 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.980+0000 7f8343fff640 1 -- 172.21.15.17:0/512799572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834406f440 msgr2=0x7f834409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.980+0000 7f8343fff640 1 --2- 172.21.15.17:0/512799572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834406f440 0x7f834409bf90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83440a3e30 tx=0x7f832c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8763f153479bba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 1 -- 172.21.15.17:0/512799572 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c007840 con 0x7f834406f440 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: dump: 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:50.989 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 1 -- 172.21.15.17:0/512799572 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c02d050 con 0x7f834406f440 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 5 monclient: authenticate success, global_id 84213 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:50.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 1 -- 172.21.15.17:0/512799572 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c007c10 con 0x7f834406f440 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f83437fe640 10 monclient: dump: 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:50.991 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 4 set_mon_vals no callback set 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 10 monclient: shutdown 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 -- 172.21.15.17:0/512799572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834406f440 msgr2=0x7f834409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 --2- 172.21.15.17:0/512799572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834406f440 0x7f834409bf90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83440a3e30 tx=0x7f832c009780 comp rx=0 tx=0).stop 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 -- 172.21.15.17:0/512799572 shutdown_connections 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 --2- 172.21.15.17:0/512799572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834406f440 0x7f834409bf90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 --2- 172.21.15.17:0/512799572 >> v2:172.21.15.67:3300/0 conn(0x7f834406eb30 0x7f834406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 -- 172.21.15.17:0/512799572 >> 172.21.15.17:0/512799572 conn(0x7f83440712f0 msgr2=0x7f83440a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 -- 172.21.15.17:0/512799572 shutdown_connections 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.981+0000 7f834ad5d640 1 -- 172.21.15.17:0/512799572 wait complete. 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.982+0000 7f834ad5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.982+0000 7f834ad5d640 10 monclient: monmap: 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:50.993 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.982+0000 7f834ad5d640 1 Processor -- start 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.982+0000 7f834ad5d640 1 -- start start 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.982+0000 7f834ad5d640 10 monclient: init 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 0x7f83441195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-b con 0x7f834406eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 0x7f8344119b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-c con 0x7f834406f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): picked mon.noname-a con 0x7f834409c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834409cd10 con 0x7f834409c4d0 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834411ae70 con 0x7f834406eb30 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8344134b30 con 0x7f834406f440 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f834ad5d640 10 monclient(hunting): authenticate will time out at 1070.979246s 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f8343fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 msgr2=0x7f8344119b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:50.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f8343fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 0x7f8344119b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37244/0 (socket says 172.21.15.17:37244) 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 -- 172.21.15.17:0/2146766296 learned_addr learned my addr 172.21.15.17:0/2146766296 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): get_auth_request con 0x7f834409c4d0 auth_method 0 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f8348af9640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 0x7f83441195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient(hunting): handle_auth_done global_id 84216 payload 824 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 -- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 msgr2=0x7f8344119b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 0x7f8344119b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 -- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 msgr2=0x7f83441195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 0x7f83441195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient: _finish_hunting 0 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 monclient: found mon.noname-a 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f83492fa640 1 -- 172.21.15.17:0/2146766296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009a20 con 0x7f834409c4d0 2024-09-15T01:56:50.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f8348af9640 10 monclient: get_auth_request con 0x7f834406eb30 auth_method 0 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.983+0000 7f8348af9640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 0x7f83441195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.985+0000 7f83492fa640 1 -- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 msgr2=0x7f834411a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.985+0000 7f83492fa640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833c00c510 tx=0x7f833c004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.985+0000 7f83492fa640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833c00c510 tx=0x7f833c004900 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f833c005cb0 mon_map magic: 0 v1 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c005cb0 con 0x7f834409c4d0 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: dump: 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:50.998 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: _finish_auth 0 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c005e30 con 0x7f834409c4d0 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 5 monclient: authenticate success, global_id 84216 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 10 monclient: _renew_subs 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834411c910 con 0x7f834409c4d0 2024-09-15T01:56:50.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8342ffd640 4 set_mon_vals no callback set 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 10 monclient: _renew_subs 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834411ba20 con 0x7f834409c4d0 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007320 con 0x7f834409c4d0 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 10 monclient: dump: 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:51.000 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.986+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f833c01b3b0 con 0x7f834409c4d0 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f8340ff9640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 0x7f832403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f834ad5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f834ad5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f834ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83440724a0 con 0x7f834409c4d0 2024-09-15T01:56:51.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f8348af9640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 0x7f832403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:51.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.987+0000 7f8348af9640 10 monclient: get_auth_request con 0x7f8324038740 auth_method 0 2024-09-15T01:56:51.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.990+0000 7f8348af9640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 0x7f832403abf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8334006fd0 tx=0x7f8334008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:51.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.990+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:56:51.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:50.992+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f833c04bac0 con 0x7f834409c4d0 2024-09-15T01:56:51.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:51.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:51.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:51.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.609+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f833c017020 con 0x7f834409c4d0 2024-09-15T01:56:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.609+0000 7f8340ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.609+0000 7f8340ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.840+0000 7f834ad5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.840+0000 7f834ad5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.840+0000 7f834ad5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.840+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f834411cdf0 con 0x7f834409c4d0 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.841+0000 7f8340ff9640 1 -- 172.21.15.17:0/2146766296 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f833c018650 con 0x7f834409c4d0 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.841+0000 7f8340ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:51.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.841+0000 7f8340ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:51.844 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 msgr2=0x7f832403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 0x7f832403abf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8334006fd0 tx=0x7f8334008040 comp rx=0 tx=0).stop 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 10 monclient: shutdown 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 msgr2=0x7f834411a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.842+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833c00c510 tx=0x7f833c004900 comp rx=0 tx=0).stop 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 shutdown_connections 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8324038740 0x7f832403abf0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f834409c4d0 0x7f834411a050 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:51.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f834406f440 0x7f8344119b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:51.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 --2- 172.21.15.17:0/2146766296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406eb30 0x7f83441195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:51.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 >> 172.21.15.17:0/2146766296 conn(0x7f83440712f0 msgr2=0x7f83440a1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:51.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 shutdown_connections 2024-09-15T01:56:51.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:51.843+0000 7f834ad5d640 1 -- 172.21.15.17:0/2146766296 wait complete. 2024-09-15T01:56:52.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:52.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:52.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:52.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:53.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:53.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:54.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:54.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:54.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:54.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:54.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:56:54.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.962+0000 7f0c1ff9d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:54.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.962+0000 7f0c1ff9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:54.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.962+0000 7f0c1ff9d640 10 monclient: monmap: 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:54.963448+0000 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:54.963448+0000 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:54.965 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 1 Processor -- start 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 1 -- start start 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 10 monclient: init 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c180a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.963+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c1809ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c18069c90 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c180a5b50 con 0x7f0c1809ccf0 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c180a18b0 con 0x7f0c18069c90 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1809c5f0 con 0x7f0c180a5780 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1dd39640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c1809bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1ff9d640 10 monclient(hunting): authenticate will time out at 1074.960069s 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1dd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1d538640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f0c1809ccf0 0x7f0c18069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42090/0 (socket says 172.21.15.17:42090) 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 learned_addr learned my addr 172.21.15.17:0/2469608650 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request con 0x7f0c18069c90 auth_method 0 2024-09-15T01:56:54.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.964+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:54.966+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 >> v1:172.21.15.17:6789/0 conn(0x7f0c1809ccf0 legacy=0x7f0c18069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:55.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1dd39640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c1809bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:55.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:55.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request con 0x7f0c18069c90 auth_method 0 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:55.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.165+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:56:55.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:55.208+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:55.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:55.208+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:55.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:55.208+0000 7fdd417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:55.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1dd39640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c1809bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:55.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:55.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request con 0x7f0c18069c90 auth_method 0 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:55.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:55.566+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:56:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:56.207+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:56.207+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:56.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1dd39640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c1809bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:56.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request con 0x7f0c18069c90 auth_method 0 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1e53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1e53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.367+0000 7f0c1e53a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.368+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.368+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.368+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:56:56.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:56.368+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:56:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:57.208+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:57.208+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562917181 0 0) 0x7f0c180a5b50 con 0x7f0c1809ccf0 2024-09-15T01:56:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): my global_id is 84219 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c040034e0 con 0x7f0c1809ccf0 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201927438 0 0) 0x7f0c040034e0 con 0x7f0c1809ccf0 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c1809bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c1809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 msgr2=0x7f0c18139880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 --2- 172.21.15.17:0/2469608650 >> v2:172.21.15.67:3300/0 conn(0x7f0c18069c90 0x7f0c18139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: _finish_hunting 0 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:56:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1809d210 con 0x7f0c1809ccf0 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: _finish_auth 0 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f0c08001720 con 0x7f0c1809ccf0 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1ff9d640 5 monclient: authenticate success, global_id 84219 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1ff9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: dump: 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 >> v1:172.21.15.17:6789/0 conn(0x7f0c1809ccf0 legacy=0x7f0c18069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c040039d0 0x7f0c04003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): picked mon.a con 0x7f0c040039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c04004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): picked mon.b con 0x7f0c04004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): picked mon.c con 0x7f0c180a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c180a18b0 con 0x7f0c040039d0 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1809c5f0 con 0x7f0c04004780 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0400bc10 con 0x7f0c180a5780 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1cd37640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:56:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1ff9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c04005200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1e53a640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c04005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.632+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c04004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): get_auth_request con 0x7f0c04004780 auth_method 0 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient(hunting): handle_auth_done global_id 84219 payload 293 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 msgr2=0x7f0c04005200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c180a5780 0x7f0c04005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c040039d0 msgr2=0x7f0c04003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c040039d0 0x7f0c04003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient: _finish_hunting 0 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 monclient: found mon.b 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040034e0 con 0x7f0c04004780 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 10 monclient: _finish_auth 0 2024-09-15T01:56:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.633+0000 7f0c1d538640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.636+0000 7f0c1d538640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 msgr2=0x7f0c04004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.636+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c04004bf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0c04004700 tx=0x7f0c080079f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9118861d7df11e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08005890 con 0x7f0c04004780 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: dump: 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c080059f0 con 0x7f0c04004780 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1ff9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1ff9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1ff9d640 4 set_mon_vals no callback set 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 1 -- 172.21.15.17:0/2469608650 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08005db0 con 0x7f0c04004780 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:57.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.638+0000 7f0c1cd37640 10 monclient: dump: 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 10 monclient: shutdown 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:57.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 msgr2=0x7f0c04004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2469608650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c04004bf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0c04004700 tx=0x7f0c080079f0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1e53a640 1 -- 172.21.15.17:0/2469608650 reap_dead start 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2469608650 shutdown_connections 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2469608650 >> 172.21.15.17:0/2469608650 conn(0x7f0c18070e10 msgr2=0x7f0c180a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.639+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2469608650 shutdown_connections 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.640+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2469608650 wait complete. 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.640+0000 7f0c1ff9d640 10 monclient: build_initial_monmap 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.640+0000 7f0c1ff9d640 10 monclient: monmap: 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 Processor -- start 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 -- start start 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient: init 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c18141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c04004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c18131ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 0x7f0c1811bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1811b6d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c180a0920 con 0x7f0c18131ac0 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1809b660 con 0x7f0c04004780 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1811d630 con 0x7f0c1811b6d0 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1ff9d640 10 monclient(hunting): authenticate will time out at 1077.637899s 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1e53a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 msgr2=0x7f0c1811bb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.641+0000 7f0c1e53a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 0x7f0c1811bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35676/0 (socket says 172.21.15.17:35676) 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 -- 172.21.15.17:0/2411275865 learned_addr learned my addr 172.21.15.17:0/2411275865 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): get_auth_request con 0x7f0c18131ac0 auth_method 0 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c18141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 10 monclient(hunting): get_auth_request con 0x7f0c04004780 auth_method 0 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient(hunting): handle_auth_done global_id 84222 payload 824 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 -- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 msgr2=0x7f0c1811bb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 0x7f0c1811bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 -- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 msgr2=0x7f0c18141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c18141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient: _finish_hunting 0 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 monclient: found mon.noname-a 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1d538640 1 -- 172.21.15.17:0/2411275865 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c18142050 con 0x7f0c18131ac0 2024-09-15T01:56:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.642+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c18141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.644+0000 7f0c1d538640 1 -- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 msgr2=0x7f0c18131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.644+0000 7f0c1d538640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c08001900 tx=0x7f0c08034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08043070 con 0x7f0c18131ac0 2024-09-15T01:56:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: dump: 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: _finish_auth 0 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 5 monclient: authenticate success, global_id 84222 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c08007720 con 0x7f0c18131ac0 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 10 monclient: _renew_subs 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1811d120 con 0x7f0c18131ac0 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c080049b0 con 0x7f0c18131ac0 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0dffb640 10 monclient: dump: 2024-09-15T01:56:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 10 monclient: _renew_subs 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c0f7fe640 4 set_mon_vals no callback set 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.645+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1811dcf0 con 0x7f0c18131ac0 2024-09-15T01:56:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.646+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0c08005030 con 0x7f0c18131ac0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.646+0000 7f0c1ff9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.646+0000 7f0c1ff9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.646+0000 7f0c1ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.646+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1811efe0 con 0x7f0c18131ac0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.647+0000 7f0c0dffb640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 0x7f0bfc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.647+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f0c0806cab0 con 0x7f0c18131ac0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.647+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 0x7f0bfc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.647+0000 7f0c1dd39640 10 monclient: get_auth_request con 0x7f0bfc0386f0 auth_method 0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.650+0000 7f0c1dd39640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 0x7f0bfc03aba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c14003940 tx=0x7f0c140095a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.651+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0c08048410 con 0x7f0c18131ac0 2024-09-15T01:56:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.652+0000 7f0c0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.652+0000 7f0c0dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:57.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.881+0000 7f0c1ff9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:56:57.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.881+0000 7f0c1ff9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:57.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.881+0000 7f0c1ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:57.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.881+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0c180724a0 con 0x7f0c18131ac0 2024-09-15T01:56:57.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.882+0000 7f0c0dffb640 1 -- 172.21.15.17:0/2411275865 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f0c0803d3a0 con 0x7f0c18131ac0 2024-09-15T01:56:57.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.882+0000 7f0c0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:56:57.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.882+0000 7f0c0dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:57.886 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:56:57.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 msgr2=0x7f0bfc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:57.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 0x7f0bfc03aba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c14003940 tx=0x7f0c140095a0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 10 monclient: shutdown 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 msgr2=0x7f0c18131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c08001900 tx=0x7f0c08034000 comp rx=0 tx=0).stop 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 shutdown_connections 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bfc0386f0 0x7f0bfc03aba0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c18131ac0 0x7f0c18131e90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c1811b6d0 0x7f0c1811bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 --2- 172.21.15.17:0/2411275865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04004780 0x7f0c18141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:57.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 >> 172.21.15.17:0/2411275865 conn(0x7f0c18070e10 msgr2=0x7f0c1809d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:57.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 shutdown_connections 2024-09-15T01:56:57.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.884+0000 7f0c1ff9d640 1 -- 172.21.15.17:0/2411275865 wait complete. 2024-09-15T01:56:57.896 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:56:57.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:56:57.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 10 monclient: get_monmap_and_config 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 10 monclient: monmap: 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:57.992932+0000 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:57.992932+0000 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:57.994 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:57.995 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:57.995 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:57.995 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:57.995 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:57.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 1 Processor -- start 2024-09-15T01:56:57.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 1 -- start start 2024-09-15T01:56:57.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.992+0000 7f82e5b5d640 10 monclient: init 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82e00a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82e0132040 0x7f82e00987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f82e0132040 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 0x7f82e01391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e0136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e00a01e0 con 0x7f82e00a58b0 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e00a1580 con 0x7f82e0132040 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00a5c80 con 0x7f82e0136dd0 2024-09-15T01:56:57.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82e5b5d640 10 monclient(hunting): authenticate will time out at 1077.989393s 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82dffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 msgr2=0x7f82e01391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82dffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 0x7f82e01391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82e0132040 0x7f82e00987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82df7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82e00a58b0 0x7f82e00980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:56550/0 (socket says 172.21.15.17:56550) 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): get_auth_request con 0x7f82e0132040 auth_method 0 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82df7fe640 1 -- 172.21.15.17:0/3609093892 learned_addr learned my addr 172.21.15.17:0/3609093892 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 10 monclient(hunting): handle_auth_done global_id 5893 payload 274 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 1 -- 172.21.15.17:0/3609093892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 msgr2=0x7f82e01391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 1 --2- 172.21.15.17:0/3609093892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 0x7f82e01391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.993+0000 7f82deffd640 1 -- 172.21.15.17:0/3609093892 >> v1:172.21.15.67:6789/0 conn(0x7f82e00a58b0 legacy=0x7f82e00980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.994+0000 7f82deffd640 10 monclient: _finish_hunting 0 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.994+0000 7f82deffd640 1 monclient: found mon.noname-c 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.994+0000 7f82deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:58.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.994+0000 7f82deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:58.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.994+0000 7f82deffd640 1 -- 172.21.15.17:0/3609093892 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e0132440 con 0x7f82e0132040 2024-09-15T01:56:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82deffd640 1 -- 172.21.15.17:0/3609093892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e0132040 msgr2=0x7f82e00987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82deffd640 1 --2- 172.21.15.17:0/3609093892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e0132040 0x7f82e00987b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009850 tx=0x7f82d40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d48635881cd1f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 1 -- 172.21.15.17:0/3609093892 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4032440 con 0x7f82e0132040 2024-09-15T01:56:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: dump: 2024-09-15T01:56:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 1 -- 172.21.15.17:0/3609093892 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d40325a0 con 0x7f82e0132040 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:58.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 1 -- 172.21.15.17:0/3609093892 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4032940 con 0x7f82e0132040 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 5 monclient: authenticate success, global_id 5893 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 4 set_mon_vals no callback set 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:58.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82de7fc640 10 monclient: dump: 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.016 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 10 monclient: shutdown 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:58.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 -- 172.21.15.17:0/3609093892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e0132040 msgr2=0x7f82e00987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/3609093892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e0132040 0x7f82e00987b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009850 tx=0x7f82d40090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 -- 172.21.15.17:0/3609093892 shutdown_connections 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/3609093892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0136dd0 0x7f82e01391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/3609093892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e0132040 0x7f82e00987b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.997+0000 7f82e5b5d640 1 -- 172.21.15.17:0/3609093892 >> 172.21.15.17:0/3609093892 conn(0x7f82e0070ed0 msgr2=0x7f82e00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:58.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.998+0000 7f82e5b5d640 1 -- 172.21.15.17:0/3609093892 shutdown_connections 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.998+0000 7f82e5b5d640 1 -- 172.21.15.17:0/3609093892 wait complete. 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.998+0000 7f82e5b5d640 10 monclient: build_initial_monmap 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.998+0000 7f82e5b5d640 10 monclient: monmap: 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:58.020 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.998+0000 7f82e5b5d640 1 Processor -- start 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 -- start start 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient: init 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:56:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e00a58b0 0x7f82e0141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e00a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0132040 0x7f82e0141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f82e0132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f82e0136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0098dc0 con 0x7f82e0136dd0 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e01364f0 con 0x7f82e00a58b0 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e01367d0 con 0x7f82e0132040 2024-09-15T01:56:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82deffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0132040 msgr2=0x7f82e0141ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82e5b5d640 10 monclient(hunting): authenticate will time out at 1077.995540s 2024-09-15T01:56:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82deffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0132040 0x7f82e0141ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35678/0 (socket says 172.21.15.17:35678) 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 learned_addr learned my addr 172.21.15.17:0/15249931 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): get_auth_request con 0x7f82e0136dd0 auth_method 0 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82df7fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e00a58b0 0x7f82e0141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82df7fe640 10 monclient(hunting): get_auth_request con 0x7f82e00a58b0 auth_method 0 2024-09-15T01:56:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient(hunting): handle_auth_done global_id 84225 payload 824 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0132040 msgr2=0x7f82e0141ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82e0132040 0x7f82e0141ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e00a58b0 msgr2=0x7f82e0141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e00a58b0 0x7f82e0141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 1 monclient: found mon.noname-a 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:57.999+0000 7f82dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.000+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4009390 con 0x7f82e0136dd0 2024-09-15T01:56:58.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.000+0000 7f82df7fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82e00a58b0 0x7f82e0141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:56:58.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 msgr2=0x7f82e0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82dffff640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82d800b490 tx=0x7f82d8006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82dffff640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82d800b490 tx=0x7f82d8006f90 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f82d8003d90 mon_map magic: 0 v1 2024-09-15T01:56:58.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8003d90 con 0x7f82e0136dd0 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 10 monclient: dump: 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 10 monclient: _finish_auth 0 2024-09-15T01:56:58.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d8006150 con 0x7f82e0136dd0 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 5 monclient: authenticate success, global_id 84225 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 10 monclient: _renew_subs 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e0134400 con 0x7f82e0136dd0 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82ddffb640 4 set_mon_vals no callback set 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 10 monclient: _renew_subs 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.002+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e0135f30 con 0x7f82e0136dd0 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8006510 con 0x7f82e0136dd0 2024-09-15T01:56:58.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 10 monclient: dump: 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.035 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f82d801b3b0 con 0x7f82e0136dd0 2024-09-15T01:56:58.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82c77fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82cc038740 0x7f82cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82e5b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:58.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 0x7f82e0133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0063b20 con 0x7f82e0133060 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82df7fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82cc038740 0x7f82cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82df7fe640 10 monclient: get_auth_request con 0x7f82cc038740 auth_method 0 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 0x7f82e0133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient: get_auth_request con 0x7f82e0133060 auth_method 0 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.003+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.004+0000 7f82deffd640 10 monclient(hunting): handle_auth_done global_id 84228 payload 824 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.004+0000 7f82deffd640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e00724a0 con 0x7f82e0133060 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82df7fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82cc038740 0x7f82cc03abf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f82d000c5a0 tx=0x7f82d0009680 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82deffd640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 msgr2=0x7f82e0133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82deffd640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 0x7f82e0133430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82e00a60e0 tx=0x7f82d4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4032440 con 0x7f82e0133060 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 10 monclient: dump: 2024-09-15T01:56:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f82d4031e70 con 0x7f82e0133060 2024-09-15T01:56:58.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:58.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:58.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 msgr2=0x7f82e0133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 0x7f82e0133430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82e00a60e0 tx=0x7f82d4032000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 msgr2=0x7f82e0133430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:58.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.006+0000 7f82c77fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133060 0x7f82e0133430 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82e00a60e0 tx=0x7f82d4032000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82e5b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:58.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 0x7f82e01374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82e5b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e01327a0 con 0x7f82e0133970 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 0x7f82e01374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient: get_auth_request con 0x7f82e0133970 auth_method 0 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 10 monclient(hunting): handle_auth_done global_id 84231 payload 824 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.012+0000 7f82deffd640 1 -- 172.21.15.17:0/15249931 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f82e0134ff0 con 0x7f82e0133970 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82deffd640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 msgr2=0x7f82e01374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82deffd640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 0x7f82e01374e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009850 tx=0x7f82d403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d403d840 con 0x7f82e0133970 2024-09-15T01:56:58.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 10 monclient: dump: 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4556 (secure 0 0 0) 0x7f82d404c070 con 0x7f82e0133970 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 msgr2=0x7f82e01374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 0x7f82e01374e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009850 tx=0x7f82d403d000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 msgr2=0x7f82e01374e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0133970 0x7f82e01374e0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009850 tx=0x7f82d403d000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:58.049 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 74, 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.050 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:58.052 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:58.053 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.054 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.055 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.056 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:58.057 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.058 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:58.059 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:58.061 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.063 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.065 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:58.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:58.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82cc038740 msgr2=0x7f82cc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f82cc038740 0x7f82cc03abf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f82d000c5a0 tx=0x7f82d0009680 comp rx=0 tx=0).stop 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 10 monclient: shutdown 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 msgr2=0x7f82e0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82d800b490 tx=0x7f82d8006f90 comp rx=0 tx=0).stop 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82dffff640 1 -- 172.21.15.17:0/15249931 reap_dead start 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82c77fe640 1 -- 172.21.15.17:0/15249931 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (unknown 0 0 0) 0x7f82d804b970 con 0x7f82e0136dd0 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 shutdown_connections 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 --2- 172.21.15.17:0/15249931 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f82e0136dd0 0x7f82e0132b20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 >> 172.21.15.17:0/15249931 conn(0x7f82e0070ed0 msgr2=0x7f82e0096e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 shutdown_connections 2024-09-15T01:56:58.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.015+0000 7f82e5b5d640 1 -- 172.21.15.17:0/15249931 wait complete. 2024-09-15T01:56:58.068 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:56:58.068 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:56:58.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:56:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.163+0000 7f2e3c8c9640 10 monclient: get_monmap_and_config 2024-09-15T01:56:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.163+0000 7f2e3c8c9640 10 monclient: build_initial_monmap 2024-09-15T01:56:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.163+0000 7f2e3c8c9640 10 monclient: monmap: 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:56:58.164313+0000 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:56:58.164313+0000 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:58.166 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 Processor -- start 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 -- start start 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient: init 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e380a5780 0x7f2e3809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f2e380a5780 addr v2:172.21.15.67:3300/0 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e3809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e38139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2e38069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380a5b50 con 0x7f2e3809ccf0 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e380a18b0 con 0x7f2e380a5780 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3809c5f0 con 0x7f2e38069c90 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e3c8c9640 10 monclient(hunting): authenticate will time out at 1078.160813s 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e36d9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 msgr2=0x7f2e38139880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:58.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.164+0000 7f2e36d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e38139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e380a5780 0x7f2e3809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e380a5780 0x7f2e3809bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:42050/0 (socket says 172.21.15.17:42050) 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 -- 172.21.15.17:0/1446292211 learned_addr learned my addr 172.21.15.17:0/1446292211 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): get_auth_request con 0x7f2e380a5780 auth_method 0 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient(hunting): handle_auth_done global_id 5899 payload 274 2024-09-15T01:56:58.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 -- 172.21.15.17:0/1446292211 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 msgr2=0x7f2e38139880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 --2- 172.21.15.17:0/1446292211 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e38139880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 -- 172.21.15.17:0/1446292211 >> v1:172.21.15.67:6789/0 conn(0x7f2e3809ccf0 legacy=0x7f2e38069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient: _finish_hunting 0 2024-09-15T01:56:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 monclient: found mon.noname-c 2024-09-15T01:56:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:58.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:56:58.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.165+0000 7f2e3659c640 1 -- 172.21.15.17:0/1446292211 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3809d210 con 0x7f2e380a5780 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.168+0000 7f2e3659c640 1 -- 172.21.15.17:0/1446292211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 msgr2=0x7f2e3809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.168+0000 7f2e3659c640 1 --2- 172.21.15.17:0/1446292211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e3809bc90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2e380a19f0 tx=0x7f2e280091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc65249cf6fc5ef1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.168+0000 7f2e35d9b640 1 -- 172.21.15.17:0/1446292211 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28032440 con 0x7f2e380a5780 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:56:58.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: dump: 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: _finish_auth 0 2024-09-15T01:56:58.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 1 -- 172.21.15.17:0/1446292211 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e280325a0 con 0x7f2e380a5780 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 1 -- 172.21.15.17:0/1446292211 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28032980 con 0x7f2e380a5780 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 5 monclient: authenticate success, global_id 5899 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 4 set_mon_vals no callback set 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e35d9b640 10 monclient: dump: 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.183 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 10 monclient: shutdown 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/1446292211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 msgr2=0x7f2e3809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/1446292211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e3809bc90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2e380a19f0 tx=0x7f2e280091d0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/1446292211 shutdown_connections 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/1446292211 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e38139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/1446292211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e3809bc90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/1446292211 >> 172.21.15.17:0/1446292211 conn(0x7f2e38070e10 msgr2=0x7f2e380a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/1446292211 shutdown_connections 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.169+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/1446292211 wait complete. 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient: build_initial_monmap 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient: monmap: 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:56:58.185 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 1 Processor -- start 2024-09-15T01:56:58.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 1 -- start start 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient: init 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e3813df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f2e38069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.170+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f2e3809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e381414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2e380a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38122d80 con 0x7f2e3809ccf0 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e381229a0 con 0x7f2e380a5780 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 msgr2=0x7f2e3813df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38120c90 con 0x7f2e38069c90 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3c8c9640 10 monclient(hunting): authenticate will time out at 1078.167045s 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e3659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e3813df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35712/0 (socket says 172.21.15.17:35712) 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 -- 172.21.15.17:0/3274415987 learned_addr learned my addr 172.21.15.17:0/3274415987 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:56:58.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): get_auth_request con 0x7f2e3809ccf0 auth_method 0 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient(hunting): handle_auth_done global_id 84234 payload 824 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e36d9d640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e381414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 msgr2=0x7f2e3813df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2e38069c90 0x7f2e3813df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 msgr2=0x7f2e381414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e380a5780 0x7f2e381414e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 monclient: found mon.noname-a 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.171+0000 7f2e2ffff640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28009450 con 0x7f2e3809ccf0 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2ffff640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 msgr2=0x7f2e3813d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2ffff640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000b4f0 tx=0x7f2e200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2ffff640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000b4f0 tx=0x7f2e200090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f2e2000f880 config(0 keys) v1 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2000f720 con 0x7f2e3809ccf0 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 10 monclient: dump: 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2000f880 con 0x7f2e3809ccf0 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 5 monclient: authenticate success, global_id 84234 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 10 monclient: _renew_subs 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e38120480 con 0x7f2e3809ccf0 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3559a640 4 set_mon_vals no callback set 2024-09-15T01:56:58.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 10 monclient: _renew_subs 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.173+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3813e980 con 0x7f2e3809ccf0 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2000fc40 con 0x7f2e3809ccf0 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e3c8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 10 monclient: dump: 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.193 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2e20011610 con 0x7f2e3809ccf0 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 0x7f2e38141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38122860 con 0x7f2e381217f0 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.174+0000 7f2e2e7fc640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e180386f0 0x7f2e1803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 0x7f2e38141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient: get_auth_request con 0x7f2e381217f0 auth_method 0 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e36d9d640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e180386f0 0x7f2e1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e36d9d640 10 monclient: get_auth_request con 0x7f2e180386f0 auth_method 0 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 10 monclient(hunting): handle_auth_done global_id 5902 payload 824 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.175+0000 7f2e3659c640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e38123060 con 0x7f2e381217f0 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.177+0000 7f2e36d9d640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e180386f0 0x7f2e1803aba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2e38122d30 tx=0x7f2e30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e3659c640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 msgr2=0x7f2e38141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e3659c640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 0x7f2e38141a20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2e28034000 tx=0x7f2e28032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28032440 con 0x7f2e381217f0 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.178+0000 7f2e2e7fc640 10 monclient: dump: 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2e28031e70 con 0x7f2e381217f0 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 msgr2=0x7f2e38141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 0x7f2e38141a20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2e28034000 tx=0x7f2e28032000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 msgr2=0x7f2e38141a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:58.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.179+0000 7f2e2e7fc640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e381217f0 0x7f2e38141a20 secure :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2e28034000 tx=0x7f2e28032000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.184+0000 7f2e3c8c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:56:58.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.184+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 0x7f2e3811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:58.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.184+0000 7f2e3c8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:58.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.184+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38063b20 con 0x7f2e38141f60 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 0x7f2e3811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient: get_auth_request con 0x7f2e38141f60 auth_method 0 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 10 monclient(hunting): handle_auth_done global_id 5905 payload 824 2024-09-15T01:56:58.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.185+0000 7f2e3659c640 1 -- 172.21.15.17:0/3274415987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e38142360 con 0x7f2e38141f60 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.187+0000 7f2e3659c640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 msgr2=0x7f2e3811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.187+0000 7f2e3659c640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 0x7f2e3811b6d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2e2802afd0 tx=0x7f2e2803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2804a070 con 0x7f2e38141f60 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 10 monclient: dump: 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4724 (secure 0 0 0) 0x7f2e2804f450 con 0x7f2e38141f60 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 msgr2=0x7f2e3811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 0x7f2e3811b6d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2e2802afd0 tx=0x7f2e2803d000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 msgr2=0x7f2e3811b6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:56:58.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.188+0000 7f2e2e7fc640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e38141f60 0x7f2e3811b6d0 secure :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2e2802afd0 tx=0x7f2e2803d000 comp rx=0 tx=0).stop 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 74, 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T01:56:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:56:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:56:58.208 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:56:58.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:56:58.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:56:58.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.208+0000 7fdd417fa640 10 monclient: tick 2024-09-15T01:56:58.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.208+0000 7fdd417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e180386f0 msgr2=0x7f2e1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2e180386f0 0x7f2e1803aba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2e38122d30 tx=0x7f2e30008040 comp rx=0 tx=0).stop 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 10 monclient: shutdown 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 msgr2=0x7f2e3813d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 --2- 172.21.15.17:0/3274415987 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e3809ccf0 0x7f2e3813d620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000b4f0 tx=0x7f2e200090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e36d9d640 1 -- 172.21.15.17:0/3274415987 reap_dead start 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e2e7fc640 1 -- 172.21.15.17:0/3274415987 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 7213+0+0 (unknown 0 0 0) 0x7f2e2004c1d0 con 0x7f2e3809ccf0 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e36d9d640 1 -- 172.21.15.17:0/3274415987 reap_dead start 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 shutdown_connections 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 >> 172.21.15.17:0/3274415987 conn(0x7f2e38070e10 msgr2=0x7f2e38098da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 shutdown_connections 2024-09-15T01:56:58.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:56:58.189+0000 7f2e3c8c9640 1 -- 172.21.15.17:0/3274415987 wait complete. 2024-09-15T01:56:58.222 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:56:58.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:56:58.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.914+0000 7fdd427fc640 1 -- 172.21.15.67:0/3332171956 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v138) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdd480102e0 con 0x7fdd540979c0 2024-09-15T01:56:58.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.914+0000 7fdd427fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:56:58.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.914+0000 7fdd427fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:56:58.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 msgr2=0x7fdd3403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 0x7fdd3403abf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009c80 tx=0x7fdd440090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 10 monclient: shutdown 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 msgr2=0x7fdd54126470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800c610 tx=0x7fdd48007ed0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 shutdown_connections 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdd34038740 0x7fdd3403abf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdd54097df0 0x7fdd541269b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd540979c0 0x7fdd54126470 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 --2- 172.21.15.67:0/3332171956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdd5406e860 0x7fdd54130e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:58.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 >> 172.21.15.67:0/3332171956 conn(0x7fdd540a13a0 msgr2=0x7fdd5409af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 shutdown_connections 2024-09-15T01:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:58.917+0000 7fdd58c8b640 1 -- 172.21.15.67:0/3332171956 wait complete. 2024-09-15T01:56:58.930 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create cool 2 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.010+0000 7f65e7268640 10 monclient: get_monmap_and_config 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.010+0000 7f65e7268640 10 monclient: build_initial_monmap 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.010+0000 7f65e7268640 10 monclient: monmap: 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:56:59.011805+0000 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:56:59.011805+0000 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:56:59.013 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:56:59.014 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:56:59.014 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:56:59.014 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:56:59.014 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 1 Processor -- start 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 1 -- start start 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 10 monclient: init 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f65e00a0b50 0x7f65e0069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-a con 0x7f65e00a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:56:59.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.011+0000 7f65e7268640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-b con 0x7f65e009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65e0131b10 addr v1:172.21.15.17:6789/0 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00a61a0 con 0x7f65e0131b10 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00a1610 con 0x7f65e00a0b50 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e009f7d0 con 0x7f65e009eea0 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e7268640 10 monclient(hunting): authenticate will time out at 1051.481054s 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e4803640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 msgr2=0x7f65e009f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e4803640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f65e00a0b50 0x7f65e0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:59.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5805640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f65e0131b10 0x7f65e0131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:42904/0 (socket says 172.21.15.67:42904) 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5805640 1 -- 172.21.15.67:0/2287331244 learned_addr learned my addr 172.21.15.67:0/2287331244 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): get_auth_request con 0x7f65e00a0b50 auth_method 0 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient(hunting): handle_auth_done global_id 84237 payload 274 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 -- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 msgr2=0x7f65e009f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 --2- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 -- 172.21.15.67:0/2287331244 >> v1:172.21.15.17:6789/0 conn(0x7f65e0131b10 legacy=0x7f65e0131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:56:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient: _finish_hunting 0 2024-09-15T01:56:59.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 monclient: found mon.noname-a 2024-09-15T01:56:59.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:59.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:56:59.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.012+0000 7f65e5004640 1 -- 172.21.15.67:0/2287331244 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00a04c0 con 0x7f65e00a0b50 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.015+0000 7f65e5004640 1 -- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e00a0b50 msgr2=0x7f65e0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.015+0000 7f65e5004640 1 --2- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e00a0b50 0x7f65e0069380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009850 tx=0x7f65dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5320e3e05c1f1c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 1 -- 172.21.15.67:0/2287331244 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc032440 con 0x7f65e00a0b50 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: dump: 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.023 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: _finish_auth 0 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 1 -- 172.21.15.67:0/2287331244 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc0325a0 con 0x7f65e00a0b50 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 1 -- 172.21.15.67:0/2287331244 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc032940 con 0x7f65e00a0b50 2024-09-15T01:56:59.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 5 monclient: authenticate success, global_id 84237 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 10 monclient: get_monmap_and_config success 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 4 set_mon_vals no callback set 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65d7fff640 10 monclient: dump: 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:59.025 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 10 monclient: shutdown 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 -- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e00a0b50 msgr2=0x7f65e0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 --2- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e00a0b50 0x7f65e0069380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009850 tx=0x7f65dc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 -- 172.21.15.67:0/2287331244 shutdown_connections 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 --2- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 --2- 172.21.15.67:0/2287331244 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e00a0b50 0x7f65e0069380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 -- 172.21.15.67:0/2287331244 >> 172.21.15.67:0/2287331244 conn(0x7f65e0071330 msgr2=0x7f65e00a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 -- 172.21.15.67:0/2287331244 shutdown_connections 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.016+0000 7f65e7268640 1 -- 172.21.15.67:0/2287331244 wait complete. 2024-09-15T01:56:59.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.017+0000 7f65e7268640 10 monclient: build_initial_monmap 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.017+0000 7f65e7268640 10 monclient: monmap: 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.017+0000 7f65e7268640 1 Processor -- start 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.017+0000 7f65e7268640 1 -- start start 2024-09-15T01:56:59.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.017+0000 7f65e7268640 10 monclient: init 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient: _reopen_session rank -1 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-c con 0x7f65e009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-b con 0x7f65e00a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e0131b10 0x7f65e011b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): picked mon.noname-a con 0x7f65e0131b10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00a12d0 con 0x7f65e0131b10 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0134c30 con 0x7f65e00a0b50 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): start opening mon connection 2024-09-15T01:56:59.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e011e050 con 0x7f65e009eea0 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): _renew_subs 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e7268640 10 monclient(hunting): authenticate will time out at 1051.487555s 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5004640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 msgr2=0x7f65e009a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5004640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47764/0 (socket says 172.21.15.67:47764) 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 1 -- 172.21.15.67:0/3365031456 learned_addr learned my addr 172.21.15.67:0/3365031456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5805640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e0131b10 0x7f65e011b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): get_auth_request con 0x7f65e00a0b50 auth_method 0 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:59.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5805640 10 monclient(hunting): get_auth_request con 0x7f65e0131b10 auth_method 0 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5805640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e5805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.018+0000 7f65e4803640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e5805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e5805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e5805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 10 monclient(hunting): handle_auth_done global_id 5908 payload 824 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 -- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 msgr2=0x7f65e009a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:56:59.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 -- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e0131b10 msgr2=0x7f65e011b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e0131b10 0x7f65e011b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 10 monclient: _finish_hunting 0 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 monclient: found mon.noname-b 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:56:59.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.019+0000 7f65e4803640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc009390 con 0x7f65e00a0b50 2024-09-15T01:56:59.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e4803640 1 -- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 msgr2=0x7f65e0130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:56:59.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e4803640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f65d000b630 tx=0x7f65d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:59.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e4803640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f65d000b630 tx=0x7f65d00091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f65d0015070 mon_map magic: 0 v1 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0015070 con 0x7f65e00a0b50 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 10 monclient: dump: 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:59.034 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 10 monclient: _finish_auth 0 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0003d90 con 0x7f65e00a0b50 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 5 monclient: authenticate success, global_id 5908 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 10 monclient: _renew_subs 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e011dd70 con 0x7f65e00a0b50 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65d77fe640 4 set_mon_vals no callback set 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 10 monclient: _renew_subs 2024-09-15T01:56:59.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.021+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e011ce10 con 0x7f65e00a0b50 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0006380 con 0x7f65e00a0b50 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 10 monclient: dump: 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:56:59.036 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f65d0010440 con 0x7f65e00a0b50 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65d57fa640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 0x7f65c403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e7268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e7268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e7268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e004bf60 con 0x7f65e00a0b50 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e5004640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 0x7f65c403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.022+0000 7f65e5004640 10 monclient: get_auth_request con 0x7f65c4038740 auth_method 0 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.025+0000 7f65e5004640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 0x7f65c403abf0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f65dc031000 tx=0x7f65dc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:56:59.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.027+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f65d004bf20 con 0x7f65e00a0b50 2024-09-15T01:56:59.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.644+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f65d0006770 con 0x7f65e00a0b50 2024-09-15T01:56:59.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.644+0000 7f65d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:56:59.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.644+0000 7f65d57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:56:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.814+0000 7f65e7268640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:56:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.814+0000 7f65e7268640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:56:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.814+0000 7f65e7268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:56:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.814+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f65e006f000 con 0x7f65e00a0b50 2024-09-15T01:56:59.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.918+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v139) v1 ==== 111+0+0 (secure 0 0 0) 0x7f65d0018780 con 0x7f65e00a0b50 2024-09-15T01:56:59.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.918+0000 7f65d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:56:59.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:56:59.918+0000 7f65d57fa640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-15T01:57:00.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.016+0000 7f65e7268640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:57:00.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.016+0000 7f65e7268640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:57:00.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.016+0000 7f65e7268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.016+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f65e0136500 con 0x7f65e00a0b50 2024-09-15T01:57:00.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.018+0000 7f65d57fa640 1 -- 172.21.15.67:0/3365031456 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v139) v1 ==== 118+0+0 (secure 0 0 0) 0x7f65d0018d60 con 0x7f65e00a0b50 2024-09-15T01:57:00.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.018+0000 7f65d57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T01:57:00.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.018+0000 7f65d57fa640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-15T01:57:00.020 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'cool' already exists 2024-09-15T01:57:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 msgr2=0x7f65c403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 0x7f65c403abf0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f65dc031000 tx=0x7f65dc032000 comp rx=0 tx=0).stop 2024-09-15T01:57:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 10 monclient: shutdown 2024-09-15T01:57:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 msgr2=0x7f65e0130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.019+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f65d000b630 tx=0x7f65d00091a0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 shutdown_connections 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f65c4038740 0x7f65c403abf0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f65e0131b10 0x7f65e011b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00a0b50 0x7f65e0130c50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 --2- 172.21.15.67:0/3365031456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f65e009eea0 0x7f65e009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 >> 172.21.15.67:0/3365031456 conn(0x7f65e0071330 msgr2=0x7f65e009d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 shutdown_connections 2024-09-15T01:57:00.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.020+0000 7f65e7268640 1 -- 172.21.15.67:0/3365031456 wait complete. 2024-09-15T01:57:00.031 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool create cold 2 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.105+0000 7f36cff5d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.105+0000 7f36cff5d640 10 monclient: build_initial_monmap 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.105+0000 7f36cff5d640 10 monclient: monmap: 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:00.106672+0000 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:00.106672+0000 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:00.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 Processor -- start 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 -- start start 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient: init 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36c8098c80 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36c809bf50 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36c8138520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-b con 0x7f36c8067590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c809e240 con 0x7f36c809bf50 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c809c340 con 0x7f36c8098c80 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c809e500 con 0x7f36c8067590 2024-09-15T01:57:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cff5d640 10 monclient(hunting): authenticate will time out at 1052.575892s 2024-09-15T01:57:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cdcf9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f36c8098c80 0x7f36c809b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38630/0 (socket says 172.21.15.67:38630) 2024-09-15T01:57:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 learned_addr learned my addr 172.21.15.67:0/1218887140 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.106+0000 7f36ce4fa640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 msgr2=0x7f36c8138520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.107+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36c8138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 >> v1:172.21.15.67:6789/0 conn(0x7f36c8098c80 legacy=0x7f36c809b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495419731 0 0) 0x7f36c4001450 con 0x7f36c8098c80 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): my global_id is 5911 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b40034e0 con 0x7f36c8098c80 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cd4f8640 1 -- 172.21.15.67:0/1218887140 >> v1:172.21.15.17:6789/0 conn(0x7f36c809bf50 legacy=0x7f36c8134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:00.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284037312 0 0) 0x7f36b40034e0 con 0x7f36c8098c80 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 msgr2=0x7f36c8138520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36c8138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 >> v1:172.21.15.17:6789/0 conn(0x7f36c809bf50 legacy=0x7f36c8134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: _finish_hunting 0 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80679f0 con 0x7f36c8098c80 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: _finish_auth 0 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f36c40018b0 con 0x7f36c8098c80 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:00.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: dump: 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:57:00.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 >> v1:172.21.15.67:6789/0 conn(0x7f36c8098c80 legacy=0x7f36c809b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 0x7f36b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): picked mon.b con 0x7f36b4003a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36b4004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): picked mon.c con 0x7f36c8067590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4004de0 0x7f36b4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): picked mon.a con 0x7f36b4004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c809e500 con 0x7f36b4004de0 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400bcf0 con 0x7f36b4003a50 2024-09-15T01:57:00.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400beb0 con 0x7f36c8067590 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cccf7640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cd4f8640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 msgr2=0x7f36b4004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 0x7f36b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cd4f8640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36b4004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): get_auth_request con 0x7f36b4003a50 auth_method 0 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4004de0 0x7f36b4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:00.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36ce4fa640 10 monclient(hunting): get_auth_request con 0x7f36b4004de0 auth_method 0 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36ce4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36ce4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36ce4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.109+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_done global_id 5911 payload 293 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 msgr2=0x7f36b4004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c8067590 0x7f36b4004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4004de0 msgr2=0x7f36b4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4004de0 0x7f36b4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 10 monclient: _finish_hunting 0 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 monclient: found mon.b 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40034e0 con 0x7f36b4003a50 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 10 monclient: _finish_auth 0 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cdcf9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cff5d640 5 monclient: authenticate success, global_id 5911 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36cff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.110+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4004de0 0x7f36b4005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.112+0000 7f36cdcf9640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 msgr2=0x7f36b4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.112+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 0x7f36b4003e20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f36c4001620 tx=0x7f36c4007c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e985a912662b1885 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4005960 con 0x7f36b4003a50 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: dump: 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c4005ae0 con 0x7f36b4003a50 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 4 set_mon_vals no callback set 2024-09-15T01:57:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 1 -- 172.21.15.67:0/1218887140 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4005ea0 con 0x7f36b4003a50 2024-09-15T01:57:00.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:00.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:00.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cccf7640 10 monclient: dump: 2024-09-15T01:57:00.129 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:00.129 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:00.132 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 10 monclient: shutdown 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 -- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 msgr2=0x7f36b4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 --2- 172.21.15.67:0/1218887140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36b4003a50 0x7f36b4003e20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f36c4001620 tx=0x7f36c4007c20 comp rx=0 tx=0).stop 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36ce4fa640 1 -- 172.21.15.67:0/1218887140 reap_dead start 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 -- 172.21.15.67:0/1218887140 shutdown_connections 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 -- 172.21.15.67:0/1218887140 >> 172.21.15.67:0/1218887140 conn(0x7f36c80a2110 msgr2=0x7f36c80a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 -- 172.21.15.67:0/1218887140 shutdown_connections 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.113+0000 7f36cff5d640 1 -- 172.21.15.67:0/1218887140 wait complete. 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient: build_initial_monmap 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient: monmap: 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:00.133 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 Processor -- start 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 -- start start 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient: init 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4003a50 0x7f36c81410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-a con 0x7f36b4003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 0x7f36c812eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-c con 0x7f36c812ec20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): picked mon.noname-b con 0x7f36c8096a70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c811e040 con 0x7f36b4003a50 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c812f530 con 0x7f36c8096a70 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c809c4e0 con 0x7f36c812ec20 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.114+0000 7f36cff5d640 10 monclient(hunting): authenticate will time out at 1052.583957s 2024-09-15T01:57:00.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cd4f8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 msgr2=0x7f36c812eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cd4f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 0x7f36c812eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47784/0 (socket says 172.21.15.67:47784) 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 -- 172.21.15.67:0/3924363229 learned_addr learned my addr 172.21.15.67:0/3924363229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): get_auth_request con 0x7f36c8096a70 auth_method 0 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4003a50 0x7f36c81410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): get_auth_request con 0x7f36b4003a50 auth_method 0 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36cdcf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient(hunting): handle_auth_done global_id 5914 payload 824 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 -- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 msgr2=0x7f36c812eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 0x7f36c812eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 -- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4003a50 msgr2=0x7f36c81410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4003a50 0x7f36c81410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient: _finish_hunting 0 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 monclient: found mon.noname-b 2024-09-15T01:57:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.115+0000 7f36ce4fa640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80a1d40 con 0x7f36c8096a70 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36ce4fa640 1 -- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 msgr2=0x7f36c8096ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36ce4fa640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f36c000dc20 tx=0x7f36c0009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0005530 con 0x7f36c8096a70 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: dump: 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0005690 con 0x7f36c8096a70 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36cff5d640 5 monclient: authenticate success, global_id 5914 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36cff5d640 10 monclient: _renew_subs 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36cff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c811dd60 con 0x7f36c8096a70 2024-09-15T01:57:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0005a50 con 0x7f36c8096a70 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bdffb640 10 monclient: dump: 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36cff5d640 10 monclient: _renew_subs 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8141dd0 con 0x7f36c8096a70 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.117+0000 7f36bffff640 4 set_mon_vals no callback set 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f36c000b430 con 0x7f36c8096a70 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36bdffb640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 0x7f36ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c804bf60 con 0x7f36c8096a70 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 7556+0+0 (secure 0 0 0) 0x7f36c0006e40 con 0x7f36c8096a70 2024-09-15T01:57:00.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.118+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 0x7f36ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:00.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.119+0000 7f36cdcf9640 10 monclient: get_auth_request con 0x7f36ac038740 auth_method 0 2024-09-15T01:57:00.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.121+0000 7f36cdcf9640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 0x7f36ac03abf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f36b400be80 tx=0x7f36c4034000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:00.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.123+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36c001b020 con 0x7f36c8096a70 2024-09-15T01:57:00.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.123+0000 7f36bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:00.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.123+0000 7f36bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:00.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.289+0000 7f36cff5d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:00.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.289+0000 7f36cff5d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:00.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.289+0000 7f36cff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:00.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.289+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f36c8066930 con 0x7f36c8096a70 2024-09-15T01:57:00.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.923+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v140) v1 ==== 111+0+0 (secure 0 0 0) 0x7f36c0013a80 con 0x7f36c8096a70 2024-09-15T01:57:00.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.923+0000 7f36bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:00.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:00.923+0000 7f36bdffb640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-15T01:57:01.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.025+0000 7f36cff5d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:01.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.025+0000 7f36cff5d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:01.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.025+0000 7f36cff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.025+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f36c81415f0 con 0x7f36c8096a70 2024-09-15T01:57:01.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.027+0000 7f36bdffb640 1 -- 172.21.15.67:0/3924363229 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v140) v1 ==== 118+0+0 (secure 0 0 0) 0x7f36c00132a0 con 0x7f36c8096a70 2024-09-15T01:57:01.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.027+0000 7f36bdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T01:57:01.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.027+0000 7f36bdffb640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-15T01:57:01.029 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'cold' already exists 2024-09-15T01:57:01.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 msgr2=0x7f36ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 0x7f36ac03abf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f36b400be80 tx=0x7f36c4034000 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 10 monclient: shutdown 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 msgr2=0x7f36c8096ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f36c000dc20 tx=0x7f36c0009760 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 shutdown_connections 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f36ac038740 0x7f36ac03abf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c8096a70 0x7f36c8096ec0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f36c812ec20 0x7f36c812eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 --2- 172.21.15.67:0/3924363229 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f36b4003a50 0x7f36c81410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 >> 172.21.15.67:0/3924363229 conn(0x7f36c80a2110 msgr2=0x7f36c8132bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:01.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 shutdown_connections 2024-09-15T01:57:01.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.029+0000 7f36cff5d640 1 -- 172.21.15.67:0/3924363229 wait complete. 2024-09-15T01:57:01.040 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool set cold size 2 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.115+0000 7f5eac95d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.115+0000 7f5eac95d640 10 monclient: build_initial_monmap 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.115+0000 7f5eac95d640 10 monclient: monmap: 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:01.116926+0000 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:01.116926+0000 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:01.119 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.116+0000 7f5eac95d640 1 Processor -- start 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.116+0000 7f5eac95d640 1 -- start start 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.116+0000 7f5eac95d640 10 monclient: init 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ea80a1a00 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:01.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ea809b1f0 0x7f5ea8133b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea809b1f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 0x7f5ea8136070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea809d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea809cc40 con 0x7f5ea80a1a00 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea809cf80 con 0x7f5ea809b1f0 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80a1dd0 con 0x7f5ea809d5a0 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5eac95d640 10 monclient(hunting): authenticate will time out at 1053.586176s 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ea80a1a00 0x7f5ea809aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43280/0 (socket says 172.21.15.67:43280) 2024-09-15T01:57:01.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea659c640 1 -- 172.21.15.67:0/1439391696 learned_addr learned my addr 172.21.15.67:0/1439391696 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 --2- 172.21.15.67:0/1439391696 >> v2:172.21.15.67:3300/0 conn(0x7f5ea809b1f0 0x7f5ea8133b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): get_auth_request con 0x7f5ea809b1f0 auth_method 0 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1439391696 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 msgr2=0x7f5ea8136070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea6d9d640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 0x7f5ea8136070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient(hunting): handle_auth_done global_id 5917 payload 274 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 -- 172.21.15.67:0/1439391696 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 msgr2=0x7f5ea8136070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 0x7f5ea8136070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 -- 172.21.15.67:0/1439391696 >> v1:172.21.15.67:6789/0 conn(0x7f5ea80a1a00 legacy=0x7f5ea809aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 monclient: found mon.noname-c 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.117+0000 7f5ea5d9b640 1 -- 172.21.15.67:0/1439391696 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea812efa0 con 0x7f5ea809b1f0 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea5d9b640 1 -- 172.21.15.67:0/1439391696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea809b1f0 msgr2=0x7f5ea8133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea5d9b640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea809b1f0 0x7f5ea8133b20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009850 tx=0x7f5e9c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deca5ed7750b25cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 1 -- 172.21.15.67:0/1439391696 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c032440 con 0x7f5ea809b1f0 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: dump: 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:01.188 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: _finish_auth 0 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 1 -- 172.21.15.67:0/1439391696 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0325a0 con 0x7f5ea809b1f0 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5eac95d640 5 monclient: authenticate success, global_id 5917 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5eac95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5eac95d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5eac95d640 4 set_mon_vals no callback set 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 1 -- 172.21.15.67:0/1439391696 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c032940 con 0x7f5ea809b1f0 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.120+0000 7f5ea559a640 10 monclient: dump: 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 10 monclient: shutdown 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 -- 172.21.15.67:0/1439391696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea809b1f0 msgr2=0x7f5ea8133b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea809b1f0 0x7f5ea8133b20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009850 tx=0x7f5e9c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 -- 172.21.15.67:0/1439391696 shutdown_connections 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809d5a0 0x7f5ea8136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1439391696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea809b1f0 0x7f5ea8133b20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 -- 172.21.15.67:0/1439391696 >> 172.21.15.67:0/1439391696 conn(0x7f5ea8071340 msgr2=0x7f5ea80a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 -- 172.21.15.67:0/1439391696 shutdown_connections 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 1 -- 172.21.15.67:0/1439391696 wait complete. 2024-09-15T01:57:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 10 monclient: build_initial_monmap 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.121+0000 7f5eac95d640 10 monclient: monmap: 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 Processor -- start 2024-09-15T01:57:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 -- start start 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient: init 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 0x7f5ea809b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea809b1f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 0x7f5ea812dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea809d5a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea80a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea811c410 con 0x7f5ea809d5a0 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea811c0a0 con 0x7f5ea80a1a00 2024-09-15T01:57:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea809c580 con 0x7f5ea809b1f0 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5eac95d640 10 monclient(hunting): authenticate will time out at 1053.591656s 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 msgr2=0x7f5ea809b9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 0x7f5ea809b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47526/0 (socket says 172.21.15.67:47526) 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1930952172 learned_addr learned my addr 172.21.15.67:0/1930952172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): get_auth_request con 0x7f5ea80a1a00 auth_method 0 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:01.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea5d9b640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 0x7f5ea812dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea5d9b640 10 monclient(hunting): get_auth_request con 0x7f5ea809d5a0 auth_method 0 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.122+0000 7f5ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 10 monclient(hunting): handle_auth_done global_id 5920 payload 824 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 msgr2=0x7f5ea809b9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 0x7f5ea809b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 msgr2=0x7f5ea812dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 0x7f5ea812dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:57:01.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 monclient: found mon.noname-b 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c009390 con 0x7f5ea80a1a00 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.123+0000 7f5ea5d9b640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 0x7f5ea812dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5ea6d9d640 1 -- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 msgr2=0x7f5ea813f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5ea6d9d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000b490 tx=0x7f5ea0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5ea6d9d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000b490 tx=0x7f5ea0009060 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f5ea0003d40 mon_map magic: 0 v1 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0003d40 con 0x7f5ea80a1a00 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 10 monclient: dump: 2024-09-15T01:57:01.196 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 5 monclient: authenticate success, global_id 5920 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0006150 con 0x7f5ea80a1a00 2024-09-15T01:57:01.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5e967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 10 monclient: _renew_subs 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea811bdc0 con 0x7f5ea80a1a00 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5ea4d99640 4 set_mon_vals no callback set 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 10 monclient: _renew_subs 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.125+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea811a9b0 con 0x7f5ea80a1a00 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5eac95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5eac95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5eac95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea811c260 con 0x7f5ea80a1a00 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0006510 con 0x7f5ea80a1a00 2024-09-15T01:57:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 10 monclient: dump: 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ea0010570 con 0x7f5ea80a1a00 2024-09-15T01:57:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5e967fc640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 0x7f5e8803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5ea659c640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 0x7f5e8803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.126+0000 7f5ea659c640 10 monclient: get_auth_request con 0x7f5e88038740 auth_method 0 2024-09-15T01:57:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.129+0000 7f5ea659c640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 0x7f5e8803abf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5e98004b50 tx=0x7f5e98005100 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.131+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 7899+0+0 (secure 0 0 0) 0x7f5ea004bc30 con 0x7f5ea80a1a00 2024-09-15T01:57:01.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.748+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ea0006770 con 0x7f5ea80a1a00 2024-09-15T01:57:01.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.748+0000 7f5e967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:01.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.748+0000 7f5e967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:01.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.915+0000 7f5eac95d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:01.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.915+0000 7f5eac95d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:01.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.915+0000 7f5eac95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:01.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.915+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f5ea804bf60 con 0x7f5ea80a1a00 2024-09-15T01:57:01.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.927+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v141) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5ea0017030 con 0x7f5ea80a1a00 2024-09-15T01:57:01.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.927+0000 7f5e967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:01.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:01.927+0000 7f5e967fc640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-15T01:57:02.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.030+0000 7f5eac95d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:02.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.030+0000 7f5eac95d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:02.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.030+0000 7f5eac95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:02.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.030+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f5ea812ecc0 con 0x7f5ea80a1a00 2024-09-15T01:57:02.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.933+0000 7f5e967fc640 1 -- 172.21.15.67:0/1930952172 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v142) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5ea0017020 con 0x7f5ea80a1a00 2024-09-15T01:57:02.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.933+0000 7f5e967fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T01:57:02.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.933+0000 7f5e967fc640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-15T01:57:02.936 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 12 size to 2 2024-09-15T01:57:02.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 msgr2=0x7f5e8803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 0x7f5e8803abf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5e98004b50 tx=0x7f5e98005100 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 10 monclient: shutdown 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 msgr2=0x7f5ea813f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000b490 tx=0x7f5ea0009060 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 shutdown_connections 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e88038740 0x7f5e8803abf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ea809d5a0 0x7f5ea812dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea80a1a00 0x7f5ea813f0f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 --2- 172.21.15.67:0/1930952172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ea809b1f0 0x7f5ea809b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 >> 172.21.15.67:0/1930952172 conn(0x7f5ea8071340 msgr2=0x7f5ea8096a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 shutdown_connections 2024-09-15T01:57:02.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:02.936+0000 7f5eac95d640 1 -- 172.21.15.67:0/1930952172 wait complete. 2024-09-15T01:57:02.950 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T01:57:03.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:03.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 10 monclient: build_initial_monmap 2024-09-15T01:57:03.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 10 monclient: monmap: 2024-09-15T01:57:03.031 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:03.030264+0000 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:03.030264+0000 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:03.032 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 1 Processor -- start 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 1 -- start start 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.029+0000 7f6f5d15d640 10 monclient: init 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f58068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f58069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f58069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f58137150 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:03.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f58096c10 con 0x7f6f58068dc0 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f58132960 con 0x7f6f58137150 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580a14d0 con 0x7f6f58069190 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5d15d640 10 monclient(hunting): authenticate will time out at 1055.499296s 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 msgr2=0x7f6f58069560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f58069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5759e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6f58137150 0x7f6f58139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43292/0 (socket says 172.21.15.67:43292) 2024-09-15T01:57:03.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.030+0000 7f6f5759e640 1 -- 172.21.15.67:0/1682493658 learned_addr learned my addr 172.21.15.67:0/1682493658 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f5759e640 1 -- 172.21.15.67:0/1682493658 >> v1:172.21.15.67:6789/0 conn(0x7f6f58137150 legacy=0x7f6f58139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339200172 0 0) 0x7f6f58132960 con 0x7f6f58137150 2024-09-15T01:57:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f55d9b640 10 monclient(hunting): my global_id is 5923 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f440034e0 con 0x7f6f58137150 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.032+0000 7f6f56d9d640 1 -- 172.21.15.67:0/1682493658 >> v1:172.21.15.17:6789/0 conn(0x7f6f58068dc0 legacy=0x7f6f58132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673323832 0 0) 0x7f6f440034e0 con 0x7f6f58137150 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 msgr2=0x7f6f58069560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f58069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 >> v1:172.21.15.17:6789/0 conn(0x7f6f58068dc0 legacy=0x7f6f58132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5809ff20 con 0x7f6f58137150 2024-09-15T01:57:03.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: _finish_auth 0 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f6f50001720 con 0x7f6f58137150 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5d15d640 5 monclient: authenticate success, global_id 5923 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5d15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: dump: 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.044 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 >> v1:172.21.15.67:6789/0 conn(0x7f6f58137150 legacy=0x7f6f58139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:03.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 0x7f6f44003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): picked mon.b con 0x7f6f44003a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f44004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): picked mon.c con 0x7f6f58069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44004ec0 0x7f6f4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): picked mon.a con 0x7f6f44004ec0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f44005640 con 0x7f6f44004ec0 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4400bf10 con 0x7f6f44003a50 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4400c080 con 0x7f6f58069190 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f55d9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5d15d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 0x7f6f44003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5659c640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 msgr2=0x7f6f44004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5659c640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f44004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): get_auth_request con 0x7f6f44003a50 auth_method 0 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44004ec0 0x7f6f4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:03.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): get_auth_request con 0x7f6f44004ec0 auth_method 0 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient(hunting): handle_auth_done global_id 5923 payload 293 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 msgr2=0x7f6f44004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f58069190 0x7f6f44004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44004ec0 msgr2=0x7f6f4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44004ec0 0x7f6f4400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 monclient: found mon.b 2024-09-15T01:57:03.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 1 -- 172.21.15.67:0/1682493658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f440034e0 con 0x7f6f44003a50 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 10 monclient: _finish_auth 0 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.033+0000 7f6f56d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f56d9d640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 msgr2=0x7f6f44003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 0x7f6f44003e60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6f480021f0 tx=0x7f6f48001890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b582ea17f5a4ba50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48005b00 con 0x7f6f44003a50 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: dump: 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.049 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4802d070 con 0x7f6f44003a50 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f5d15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f5d15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 1 -- 172.21.15.67:0/1682493658 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48005cc0 con 0x7f6f44003a50 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f5d15d640 4 set_mon_vals no callback set 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.036+0000 7f6f55d9b640 10 monclient: dump: 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.051 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 10 monclient: shutdown 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 -- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 msgr2=0x7f6f44003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/1682493658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f44003a50 0x7f6f44003e60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6f480021f0 tx=0x7f6f48001890 comp rx=0 tx=0).stop 2024-09-15T01:57:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5759e640 1 -- 172.21.15.67:0/1682493658 reap_dead start 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 -- 172.21.15.67:0/1682493658 shutdown_connections 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 -- 172.21.15.67:0/1682493658 >> 172.21.15.67:0/1682493658 conn(0x7f6f580711b0 msgr2=0x7f6f580a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 -- 172.21.15.67:0/1682493658 shutdown_connections 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 1 -- 172.21.15.67:0/1682493658 wait complete. 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 10 monclient: build_initial_monmap 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.037+0000 7f6f5d15d640 10 monclient: monmap: 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 Processor -- start 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 -- start start 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient: init 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 0x7f6f58098b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f44003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f580990a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 0x7f6f581475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f580999b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5809abd0 con 0x7f6f44003a50 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580a3170 con 0x7f6f580990a0 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580a0890 con 0x7f6f580999b0 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5d15d640 10 monclient(hunting): authenticate will time out at 1055.507696s 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5759e640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 msgr2=0x7f6f581475f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5759e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 0x7f6f581475f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47546/0 (socket says 172.21.15.67:47546) 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 1 -- 172.21.15.67:0/3696508404 learned_addr learned my addr 172.21.15.67:0/3696508404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): get_auth_request con 0x7f6f580990a0 auth_method 0 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.038+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 0x7f6f58098b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f56d9d640 10 monclient(hunting): get_auth_request con 0x7f6f44003a50 auth_method 0 2024-09-15T01:57:03.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 10 monclient(hunting): handle_auth_done global_id 5926 payload 824 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 -- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 msgr2=0x7f6f581475f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 0x7f6f581475f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 -- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 msgr2=0x7f6f58098b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 0x7f6f58098b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 10 monclient: _finish_hunting 0 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 monclient: found mon.noname-b 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f5659c640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5809b2c0 con 0x7f6f580990a0 2024-09-15T01:57:03.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.039+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 0x7f6f58098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5659c640 1 -- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 msgr2=0x7f6f58099470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5659c640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b810 tx=0x7f6f40009270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40003e20 con 0x7f6f580990a0 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: dump: 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: _finish_auth 0 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f400050d0 con 0x7f6f580990a0 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40005490 con 0x7f6f580990a0 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 5 monclient: authenticate success, global_id 5926 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 10 monclient: _renew_subs 2024-09-15T01:57:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5811b830 con 0x7f6f580990a0 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f4effd640 10 monclient: dump: 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:03.062 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5559a640 4 set_mon_vals no callback set 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 10 monclient: _renew_subs 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.041+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f581308a0 con 0x7f6f580990a0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6f40012070 con 0x7f6f580990a0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f5d15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f5d15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f5d15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5804bf60 con 0x7f6f580990a0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f4effd640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 0x7f6f3803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 7899+0+0 (secure 0 0 0) 0x7f6f4004bcc0 con 0x7f6f580990a0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 0x7f6f3803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.042+0000 7f6f56d9d640 10 monclient: get_auth_request con 0x7f6f380386f0 auth_method 0 2024-09-15T01:57:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.045+0000 7f6f56d9d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 0x7f6f3803aba0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6f4802f000 tx=0x7f6f48033000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.047+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6f40012920 con 0x7f6f580990a0 2024-09-15T01:57:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.047+0000 7f6f4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.047+0000 7f6f4effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:03.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.219+0000 7f6f5d15d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:03.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.219+0000 7f6f5d15d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:03.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.219+0000 7f6f5d15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:03.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:03.219+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f6f581353f0 con 0x7f6f580990a0 2024-09-15T01:57:04.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.163+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v144) v1 ==== 82+0+0 (secure 0 0 0) 0x7f6f4001a020 con 0x7f6f580990a0 2024-09-15T01:57:04.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.163+0000 7f6f4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:04.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.163+0000 7f6f4effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:04.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.269+0000 7f6f5d15d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:04.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.269+0000 7f6f5d15d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:04.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.269+0000 7f6f5d15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.269+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f6f580008d0 con 0x7f6f580990a0 2024-09-15T01:57:04.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.270+0000 7f6f4effd640 1 -- 172.21.15.67:0/3696508404 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v144) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6f4001d350 con 0x7f6f580990a0 2024-09-15T01:57:04.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.270+0000 7f6f4effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:57:04.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.270+0000 7f6f4effd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T01:57:04.273 INFO:tasks.workunit.client.0.smithi067.stderr:compat weight-set already created 2024-09-15T01:57:04.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 msgr2=0x7f6f3803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:04.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 0x7f6f3803aba0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6f4802f000 tx=0x7f6f48033000 comp rx=0 tx=0).stop 2024-09-15T01:57:04.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 10 monclient: shutdown 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 msgr2=0x7f6f58099470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b810 tx=0x7f6f40009270 comp rx=0 tx=0).stop 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 shutdown_connections 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6f380386f0 0x7f6f3803aba0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6f580999b0 0x7f6f581475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f580990a0 0x7f6f58099470 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 --2- 172.21.15.67:0/3696508404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f44003a50 0x7f6f58098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 >> 172.21.15.67:0/3696508404 conn(0x7f6f580711b0 msgr2=0x7f6f5809ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 shutdown_connections 2024-09-15T01:57:04.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.272+0000 7f6f5d15d640 1 -- 172.21.15.67:0/3696508404 wait complete. 2024-09-15T01:57:04.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create cool flat 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.359+0000 7f898b75d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.359+0000 7f898b75d640 10 monclient: build_initial_monmap 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.359+0000 7f898b75d640 10 monclient: monmap: 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:04.360667+0000 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:04.360667+0000 2024-09-15T01:57:04.362 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 Processor -- start 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 -- start start 2024-09-15T01:57:04.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient: init 2024-09-15T01:57:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:57:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8984131e70 0x7f8984132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-c con 0x7f8984131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8984098420 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 0x7f8984139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-a con 0x7f89841371a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89840a13e0 con 0x7f8984098420 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840a0440 con 0x7f89841371a0 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840a01e0 con 0x7f8984131e70 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f898b75d640 10 monclient(hunting): authenticate will time out at 1056.829819s 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8984131e70 0x7f8984132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8984131e70 0x7f8984132240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47558/0 (socket says 172.21.15.67:47558) 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 1 -- 172.21.15.67:0/3758919877 learned_addr learned my addr 172.21.15.67:0/3758919877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 10 monclient(hunting): get_auth_request con 0x7f8984131e70 auth_method 0 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.360+0000 7f89894f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient(hunting): handle_auth_done global_id 5929 payload 274 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f8989cfa640 1 --2- 172.21.15.67:0/3758919877 >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 0x7f8984139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 1 -- 172.21.15.67:0/3758919877 >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 msgr2=0x7f8984139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 1 --2- 172.21.15.67:0/3758919877 >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 0x7f8984139580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 1 -- 172.21.15.67:0/3758919877 >> v1:172.21.15.17:6789/0 conn(0x7f8984098420 legacy=0x7f8984098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient: _finish_hunting 0 2024-09-15T01:57:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 1 monclient: found mon.noname-c 2024-09-15T01:57:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f89894f9640 1 -- 172.21.15.67:0/3758919877 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840a19b0 con 0x7f8984131e70 2024-09-15T01:57:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f8989cfa640 10 monclient: get_auth_request con 0x7f89841371a0 auth_method 0 2024-09-15T01:57:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.361+0000 7f8989cfa640 1 --2- 172.21.15.67:0/3758919877 >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 0x7f8984139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f89894f9640 1 -- 172.21.15.67:0/3758919877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984131e70 msgr2=0x7f8984132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f89894f9640 1 --2- 172.21.15.67:0/3758919877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984131e70 0x7f8984132240 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f89840a6250 tx=0x7f89800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc50ed7cd8ee6b7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 1 -- 172.21.15.67:0/3758919877 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980032440 con 0x7f8984131e70 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: dump: 2024-09-15T01:57:04.373 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: _finish_auth 0 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 5 monclient: authenticate success, global_id 5929 2024-09-15T01:57:04.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 1 -- 172.21.15.67:0/3758919877 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89800325a0 con 0x7f8984131e70 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 4 set_mon_vals no callback set 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 1 -- 172.21.15.67:0/3758919877 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980032940 con 0x7f8984131e70 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f897bfff640 10 monclient: dump: 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:04.375 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 10 monclient: shutdown 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 -- 172.21.15.67:0/3758919877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984131e70 msgr2=0x7f8984132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 --2- 172.21.15.67:0/3758919877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984131e70 0x7f8984132240 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f89840a6250 tx=0x7f89800091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 -- 172.21.15.67:0/3758919877 shutdown_connections 2024-09-15T01:57:04.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 --2- 172.21.15.67:0/3758919877 >> v2:172.21.15.17:3300/0 conn(0x7f89841371a0 0x7f8984139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 --2- 172.21.15.67:0/3758919877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984131e70 0x7f8984132240 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 -- 172.21.15.67:0/3758919877 >> 172.21.15.67:0/3758919877 conn(0x7f89840a5c10 msgr2=0x7f89840a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 -- 172.21.15.67:0/3758919877 shutdown_connections 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.364+0000 7f898b75d640 1 -- 172.21.15.67:0/3758919877 wait complete. 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.365+0000 7f898b75d640 10 monclient: build_initial_monmap 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.365+0000 7f898b75d640 10 monclient: monmap: 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:04.377 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.365+0000 7f898b75d640 1 Processor -- start 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.365+0000 7f898b75d640 1 -- start start 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.365+0000 7f898b75d640 10 monclient: init 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-b con 0x7f8984098420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 0x7f8984132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-c con 0x7f8984131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89841371a0 0x7f8984133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): picked mon.noname-a con 0x7f89841371a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:04.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840a0df0 con 0x7f89841371a0 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840a3040 con 0x7f8984098420 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89841364d0 con 0x7f8984131e70 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f898b75d640 10 monclient(hunting): authenticate will time out at 1056.835351s 2024-09-15T01:57:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f8988cf8640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 msgr2=0x7f8984132ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47566/0 (socket says 172.21.15.67:47566) 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f8988cf8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 0x7f8984132ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 -- 172.21.15.67:0/3171108387 learned_addr learned my addr 172.21.15.67:0/3171108387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): get_auth_request con 0x7f8984098420 auth_method 0 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient(hunting): handle_auth_done global_id 5932 payload 824 2024-09-15T01:57:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f8989cfa640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89841371a0 0x7f8984133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 -- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 msgr2=0x7f8984132ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 0x7f8984132ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 -- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89841371a0 msgr2=0x7f8984133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89841371a0 0x7f8984133430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient: _finish_hunting 0 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 monclient: found mon.noname-b 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.366+0000 7f89894f9640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8980009450 con 0x7f8984098420 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.368+0000 7f89894f9640 1 -- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 msgr2=0x7f898412ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89894f9640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8980035040 tx=0x7f8980032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89894f9640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8980035040 tx=0x7f8980032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8980031c70 mon_map magic: 0 v1 2024-09-15T01:57:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980031c70 con 0x7f8984098420 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 10 monclient: dump: 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.382 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 10 monclient: _finish_auth 0 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980031dd0 con 0x7f8984098420 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 5 monclient: authenticate success, global_id 5932 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f89797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 10 monclient: _renew_subs 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8984142430 con 0x7f8984098420 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f897b7fe640 4 set_mon_vals no callback set 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 10 monclient: _renew_subs 2024-09-15T01:57:04.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.369+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89841361f0 con 0x7f8984098420 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898003b650 con 0x7f8984098420 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 10 monclient: dump: 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:04.384 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8980039040 con 0x7f8984098420 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f89797fa640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 0x7f896803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f898b75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f898b75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f898b75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89841368b0 con 0x7f8984098420 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f8988cf8640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 0x7f896803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:04.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.370+0000 7f8988cf8640 10 monclient: get_auth_request con 0x7f8968038740 auth_method 0 2024-09-15T01:57:04.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.373+0000 7f8988cf8640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 0x7f896803abf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8974002ee0 tx=0x7f8974002aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:04.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.374+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 8231+0+0 (secure 0 0 0) 0x7f898006ce40 con 0x7f8984098420 2024-09-15T01:57:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.993+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f898003f080 con 0x7f8984098420 2024-09-15T01:57:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.993+0000 7f89797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:04.993+0000 7f89797fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:05.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:05.169+0000 7f898b75d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:05.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:05.169+0000 7f898b75d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:05.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:05.169+0000 7f898b75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:05.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:05.169+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f89840639e0 con 0x7f8984098420 2024-09-15T01:57:06.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.171+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v145) v1 ==== 107+0+0 (secure 0 0 0) 0x7f8980037070 con 0x7f8984098420 2024-09-15T01:57:06.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.171+0000 7f89797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:06.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.171+0000 7f89797fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:06.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.280+0000 7f898b75d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.280+0000 7f898b75d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.280+0000 7f898b75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.280+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f89840639e0 con 0x7f8984098420 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.281+0000 7f89797fa640 1 -- 172.21.15.67:0/3171108387 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v145) v1 ==== 149+0+0 (secure 0 0 0) 0x7f898003ba40 con 0x7f8984098420 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.281+0000 7f89797fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.281+0000 7f89797fa640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-15T01:57:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:weight-set for pool 'cool' already created 2024-09-15T01:57:06.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 msgr2=0x7f896803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 0x7f896803abf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8974002ee0 tx=0x7f8974002aa0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 10 monclient: shutdown 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 msgr2=0x7f898412ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8980035040 tx=0x7f8980032000 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 shutdown_connections 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8968038740 0x7f896803abf0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89841371a0 0x7f8984133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8984131e70 0x7f8984132ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 --2- 172.21.15.67:0/3171108387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8984098420 0x7f898412ea60 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 >> 172.21.15.67:0/3171108387 conn(0x7f89840a5c10 msgr2=0x7f8984068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:06.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 shutdown_connections 2024-09-15T01:57:06.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.284+0000 7f898b75d640 1 -- 172.21.15.67:0/3171108387 wait complete. 2024-09-15T01:57:06.296 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create cold positional 2024-09-15T01:57:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.372+0000 7f49b3f09640 10 monclient: get_monmap_and_config 2024-09-15T01:57:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.372+0000 7f49b3f09640 10 monclient: build_initial_monmap 2024-09-15T01:57:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.372+0000 7f49b3f09640 10 monclient: monmap: 2024-09-15T01:57:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:06.373870+0000 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:06.373870+0000 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:06.377 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 Processor -- start 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 -- start start 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient: init 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T01:57:06.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49ac0a0b50 0x7f49ac069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-c con 0x7f49ac0a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 0x7f49ac09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-b con 0x7f49ac09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49ac131b10 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac06a270 con 0x7f49ac131b10 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac06f290 con 0x7f49ac0a0b50 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac09f7e0 con 0x7f49ac09eea0 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.373+0000 7f49b3f09640 10 monclient(hunting): authenticate will time out at 1058.842971s 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b14a4640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 msgr2=0x7f49ac09f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b14a4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 0x7f49ac09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49ac0a0b50 0x7f49ac069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49ac0a0b50 0x7f49ac069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47570/0 (socket says 172.21.15.67:47570) 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3775877275 learned_addr learned my addr 172.21.15.67:0/3775877275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): get_auth_request con 0x7f49ac0a0b50 auth_method 0 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:06.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_done global_id 5935 payload 274 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3775877275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 msgr2=0x7f49ac09f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3775877275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 0x7f49ac09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3775877275 >> v1:172.21.15.17:6789/0 conn(0x7f49ac131b10 legacy=0x7f49ac131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient: _finish_hunting 0 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 monclient: found mon.noname-c 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.374+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3775877275 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0a0f20 con 0x7f49ac0a0b50 2024-09-15T01:57:06.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3775877275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac0a0b50 msgr2=0x7f49ac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:06.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3775877275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac0a0b50 0x7f49ac069340 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009910 tx=0x7f49a80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6569dbc57c1147dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:06.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 1 -- 172.21.15.67:0/3775877275 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a802ae30 con 0x7f49ac0a0b50 2024-09-15T01:57:06.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:06.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:06.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: dump: 2024-09-15T01:57:06.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:06.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: _finish_auth 0 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 1 -- 172.21.15.67:0/3775877275 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a802f4b0 con 0x7f49ac0a0b50 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 5 monclient: authenticate success, global_id 5935 2024-09-15T01:57:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 4 set_mon_vals no callback set 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 1 -- 172.21.15.67:0/3775877275 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a802f850 con 0x7f49ac0a0b50 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b0ca3640 10 monclient: dump: 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:06.394 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 10 monclient: shutdown 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 -- 172.21.15.67:0/3775877275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac0a0b50 msgr2=0x7f49ac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3775877275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac0a0b50 0x7f49ac069340 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009910 tx=0x7f49a80091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 -- 172.21.15.67:0/3775877275 shutdown_connections 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3775877275 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac09eea0 0x7f49ac09f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3775877275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac0a0b50 0x7f49ac069340 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.377+0000 7f49b3f09640 1 -- 172.21.15.67:0/3775877275 >> 172.21.15.67:0/3775877275 conn(0x7f49ac071290 msgr2=0x7f49ac0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 1 -- 172.21.15.67:0/3775877275 shutdown_connections 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 1 -- 172.21.15.67:0/3775877275 wait complete. 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 10 monclient: build_initial_monmap 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 10 monclient: monmap: 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:06.396 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 1 Processor -- start 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 1 -- start start 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.378+0000 7f49b3f09640 10 monclient: init 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-b con 0x7f49ac09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 0x7f49ac130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-a con 0x7f49ac0a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 0x7f49ac11b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): picked mon.noname-c con 0x7f49ac131b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac134ca0 con 0x7f49ac0a0b50 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac11e050 con 0x7f49ac09eea0 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac09b440 con 0x7f49ac131b10 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b3f09640 10 monclient(hunting): authenticate will time out at 1058.848278s 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47574/0 (socket says 172.21.15.67:47574) 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3079638521 learned_addr learned my addr 172.21.15.67:0/3079638521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): get_auth_request con 0x7f49ac09eea0 auth_method 0 2024-09-15T01:57:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b24a6640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 msgr2=0x7f49ac11b940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b24a6640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 0x7f49ac11b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient(hunting): handle_auth_done global_id 5938 payload 824 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 msgr2=0x7f49ac11b940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 0x7f49ac11b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b14a4640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 0x7f49ac130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 msgr2=0x7f49ac130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 0x7f49ac130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:06.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient: _finish_hunting 0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 monclient: found mon.noname-b 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8009450 con 0x7f49ac09eea0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b14a4640 10 monclient: get_auth_request con 0x7f49ac0a0b50 auth_method 0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.379+0000 7f49b14a4640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 0x7f49ac130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.381+0000 7f49b1ca5640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 msgr2=0x7f49ac09a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.381+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009970 tx=0x7f49a8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8032a90 con 0x7f49ac09eea0 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b1ca5640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009970 tx=0x7f49a8032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f49a8032bf0 config(0 keys) v1 2024-09-15T01:57:06.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 10 monclient: dump: 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:06.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 5 monclient: authenticate success, global_id 5938 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8032bf0 con 0x7f49ac09eea0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 10 monclient: _renew_subs 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac141fc0 con 0x7f49ac09eea0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49a3fff640 4 set_mon_vals no callback set 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 10 monclient: _renew_subs 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.382+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac11ce80 con 0x7f49ac09eea0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a803b400 con 0x7f49ac09eea0 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:06.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 10 monclient: dump: 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f49a8039040 con 0x7f49ac09eea0 2024-09-15T01:57:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b3f09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b3f09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b3f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac04bf60 con 0x7f49ac09eea0 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49a1ffb640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 0x7f499003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b14a4640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 0x7f499003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.383+0000 7f49b14a4640 10 monclient: get_auth_request con 0x7f49900386f0 auth_method 0 2024-09-15T01:57:06.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.386+0000 7f49b14a4640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 0x7f499003aba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f499c006fd0 tx=0x7f499c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:06.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:06.387+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 8563+0+0 (secure 0 0 0) 0x7f49a806cff0 con 0x7f49ac09eea0 2024-09-15T01:57:07.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.005+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f49a803b830 con 0x7f49ac09eea0 2024-09-15T01:57:07.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.005+0000 7f49a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:07.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.005+0000 7f49a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:07.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.181+0000 7f49b3f09640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:07.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.181+0000 7f49b3f09640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:07.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.181+0000 7f49b3f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:07.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:07.181+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f49ac06f000 con 0x7f49ac09eea0 2024-09-15T01:57:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.176+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v146) v1 ==== 113+0+0 (secure 0 0 0) 0x7f49a803f020 con 0x7f49ac09eea0 2024-09-15T01:57:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.176+0000 7f49a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:08.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.176+0000 7f49a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:08.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.284+0000 7f49b3f09640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:08.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.284+0000 7f49b3f09640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:08.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.284+0000 7f49b3f09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.284+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f49ac06f000 con 0x7f49ac09eea0 2024-09-15T01:57:08.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.285+0000 7f49a1ffb640 1 -- 172.21.15.67:0/3079638521 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v146) v1 ==== 155+0+0 (secure 0 0 0) 0x7f49a803f030 con 0x7f49ac09eea0 2024-09-15T01:57:08.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.285+0000 7f49a1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T01:57:08.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.285+0000 7f49a1ffb640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-15T01:57:08.288 INFO:tasks.workunit.client.0.smithi067.stderr:weight-set for pool 'cold' already created 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 msgr2=0x7f499003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 0x7f499003aba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f499c006fd0 tx=0x7f499c008040 comp rx=0 tx=0).stop 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 10 monclient: shutdown 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 msgr2=0x7f49ac09a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009970 tx=0x7f49a8032000 comp rx=0 tx=0).stop 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 shutdown_connections 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49900386f0 0x7f499003aba0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f49ac0a0b50 0x7f49ac130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f49ac131b10 0x7f49ac11b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 --2- 172.21.15.67:0/3079638521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ac09eea0 0x7f49ac09a840 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 >> 172.21.15.67:0/3079638521 conn(0x7f49ac071290 msgr2=0x7f49ac134eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 shutdown_connections 2024-09-15T01:57:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.287+0000 7f49b3f09640 1 -- 172.21.15.67:0/3079638521 wait complete. 2024-09-15T01:57:08.299 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T01:57:08.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.374+0000 7f559699d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:08.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.374+0000 7f559699d640 10 monclient: build_initial_monmap 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.374+0000 7f559699d640 10 monclient: monmap: 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:08.375463+0000 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:08.375463+0000 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:08.377 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 Processor -- start 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 -- start start 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient: init 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f559009d150 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5590132900 0x7f5590134ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): picked mon.noname-a con 0x7f5590132900 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5590135220 0x7f5590137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): picked mon.noname-c con 0x7f5590135220 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559009b7f0 con 0x7f559009d150 2024-09-15T01:57:08.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559009f590 con 0x7f5590132900 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559009cd40 con 0x7f5590135220 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f559699d640 10 monclient(hunting): authenticate will time out at 1060.844712s 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5590135220 0x7f5590137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5590135220 0x7f5590137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47580/0 (socket says 172.21.15.67:47580) 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 1 -- 172.21.15.67:0/773797831 learned_addr learned my addr 172.21.15.67:0/773797831 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:08.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): get_auth_request con 0x7f5590135220 auth_method 0 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.375+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f558f7fe640 1 --2- 172.21.15.67:0/773797831 >> v2:172.21.15.17:3300/0 conn(0x7f5590132900 0x7f5590134ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 10 monclient(hunting): handle_auth_done global_id 5941 payload 274 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 1 -- 172.21.15.67:0/773797831 >> v2:172.21.15.17:3300/0 conn(0x7f5590132900 msgr2=0x7f5590134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 1 --2- 172.21.15.67:0/773797831 >> v2:172.21.15.17:3300/0 conn(0x7f5590132900 0x7f5590134ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 1 -- 172.21.15.67:0/773797831 >> v1:172.21.15.17:6789/0 conn(0x7f559009d150 legacy=0x7f559009c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 10 monclient: _finish_hunting 0 2024-09-15T01:57:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 1 monclient: found mon.noname-c 2024-09-15T01:57:08.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:08.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.376+0000 7f5594f3a640 1 -- 172.21.15.67:0/773797831 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559009bad0 con 0x7f5590135220 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.378+0000 7f5594f3a640 1 -- 172.21.15.67:0/773797831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590135220 msgr2=0x7f5590137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.378+0000 7f5594f3a640 1 --2- 172.21.15.67:0/773797831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590135220 0x7f5590137600 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f559009f220 tx=0x7f5588009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87091148ac796e83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 1 -- 172.21.15.67:0/773797831 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5588007610 con 0x7f5590135220 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:08.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: dump: 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: _finish_auth 0 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:08.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 1 -- 172.21.15.67:0/773797831 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558802d050 con 0x7f5590135220 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 5 monclient: authenticate success, global_id 5941 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 4 set_mon_vals no callback set 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 1 -- 172.21.15.67:0/773797831 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55880079a0 con 0x7f5590135220 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f558effd640 10 monclient: dump: 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.390 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 10 monclient: shutdown 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 -- 172.21.15.67:0/773797831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590135220 msgr2=0x7f5590137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 --2- 172.21.15.67:0/773797831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590135220 0x7f5590137600 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f559009f220 tx=0x7f5588009840 comp rx=0 tx=0).stop 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 -- 172.21.15.67:0/773797831 shutdown_connections 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 --2- 172.21.15.67:0/773797831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590135220 0x7f5590137600 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 --2- 172.21.15.67:0/773797831 >> v2:172.21.15.17:3300/0 conn(0x7f5590132900 0x7f5590134ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 -- 172.21.15.67:0/773797831 >> 172.21.15.67:0/773797831 conn(0x7f5590070ff0 msgr2=0x7f55900a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 -- 172.21.15.67:0/773797831 shutdown_connections 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.379+0000 7f559699d640 1 -- 172.21.15.67:0/773797831 wait complete. 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.380+0000 7f559699d640 10 monclient: build_initial_monmap 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.380+0000 7f559699d640 10 monclient: monmap: 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:08.392 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.380+0000 7f559699d640 1 Processor -- start 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.380+0000 7f559699d640 1 -- start start 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.380+0000 7f559699d640 10 monclient: init 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 0x7f559011a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): picked mon.noname-c con 0x7f559009d150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): picked mon.noname-b con 0x7f5590132900 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5590135220 0x7f559011abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): picked mon.noname-a con 0x7f5590135220 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:08.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559011d4f0 con 0x7f5590135220 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559011bfb0 con 0x7f5590132900 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5590099ee0 con 0x7f559009d150 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f559699d640 10 monclient(hunting): authenticate will time out at 1060.850162s 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47586/0 (socket says 172.21.15.67:47586) 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 -- 172.21.15.67:0/300125670 learned_addr learned my addr 172.21.15.67:0/300125670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): get_auth_request con 0x7f5590132900 auth_method 0 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:08.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558ffff640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 msgr2=0x7f559011a160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558ffff640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 0x7f559011a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5590135220 0x7f559011abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): get_auth_request con 0x7f5590135220 auth_method 0 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:08.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f5594f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient(hunting): handle_auth_done global_id 5944 payload 824 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 msgr2=0x7f559011a160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 0x7f559011a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5590135220 msgr2=0x7f559011abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5590135220 0x7f559011abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 monclient: found mon.noname-b 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.381+0000 7f558f7fe640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588009ae0 con 0x7f5590132900 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.383+0000 7f558f7fe640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 msgr2=0x7f559011a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:08.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.383+0000 7f558f7fe640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f558400c880 tx=0x7f5584008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558400ce10 con 0x7f5590132900 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: dump: 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:08.397 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55840079c0 con 0x7f5590132900 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584007d60 con 0x7f5590132900 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 5 monclient: authenticate success, global_id 5944 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 10 monclient: dump: 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:08.398 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 10 monclient: _renew_subs 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f558e7fc640 4 set_mon_vals no callback set 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5590133fa0 con 0x7f5590132900 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 10 monclient: _renew_subs 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55901325d0 con 0x7f5590132900 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.384+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5584011600 con 0x7f5590132900 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f559699d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f559699d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f559699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559004bf60 con 0x7f5590132900 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f5577fff640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 0x7f557c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 9035+0+0 (secure 0 0 0) 0x7f558404c930 con 0x7f5590132900 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f558ffff640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 0x7f557c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:08.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.385+0000 7f558ffff640 10 monclient: get_auth_request con 0x7f557c038740 auth_method 0 2024-09-15T01:57:08.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.387+0000 7f558ffff640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 0x7f557c03abf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5580006fd0 tx=0x7f5580006cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:08.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.387+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:57:08.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.389+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5584020ae0 con 0x7f5590132900 2024-09-15T01:57:08.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.389+0000 7f5577fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:08.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.389+0000 7f5577fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:08.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.558+0000 7f559699d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.558+0000 7f559699d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.558+0000 7f559699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:08.558+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f559009f980 con 0x7f5590132900 2024-09-15T01:57:09.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.177+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v147) v1 ==== 123+0+0 (secure 0 0 0) 0x7f5584020300 con 0x7f5590132900 2024-09-15T01:57:09.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.177+0000 7f5577fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:09.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.177+0000 7f5577fff640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T01:57:09.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.283+0000 7f559699d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:09.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.283+0000 7f559699d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:09.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.283+0000 7f559699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:09.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:09.283+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f559009a640 con 0x7f5590132900 2024-09-15T01:57:10.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.181+0000 7f5577fff640 1 -- 172.21.15.67:0/300125670 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v148) v1 ==== 149+0+0 (secure 0 0 0) 0x7f55840204e0 con 0x7f5590132900 2024-09-15T01:57:10.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.181+0000 7f5577fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:10.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.181+0000 7f5577fff640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:57:10.184 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:57:10.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 msgr2=0x7f557c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 0x7f557c03abf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5580006fd0 tx=0x7f5580006cf0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 10 monclient: shutdown 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 msgr2=0x7f559011a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f558400c880 tx=0x7f5584008040 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 shutdown_connections 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f557c038740 0x7f557c03abf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5590135220 0x7f559011abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5590132900 0x7f559011a6a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 --2- 172.21.15.67:0/300125670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f559009d150 0x7f559011a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 >> 172.21.15.67:0/300125670 conn(0x7f5590070ff0 msgr2=0x7f55900a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 shutdown_connections 2024-09-15T01:57:10.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.184+0000 7f559699d640 1 -- 172.21.15.67:0/300125670 wait complete. 2024-09-15T01:57:10.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-15T01:57:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 10 monclient: build_initial_monmap 2024-09-15T01:57:10.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 10 monclient: monmap: 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:10.281466+0000 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:10.281466+0000 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:10.283 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 1 Processor -- start 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 1 -- start start 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.280+0000 7f543459d640 10 monclient: init 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f542c098140 0x7f542c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): picked mon.noname-a con 0x7f542c098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f542c0a0370 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 0x7f542c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): picked mon.noname-b con 0x7f542c098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0a0b80 con 0x7f542c0a0370 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:10.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c0a1730 con 0x7f542c098140 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0a00b0 con 0x7f542c098c20 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f543459d640 10 monclient(hunting): authenticate will time out at 1062.750653s 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432b3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 msgr2=0x7f542c1390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432b3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 0x7f542c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5431b38640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f542c0a0370 0x7f542c098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43302/0 (socket says 172.21.15.67:43302) 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5431b38640 1 -- 172.21.15.67:0/1343479830 learned_addr learned my addr 172.21.15.67:0/1343479830 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432339640 1 --2- 172.21.15.67:0/1343479830 >> v2:172.21.15.17:3300/0 conn(0x7f542c098140 0x7f542c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432339640 10 monclient(hunting): get_auth_request con 0x7f542c098140 auth_method 0 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:10.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.281+0000 7f5432339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient(hunting): handle_auth_done global_id 84261 payload 274 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 1 -- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 msgr2=0x7f542c1390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 1 --2- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 0x7f542c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 1 -- 172.21.15.67:0/1343479830 >> v1:172.21.15.67:6789/0 conn(0x7f542c0a0370 legacy=0x7f542c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient: _finish_hunting 0 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 1 monclient: found mon.noname-a 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:10.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.282+0000 7f5432339640 1 -- 172.21.15.67:0/1343479830 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c071390 con 0x7f542c098140 2024-09-15T01:57:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5432339640 1 -- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098140 msgr2=0x7f542c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5432339640 1 --2- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098140 0x7f542c1321e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5428009850 tx=0x7f54280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a856c37b0f3b61c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 1 -- 172.21.15.67:0/1343479830 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428032440 con 0x7f542c098140 2024-09-15T01:57:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: dump: 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: _finish_auth 0 2024-09-15T01:57:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 1 -- 172.21.15.67:0/1343479830 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54280325a0 con 0x7f542c098140 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 1 -- 172.21.15.67:0/1343479830 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428032940 con 0x7f542c098140 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f5431337640 10 monclient: dump: 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.296 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f543459d640 5 monclient: authenticate success, global_id 84261 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f543459d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.285+0000 7f543459d640 4 set_mon_vals no callback set 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 10 monclient: shutdown 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 -- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098140 msgr2=0x7f542c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 --2- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098140 0x7f542c1321e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5428009850 tx=0x7f54280090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 -- 172.21.15.67:0/1343479830 shutdown_connections 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 --2- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098c20 0x7f542c1390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 --2- 172.21.15.67:0/1343479830 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098140 0x7f542c1321e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 -- 172.21.15.67:0/1343479830 >> 172.21.15.67:0/1343479830 conn(0x7f542c0a5780 msgr2=0x7f542c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 -- 172.21.15.67:0/1343479830 shutdown_connections 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 -- 172.21.15.67:0/1343479830 wait complete. 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 10 monclient: build_initial_monmap 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 10 monclient: monmap: 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:10.298 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.286+0000 7f543459d640 1 Processor -- start 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 -- start start 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient: init 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 0x7f542c132e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): picked mon.noname-c con 0x7f542c098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098c20 0x7f542c133340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): picked mon.noname-a con 0x7f542c098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): picked mon.noname-b con 0x7f542c0a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c099090 con 0x7f542c098c20 2024-09-15T01:57:10.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0a3b60 con 0x7f542c0a0370 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c135200 con 0x7f542c098140 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f543459d640 10 monclient(hunting): authenticate will time out at 1062.756488s 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432339640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 msgr2=0x7f542c132e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 0x7f542c132e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47602/0 (socket says 172.21.15.67:47602) 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 -- 172.21.15.67:0/3265122641 learned_addr learned my addr 172.21.15.67:0/3265122641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): get_auth_request con 0x7f542c0a0370 auth_method 0 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:10.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098c20 0x7f542c133340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): get_auth_request con 0x7f542c098c20 auth_method 0 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5431b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:10.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient(hunting): handle_auth_done global_id 5947 payload 824 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 -- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 msgr2=0x7f542c132e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 0x7f542c132e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 -- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098c20 msgr2=0x7f542c133340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098c20 0x7f542c133340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 1 monclient: found mon.noname-b 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.287+0000 7f5432b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.288+0000 7f5432b3a640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428009390 con 0x7f542c0a0370 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f5432b3a640 1 -- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 msgr2=0x7f542c133880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f5432b3a640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f542400bb30 tx=0x7f54240097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542400cbc0 con 0x7f542c0a0370 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: dump: 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:10.303 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 5 monclient: authenticate success, global_id 5947 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542400cd20 con 0x7f542c0a0370 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 10 monclient: _renew_subs 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424006150 con 0x7f542c0a0370 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c131980 con 0x7f542c0a0370 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f54227fc640 10 monclient: dump: 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f5430b36640 4 set_mon_vals no callback set 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 10 monclient: _renew_subs 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.290+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c136080 con 0x7f542c0a0370 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f543459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f543459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f543459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5424012020 con 0x7f542c0a0370 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f54227fc640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 0x7f541003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.291+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 9003+0+0 (secure 0 0 0) 0x7f542404ca30 con 0x7f542c0a0370 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.292+0000 7f5432339640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 0x7f541003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.292+0000 7f5432339640 10 monclient: get_auth_request con 0x7f54100386f0 auth_method 0 2024-09-15T01:57:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.294+0000 7f5432339640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 0x7f541003aba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f542c0717a0 tx=0x7f5428032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.294+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:57:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.296+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.296+0000 7f54227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.296+0000 7f54227fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.470+0000 7f543459d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.470+0000 7f543459d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.470+0000 7f543459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:10.470+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:11.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.381+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v149) v1 ==== 119+0+0 (secure 0 0 0) 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:11.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.382+0000 7f54227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:11.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.382+0000 7f54227fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:11.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.496+0000 7f543459d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:11.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.496+0000 7f543459d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:11.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.496+0000 7f543459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:11.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:11.496+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:12.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.382+0000 7f54227fc640 1 -- 172.21.15.67:0/3265122641 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v150) v1 ==== 119+0+0 (secure 0 0 0) 0x7f542c1353b0 con 0x7f542c0a0370 2024-09-15T01:57:12.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.382+0000 7f54227fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T01:57:12.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.382+0000 7f54227fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:12.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 msgr2=0x7f541003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 0x7f541003aba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f542c0717a0 tx=0x7f5428032000 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 10 monclient: shutdown 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 msgr2=0x7f542c133880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.388+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f542400bb30 tx=0x7f54240097a0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 shutdown_connections 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f54100386f0 0x7f541003aba0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f542c098c20 0x7f542c133340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f542c0a0370 0x7f542c133880 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 --2- 172.21.15.67:0/3265122641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f542c098140 0x7f542c132e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 >> 172.21.15.67:0/3265122641 conn(0x7f542c0a5780 msgr2=0x7f542c09b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:12.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 shutdown_connections 2024-09-15T01:57:12.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.389+0000 7f543459d640 1 -- 172.21.15.67:0/3265122641 wait complete. 2024-09-15T01:57:12.406 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.488+0000 7f3ba255d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.488+0000 7f3ba255d640 10 monclient: build_initial_monmap 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.488+0000 7f3ba255d640 10 monclient: monmap: 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:12.489836+0000 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:12.489836+0000 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:12.491 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:12.492 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:12.492 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:12.492 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:12.492 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.489+0000 7f3ba255d640 1 Processor -- start 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.489+0000 7f3ba255d640 1 -- start start 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.489+0000 7f3ba255d640 10 monclient: init 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b9c0a3850 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9c06a6e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b9c137240 0x7f3b9c139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9c137240 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c09f9a0 con 0x7f3b9c0a3850 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c069c20 con 0x7f3b9c137240 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0696c0 con 0x7f3b9c06a6e0 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba255d640 10 monclient(hunting): authenticate will time out at 1064.959134s 2024-09-15T01:57:12.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b9c137240 0x7f3b9c139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3b9b7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 msgr2=0x7f3b9c136d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3b9bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b9c0a3850 0x7f3b9c06e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59480/0 (socket says 172.21.15.67:59480) 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3b9bfff640 1 -- 172.21.15.67:0/778746723 learned_addr learned my addr 172.21.15.67:0/778746723 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3b9b7fe640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c136d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): get_auth_request con 0x7f3b9c137240 auth_method 0 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:12.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_done global_id 5950 payload 274 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 -- 172.21.15.67:0/778746723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 msgr2=0x7f3b9c136d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c136d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 -- 172.21.15.67:0/778746723 >> v1:172.21.15.67:6789/0 conn(0x7f3b9c0a3850 legacy=0x7f3b9c06e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient: _finish_hunting 0 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 monclient: found mon.noname-c 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.490+0000 7f3ba0afa640 1 -- 172.21.15.67:0/778746723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0a3c20 con 0x7f3b9c137240 2024-09-15T01:57:12.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba0afa640 1 -- 172.21.15.67:0/778746723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 msgr2=0x7f3b9c139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:12.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba0afa640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c139620 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009e60 tx=0x7f3b94009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=298a3f74406f6170 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 1 -- 172.21.15.67:0/778746723 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94007840 con 0x7f3b9c137240 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: dump: 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:12.503 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: _finish_auth 0 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 1 -- 172.21.15.67:0/778746723 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9402d050 con 0x7f3b9c137240 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba255d640 5 monclient: authenticate success, global_id 5950 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba255d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba255d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 1 -- 172.21.15.67:0/778746723 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94007c10 con 0x7f3b9c137240 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3ba255d640 4 set_mon_vals no callback set 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.493+0000 7f3b9affd640 10 monclient: dump: 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:12.504 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 10 monclient: shutdown 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 -- 172.21.15.67:0/778746723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 msgr2=0x7f3b9c139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c139620 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009e60 tx=0x7f3b94009780 comp rx=0 tx=0).stop 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 -- 172.21.15.67:0/778746723 shutdown_connections 2024-09-15T01:57:12.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c139620 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 --2- 172.21.15.67:0/778746723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 -- 172.21.15.67:0/778746723 >> 172.21.15.67:0/778746723 conn(0x7f3b9c070ff0 msgr2=0x7f3b9c0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 -- 172.21.15.67:0/778746723 shutdown_connections 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 1 -- 172.21.15.67:0/778746723 wait complete. 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 10 monclient: build_initial_monmap 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.494+0000 7f3ba255d640 10 monclient: monmap: 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.506 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 Processor -- start 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 -- start start 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient: init 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c11e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9c06a6e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 0x7f3b9c11eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9c0a3850 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9c137240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c06e9e0 con 0x7f3b9c0a3850 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0a23f0 con 0x7f3b9c137240 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c121f10 con 0x7f3b9c06a6e0 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba255d640 10 monclient(hunting): authenticate will time out at 1064.964734s 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3b9bfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 msgr2=0x7f3b9c11e5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:12.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3b9bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c11e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47692/0 (socket says 172.21.15.67:47692) 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 1 -- 172.21.15.67:0/1131979690 learned_addr learned my addr 172.21.15.67:0/1131979690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): get_auth_request con 0x7f3b9c137240 auth_method 0 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.495+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 0x7f3b9c11eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:12.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 10 monclient(hunting): get_auth_request con 0x7f3b9c0a3850 auth_method 0 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 10 monclient(hunting): handle_auth_done global_id 5953 payload 824 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 -- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 msgr2=0x7f3b9c11e5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c11e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 -- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 msgr2=0x7f3b9c11eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 0x7f3b9c11eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 10 monclient: _finish_hunting 0 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 monclient: found mon.noname-b 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3ba0afa640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94009a20 con 0x7f3b9c137240 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.496+0000 7f3b9b7fe640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 0x7f3b9c11eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba0afa640 1 -- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 msgr2=0x7f3b9c11f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba0afa640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009e60 tx=0x7f3b94037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94031cd0 con 0x7f3b9c137240 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: dump: 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.511 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94040070 con 0x7f3b9c137240 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba255d640 5 monclient: authenticate success, global_id 5953 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba255d640 10 monclient: _renew_subs 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba255d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c1229d0 con 0x7f3b9c137240 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:12.512 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94045640 con 0x7f3b9c137240 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b83fff640 10 monclient: dump: 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:12.513 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 10 monclient: _renew_subs 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c120d40 con 0x7f3b9c137240 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.498+0000 7f3b9a7fc640 4 set_mon_vals no callback set 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3b94003020 con 0x7f3b9c137240 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.499+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c04bf60 con 0x7f3b9c137240 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.500+0000 7f3b83fff640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 0x7f3b8803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.500+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f3b9402ae40 con 0x7f3b9c137240 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.500+0000 7f3b9bfff640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 0x7f3b8803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:12.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.500+0000 7f3b9bfff640 10 monclient: get_auth_request con 0x7f3b88038720 auth_method 0 2024-09-15T01:57:12.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.502+0000 7f3b9bfff640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 0x7f3b8803abd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0047d0 tx=0x7f3b8c007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:12.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.504+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3b94031e30 con 0x7f3b9c137240 2024-09-15T01:57:12.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.504+0000 7f3b83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:12.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.504+0000 7f3b83fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.675+0000 7f3ba255d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.676+0000 7f3ba255d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.676+0000 7f3ba255d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:12.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:12.676+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f3b9c120f30 con 0x7f3b9c137240 2024-09-15T01:57:13.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.159+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v151) v1 ==== 128+0+0 (secure 0 0 0) 0x7f3b940393c0 con 0x7f3b9c137240 2024-09-15T01:57:13.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.159+0000 7f3b83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:13.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.159+0000 7f3b83fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:13.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.268+0000 7f3ba255d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:13.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.268+0000 7f3ba255d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:13.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.268+0000 7f3ba255d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:13.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:13.268+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f3b9c120f30 con 0x7f3b9c137240 2024-09-15T01:57:14.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.166+0000 7f3b83fff640 1 -- 172.21.15.67:0/1131979690 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v152) v1 ==== 128+0+0 (secure 0 0 0) 0x7f3b9403a3a0 con 0x7f3b9c137240 2024-09-15T01:57:14.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.167+0000 7f3b83fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T01:57:14.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.167+0000 7f3b83fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 msgr2=0x7f3b8803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 0x7f3b8803abd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0047d0 tx=0x7f3b8c007b80 comp rx=0 tx=0).stop 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 10 monclient: shutdown 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 msgr2=0x7f3b9c11f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009e60 tx=0x7f3b94037000 comp rx=0 tx=0).stop 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 shutdown_connections 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b88038720 0x7f3b8803abd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b9c0a3850 0x7f3b9c11eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b9c137240 0x7f3b9c11f060 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 --2- 172.21.15.67:0/1131979690 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b9c06a6e0 0x7f3b9c11e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 >> 172.21.15.67:0/1131979690 conn(0x7f3b9c070ff0 msgr2=0x7f3b9c09fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:14.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 shutdown_connections 2024-09-15T01:57:14.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.170+0000 7f3ba255d640 1 -- 172.21.15.67:0/1131979690 wait complete. 2024-09-15T01:57:14.184 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: get_monmap_and_config 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: build_initial_monmap 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: monmap: 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:14.268882+0000 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:14.268882+0000 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:14.270 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:14.271 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 1 Processor -- start 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 1 -- start start 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: init 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0ac0a3830 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb0ac06b9b0 0x7fb0ac136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac06b9b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0ac137240 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.268+0000 7fb0b144f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac069c60 con 0x7fb0ac0a3830 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0b144f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0a3c20 con 0x7fb0ac137240 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0b144f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0b144f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac069650 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0b144f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0b144f640 10 monclient(hunting): authenticate will time out at 1066.738028s 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0ab7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb0ac137240 0x7fb0ac139620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59496/0 (socket says 172.21.15.67:59496) 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/2078033824 learned_addr learned my addr 172.21.15.67:0/2078033824 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 1 --2- 172.21.15.67:0/2078033824 >> v2:172.21.15.17:3300/0 conn(0x7fb0ac06b9b0 0x7fb0ac136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:14.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): get_auth_request con 0x7fb0ac06b9b0 auth_method 0 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient(hunting): handle_auth_done global_id 84270 payload 274 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 1 -- 172.21.15.67:0/2078033824 >> v1:172.21.15.67:6789/0 conn(0x7fb0ac137240 legacy=0x7fb0ac139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 1 -- 172.21.15.67:0/2078033824 >> v1:172.21.15.17:6789/0 conn(0x7fb0ac0a3830 legacy=0x7fb0ac06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 1 monclient: found mon.noname-a 2024-09-15T01:57:14.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:14.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:14.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.269+0000 7fb0aa7fc640 1 -- 172.21.15.67:0/2078033824 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac09f500 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.272+0000 7fb0aa7fc640 1 -- 172.21.15.67:0/2078033824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 msgr2=0x7fb0ac136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:14.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.272+0000 7fb0aa7fc640 1 --2- 172.21.15.67:0/2078033824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac136d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009850 tx=0x7fb0a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a303fc10293f65e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:14.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.272+0000 7fb0a9ffb640 1 -- 172.21.15.67:0/2078033824 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0032440 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: dump: 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:14.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 1 -- 172.21.15.67:0/2078033824 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a00325a0 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 5 monclient: authenticate success, global_id 84270 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 1 -- 172.21.15.67:0/2078033824 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0032940 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0a9ffb640 10 monclient: dump: 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 4 set_mon_vals no callback set 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 10 monclient: shutdown 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 -- 172.21.15.67:0/2078033824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 msgr2=0x7fb0ac136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 --2- 172.21.15.67:0/2078033824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac136d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009850 tx=0x7fb0a00090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 -- 172.21.15.67:0/2078033824 shutdown_connections 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 --2- 172.21.15.67:0/2078033824 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac136d00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 -- 172.21.15.67:0/2078033824 >> 172.21.15.67:0/2078033824 conn(0x7fb0ac070ff0 msgr2=0x7fb0ac0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 -- 172.21.15.67:0/2078033824 shutdown_connections 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.273+0000 7fb0b144f640 1 -- 172.21.15.67:0/2078033824 wait complete. 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.274+0000 7fb0b144f640 10 monclient: build_initial_monmap 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.274+0000 7fb0b144f640 10 monclient: monmap: 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:14.286 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.274+0000 7fb0b144f640 1 Processor -- start 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.274+0000 7fb0b144f640 1 -- start start 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.274+0000 7fb0b144f640 10 monclient: init 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac06b9b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 0x7fb0ac097f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ac0a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:14.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac137240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac099e80 con 0x7fb0ac06b9b0 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac13a640 con 0x7fb0ac137240 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac11efd0 con 0x7fb0ac0a3830 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0b144f640 10 monclient(hunting): authenticate will time out at 1066.744347s 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aa7fc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 msgr2=0x7fb0ac097f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aa7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 0x7fb0ac097f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:14.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47708/0 (socket says 172.21.15.67:47708) 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/773290937 learned_addr learned my addr 172.21.15.67:0/773290937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb0ac137240 auth_method 0 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aaffd640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aaffd640 10 monclient(hunting): get_auth_request con 0x7fb0ac06b9b0 auth_method 0 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:14.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient(hunting): handle_auth_done global_id 5956 payload 824 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 msgr2=0x7fb0ac097f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 0x7fb0ac097f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 msgr2=0x7fb0ac097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 monclient: found mon.noname-b 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.275+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0009390 con 0x7fb0ac137240 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0ab7fe640 1 -- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 msgr2=0x7fb0ac0984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0ab7fe640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400ccc0 tx=0x7fb0a4005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:14.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400ce80 con 0x7fb0ac137240 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: dump: 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:14.291 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a400aa60 con 0x7fb0ac137240 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 5 monclient: authenticate success, global_id 5956 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400ae00 con 0x7fb0ac137240 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 10 monclient: _renew_subs 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:14.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac099190 con 0x7fb0ac137240 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb092ffd640 10 monclient: dump: 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0a97fa640 4 set_mon_vals no callback set 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 10 monclient: _renew_subs 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.278+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac11dd90 con 0x7fb0ac137240 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb0a4007300 con 0x7fb0ac137240 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0b144f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0b144f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0b144f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac11f190 con 0x7fb0ac137240 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb092ffd640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 0x7fb09803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fb0a404ce60 con 0x7fb0ac137240 2024-09-15T01:57:14.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0aaffd640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 0x7fb09803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.279+0000 7fb0aaffd640 10 monclient: get_auth_request con 0x7fb0980386f0 auth_method 0 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.282+0000 7fb0aaffd640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 0x7fb09803aba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0037d0 tx=0x7fb09c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.282+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.284+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb0a40075b0 con 0x7fb0ac137240 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.284+0000 7fb092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:14.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.284+0000 7fb092ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:14.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.459+0000 7fb0b144f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:14.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.459+0000 7fb0b144f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:14.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.459+0000 7fb0b144f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:14.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:14.459+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7fb0ac04bf60 con 0x7fb0ac137240 2024-09-15T01:57:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.405+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v153) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb0a4010300 con 0x7fb0ac137240 2024-09-15T01:57:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.405+0000 7fb092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:15.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.405+0000 7fb092ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:15.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.515+0000 7fb0b144f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:15.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.515+0000 7fb0b144f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:15.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.515+0000 7fb0b144f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:15.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:15.515+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7fb0ac068a20 con 0x7fb0ac137240 2024-09-15T01:57:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.404+0000 7fb092ffd640 1 -- 172.21.15.67:0/773290937 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v154) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb0a4029070 con 0x7fb0ac137240 2024-09-15T01:57:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.405+0000 7fb092ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T01:57:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.405+0000 7fb092ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.407+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 msgr2=0x7fb09803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.407+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 0x7fb09803aba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0037d0 tx=0x7fb09c009040 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.407+0000 7fb0b144f640 10 monclient: shutdown 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.407+0000 7fb0b144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 msgr2=0x7fb0ac0984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400ccc0 tx=0x7fb0a4005370 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 shutdown_connections 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb0980386f0 0x7fb09803aba0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0ac137240 0x7fb0ac0984c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0ac0a3830 0x7fb0ac097f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 --2- 172.21.15.67:0/773290937 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0ac06b9b0 0x7fb0ac097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 >> 172.21.15.67:0/773290937 conn(0x7fb0ac070ff0 msgr2=0x7fb0ac139620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 shutdown_connections 2024-09-15T01:57:16.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.408+0000 7fb0b144f640 1 -- 172.21.15.67:0/773290937 wait complete. 2024-09-15T01:57:16.422 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.505+0000 7fe94715d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.505+0000 7fe94715d640 10 monclient: build_initial_monmap 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.505+0000 7fe94715d640 10 monclient: monmap: 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:16.505918+0000 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:16.505918+0000 2024-09-15T01:57:16.507 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:16.508 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.506+0000 7fe94715d640 1 Processor -- start 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.506+0000 7fe94715d640 1 -- start start 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.506+0000 7fe94715d640 10 monclient: init 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9400a1a00 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe940133b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-b con 0x7fe94009b1f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe94009d5a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94009cc40 con 0x7fe9400a1a00 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400a1dd0 con 0x7fe94009d5a0 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94009cf80 con 0x7fe94009b1f0 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe94715d640 10 monclient(hunting): authenticate will time out at 1068.975173s 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe9456fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe94009d5a0 0x7fe940136070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59508/0 (socket says 172.21.15.67:59508) 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe9456fa640 1 -- 172.21.15.67:0/1984153636 learned_addr learned my addr 172.21.15.67:0/1984153636 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe93ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 msgr2=0x7fe940133b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.507+0000 7fe93ffff640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe940133b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe9456fa640 1 -- 172.21.15.67:0/1984153636 >> v1:172.21.15.67:6789/0 conn(0x7fe94009d5a0 legacy=0x7fe940136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe944ef9640 1 -- 172.21.15.67:0/1984153636 >> v1:172.21.15.17:6789/0 conn(0x7fe9400a1a00 legacy=0x7fe94009aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313626363 0 0) 0x7fe9400a1dd0 con 0x7fe94009d5a0 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:16.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): my global_id is 5959 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9300034e0 con 0x7fe94009d5a0 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251571289 0 0) 0x7fe94009cc40 con 0x7fe9400a1a00 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 10 monclient(hunting): my global_id is 84276 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.509+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9400a1dd0 con 0x7fe9400a1a00 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931944185 0 0) 0x7fe9300034e0 con 0x7fe94009d5a0 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 msgr2=0x7fe940133b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe940133b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 >> v1:172.21.15.17:6789/0 conn(0x7fe9400a1a00 legacy=0x7fe94009aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:16.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94012efa0 con 0x7fe94009d5a0 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe938001aa0 con 0x7fe94009d5a0 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe94715d640 5 monclient: authenticate success, global_id 5959 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe94715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: dump: 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.518 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 >> v1:172.21.15.67:6789/0 conn(0x7fe94009d5a0 legacy=0x7fe940136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe930003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): picked mon.c con 0x7fe94009b1f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe930004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): picked mon.b con 0x7fe9300040e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe930004de0 0x7fe930005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): picked mon.a con 0x7fe930004de0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94009cf80 con 0x7fe930004de0 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93000bd30 con 0x7fe9300040e0 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93000bef0 con 0x7fe94009b1f0 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe94715d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:16.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe944ef9640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 msgr2=0x7fe930003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe944ef9640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe930003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe930004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): get_auth_request con 0x7fe9300040e0 auth_method 0 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe930004de0 0x7fe930005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:16.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): get_auth_request con 0x7fe930004de0 auth_method 0 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe9456fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient(hunting): handle_auth_done global_id 5959 payload 293 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 msgr2=0x7fe930003b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94009b1f0 0x7fe930003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe930004de0 msgr2=0x7fe930005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe930004de0 0x7fe930005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:57:16.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 monclient: found mon.b 2024-09-15T01:57:16.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 1 -- 172.21.15.67:0/1984153636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300034e0 con 0x7fe9300040e0 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 10 monclient: _finish_auth 0 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.510+0000 7fe93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93ffff640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 msgr2=0x7fe930004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93ffff640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe930004780 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0038f0 tx=0x7fe92c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29957999026f1f2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c003b80 con 0x7fe9300040e0 2024-09-15T01:57:16.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: dump: 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c003d00 con 0x7fe9300040e0 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 1 -- 172.21.15.67:0/1984153636 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c02ab70 con 0x7fe9300040e0 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 4 set_mon_vals no callback set 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe93f7fe640 10 monclient: dump: 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:16.526 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 10 monclient: shutdown 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 1 -- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 msgr2=0x7fe930004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe94715d640 1 --2- 172.21.15.67:0/1984153636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe930004780 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0038f0 tx=0x7fe92c009060 comp rx=0 tx=0).stop 2024-09-15T01:57:16.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.513+0000 7fe9456fa640 1 -- 172.21.15.67:0/1984153636 reap_dead start 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 1 -- 172.21.15.67:0/1984153636 shutdown_connections 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 1 -- 172.21.15.67:0/1984153636 >> 172.21.15.67:0/1984153636 conn(0x7fe940071340 msgr2=0x7fe9400a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 1 -- 172.21.15.67:0/1984153636 shutdown_connections 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 1 -- 172.21.15.67:0/1984153636 wait complete. 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 10 monclient: build_initial_monmap 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 10 monclient: monmap: 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:16.528 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.514+0000 7fe94715d640 1 Processor -- start 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 -- start start 2024-09-15T01:57:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient: init 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9300040e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 0x7fe94012dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94013f500 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe94012e170 0x7fe940117700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): picked mon.noname-a con 0x7fe94012e170 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400985b0 con 0x7fe94012e170 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94012e990 con 0x7fe9300040e0 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94012e5e0 con 0x7fe94013f500 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe94715d640 10 monclient(hunting): authenticate will time out at 1068.983781s 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47726/0 (socket says 172.21.15.67:47726) 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe93ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 msgr2=0x7fe94012dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 1 -- 172.21.15.67:0/220263401 learned_addr learned my addr 172.21.15.67:0/220263401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:16.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): get_auth_request con 0x7fe9300040e0 auth_method 0 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe93ffff640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 0x7fe94012dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.515+0000 7fe944ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe9456fa640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe94012e170 0x7fe940117700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 10 monclient(hunting): handle_auth_done global_id 5962 payload 824 2024-09-15T01:57:16.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 -- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 msgr2=0x7fe94012dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 0x7fe94012dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 -- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe94012e170 msgr2=0x7fe940117700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe94012e170 0x7fe940117700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 10 monclient: _finish_hunting 0 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 monclient: found mon.noname-b 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.516+0000 7fe944ef9640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940119220 con 0x7fe9300040e0 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe944ef9640 1 -- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 msgr2=0x7fe94013efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe944ef9640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe93400c7c0 tx=0x7fe934001ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934007510 con 0x7fe9300040e0 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:16.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: _finish_auth 0 2024-09-15T01:57:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934005780 con 0x7fe9300040e0 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934005b40 con 0x7fe9300040e0 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 5 monclient: authenticate success, global_id 5962 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 10 monclient: _renew_subs 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe93effd640 4 set_mon_vals no callback set 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94011ad50 con 0x7fe9300040e0 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 10 monclient: _renew_subs 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.518+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe940119df0 con 0x7fe9300040e0 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe93401b400 con 0x7fe9300040e0 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe94715d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe94715d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe94715d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94004bf60 con 0x7fe9300040e0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe93cff9640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 0x7fe92403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fe93404c8e0 con 0x7fe9300040e0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.519+0000 7fe93ffff640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 0x7fe92403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.520+0000 7fe93ffff640 10 monclient: get_auth_request con 0x7fe924038740 auth_method 0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.522+0000 7fe93ffff640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 0x7fe92403abf0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe94009b5e0 tx=0x7fe92c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.524+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe93401b6b0 con 0x7fe9300040e0 2024-09-15T01:57:16.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.524+0000 7fe93cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:16.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.524+0000 7fe93cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:16.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.694+0000 7fe94715d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:16.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.694+0000 7fe94715d640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:16.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.694+0000 7fe94715d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:16.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:16.694+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fe94011b1d0 con 0x7fe9300040e0 2024-09-15T01:57:17.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.414+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v155) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe934010300 con 0x7fe9300040e0 2024-09-15T01:57:17.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.414+0000 7fe93cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:17.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.414+0000 7fe93cff9640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-15T01:57:17.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.518+0000 7fe94715d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:17.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.518+0000 7fe94715d640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:17.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.518+0000 7fe94715d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:17.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:17.518+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fe940067180 con 0x7fe9300040e0 2024-09-15T01:57:18.223 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T01:57:18.223 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T01:57:18.223 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T01:57:18.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T01:57:18.226 INFO:tasks.ceph.mon.c:Started 2024-09-15T01:57:18.227 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:57:18.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:57:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.332+0000 7f4bbba4f640 10 monclient: get_monmap_and_config 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.332+0000 7f4bbba4f640 10 monclient: build_initial_monmap 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.332+0000 7f4bbba4f640 10 monclient: monmap: 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:18.333786+0000 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:18.333786+0000 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:18.336 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:18.336 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:18.336 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.333+0000 7f4bbba4f640 1 Processor -- start 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.333+0000 7f4bbba4f640 1 -- start start 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.333+0000 7f4bbba4f640 10 monclient: init 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb40a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:18.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bb40980a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4bb4136dd0 0x7f4bb41391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb4136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb40a01e0 con 0x7f4bb40980a0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb40a1580 con 0x7f4bb4136dd0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40a5c80 con 0x7f4bb40a58b0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bbba4f640 10 monclient(hunting): authenticate will time out at 1098.330452s 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4bb4136dd0 0x7f4bb41391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb8fea640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4bb40980a0 0x7f4bb4098470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56620/0 (socket says 172.21.15.17:56620) 2024-09-15T01:57:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): get_auth_request con 0x7f4bb4136dd0 auth_method 0 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb8fea640 1 -- 172.21.15.17:0/1854779857 learned_addr learned my addr 172.21.15.17:0/1854779857 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 10 monclient(hunting): get_auth_request con 0x7f4bb40a58b0 auth_method 0 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 1 -- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 1 -- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:18.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb97eb640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient(hunting): handle_auth_done global_id 84279 payload 274 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 -- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 -- 172.21.15.17:0/1854779857 >> v1:172.21.15.17:6789/0 conn(0x7f4bb40980a0 legacy=0x7f4bb4098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient: _finish_hunting 0 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 monclient: found mon.noname-a 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.334+0000 7f4bb9fec640 1 -- 172.21.15.17:0/1854779857 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb41326c0 con 0x7f4bb4136dd0 2024-09-15T01:57:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bb9fec640 1 -- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb4136dd0 msgr2=0x7f4bb41391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bb9fec640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41391b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009e60 tx=0x7f4bac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a593c8787c69b69 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 1 -- 172.21.15.17:0/1854779857 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac007760 con 0x7f4bb4136dd0 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: dump: 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bbba4f640 5 monclient: authenticate success, global_id 84279 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bbba4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 1 -- 172.21.15.17:0/1854779857 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac02d050 con 0x7f4bb4136dd0 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 1 -- 172.21.15.17:0/1854779857 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac007af0 con 0x7f4bb4136dd0 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bbba4f640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4bbba4f640 4 set_mon_vals no callback set 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.337+0000 7f4ba7fff640 10 monclient: dump: 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 10 monclient: shutdown 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 -- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb4136dd0 msgr2=0x7f4bb41391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41391b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009e60 tx=0x7f4bac009780 comp rx=0 tx=0).stop 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 -- 172.21.15.17:0/1854779857 shutdown_connections 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41391b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/1854779857 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 -- 172.21.15.17:0/1854779857 >> 172.21.15.17:0/1854779857 conn(0x7f4bb4070ed0 msgr2=0x7f4bb40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 -- 172.21.15.17:0/1854779857 shutdown_connections 2024-09-15T01:57:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.338+0000 7f4bbba4f640 1 -- 172.21.15.17:0/1854779857 wait complete. 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.339+0000 7f4bbba4f640 10 monclient: build_initial_monmap 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.339+0000 7f4bbba4f640 10 monclient: monmap: 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.339+0000 7f4bbba4f640 1 Processor -- start 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.339+0000 7f4bbba4f640 1 -- start start 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.339+0000 7f4bbba4f640 10 monclient: init 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb40980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb40a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb4136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4135f60 con 0x7f4bb40980a0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb41393f0 con 0x7f4bb4136dd0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4135bf0 con 0x7f4bb40a58b0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bbba4f640 10 monclient(hunting): authenticate will time out at 1098.336193s 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:44234/0 (socket says 172.21.15.17:44234) 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 -- 172.21.15.17:0/4162106810 learned_addr learned my addr 172.21.15.17:0/4162106810 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): get_auth_request con 0x7f4bb40980a0 auth_method 0 2024-09-15T01:57:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 10 monclient(hunting): get_auth_request con 0x7f4bb40a58b0 auth_method 0 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb8fea640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb9fec640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb9fec640 10 monclient(hunting): get_auth_request con 0x7f4bb4136dd0 auth_method 0 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb9fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb9fec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb9fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient(hunting): handle_auth_done global_id 84282 payload 824 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 msgr2=0x7f4bb4133060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb4136dd0 msgr2=0x7f4bb41335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient: _finish_hunting 0 2024-09-15T01:57:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 monclient: found mon.noname-a 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.340+0000 7f4bb97eb640 1 -- 172.21.15.17:0/4162106810 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac009a20 con 0x7f4bb40980a0 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.342+0000 7f4bb97eb640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 msgr2=0x7f4bb4132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.342+0000 7f4bb97eb640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000d890 tx=0x7f4bb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000de20 con 0x7f4bb40980a0 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: dump: 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: _finish_auth 0 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb000bae0 con 0x7f4bb40980a0 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 5 monclient: authenticate success, global_id 84282 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 10 monclient: _renew_subs 2024-09-15T01:57:18.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb4134c90 con 0x7f4bb40980a0 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000bea0 con 0x7f4bb40980a0 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba57fa640 10 monclient: dump: 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.358 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4ba77fe640 4 set_mon_vals no callback set 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 10 monclient: _renew_subs 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.343+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb4136ac0 con 0x7f4bb40980a0 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4bbba4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4bbba4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4bbba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb40724a0 con 0x7f4bb40980a0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4bb0014640 con 0x7f4bb40980a0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4ba57fa640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 0x7f4b9803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.344+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f4bb004d580 con 0x7f4bb40980a0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.345+0000 7f4bb8fea640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 0x7f4b9803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.345+0000 7f4bb8fea640 10 monclient: get_auth_request con 0x7f4b98038740 auth_method 0 2024-09-15T01:57:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.347+0000 7f4bb8fea640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 0x7f4b9803abf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8006fd0 tx=0x7f4ba8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.349+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4bb00148f0 con 0x7f4bb40980a0 2024-09-15T01:57:18.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.349+0000 7f4ba57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.349+0000 7f4ba57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.416+0000 7fe93cff9640 1 -- 172.21.15.67:0/220263401 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v156) v1 ==== 241+0+0 (secure 0 0 0) 0x7fe9340104e0 con 0x7fe9300040e0 2024-09-15T01:57:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.416+0000 7fe93cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T01:57:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.416+0000 7fe93cff9640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-15T01:57:18.417 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-15T01:57:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 msgr2=0x7fe92403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 0x7fe92403abf0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe94009b5e0 tx=0x7fe92c030000 comp rx=0 tx=0).stop 2024-09-15T01:57:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 10 monclient: shutdown 2024-09-15T01:57:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:18.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 msgr2=0x7fe94013efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe93400c7c0 tx=0x7fe934001ba0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 shutdown_connections 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe924038740 0x7fe92403abf0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe94012e170 0x7fe940117700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe94013f500 0x7fe94012dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 --2- 172.21.15.67:0/220263401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9300040e0 0x7fe94013efc0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.418+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 >> 172.21.15.67:0/220263401 conn(0x7fe940071340 msgr2=0x7fe940131880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.419+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 shutdown_connections 2024-09-15T01:57:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.419+0000 7fe94715d640 1 -- 172.21.15.67:0/220263401 wait complete. 2024-09-15T01:57:18.432 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:18.432 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:18.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep fish 2024-09-15T01:57:18.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '10\.' 2024-09-15T01:57:18.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 10 monclient: build_initial_monmap 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 10 monclient: monmap: 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:18.512492+0000 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:18.512492+0000 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:18.514 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:18.515 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:18.515 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 1 Processor -- start 2024-09-15T01:57:18.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 1 -- start start 2024-09-15T01:57:18.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.512+0000 7feb24d9d640 10 monclient: init 2024-09-15T01:57:18.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb200a0b50 0x7feb20069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-c con 0x7feb200a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7feb2009eea0 0x7feb2009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-a con 0x7feb2009eea0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb20131b10 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200a61a0 con 0x7feb20131b10 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2009f800 con 0x7feb2009eea0 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200a1610 con 0x7feb200a0b50 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:18.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb24d9d640 10 monclient(hunting): authenticate will time out at 1070.981654s 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb200a0b50 0x7feb20069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb200a0b50 0x7feb20069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47730/0 (socket says 172.21.15.67:47730) 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 1 -- 172.21.15.67:0/34449188 learned_addr learned my addr 172.21.15.67:0/34449188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): get_auth_request con 0x7feb200a0b50 auth_method 0 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1dd9b640 1 --2- 172.21.15.67:0/34449188 >> v2:172.21.15.17:3300/0 conn(0x7feb2009eea0 0x7feb2009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1dd9b640 10 monclient(hunting): get_auth_request con 0x7feb2009eea0 auth_method 0 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.513+0000 7feb1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 10 monclient(hunting): handle_auth_done global_id 5968 payload 274 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 1 -- 172.21.15.67:0/34449188 >> v2:172.21.15.17:3300/0 conn(0x7feb2009eea0 msgr2=0x7feb2009f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 1 --2- 172.21.15.67:0/34449188 >> v2:172.21.15.17:3300/0 conn(0x7feb2009eea0 0x7feb2009f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 1 -- 172.21.15.67:0/34449188 >> v1:172.21.15.17:6789/0 conn(0x7feb20131b10 legacy=0x7feb20131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 1 monclient: found mon.noname-c 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.514+0000 7feb1e59c640 1 -- 172.21.15.67:0/34449188 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200a04c0 con 0x7feb200a0b50 2024-09-15T01:57:18.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.516+0000 7feb1e59c640 1 -- 172.21.15.67:0/34449188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb200a0b50 msgr2=0x7feb20069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.516+0000 7feb1e59c640 1 --2- 172.21.15.67:0/34449188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb200a0b50 0x7feb20069380 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7feb2006a5c0 tx=0x7feb100091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=655cacafb9296972 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 1 -- 172.21.15.67:0/34449188 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10032440 con 0x7feb200a0b50 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: dump: 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.527 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: _finish_auth 0 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 1 -- 172.21.15.67:0/34449188 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb100325a0 con 0x7feb200a0b50 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 5 monclient: authenticate success, global_id 5968 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 1 -- 172.21.15.67:0/34449188 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10032940 con 0x7feb200a0b50 2024-09-15T01:57:18.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 4 set_mon_vals no callback set 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb1d59a640 10 monclient: dump: 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.529 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 10 monclient: shutdown 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 -- 172.21.15.67:0/34449188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb200a0b50 msgr2=0x7feb20069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 --2- 172.21.15.67:0/34449188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb200a0b50 0x7feb20069380 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7feb2006a5c0 tx=0x7feb100091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 -- 172.21.15.67:0/34449188 shutdown_connections 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 --2- 172.21.15.67:0/34449188 >> v2:172.21.15.17:3300/0 conn(0x7feb2009eea0 0x7feb2009f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 --2- 172.21.15.67:0/34449188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb200a0b50 0x7feb20069380 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 -- 172.21.15.67:0/34449188 >> 172.21.15.67:0/34449188 conn(0x7feb20071330 msgr2=0x7feb200a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 -- 172.21.15.67:0/34449188 shutdown_connections 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.517+0000 7feb24d9d640 1 -- 172.21.15.67:0/34449188 wait complete. 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.518+0000 7feb24d9d640 10 monclient: build_initial_monmap 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.518+0000 7feb24d9d640 10 monclient: monmap: 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:18.530 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.518+0000 7feb24d9d640 1 Processor -- start 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.518+0000 7feb24d9d640 1 -- start start 2024-09-15T01:57:18.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.518+0000 7feb24d9d640 10 monclient: init 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-c con 0x7feb2009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feb200a0b50 0x7feb20130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-a con 0x7feb200a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:18.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): picked mon.noname-b con 0x7feb20131b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2009b460 con 0x7feb200a0b50 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb20134bb0 con 0x7feb20131b10 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2011dff0 con 0x7feb2009eea0 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb24d9d640 10 monclient(hunting): authenticate will time out at 1070.987212s 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47742/0 (socket says 172.21.15.67:47742) 2024-09-15T01:57:18.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 -- 172.21.15.67:0/4289919372 learned_addr learned my addr 172.21.15.67:0/4289919372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): get_auth_request con 0x7feb20131b10 auth_method 0 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feb200a0b50 0x7feb20130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 10 monclient(hunting): get_auth_request con 0x7feb2009eea0 auth_method 0 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): get_auth_request con 0x7feb200a0b50 auth_method 0 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 msgr2=0x7feb2009a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 msgr2=0x7feb2009a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1e59c640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient(hunting): handle_auth_done global_id 5971 payload 824 2024-09-15T01:57:18.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 msgr2=0x7feb2009a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feb200a0b50 msgr2=0x7feb20130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feb200a0b50 0x7feb20130bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 monclient: found mon.noname-b 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.519+0000 7feb1ed9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10009450 con 0x7feb20131b10 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.521+0000 7feb1ed9d640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 msgr2=0x7feb2011b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.521+0000 7feb1ed9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c8c0 tx=0x7feb18002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1800aa00 con 0x7feb20131b10 2024-09-15T01:57:18.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: dump: 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1800ab80 con 0x7feb20131b10 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18012420 con 0x7feb20131b10 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 5 monclient: authenticate success, global_id 5971 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 10 monclient: _renew_subs 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb20141ed0 con 0x7feb20131b10 2024-09-15T01:57:18.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 10 monclient: dump: 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:18.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb1cd99640 4 set_mon_vals no callback set 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 10 monclient: _renew_subs 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201364b0 con 0x7feb20131b10 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.522+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7feb18012890 con 0x7feb20131b10 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb24d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb24d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb24d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb0e7fc640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 0x7feb0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 9825+0+0 (secure 0 0 0) 0x7feb1804d1d0 con 0x7feb20131b10 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb1e59c640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 0x7feb0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.523+0000 7feb1e59c640 10 monclient: get_auth_request con 0x7feb00038740 auth_method 0 2024-09-15T01:57:18.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.525+0000 7feb1e59c640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 0x7feb0003abf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7feb200a1200 tx=0x7feb10032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.527+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.527+0000 7feb0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.527+0000 7feb0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:18.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.577+0000 7f4bbba4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:57:18.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.577+0000 7f4bbba4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:18.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.577+0000 7f4bbba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:18.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:18.577+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bb409fc90 con 0x7f4bb40980a0 2024-09-15T01:57:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.697+0000 7feb24d9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.697+0000 7feb24d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.697+0000 7feb24d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:18.697+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:21.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:21.339+0000 7f4b9ffff640 10 monclient: tick 2024-09-15T01:57:21.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:21.339+0000 7f4b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:21.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:21.339+0000 7f4b9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:21.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:21.518+0000 7feb0d7fa640 10 monclient: tick 2024-09-15T01:57:21.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:21.518+0000 7feb0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:21.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:21.518+0000 7feb0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:22.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:22.339+0000 7f4b9ffff640 10 monclient: tick 2024-09-15T01:57:22.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:22.339+0000 7f4b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:22.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:22.518+0000 7feb0d7fa640 10 monclient: tick 2024-09-15T01:57:22.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:22.518+0000 7feb0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:23.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.340+0000 7f4b9ffff640 10 monclient: tick 2024-09-15T01:57:23.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.340+0000 7f4b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:23.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.518+0000 7feb0d7fa640 10 monclient: tick 2024-09-15T01:57:23.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.518+0000 7feb0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:23.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.541+0000 7f4ba57fa640 1 -- 172.21.15.17:0/4162106810 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f4bb001dbb0 con 0x7f4bb40980a0 2024-09-15T01:57:23.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.541+0000 7f4ba57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:23.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.541+0000 7f4ba57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:23.545 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":78,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:57:23.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 msgr2=0x7f4b9803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 0x7f4b9803abf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8006fd0 tx=0x7f4ba8008040 comp rx=0 tx=0).stop 2024-09-15T01:57:23.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 10 monclient: shutdown 2024-09-15T01:57:23.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 msgr2=0x7f4bb4132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000d890 tx=0x7f4bb0008040 comp rx=0 tx=0).stop 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 shutdown_connections 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b98038740 0x7f4b9803abf0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb4136dd0 0x7f4bb41335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bb40a58b0 0x7f4bb4133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 --2- 172.21.15.17:0/4162106810 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bb40980a0 0x7f4bb4132b20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 >> 172.21.15.17:0/4162106810 conn(0x7f4bb4070ed0 msgr2=0x7f4bb409afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 shutdown_connections 2024-09-15T01:57:23.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:23.544+0000 7f4bbba4f640 1 -- 172.21.15.17:0/4162106810 wait complete. 2024-09-15T01:57:23.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.545+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:23.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.545+0000 7feb0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.545+0000 7feb0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.658+0000 7feb24d9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.658+0000 7feb24d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.658+0000 7feb24d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.658+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.659+0000 7feb0e7fc640 1 -- 172.21.15.67:0/4289919372 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7feb200a12d0 con 0x7feb20131b10 2024-09-15T01:57:23.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.659+0000 7feb0e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.659+0000 7feb0e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:23.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 msgr2=0x7feb0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 0x7feb0003abf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7feb200a1200 tx=0x7feb10032000 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 10 monclient: shutdown 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 msgr2=0x7feb2011b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c8c0 tx=0x7feb18002640 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 shutdown_connections 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7feb00038740 0x7feb0003abf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feb200a0b50 0x7feb20130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb20131b10 0x7feb2011b940 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 --2- 172.21.15.67:0/4289919372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feb2009eea0 0x7feb2009a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 >> 172.21.15.67:0/4289919372 conn(0x7feb20071330 msgr2=0x7feb20134dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:23.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 shutdown_connections 2024-09-15T01:57:23.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.661+0000 7feb24d9d640 1 -- 172.21.15.67:0/4289919372 wait complete. 2024-09-15T01:57:23.673 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:23.673 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:23.673 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:23.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:23.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep fish 2024-09-15T01:57:23.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '11\.' 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.747+0000 7f527771d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.747+0000 7f527771d640 10 monclient: build_initial_monmap 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.747+0000 7f527771d640 10 monclient: monmap: 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:23.748990+0000 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:23.748990+0000 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:23.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:23.751 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.748+0000 7f527771d640 1 Processor -- start 2024-09-15T01:57:23.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.748+0000 7f527771d640 1 -- start start 2024-09-15T01:57:23.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.748+0000 7f527771d640 10 monclient: init 2024-09-15T01:57:23.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 0x7f527006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): picked mon.noname-a con 0x7f52700a3830 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f527006bef0 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5270136d00 0x7f52701390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): picked mon.noname-c con 0x7f5270136d00 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52700a3c00 con 0x7f527006bef0 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52700696a0 con 0x7f52700a3830 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5270069940 con 0x7f5270136d00 2024-09-15T01:57:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f527771d640 10 monclient(hunting): authenticate will time out at 1076.218117s 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5270136d00 0x7f52701390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5274cb8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f527006bef0 0x7f5270099aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35124/0 (socket says 172.21.15.67:35124) 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5274cb8640 1 -- 172.21.15.67:0/3490974522 learned_addr learned my addr 172.21.15.67:0/3490974522 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5270136d00 0x7f52701390e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45148/0 (socket says 172.21.15.67:45148) 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): get_auth_request con 0x7f5270136d00 auth_method 0 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f52754b9640 1 --2- 172.21.15.67:0/3490974522 >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 0x7f527006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient(hunting): handle_auth_done global_id 5974 payload 274 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 -- 172.21.15.67:0/3490974522 >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 msgr2=0x7f527006a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 --2- 172.21.15.67:0/3490974522 >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 0x7f527006a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 -- 172.21.15.67:0/3490974522 >> v1:172.21.15.67:6789/0 conn(0x7f527006bef0 legacy=0x7f5270099aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient: _finish_hunting 0 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 monclient: found mon.noname-c 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f5275cba640 1 -- 172.21.15.67:0/3490974522 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270069c20 con 0x7f5270136d00 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f52754b9640 10 monclient: get_auth_request con 0x7f52700a3830 auth_method 0 2024-09-15T01:57:23.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.749+0000 7f52754b9640 1 --2- 172.21.15.67:0/3490974522 >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 0x7f527006a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5275cba640 1 -- 172.21.15.67:0/3490974522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5270136d00 msgr2=0x7f52701390e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5275cba640 1 --2- 172.21.15.67:0/3490974522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5270136d00 0x7f52701390e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f527006ab40 tx=0x7f5268009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87f4804d6a8df6b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 1 -- 172.21.15.67:0/3490974522 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268007840 con 0x7f5270136d00 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: dump: 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 1 -- 172.21.15.67:0/3490974522 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526802d050 con 0x7f5270136d00 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f527771d640 5 monclient: authenticate success, global_id 5974 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f527771d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f527771d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 1 -- 172.21.15.67:0/3490974522 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268007bd0 con 0x7f5270136d00 2024-09-15T01:57:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f527771d640 4 set_mon_vals no callback set 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.752+0000 7f5267fff640 10 monclient: dump: 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 10 monclient: shutdown 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 -- 172.21.15.67:0/3490974522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5270136d00 msgr2=0x7f52701390e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 --2- 172.21.15.67:0/3490974522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5270136d00 0x7f52701390e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f527006ab40 tx=0x7f5268009780 comp rx=0 tx=0).stop 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 -- 172.21.15.67:0/3490974522 shutdown_connections 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 --2- 172.21.15.67:0/3490974522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5270136d00 0x7f52701390e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 --2- 172.21.15.67:0/3490974522 >> v2:172.21.15.17:3300/0 conn(0x7f52700a3830 0x7f527006a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 -- 172.21.15.67:0/3490974522 >> 172.21.15.67:0/3490974522 conn(0x7f5270070fb0 msgr2=0x7f52700a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 -- 172.21.15.67:0/3490974522 shutdown_connections 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 1 -- 172.21.15.67:0/3490974522 wait complete. 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 10 monclient: build_initial_monmap 2024-09-15T01:57:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.753+0000 7f527771d640 10 monclient: monmap: 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 Processor -- start 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 -- start start 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient: init 2024-09-15T01:57:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): picked mon.noname-b con 0x7f527006bef0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52700a3830 0x7f5270131850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): picked mon.noname-a con 0x7f52700a3830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5270136d00 0x7f5270131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): picked mon.noname-c con 0x7f5270136d00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52701327f0 con 0x7f52700a3830 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527006ea50 con 0x7f527006bef0 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527009c4b0 con 0x7f5270136d00 2024-09-15T01:57:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f527771d640 10 monclient(hunting): authenticate will time out at 1076.223823s 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45162/0 (socket says 172.21.15.67:45162) 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 1 -- 172.21.15.67:0/865861643 learned_addr learned my addr 172.21.15.67:0/865861643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 10 monclient(hunting): get_auth_request con 0x7f527006bef0 auth_method 0 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f52754b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.754+0000 7f5274cb8640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52700a3830 0x7f5270131850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:23.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): get_auth_request con 0x7f52700a3830 auth_method 0 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5275cba640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5270136d00 0x7f5270131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5275cba640 10 monclient(hunting): get_auth_request con 0x7f5270136d00 auth_method 0 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5275cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5275cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5275cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f5274cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient(hunting): handle_auth_done global_id 5977 payload 824 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 -- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5270136d00 msgr2=0x7f5270131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5270136d00 0x7f5270131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 -- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52700a3830 msgr2=0x7f5270131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52700a3830 0x7f5270131850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient: _finish_hunting 0 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 monclient: found mon.noname-b 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.755+0000 7f52754b9640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268009a20 con 0x7f527006bef0 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52754b9640 1 -- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 msgr2=0x7f5270131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52754b9640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c880 tx=0x7f526c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c00ce10 con 0x7f527006bef0 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:23.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: dump: 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: _finish_auth 0 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c007aa0 con 0x7f527006bef0 2024-09-15T01:57:23.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 5 monclient: authenticate success, global_id 5977 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c007e60 con 0x7f527006bef0 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 10 monclient: _renew_subs 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527009a940 con 0x7f527006bef0 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52657fa640 10 monclient: dump: 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:23.772 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f52677fe640 4 set_mon_vals no callback set 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 10 monclient: _renew_subs 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.757+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527009ab80 con 0x7f527006bef0 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f526c011600 con 0x7f527006bef0 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f527771d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f527771d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f527771d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527009c080 con 0x7f527006bef0 2024-09-15T01:57:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f52657fa640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 0x7f525403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f526c04cf90 con 0x7f527006bef0 2024-09-15T01:57:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f5274cb8640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 0x7f525403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.758+0000 7f5274cb8640 10 monclient: get_auth_request con 0x7f5254038740 auth_method 0 2024-09-15T01:57:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.761+0000 7f5274cb8640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 0x7f525403abf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5270132aa0 tx=0x7f5260006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.763+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f526c011be0 con 0x7f527006bef0 2024-09-15T01:57:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.763+0000 7f52657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.763+0000 7f52657fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.933+0000 7f527771d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.933+0000 7f527771d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.933+0000 7f527771d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.933+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f527004bf60 con 0x7f527006bef0 2024-09-15T01:57:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.934+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f526c01e070 con 0x7f527006bef0 2024-09-15T01:57:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.934+0000 7f52657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:23.934+0000 7f52657fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:24.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.039+0000 7f527771d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.039+0000 7f527771d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.039+0000 7f527771d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.039+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f527006e3b0 con 0x7f527006bef0 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.040+0000 7f52657fa640 1 -- 172.21.15.67:0/865861643 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f526c017300 con 0x7f527006bef0 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.040+0000 7f52657fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.040+0000 7f52657fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:24.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 msgr2=0x7f525403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 0x7f525403abf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5270132aa0 tx=0x7f5260006f50 comp rx=0 tx=0).stop 2024-09-15T01:57:24.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 10 monclient: shutdown 2024-09-15T01:57:24.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:24.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 msgr2=0x7f5270131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c880 tx=0x7f526c008040 comp rx=0 tx=0).stop 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.041+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 shutdown_connections 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5254038740 0x7f525403abf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52700a3830 0x7f5270131850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5270136d00 0x7f5270131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 --2- 172.21.15.67:0/865861643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f527006bef0 0x7f5270131310 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 >> 172.21.15.67:0/865861643 conn(0x7f5270070fb0 msgr2=0x7f52700a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 shutdown_connections 2024-09-15T01:57:24.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.042+0000 7f527771d640 1 -- 172.21.15.67:0/865861643 wait complete. 2024-09-15T01:57:24.053 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:24.053 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:24.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:24.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:24.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep fish 2024-09-15T01:57:24.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '12\.' 2024-09-15T01:57:24.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 10 monclient: get_monmap_and_config 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 10 monclient: build_initial_monmap 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 10 monclient: monmap: 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:24.128205+0000 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:24.128205+0000 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 1 Processor -- start 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 1 -- start start 2024-09-15T01:57:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.127+0000 7f1ea5834640 10 monclient: init 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea0068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1ea0069190 0x7f1ea0069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea0069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ea012e960 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0096c10 con 0x7f1ea012e960 2024-09-15T01:57:24.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea00a14d0 con 0x7f1ea0069190 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea009ff20 con 0x7f1ea0068dc0 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1ea5834640 10 monclient(hunting): authenticate will time out at 1076.597183s 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1ea0069190 0x7f1ea0069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:49406/0 (socket says 172.21.15.67:49406) 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): get_auth_request con 0x7f1ea0069190 auth_method 0 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 -- 172.21.15.67:0/735363776 learned_addr learned my addr 172.21.15.67:0/735363776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9f7fe640 1 --1- 172.21.15.67:0/735363776 >> v1:172.21.15.17:6789/0 conn(0x7f1ea012e960 0x7f1ea012edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:42584/0 (socket says 172.21.15.67:42584) 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 10 monclient(hunting): get_auth_request con 0x7f1ea0068dc0 auth_method 0 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 -- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 msgr2=0x7f1ea0132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 -- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 msgr2=0x7f1ea0132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9effd640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:24.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient(hunting): handle_auth_done global_id 84294 payload 274 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 -- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 msgr2=0x7f1ea0132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 -- 172.21.15.67:0/735363776 >> v1:172.21.15.17:6789/0 conn(0x7f1ea012e960 legacy=0x7f1ea012edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 monclient: found mon.noname-a 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:24.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.128+0000 7f1e9e7fc640 1 -- 172.21.15.67:0/735363776 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00a0180 con 0x7f1ea0069190 2024-09-15T01:57:24.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.131+0000 7f1e9e7fc640 1 -- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea0069190 msgr2=0x7f1ea0069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:24.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.131+0000 7f1e9e7fc640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea0069190 0x7f1ea0069560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2840cf08cf0218 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 1 -- 172.21.15.67:0/735363776 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9402adf0 con 0x7f1ea0069190 2024-09-15T01:57:24.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: dump: 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 1 -- 172.21.15.67:0/735363776 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e94032440 con 0x7f1ea0069190 2024-09-15T01:57:24.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 5 monclient: authenticate success, global_id 84294 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 4 set_mon_vals no callback set 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 1 -- 172.21.15.67:0/735363776 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940327e0 con 0x7f1ea0069190 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1e9dffb640 10 monclient: dump: 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.143 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 10 monclient: shutdown 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 -- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea0069190 msgr2=0x7f1ea0069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea0069190 0x7f1ea0069560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e940090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 -- 172.21.15.67:0/735363776 shutdown_connections 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea0069190 0x7f1ea0069560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 --2- 172.21.15.67:0/735363776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0068dc0 0x7f1ea0132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 -- 172.21.15.67:0/735363776 >> 172.21.15.67:0/735363776 conn(0x7f1ea00711b0 msgr2=0x7f1ea00a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:24.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 -- 172.21.15.67:0/735363776 shutdown_connections 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.132+0000 7f1ea5834640 1 -- 172.21.15.67:0/735363776 wait complete. 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.133+0000 7f1ea5834640 10 monclient: build_initial_monmap 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.133+0000 7f1ea5834640 10 monclient: monmap: 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:24.145 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.133+0000 7f1ea5834640 1 Processor -- start 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.133+0000 7f1ea5834640 1 -- start start 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.133+0000 7f1ea5834640 10 monclient: init 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea0068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0069190 0x7f1ea0097f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea0069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea012e960 0x7f1ea0098490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea012e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea00a0890 con 0x7f1ea012e960 2024-09-15T01:57:24.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea0134d30 con 0x7f1ea0068dc0 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea009a580 con 0x7f1ea0069190 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1ea5834640 10 monclient(hunting): authenticate will time out at 1076.603480s 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45172/0 (socket says 172.21.15.67:45172) 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 -- 172.21.15.67:0/2302461221 learned_addr learned my addr 172.21.15.67:0/2302461221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): get_auth_request con 0x7f1ea0068dc0 auth_method 0 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:24.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9e7fc640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0069190 0x7f1ea0097f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9f7fe640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea012e960 0x7f1ea0098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9e7fc640 10 monclient(hunting): get_auth_request con 0x7f1ea0069190 auth_method 0 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9f7fe640 10 monclient(hunting): get_auth_request con 0x7f1ea012e960 auth_method 0 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient(hunting): handle_auth_done global_id 5980 payload 824 2024-09-15T01:57:24.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 -- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0069190 msgr2=0x7f1ea0097f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0069190 0x7f1ea0097f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 -- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea012e960 msgr2=0x7f1ea0098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea012e960 0x7f1ea0098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 monclient: found mon.noname-b 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.134+0000 7f1e9effd640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94009390 con 0x7f1ea0068dc0 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e9effd640 1 -- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 msgr2=0x7f1ea012d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e9effd640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e8a0 tx=0x7f1e90008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000c6d0 con 0x7f1ea0068dc0 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: dump: 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:24.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 5 monclient: authenticate success, global_id 5980 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 10 monclient: _renew_subs 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea0099ec0 con 0x7f1ea0068dc0 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9000c830 con 0x7f1ea0068dc0 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000cbf0 con 0x7f1ea0068dc0 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e86ffd640 10 monclient: dump: 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.151 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 10 monclient: _renew_subs 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea00999b0 con 0x7f1ea0068dc0 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.137+0000 7f1e9d7fa640 4 set_mon_vals no callback set 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1e90013410 con 0x7f1ea0068dc0 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1ea5834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1ea5834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1ea5834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea004bf60 con 0x7f1ea0068dc0 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1e86ffd640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f1e9004ce30 con 0x7f1ea0068dc0 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1e9e7fc640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.138+0000 7f1e9e7fc640 10 monclient: get_auth_request con 0x7f1e8c038740 auth_method 0 2024-09-15T01:57:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.141+0000 7f1e9e7fc640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1e94031000 tx=0x7f1e94032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.142+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1e90023300 con 0x7f1ea0068dc0 2024-09-15T01:57:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.142+0000 7f1e86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.142+0000 7f1e86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.310+0000 7f1ea5834640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.310+0000 7f1ea5834640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.310+0000 7f1ea5834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.310+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f1ea0069aa0 con 0x7f1ea0068dc0 2024-09-15T01:57:24.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.311+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f1e90023300 con 0x7f1ea0068dc0 2024-09-15T01:57:24.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.311+0000 7f1e86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.311+0000 7f1e86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:24.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1ea5834640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1ea5834640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1ea5834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f1ea009b850 con 0x7f1ea0068dc0 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1e86ffd640 1 -- 172.21.15.67:0/2302461221 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f1e90023300 con 0x7f1ea0068dc0 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1e86ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:24.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.414+0000 7f1e86ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:24.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 msgr2=0x7f1e8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1e94031000 tx=0x7f1e94032000 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 10 monclient: shutdown 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 msgr2=0x7f1ea012d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e8a0 tx=0x7f1e90008040 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 shutdown_connections 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1e8c038740 0x7f1e8c03abf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ea012e960 0x7f1ea0098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ea0069190 0x7f1ea0097f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 --2- 172.21.15.67:0/2302461221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ea0068dc0 0x7f1ea012d9e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 >> 172.21.15.67:0/2302461221 conn(0x7f1ea00711b0 msgr2=0x7f1ea0134f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 shutdown_connections 2024-09-15T01:57:24.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.416+0000 7f1ea5834640 1 -- 172.21.15.67:0/2302461221 wait complete. 2024-09-15T01:57:24.427 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:24.427 INFO:tasks.workunit.client.0.smithi067.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:24.428 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.500+0000 7f157a71d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.500+0000 7f157a71d640 10 monclient: build_initial_monmap 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.500+0000 7f157a71d640 10 monclient: monmap: 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:24.501999+0000 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:24.501999+0000 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:24.503 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:24.504 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.501+0000 7f157a71d640 1 Processor -- start 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.501+0000 7f157a71d640 1 -- start start 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.501+0000 7f157a71d640 10 monclient: init 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-b con 0x7f15740a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f157409eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f157409f290 0x7f1574132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-a con 0x7f157409f290 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740a61a0 con 0x7f157409eea0 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.507 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15740a1610 con 0x7f157409f290 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15740a04c0 con 0x7f15740a0b50 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f157a71d640 10 monclient(hunting): authenticate will time out at 1076.971190s 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f15737fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f157409eea0 0x7f1574131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35130/0 (socket says 172.21.15.67:35130) 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f15737fe640 1 -- 172.21.15.67:0/3454845754 learned_addr learned my addr 172.21.15.67:0/3454845754 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 10 monclient(hunting): get_auth_request con 0x7f15740a0b50 auth_method 0 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 --2- 172.21.15.67:0/3454845754 >> v2:172.21.15.17:3300/0 conn(0x7f157409f290 0x7f1574132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): get_auth_request con 0x7f157409f290 auth_method 0 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 1 -- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 msgr2=0x7f1574069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 1 -- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 msgr2=0x7f1574069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1573fff640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient(hunting): handle_auth_done global_id 84300 payload 274 2024-09-15T01:57:24.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 -- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 msgr2=0x7f1574069380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 -- 172.21.15.67:0/3454845754 >> v1:172.21.15.67:6789/0 conn(0x7f157409eea0 legacy=0x7f1574131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient: _finish_hunting 0 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 monclient: found mon.noname-a 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:24.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.502+0000 7f1578cba640 1 -- 172.21.15.67:0/3454845754 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740a0f20 con 0x7f157409f290 2024-09-15T01:57:24.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.505+0000 7f1578cba640 1 -- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 msgr2=0x7f1574132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:24.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.505+0000 7f1578cba640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574132220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f156c009e60 tx=0x7f156c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6510372182e7c418 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 1 -- 172.21.15.67:0/3454845754 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c007760 con 0x7f157409f290 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: dump: 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.518 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 1 -- 172.21.15.67:0/3454845754 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c02d050 con 0x7f157409f290 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 1 -- 172.21.15.67:0/3454845754 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c007af0 con 0x7f157409f290 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f1572ffd640 10 monclient: dump: 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 5 monclient: authenticate success, global_id 84300 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 4 set_mon_vals no callback set 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 10 monclient: shutdown 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 -- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 msgr2=0x7f1574132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574132220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f156c009e60 tx=0x7f156c009780 comp rx=0 tx=0).stop 2024-09-15T01:57:24.520 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 -- 172.21.15.67:0/3454845754 shutdown_connections 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574132220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 --2- 172.21.15.67:0/3454845754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15740a0b50 0x7f1574069380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 -- 172.21.15.67:0/3454845754 >> 172.21.15.67:0/3454845754 conn(0x7f1574071330 msgr2=0x7f15740a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 -- 172.21.15.67:0/3454845754 shutdown_connections 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.506+0000 7f157a71d640 1 -- 172.21.15.67:0/3454845754 wait complete. 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.507+0000 7f157a71d640 10 monclient: build_initial_monmap 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.507+0000 7f157a71d640 10 monclient: monmap: 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:24.521 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.507+0000 7f157a71d640 1 Processor -- start 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.507+0000 7f157a71d640 1 -- start start 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.507+0000 7f157a71d640 10 monclient: init 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-c con 0x7f157409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-a con 0x7f157409f290 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): picked mon.noname-b con 0x7f15740a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574122610 con 0x7f157409f290 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741222a0 con 0x7f15740a0b50 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1574141f30 con 0x7f157409eea0 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f157a71d640 10 monclient(hunting): authenticate will time out at 1076.977294s 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45178/0 (socket says 172.21.15.67:45178) 2024-09-15T01:57:24.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 -- 172.21.15.67:0/1397015280 learned_addr learned my addr 172.21.15.67:0/1397015280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): get_auth_request con 0x7f15740a0b50 auth_method 0 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 10 monclient(hunting): get_auth_request con 0x7f157409eea0 auth_method 0 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574133e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): get_auth_request con 0x7f157409f290 auth_method 0 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 msgr2=0x7f15741410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 msgr2=0x7f15741410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1573fff640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f15737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient(hunting): handle_auth_done global_id 5983 payload 824 2024-09-15T01:57:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 msgr2=0x7f15741410d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 msgr2=0x7f1574133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574133e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient: _finish_hunting 0 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 monclient: found mon.noname-b 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.508+0000 7f1578cba640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c009a20 con 0x7f15740a0b50 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.511+0000 7f1578cba640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 msgr2=0x7f157411e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.511+0000 7f1578cba640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f156c0099f0 tx=0x7f156c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c031c10 con 0x7f15740a0b50 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: dump: 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: _finish_auth 0 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 5 monclient: authenticate success, global_id 5983 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 10 monclient: _renew_subs 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1574120490 con 0x7f15740a0b50 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c040070 con 0x7f15740a0b50 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c045430 con 0x7f15740a0b50 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f156bfff640 10 monclient: dump: 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f15727fc640 4 set_mon_vals no callback set 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 10 monclient: _renew_subs 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.513+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1574121060 con 0x7f15740a0b50 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f156c042030 con 0x7f15740a0b50 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f157a71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f157a71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f157a71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574122460 con 0x7f15740a0b50 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f156bfff640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 0x7f155403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f156c06e820 con 0x7f15740a0b50 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f1573fff640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 0x7f155403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.514+0000 7f1573fff640 10 monclient: get_auth_request con 0x7f1554038740 auth_method 0 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.516+0000 7f1573fff640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 0x7f155403abf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1564006fd0 tx=0x7f1564008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.518+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f156c044bb0 con 0x7f15740a0b50 2024-09-15T01:57:24.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.518+0000 7f156bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:24.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.518+0000 7f156bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:24.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.687+0000 7f157a71d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:24.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.688+0000 7f157a71d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:24.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.688+0000 7f157a71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:24.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:24.688+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f157404bf60 con 0x7f15740a0b50 2024-09-15T01:57:25.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.613+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v157) v1 ==== 123+0+0 (secure 0 0 0) 0x7f156c0443d0 con 0x7f15740a0b50 2024-09-15T01:57:25.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.613+0000 7f156bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:25.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.613+0000 7f156bfff640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T01:57:25.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.719+0000 7f157a71d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:25.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.719+0000 7f157a71d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:25.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.719+0000 7f157a71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:25.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:25.719+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f157404bf60 con 0x7f15740a0b50 2024-09-15T01:57:26.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:57:26.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.615+0000 7f156bfff640 1 -- 172.21.15.67:0/1397015280 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v158) v1 ==== 149+0+0 (secure 0 0 0) 0x7f156c0445b0 con 0x7f15740a0b50 2024-09-15T01:57:26.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.615+0000 7f156bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:57:26.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.615+0000 7f156bfff640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:57:26.618 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:57:26.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 msgr2=0x7f155403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 0x7f155403abf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1564006fd0 tx=0x7f1564008040 comp rx=0 tx=0).stop 2024-09-15T01:57:26.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 10 monclient: shutdown 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 msgr2=0x7f157411e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.618+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f156c0099f0 tx=0x7f156c037000 comp rx=0 tx=0).stop 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 shutdown_connections 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1554038740 0x7f155403abf0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f157409f290 0x7f1574133e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15740a0b50 0x7f157411e990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 --2- 172.21.15.67:0/1397015280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f157409eea0 0x7f15741410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 >> 172.21.15.67:0/1397015280 conn(0x7f1574071330 msgr2=0x7f157409ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 shutdown_connections 2024-09-15T01:57:26.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.619+0000 7f157a71d640 1 -- 172.21.15.67:0/1397015280 wait complete. 2024-09-15T01:57:26.633 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 10 monclient: get_monmap_and_config 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 10 monclient: build_initial_monmap 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 10 monclient: monmap: 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:26.671906+0000 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:26.671906+0000 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.673 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:26.674 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 1 Processor -- start 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 1 -- start start 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.670+0000 7f6d09ac9640 10 monclient: init 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d041319a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0409eac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6d04136cd0 0x7f6d041390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f6d04136cd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040a1530 con 0x7f6d041319a0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0409fca0 con 0x7f6d04136cd0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040a00e0 con 0x7f6d0409eac0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d09ac9640 10 monclient(hunting): authenticate will time out at 1106.668465s 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6d04136cd0 0x7f6d041390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d037fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6d041319a0 0x7f6d04131d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40388/0 (socket says 172.21.15.17:40388) 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d037fe640 1 -- 172.21.15.17:0/4055030430 learned_addr learned my addr 172.21.15.17:0/4055030430 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): get_auth_request con 0x7f6d04136cd0 auth_method 0 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request con 0x7f6d0409eac0 auth_method 0 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 1 -- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 msgr2=0x7f6d0409ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 1 -- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 msgr2=0x7f6d0409ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d02ffd640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient(hunting): handle_auth_done global_id 84306 payload 274 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 -- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 msgr2=0x7f6d0409ee90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 -- 172.21.15.17:0/4055030430 >> v1:172.21.15.17:6789/0 conn(0x7f6d041319a0 legacy=0x7f6d04131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient: _finish_hunting 0 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 monclient: found mon.noname-a 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.671+0000 7f6d03fff640 1 -- 172.21.15.17:0/4055030430 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040a1250 con 0x7f6d04136cd0 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d03fff640 1 -- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d04136cd0 msgr2=0x7f6d041390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d03fff640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d04136cd0 0x7f6d041390b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009e60 tx=0x7f6cfc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ff34fe29e4a934 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 1 -- 172.21.15.17:0/4055030430 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc007760 con 0x7f6d04136cd0 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: dump: 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.684 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 1 -- 172.21.15.17:0/4055030430 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc02d050 con 0x7f6d04136cd0 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 1 -- 172.21.15.17:0/4055030430 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc007af0 con 0x7f6d04136cd0 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d09ac9640 5 monclient: authenticate success, global_id 84306 2024-09-15T01:57:26.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d09ac9640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d09ac9640 4 set_mon_vals no callback set 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.674+0000 7f6d027fc640 10 monclient: dump: 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.686 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 10 monclient: shutdown 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 -- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d04136cd0 msgr2=0x7f6d041390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d04136cd0 0x7f6d041390b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009e60 tx=0x7f6cfc009780 comp rx=0 tx=0).stop 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 -- 172.21.15.17:0/4055030430 shutdown_connections 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d04136cd0 0x7f6d041390b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/4055030430 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d0409eac0 0x7f6d0409ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 -- 172.21.15.17:0/4055030430 >> 172.21.15.17:0/4055030430 conn(0x7f6d040a5000 msgr2=0x7f6d040a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 -- 172.21.15.17:0/4055030430 shutdown_connections 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 1 -- 172.21.15.17:0/4055030430 wait complete. 2024-09-15T01:57:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 10 monclient: build_initial_monmap 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.675+0000 7f6d09ac9640 10 monclient: monmap: 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 Processor -- start 2024-09-15T01:57:26.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 -- start start 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient: init 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 0x7f6d0409b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0409eac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f6d041319a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f6d04136cd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0409a670 con 0x7f6d041319a0 2024-09-15T01:57:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0409b870 con 0x7f6d0409eac0 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0409acc0 con 0x7f6d04136cd0 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.676+0000 7f6d09ac9640 10 monclient(hunting): authenticate will time out at 1106.673959s 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53856/0 (socket says 172.21.15.17:53856) 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34272/0 (socket says 172.21.15.17:34272) 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 -- 172.21.15.17:0/1129515315 learned_addr learned my addr 172.21.15.17:0/1129515315 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request con 0x7f6d041319a0 auth_method 0 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 10 monclient(hunting): get_auth_request con 0x7f6d04136cd0 auth_method 0 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 msgr2=0x7f6d040975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 msgr2=0x7f6d040975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 0x7f6d0409b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d03fff640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 10 monclient(hunting): get_auth_request con 0x7f6d0409eac0 auth_method 0 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_done global_id 84309 payload 824 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 msgr2=0x7f6d040975e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 msgr2=0x7f6d0409b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 0x7f6d0409b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 monclient: found mon.noname-a 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d02ffd640 1 -- 172.21.15.17:0/1129515315 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc009a20 con 0x7f6d041319a0 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.677+0000 7f6d037fe640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 0x7f6d0409b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d02ffd640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 msgr2=0x7f6d040970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d02ffd640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800d890 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf800de20 con 0x7f6d041319a0 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: dump: 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf800bae0 con 0x7f6d041319a0 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf800bea0 con 0x7f6d041319a0 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d09ac9640 5 monclient: authenticate success, global_id 84309 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d09ac9640 10 monclient: _renew_subs 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d09ac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d040990c0 con 0x7f6d041319a0 2024-09-15T01:57:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6ceb7fe640 10 monclient: dump: 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.679+0000 7f6d01ffb640 4 set_mon_vals no callback set 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 10 monclient: _renew_subs 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d04139380 con 0x7f6d041319a0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6cf8014640 con 0x7f6d041319a0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.680+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d040724a0 con 0x7f6d041319a0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.681+0000 7f6ceb7fe640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 0x7f6cf003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.681+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f6cf804d2e0 con 0x7f6d041319a0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.681+0000 7f6d037fe640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 0x7f6cf003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.681+0000 7f6d037fe640 10 monclient: get_auth_request con 0x7f6cf0038740 auth_method 0 2024-09-15T01:57:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.683+0000 7f6d037fe640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 0x7f6cf003abf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007fd0 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.685+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6cf80148f0 con 0x7f6d041319a0 2024-09-15T01:57:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.685+0000 7f6ceb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.685+0000 7f6ceb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:26.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.713+0000 7f4c47620640 10 monclient: get_monmap_and_config 2024-09-15T01:57:26.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.713+0000 7f4c47620640 10 monclient: build_initial_monmap 2024-09-15T01:57:26.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.713+0000 7f4c47620640 10 monclient: monmap: 2024-09-15T01:57:26.715 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:26.714476+0000 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:26.714476+0000 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:26.716 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.713+0000 7f4c47620640 1 Processor -- start 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 1 -- start start 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient: init 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T01:57:26.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c400a1a00 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c4009b1f0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4c4009d5a0 0x7f4c40137250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-a con 0x7f4c4009d5a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4009cc40 con 0x7f4c4009b1f0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c400a1dd0 con 0x7f4c400a1a00 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4009cf80 con 0x7f4c4009d5a0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c47620640 10 monclient(hunting): authenticate will time out at 1079.183788s 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c453bc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c400a1a00 0x7f4c4009aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35142/0 (socket says 172.21.15.67:35142) 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c453bc640 1 -- 172.21.15.67:0/4174216539 learned_addr learned my addr 172.21.15.67:0/4174216539 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.714+0000 7f4c45bbd640 1 --2- 172.21.15.67:0/4174216539 >> v2:172.21.15.17:3300/0 conn(0x7f4c4009d5a0 0x7f4c40137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): get_auth_request con 0x7f4c4009d5a0 auth_method 0 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_done global_id 84312 payload 274 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 1 -- 172.21.15.67:0/4174216539 >> v1:172.21.15.67:6789/0 conn(0x7f4c400a1a00 legacy=0x7f4c4009aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 1 -- 172.21.15.67:0/4174216539 >> v1:172.21.15.17:6789/0 conn(0x7f4c4009b1f0 legacy=0x7f4c40133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 1 monclient: found mon.noname-a 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.715+0000 7f4c45bbd640 1 -- 172.21.15.67:0/4174216539 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4012efa0 con 0x7f4c4009d5a0 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c45bbd640 1 -- 172.21.15.67:0/4174216539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c4009d5a0 msgr2=0x7f4c40137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c45bbd640 1 --2- 172.21.15.67:0/4174216539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c4009d5a0 0x7f4c40137250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009e60 tx=0x7f4c38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e24e43cf7ae2d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 1 -- 172.21.15.67:0/4174216539 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38007630 con 0x7f4c4009d5a0 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: dump: 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 1 -- 172.21.15.67:0/4174216539 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3802d050 con 0x7f4c4009d5a0 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 1 -- 172.21.15.67:0/4174216539 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380079c0 con 0x7f4c4009d5a0 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c37fff640 10 monclient: dump: 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c47620640 5 monclient: authenticate success, global_id 84312 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c47620640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.718+0000 7f4c47620640 4 set_mon_vals no callback set 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 10 monclient: shutdown 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 -- 172.21.15.67:0/4174216539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c4009d5a0 msgr2=0x7f4c40137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 --2- 172.21.15.67:0/4174216539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c4009d5a0 0x7f4c40137250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009e60 tx=0x7f4c38009780 comp rx=0 tx=0).stop 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 -- 172.21.15.67:0/4174216539 shutdown_connections 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 --2- 172.21.15.67:0/4174216539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c4009d5a0 0x7f4c40137250 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 -- 172.21.15.67:0/4174216539 >> 172.21.15.67:0/4174216539 conn(0x7f4c40071340 msgr2=0x7f4c400a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 -- 172.21.15.67:0/4174216539 shutdown_connections 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 1 -- 172.21.15.67:0/4174216539 wait complete. 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 10 monclient: build_initial_monmap 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.719+0000 7f4c47620640 10 monclient: monmap: 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 Processor -- start 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 -- start start 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient: init 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-b con 0x7f4c4009b1f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-c con 0x7f4c4009d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 0x7f4c40119a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): picked mon.noname-a con 0x7f4c400a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4011c1b0 con 0x7f4c400a1a00 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4013f4b0 con 0x7f4c4009b1f0 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4011c6e0 con 0x7f4c4009d5a0 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c47620640 10 monclient(hunting): authenticate will time out at 1079.189771s 2024-09-15T01:57:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45182/0 (socket says 172.21.15.67:45182) 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 1 -- 172.21.15.67:0/1107122207 learned_addr learned my addr 172.21.15.67:0/1107122207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 10 monclient(hunting): get_auth_request con 0x7f4c4009b1f0 auth_method 0 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c453bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c45bbd640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 0x7f4c40119a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c45bbd640 10 monclient(hunting): get_auth_request con 0x7f4c400a1a00 auth_method 0 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c45bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c45bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c45bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.720+0000 7f4c44bbb640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 10 monclient(hunting): get_auth_request con 0x7f4c4009d5a0 auth_method 0 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c45bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 msgr2=0x7f4c4013f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 msgr2=0x7f4c4013f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:26.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c44bbb640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient(hunting): handle_auth_done global_id 5989 payload 824 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 msgr2=0x7f4c4013f780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 msgr2=0x7f4c40119a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 0x7f4c40119a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient: _finish_hunting 0 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 monclient: found mon.noname-b 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c453bc640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38009a20 con 0x7f4c4009b1f0 2024-09-15T01:57:26.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.721+0000 7f4c45bbd640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 0x7f4c40119a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c453bc640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 msgr2=0x7f4c4009b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c453bc640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c5e0 tx=0x7f4c3c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c005cb0 con 0x7f4c4009b1f0 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: dump: 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: _finish_auth 0 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c005e30 con 0x7f4c4009b1f0 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 5 monclient: authenticate success, global_id 5989 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 10 monclient: _renew_subs 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4009c880 con 0x7f4c4009b1f0 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c007320 con 0x7f4c4009b1f0 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c357fa640 10 monclient: dump: 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 10 monclient: _renew_subs 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4011be40 con 0x7f4c4009b1f0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.723+0000 7f4c377fe640 4 set_mon_vals no callback set 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.724+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4c3c01b3b0 con 0x7f4c4009b1f0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.724+0000 7f4c47620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.724+0000 7f4c47620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.724+0000 7f4c47620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.724+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4004bf60 con 0x7f4c4009b1f0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.725+0000 7f4c357fa640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 0x7f4c2403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.725+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f4c3c04c1e0 con 0x7f4c4009b1f0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.725+0000 7f4c44bbb640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 0x7f4c2403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.725+0000 7f4c44bbb640 10 monclient: get_auth_request con 0x7f4c24038740 auth_method 0 2024-09-15T01:57:26.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.727+0000 7f4c44bbb640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 0x7f4c2403abf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4c4012ec30 tx=0x7f4c30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:26.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.729+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4c3c018810 con 0x7f4c4009b1f0 2024-09-15T01:57:26.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.729+0000 7f4c357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:26.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.729+0000 7f4c357fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:26.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.895+0000 7f4c47620640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:26.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.895+0000 7f4c47620640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:26.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.895+0000 7f4c47620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:26.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:26.895+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f4c4012ecc0 con 0x7f4c4009b1f0 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6d09ac9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6d09ac9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6d09ac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6d04098b40 con 0x7f6d041319a0 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6ceb7fe640 1 -- 172.21.15.17:0/1129515315 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f6cf801dbb0 con 0x7f6d041319a0 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6ceb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.913+0000 7f6ceb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:26.917 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 msgr2=0x7f6cf003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 0x7f6cf003abf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007fd0 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 10 monclient: shutdown 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 msgr2=0x7f6d040970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800d890 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 shutdown_connections 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6cf0038740 0x7f6cf003abf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6d041319a0 0x7f6d040970a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6d04136cd0 0x7f6d040975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 --2- 172.21.15.17:0/1129515315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0409eac0 0x7f6d0409b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:26.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 >> 172.21.15.17:0/1129515315 conn(0x7f6d040a5000 msgr2=0x7f6d040a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:26.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 shutdown_connections 2024-09-15T01:57:26.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:26.915+0000 7f6d09ac9640 1 -- 172.21.15.17:0/1129515315 wait complete. 2024-09-15T01:57:26.929 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:57:26.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:57:26.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 10 monclient: get_monmap_and_config 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 10 monclient: build_initial_monmap 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 10 monclient: monmap: 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:27.025100+0000 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:27.025100+0000 2024-09-15T01:57:27.026 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:27.027 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 1 Processor -- start 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 1 -- start start 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.023+0000 7f5f5140f640 10 monclient: init 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f4c0a0280 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f4c09f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0a0040 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c09bff0 con 0x7f5f4c0a0280 2024-09-15T01:57:27.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0a14a0 con 0x7f5f4c131b20 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f5140f640 10 monclient(hunting): authenticate will time out at 1107.021423s 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5f4c09f1e0 0x7f5f4c138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40398/0 (socket says 172.21.15.17:40398) 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34280/0 (socket says 172.21.15.17:34280) 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2587588393 learned_addr learned my addr 172.21.15.17:0/2587588393 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 10 monclient(hunting): get_auth_request con 0x7f5f4c131b20 auth_method 0 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:27.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.024+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2587588393 >> v1:172.21.15.17:6789/0 conn(0x7f5f4c09f1e0 legacy=0x7f5f4c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173557039 0 0) 0x7f5f4c0a0040 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f49ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f49ffb640 10 monclient(hunting): my global_id is 84318 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f380034e0 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.026+0000 7f5f4a7fc640 1 -- 172.21.15.17:0/2587588393 >> v1:172.21.15.67:6789/0 conn(0x7f5f4c0a0280 legacy=0x7f5f4c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1984882433 0 0) 0x7f5f380034e0 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 >> v1:172.21.15.67:6789/0 conn(0x7f5f4c0a0280 legacy=0x7f5f4c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: _finish_hunting 0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c132600 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f5140f640 5 monclient: authenticate success, global_id 84318 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f5f3c001720 con 0x7f5f4c09f1e0 2024-09-15T01:57:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f5140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: dump: 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 >> v1:172.21.15.17:6789/0 conn(0x7f5f4c09f1e0 legacy=0x7f5f4c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f38003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): picked mon.c con 0x7f5f4c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f38004180 0x7f5f38004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): picked mon.a con 0x7f5f38004180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f38004ec0 0x7f5f3800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): picked mon.b con 0x7f5f38004ec0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0a14a0 con 0x7f5f38004180 2024-09-15T01:57:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3800bf10 con 0x7f5f38004ec0 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3800c050 con 0x7f5f4c131b20 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f49ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f5140f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f38003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f38004180 0x7f5f38004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): get_auth_request con 0x7f5f4c131b20 auth_method 0 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): get_auth_request con 0x7f5f38004180 auth_method 0 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f38004ec0 0x7f5f3800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f38004ec0 auth_method 0 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient(hunting): handle_auth_done global_id 84318 payload 293 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f38004ec0 msgr2=0x7f5f3800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f38004ec0 0x7f5f3800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f38004180 msgr2=0x7f5f38004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f38004180 0x7f5f38004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 monclient: found mon.c 2024-09-15T01:57:27.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380034e0 con 0x7f5f4c131b20 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 10 monclient: _finish_auth 0 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.027+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f38004180 0x7f5f38004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f4affd640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f38003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f38003b00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5f44007440 tx=0x7f5f44009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59c12bc74b3f20d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4402e660 con 0x7f5f4c131b20 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: dump: 2024-09-15T01:57:27.044 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f5140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4402e800 con 0x7f5f4c131b20 2024-09-15T01:57:27.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f5140f640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f5140f640 4 set_mon_vals no callback set 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 1 -- 172.21.15.17:0/2587588393 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4402ebc0 con 0x7f5f4c131b20 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.030+0000 7f5f49ffb640 10 monclient: dump: 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.046 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 10 monclient: shutdown 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 -- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f38003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 --2- 172.21.15.17:0/2587588393 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f38003b00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5f44007440 tx=0x7f5f44009720 comp rx=0 tx=0).stop 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2587588393 reap_dead start 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 -- 172.21.15.17:0/2587588393 shutdown_connections 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 -- 172.21.15.17:0/2587588393 >> 172.21.15.17:0/2587588393 conn(0x7f5f4c071270 msgr2=0x7f5f4c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 -- 172.21.15.17:0/2587588393 shutdown_connections 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 1 -- 172.21.15.17:0/2587588393 wait complete. 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 10 monclient: build_initial_monmap 2024-09-15T01:57:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.031+0000 7f5f5140f640 10 monclient: monmap: 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 Processor -- start 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 -- start start 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient: init 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4c131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c09b4c0 0x7f5f4c0992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4c09b4c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 0x7f5f4c140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4c099830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0a09b0 con 0x7f5f4c099830 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c130c70 con 0x7f5f4c09b4c0 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c11c620 con 0x7f5f4c131b20 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f5140f640 10 monclient(hunting): authenticate will time out at 1107.029712s 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 0x7f5f4c140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 0x7f5f4c140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53870/0 (socket says 172.21.15.17:53870) 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34300/0 (socket says 172.21.15.17:34300) 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 learned_addr learned my addr 172.21.15.17:0/2290964394 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f4c099830 auth_method 0 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 10 monclient(hunting): get_auth_request con 0x7f5f4c131b20 auth_method 0 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c09b4c0 0x7f5f4c0992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4a7fc640 10 monclient(hunting): get_auth_request con 0x7f5f4c09b4c0 auth_method 0 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c098db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c098db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.032+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 10 monclient(hunting): handle_auth_done global_id 84321 payload 824 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 msgr2=0x7f5f4c098db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5f4c131b20 0x7f5f4c098db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c09b4c0 msgr2=0x7f5f4c0992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c09b4c0 0x7f5f4c0992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:27.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 monclient: found mon.noname-a 2024-09-15T01:57:27.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:27.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.033+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c09ae00 con 0x7f5f4c099830 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 msgr2=0x7f5f4c140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f4b7fe640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 0x7f5f4c140f10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c001490 tx=0x7f5f3c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c001ce0 con 0x7f5f4c099830 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: dump: 2024-09-15T01:57:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c0076b0 con 0x7f5f4c099830 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 5 monclient: authenticate success, global_id 84321 2024-09-15T01:57:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 10 monclient: _renew_subs 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c09a950 con 0x7f5f4c099830 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c001720 con 0x7f5f4c099830 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f42ffd640 10 monclient: dump: 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.059 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 10 monclient: _renew_subs 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f497fa640 4 set_mon_vals no callback set 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.035+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c12f5e0 con 0x7f5f4c099830 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f5140f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f5140f640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 0x7f5f4c141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c063a40 con 0x7f5f4c141450 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5f3c013870 con 0x7f5f4c099830 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 0x7f5f4c141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient: get_auth_request con 0x7f5f4c141450 auth_method 0 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f42ffd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 0x7f5f2c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 10 monclient(hunting): handle_auth_done global_id 84324 payload 824 2024-09-15T01:57:27.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f4c0724a0 con 0x7f5f4c141450 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.036+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f5f3c04c350 con 0x7f5f4c099830 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 0x7f5f2c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 10 monclient: get_auth_request con 0x7f5f2c038790 auth_method 0 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 msgr2=0x7f5f2c03ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 msgr2=0x7f5f2c03ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 0x7f5f2c03ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.037+0000 7f5f4a7fc640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 0x7f5f2c03ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 msgr2=0x7f5f4c141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 0x7f5f4c141820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5f44035000 tx=0x7f5f44036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44045070 con 0x7f5f4c141450 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:27.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 10 monclient: dump: 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5f4402c8b0 con 0x7f5f4c141450 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:27.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 msgr2=0x7f5f4c141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 0x7f5f4c141820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5f44035000 tx=0x7f5f44036000 comp rx=0 tx=0).stop 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 msgr2=0x7f5f4c141820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.039+0000 7f5f42ffd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c141450 0x7f5f4c141820 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5f44035000 tx=0x7f5f44036000 comp rx=0 tx=0).stop 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f5140f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f5140f640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 0x7f5f4c09ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f5140f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c141d60 con 0x7f5f4c09ca40 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 0x7f5f4c09ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient: get_auth_request con 0x7f5f4c09ca40 auth_method 0 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 10 monclient(hunting): handle_auth_done global_id 84327 payload 824 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.045+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f4c06efc0 con 0x7f5f4c09ca40 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f4affd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 msgr2=0x7f5f4c09ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f4affd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 0x7f5f4c09ce40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f44007440 tx=0x7f5f4403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4402cea0 con 0x7f5f4c09ca40 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.047+0000 7f5f42ffd640 10 monclient: dump: 2024-09-15T01:57:27.065 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f5f4404e880 con 0x7f5f4c09ca40 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 msgr2=0x7f5f4c09ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 0x7f5f4c09ce40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f44007440 tx=0x7f5f4403f000 comp rx=0 tx=0).stop 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 msgr2=0x7f5f4c09ce40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f42ffd640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c09ca40 0x7f5f4c09ce40 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f44007440 tx=0x7f5f4403f000 comp rx=0 tx=0).stop 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:27.079 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:27.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 msgr2=0x7f5f2c03ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f2c038790 0x7f5f2c03ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 10 monclient: shutdown 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 msgr2=0x7f5f4c140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 --2- 172.21.15.17:0/2290964394 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f4c099830 0x7f5f4c140f10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c001490 tx=0x7f5f3c007a50 comp rx=0 tx=0).stop 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 reap_dead start 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f4b7fe640 1 -- 172.21.15.17:0/2290964394 reap_dead start 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 shutdown_connections 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 >> 172.21.15.17:0/2290964394 conn(0x7f5f4c071270 msgr2=0x7f5f4c130390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 shutdown_connections 2024-09-15T01:57:27.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.048+0000 7f5f5140f640 1 -- 172.21.15.17:0/2290964394 wait complete. 2024-09-15T01:57:27.082 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:27.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:57:27.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.176+0000 7fcef7b9d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.176+0000 7fcef7b9d640 10 monclient: build_initial_monmap 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.176+0000 7fcef7b9d640 10 monclient: monmap: 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:27.178231+0000 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:27.178231+0000 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:27.180 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 Processor -- start 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 -- start start 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient: init 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T01:57:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcef00a5780 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcef009ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcef009bc90 0x7fcef013a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcef009bc90 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef00a5b50 con 0x7fcef009ccf0 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef009c5f0 con 0x7fcef00a5780 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef00a18b0 con 0x7fcef009bc90 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef7b9d640 10 monclient(hunting): authenticate will time out at 1107.174804s 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef613a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcef009bc90 0x7fcef013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef5138640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fcef009ccf0 0x7fcef0069c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40402/0 (socket says 172.21.15.17:40402) 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef5138640 1 -- 172.21.15.17:0/3928071481 learned_addr learned my addr 172.21.15.17:0/3928071481 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:57:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef613a640 10 monclient(hunting): get_auth_request con 0x7fcef009bc90 auth_method 0 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef613a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef613a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.177+0000 7fcef613a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient(hunting): handle_auth_done global_id 84330 payload 274 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 1 -- 172.21.15.17:0/3928071481 >> v1:172.21.15.67:6789/0 conn(0x7fcef00a5780 legacy=0x7fcef0069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 1 -- 172.21.15.17:0/3928071481 >> v1:172.21.15.17:6789/0 conn(0x7fcef009ccf0 legacy=0x7fcef0069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient: _finish_hunting 0 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 1 monclient: found mon.noname-a 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.178+0000 7fcef613a640 1 -- 172.21.15.17:0/3928071481 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef006a3a0 con 0x7fcef009bc90 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef613a640 1 -- 172.21.15.17:0/3928071481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef009bc90 msgr2=0x7fcef013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef613a640 1 --2- 172.21.15.17:0/3928071481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef009bc90 0x7fcef013a890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009e60 tx=0x7fcee8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27633fb32c4bbbb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 1 -- 172.21.15.17:0/3928071481 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8007630 con 0x7fcef009bc90 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: dump: 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: _finish_auth 0 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 1 -- 172.21.15.17:0/3928071481 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee802d050 con 0x7fcef009bc90 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 5 monclient: authenticate success, global_id 84330 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 4 set_mon_vals no callback set 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 1 -- 172.21.15.17:0/3928071481 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80079c0 con 0x7fcef009bc90 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef4937640 10 monclient: dump: 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 10 monclient: shutdown 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 -- 172.21.15.17:0/3928071481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef009bc90 msgr2=0x7fcef013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/3928071481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef009bc90 0x7fcef013a890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009e60 tx=0x7fcee8009780 comp rx=0 tx=0).stop 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 -- 172.21.15.17:0/3928071481 shutdown_connections 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/3928071481 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef009bc90 0x7fcef013a890 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 -- 172.21.15.17:0/3928071481 >> 172.21.15.17:0/3928071481 conn(0x7fcef0070e10 msgr2=0x7fcef00a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.181+0000 7fcef7b9d640 1 -- 172.21.15.17:0/3928071481 shutdown_connections 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 1 -- 172.21.15.17:0/3928071481 wait complete. 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 10 monclient: build_initial_monmap 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 10 monclient: monmap: 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 1 Processor -- start 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 1 -- start start 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.182+0000 7fcef7b9d640 10 monclient: init 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcef009bc90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef009ccf0 0x7fcef013d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcef009ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 0x7fcef01411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcef00a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0122a90 con 0x7fcef00a5780 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef01226b0 con 0x7fcef009ccf0 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef01209a0 con 0x7fcef009bc90 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef7b9d640 10 monclient(hunting): authenticate will time out at 1107.180355s 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 0x7fcef01411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34304/0 (socket says 172.21.15.17:34304) 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 learned_addr learned my addr 172.21.15.17:0/945618155 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 10 monclient(hunting): get_auth_request con 0x7fcef009bc90 auth_method 0 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): get_auth_request con 0x7fcef00a5780 auth_method 0 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 msgr2=0x7fcef013dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 msgr2=0x7fcef013dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef009ccf0 0x7fcef013d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): get_auth_request con 0x7fcef009ccf0 auth_method 0 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef5138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient(hunting): handle_auth_done global_id 84333 payload 824 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 msgr2=0x7fcef013dc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef009bc90 0x7fcef013dc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef009ccf0 msgr2=0x7fcef013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef009ccf0 0x7fcef013d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient: _finish_hunting 0 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 monclient: found mon.noname-a 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.183+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8009a20 con 0x7fcef00a5780 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 msgr2=0x7fcef01411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef613a640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 0x7fcef01411f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcee8031000 tx=0x7fcee8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8007ec0 con 0x7fcef00a5780 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: dump: 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 5 monclient: authenticate success, global_id 84333 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee8040070 con 0x7fcef00a5780 2024-09-15T01:57:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 10 monclient: _renew_subs 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef0120190 con 0x7fcef00a5780 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8031d10 con 0x7fcef00a5780 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee1ffb640 10 monclient: dump: 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:27.203 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 10 monclient: _renew_subs 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcee3fff640 4 set_mon_vals no callback set 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.186+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef013e690 con 0x7fcef00a5780 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef7b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0122570 con 0x7fcef0121500 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcee8042030 con 0x7fcef00a5780 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 10 monclient: get_auth_request con 0x7fcef0121500 auth_method 0 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcee1ffb640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fced4038790 0x7fced403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fcee806e100 con 0x7fcef00a5780 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5138640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fced4038790 0x7fced403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.187+0000 7fcef5138640 10 monclient: get_auth_request con 0x7fced4038790 auth_method 0 2024-09-15T01:57:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.190+0000 7fcef5138640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fced4038790 0x7fced403ac40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcef0066470 tx=0x7fceec007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 10 monclient: get_auth_request con 0x7fcef0121500 auth_method 0 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:57:27.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.388+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:57:27.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.627+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v159) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4c3c017020 con 0x7f4c4009b1f0 2024-09-15T01:57:27.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.627+0000 7f4c357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:27.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.627+0000 7f4c357fa640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-15T01:57:27.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.730+0000 7f4c47620640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:27.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.730+0000 7f4c47620640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:27.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.730+0000 7f4c47620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:27.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:27.730+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f4c4009a870 con 0x7f4c4009b1f0 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 10 monclient: get_auth_request con 0x7fcef0121500 auth_method 0 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:57:27.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:27.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:27.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:27.789+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:57:28.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient: get_auth_request con 0x7fcef0121500 auth_method 0 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 10 monclient(hunting): handle_auth_done global_id 94100 payload 824 2024-09-15T01:57:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.590+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcef00724a0 con 0x7fcef0121500 2024-09-15T01:57:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.592+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.592+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcee4007ce0 tx=0x7fcee4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4007450 con 0x7fcef0121500 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 10 monclient: dump: 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcee40075b0 con 0x7fcef0121500 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcee4007ce0 tx=0x7fcee4009040 comp rx=0 tx=0).stop 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 msgr2=0x7fcef0141730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:28.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.593+0000 7fcee1ffb640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0121500 0x7fcef0141730 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.599+0000 7fcef7b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.599+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 0x7fcef011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0063b20 con 0x7fcef0141c70 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 0x7fcef011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient: get_auth_request con 0x7fcef0141c70 auth_method 0 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 10 monclient(hunting): handle_auth_done global_id 94103 payload 824 2024-09-15T01:57:28.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.600+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcef0142070 con 0x7fcef0141c70 2024-09-15T01:57:28.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.602+0000 7fcef5939640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 msgr2=0x7fcef011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.602+0000 7fcef5939640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 0x7fcef011b6d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcee401d000 tx=0x7fcee4020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4017880 con 0x7fcef0141c70 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 10 monclient: dump: 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7fcee4007240 con 0x7fcef0141c70 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:28.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 msgr2=0x7fcef011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 0x7fcef011b6d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcee401d000 tx=0x7fcee4020000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 msgr2=0x7fcef011b6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.603+0000 7fcee1ffb640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcef0141c70 0x7fcef011b6d0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcee401d000 tx=0x7fcee4020000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:57:28.608 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.609 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.610 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:28.611 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:28.612 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.613 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:28.614 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:28.617 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.618 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.619 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:28.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.621 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:28.621 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:28.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fced4038790 msgr2=0x7fced403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fced4038790 0x7fced403ac40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcef0066470 tx=0x7fceec007f00 comp rx=0 tx=0).stop 2024-09-15T01:57:28.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 10 monclient: shutdown 2024-09-15T01:57:28.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 msgr2=0x7fcef01411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 --2- 172.21.15.17:0/945618155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcef00a5780 0x7fcef01411f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcee8031000 tx=0x7fcee8037000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 reap_dead start 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef613a640 1 -- 172.21.15.17:0/945618155 reap_dead start 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 shutdown_connections 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 >> 172.21.15.17:0/945618155 conn(0x7fcef0070e10 msgr2=0x7fcef009a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 shutdown_connections 2024-09-15T01:57:28.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.604+0000 7fcef7b9d640 1 -- 172.21.15.17:0/945618155 wait complete. 2024-09-15T01:57:28.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:28.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:57:28.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:57:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.626+0000 7f4c357fa640 1 -- 172.21.15.67:0/1107122207 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v160) v1 ==== 257+0+0 (secure 0 0 0) 0x7f4c3c017070 con 0x7f4c4009b1f0 2024-09-15T01:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.626+0000 7f4c357fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T01:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.626+0000 7f4c357fa640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-15T01:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 msgr2=0x7f4c2403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 0x7f4c2403abf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4c4012ec30 tx=0x7f4c30008040 comp rx=0 tx=0).stop 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 10 monclient: shutdown 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 msgr2=0x7f4c4009b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c5e0 tx=0x7f4c3c004960 comp rx=0 tx=0).stop 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 shutdown_connections 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c24038740 0x7f4c2403abf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4c400a1a00 0x7f4c40119a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4c4009d5a0 0x7f4c4013f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 --2- 172.21.15.67:0/1107122207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c4009b1f0 0x7f4c4009b9a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 >> 172.21.15.67:0/1107122207 conn(0x7f4c40071340 msgr2=0x7f4c40131880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 shutdown_connections 2024-09-15T01:57:28.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.630+0000 7f4c47620640 1 -- 172.21.15.67:0/1107122207 wait complete. 2024-09-15T01:57:28.645 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:28.645 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:28.645 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:57:28.645 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '10\.' 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.723+0000 7fcf36c2b640 10 monclient: get_monmap_and_config 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.723+0000 7fcf36c2b640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.723+0000 7fcf36c2b640 10 monclient: monmap: 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.726 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:28.725131+0000 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:28.725131+0000 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:28.727 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.724+0000 7fcf36c2b640 1 Processor -- start 2024-09-15T01:57:28.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.724+0000 7fcf36c2b640 1 -- start start 2024-09-15T01:57:28.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.724+0000 7fcf36c2b640 10 monclient: init 2024-09-15T01:57:28.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf300a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 0x7fcf3009f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3009eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcf30136e40 0x7fcf30139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf30136e40 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3006a270 con 0x7fcf300a0b50 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3006f290 con 0x7fcf30136e40 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3009f7e0 con 0x7fcf3009eea0 2024-09-15T01:57:28.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf36c2b640 10 monclient(hunting): authenticate will time out at 1081.194329s 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf349c7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf300a0b50 0x7fcf30069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35154/0 (socket says 172.21.15.67:35154) 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf349c7640 1 -- 172.21.15.67:0/1310811813 learned_addr learned my addr 172.21.15.67:0/1310811813 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 0x7fcf3009f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 1 --2- 172.21.15.67:0/1310811813 >> v2:172.21.15.17:3300/0 conn(0x7fcf30136e40 0x7fcf30139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): get_auth_request con 0x7fcf3009eea0 auth_method 0 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): get_auth_request con 0x7fcf30136e40 auth_method 0 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_done global_id 94106 payload 274 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 -- 172.21.15.67:0/1310811813 >> v2:172.21.15.17:3300/0 conn(0x7fcf30136e40 msgr2=0x7fcf30139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/1310811813 >> v2:172.21.15.17:3300/0 conn(0x7fcf30136e40 0x7fcf30139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 -- 172.21.15.67:0/1310811813 >> v1:172.21.15.67:6789/0 conn(0x7fcf300a0b50 legacy=0x7fcf30069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 monclient: found mon.noname-b 2024-09-15T01:57:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.725+0000 7fcf2ffff640 1 -- 172.21.15.67:0/1310811813 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300a0f20 con 0x7fcf3009eea0 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.728+0000 7fcf2ffff640 1 -- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 msgr2=0x7fcf3009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.728+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 0x7fcf3009f2a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009850 tx=0x7fcf180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d82c49d32697ffde server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 1 -- 172.21.15.67:0/1310811813 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1802adf0 con 0x7fcf3009eea0 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:28.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: dump: 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 1 -- 172.21.15.67:0/1310811813 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18032440 con 0x7fcf3009eea0 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 5 monclient: authenticate success, global_id 94106 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 1 -- 172.21.15.67:0/1310811813 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180327e0 con 0x7fcf3009eea0 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 4 set_mon_vals no callback set 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf2f7fe640 10 monclient: dump: 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 10 monclient: shutdown 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 -- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 msgr2=0x7fcf3009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 0x7fcf3009f2a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009850 tx=0x7fcf180090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 -- 172.21.15.67:0/1310811813 shutdown_connections 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/1310811813 >> v2:172.21.15.17:3300/0 conn(0x7fcf30136e40 0x7fcf30139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/1310811813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3009eea0 0x7fcf3009f2a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 -- 172.21.15.67:0/1310811813 >> 172.21.15.67:0/1310811813 conn(0x7fcf30071290 msgr2=0x7fcf300a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 -- 172.21.15.67:0/1310811813 shutdown_connections 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.729+0000 7fcf36c2b640 1 -- 172.21.15.67:0/1310811813 wait complete. 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.730+0000 7fcf36c2b640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.730+0000 7fcf36c2b640 10 monclient: monmap: 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.730+0000 7fcf36c2b640 1 Processor -- start 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.730+0000 7fcf36c2b640 1 -- start start 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.730+0000 7fcf36c2b640 10 monclient: init 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3009eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 0x7fcf3011e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf300a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf30136e40 0x7fcf3011eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf30136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300a1260 con 0x7fcf300a0b50 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300a3900 con 0x7fcf3009eea0 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30121b90 con 0x7fcf30136e40 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf36c2b640 10 monclient(hunting): authenticate will time out at 1081.200426s 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45198/0 (socket says 172.21.15.67:45198) 2024-09-15T01:57:28.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 -- 172.21.15.67:0/2176206527 learned_addr learned my addr 172.21.15.67:0/2176206527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): get_auth_request con 0x7fcf3009eea0 auth_method 0 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 0x7fcf3011e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): get_auth_request con 0x7fcf300a0b50 auth_method 0 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf30136e40 0x7fcf3011eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): get_auth_request con 0x7fcf30136e40 auth_method 0 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf351c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient(hunting): handle_auth_done global_id 6001 payload 824 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 -- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf30136e40 msgr2=0x7fcf3011eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf30136e40 0x7fcf3011eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 -- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 msgr2=0x7fcf3011e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 0x7fcf3011e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 monclient: found mon.noname-b 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.731+0000 7fcf349c7640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18009390 con 0x7fcf3009eea0 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.732+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 0x7fcf3011e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf349c7640 1 -- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 msgr2=0x7fcf30135a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf349c7640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000c5e0 tx=0x7fcf20004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20005cb0 con 0x7fcf3009eea0 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: dump: 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 5 monclient: authenticate success, global_id 6001 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf20005e30 con 0x7fcf3009eea0 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 10 monclient: _renew_subs 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf301205e0 con 0x7fcf3009eea0 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20007340 con 0x7fcf3009eea0 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2cff9640 10 monclient: dump: 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 10 monclient: _renew_subs 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf30122200 con 0x7fcf3009eea0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.734+0000 7fcf2effd640 4 set_mon_vals no callback set 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf36c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf36c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf36c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcf2001b400 con 0x7fcf3009eea0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3004bf60 con 0x7fcf3009eea0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf2cff9640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 0x7fcf1003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7fcf2004ca10 con 0x7fcf3009eea0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 0x7fcf1003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.735+0000 7fcf2ffff640 10 monclient: get_auth_request con 0x7fcf10038770 auth_method 0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.738+0000 7fcf2ffff640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 0x7fcf1003ac20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcf3006a450 tx=0x7fcf18032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.740+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcf2001ab40 con 0x7fcf3009eea0 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.740+0000 7fcf2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.740+0000 7fcf2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:28.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.729+0000 7f4187354640 10 monclient: get_monmap_and_config 2024-09-15T01:57:28.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.729+0000 7f4187354640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.729+0000 7f4187354640 10 monclient: monmap: 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:28.731397+0000 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:28.731397+0000 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:28.766 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 Processor -- start 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 -- start start 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient: init 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f41800a5780 0x7f418009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): picked mon.noname-a con 0x7f41800a5780 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f418009ccf0 0x7f4180069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): picked mon.noname-c con 0x7f418009ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f4180069f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): picked mon.noname-b con 0x7f4180069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418006a460 con 0x7f41800a5780 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41800a18b0 con 0x7f418009ccf0 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41800a5b50 con 0x7f4180069ac0 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.730+0000 7f4187354640 10 monclient(hunting): authenticate will time out at 1108.727917s 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f41800a5780 0x7f418009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f41800a5780 0x7f418009bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53920/0 (socket says 172.21.15.17:53920) 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 -- 172.21.15.17:0/1065500609 learned_addr learned my addr 172.21.15.17:0/1065500609 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f4180069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): get_auth_request con 0x7f41800a5780 auth_method 0 2024-09-15T01:57:28.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 1 --2- 172.21.15.17:0/1065500609 >> v2:172.21.15.67:3300/0 conn(0x7f418009ccf0 0x7f4180069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): get_auth_request con 0x7f4180069ac0 auth_method 0 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): get_auth_request con 0x7f418009ccf0 auth_method 0 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient(hunting): handle_auth_done global_id 84342 payload 274 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 -- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 msgr2=0x7f4180069f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 --2- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f4180069f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 -- 172.21.15.17:0/1065500609 >> v2:172.21.15.67:3300/0 conn(0x7f418009ccf0 msgr2=0x7f4180069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 --2- 172.21.15.17:0/1065500609 >> v2:172.21.15.67:3300/0 conn(0x7f418009ccf0 0x7f4180069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 monclient: found mon.noname-a 2024-09-15T01:57:28.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.731+0000 7f41850f0640 1 -- 172.21.15.17:0/1065500609 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418009c5f0 con 0x7f41800a5780 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f41850f0640 1 -- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 msgr2=0x7f418009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f41850f0640 1 --2- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f418009bc90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4180068e70 tx=0x7f417c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a514b71cd1b33a42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 1 -- 172.21.15.17:0/1065500609 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c032440 con 0x7f41800a5780 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: dump: 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.771 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4187354640 5 monclient: authenticate success, global_id 84342 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 1 -- 172.21.15.17:0/1065500609 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c0325a0 con 0x7f41800a5780 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4187354640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 1 -- 172.21.15.17:0/1065500609 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c032940 con 0x7f41800a5780 2024-09-15T01:57:28.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4187354640 4 set_mon_vals no callback set 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.734+0000 7f4177fff640 10 monclient: dump: 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.773 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 10 monclient: shutdown 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 -- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 msgr2=0x7f418009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 --2- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f418009bc90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4180068e70 tx=0x7f417c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 -- 172.21.15.17:0/1065500609 shutdown_connections 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 --2- 172.21.15.17:0/1065500609 >> v2:172.21.15.67:3300/0 conn(0x7f418009ccf0 0x7f4180069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 --2- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f4180069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 --2- 172.21.15.17:0/1065500609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f418009bc90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 -- 172.21.15.17:0/1065500609 >> 172.21.15.17:0/1065500609 conn(0x7f4180070e10 msgr2=0x7f41800a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 -- 172.21.15.17:0/1065500609 shutdown_connections 2024-09-15T01:57:28.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 1 -- 172.21.15.17:0/1065500609 wait complete. 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.735+0000 7f4187354640 10 monclient: monmap: 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:28.775 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 Processor -- start 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 -- start start 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient: init 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f418013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): picked mon.noname-c con 0x7f4180069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418009ccf0 0x7f418013d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): picked mon.noname-b con 0x7f418009ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f4180141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): picked mon.noname-a con 0x7f41800a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4180122eb0 con 0x7f41800a5780 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41801228f0 con 0x7f418009ccf0 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41801203d0 con 0x7f4180069ac0 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f4187354640 10 monclient(hunting): authenticate will time out at 1108.733803s 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f418013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f418013dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34368/0 (socket says 172.21.15.17:34368) 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 1 -- 172.21.15.17:0/2447229274 learned_addr learned my addr 172.21.15.17:0/2447229274 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:28.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): get_auth_request con 0x7f4180069ac0 auth_method 0 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418009ccf0 0x7f418013d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41858f1640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f4180141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41848ef640 10 monclient(hunting): get_auth_request con 0x7f418009ccf0 auth_method 0 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41848ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41848ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41848ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41858f1640 10 monclient(hunting): get_auth_request con 0x7f41800a5780 auth_method 0 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41858f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41858f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41858f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.736+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41858f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 10 monclient(hunting): handle_auth_done global_id 94115 payload 824 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418009ccf0 msgr2=0x7f418013d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418009ccf0 0x7f418013d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 msgr2=0x7f4180141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f4180141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 monclient: found mon.noname-c 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41850f0640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009450 con 0x7f4180069ac0 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418009ccf0 0x7f418013d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.737+0000 7f41858f1640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f41800a5780 0x7f4180141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41850f0640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 msgr2=0x7f418013dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41850f0640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f418013dec0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f417c009940 tx=0x7f417c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0319c0 con 0x7f4180069ac0 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: dump: 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 5 monclient: authenticate success, global_id 94115 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 10 monclient: _renew_subs 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4180122610 con 0x7f4180069ac0 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c031b20 con 0x7f4180069ac0 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c03d440 con 0x7f4180069ac0 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41757fa640 10 monclient: dump: 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.782 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 10 monclient: _renew_subs 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f41777fe640 4 set_mon_vals no callback set 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.739+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418013dc50 con 0x7f4180069ac0 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f417c039040 con 0x7f4180069ac0 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f4187354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f4187354640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 0x7f4180141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4180063b20 con 0x7f4180121740 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41757fa640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4164038740 0x7f416403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f417c06cb80 con 0x7f4180069ac0 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 0x7f4180141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient: get_auth_request con 0x7f4180121740 auth_method 0 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41858f1640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4164038740 0x7f416403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41858f1640 10 monclient: get_auth_request con 0x7f4164038740 auth_method 0 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.740+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.741+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.741+0000 7f41848ef640 10 monclient(hunting): handle_auth_done global_id 6010 payload 824 2024-09-15T01:57:28.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.741+0000 7f41848ef640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4180123020 con 0x7f4180121740 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41858f1640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4164038740 0x7f416403abf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f418013e440 tx=0x7f417800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41848ef640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 msgr2=0x7f4180141970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 0x7f4180141970 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f418013ef30 tx=0x7f41700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170014860 con 0x7f4180121740 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 10 monclient: dump: 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.785 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f41700149c0 con 0x7f4180121740 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 msgr2=0x7f4180141970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 0x7f4180141970 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f418013ef30 tx=0x7f41700090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 msgr2=0x7f4180141970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.743+0000 7f41757fa640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4180121740 0x7f4180141970 secure :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f418013ef30 tx=0x7f41700090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f4187354640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f4187354640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 0x7f4180141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f4187354640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4180122b30 con 0x7f418013e840 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 0x7f4180141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f41848ef640 10 monclient: get_auth_request con 0x7f418013e840 auth_method 0 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f41848ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f41848ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.749+0000 7f41848ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.750+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.750+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.750+0000 7f41848ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.750+0000 7f41848ef640 10 monclient(hunting): handle_auth_done global_id 6013 payload 824 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.750+0000 7f41848ef640 1 -- 172.21.15.17:0/2447229274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f41800724a0 con 0x7f418013e840 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41848ef640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 msgr2=0x7f4180141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41848ef640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 0x7f4180141eb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f417001b000 tx=0x7f417001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170024040 con 0x7f418013e840 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.752+0000 7f41757fa640 10 monclient: dump: 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f417001cd10 con 0x7f418013e840 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 msgr2=0x7f4180141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 0x7f4180141eb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f417001b000 tx=0x7f417001c000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 msgr2=0x7f4180141eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:28.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41757fa640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f418013e840 0x7f4180141eb0 secure :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f417001b000 tx=0x7f417001c000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T01:57:28.790 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.791 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.792 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:28.793 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:28.794 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.796 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:28.798 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.799 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.800 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.801 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:28.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:28.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:28.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:28.803 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:28.803 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:28.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4164038740 msgr2=0x7f416403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4164038740 0x7f416403abf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f418013e440 tx=0x7f417800b040 comp rx=0 tx=0).stop 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 10 monclient: shutdown 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 msgr2=0x7f418013dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 --2- 172.21.15.17:0/2447229274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4180069ac0 0x7f418013dec0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f417c009940 tx=0x7f417c032000 comp rx=0 tx=0).stop 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41858f1640 1 -- 172.21.15.17:0/2447229274 reap_dead start 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f41858f1640 1 -- 172.21.15.17:0/2447229274 reap_dead start 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 shutdown_connections 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 >> 172.21.15.17:0/2447229274 conn(0x7f4180070e10 msgr2=0x7f41800a3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 shutdown_connections 2024-09-15T01:57:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.753+0000 7f4187354640 1 -- 172.21.15.17:0/2447229274 wait complete. 2024-09-15T01:57:28.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:28.806 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:57:28.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.899+0000 7f3a1299d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.899+0000 7f3a1299d640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.899+0000 7f3a1299d640 10 monclient: monmap: 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:28.901109+0000 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:28.901109+0000 2024-09-15T01:57:28.902 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:28.903 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.899+0000 7f3a1299d640 1 Processor -- start 2024-09-15T01:57:28.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 -- start start 2024-09-15T01:57:28.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient: init 2024-09-15T01:57:28.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 0x7f3a0c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c0a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c0980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a0c0989e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0a01e0 con 0x7f3a0c0989e0 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c0a1580 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0a5c80 con 0x7f3a0c0980a0 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a1299d640 10 monclient(hunting): authenticate will time out at 1108.897610s 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 0x7f3a0c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34370/0 (socket says 172.21.15.17:34370) 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/1011267777 learned_addr learned my addr 172.21.15.17:0/1011267777 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f3a0c0980a0 auth_method 0 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request con 0x7f3a0c0a58b0 auth_method 0 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_done global_id 94118 payload 274 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/1011267777 >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 msgr2=0x7f3a0c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 --2- 172.21.15.17:0/1011267777 >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 0x7f3a0c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/1011267777 >> v1:172.21.15.67:6789/0 conn(0x7f3a0c0989e0 legacy=0x7f3a0c068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.900+0000 7f3a0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.901+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/1011267777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c1326c0 con 0x7f3a0c0980a0 2024-09-15T01:57:28.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.901+0000 7f3a0bfff640 1 --2- 172.21.15.17:0/1011267777 >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 0x7f3a0c132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:28.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.910+0000 7fcf36c2b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:28.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.910+0000 7fcf36c2b640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:28.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.910+0000 7fcf36c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:28.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.910+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcf30120d00 con 0x7fcf3009eea0 2024-09-15T01:57:28.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.911+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcf2001a360 con 0x7fcf3009eea0 2024-09-15T01:57:28.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.911+0000 7fcf2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:28.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:28.911+0000 7fcf2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:28.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.903+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/1011267777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 msgr2=0x7f3a0c0984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.903+0000 7f3a0b7fe640 1 --2- 172.21.15.17:0/1011267777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009940 tx=0x7f3a000091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71bbe8da087a99ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 1 -- 172.21.15.17:0/1011267777 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00032440 con 0x7f3a0c0980a0 2024-09-15T01:57:28.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: dump: 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 5 monclient: authenticate success, global_id 94118 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 1 -- 172.21.15.17:0/1011267777 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a000325a0 con 0x7f3a0c0980a0 2024-09-15T01:57:28.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 1 -- 172.21.15.17:0/1011267777 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00032940 con 0x7f3a0c0980a0 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 4 set_mon_vals no callback set 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a0affd640 10 monclient: dump: 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.922 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 10 monclient: shutdown 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 -- 172.21.15.17:0/1011267777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 msgr2=0x7f3a0c0984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 --2- 172.21.15.17:0/1011267777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009940 tx=0x7f3a000091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 -- 172.21.15.17:0/1011267777 shutdown_connections 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 --2- 172.21.15.17:0/1011267777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0980a0 0x7f3a0c0984a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 --2- 172.21.15.17:0/1011267777 >> v2:172.21.15.17:3300/0 conn(0x7f3a0c0a58b0 0x7f3a0c132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 -- 172.21.15.17:0/1011267777 >> 172.21.15.17:0/1011267777 conn(0x7f3a0c070ed0 msgr2=0x7f3a0c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 -- 172.21.15.17:0/1011267777 shutdown_connections 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.904+0000 7f3a1299d640 1 -- 172.21.15.17:0/1011267777 wait complete. 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.905+0000 7f3a1299d640 10 monclient: build_initial_monmap 2024-09-15T01:57:28.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.905+0000 7f3a1299d640 10 monclient: monmap: 2024-09-15T01:57:28.925 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:28.925 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:28.925 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:28.925 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:28.925 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.905+0000 7f3a1299d640 1 Processor -- start 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.905+0000 7f3a1299d640 1 -- start start 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.905+0000 7f3a1299d640 10 monclient: init 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c0980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 0x7f3a0c130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a0c1311a0 0x7f3a0c11b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c1311a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c11dd80 con 0x7f3a0c0980a0 2024-09-15T01:57:28.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c11cc50 con 0x7f3a0c1311a0 2024-09-15T01:57:28.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:28.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c142190 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:28.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a1299d640 10 monclient(hunting): authenticate will time out at 1108.903484s 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 0x7f3a0c130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53952/0 (socket says 172.21.15.17:53952) 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 1 -- 172.21.15.17:0/3623907445 learned_addr learned my addr 172.21.15.17:0/3623907445 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request con 0x7f3a0c0980a0 auth_method 0 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f3a0c0a58b0 auth_method 0 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a0c1311a0 0x7f3a0c11b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): get_auth_request con 0x7f3a0c1311a0 auth_method 0 2024-09-15T01:57:28.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a10f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient(hunting): handle_auth_done global_id 94121 payload 824 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a0c1311a0 msgr2=0x7f3a0c11b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a0c1311a0 0x7f3a0c11b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 msgr2=0x7f3a0c09af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 monclient: found mon.noname-c 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/3623907445 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc009390 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.906+0000 7f3a0bfff640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a0b7fe640 1 -- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 msgr2=0x7f3a0c130c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a0b7fe640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 0x7f3a0c130c60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a00031000 tx=0x7f3a00032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00031a10 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: dump: 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.935 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:28.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:28.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00031b70 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:28.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0003d420 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:28.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f39f3fff640 10 monclient: dump: 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:28.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 5 monclient: authenticate success, global_id 94121 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 10 monclient: _renew_subs 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a09ffb640 4 set_mon_vals no callback set 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a00009450 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 10 monclient: _renew_subs 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.909+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c09eaa0 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a00039040 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f39f3fff640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 0x7f39f803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f3a0006c260 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a1299d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a1299d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a1299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:28.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c11e0c0 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a0bfff640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 0x7f39f803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.910+0000 7f3a0bfff640 10 monclient: get_auth_request con 0x7f39f8038740 auth_method 0 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.913+0000 7f3a0bfff640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 0x7f39f803abf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f39fc011640 tx=0x7f39fc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.915+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3a0c11e0c0 con 0x7f3a0c0a58b0 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.915+0000 7f39f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:28.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:28.915+0000 7f39f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf36c2b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf36c2b640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf36c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:29.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcf300008d0 con 0x7fcf3009eea0 2024-09-15T01:57:29.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf2cff9640 1 -- 172.21.15.67:0/2176206527 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcf2002a090 con 0x7fcf3009eea0 2024-09-15T01:57:29.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf2cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:29.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.014+0000 7fcf2cff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:29.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 msgr2=0x7fcf1003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 0x7fcf1003ac20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcf3006a450 tx=0x7fcf18032000 comp rx=0 tx=0).stop 2024-09-15T01:57:29.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 10 monclient: shutdown 2024-09-15T01:57:29.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 msgr2=0x7fcf30135a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000c5e0 tx=0x7fcf20004960 comp rx=0 tx=0).stop 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 shutdown_connections 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf10038770 0x7fcf1003ac20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf300a0b50 0x7fcf3011e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf30136e40 0x7fcf3011eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 --2- 172.21.15.67:0/2176206527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3009eea0 0x7fcf30135a80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 >> 172.21.15.67:0/2176206527 conn(0x7fcf30071290 msgr2=0x7fcf3009ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 shutdown_connections 2024-09-15T01:57:29.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.016+0000 7fcf36c2b640 1 -- 172.21.15.67:0/2176206527 wait complete. 2024-09-15T01:57:29.027 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:29.028 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:57:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '11\.' 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.101+0000 7f87be2ef640 10 monclient: get_monmap_and_config 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.101+0000 7f87be2ef640 10 monclient: build_initial_monmap 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.101+0000 7f87be2ef640 10 monclient: monmap: 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:29.104 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:29.103005+0000 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:29.103005+0000 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:29.105 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:29.106 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.102+0000 7f87be2ef640 1 Processor -- start 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.102+0000 7f87be2ef640 1 -- start start 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.102+0000 7f87be2ef640 10 monclient: init 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87b80a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-b con 0x7f87b809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 0x7f87b8139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-a con 0x7f87b8136e40 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b806a270 con 0x7f87b80a0b50 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b806f290 con 0x7f87b8136e40 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b809f7e0 con 0x7f87b809eea0 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:29.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87be2ef640 10 monclient(hunting): authenticate will time out at 1081.572190s 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 0x7f87b8139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:49490/0 (socket says 172.21.15.67:49490) 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 -- 172.21.15.67:0/1555445668 learned_addr learned my addr 172.21.15.67:0/1555445668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): get_auth_request con 0x7f87b809eea0 auth_method 0 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 0x7f87b8139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:33860/0 (socket says 172.21.15.67:33860) 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b7fff640 1 --1- 172.21.15.67:0/1555445668 >> v1:172.21.15.17:6789/0 conn(0x7f87b80a0b50 0x7f87b8069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:42600/0 (socket says 172.21.15.67:42600) 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): get_auth_request con 0x7f87b8136e40 auth_method 0 2024-09-15T01:57:29.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:29.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient(hunting): handle_auth_done global_id 94124 payload 274 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 -- 172.21.15.67:0/1555445668 >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 msgr2=0x7f87b8139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 --2- 172.21.15.67:0/1555445668 >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 0x7f87b8139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 -- 172.21.15.67:0/1555445668 >> v1:172.21.15.17:6789/0 conn(0x7f87b80a0b50 legacy=0x7f87b8069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 monclient: found mon.noname-b 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.103+0000 7f87b77fe640 1 -- 172.21.15.67:0/1555445668 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b80a0f20 con 0x7f87b809eea0 2024-09-15T01:57:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b77fe640 1 -- 172.21.15.67:0/1555445668 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 msgr2=0x7f87b809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b77fe640 1 --2- 172.21.15.67:0/1555445668 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009910 tx=0x7f87ac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63eb2abb38d9b9ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 1 -- 172.21.15.67:0/1555445668 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac02aea0 con 0x7f87b809eea0 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: dump: 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.118 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 1 -- 172.21.15.67:0/1555445668 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac032440 con 0x7f87b809eea0 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87be2ef640 5 monclient: authenticate success, global_id 94124 2024-09-15T01:57:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87be2ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87be2ef640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87be2ef640 4 set_mon_vals no callback set 2024-09-15T01:57:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 1 -- 172.21.15.67:0/1555445668 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac0327e0 con 0x7f87b809eea0 2024-09-15T01:57:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.106+0000 7f87b6ffd640 10 monclient: dump: 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:29.121 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 10 monclient: shutdown 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 -- 172.21.15.67:0/1555445668 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 msgr2=0x7f87b809f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 --2- 172.21.15.67:0/1555445668 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009910 tx=0x7f87ac0091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 -- 172.21.15.67:0/1555445668 shutdown_connections 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 --2- 172.21.15.67:0/1555445668 >> v2:172.21.15.17:3300/0 conn(0x7f87b8136e40 0x7f87b8139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 --2- 172.21.15.67:0/1555445668 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b809eea0 0x7f87b809f2a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 -- 172.21.15.67:0/1555445668 >> 172.21.15.67:0/1555445668 conn(0x7f87b8071290 msgr2=0x7f87b80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 -- 172.21.15.67:0/1555445668 shutdown_connections 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 1 -- 172.21.15.67:0/1555445668 wait complete. 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 10 monclient: build_initial_monmap 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.107+0000 7f87be2ef640 10 monclient: monmap: 2024-09-15T01:57:29.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 Processor -- start 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 -- start start 2024-09-15T01:57:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient: init 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-b con 0x7f87b809eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-c con 0x7f87b80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 0x7f87b811eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): picked mon.noname-a con 0x7f87b8136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8122240 con 0x7f87b8136e40 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b80a1260 con 0x7f87b809eea0 2024-09-15T01:57:29.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b80a3930 con 0x7f87b80a0b50 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87be2ef640 10 monclient(hunting): authenticate will time out at 1081.577692s 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87bc88c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 0x7f87b811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45206/0 (socket says 172.21.15.67:45206) 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 1 -- 172.21.15.67:0/2766995573 learned_addr learned my addr 172.21.15.67:0/2766995573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): get_auth_request con 0x7f87b809eea0 auth_method 0 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:29.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87bc88c640 10 monclient(hunting): get_auth_request con 0x7f87b8136e40 auth_method 0 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87bc88c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87bc88c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87bc88c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): get_auth_request con 0x7f87b80a0b50 auth_method 0 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:29.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.108+0000 7f87b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87bc88c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 10 monclient(hunting): handle_auth_done global_id 94127 payload 824 2024-09-15T01:57:29.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 -- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 msgr2=0x7f87b8135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 -- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 msgr2=0x7f87b811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 0x7f87b811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 monclient: found mon.noname-c 2024-09-15T01:57:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:29.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b77fe640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac009450 con 0x7f87b80a0b50 2024-09-15T01:57:29.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87b7fff640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:29.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.109+0000 7f87bc88c640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 0x7f87b811eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:29.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87b77fe640 1 -- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 msgr2=0x7f87b811e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:29.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87b77fe640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f87b806a730 tx=0x7f87ac032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:29.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac02aea0 con 0x7f87b80a0b50 2024-09-15T01:57:29.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:29.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: dump: 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.133 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: _finish_auth 0 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac006b30 con 0x7f87b80a0b50 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:29.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac03c400 con 0x7f87b80a0b50 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f879ffff640 10 monclient: dump: 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:29.135 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 5 monclient: authenticate success, global_id 94127 2024-09-15T01:57:29.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 10 monclient: _renew_subs 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b8120130 con 0x7f87b80a0b50 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87b67fc640 4 set_mon_vals no callback set 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 10 monclient: _renew_subs 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.111+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b8120d00 con 0x7f87b80a0b50 2024-09-15T01:57:29.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87b77fe640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f87b806a730 tx=0x7f87ac032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f87ac06d4a0 osd_map(160..160 src has 1..160) v4 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f87ac03c870 con 0x7f87b80a0b50 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87be2ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87be2ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87be2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b804bf60 con 0x7f87b80a0b50 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f879ffff640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 0x7f87a403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87b7fff640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 0x7f87a403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:29.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.112+0000 7f87b7fff640 10 monclient: get_auth_request con 0x7f87a4038770 auth_method 0 2024-09-15T01:57:29.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.115+0000 7f87b7fff640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 0x7f87a403ac20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007fd0 tx=0x7f87a8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:29.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.117+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f87ac06d4a0 con 0x7f87b80a0b50 2024-09-15T01:57:29.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.145+0000 7f3a1299d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:57:29.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.145+0000 7f3a1299d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:57:29.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.145+0000 7f3a1299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:29.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.145+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3a0c11e0c0 con 0x7f3a0c0a58b0 2024-09-15T01:57:29.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=3052264433,config=3140514009,health=1769316538,kv=947847570,logm=3303116181}) 2024-09-15T01:57:29.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=8,logm=62} crc {auth=3052264433,config=55734535,health=93589523,kv=1529054990,logm=3269997362}) 2024-09-15T01:57:29.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=3052264433,config=3140514009,health=1769316538,kv=947847570,logm=3303116181}) 2024-09-15T01:57:29.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.147+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=1,health=12,kv=8,logm=64} crc {auth=4005406800,config=1228700967,health=93589523,kv=1529054990,logm=3840490925}) 2024-09-15T01:57:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.147+0000 7f39f3fff640 1 -- 172.21.15.17:0/3623907445 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3a0c11e0c0 con 0x7f3a0c0a58b0 2024-09-15T01:57:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.147+0000 7f39f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:57:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.147+0000 7f39f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:29.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701207182}) 2024-09-15T01:57:29.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905217990}) 2024-09-15T01:57:29.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701207182}) 2024-09-15T01:57:29.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.148+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898390096}) 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 msgr2=0x7f39f803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 0x7f39f803abf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f39fc011640 tx=0x7f39fc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 10 monclient: shutdown 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 msgr2=0x7f3a0c130c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 0x7f3a0c130c60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a00031000 tx=0x7f3a00032000 comp rx=0 tx=0).stop 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 shutdown_connections 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39f8038740 0x7f39f803abf0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a0c1311a0 0x7f3a0c11b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a0c0a58b0 0x7f3a0c130c60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 --2- 172.21.15.17:0/3623907445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a0c0980a0 0x7f3a0c09af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 >> 172.21.15.17:0/3623907445 conn(0x7f3a0c070ed0 msgr2=0x7f3a0c134d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:29.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989143273}) 2024-09-15T01:57:29.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154003206}) 2024-09-15T01:57:29.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989143273}) 2024-09-15T01:57:29.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.149+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988572277}) 2024-09-15T01:57:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 shutdown_connections 2024-09-15T01:57:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:29.149+0000 7f3a1299d640 1 -- 172.21.15.17:0/3623907445 wait complete. 2024-09-15T01:57:29.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908852013}) 2024-09-15T01:57:29.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2158820454}) 2024-09-15T01:57:29.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908852013}) 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.150+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450055166}) 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342850983}) 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993437458}) 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342850983}) 2024-09-15T01:57:29.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.151+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644804073}) 2024-09-15T01:57:29.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:57:29.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774623554}) 2024-09-15T01:57:29.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502453685}) 2024-09-15T01:57:29.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774623554}) 2024-09-15T01:57:29.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061379666}) 2024-09-15T01:57:29.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089515105}) 2024-09-15T01:57:29.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76901564}) 2024-09-15T01:57:29.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089515105}) 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.488+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124432091}) 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172260943}) 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032869409}) 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172260943}) 2024-09-15T01:57:29.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.489+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237684106}) 2024-09-15T01:57:29.494 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383201448}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769085863}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383201448}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.490+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540052949}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743831702}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=17664312}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743831702}) 2024-09-15T01:57:29.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.491+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229665415}) 2024-09-15T01:57:29.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386787034}) 2024-09-15T01:57:29.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2683613163}) 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386787034}) 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.492+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=163660484}) 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861401277}) 2024-09-15T01:57:29.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259508005}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861401277}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.493+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281867589}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257188356}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227385774}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257188356}) 2024-09-15T01:57:29.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.494+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173439649}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137486854}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136742630}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137486854}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.495+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346151562}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343186411}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3361502481}) 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343186411}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737672530}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558159284}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=336244314}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558159284}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.496+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676970764}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835382937}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=9863307}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835382937}) 2024-09-15T01:57:29.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.497+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1272769390}) 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916814592}) 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895318205}) 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916814592}) 2024-09-15T01:57:29.504 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.498+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955025747}) 2024-09-15T01:57:29.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878485979}) 2024-09-15T01:57:29.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521014093}) 2024-09-15T01:57:29.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878485979}) 2024-09-15T01:57:29.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.627+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447500312}) 2024-09-15T01:57:29.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929599543}) 2024-09-15T01:57:29.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=868923379}) 2024-09-15T01:57:29.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929599543}) 2024-09-15T01:57:29.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.630+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3245322575}) 2024-09-15T01:57:29.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318866446}) 2024-09-15T01:57:29.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2019372926}) 2024-09-15T01:57:29.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318866446}) 2024-09-15T01:57:29.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.636+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666869048}) 2024-09-15T01:57:29.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.940+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f87ac03b3a0 con 0x7f87b80a0b50 2024-09-15T01:57:29.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.940+0000 7f879ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:29.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:29.940+0000 7f879ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:29.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595904456}) 2024-09-15T01:57:29.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071980928}) 2024-09-15T01:57:29.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595904456}) 2024-09-15T01:57:29.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.949+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=19972343}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212045722}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299926146}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212045722}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.950+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241440842}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073323958}) 2024-09-15T01:57:29.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3957634565}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073323958}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.951+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234820672}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538181395}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170196920}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538181395}) 2024-09-15T01:57:29.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:29.952+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470735028}) 2024-09-15T01:57:30.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841972229}) 2024-09-15T01:57:30.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258653916}) 2024-09-15T01:57:30.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841972229}) 2024-09-15T01:57:30.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.026+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2770504867}) 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223845318}) 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083211448}) 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223845318}) 2024-09-15T01:57:30.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.027+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964348106}) 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5} crc {logm=3143390364,mdsmap=1531151339,mgr=1085909001,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=81982431,mon_config_key=1583035879}) 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5,mon_config_key=4} crc {logm=2560076502,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2550716223,mon_config_key=3545683659}) 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5} crc {logm=3143390364,mdsmap=1531151339,mgr=1085909001,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=81982431,mon_config_key=1583035879}) 2024-09-15T01:57:30.033 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.028+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=718645741,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2229878038,mon_config_key=1386283410}) 2024-09-15T01:57:30.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1180979167,osd_snap=1063953654,osdmap=2925561275}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1180979167,osd_snap=1063953654,osdmap=665212137}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1180979167,osd_snap=1063953654,osdmap=2925561275}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.029+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1180979167,osd_snap=1063953654,osdmap=3478316313}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3594503110}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=605451552}) 2024-09-15T01:57:30.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3594503110}) 2024-09-15T01:57:30.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.031+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1900357390}) 2024-09-15T01:57:30.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=4044463415}) 2024-09-15T01:57:30.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=2237860725}) 2024-09-15T01:57:30.040 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:57:30.040 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=4044463415}) 2024-09-15T01:57:30.040 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:57:30.036+0000 7fe8b07f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=1616559050}) 2024-09-15T01:57:30.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.112+0000 7f87be2ef640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.112+0000 7f87be2ef640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.112+0000 7f87be2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.112+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f87b806f000 con 0x7f87b80a0b50 2024-09-15T01:57:30.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.114+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f87ac037030 con 0x7f87b80a0b50 2024-09-15T01:57:30.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.114+0000 7f879ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.114+0000 7f879ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:57:30.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:57:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.216+0000 7f87be2ef640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.216+0000 7f87be2ef640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.216+0000 7f87be2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.216+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f87b806f000 con 0x7f87b80a0b50 2024-09-15T01:57:30.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.217+0000 7f879ffff640 1 -- 172.21.15.67:0/2766995573 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f87ac037030 con 0x7f87b80a0b50 2024-09-15T01:57:30.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.217+0000 7f879ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.217+0000 7f879ffff640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 msgr2=0x7f87a403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 0x7f87a403ac20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007fd0 tx=0x7f87a8007f00 comp rx=0 tx=0).stop 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 10 monclient: shutdown 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 msgr2=0x7f87b811e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f87b806a730 tx=0x7f87ac032000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 shutdown_connections 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f87a4038770 0x7f87a403ac20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87b8136e40 0x7f87b811eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f87b80a0b50 0x7f87b811e5e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 --2- 172.21.15.67:0/2766995573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b809eea0 0x7f87b8135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 >> 172.21.15.67:0/2766995573 conn(0x7f87b8071290 msgr2=0x7f87b809ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 shutdown_connections 2024-09-15T01:57:30.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.219+0000 7f87be2ef640 1 -- 172.21.15.67:0/2766995573 wait complete. 2024-09-15T01:57:30.230 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:30.230 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:30.231 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:57:30.231 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:57:30.231 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:57:30.231 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '12\.' 2024-09-15T01:57:30.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.269+0000 7f19d191d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.269+0000 7f19d191d640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.269+0000 7f19d191d640 10 monclient: monmap: 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:30.271661+0000 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:30.271661+0000 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.274 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.274 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 Processor -- start 2024-09-15T01:57:30.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 -- start start 2024-09-15T01:57:30.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient: init 2024-09-15T01:57:30.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-b con 0x7f19cc0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19cc0980a0 0x7f19cc098470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-c con 0x7f19cc0980a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f19cc0989b0 0x7f19cc068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-a con 0x7f19cc0989b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0a01e0 con 0x7f19cc0989b0 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0a1580 con 0x7f19cc0980a0 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0a5c80 con 0x7f19cc0a58b0 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.270+0000 7f19d191d640 10 monclient(hunting): authenticate will time out at 1110.267992s 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f19cc0989b0 0x7f19cc068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f19cc0989b0 0x7f19cc068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53960/0 (socket says 172.21.15.17:53960) 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34392/0 (socket says 172.21.15.17:34392) 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 -- 172.21.15.17:0/894150173 learned_addr learned my addr 172.21.15.17:0/894150173 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): get_auth_request con 0x7f19cc0989b0 auth_method 0 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): get_auth_request con 0x7f19cc0a58b0 auth_method 0 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 1 --2- 172.21.15.17:0/894150173 >> v2:172.21.15.67:3300/0 conn(0x7f19cc0980a0 0x7f19cc098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): get_auth_request con 0x7f19cc0980a0 auth_method 0 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_done global_id 84360 payload 274 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 -- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 msgr2=0x7f19cc132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 -- 172.21.15.17:0/894150173 >> v2:172.21.15.67:3300/0 conn(0x7f19cc0980a0 msgr2=0x7f19cc098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 --2- 172.21.15.17:0/894150173 >> v2:172.21.15.67:3300/0 conn(0x7f19cc0980a0 0x7f19cc098470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 monclient: found mon.noname-a 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:30.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19cb7fe640 1 -- 172.21.15.17:0/894150173 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc1326c0 con 0x7f19cc0989b0 2024-09-15T01:57:30.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.271+0000 7f19caffd640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:30.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19cb7fe640 1 -- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0989b0 msgr2=0x7f19cc068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19cb7fe640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0989b0 0x7f19cc068ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0a5ff0 tx=0x7f19c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56c8a18dc0f38b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 1 -- 172.21.15.17:0/894150173 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4007610 con 0x7f19cc0989b0 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: dump: 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.288 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 1 -- 172.21.15.17:0/894150173 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c402d050 con 0x7f19cc0989b0 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19d191d640 5 monclient: authenticate success, global_id 84360 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19d191d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19d191d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 1 -- 172.21.15.17:0/894150173 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c40079e0 con 0x7f19cc0989b0 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19d191d640 4 set_mon_vals no callback set 2024-09-15T01:57:30.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.274+0000 7f19c9ffb640 10 monclient: dump: 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 10 monclient: shutdown 2024-09-15T01:57:30.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 -- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0989b0 msgr2=0x7f19cc068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0989b0 0x7f19cc068ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0a5ff0 tx=0x7f19c4009840 comp rx=0 tx=0).stop 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 -- 172.21.15.17:0/894150173 shutdown_connections 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 --2- 172.21.15.17:0/894150173 >> v2:172.21.15.67:3300/0 conn(0x7f19cc0980a0 0x7f19cc098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0989b0 0x7f19cc068ef0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 --2- 172.21.15.17:0/894150173 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0a58b0 0x7f19cc132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 -- 172.21.15.17:0/894150173 >> 172.21.15.17:0/894150173 conn(0x7f19cc070ed0 msgr2=0x7f19cc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 -- 172.21.15.17:0/894150173 shutdown_connections 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 1 -- 172.21.15.17:0/894150173 wait complete. 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.275+0000 7f19d191d640 10 monclient: monmap: 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 Processor -- start 2024-09-15T01:57:30.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 -- start start 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient: init 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-c con 0x7f19cc0980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0989b0 0x7f19cc1309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-b con 0x7f19cc0989b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0a58b0 0x7f19cc11b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): picked mon.noname-a con 0x7f19cc0a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc11db10 con 0x7f19cc0a58b0 2024-09-15T01:57:30.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc11e140 con 0x7f19cc0989b0 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc131790 con 0x7f19cc0980a0 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19d191d640 10 monclient(hunting): authenticate will time out at 1110.273766s 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34406/0 (socket says 172.21.15.17:34406) 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 1 -- 172.21.15.17:0/1234479031 learned_addr learned my addr 172.21.15.17:0/1234479031 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): get_auth_request con 0x7f19cc0980a0 auth_method 0 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19cb7fe640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0a58b0 0x7f19cc11b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19cb7fe640 10 monclient(hunting): get_auth_request con 0x7f19cc0a58b0 auth_method 0 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19ca7fc640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0989b0 0x7f19cc1309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19ca7fc640 10 monclient(hunting): get_auth_request con 0x7f19cc0989b0 auth_method 0 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.276+0000 7f19caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 10 monclient(hunting): handle_auth_done global_id 94133 payload 824 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 -- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0989b0 msgr2=0x7f19cc1309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0989b0 0x7f19cc1309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 -- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0a58b0 msgr2=0x7f19cc11b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0a58b0 0x7f19cc11b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 monclient: found mon.noname-c 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.277+0000 7f19caffd640 1 -- 172.21.15.17:0/1234479031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc009390 con 0x7f19cc0980a0 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19caffd640 1 -- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 msgr2=0x7f19cc09ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19caffd640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00ec80 tx=0x7f19bc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00eec0 con 0x7f19cc0980a0 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: dump: 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc00cd70 con 0x7f19cc0980a0 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 5 monclient: authenticate success, global_id 94133 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 10 monclient: _renew_subs 2024-09-15T01:57:30.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c4009ae0 con 0x7f19cc0980a0 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00cee0 con 0x7f19cc0980a0 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19b2ffd640 10 monclient: dump: 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 10 monclient: _renew_subs 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19c97fa640 4 set_mon_vals no callback set 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.279+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc136540 con 0x7f19cc0980a0 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f19bc013890 con 0x7f19cc0980a0 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19d191d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19d191d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19d191d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc0724a0 con 0x7f19cc0980a0 2024-09-15T01:57:30.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19b2ffd640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 0x7f19b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.280+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f19bc04d350 con 0x7f19cc0980a0 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.281+0000 7f19ca7fc640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 0x7f19b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.281+0000 7f19ca7fc640 10 monclient: get_auth_request con 0x7f19b8038740 auth_method 0 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.283+0000 7f19ca7fc640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 0x7f19b803abf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009c80 tx=0x7f19c00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.285+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f19bc0236b0 con 0x7f19cc0980a0 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.285+0000 7f19b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.285+0000 7f19b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.304+0000 7fcc44c78640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.304+0000 7fcc44c78640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.304+0000 7fcc44c78640 10 monclient: monmap: 2024-09-15T01:57:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:30.305558+0000 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:30.305558+0000 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.307 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.304+0000 7fcc44c78640 1 Processor -- start 2024-09-15T01:57:30.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 1 -- start start 2024-09-15T01:57:30.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient: init 2024-09-15T01:57:30.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 0x7fcc40069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-b con 0x7fcc400a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc4009eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc4009f290 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc400a61a0 con 0x7fcc4009eea0 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc400a04c0 con 0x7fcc4009f290 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400a1610 con 0x7fcc400a0b50 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc44c78640 10 monclient(hunting): authenticate will time out at 1082.774666s 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc4009f290 0x7fcc40132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34072/0 (socket says 172.21.15.67:34072) 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3ed9d640 1 -- 172.21.15.67:0/89831812 learned_addr learned my addr 172.21.15.67:0/89831812 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:30.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 1 --2- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 0x7fcc40069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): get_auth_request con 0x7fcc400a0b50 auth_method 0 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.305+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_done global_id 94136 payload 274 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 1 -- 172.21.15.67:0/89831812 >> v1:172.21.15.67:6789/0 conn(0x7fcc4009f290 legacy=0x7fcc40132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 1 -- 172.21.15.67:0/89831812 >> v1:172.21.15.17:6789/0 conn(0x7fcc4009eea0 legacy=0x7fcc40131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 1 monclient: found mon.noname-b 2024-09-15T01:57:30.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.306+0000 7fcc3e59c640 1 -- 172.21.15.67:0/89831812 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400a0f20 con 0x7fcc400a0b50 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3e59c640 1 -- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 msgr2=0x7fcc40069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3e59c640 1 --2- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 0x7fcc40069380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e90 tx=0x7fcc30009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fd260c7da58c14b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 1 -- 172.21.15.67:0/89831812 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30007840 con 0x7fcc400a0b50 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: dump: 2024-09-15T01:57:30.321 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 1 -- 172.21.15.67:0/89831812 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3002d050 con 0x7fcc400a0b50 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 1 -- 172.21.15.67:0/89831812 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30007bd0 con 0x7fcc400a0b50 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc3d59a640 10 monclient: dump: 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.324 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 5 monclient: authenticate success, global_id 94136 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 4 set_mon_vals no callback set 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 10 monclient: shutdown 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 1 -- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 msgr2=0x7fcc40069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.309+0000 7fcc44c78640 1 --2- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 0x7fcc40069380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e90 tx=0x7fcc30009780 comp rx=0 tx=0).stop 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 -- 172.21.15.67:0/89831812 shutdown_connections 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 --2- 172.21.15.67:0/89831812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc400a0b50 0x7fcc40069380 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 -- 172.21.15.67:0/89831812 >> 172.21.15.67:0/89831812 conn(0x7fcc40071330 msgr2=0x7fcc400a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 -- 172.21.15.67:0/89831812 shutdown_connections 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 -- 172.21.15.67:0/89831812 wait complete. 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 10 monclient: monmap: 2024-09-15T01:57:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 Processor -- start 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 1 -- start start 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.310+0000 7fcc44c78640 10 monclient: init 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4009eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 0x7fcc40130cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-c con 0x7fcc4009f290 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc400a0b50 0x7fcc40141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): picked mon.noname-a con 0x7fcc400a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4009b520 con 0x7fcc400a0b50 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400a1260 con 0x7fcc4009eea0 2024-09-15T01:57:30.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc40131ac0 con 0x7fcc4009f290 2024-09-15T01:57:30.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc44c78640 10 monclient(hunting): authenticate will time out at 1082.780384s 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34270/0 (socket says 172.21.15.67:34270) 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 -- 172.21.15.67:0/3588765468 learned_addr learned my addr 172.21.15.67:0/3588765468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): get_auth_request con 0x7fcc4009eea0 auth_method 0 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 0x7fcc40130cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc400a0b50 0x7fcc40141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): get_auth_request con 0x7fcc4009f290 auth_method 0 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc400a0b50 auth_method 0 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient(hunting): handle_auth_done global_id 6028 payload 824 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 -- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 msgr2=0x7fcc40130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 0x7fcc40130cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 -- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc400a0b50 msgr2=0x7fcc40141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc400a0b50 0x7fcc40141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 monclient: found mon.noname-b 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3e59c640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009a20 con 0x7fcc4009eea0 2024-09-15T01:57:30.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.311+0000 7fcc3dd9b640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 0x7fcc40130cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc3e59c640 1 -- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 msgr2=0x7fcc4009a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc3e59c640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009de0 tx=0x7fcc30037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30009460 con 0x7fcc4009eea0 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: dump: 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.336 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.337 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 5 monclient: authenticate success, global_id 6028 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30040070 con 0x7fcc4009eea0 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30031cf0 con 0x7fcc4009eea0 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc367fc640 10 monclient: dump: 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 10 monclient: _renew_subs 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc3cd99640 4 set_mon_vals no callback set 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4011dde0 con 0x7fcc4009eea0 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 10 monclient: _renew_subs 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.314+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4011c960 con 0x7fcc4009eea0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcc30042030 con 0x7fcc4009eea0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc44c78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc44c78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc44c78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4011e1e0 con 0x7fcc4009eea0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc367fc640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 0x7fcc2003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7fcc3006e770 con 0x7fcc4009eea0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc3dd9b640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 0x7fcc2003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.315+0000 7fcc3dd9b640 10 monclient: get_auth_request con 0x7fcc20038770 auth_method 0 2024-09-15T01:57:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.318+0000 7fcc3dd9b640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 0x7fcc2003ac20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcc4009b120 tx=0x7fcc28007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.320+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcc3003b420 con 0x7fcc4009eea0 2024-09-15T01:57:30.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.320+0000 7fcc367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.320+0000 7fcc367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.493+0000 7fcc44c78640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.493+0000 7fcc44c78640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.493+0000 7fcc44c78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.493+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcc4011e1e0 con 0x7fcc4009eea0 2024-09-15T01:57:30.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.494+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcc300443d0 con 0x7fcc4009eea0 2024-09-15T01:57:30.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.494+0000 7fcc367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.494+0000 7fcc367fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.512+0000 7f19d191d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:57:30.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.512+0000 7f19d191d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:30.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.512+0000 7f19d191d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.512+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f19cc09fc90 con 0x7f19cc0980a0 2024-09-15T01:57:30.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.513+0000 7f19b2ffd640 1 -- 172.21.15.17:0/1234479031 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f19bc0232f0 con 0x7f19cc0980a0 2024-09-15T01:57:30.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.513+0000 7f19b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:30.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.513+0000 7f19b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.516 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 msgr2=0x7f19b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 0x7f19b803abf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009c80 tx=0x7f19c00090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 10 monclient: shutdown 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 msgr2=0x7f19cc09ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.514+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00ec80 tx=0x7f19bc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 shutdown_connections 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f19b8038740 0x7f19b803abf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19cc0a58b0 0x7f19cc11b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0989b0 0x7f19cc1309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 --2- 172.21.15.17:0/1234479031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f19cc0980a0 0x7f19cc09ae20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 >> 172.21.15.17:0/1234479031 conn(0x7f19cc070ed0 msgr2=0x7f19cc09eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 shutdown_connections 2024-09-15T01:57:30.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.515+0000 7f19d191d640 1 -- 172.21.15.17:0/1234479031 wait complete. 2024-09-15T01:57:30.528 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:57:30.528 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:57:30.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:57:30.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:57:30.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.596+0000 7fcc44c78640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.596+0000 7fcc44c78640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.596+0000 7fcc44c78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.596+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcc4011e1e0 con 0x7fcc4009eea0 2024-09-15T01:57:30.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.597+0000 7fcc367fc640 1 -- 172.21.15.67:0/3588765468 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v160) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcc300445b0 con 0x7fcc4009eea0 2024-09-15T01:57:30.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.597+0000 7fcc367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:57:30.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.597+0000 7fcc367fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:30.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 msgr2=0x7fcc2003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 0x7fcc2003ac20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcc4009b120 tx=0x7fcc28007e10 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 10 monclient: shutdown 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 msgr2=0x7fcc4009a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009de0 tx=0x7fcc30037000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 shutdown_connections 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcc20038770 0x7fcc2003ac20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc400a0b50 0x7fcc40141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcc4009f290 0x7fcc40130cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 --2- 172.21.15.67:0/3588765468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc4009eea0 0x7fcc4009a950 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 >> 172.21.15.67:0/3588765468 conn(0x7fcc40071330 msgr2=0x7fcc40131dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 shutdown_connections 2024-09-15T01:57:30.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.599+0000 7fcc44c78640 1 -- 172.21.15.67:0/3588765468 wait complete. 2024-09-15T01:57:30.610 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:30.610 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T01:57:30.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-15T01:57:30.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.622+0000 7f2028d5d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.622+0000 7f2028d5d640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.622+0000 7f2028d5d640 10 monclient: monmap: 2024-09-15T01:57:30.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:30.624630+0000 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:30.624630+0000 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.626 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.627 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.627 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.623+0000 7f2028d5d640 1 Processor -- start 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.623+0000 7f2028d5d640 1 -- start start 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.623+0000 7f2028d5d640 10 monclient: init 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.623+0000 7f2028d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.623+0000 7f2028d5d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2024097320 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:30.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f202409a590 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f20241350f0 0x7f20241374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20241350f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240985d0 con 0x7f2024097320 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202409d200 con 0x7f202409a590 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20240992f0 con 0x7f20241350f0 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2028d5d640 10 monclient(hunting): authenticate will time out at 1110.621077s 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f20241350f0 0x7f20241374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f20241350f0 0x7f20241374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:53978/0 (socket says 172.21.15.17:53978) 2024-09-15T01:57:30.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 -- 172.21.15.17:0/2654962534 learned_addr learned my addr 172.21.15.17:0/2654962534 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): get_auth_request con 0x7f20241350f0 auth_method 0 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f202259c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2024097320 0x7f20240976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40404/0 (socket says 172.21.15.17:40404) 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient(hunting): handle_auth_done global_id 84369 payload 274 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 -- 172.21.15.17:0/2654962534 >> v1:172.21.15.67:6789/0 conn(0x7f202409a590 legacy=0x7f20241339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 -- 172.21.15.17:0/2654962534 >> v1:172.21.15.17:6789/0 conn(0x7f2024097320 legacy=0x7f20240976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 monclient: found mon.noname-a 2024-09-15T01:57:30.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:30.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.624+0000 7f2022d9d640 1 -- 172.21.15.17:0/2654962534 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202409cb90 con 0x7f20241350f0 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2022d9d640 1 -- 172.21.15.17:0/2654962534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 msgr2=0x7f20241374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2022d9d640 1 --2- 172.21.15.17:0/2654962534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f20241374d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2014009f20 tx=0x7f2014009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350dadd6d5e9fede server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 1 -- 172.21.15.17:0/2654962534 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201402aea0 con 0x7f20241350f0 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: dump: 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 1 -- 172.21.15.17:0/2654962534 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201402d050 con 0x7f20241350f0 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2028d5d640 5 monclient: authenticate success, global_id 84369 2024-09-15T01:57:30.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2028d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2028d5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 1 -- 172.21.15.17:0/2654962534 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014007840 con 0x7f20241350f0 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f2028d5d640 4 set_mon_vals no callback set 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.627+0000 7f202159a640 10 monclient: dump: 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.640 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 10 monclient: shutdown 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 -- 172.21.15.17:0/2654962534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 msgr2=0x7f20241374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 --2- 172.21.15.17:0/2654962534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f20241374d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2014009f20 tx=0x7f2014009840 comp rx=0 tx=0).stop 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 -- 172.21.15.17:0/2654962534 shutdown_connections 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 --2- 172.21.15.17:0/2654962534 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f20241374d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 -- 172.21.15.17:0/2654962534 >> 172.21.15.17:0/2654962534 conn(0x7f20240a1000 msgr2=0x7f20240a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 -- 172.21.15.17:0/2654962534 shutdown_connections 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 1 -- 172.21.15.17:0/2654962534 wait complete. 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.628+0000 7f2028d5d640 10 monclient: monmap: 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 Processor -- start 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 -- start start 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient: init 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024097320 0x7f202406d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2024097320 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 0x7f202406d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f202409a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f202406df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20241350f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024134530 con 0x7f20241350f0 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024118700 con 0x7f2024097320 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202409ad50 con 0x7f202409a590 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2028d5d640 10 monclient(hunting): authenticate will time out at 1110.626668s 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 0x7f202406d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 0x7f202406d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34410/0 (socket says 172.21.15.17:34410) 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 1 -- 172.21.15.17:0/1352001176 learned_addr learned my addr 172.21.15.17:0/1352001176 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): get_auth_request con 0x7f202409a590 auth_method 0 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f202406df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024097320 0x7f202406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): get_auth_request con 0x7f20241350f0 auth_method 0 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): get_auth_request con 0x7f2024097320 auth_method 0 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2021d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f2022d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.629+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 10 monclient(hunting): handle_auth_done global_id 94142 payload 824 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024097320 msgr2=0x7f202406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024097320 0x7f202406d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 msgr2=0x7f202406df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f20241350f0 0x7f202406df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 monclient: found mon.noname-c 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.630+0000 7f2021d9b640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2014009ae0 con 0x7f202409a590 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2021d9b640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 msgr2=0x7f202406d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2021d9b640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 0x7f202406d9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f201800c5e0 tx=0x7f2018007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018012840 con 0x7f202409a590 2024-09-15T01:57:30.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: dump: 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20180129c0 con 0x7f202409a590 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 5 monclient: authenticate success, global_id 94142 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018012d80 con 0x7f202409a590 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f20127fc640 10 monclient: dump: 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2020d99640 4 set_mon_vals no callback set 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 10 monclient: _renew_subs 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2024118cc0 con 0x7f202409a590 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 10 monclient: _renew_subs 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.632+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202412f4e0 con 0x7f202409a590 2024-09-15T01:57:30.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2018011590 con 0x7f202409a590 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2028d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2028d5d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 0x7f20240698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024063c20 con 0x7f202406e450 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f20127fc640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2004038740 0x7f200403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f201804d0c0 con 0x7f202409a590 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 0x7f20240698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient: get_auth_request con 0x7f202406e450 auth_method 0 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2022d9d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2004038740 0x7f200403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.633+0000 7f2022d9d640 10 monclient: get_auth_request con 0x7f2004038740 auth_method 0 2024-09-15T01:57:30.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.634+0000 7f202259c640 10 monclient(hunting): handle_auth_done global_id 84375 payload 824 2024-09-15T01:57:30.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.634+0000 7f202259c640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20240724a0 con 0x7f202406e450 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.636+0000 7f202259c640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 msgr2=0x7f20240698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.636+0000 7f202259c640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 0x7f20240698d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.636+0000 7f2022d9d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2004038740 0x7f200403abf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2024098070 tx=0x7f2014036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.637+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c005050 con 0x7f202406e450 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.637+0000 7f20127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.637+0000 7f20127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.637+0000 7f20127fc640 10 monclient: dump: 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.653 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f201c0051b0 con 0x7f202406e450 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 msgr2=0x7f20240698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 0x7f20240698d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c007e40 comp rx=0 tx=0).stop 2024-09-15T01:57:30.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 msgr2=0x7f20240698d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.647+0000 7f20127fc640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f202406e450 0x7f20240698d0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c007e40 comp rx=0 tx=0).stop 2024-09-15T01:57:30.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.653+0000 7f2028d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:30.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.653+0000 7f2028d5d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 0x7f202406a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.653+0000 7f2028d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.653+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20241184e0 con 0x7f2024069e10 2024-09-15T01:57:30.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 0x7f202406a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient: get_auth_request con 0x7f2024069e10 auth_method 0 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 10 monclient(hunting): handle_auth_done global_id 84378 payload 824 2024-09-15T01:57:30.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.654+0000 7f202259c640 1 -- 172.21.15.17:0/1352001176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f202406f610 con 0x7f2024069e10 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f202259c640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 msgr2=0x7f202406a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f202259c640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 0x7f202406a210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c01d610 con 0x7f2024069e10 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 10 monclient: dump: 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f201c025070 con 0x7f2024069e10 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 msgr2=0x7f202406a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 0x7f202406a210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c019000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 msgr2=0x7f202406a210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.656+0000 7f20127fc640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2024069e10 0x7f202406a210 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e740 tx=0x7f201c019000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:30.662 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:30.663 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:30.664 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.665 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:30.666 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.667 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.669 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.671 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.673 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.674 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:30.675 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:30.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2004038740 msgr2=0x7f200403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2004038740 0x7f200403abf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2024098070 tx=0x7f2014036000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 10 monclient: shutdown 2024-09-15T01:57:30.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 msgr2=0x7f202406d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 --2- 172.21.15.17:0/1352001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f202409a590 0x7f202406d9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f201800c5e0 tx=0x7f2018007ed0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2022d9d640 1 -- 172.21.15.17:0/1352001176 reap_dead start 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2022d9d640 1 -- 172.21.15.17:0/1352001176 reap_dead start 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 shutdown_connections 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 >> 172.21.15.17:0/1352001176 conn(0x7f20240a1000 msgr2=0x7f202409b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 shutdown_connections 2024-09-15T01:57:30.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.657+0000 7f2028d5d640 1 -- 172.21.15.17:0/1352001176 wait complete. 2024-09-15T01:57:30.677 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:30.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:57:30.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.685+0000 7fb3bf0dd640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.685+0000 7fb3bf0dd640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.685+0000 7fb3bf0dd640 10 monclient: monmap: 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:30.687036+0000 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:30.687036+0000 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.689 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.689 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.689 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.689 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.689 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.686+0000 7fb3bf0dd640 1 Processor -- start 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.686+0000 7fb3bf0dd640 1 -- start start 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.686+0000 7fb3bf0dd640 10 monclient: init 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b8069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3b809eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:30.692 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3b809f270 0x7fb3b8132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b809f270 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b806a270 con 0x7fb3b809eea0 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b806f290 con 0x7fb3b809f270 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b80a0f20 con 0x7fb3b80a0b50 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bf0dd640 10 monclient(hunting): authenticate will time out at 1083.156256s 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3b809f270 0x7fb3b8132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3b809f270 0x7fb3b8132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34278/0 (socket says 172.21.15.67:34278) 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 -- 172.21.15.67:0/4194191539 learned_addr learned my addr 172.21.15.67:0/4194191539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): get_auth_request con 0x7fb3b809f270 auth_method 0 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3b7fff640 1 --1- 172.21.15.67:0/4194191539 >> v1:172.21.15.17:6789/0 conn(0x7fb3b809eea0 0x7fb3b8131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:36004/0 (socket says 172.21.15.67:36004) 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): get_auth_request con 0x7fb3b80a0b50 auth_method 0 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_done global_id 6034 payload 274 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 -- 172.21.15.67:0/4194191539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 msgr2=0x7fb3b8069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 --2- 172.21.15.67:0/4194191539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b8069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 -- 172.21.15.67:0/4194191539 >> v1:172.21.15.17:6789/0 conn(0x7fb3b809eea0 legacy=0x7fb3b8131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 monclient: found mon.noname-c 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.687+0000 7fb3bd67a640 1 -- 172.21.15.67:0/4194191539 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80a6160 con 0x7fb3b809f270 2024-09-15T01:57:30.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bd67a640 1 -- 172.21.15.67:0/4194191539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 msgr2=0x7fb3b8132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bd67a640 1 --2- 172.21.15.67:0/4194191539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b8132220 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009f20 tx=0x7fb3b0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ceed34e932ae12b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.701 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 1 -- 172.21.15.67:0/4194191539 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007610 con 0x7fb3b809f270 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: dump: 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.702 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 1 -- 172.21.15.67:0/4194191539 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b002d050 con 0x7fb3b809f270 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bf0dd640 5 monclient: authenticate success, global_id 6034 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bf0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 1 -- 172.21.15.67:0/4194191539 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00079e0 con 0x7fb3b809f270 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3b77fe640 10 monclient: dump: 2024-09-15T01:57:30.703 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bf0dd640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.690+0000 7fb3bf0dd640 4 set_mon_vals no callback set 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 10 monclient: shutdown 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/4194191539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 msgr2=0x7fb3b8132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/4194191539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b8132220 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009f20 tx=0x7fb3b0009840 comp rx=0 tx=0).stop 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/4194191539 shutdown_connections 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/4194191539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b8132220 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/4194191539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b8069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/4194191539 >> 172.21.15.67:0/4194191539 conn(0x7fb3b8071290 msgr2=0x7fb3b80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/4194191539 shutdown_connections 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/4194191539 wait complete. 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.705 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.691+0000 7fb3bf0dd640 10 monclient: monmap: 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 Processor -- start 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 -- start start 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient: init 2024-09-15T01:57:30.706 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3b809eea0 0x7fb3b8133be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b809f270 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b811ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b80a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8121ff0 con 0x7fb3b809eea0 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8120730 con 0x7fb3b809f270 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.707 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8141c80 con 0x7fb3b80a0b50 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bf0dd640 10 monclient(hunting): authenticate will time out at 1083.161760s 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34288/0 (socket says 172.21.15.67:34288) 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 1 -- 172.21.15.67:0/3950720171 learned_addr learned my addr 172.21.15.67:0/3950720171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): get_auth_request con 0x7fb3b809f270 auth_method 0 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bce79640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3b809eea0 0x7fb3b8133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bd67a640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b811ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bce79640 10 monclient(hunting): get_auth_request con 0x7fb3b809eea0 auth_method 0 2024-09-15T01:57:30.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bce79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bce79640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bce79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bd67a640 10 monclient(hunting): get_auth_request con 0x7fb3b80a0b50 auth_method 0 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bd67a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bd67a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3bd67a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.692+0000 7fb3b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.709 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bce79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3bd67a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 10 monclient(hunting): handle_auth_done global_id 6037 payload 824 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 -- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 msgr2=0x7fb3b811ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b811ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 -- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3b809eea0 msgr2=0x7fb3b8133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3b809eea0 0x7fb3b8133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 monclient: found mon.noname-b 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.693+0000 7fb3b7fff640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0009ae0 con 0x7fb3b809f270 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b7fff640 1 -- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 msgr2=0x7fb3b811e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b7fff640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c5e0 tx=0x7fb3a4007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4007510 con 0x7fb3b809f270 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: dump: 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.711 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a40057d0 con 0x7fb3b809f270 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4007690 con 0x7fb3b809f270 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 5 monclient: authenticate success, global_id 6037 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 10 monclient: _renew_subs 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b80a1260 con 0x7fb3b809f270 2024-09-15T01:57:30.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b6ffd640 4 set_mon_vals no callback set 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3b4ff9640 10 monclient: dump: 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 10 monclient: _renew_subs 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.695+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b8120db0 con 0x7fb3b809f270 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bf0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb3a401b400 con 0x7fb3b809f270 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bf0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bf0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b81223d0 con 0x7fb3b809f270 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3b4ff9640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 0x7fb39c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7fb3a404ca20 con 0x7fb3b809f270 2024-09-15T01:57:30.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bce79640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 0x7fb39c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.696+0000 7fb3bce79640 10 monclient: get_auth_request con 0x7fb39c038740 auth_method 0 2024-09-15T01:57:30.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.699+0000 7fb3bce79640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 0x7fb39c03abf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009d20 tx=0x7fb3ac0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.701+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb3a4005e60 con 0x7fb3b809f270 2024-09-15T01:57:30.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.701+0000 7fb3b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.701+0000 7fb3b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.774+0000 7f1a95489640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.774+0000 7f1a95489640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.774+0000 7f1a95489640 10 monclient: monmap: 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:30.776694+0000 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:30.776694+0000 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.779 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.779 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.779 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.779 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 1 Processor -- start 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 1 -- start start 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 10 monclient: init 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:57:30.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.775+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-b con 0x7f1a90068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 0x7f1a900696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-c con 0x7f1a900692f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a9012e900 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a900a1950 con 0x7f1a9012e900 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a900a1670 con 0x7f1a900692f0 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a900a0020 con 0x7f1a90068f20 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a95489640 10 monclient(hunting): authenticate will time out at 1110.773070s 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34416/0 (socket says 172.21.15.17:34416) 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 -- 172.21.15.17:0/3410844021 learned_addr learned my addr 172.21.15.17:0/3410844021 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): get_auth_request con 0x7f1a90068f20 auth_method 0 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/3410844021 >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 0x7f1a900696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f1a900692f0 auth_method 0 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient(hunting): handle_auth_done global_id 94151 payload 274 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 -- 172.21.15.17:0/3410844021 >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 msgr2=0x7f1a900696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 --2- 172.21.15.17:0/3410844021 >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 0x7f1a900696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 -- 172.21.15.17:0/3410844021 >> v1:172.21.15.67:6789/0 conn(0x7f1a9012e900 legacy=0x7f1a9012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 monclient: found mon.noname-b 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8effd640 1 -- 172.21.15.17:0/3410844021 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a900a1390 con 0x7f1a90068f20 2024-09-15T01:57:30.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.776+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/3410844021 >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 0x7f1a900696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8effd640 1 -- 172.21.15.17:0/3410844021 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 msgr2=0x7f1a901321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8effd640 1 --2- 172.21.15.17:0/3410844021 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009f20 tx=0x7f1a80009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32fde88e2b669aa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 1 -- 172.21.15.17:0/3410844021 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80007610 con 0x7f1a90068f20 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: dump: 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 5 monclient: authenticate success, global_id 94151 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 1 -- 172.21.15.17:0/3410844021 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8002d050 con 0x7f1a90068f20 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 4 set_mon_vals no callback set 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 1 -- 172.21.15.17:0/3410844021 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800079a0 con 0x7f1a90068f20 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a8dffb640 10 monclient: dump: 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.793 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 10 monclient: shutdown 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 1 -- 172.21.15.17:0/3410844021 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 msgr2=0x7f1a901321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.779+0000 7f1a95489640 1 --2- 172.21.15.17:0/3410844021 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009f20 tx=0x7f1a80009840 comp rx=0 tx=0).stop 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 -- 172.21.15.17:0/3410844021 shutdown_connections 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 --2- 172.21.15.17:0/3410844021 >> v2:172.21.15.67:3300/0 conn(0x7f1a900692f0 0x7f1a900696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 --2- 172.21.15.17:0/3410844021 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a90068f20 0x7f1a901321c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 -- 172.21.15.17:0/3410844021 >> 172.21.15.17:0/3410844021 conn(0x7f1a90070fb0 msgr2=0x7f1a900a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 -- 172.21.15.17:0/3410844021 shutdown_connections 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 -- 172.21.15.17:0/3410844021 wait complete. 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 10 monclient: monmap: 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.780+0000 7f1a95489640 1 Processor -- start 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 -- start start 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient: init 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-b con 0x7f1a90068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 0x7f1a9012dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-a con 0x7f1a900692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 0x7f1a90097f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): picked mon.noname-c con 0x7f1a9012e900 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90130c40 con 0x7f1a900692f0 2024-09-15T01:57:30.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a900a0750 con 0x7f1a90068f20 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90141f20 con 0x7f1a9012e900 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a95489640 10 monclient(hunting): authenticate will time out at 1110.778467s 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 0x7f1a90097f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 0x7f1a9012dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:60596/0 (socket says 172.21.15.17:60596) 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 0x7f1a90097f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34424/0 (socket says 172.21.15.17:34424) 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 -- 172.21.15.17:0/333870199 learned_addr learned my addr 172.21.15.17:0/333870199 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 0x7f1a9012dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54006/0 (socket says 172.21.15.17:54006) 2024-09-15T01:57:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): get_auth_request con 0x7f1a90068f20 auth_method 0 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f1a900692f0 auth_method 0 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a9012e900 auth_method 0 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient(hunting): handle_auth_done global_id 6043 payload 824 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 msgr2=0x7f1a90097f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 0x7f1a90097f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 msgr2=0x7f1a9012dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 0x7f1a9012dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 monclient: found mon.noname-b 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8effd640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80009ae0 con 0x7f1a90068f20 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8f7fe640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e900 0x7f1a90097f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.781+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a900692f0 0x7f1a9012dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a8effd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 msgr2=0x7f1a901410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a8effd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009f20 tx=0x7f1a80036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800328b0 con 0x7f1a90068f20 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: dump: 2024-09-15T01:57:30.800 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 5 monclient: authenticate success, global_id 6043 2024-09-15T01:57:30.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 10 monclient: _renew_subs 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a80032a30 con 0x7f1a90068f20 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9009a5d0 con 0x7f1a90068f20 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80032df0 con 0x7f1a90068f20 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a86ffd640 10 monclient: dump: 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 10 monclient: _renew_subs 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a8d7fa640 4 set_mon_vals no callback set 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.784+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a90099a00 con 0x7f1a90068f20 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a95489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a95489640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 0x7f1a90098490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9009b6d0 con 0x7f1a9012e110 2024-09-15T01:57:30.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8effd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009f20 tx=0x7f1a80036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f1a80038050 mgrmap(e 5) v1 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 0x7f1a90098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1a80038050 con 0x7f1a90068f20 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient: get_auth_request con 0x7f1a9012e110 auth_method 0 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_done global_id 94157 payload 824 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8e7fc640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a90099f90 con 0x7f1a9012e110 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a86ffd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a70038740 0x7f1a7003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8f7fe640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a70038740 0x7f1a7003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.785+0000 7f1a8f7fe640 10 monclient: get_auth_request con 0x7f1a70038740 auth_method 0 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.787+0000 7f1a8e7fc640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 msgr2=0x7f1a90098490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.787+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 0x7f1a90098490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009b460 tx=0x7f1a78008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800eec0 con 0x7f1a9012e110 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 10 monclient: dump: 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.805 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1a78010ea0 con 0x7f1a9012e110 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 msgr2=0x7f1a90098490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 0x7f1a90098490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009b460 tx=0x7f1a78008040 comp rx=0 tx=0).stop 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 msgr2=0x7f1a90098490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a86ffd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a9012e110 0x7f1a90098490 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009b460 tx=0x7f1a78008040 comp rx=0 tx=0).stop 2024-09-15T01:57:30.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.788+0000 7f1a8f7fe640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a70038740 0x7f1a7003abf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009aa70 tx=0x7f1a88007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.793+0000 7f1a95489640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a95489640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 0x7f1a9009c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a95489640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90063b90 con 0x7f1a900989d0 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 0x7f1a9009c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient: get_auth_request con 0x7f1a900989d0 auth_method 0 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 10 monclient(hunting): handle_auth_done global_id 94160 payload 824 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.794+0000 7f1a8e7fc640 1 -- 172.21.15.17:0/333870199 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1a9009b8a0 con 0x7f1a900989d0 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a8e7fc640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 msgr2=0x7f1a9009c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a8e7fc640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 0x7f1a9009c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009b460 tx=0x7f1a7801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7801ddd0 con 0x7f1a900989d0 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 10 monclient: dump: 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.809 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4395 (secure 0 0 0) 0x7f1a78023040 con 0x7f1a900989d0 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 msgr2=0x7f1a9009c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 0x7f1a9009c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009b460 tx=0x7f1a7801d000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.796+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 msgr2=0x7f1a9009c550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a86ffd640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a900989d0 0x7f1a9009c550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:57:30.811 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.812 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.813 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:30.814 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.815 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.816 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.818 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.820 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.821 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.822 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:30.823 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a70038740 msgr2=0x7f1a7003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a70038740 0x7f1a7003abf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1a9009aa70 tx=0x7f1a88007e10 comp rx=0 tx=0).stop 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 10 monclient: shutdown 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 msgr2=0x7f1a901410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 --2- 172.21.15.17:0/333870199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a90068f20 0x7f1a901410d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009f20 tx=0x7f1a80036000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a8f7fe640 1 -- 172.21.15.17:0/333870199 reap_dead start 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a86ffd640 1 -- 172.21.15.17:0/333870199 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (unknown 0 0 0) 0x7f1a8006d8c0 con 0x7f1a90068f20 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a8f7fe640 1 -- 172.21.15.17:0/333870199 reap_dead start 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 shutdown_connections 2024-09-15T01:57:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 >> 172.21.15.17:0/333870199 conn(0x7f1a90070fb0 msgr2=0x7f1a90130e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 shutdown_connections 2024-09-15T01:57:30.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.797+0000 7f1a95489640 1 -- 172.21.15.17:0/333870199 wait complete. 2024-09-15T01:57:30.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:30.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:57:30.826 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:57:30.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.870+0000 7fb3bf0dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:30.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.870+0000 7fb3bf0dd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:30.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.870+0000 7fb3bf0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:30.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:30.870+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7fb3b804bf60 con 0x7fb3b809f270 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 10 monclient: get_monmap_and_config 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 10 monclient: monmap: 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:30.922086+0000 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:30.922086+0000 2024-09-15T01:57:30.923 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:30.924 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 1 Processor -- start 2024-09-15T01:57:30.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 1 -- start start 2024-09-15T01:57:30.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.920+0000 7f3d3942b640 10 monclient: init 2024-09-15T01:57:30.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d34131b20 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3d340a0280 0x7f3d34131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d340a0280 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d34132430 0x7f3d34138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d34132430 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d340a0040 con 0x7f3d34131b20 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3409bff0 con 0x7f3d340a0280 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d340a14a0 con 0x7f3d34132430 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d3942b640 10 monclient(hunting): authenticate will time out at 1110.918425s 2024-09-15T01:57:30.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3d340a0280 0x7f3d34131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3d340a0280 0x7f3d34131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54010/0 (socket says 172.21.15.17:54010) 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 -- 172.21.15.17:0/931558005 learned_addr learned my addr 172.21.15.17:0/931558005 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f3d340a0280 auth_method 0 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 1 --2- 172.21.15.17:0/931558005 >> v2:172.21.15.67:3300/0 conn(0x7f3d34132430 0x7f3d34138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): get_auth_request con 0x7f3d34132430 auth_method 0 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_done global_id 84387 payload 274 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 -- 172.21.15.17:0/931558005 >> v2:172.21.15.67:3300/0 conn(0x7f3d34132430 msgr2=0x7f3d34138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 --2- 172.21.15.17:0/931558005 >> v2:172.21.15.67:3300/0 conn(0x7f3d34132430 0x7f3d34138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 -- 172.21.15.17:0/931558005 >> v1:172.21.15.67:6789/0 conn(0x7f3d34131b20 legacy=0x7f3d3409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 monclient: found mon.noname-a 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:57:30.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.921+0000 7f3d2a7fc640 1 -- 172.21.15.17:0/931558005 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3409f3b0 con 0x7f3d340a0280 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d2a7fc640 1 -- 172.21.15.17:0/931558005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d340a0280 msgr2=0x7f3d34131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d2a7fc640 1 --2- 172.21.15.17:0/931558005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d340a0280 0x7f3d34131ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3d340a13a0 tx=0x7f3d1c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1caff03dd2cb8b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 1 -- 172.21.15.17:0/931558005 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c032440 con 0x7f3d340a0280 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: dump: 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.935 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 1 -- 172.21.15.17:0/931558005 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0325a0 con 0x7f3d340a0280 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d3942b640 5 monclient: authenticate success, global_id 84387 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d3942b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 1 -- 172.21.15.17:0/931558005 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c032940 con 0x7f3d340a0280 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d327fc640 10 monclient: dump: 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.937 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d3942b640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.924+0000 7f3d3942b640 4 set_mon_vals no callback set 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 10 monclient: shutdown 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 -- 172.21.15.17:0/931558005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d340a0280 msgr2=0x7f3d34131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 --2- 172.21.15.17:0/931558005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d340a0280 0x7f3d34131ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3d340a13a0 tx=0x7f3d1c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 -- 172.21.15.17:0/931558005 shutdown_connections 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 --2- 172.21.15.17:0/931558005 >> v2:172.21.15.67:3300/0 conn(0x7f3d34132430 0x7f3d34138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 --2- 172.21.15.17:0/931558005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d340a0280 0x7f3d34131ef0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 -- 172.21.15.17:0/931558005 >> 172.21.15.17:0/931558005 conn(0x7f3d34071270 msgr2=0x7f3d340a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 -- 172.21.15.17:0/931558005 shutdown_connections 2024-09-15T01:57:30.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.925+0000 7f3d3942b640 1 -- 172.21.15.17:0/931558005 wait complete. 2024-09-15T01:57:30.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient: build_initial_monmap 2024-09-15T01:57:30.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient: monmap: 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 1 Processor -- start 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 1 -- start start 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient: init 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340a0280 0x7f3d341416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d340a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.926+0000 7f3d3942b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 0x7f3d34141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d34131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d34132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34136240 con 0x7f3d34131b20 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340a09b0 con 0x7f3d340a0280 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340a5140 con 0x7f3d34132430 2024-09-15T01:57:30.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d3942b640 10 monclient(hunting): authenticate will time out at 1110.924037s 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 0x7f3d34141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34456/0 (socket says 172.21.15.17:34456) 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 0x7f3d34141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54026/0 (socket says 172.21.15.17:54026) 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 learned_addr learned my addr 172.21.15.17:0/2672696752 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): get_auth_request con 0x7f3d34132430 auth_method 0 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f3d34131b20 auth_method 0 2024-09-15T01:57:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340a0280 0x7f3d341416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): get_auth_request con 0x7f3d340a0280 auth_method 0 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient(hunting): handle_auth_done global_id 94163 payload 824 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340a0280 msgr2=0x7f3d341416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340a0280 0x7f3d341416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 msgr2=0x7f3d34141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 0x7f3d34141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 monclient: found mon.noname-c 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009450 con 0x7f3d34132430 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.927+0000 7f3d2a7fc640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d34131b20 0x7f3d34141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 msgr2=0x7f3d34142140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d337fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009ef0 tx=0x7f3d2c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c014480 con 0x7f3d34132430 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: dump: 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.947 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d3942b640 5 monclient: authenticate success, global_id 94163 2024-09-15T01:57:30.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d3942b640 10 monclient: _renew_subs 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d3942b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d341340e0 con 0x7f3d34132430 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00f040 con 0x7f3d34132430 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c014810 con 0x7f3d34132430 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:30.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d2b7fe640 10 monclient: dump: 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d31ffb640 4 set_mon_vals no callback set 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.929+0000 7f3d3942b640 10 monclient: _renew_subs 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d34134cb0 con 0x7f3d34132430 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 0x7f3d34097df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34063a40 con 0x7f3d34097a20 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3d2c014bc0 con 0x7f3d34132430 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d32ffd640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 0x7f3d34097df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d32ffd640 10 monclient: get_auth_request con 0x7f3d34097a20 auth_method 0 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.930+0000 7f3d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d2b7fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d14038740 0x7f3d1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 9907+0+0 (secure 0 0 0) 0x7f3d2c04da80 con 0x7f3d34132430 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_done global_id 6052 payload 824 2024-09-15T01:57:30.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d32ffd640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d340724a0 con 0x7f3d34097a20 2024-09-15T01:57:30.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d2a7fc640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d14038740 0x7f3d1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.931+0000 7f3d2a7fc640 10 monclient: get_auth_request con 0x7f3d14038740 auth_method 0 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d32ffd640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 msgr2=0x7f3d34097df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d32ffd640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 0x7f3d34097df0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400f780 tx=0x7f3d24007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400d8b0 con 0x7f3d34097a20 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d2b7fe640 10 monclient: dump: 2024-09-15T01:57:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.933+0000 7f3d2a7fc640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d14038740 0x7f3d1403abf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3d3409f710 tx=0x7f3d1c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3d2400da10 con 0x7f3d34097a20 2024-09-15T01:57:30.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 msgr2=0x7f3d34097df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 0x7f3d34097df0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400f780 tx=0x7f3d24007e10 comp rx=0 tx=0).stop 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 msgr2=0x7f3d34097df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.934+0000 7f3d2b7fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d34097a20 0x7f3d34097df0 secure :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400f780 tx=0x7f3d24007e10 comp rx=0 tx=0).stop 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.939+0000 7f3d3942b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.939+0000 7f3d3942b640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 0x7f3d3409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.939+0000 7f3d3942b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:30.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.939+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34136f00 con 0x7f3d3409e6e0 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 0x7f3d3409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient: get_auth_request con 0x7f3d3409e6e0 auth_method 0 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 10 monclient(hunting): handle_auth_done global_id 6055 payload 824 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.940+0000 7f3d32ffd640 1 -- 172.21.15.17:0/2672696752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d3406efc0 con 0x7f3d3409e6e0 2024-09-15T01:57:30.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d32ffd640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 msgr2=0x7f3d3409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d32ffd640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 0x7f3d3409eae0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003420 tx=0x7f3d24019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2401d610 con 0x7f3d3409e6e0 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.942+0000 7f3d2b7fe640 10 monclient: dump: 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:30.959 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f3d2401d770 con 0x7f3d3409e6e0 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 msgr2=0x7f3d3409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 0x7f3d3409eae0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003420 tx=0x7f3d24019000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 msgr2=0x7f3d3409eae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d2b7fe640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3409e6e0 0x7f3d3409eae0 secure :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003420 tx=0x7f3d24019000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.960 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 80, 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T01:57:30.961 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.962 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:30.964 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:30.965 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:30.966 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:30.967 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:30.968 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.969 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:30.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:30.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.972 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:57:30.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:30.974 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:30.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:30.975 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:30.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:30.975 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:30.975 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:30.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d14038740 msgr2=0x7f3d1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3d14038740 0x7f3d1403abf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3d3409f710 tx=0x7f3d1c032000 comp rx=0 tx=0).stop 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 10 monclient: shutdown 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 msgr2=0x7f3d34142140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d337fe640 1 -- 172.21.15.17:0/2672696752 reap_dead start 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009ef0 tx=0x7f3d2c00b040 comp rx=0 tx=0).stop 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 shutdown_connections 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 --2- 172.21.15.17:0/2672696752 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3d34132430 0x7f3d34142140 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 >> 172.21.15.17:0/2672696752 conn(0x7f3d34071270 msgr2=0x7f3d3409dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 shutdown_connections 2024-09-15T01:57:30.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:30.943+0000 7f3d3942b640 1 -- 172.21.15.17:0/2672696752 wait complete. 2024-09-15T01:57:30.978 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:57:30.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T01:57:30.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T01:57:30.978 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T01:57:30.978 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:57:30.979 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T01:57:30.979 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:57:30.990 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T01:57:30.990 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T01:57:30.990 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T01:57:30.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T01:57:33.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:33.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:33.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:33.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:33.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:33.693+0000 7fb3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:34.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:34.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:34.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:34.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:35.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:35.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:35.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:35.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:36.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:36.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:37.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:37.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:37.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:37.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:38.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:38.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:38.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:38.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:39.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:39.694+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:39.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:39.694+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:40.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:40.693+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:40.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:40.693+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:41.694+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:41.694+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:42.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:42.694+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:42.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:42.694+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:43.694+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:43.694+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:43.694+0000 7fb3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:44.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:44.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:44.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:44.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:45.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:45.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:45.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:45.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:46.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:46.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:46.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:46.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:47.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:47.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:47.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:47.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:48.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:48.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:48.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:48.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:49.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:49.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:49.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:49.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:50.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:50.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:50.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:50.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:50.991 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T01:57:50.992 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T01:57:50.992 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:57:50.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:57:51.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.098+0000 7fc6e2a4f640 10 monclient: get_monmap_and_config 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.098+0000 7fc6e2a4f640 10 monclient: build_initial_monmap 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.098+0000 7fc6e2a4f640 10 monclient: monmap: 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:51.099573+0000 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:51.099573+0000 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:51.101 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:51.102 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:51.102 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.099+0000 7fc6e2a4f640 1 Processor -- start 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.099+0000 7fc6e2a4f640 1 -- start start 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.099+0000 7fc6e2a4f640 10 monclient: init 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T01:57:51.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fc6dc06eb30 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6dc06f440 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6dc09e970 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc09f6a0 con 0x7fc6dc06f440 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc09ed70 con 0x7fc6dc09e970 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc069f40 con 0x7fc6dc06eb30 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e2a4f640 10 monclient(hunting): authenticate will time out at 1131.096247s 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6dbfff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 msgr2=0x7fc6dc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6dbfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6db7fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6dc09bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:51.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e0fec640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6dc09e970 0x7fc6dc139b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:53752/0 (socket says 172.21.15.17:53752) 2024-09-15T01:57:51.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.100+0000 7fc6e0fec640 1 -- 172.21.15.17:0/834871876 learned_addr learned my addr 172.21.15.17:0/834871876 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:57:51.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.102+0000 7fc6e0fec640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.67:6789/0 conn(0x7fc6dc09e970 legacy=0x7fc6dc139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:51.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.301+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6dc09bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:51.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.301+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 msgr2=0x7fc6dc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:51.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.301+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:57:51.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:51.696+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:51.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:51.696+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:51.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.702+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6dc09bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:51.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.702+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 msgr2=0x7fc6dc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:51.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:51.702+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:57:52.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:52.503+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 msgr2=0x7fc6dc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:52.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:52.503+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6dc09bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:52.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:52.503+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:57:52.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:52.696+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:52.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:52.696+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:53.695+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:53.695+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:53.695+0000 7fb3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): tick 2024-09-15T01:57:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:57:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:57:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 msgr2=0x7fc6dc06ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.17:3300/0 conn(0x7fc6dc06eb30 0x7fc6dc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.67:6789/0 conn(0x7fc6dc09e970 legacy=0x7fc6dc139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6dc09bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient: _add_conns ranks=[4,2] 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fc6c8003700 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c8006200 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:54.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc069f40 con 0x7fc6c8003700 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c8008de0 con 0x7fc6c8006200 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6d9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6c8006200 auth_method 0 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request con 0x7fc6c8003700 auth_method 0 2024-09-15T01:57:54.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.100+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6c8006200 auth_method 0 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request con 0x7fc6c8003700 auth_method 0 2024-09-15T01:57:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.301+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:57:54.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:54.696+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:54.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:54.696+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6c8006200 auth_method 0 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request con 0x7fc6c8003700 auth_method 0 2024-09-15T01:57:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:54.702+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:57:55.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:55.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6c8006200 auth_method 0 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request con 0x7fc6c8003700 auth_method 0 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:55.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:57:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:57:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:57:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.503+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:55.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:55.504+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:57:55.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:55.696+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:55.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:55.696+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:56.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.056+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v161) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb3a4029070 con 0x7fb3b809f270 2024-09-15T01:57:56.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.056+0000 7fb3b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:56.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.056+0000 7fb3b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:56.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.165+0000 7fb3bf0dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:56.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.165+0000 7fb3bf0dd640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:56.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.165+0000 7fb3bf0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:56.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.165+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7fb3b80a04e0 con 0x7fb3b809f270 2024-09-15T01:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.696+0000 7fb3a37fe640 10 monclient: tick 2024-09-15T01:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:56.696+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.068+0000 7fb3b4ff9640 1 -- 172.21.15.67:0/3950720171 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v162) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb3a401a360 con 0x7fb3b809f270 2024-09-15T01:57:57.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.068+0000 7fb3b4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T01:57:57.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.068+0000 7fb3b4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 msgr2=0x7fb39c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 0x7fb39c03abf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009d20 tx=0x7fb3ac0090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 10 monclient: shutdown 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 msgr2=0x7fb3b811e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c5e0 tx=0x7fb3a4007f00 comp rx=0 tx=0).stop 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 shutdown_connections 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb39c038740 0x7fb39c03abf0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3b80a0b50 0x7fb3b811ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3b809f270 0x7fb3b811e6a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 --2- 172.21.15.67:0/3950720171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3b809eea0 0x7fb3b8133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 >> 172.21.15.67:0/3950720171 conn(0x7fb3b8071290 msgr2=0x7fb3b80a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 shutdown_connections 2024-09-15T01:57:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.071+0000 7fb3bf0dd640 1 -- 172.21.15.67:0/3950720171 wait complete. 2024-09-15T01:57:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): tick 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 msgr2=0x7fc6c8005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> v2:172.21.15.67:3300/0 conn(0x7fc6c8003700 0x7fc6c8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6e0fec640 1 -- 172.21.15.17:0/834871876 reap_dead start 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6dc09e970 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:57.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c8006200 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6dc06f440 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc069950 con 0x7fc6dc06f440 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc09f6a0 con 0x7fc6dc09e970 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 1 -- 172.21.15.17:0/834871876 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc069f40 con 0x7fc6c8006200 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.100+0000 7fc6d9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6e0fec640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6c800e960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6c8006200 auth_method 0 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_done global_id 94166 payload 274 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.67:6789/0 conn(0x7fc6dc09e970 legacy=0x7fc6c800b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> v1:172.21.15.17:6789/0 conn(0x7fc6dc06f440 legacy=0x7fc6c800e960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 1 monclient: found mon.noname-b 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.101+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8004b70 con 0x7fc6c8006200 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6db7fe640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009bd0 tx=0x7fc6d000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99e301d0258ea12a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 1 -- 172.21.15.17:0/834871876 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0031420 con 0x7fc6c8006200 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: dump: 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 1 -- 172.21.15.17:0/834871876 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d002f070 con 0x7fc6c8006200 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6e2a4f640 5 monclient: authenticate success, global_id 94166 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6e2a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:57.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 1 -- 172.21.15.17:0/834871876 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00317f0 con 0x7fc6c8006200 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6e2a4f640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6e2a4f640 4 set_mon_vals no callback set 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.104+0000 7fc6daffd640 10 monclient: dump: 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.110 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 10 monclient: shutdown 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 msgr2=0x7fc6c80086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009bd0 tx=0x7fc6d000b040 comp rx=0 tx=0).stop 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/834871876 shutdown_connections 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/834871876 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6c8006200 0x7fc6c80086b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/834871876 >> 172.21.15.17:0/834871876 conn(0x7fc6dc0712f0 msgr2=0x7fc6dc0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/834871876 shutdown_connections 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.105+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/834871876 wait complete. 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.106+0000 7fc6e2a4f640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.106+0000 7fc6e2a4f640 10 monclient: monmap: 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.112 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.113 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:57.113 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:57.113 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:57.113 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.106+0000 7fc6e2a4f640 1 Processor -- start 2024-09-15T01:57:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.106+0000 7fc6e2a4f640 1 -- start start 2024-09-15T01:57:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.106+0000 7fc6e2a4f640 10 monclient: init 2024-09-15T01:57:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 0x7fc6dc134c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fc6c8006200 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-c con 0x7fc6dc06f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6dc09e970 0x7fc6dc11ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): picked mon.noname-b con 0x7fc6dc09e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc09cd10 con 0x7fc6c8006200 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0a0700 con 0x7fc6dc09e970 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1202a0 con 0x7fc6dc06f440 2024-09-15T01:57:57.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e2a4f640 10 monclient(hunting): authenticate will time out at 1137.103464s 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6dbfff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 msgr2=0x7fc6dc134c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6dbfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 0x7fc6dc134c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6dc09e970 0x7fc6dc11ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34712/0 (socket says 172.21.15.17:34712) 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 1 -- 172.21.15.17:0/2316115282 learned_addr learned my addr 172.21.15.17:0/2316115282 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request con 0x7fc6dc06f440 auth_method 0 2024-09-15T01:57:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): get_auth_request con 0x7fc6dc09e970 auth_method 0 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6e0fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.107+0000 7fc6db7fe640 10 monclient(hunting): handle_auth_done global_id 94169 payload 824 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 -- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6dc09e970 msgr2=0x7fc6dc11ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6dc09e970 0x7fc6dc11ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 -- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 msgr2=0x7fc6dc134c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 0x7fc6dc134c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 monclient: found mon.noname-c 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.108+0000 7fc6db7fe640 1 -- 172.21.15.17:0/2316115282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d0009830 con 0x7fc6dc06f440 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6db7fe640 1 -- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 msgr2=0x7fc6dc11e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6db7fe640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009770 tx=0x7fc6d0037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d003e840 con 0x7fc6dc06f440 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: dump: 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.126 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d003e9a0 con 0x7fc6dc06f440 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 5 monclient: authenticate success, global_id 94169 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 10 monclient: _renew_subs 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d003ed60 con 0x7fc6dc06f440 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc141c60 con 0x7fc6dc06f440 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6c3fff640 10 monclient: dump: 2024-09-15T01:57:57.127 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6da7fc640 4 set_mon_vals no callback set 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 10 monclient: _renew_subs 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.110+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc121e10 con 0x7fc6dc06f440 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.111+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc6d0035610 con 0x7fc6dc06f440 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.111+0000 7fc6e2a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.111+0000 7fc6e2a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.111+0000 7fc6e2a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.111+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc0724a0 con 0x7fc6dc06f440 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.112+0000 7fc6c3fff640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 0x7fc6c803b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.112+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fc6d0044070 con 0x7fc6dc06f440 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.112+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 0x7fc6c803b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.112+0000 7fc6dbfff640 10 monclient: get_auth_request con 0x7fc6c80394b0 auth_method 0 2024-09-15T01:57:57.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.114+0000 7fc6dbfff640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 0x7fc6c803b960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc003940 tx=0x7fc6cc003a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.116+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc6d0031070 con 0x7fc6dc06f440 2024-09-15T01:57:57.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.116+0000 7fc6c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.116+0000 7fc6c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.168+0000 7f2b3419d640 10 monclient: get_monmap_and_config 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.168+0000 7f2b3419d640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.168+0000 7f2b3419d640 10 monclient: monmap: 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.171 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:57.170080+0000 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:57.170080+0000 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:57.172 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.169+0000 7f2b3419d640 1 Processor -- start 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.169+0000 7f2b3419d640 1 -- start start 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.169+0000 7f2b3419d640 10 monclient: init 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 0x7f2b2c069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b2c09eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b2c09f270 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c06a270 con 0x7f2b2c09f270 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c0a0f20 con 0x7f2b2c09eea0 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c06f290 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3419d640 10 monclient(hunting): authenticate will time out at 1109.639288s 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b3273a640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f2b2c09f270 legacy=0x7f2b2c132220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31738640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b2c09eea0 0x7f2b2c131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37848/0 (socket says 172.21.15.67:37848) 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31738640 1 -- 172.21.15.67:0/191694875 learned_addr learned my addr 172.21.15.67:0/191694875 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 1 --2- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 0x7f2b2c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): get_auth_request con 0x7f2b2c0a0b50 auth_method 0 2024-09-15T01:57:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient(hunting): handle_auth_done global_id 94172 payload 274 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 1 -- 172.21.15.67:0/191694875 >> v1:172.21.15.67:6789/0 conn(0x7f2b2c09eea0 legacy=0x7f2b2c131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 1 -- 172.21.15.67:0/191694875 >> v1:172.21.15.17:6789/0 conn(0x7f2b2c09f270 legacy=0x7f2b2c132220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 1 monclient: found mon.noname-b 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.170+0000 7f2b31f39640 1 -- 172.21.15.67:0/191694875 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c132930 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b31f39640 1 -- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 msgr2=0x7f2b2c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b31f39640 1 --2- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 0x7f2b2c069340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009880 tx=0x7f2b240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b267ee4ec11f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 1 -- 172.21.15.67:0/191694875 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24032440 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: dump: 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.186 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 1 -- 172.21.15.67:0/191694875 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b240325a0 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 1 -- 172.21.15.67:0/191694875 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24032940 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b30f37640 10 monclient: dump: 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:57.188 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 5 monclient: authenticate success, global_id 94172 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:57.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 4 set_mon_vals no callback set 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 10 monclient: shutdown 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 -- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 msgr2=0x7f2b2c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 --2- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 0x7f2b2c069340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009880 tx=0x7f2b240090f0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 -- 172.21.15.67:0/191694875 shutdown_connections 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 --2- 172.21.15.67:0/191694875 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c0a0b50 0x7f2b2c069340 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 -- 172.21.15.67:0/191694875 >> 172.21.15.67:0/191694875 conn(0x7f2b2c071290 msgr2=0x7f2b2c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 -- 172.21.15.67:0/191694875 shutdown_connections 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.174+0000 7f2b3419d640 1 -- 172.21.15.67:0/191694875 wait complete. 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.175+0000 7f2b3419d640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.175+0000 7f2b3419d640 10 monclient: monmap: 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:57.191 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.175+0000 7f2b3419d640 1 Processor -- start 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.175+0000 7f2b3419d640 1 -- start start 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.175+0000 7f2b3419d640 10 monclient: init 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 0x7f2b2c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2c09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2c09f270 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 0x7f2b2c1411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2c0a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c11e480 con 0x7f2b2c0a0b50 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c09b4b0 con 0x7f2b2c09f270 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c11de50 con 0x7f2b2c09eea0 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3419d640 10 monclient(hunting): authenticate will time out at 1109.645123s 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3273a640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 msgr2=0x7f2b2c1411f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:57.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b3273a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 0x7f2b2c1411f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43936/0 (socket says 172.21.15.67:43936) 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 -- 172.21.15.67:0/2740835485 learned_addr learned my addr 172.21.15.67:0/2740835485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): get_auth_request con 0x7f2b2c09f270 auth_method 0 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 0x7f2b2c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): get_auth_request con 0x7f2b2c09eea0 auth_method 0 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient(hunting): handle_auth_done global_id 6061 payload 824 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 msgr2=0x7f2b2c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 0x7f2b2c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 msgr2=0x7f2b2c1411f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 0x7f2b2c1411f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 monclient: found mon.noname-b 2024-09-15T01:57:57.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31738640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24009390 con 0x7f2b2c09f270 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.176+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 0x7f2b2c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.178+0000 7f2b31738640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 msgr2=0x7f2b2c130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.178+0000 7f2b31738640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c5e0 tx=0x7f2b20004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20005cb0 con 0x7f2b2c09f270 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: dump: 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.196 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 5 monclient: authenticate success, global_id 6061 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b20005e30 con 0x7f2b2c09f270 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 10 monclient: _renew_subs 2024-09-15T01:57:57.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20007340 con 0x7f2b2c09f270 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c11c990 con 0x7f2b2c09f270 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b29ffb640 10 monclient: dump: 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b2bfff640 4 set_mon_vals no callback set 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 10 monclient: _renew_subs 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.179+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c11bff0 con 0x7f2b2c09f270 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2b2001b400 con 0x7f2b2c09f270 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b29ffb640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f2b2004c9d0 con 0x7f2b2c09f270 2024-09-15T01:57:57.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b3419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b3419d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b3419d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 10 monclient: get_auth_request con 0x7f2b180386f0 auth_method 0 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 msgr2=0x7f2b1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 msgr2=0x7f2b1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c09ccd0 tx=0x7f2b24032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:57:57.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.180+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.185+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:57.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.185+0000 7f2b29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.185+0000 7f2b29ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:57.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6e2a4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:57:57.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6e2a4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:57.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6e2a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc6dc067220 con 0x7fc6dc06f440 2024-09-15T01:57:57.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6c3fff640 1 -- 172.21.15.17:0/2316115282 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7fc6d0006000 con 0x7fc6dc06f440 2024-09-15T01:57:57.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:57:57.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.349+0000 7fc6c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:57.352 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 msgr2=0x7fc6c803b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 0x7fc6c803b960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc003940 tx=0x7fc6cc003a00 comp rx=0 tx=0).stop 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 10 monclient: shutdown 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 msgr2=0x7fc6dc11e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009770 tx=0x7fc6d0037000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 shutdown_connections 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc6c80394b0 0x7fc6c803b960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6dc09e970 0x7fc6dc11ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc6dc06f440 0x7fc6dc11e750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 --2- 172.21.15.17:0/2316115282 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc6c8006200 0x7fc6dc134c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.351+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 >> 172.21.15.17:0/2316115282 conn(0x7fc6dc0712f0 msgr2=0x7fc6dc06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.352+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 shutdown_connections 2024-09-15T01:57:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.352+0000 7fc6e2a4f640 1 -- 172.21.15.17:0/2316115282 wait complete. 2024-09-15T01:57:57.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.358+0000 7f2b3419d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:57.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.358+0000 7f2b3419d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:57.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.358+0000 7f2b3419d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.358+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:57.364 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:57:57.364 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T01:57:57.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:57:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.382+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.382+0000 7f2b31f39640 10 monclient: get_auth_request con 0x7f2b180386f0 auth_method 0 2024-09-15T01:57:57.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:57.384+0000 7f2b31f39640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009360 tx=0x7f2b2403d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 10 monclient: get_monmap_and_config 2024-09-15T01:57:57.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 10 monclient: monmap: 2024-09-15T01:57:57.470 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:57.460885+0000 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:57.460885+0000 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:57.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 1 Processor -- start 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 1 -- start start 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.460+0000 7fdaa462b640 10 monclient: init 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda9c0a58b0 0x7fda9c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-c con 0x7fda9c0a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda9c0980a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 0x7fda9c1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-a con 0x7fda9c136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0a01e0 con 0x7fda9c0980a0 2024-09-15T01:57:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c0a1580 con 0x7fda9c136dd0 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c0a5c80 con 0x7fda9c0a58b0 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa462b640 10 monclient(hunting): authenticate will time out at 1137.457337s 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa1bc6640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fda9c0980a0 legacy=0x7fda9c0984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa2bc8640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 msgr2=0x7fda9c1391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa2bc8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 0x7fda9c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda9c0a58b0 0x7fda9c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda9c0a58b0 0x7fda9c132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:58802/0 (socket says 172.21.15.17:58802) 2024-09-15T01:57:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 1 -- 172.21.15.17:0/235598656 learned_addr learned my addr 172.21.15.17:0/235598656 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request con 0x7fda9c0a58b0 auth_method 0 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_done global_id 6064 payload 274 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 1 -- 172.21.15.17:0/235598656 >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 msgr2=0x7fda9c1391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.461+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/235598656 >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 0x7fda9c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 1 -- 172.21.15.17:0/235598656 >> v1:172.21.15.17:6789/0 conn(0x7fda9c0980a0 legacy=0x7fda9c0984a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 1 monclient: found mon.noname-c 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.462+0000 7fdaa23c7640 1 -- 172.21.15.17:0/235598656 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c098bb0 con 0x7fda9c0a58b0 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.464+0000 7fdaa23c7640 1 -- 172.21.15.17:0/235598656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0a58b0 msgr2=0x7fda9c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.464+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/235598656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0a58b0 0x7fda9c132040 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a60e0 tx=0x7fda980091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb4170aa7dccf85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 1 -- 172.21.15.17:0/235598656 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9802ae50 con 0x7fda9c0a58b0 2024-09-15T01:57:57.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:57.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: dump: 2024-09-15T01:57:57.487 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.487 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 1 -- 172.21.15.17:0/235598656 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9802f4b0 con 0x7fda9c0a58b0 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 5 monclient: authenticate success, global_id 6064 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 4 set_mon_vals no callback set 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 1 -- 172.21.15.17:0/235598656 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9802f850 con 0x7fda9c0a58b0 2024-09-15T01:57:57.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa13c5640 10 monclient: dump: 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.490 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 10 monclient: shutdown 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 -- 172.21.15.17:0/235598656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0a58b0 msgr2=0x7fda9c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 --2- 172.21.15.17:0/235598656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0a58b0 0x7fda9c132040 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a60e0 tx=0x7fda980091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 -- 172.21.15.17:0/235598656 shutdown_connections 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 --2- 172.21.15.17:0/235598656 >> v2:172.21.15.17:3300/0 conn(0x7fda9c136dd0 0x7fda9c1391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 --2- 172.21.15.17:0/235598656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0a58b0 0x7fda9c132040 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 -- 172.21.15.17:0/235598656 >> 172.21.15.17:0/235598656 conn(0x7fda9c070ed0 msgr2=0x7fda9c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 -- 172.21.15.17:0/235598656 shutdown_connections 2024-09-15T01:57:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.465+0000 7fdaa462b640 1 -- 172.21.15.17:0/235598656 wait complete. 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.466+0000 7fdaa462b640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.466+0000 7fdaa462b640 10 monclient: monmap: 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:57.492 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.466+0000 7fdaa462b640 1 Processor -- start 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.466+0000 7fdaa462b640 1 -- start start 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.466+0000 7fdaa462b640 10 monclient: init 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0980a0 0x7fda9c141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-b con 0x7fda9c0980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 0x7fda9c132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-c con 0x7fda9c0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda9c136dd0 0x7fda9c133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): picked mon.noname-a con 0x7fda9c136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c098dc0 con 0x7fda9c136dd0 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c136620 con 0x7fda9c0980a0 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c136060 con 0x7fda9c0a58b0 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa462b640 10 monclient(hunting): authenticate will time out at 1137.463121s 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa2bc8640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda9c136dd0 msgr2=0x7fda9c133060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa2bc8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda9c136dd0 0x7fda9c133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 0x7fda9c132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 0x7fda9c132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34718/0 (socket says 172.21.15.17:34718) 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 -- 172.21.15.17:0/3955589432 learned_addr learned my addr 172.21.15.17:0/3955589432 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): get_auth_request con 0x7fda9c0a58b0 auth_method 0 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0980a0 0x7fda9c141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request con 0x7fda9c0980a0 auth_method 0 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient(hunting): handle_auth_done global_id 94178 payload 824 2024-09-15T01:57:57.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0980a0 msgr2=0x7fda9c141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9c0980a0 0x7fda9c141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda9c136dd0 msgr2=0x7fda9c133060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda9c136dd0 0x7fda9c133060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 monclient: found mon.noname-c 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.467+0000 7fdaa1bc6640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98009450 con 0x7fda9c0a58b0 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa1bc6640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 msgr2=0x7fda9c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa1bc6640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 0x7fda9c132b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00b600 tx=0x7fda8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c015070 con 0x7fda9c0a58b0 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: dump: 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.498 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa462b640 5 monclient: authenticate success, global_id 94178 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c003d90 con 0x7fda9c0a58b0 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c006380 con 0x7fda9c0a58b0 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa462b640 10 monclient: _renew_subs 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa462b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9c136340 con 0x7fda9c0a58b0 2024-09-15T01:57:57.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fdaa0bc4640 4 set_mon_vals no callback set 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.469+0000 7fda927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fda927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fda927fc640 10 monclient: dump: 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 10 monclient: _renew_subs 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c131150 con 0x7fda9c0a58b0 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fda8c010440 con 0x7fda9c0a58b0 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 0x7fda9c1335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.470+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c1367b0 con 0x7fda9c135100 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 0x7fda9c1335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fda927fc640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda800386f0 0x7fda8003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient: get_auth_request con 0x7fda9c135100 auth_method 0 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fda8c04c960 con 0x7fda9c0a58b0 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa2bc8640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda800386f0 0x7fda8003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa2bc8640 10 monclient: get_auth_request con 0x7fda800386f0 auth_method 0 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_done global_id 94181 payload 824 2024-09-15T01:57:57.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.471+0000 7fdaa23c7640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda9c0a3520 con 0x7fda9c135100 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fdaa23c7640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 msgr2=0x7fda9c1335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 0x7fda9c1335a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fda9c09f780 tx=0x7fda98032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fdaa2bc8640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda800386f0 0x7fda8003aba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fda9c134960 tx=0x7fda940092d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98032a10 con 0x7fda9c135100 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fda927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fda927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.473+0000 7fda927fc640 10 monclient: dump: 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.503 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fda98041070 con 0x7fda9c135100 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 msgr2=0x7fda9c1335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 0x7fda9c1335a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fda9c09f780 tx=0x7fda98032000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 msgr2=0x7fda9c1335a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.474+0000 7fda927fc640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c135100 0x7fda9c1335a0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fda9c09f780 tx=0x7fda98032000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa462b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa462b640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 0x7fda9c133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa462b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c063ae0 con 0x7fda9c142390 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 0x7fda9c133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient: get_auth_request con 0x7fda9c142390 auth_method 0 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.479+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.480+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.480+0000 7fdaa23c7640 10 monclient(hunting): handle_auth_done global_id 94184 payload 824 2024-09-15T01:57:57.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.480+0000 7fdaa23c7640 1 -- 172.21.15.17:0/3955589432 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fda9c0724a0 con 0x7fda9c142390 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fdaa23c7640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 msgr2=0x7fda9c133ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fdaa23c7640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 0x7fda9c133ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0638f0 tx=0x7fda9803f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98032e80 con 0x7fda9c142390 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 10 monclient: dump: 2024-09-15T01:57:57.507 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4554 (secure 0 0 0) 0x7fda9803d810 con 0x7fda9c142390 2024-09-15T01:57:57.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 msgr2=0x7fda9c133ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 0x7fda9c133ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0638f0 tx=0x7fda9803f070 comp rx=0 tx=0).stop 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 msgr2=0x7fda9c133ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.482+0000 7fda927fc640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c142390 0x7fda9c133ae0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0638f0 tx=0x7fda9803f070 comp rx=0 tx=0).stop 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:57.509 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 82, 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.510 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:57.513 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.514 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.516 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.517 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:57.518 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:57.519 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.521 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.522 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:57.523 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:57.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda800386f0 msgr2=0x7fda8003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fda800386f0 0x7fda8003aba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fda9c134960 tx=0x7fda940092d0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 10 monclient: shutdown 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 msgr2=0x7fda9c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 --2- 172.21.15.17:0/3955589432 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fda9c0a58b0 0x7fda9c132b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00b600 tx=0x7fda8c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa2bc8640 1 -- 172.21.15.17:0/3955589432 reap_dead start 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa2bc8640 1 -- 172.21.15.17:0/3955589432 reap_dead start 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 shutdown_connections 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 >> 172.21.15.17:0/3955589432 conn(0x7fda9c070ed0 msgr2=0x7fda9c0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 shutdown_connections 2024-09-15T01:57:57.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.483+0000 7fdaa462b640 1 -- 172.21.15.17:0/3955589432 wait complete. 2024-09-15T01:57:57.526 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:57:57.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T01:57:57.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.622+0000 7fd7a1434640 10 monclient: get_monmap_and_config 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.622+0000 7fd7a1434640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.622+0000 7fd7a1434640 10 monclient: monmap: 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:57:57.623587+0000 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:57:57.623587+0000 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.625 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 Processor -- start 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 -- start start 2024-09-15T01:57:57.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient: init 2024-09-15T01:57:57.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T01:57:57.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd79c09a590 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd79c09a960 0x7fd79c12edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c09a960 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0985d0 con 0x7fd79c09a590 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79c09d200 con 0x7fd79c09a960 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c097c30 con 0x7fd79c097320 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd7a1434640 10 monclient(hunting): authenticate will time out at 1137.619929s 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.623+0000 7fd79a7fc640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fd79c09a590 legacy=0x7fd79c12e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34738/0 (socket says 172.21.15.17:34738) 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 -- 172.21.15.17:0/724628028 learned_addr learned my addr 172.21.15.17:0/724628028 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): get_auth_request con 0x7fd79c097320 auth_method 0 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 1 --2- 172.21.15.17:0/724628028 >> v2:172.21.15.67:3300/0 conn(0x7fd79c09a960 0x7fd79c12edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c09a960 auth_method 0 2024-09-15T01:57:57.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient(hunting): handle_auth_done global_id 94187 payload 274 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 -- 172.21.15.17:0/724628028 >> v2:172.21.15.67:3300/0 conn(0x7fd79c09a960 msgr2=0x7fd79c12edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 --2- 172.21.15.17:0/724628028 >> v2:172.21.15.67:3300/0 conn(0x7fd79c09a960 0x7fd79c12edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 -- 172.21.15.17:0/724628028 >> v1:172.21.15.17:6789/0 conn(0x7fd79c09a590 legacy=0x7fd79c12e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 monclient: found mon.noname-b 2024-09-15T01:57:57.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.624+0000 7fd79affd640 1 -- 172.21.15.17:0/724628028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c0992f0 con 0x7fd79c097320 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd79affd640 1 -- 172.21.15.17:0/724628028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 msgr2=0x7fd79c0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd79affd640 1 --2- 172.21.15.17:0/724628028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a48513d5a7a0b229 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 1 -- 172.21.15.17:0/724628028 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c02aea0 con 0x7fd79c097320 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: dump: 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.636 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 5 monclient: authenticate success, global_id 94187 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 1 -- 172.21.15.17:0/724628028 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c032440 con 0x7fd79c097320 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 1 -- 172.21.15.17:0/724628028 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0327e0 con 0x7fd79c097320 2024-09-15T01:57:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 4 set_mon_vals no callback set 2024-09-15T01:57:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd799ffb640 10 monclient: dump: 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 10 monclient: shutdown 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 -- 172.21.15.17:0/724628028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 msgr2=0x7fd79c0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 --2- 172.21.15.17:0/724628028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 -- 172.21.15.17:0/724628028 shutdown_connections 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 --2- 172.21.15.17:0/724628028 >> v2:172.21.15.67:3300/0 conn(0x7fd79c09a960 0x7fd79c12edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 --2- 172.21.15.17:0/724628028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c097320 0x7fd79c0976f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 -- 172.21.15.17:0/724628028 >> 172.21.15.17:0/724628028 conn(0x7fd79c0a1000 msgr2=0x7fd79c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.627+0000 7fd7a1434640 1 -- 172.21.15.17:0/724628028 shutdown_connections 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.628+0000 7fd7a1434640 1 -- 172.21.15.17:0/724628028 wait complete. 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.628+0000 7fd7a1434640 10 monclient: build_initial_monmap 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.628+0000 7fd7a1434640 10 monclient: monmap: 2024-09-15T01:57:57.641 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:57.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 Processor -- start 2024-09-15T01:57:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 -- start start 2024-09-15T01:57:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient: init 2024-09-15T01:57:57.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd79c097320 0x7fd79c141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-a con 0x7fd79c097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 0x7fd79c12da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c09a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c09a960 0x7fd79c11e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c09a960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c12e8d0 con 0x7fd79c097320 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c1205a0 con 0x7fd79c09a960 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c141f70 con 0x7fd79c09a590 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd7a1434640 10 monclient(hunting): authenticate will time out at 1137.625582s 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79affd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd79c097320 msgr2=0x7fd79c141a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd79c097320 0x7fd79c141a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 0x7fd79c12da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 0x7fd79c12da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34740/0 (socket says 172.21.15.17:34740) 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 1 -- 172.21.15.17:0/4273529079 learned_addr learned my addr 172.21.15.17:0/4273529079 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:57:57.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): get_auth_request con 0x7fd79c09a590 auth_method 0 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c09a960 0x7fd79c11e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c09a960 auth_method 0 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 10 monclient(hunting): handle_auth_done global_id 94190 payload 824 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c09a960 msgr2=0x7fd79c11e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.629+0000 7fd79a7fc640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c09a960 0x7fd79c11e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd79c097320 msgr2=0x7fd79c141a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd79c097320 0x7fd79c141a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 1 monclient: found mon.noname-c 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.630+0000 7fd79a7fc640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c009450 con 0x7fd79c09a590 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd79a7fc640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 msgr2=0x7fd79c12da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd79a7fc640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 0x7fd79c12da30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd79000b630 tx=0x7fd7900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790015070 con 0x7fd79c09a590 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: dump: 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: _finish_auth 0 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790003d90 con 0x7fd79c09a590 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 5 monclient: authenticate success, global_id 94190 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 10 monclient: _renew_subs 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c120c60 con 0x7fd79c09a590 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790006380 con 0x7fd79c09a590 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd782ffd640 10 monclient: dump: 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 10 monclient: _renew_subs 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c120090 con 0x7fd79c09a590 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.632+0000 7fd7997fa640 4 set_mon_vals no callback set 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd7a1434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd7a1434640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 0x7fd79c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c063c20 con 0x7fd79c12df70 2024-09-15T01:57:57.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd790010440 con 0x7fd79c09a590 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 0x7fd79c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient: get_auth_request con 0x7fd79c12df70 auth_method 0 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd782ffd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7880386f0 0x7fd78803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fd79004c960 con 0x7fd79c09a590 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 10 monclient(hunting): handle_auth_done global_id 6076 payload 824 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79b7fe640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7880386f0 0x7fd78803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79affd640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd79c1222f0 con 0x7fd79c12df70 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.633+0000 7fd79b7fe640 10 monclient: get_auth_request con 0x7fd7880386f0 auth_method 0 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd79affd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 msgr2=0x7fd79c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd79affd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 0x7fd79c0298c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034000 tx=0x7fd78c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c02aea0 con 0x7fd79c12df70 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 10 monclient: dump: 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.661 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd79b7fe640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7880386f0 0x7fd78803aba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd794009fd0 tx=0x7fd79400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd78c0327c0 con 0x7fd79c12df70 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 msgr2=0x7fd79c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 0x7fd79c0298c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034000 tx=0x7fd78c032000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 msgr2=0x7fd79c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.636+0000 7fd782ffd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c12df70 0x7fd79c0298c0 secure :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034000 tx=0x7fd78c032000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd7a1434640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd7a1434640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 0x7fd79c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd7a1434640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c1220e0 con 0x7fd79c029e00 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 0x7fd79c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient: get_auth_request con 0x7fd79c029e00 auth_method 0 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 10 monclient(hunting): handle_auth_done global_id 6079 payload 824 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.642+0000 7fd79affd640 1 -- 172.21.15.17:0/4273529079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd79c0724a0 con 0x7fd79c029e00 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd79affd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 msgr2=0x7fd79c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd79affd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 0x7fd79c02a200 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c03c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c031eb0 con 0x7fd79c029e00 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 10 monclient: dump: 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:57.664 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4726 (secure 0 0 0) 0x7fd78c03de00 con 0x7fd79c029e00 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 msgr2=0x7fd79c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 0x7fd79c02a200 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c03c000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 msgr2=0x7fd79c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:57:57.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd782ffd640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79c029e00 0x7fd79c02a200 secure :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c03c000 comp rx=0 tx=0).stop 2024-09-15T01:57:57.666 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:57:57.666 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:57.666 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:57.667 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:57:57.667 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 82, 2024-09-15T01:57:57.667 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:57:57.667 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:57:57.667 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.668 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:57:57.672 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:57:57.673 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:57.673 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:57:57.673 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:57:57.673 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:57:57.673 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:57:57.674 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:57:57.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:57:57.676 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.677 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.679 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:57:57.680 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.682 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:57:57.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:57:57.684 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7880386f0 msgr2=0x7fd78803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7880386f0 0x7fd78803aba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd794009fd0 tx=0x7fd79400b040 comp rx=0 tx=0).stop 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 10 monclient: shutdown 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 msgr2=0x7fd79c12da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 --2- 172.21.15.17:0/4273529079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd79c09a590 0x7fd79c12da30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd79000b630 tx=0x7fd7900091a0 comp rx=0 tx=0).stop 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd79b7fe640 1 -- 172.21.15.17:0/4273529079 reap_dead start 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd79b7fe640 1 -- 172.21.15.17:0/4273529079 reap_dead start 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 shutdown_connections 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.645+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 >> 172.21.15.17:0/4273529079 conn(0x7fd79c0a1000 msgr2=0x7fd79c09f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:57.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.646+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 shutdown_connections 2024-09-15T01:57:57.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:57:57.646+0000 7fd7a1434640 1 -- 172.21.15.17:0/4273529079 wait complete. 2024-09-15T01:57:57.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:57:57.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:57:58.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.077+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v163) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:58.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.077+0000 7f2b29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:58.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.077+0000 7f2b29ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:57:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.186+0000 7f2b3419d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.186+0000 7f2b3419d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.186+0000 7f2b3419d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:58.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:58.186+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.081+0000 7f2b29ffb640 1 -- 172.21.15.67:0/2740835485 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v164) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2b2c11e070 con 0x7f2b2c09f270 2024-09-15T01:57:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.081+0000 7f2b29ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T01:57:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.081+0000 7f2b29ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 msgr2=0x7f2b1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009360 tx=0x7f2b2403d000 comp rx=0 tx=0).stop 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 10 monclient: shutdown 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 msgr2=0x7f2b2c130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c5e0 tx=0x7f2b20004960 comp rx=0 tx=0).stop 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 shutdown_connections 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b180386f0 0x7f2b1803aba0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b2c0a0b50 0x7f2b2c1411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c09f270 0x7f2b2c130d30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 --2- 172.21.15.67:0/2740835485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b2c09eea0 0x7f2b2c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 >> 172.21.15.67:0/2740835485 conn(0x7f2b2c071290 msgr2=0x7f2b2c09d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:59.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 shutdown_connections 2024-09-15T01:57:59.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.084+0000 7f2b3419d640 1 -- 172.21.15.67:0/2740835485 wait complete. 2024-09-15T01:57:59.097 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.180+0000 7ff452ec0640 10 monclient: get_monmap_and_config 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.180+0000 7ff452ec0640 10 monclient: build_initial_monmap 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.180+0000 7ff452ec0640 10 monclient: monmap: 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:57:59.180752+0000 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:57:59.180752+0000 2024-09-15T01:57:59.182 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:57:59.183 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 Processor -- start 2024-09-15T01:57:59.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 -- start start 2024-09-15T01:57:59.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient: init 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c06e210 0x7ff44c06b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-c con 0x7ff44c06e210 addr v2:172.21.15.67:3300/0 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 0x7ff44c09db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-b con 0x7ff44c06e5e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff44c09e050 addr v1:172.21.15.17:6789/0 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44c06aa60 con 0x7ff44c09e050 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44c06a190 con 0x7ff44c06e210 2024-09-15T01:57:59.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c069bd0 con 0x7ff44c06e5e0 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.181+0000 7ff452ec0640 10 monclient(hunting): authenticate will time out at 1111.649882s 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c06e210 0x7ff44c06b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff45145d640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7ff44c09e050 legacy=0x7ff44c138d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c06e210 0x7ff44c06b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43938/0 (socket says 172.21.15.67:43938) 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 -- 172.21.15.67:0/2442469892 learned_addr learned my addr 172.21.15.67:0/2442469892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): get_auth_request con 0x7ff44c06e210 auth_method 0 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:59.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 0x7ff44c09db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 10 monclient(hunting): get_auth_request con 0x7ff44c06e5e0 auth_method 0 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient(hunting): handle_auth_done global_id 6082 payload 274 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 -- 172.21.15.67:0/2442469892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 msgr2=0x7ff44c09db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 0x7ff44c09db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 -- 172.21.15.67:0/2442469892 >> v1:172.21.15.17:6789/0 conn(0x7ff44c09e050 legacy=0x7ff44c138d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient: _finish_hunting 0 2024-09-15T01:57:59.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 monclient: found mon.noname-c 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff450c5c640 1 -- 172.21.15.67:0/2442469892 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c0698f0 con 0x7ff44c06e210 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.182+0000 7ff443fff640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 0x7ff44c09db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff450c5c640 1 -- 172.21.15.67:0/2442469892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e210 msgr2=0x7ff44c06b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff450c5c640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e210 0x7ff44c06b9b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff448009910 tx=0x7ff4480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d148da1e46b00f7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 1 -- 172.21.15.67:0/2442469892 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448032440 con 0x7ff44c06e210 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:59.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: dump: 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:59.196 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: _finish_auth 0 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 1 -- 172.21.15.67:0/2442469892 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4480325a0 con 0x7ff44c06e210 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 5 monclient: authenticate success, global_id 6082 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 10 monclient: get_monmap_and_config success 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 1 -- 172.21.15.67:0/2442469892 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448032940 con 0x7ff44c06e210 2024-09-15T01:57:59.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 4 set_mon_vals no callback set 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff4437fe640 10 monclient: dump: 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:59.198 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 10 monclient: shutdown 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 -- 172.21.15.67:0/2442469892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e210 msgr2=0x7ff44c06b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e210 0x7ff44c06b9b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff448009910 tx=0x7ff4480091d0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 -- 172.21.15.67:0/2442469892 shutdown_connections 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c06e5e0 0x7ff44c09db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 --2- 172.21.15.67:0/2442469892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e210 0x7ff44c06b9b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 -- 172.21.15.67:0/2442469892 >> 172.21.15.67:0/2442469892 conn(0x7ff44c070ff0 msgr2=0x7ff44c0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.185+0000 7ff452ec0640 1 -- 172.21.15.67:0/2442469892 shutdown_connections 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.186+0000 7ff452ec0640 1 -- 172.21.15.67:0/2442469892 wait complete. 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.186+0000 7ff452ec0640 10 monclient: build_initial_monmap 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.186+0000 7ff452ec0640 10 monclient: monmap: 2024-09-15T01:57:59.199 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.186+0000 7ff452ec0640 1 Processor -- start 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 -- start start 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient: init 2024-09-15T01:57:59.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient: _reopen_session rank -1 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 0x7ff44c13e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-a con 0x7ff44c06e210 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-b con 0x7ff44c06e5e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c09e050 0x7ff44c141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): picked mon.noname-c con 0x7ff44c09e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c099f50 con 0x7ff44c06e210 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c122d20 con 0x7ff44c06e5e0 2024-09-15T01:57:59.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c120630 con 0x7ff44c09e050 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): _renew_subs 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff452ec0640 10 monclient(hunting): authenticate will time out at 1111.655463s 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff450c5c640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 msgr2=0x7ff44c13e010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff450c5c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 0x7ff44c13e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43940/0 (socket says 172.21.15.67:43940) 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 -- 172.21.15.67:0/3493173982 learned_addr learned my addr 172.21.15.67:0/3493173982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): get_auth_request con 0x7ff44c06e5e0 auth_method 0 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c09e050 0x7ff44c141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:59.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): get_auth_request con 0x7ff44c09e050 auth_method 0 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff45145d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient(hunting): handle_auth_done global_id 6085 payload 824 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 -- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c09e050 msgr2=0x7ff44c141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c09e050 0x7ff44c141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 -- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 msgr2=0x7ff44c13e010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 0x7ff44c13e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient: _finish_hunting 0 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 monclient: found mon.noname-b 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.187+0000 7ff443fff640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff448009450 con 0x7ff44c06e5e0 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.190+0000 7ff443fff640 1 -- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 msgr2=0x7ff44c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.190+0000 7ff443fff640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00d890 tx=0x7ff43c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:59.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.190+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:57:59.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00de80 con 0x7ff44c06e5e0 2024-09-15T01:57:59.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: dump: 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: _finish_auth 0 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 5 monclient: authenticate success, global_id 6085 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c00bae0 con 0x7ff44c06e5e0 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 10 monclient: _renew_subs 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44c122760 con 0x7ff44c06e5e0 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00bea0 con 0x7ff44c06e5e0 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff440ff9640 10 monclient: dump: 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:57:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 10 monclient: _renew_subs 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44c13dc00 con 0x7ff44c06e5e0 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.192+0000 7ff4427fc640 4 set_mon_vals no callback set 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff43c014640 con 0x7ff44c06e5e0 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff452ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:59.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff452ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff452ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44c04bf60 con 0x7ff44c06e5e0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff440ff9640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 0x7ff43803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 9845+0+0 (secure 0 0 0) 0x7ff43c04d530 con 0x7ff44c06e5e0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff450c5c640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 0x7ff43803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.193+0000 7ff450c5c640 10 monclient: get_auth_request con 0x7ff438038740 auth_method 0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.195+0000 7ff450c5c640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 0x7ff43803abf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff448034000 tx=0x7ff448032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.197+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff43c01e460 con 0x7ff44c06e5e0 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.197+0000 7ff440ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:57:59.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.197+0000 7ff440ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:57:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.367+0000 7ff452ec0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:57:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.367+0000 7ff452ec0640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:57:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.367+0000 7ff452ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:57:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:57:59.367+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7ff44c09f980 con 0x7ff44c06e5e0 2024-09-15T01:58:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.086+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v165) v1 ==== 132+0+0 (secure 0 0 0) 0x7ff43c01dbb0 con 0x7ff44c06e5e0 2024-09-15T01:58:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.086+0000 7ff440ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:58:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.086+0000 7ff440ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:00.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.193+0000 7ff452ec0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:58:00.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.193+0000 7ff452ec0640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:58:00.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.193+0000 7ff452ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:00.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:00.193+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7ff44c068a20 con 0x7ff44c06e5e0 2024-09-15T01:58:01.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.088+0000 7ff440ff9640 1 -- 172.21.15.67:0/3493173982 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v166) v1 ==== 132+0+0 (secure 0 0 0) 0x7ff43c011340 con 0x7ff44c06e5e0 2024-09-15T01:58:01.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.088+0000 7ff440ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T01:58:01.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.088+0000 7ff440ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 msgr2=0x7ff43803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 0x7ff43803abf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff448034000 tx=0x7ff448032000 comp rx=0 tx=0).stop 2024-09-15T01:58:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 10 monclient: shutdown 2024-09-15T01:58:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 msgr2=0x7ff44c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.091+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00d890 tx=0x7ff43c008040 comp rx=0 tx=0).stop 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 shutdown_connections 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff438038740 0x7ff43803abf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff44c09e050 0x7ff44c141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06e5e0 0x7ff44c13d6c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 --2- 172.21.15.67:0/3493173982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff44c06e210 0x7ff44c13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 >> 172.21.15.67:0/3493173982 conn(0x7ff44c070ff0 msgr2=0x7ff44c0a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 shutdown_connections 2024-09-15T01:58:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.092+0000 7ff452ec0640 1 -- 172.21.15.67:0/3493173982 wait complete. 2024-09-15T01:58:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:58:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:58:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:58:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '7\.' 2024-09-15T01:58:01.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.185+0000 7f2df7b9d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:01.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.185+0000 7f2df7b9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.185+0000 7f2df7b9d640 10 monclient: monmap: 2024-09-15T01:58:01.186 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:01.185420+0000 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:01.185420+0000 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:01.187 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.185+0000 7f2df7b9d640 1 Processor -- start 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 1 -- start start 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient: init 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2df0068dc0 0x7f2df0132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2df0068dc0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2df0069190 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2df0137150 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df0096c10 con 0x7f2df0137150 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df009ff20 con 0x7f2df0069190 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df006a2a0 con 0x7f2df0068dc0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df7b9d640 10 monclient(hunting): authenticate will time out at 1113.654651s 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2df0068dc0 0x7f2df0132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df613a640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f2df0137150 legacy=0x7f2df0139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5138640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2df0069190 0x7f2df0069590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47516/0 (socket says 172.21.15.67:47516) 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5138640 1 -- 172.21.15.67:0/2246303699 learned_addr learned my addr 172.21.15.67:0/2246303699 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:01.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): get_auth_request con 0x7f2df0068dc0 auth_method 0 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.186+0000 7f2df5939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 10 monclient(hunting): handle_auth_done global_id 6088 payload 274 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 1 -- 172.21.15.67:0/2246303699 >> v1:172.21.15.67:6789/0 conn(0x7f2df0069190 legacy=0x7f2df0069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 1 -- 172.21.15.67:0/2246303699 >> v1:172.21.15.17:6789/0 conn(0x7f2df0137150 legacy=0x7f2df0139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 1 monclient: found mon.noname-c 2024-09-15T01:58:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.187+0000 7f2df5939640 1 -- 172.21.15.67:0/2246303699 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df00a0180 con 0x7f2df0068dc0 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.189+0000 7f2df5939640 1 -- 172.21.15.67:0/2246303699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0068dc0 msgr2=0x7f2df0132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.189+0000 7f2df5939640 1 --2- 172.21.15.67:0/2246303699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0068dc0 0x7f2df0132250 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009850 tx=0x7f2de40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b420e65567631b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 1 -- 172.21.15.67:0/2246303699 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4032440 con 0x7f2df0068dc0 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: dump: 2024-09-15T01:58:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 5 monclient: authenticate success, global_id 6088 2024-09-15T01:58:01.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 1 -- 172.21.15.67:0/2246303699 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de40325a0 con 0x7f2df0068dc0 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 1 -- 172.21.15.67:0/2246303699 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4032940 con 0x7f2df0068dc0 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 4 set_mon_vals no callback set 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df4937640 10 monclient: dump: 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 10 monclient: shutdown 2024-09-15T01:58:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2246303699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0068dc0 msgr2=0x7f2df0132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2246303699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0068dc0 0x7f2df0132250 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009850 tx=0x7f2de40090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2246303699 shutdown_connections 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2246303699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0068dc0 0x7f2df0132250 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2246303699 >> 172.21.15.67:0/2246303699 conn(0x7f2df00711b0 msgr2=0x7f2df00a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2246303699 shutdown_connections 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.190+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2246303699 wait complete. 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 10 monclient: monmap: 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.202 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 1 Processor -- start 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 1 -- start start 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 10 monclient: init 2024-09-15T01:58:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.191+0000 7f2df7b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 0x7f2df012eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2df0068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2df0069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0137150 0x7f2df0098480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2df0137150 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df009bba0 con 0x7f2df0068dc0 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df012fc20 con 0x7f2df0069190 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df009b2f0 con 0x7f2df0137150 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df7b9d640 10 monclient(hunting): authenticate will time out at 1113.660063s 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39072/0 (socket says 172.21.15.67:39072) 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 -- 172.21.15.67:0/2747709009 learned_addr learned my addr 172.21.15.67:0/2747709009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): get_auth_request con 0x7f2df0069190 auth_method 0 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5939640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 msgr2=0x7f2df012eaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5939640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 0x7f2df012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df613a640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0137150 0x7f2df0098480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df613a640 10 monclient(hunting): get_auth_request con 0x7f2df0137150 auth_method 0 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df613a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df613a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df613a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient(hunting): handle_auth_done global_id 6091 payload 824 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0137150 msgr2=0x7f2df0098480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0137150 0x7f2df0098480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 msgr2=0x7f2df012eaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 0x7f2df012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 monclient: found mon.noname-b 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.192+0000 7f2df5138640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de4009390 con 0x7f2df0069190 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df5138640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 msgr2=0x7f2df0097f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df5138640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00cca0 tx=0x7f2dec0053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00aa60 con 0x7f2df0069190 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: dump: 2024-09-15T01:58:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec00abe0 con 0x7f2df0069190 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0070f0 con 0x7f2df0069190 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df7b9d640 5 monclient: authenticate success, global_id 6091 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df7b9d640 10 monclient: _renew_subs 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df009b5e0 con 0x7f2df0069190 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de3fff640 4 set_mon_vals no callback set 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.194+0000 7f2de1ffb640 10 monclient: dump: 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.209 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.195+0000 7f2df7b9d640 10 monclient: _renew_subs 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.195+0000 7f2df7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.195+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df0099ab0 con 0x7f2df0069190 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.195+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2dec007520 con 0x7f2df0069190 2024-09-15T01:58:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2de1ffb640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 0x7f2dd403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(166..166 src has 140..166) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f2dec04d230 con 0x7f2df0069190 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df7b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df7b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df5939640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 0x7f2dd403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df009a740 con 0x7f2df0069190 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.196+0000 7f2df5939640 10 monclient: get_auth_request con 0x7f2dd40386f0 auth_method 0 2024-09-15T01:58:01.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.198+0000 7f2df5939640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 0x7f2dd403aba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034000 tx=0x7f2de4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.200+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2dec01a590 con 0x7f2df0069190 2024-09-15T01:58:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.200+0000 7f2de1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.200+0000 7f2de1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:01.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2df7b9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2df7b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2df7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2df004bf60 con 0x7f2df0069190 2024-09-15T01:58:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f2dec010af0 con 0x7f2df0069190 2024-09-15T01:58:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2de1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.370+0000 7f2de1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.473+0000 7f2df7b9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.473+0000 7f2df7b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.473+0000 7f2df7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.473+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2df009b770 con 0x7f2df0069190 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.474+0000 7f2de1ffb640 1 -- 172.21.15.67:0/2747709009 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f2dec010cd0 con 0x7f2df0069190 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.474+0000 7f2de1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.474+0000 7f2de1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:01.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 msgr2=0x7f2dd403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 0x7f2dd403aba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034000 tx=0x7f2de4032000 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 10 monclient: shutdown 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 msgr2=0x7f2df0097f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00cca0 tx=0x7f2dec0053f0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 shutdown_connections 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2dd40386f0 0x7f2dd403aba0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2df0137150 0x7f2df0098480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2df0069190 0x7f2df0097f40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 --2- 172.21.15.67:0/2747709009 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2df0068dc0 0x7f2df012eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 >> 172.21.15.67:0/2747709009 conn(0x7f2df00711b0 msgr2=0x7f2df009e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 shutdown_connections 2024-09-15T01:58:01.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.476+0000 7f2df7b9d640 1 -- 172.21.15.67:0/2747709009 wait complete. 2024-09-15T01:58:01.486 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:01.486 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:01.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:58:01.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:58:01.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:58:01.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '8\.' 2024-09-15T01:58:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 10 monclient: get_monmap_and_config 2024-09-15T01:58:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 10 monclient: monmap: 2024-09-15T01:58:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:01.561320+0000 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:01.561320+0000 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:01.563 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 1 Processor -- start 2024-09-15T01:58:01.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 1 -- start start 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.561+0000 7fa876c6f640 10 monclient: init 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 0x7fa8701321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa870098140 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8700a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa870098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8700a0b80 con 0x7fa8700a0370 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8700a1730 con 0x7fa870098140 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700a00b0 con 0x7fa870098c20 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa876c6f640 10 monclient(hunting): authenticate will time out at 1114.030447s 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa86ffff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fa8700a0370 legacy=0x7fa870098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa874a0b640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 msgr2=0x7fa8701321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa874a0b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 0x7fa8701321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35752/0 (socket says 172.21.15.67:35752) 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 1 -- 172.21.15.67:0/4131012148 learned_addr learned my addr 172.21.15.67:0/4131012148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): get_auth_request con 0x7fa870098c20 auth_method 0 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.562+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 10 monclient(hunting): handle_auth_done global_id 94202 payload 274 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 1 -- 172.21.15.67:0/4131012148 >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 msgr2=0x7fa8701321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 1 --2- 172.21.15.67:0/4131012148 >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 0x7fa8701321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 1 -- 172.21.15.67:0/4131012148 >> v1:172.21.15.17:6789/0 conn(0x7fa8700a0370 legacy=0x7fa870098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 1 monclient: found mon.noname-b 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:01.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.563+0000 7fa87520c640 1 -- 172.21.15.67:0/4131012148 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870071390 con 0x7fa870098c20 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa87520c640 1 -- 172.21.15.67:0/4131012148 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 msgr2=0x7fa8701390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa87520c640 1 --2- 172.21.15.67:0/4131012148 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa868009f20 tx=0x7fa868009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d3656ae403ddbb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 1 -- 172.21.15.67:0/4131012148 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868007610 con 0x7fa870098c20 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: dump: 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.572 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 1 -- 172.21.15.67:0/4131012148 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86802d050 con 0x7fa870098c20 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 1 -- 172.21.15.67:0/4131012148 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680079a0 con 0x7fa870098c20 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa86f7fe640 10 monclient: dump: 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.574 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa876c6f640 5 monclient: authenticate success, global_id 94202 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa876c6f640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.566+0000 7fa876c6f640 4 set_mon_vals no callback set 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 10 monclient: shutdown 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 -- 172.21.15.67:0/4131012148 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 msgr2=0x7fa8701390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 --2- 172.21.15.67:0/4131012148 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa868009f20 tx=0x7fa868009840 comp rx=0 tx=0).stop 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 -- 172.21.15.67:0/4131012148 shutdown_connections 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 --2- 172.21.15.67:0/4131012148 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa8701390c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 --2- 172.21.15.67:0/4131012148 >> v2:172.21.15.17:3300/0 conn(0x7fa870098140 0x7fa8701321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 -- 172.21.15.67:0/4131012148 >> 172.21.15.67:0/4131012148 conn(0x7fa8700a5780 msgr2=0x7fa8700a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 -- 172.21.15.67:0/4131012148 shutdown_connections 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 1 -- 172.21.15.67:0/4131012148 wait complete. 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.567+0000 7fa876c6f640 10 monclient: monmap: 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:01.576 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 Processor -- start 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 -- start start 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient: init 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 0x7fa870138930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa870098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa870130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa870098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8700a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa870133420 con 0x7fa870098140 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa870135620 con 0x7fa8700a0370 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa870134960 con 0x7fa870098c20 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa876c6f640 10 monclient(hunting): authenticate will time out at 1114.036875s 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa874a0b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 msgr2=0x7fa870138930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa874a0b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 0x7fa870138930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.568+0000 7fa87520c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39080/0 (socket says 172.21.15.67:39080) 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 -- 172.21.15.67:0/3751775180 learned_addr learned my addr 172.21.15.67:0/3751775180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): get_auth_request con 0x7fa8700a0370 auth_method 0 2024-09-15T01:58:01.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa870130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): get_auth_request con 0x7fa870098c20 auth_method 0 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient(hunting): handle_auth_done global_id 6094 payload 824 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 -- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 msgr2=0x7fa870130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa870130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 -- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 msgr2=0x7fa870138930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 0x7fa870138930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 monclient: found mon.noname-b 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.569+0000 7fa87520c640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868009ae0 con 0x7fa8700a0370 2024-09-15T01:58:01.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa87520c640 1 -- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 msgr2=0x7fa8701311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa87520c640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa868009ab0 tx=0x7fa868036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868031cd0 con 0x7fa8700a0370 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: dump: 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.583 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 5 monclient: authenticate success, global_id 6094 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 10 monclient: _renew_subs 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa870137e20 con 0x7fa8700a0370 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868031e30 con 0x7fa8700a0370 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86803c670 con 0x7fa8700a0370 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:01.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86cff9640 10 monclient: dump: 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 10 monclient: _renew_subs 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa86effd640 4 set_mon_vals no callback set 2024-09-15T01:58:01.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.571+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa870134c40 con 0x7fa8700a0370 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa86803ca60 con 0x7fa8700a0370 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa86cff9640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 0x7fa85403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa876c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa876c6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa876c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87004bf60 con 0x7fa8700a0370 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(166..166 src has 140..166) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fa86806d820 con 0x7fa8700a0370 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa874a0b640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 0x7fa85403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.572+0000 7fa874a0b640 10 monclient: get_auth_request con 0x7fa8540386f0 auth_method 0 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.575+0000 7fa874a0b640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 0x7fa85403aba0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa864006fd0 tx=0x7fa864006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.577+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa868049080 con 0x7fa8700a0370 2024-09-15T01:58:01.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.577+0000 7fa86cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.577+0000 7fa86cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:01.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.747+0000 7fa876c6f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.747+0000 7fa876c6f640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.747+0000 7fa876c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.747+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa8701326e0 con 0x7fa8700a0370 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.748+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fa868040070 con 0x7fa8700a0370 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.748+0000 7fa86cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.748+0000 7fa86cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.850+0000 7fa876c6f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.850+0000 7fa876c6f640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.850+0000 7fa876c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.850+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa8700008d0 con 0x7fa8700a0370 2024-09-15T01:58:01.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.851+0000 7fa86cff9640 1 -- 172.21.15.67:0/3751775180 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fa86803a6d0 con 0x7fa8700a0370 2024-09-15T01:58:01.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.851+0000 7fa86cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:01.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.851+0000 7fa86cff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 msgr2=0x7fa85403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 0x7fa85403aba0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa864006fd0 tx=0x7fa864006d30 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 10 monclient: shutdown 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 msgr2=0x7fa8701311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa868009ab0 tx=0x7fa868036000 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 shutdown_connections 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa8540386f0 0x7fa85403aba0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8700a0370 0x7fa8701311b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa870098c20 0x7fa870130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 --2- 172.21.15.67:0/3751775180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa870098140 0x7fa870138930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 >> 172.21.15.67:0/3751775180 conn(0x7fa8700a5780 msgr2=0x7fa87009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 shutdown_connections 2024-09-15T01:58:01.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.853+0000 7fa876c6f640 1 -- 172.21.15.67:0/3751775180 wait complete. 2024-09-15T01:58:01.863 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:01.863 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:01.863 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T01:58:01.864 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep 'osd\.0' 2024-09-15T01:58:01.864 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep globster 2024-09-15T01:58:01.864 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep '6\.' 2024-09-15T01:58:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.938+0000 7f258e634640 10 monclient: get_monmap_and_config 2024-09-15T01:58:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.938+0000 7f258e634640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.938+0000 7f258e634640 10 monclient: monmap: 2024-09-15T01:58:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:01.938413+0000 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:01.938413+0000 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:01.940 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 Processor -- start 2024-09-15T01:58:01.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 -- start start 2024-09-15T01:58:01.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient: init 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): picked mon.noname-b con 0x7f2588068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 0x7f2588069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): picked mon.noname-a con 0x7f2588069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f258812e960 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2588096c10 con 0x7f258812e960 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880a14d0 con 0x7f2588069190 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258809ff20 con 0x7f2588068dc0 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258e634640 10 monclient(hunting): authenticate will time out at 1114.407701s 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f258cbd1640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f258812e960 legacy=0x7f258812edc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f257ffff640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 msgr2=0x7f2588069560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.939+0000 7f257ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 0x7f2588069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35774/0 (socket says 172.21.15.67:35774) 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 -- 172.21.15.67:0/381537704 learned_addr learned my addr 172.21.15.67:0/381537704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): get_auth_request con 0x7f2588068dc0 auth_method 0 2024-09-15T01:58:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient(hunting): handle_auth_done global_id 94208 payload 274 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 -- 172.21.15.67:0/381537704 >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 msgr2=0x7f2588069560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:01.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 --2- 172.21.15.67:0/381537704 >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 0x7f2588069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 -- 172.21.15.67:0/381537704 >> v1:172.21.15.17:6789/0 conn(0x7f258812e960 legacy=0x7f258812edc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 monclient: found mon.noname-b 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:01.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.940+0000 7f2587fff640 1 -- 172.21.15.67:0/381537704 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880a0180 con 0x7f2588068dc0 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f2587fff640 1 -- 172.21.15.67:0/381537704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 msgr2=0x7f2588132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f2587fff640 1 --2- 172.21.15.67:0/381537704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2578009910 tx=0x7f25780091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a59dfc3bb7d6e58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 1 -- 172.21.15.67:0/381537704 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578032440 con 0x7f2588068dc0 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: dump: 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 1 -- 172.21.15.67:0/381537704 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25780325a0 con 0x7f2588068dc0 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f258e634640 5 monclient: authenticate success, global_id 94208 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f258e634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f258e634640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 1 -- 172.21.15.67:0/381537704 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578032940 con 0x7f2588068dc0 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f258e634640 4 set_mon_vals no callback set 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.943+0000 7f25877fe640 10 monclient: dump: 2024-09-15T01:58:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 10 monclient: shutdown 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 -- 172.21.15.67:0/381537704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 msgr2=0x7f2588132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 --2- 172.21.15.67:0/381537704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2578009910 tx=0x7f25780091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 -- 172.21.15.67:0/381537704 shutdown_connections 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 --2- 172.21.15.67:0/381537704 >> v2:172.21.15.17:3300/0 conn(0x7f2588069190 0x7f2588069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 --2- 172.21.15.67:0/381537704 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f2588132250 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 -- 172.21.15.67:0/381537704 >> 172.21.15.67:0/381537704 conn(0x7f25880711b0 msgr2=0x7f25880a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 -- 172.21.15.67:0/381537704 shutdown_connections 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.944+0000 7f258e634640 1 -- 172.21.15.67:0/381537704 wait complete. 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient: build_initial_monmap 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient: monmap: 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 1 Processor -- start 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 1 -- start start 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient: init 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f258812dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient(hunting): picked mon.noname-c con 0x7f2588068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient(hunting): picked mon.noname-b con 0x7f2588069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 0x7f2588141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.945+0000 7f258e634640 10 monclient(hunting): picked mon.noname-a con 0x7f258812e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258811c680 con 0x7f258812e960 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258811c0c0 con 0x7f2588069190 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2588141f00 con 0x7f2588068dc0 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258e634640 10 monclient(hunting): authenticate will time out at 1114.414027s 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258cbd1640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 msgr2=0x7f2588141370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f258cbd1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 0x7f2588141370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39088/0 (socket says 172.21.15.67:39088) 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 -- 172.21.15.67:0/3111305826 learned_addr learned my addr 172.21.15.67:0/3111305826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): get_auth_request con 0x7f2588069190 auth_method 0 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f258812dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): get_auth_request con 0x7f2588068dc0 auth_method 0 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient(hunting): handle_auth_done global_id 6097 payload 824 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 -- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 msgr2=0x7f258812dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f258812dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 -- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 msgr2=0x7f2588141370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 0x7f2588141370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 monclient: found mon.noname-b 2024-09-15T01:58:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.946+0000 7f257ffff640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578009450 con 0x7f2588069190 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.948+0000 7f257ffff640 1 -- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 msgr2=0x7f258812e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.948+0000 7f257ffff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f257000b5c0 tx=0x7f2570009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.948+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570003cc0 con 0x7f2588069190 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: dump: 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2570003e40 con 0x7f2588069190 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 5 monclient: authenticate success, global_id 6097 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570005340 con 0x7f2588069190 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _renew_subs 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258811bde0 con 0x7f2588069190 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2586ffd640 4 set_mon_vals no callback set 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 10 monclient: dump: 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _renew_subs 2024-09-15T01:58:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25881418b0 con 0x7f2588069190 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f257001a460 con 0x7f2588069190 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.949+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258804bf60 con 0x7f2588069190 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.950+0000 7f2584ff9640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 0x7f256803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.950+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(166..166 src has 140..166) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f257004c950 con 0x7f2588069190 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.950+0000 7f2587fff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 0x7f256803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:01.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.950+0000 7f2587fff640 10 monclient: get_auth_request con 0x7f2568038720 auth_method 0 2024-09-15T01:58:01.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.952+0000 7f2587fff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 0x7f256803abd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f258806a4e0 tx=0x7f2578032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:01.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:01.954+0000 7f257ffff640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f257000b5c0 tx=0x7f2570009120 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f2570017ad0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:58:02.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.782+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2570017ad0 con 0x7f2588069190 2024-09-15T01:58:02.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.782+0000 7f2584ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:02.784 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.782+0000 7f2584ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:02.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.956+0000 7f258e634640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.956+0000 7f258e634640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.956+0000 7f258e634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.956+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2588069aa0 con 0x7f2588069190 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.957+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f2570017310 con 0x7f2588069190 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.957+0000 7f2584ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:02.958 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:02.957+0000 7f2584ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f258e634640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f258e634640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f258e634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f258812e740 con 0x7f2588069190 2024-09-15T01:58:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f2584ff9640 1 -- 172.21.15.67:0/3111305826 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v166) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f25700174f0 con 0x7f2588069190 2024-09-15T01:58:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f2584ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T01:58:03.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.059+0000 7f2584ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 msgr2=0x7f256803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 0x7f256803abd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f258806a4e0 tx=0x7f2578032000 comp rx=0 tx=0).stop 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 10 monclient: shutdown 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 msgr2=0x7f258812e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f257000b5c0 tx=0x7f2570009120 comp rx=0 tx=0).stop 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 shutdown_connections 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038720 0x7f256803abd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f258812e960 0x7f2588141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2588069190 0x7f258812e1f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 --2- 172.21.15.67:0/3111305826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2588068dc0 0x7f258812dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 >> 172.21.15.67:0/3111305826 conn(0x7f25880711b0 msgr2=0x7f25881302f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 shutdown_connections 2024-09-15T01:58:03.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.061+0000 7f258e634640 1 -- 172.21.15.67:0/3111305826 wait complete. 2024-09-15T01:58:03.071 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:03.071 INFO:tasks.workunit.client.0.smithi067.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T01:58:03.072 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T01:58:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 10 monclient: get_monmap_and_config 2024-09-15T01:58:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 10 monclient: build_initial_monmap 2024-09-15T01:58:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 10 monclient: monmap: 2024-09-15T01:58:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:03.146340+0000 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:03.146340+0000 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:03.148 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 1 Processor -- start 2024-09-15T01:58:03.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 1 -- start start 2024-09-15T01:58:03.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.146+0000 7f8463e2b640 10 monclient: init 2024-09-15T01:58:03.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 0x7f845c132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-a con 0x7f845c068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f845c069190 0x7f845c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f845c069190 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f845c12e960 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c096c10 con 0x7f845c12e960 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c0a14d0 con 0x7f845c068dc0 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c09ff20 con 0x7f845c069190 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:03.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8463e2b640 10 monclient(hunting): authenticate will time out at 1115.615462s 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f845c069190 0x7f845c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8461bc7640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 msgr2=0x7f845c132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84623c8640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f845c12e960 legacy=0x7f845c12ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f845c069190 0x7f845c069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39096/0 (socket says 172.21.15.67:39096) 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 -- 172.21.15.67:0/3369512443 learned_addr learned my addr 172.21.15.67:0/3369512443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): get_auth_request con 0x7f845c069190 auth_method 0 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f8461bc7640 1 --2- 172.21.15.67:0/3369512443 >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 0x7f845c132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:03.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient(hunting): handle_auth_done global_id 6100 payload 274 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 -- 172.21.15.67:0/3369512443 >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 msgr2=0x7f845c132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 --2- 172.21.15.67:0/3369512443 >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 0x7f845c132250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 -- 172.21.15.67:0/3369512443 >> v1:172.21.15.17:6789/0 conn(0x7f845c12e960 legacy=0x7f845c12ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient: _finish_hunting 0 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 monclient: found mon.noname-c 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.147+0000 7f84613c6640 1 -- 172.21.15.67:0/3369512443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0a0180 con 0x7f845c069190 2024-09-15T01:58:03.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.150+0000 7f84613c6640 1 -- 172.21.15.67:0/3369512443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c069190 msgr2=0x7f845c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:03.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.150+0000 7f84613c6640 1 --2- 172.21.15.67:0/3369512443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c069190 0x7f845c069590 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f845c06a680 tx=0x7f844c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac8c8205c72c79c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.150+0000 7f8460bc5640 1 -- 172.21.15.67:0/3369512443 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c032440 con 0x7f845c069190 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: dump: 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:03.158 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: _finish_auth 0 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 1 -- 172.21.15.67:0/3369512443 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c0325a0 con 0x7f845c069190 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 5 monclient: authenticate success, global_id 6100 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 1 -- 172.21.15.67:0/3369512443 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c032940 con 0x7f845c069190 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 4 set_mon_vals no callback set 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8460bc5640 10 monclient: dump: 2024-09-15T01:58:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 10 monclient: shutdown 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 -- 172.21.15.67:0/3369512443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c069190 msgr2=0x7f845c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:03.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 --2- 172.21.15.67:0/3369512443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c069190 0x7f845c069590 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f845c06a680 tx=0x7f844c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 -- 172.21.15.67:0/3369512443 shutdown_connections 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 --2- 172.21.15.67:0/3369512443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c069190 0x7f845c069590 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 --2- 172.21.15.67:0/3369512443 >> v2:172.21.15.17:3300/0 conn(0x7f845c068dc0 0x7f845c132250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 -- 172.21.15.67:0/3369512443 >> 172.21.15.67:0/3369512443 conn(0x7f845c0711b0 msgr2=0x7f845c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 -- 172.21.15.67:0/3369512443 shutdown_connections 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.151+0000 7f8463e2b640 1 -- 172.21.15.67:0/3369512443 wait complete. 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 10 monclient: build_initial_monmap 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 10 monclient: monmap: 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:03.161 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 1 Processor -- start 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 1 -- start start 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 10 monclient: init 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.152+0000 7f8463e2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-b con 0x7f845c068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f845c069190 0x7f845c12db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f845c069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 0x7f845c0980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): picked mon.noname-a con 0x7f845c12e960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0a0890 con 0x7f845c12e960 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c134cb0 con 0x7f845c068dc0 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c09b740 con 0x7f845c069190 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8463e2b640 10 monclient(hunting): authenticate will time out at 1115.621077s 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39098/0 (socket says 172.21.15.67:39098) 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 -- 172.21.15.67:0/1191239290 learned_addr learned my addr 172.21.15.67:0/1191239290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:03.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): get_auth_request con 0x7f845c068dc0 auth_method 0 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f84623c8640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 msgr2=0x7f845c0980f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f84623c8640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 0x7f845c0980f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient(hunting): handle_auth_done global_id 6103 payload 824 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f84613c6640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f845c069190 0x7f845c12db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f845c069190 msgr2=0x7f845c12db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f845c069190 0x7f845c12db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 msgr2=0x7f845c0980f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 0x7f845c0980f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient: _finish_hunting 0 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 monclient: found mon.noname-b 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.153+0000 7f8461bc7640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c009450 con 0x7f845c068dc0 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8461bc7640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 msgr2=0x7f845c141380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8461bc7640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f845800b600 tx=0x7f84580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8458015070 con 0x7f845c068dc0 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: dump: 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8458003d90 con 0x7f845c068dc0 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8463e2b640 5 monclient: authenticate success, global_id 6103 2024-09-15T01:58:03.166 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8463e2b640 10 monclient: _renew_subs 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8463e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c09ba20 con 0x7f845c068dc0 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8458006380 con 0x7f845c068dc0 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8451ffb640 10 monclient: dump: 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.156+0000 7f8463e2b640 10 monclient: _renew_subs 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.156+0000 7f8463e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.155+0000 7f8453fff640 4 set_mon_vals no callback set 2024-09-15T01:58:03.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.156+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c1418c0 con 0x7f845c068dc0 2024-09-15T01:58:03.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.156+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8458010440 con 0x7f845c068dc0 2024-09-15T01:58:03.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8463e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:03.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8463e2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:03.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8463e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c04bf60 con 0x7f845c068dc0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8451ffb640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 0x7f844003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(166..166 src has 140..166) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f845804ca50 con 0x7f845c068dc0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f84613c6640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 0x7f844003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.157+0000 7f84613c6640 10 monclient: get_auth_request con 0x7f84400386f0 auth_method 0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.159+0000 7f84613c6640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 0x7f844003aba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f845c0a0510 tx=0x7f844c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.161+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8458018ab0 con 0x7f845c068dc0 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.161+0000 7f8451ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:03.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.161+0000 7f8451ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:03.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.328+0000 7f8463e2b640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:03.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.328+0000 7f8463e2b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:03.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.328+0000 7f8463e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:03.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:03.328+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f845c069ad0 con 0x7f845c068dc0 2024-09-15T01:58:04.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.100+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v167) v1 ==== 123+0+0 (secure 0 0 0) 0x7f8458018690 con 0x7f845c068dc0 2024-09-15T01:58:04.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.100+0000 7f8451ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:04.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.100+0000 7f8451ffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T01:58:04.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.205+0000 7f8463e2b640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:04.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.205+0000 7f8463e2b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:04.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.205+0000 7f8463e2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:04.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:04.205+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f845c0008d0 con 0x7f845c068dc0 2024-09-15T01:58:05.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.102+0000 7f8451ffb640 1 -- 172.21.15.67:0/1191239290 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v168) v1 ==== 149+0+0 (secure 0 0 0) 0x7f84580280a0 con 0x7f845c068dc0 2024-09-15T01:58:05.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.102+0000 7f8451ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T01:58:05.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.102+0000 7f8451ffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:58:05.105 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 msgr2=0x7f844003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 0x7f844003aba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f845c0a0510 tx=0x7f844c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 10 monclient: shutdown 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 msgr2=0x7f845c141380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f845800b600 tx=0x7f84580091a0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 shutdown_connections 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f84400386f0 0x7f844003aba0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f845c12e960 0x7f845c0980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f845c069190 0x7f845c12db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 --2- 172.21.15.67:0/1191239290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c068dc0 0x7f845c141380 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 >> 172.21.15.67:0/1191239290 conn(0x7f845c0711b0 msgr2=0x7f845c134f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:05.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 shutdown_connections 2024-09-15T01:58:05.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.106+0000 7f8463e2b640 1 -- 172.21.15.67:0/1191239290 wait complete. 2024-09-15T01:58:05.119 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-15T01:58:05.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.200+0000 7fd298a99640 10 monclient: get_monmap_and_config 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.200+0000 7fd298a99640 10 monclient: build_initial_monmap 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.200+0000 7fd298a99640 10 monclient: monmap: 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:05.201688+0000 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:05.201688+0000 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:05.203 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:05.204 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:05.204 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 Processor -- start 2024-09-15T01:58:05.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 -- start start 2024-09-15T01:58:05.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient: init 2024-09-15T01:58:05.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 0x7fd29409bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-b con 0x7fd29409b840 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd29412fef0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd294130300 0x7fd2940677a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-c con 0x7fd294130300 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29409e8b0 con 0x7fd29412fef0 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29409e490 con 0x7fd294130300 2024-09-15T01:58:05.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29409db80 con 0x7fd29409b840 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd298a99640 10 monclient(hunting): authenticate will time out at 1117.670841s 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.201+0000 7fd292d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd294130300 0x7fd2940677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd28bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd29412fef0 0x7fd294067090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47532/0 (socket says 172.21.15.67:47532) 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd28bfff640 1 -- 172.21.15.67:0/3751828007 learned_addr learned my addr 172.21.15.67:0/3751828007 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): get_auth_request con 0x7fd294130300 auth_method 0 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:05.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 0x7fd29409bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 10 monclient(hunting): get_auth_request con 0x7fd29409b840 auth_method 0 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient(hunting): handle_auth_done global_id 6106 payload 274 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 1 -- 172.21.15.67:0/3751828007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 msgr2=0x7fd29409bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 0x7fd29409bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 1 -- 172.21.15.67:0/3751828007 >> v1:172.21.15.67:6789/0 conn(0x7fd29412fef0 legacy=0x7fd294067090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 1 monclient: found mon.noname-c 2024-09-15T01:58:05.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:05.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd292d9d640 1 -- 172.21.15.67:0/3751828007 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29409de60 con 0x7fd294130300 2024-09-15T01:58:05.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.202+0000 7fd29259c640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 0x7fd29409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:58:05.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd292d9d640 1 -- 172.21.15.67:0/3751828007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd294130300 msgr2=0x7fd2940677a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:05.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd292d9d640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd294130300 0x7fd2940677a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009e60 tx=0x7fd28c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f76ac989aa03afd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:05.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 1 -- 172.21.15.67:0/3751828007 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c007760 con 0x7fd294130300 2024-09-15T01:58:05.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: dump: 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:05.216 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 5 monclient: authenticate success, global_id 6106 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 1 -- 172.21.15.67:0/3751828007 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c02d050 con 0x7fd294130300 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 4 set_mon_vals no callback set 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 1 -- 172.21.15.67:0/3751828007 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c007af0 con 0x7fd294130300 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd291d9b640 10 monclient: dump: 2024-09-15T01:58:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 10 monclient: shutdown 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 -- 172.21.15.67:0/3751828007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd294130300 msgr2=0x7fd2940677a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd294130300 0x7fd2940677a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009e60 tx=0x7fd28c009780 comp rx=0 tx=0).stop 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 -- 172.21.15.67:0/3751828007 shutdown_connections 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd294130300 0x7fd2940677a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 --2- 172.21.15.67:0/3751828007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd29409b840 0x7fd29409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.205+0000 7fd298a99640 1 -- 172.21.15.67:0/3751828007 >> 172.21.15.67:0/3751828007 conn(0x7fd2940a2110 msgr2=0x7fd2940a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 1 -- 172.21.15.67:0/3751828007 shutdown_connections 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 1 -- 172.21.15.67:0/3751828007 wait complete. 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 10 monclient: build_initial_monmap 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 10 monclient: monmap: 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 1 Processor -- start 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 1 -- start start 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.206+0000 7fd298a99640 10 monclient: init 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-b con 0x7fd29409b840 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 0x7fd29412eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-a con 0x7fd29412fef0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 0x7fd294096a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): picked mon.noname-c con 0x7fd294130300 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29411e2c0 con 0x7fd29412fef0 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29411dee0 con 0x7fd29409b840 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29411dc00 con 0x7fd294130300 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd298a99640 10 monclient(hunting): authenticate will time out at 1117.676465s 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:05.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39122/0 (socket says 172.21.15.67:39122) 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 -- 172.21.15.67:0/445221277 learned_addr learned my addr 172.21.15.67:0/445221277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): get_auth_request con 0x7fd29409b840 auth_method 0 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd28bfff640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 msgr2=0x7fd29412eac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd28bfff640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 0x7fd29412eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient(hunting): handle_auth_done global_id 6109 payload 824 2024-09-15T01:58:05.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd292d9d640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 0x7fd294096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 msgr2=0x7fd294096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 0x7fd294096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 msgr2=0x7fd29412eac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 0x7fd29412eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 monclient: found mon.noname-b 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd29259c640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c009a20 con 0x7fd29409b840 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd292d9d640 10 monclient: get_auth_request con 0x7fd294130300 auth_method 0 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.207+0000 7fd292d9d640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 0x7fd294096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd29259c640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 msgr2=0x7fd294140dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:05.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd29259c640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd28400d890 tx=0x7fd284008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28400de80 con 0x7fd29409b840 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: dump: 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:05.224 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 5 monclient: authenticate success, global_id 6109 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28400bae0 con 0x7fd29409b840 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 10 monclient: _renew_subs 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28400bea0 con 0x7fd29409b840 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29411cd10 con 0x7fd29409b840 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:05.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd28a7fc640 10 monclient: dump: 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd29159a640 4 set_mon_vals no callback set 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 10 monclient: _renew_subs 2024-09-15T01:58:05.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.210+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd294141300 con 0x7fd29409b840 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd298a99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd298a99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd298a99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29404bf60 con 0x7fd29409b840 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd284014640 con 0x7fd29409b840 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd28a7fc640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 0x7fd27403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 140..168) v4 ==== 9156+0+0 (secure 0 0 0) 0x7fd28404d370 con 0x7fd29409b840 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd28bfff640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 0x7fd27403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:05.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.211+0000 7fd28bfff640 10 monclient: get_auth_request con 0x7fd2740386f0 auth_method 0 2024-09-15T01:58:05.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.214+0000 7fd28bfff640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 0x7fd27403aba0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0047d0 tx=0x7fd27c007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:05.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.215+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd2840148f0 con 0x7fd29409b840 2024-09-15T01:58:05.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.215+0000 7fd28a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:05.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.215+0000 7fd28a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:05.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.378+0000 7fd298a99640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:05.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.378+0000 7fd298a99640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:05.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.378+0000 7fd298a99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:05.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:05.378+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd29409c150 con 0x7fd29409b840 2024-09-15T01:58:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.156+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v169) v1 ==== 148+0+0 (secure 0 0 0) 0x7fd28401dbb0 con 0x7fd29409b840 2024-09-15T01:58:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.156+0000 7fd28a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.156+0000 7fd28a7fc640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-15T01:58:06.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd298a99640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd298a99640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd298a99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd294066910 con 0x7fd29409b840 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd28a7fc640 1 -- 172.21.15.67:0/445221277 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v169) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd28401dd90 con 0x7fd29409b840 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd28a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.256+0000 7fd28a7fc640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-15T01:58:06.259 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'cool' does not exist 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 msgr2=0x7fd27403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 0x7fd27403aba0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0047d0 tx=0x7fd27c007b80 comp rx=0 tx=0).stop 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 10 monclient: shutdown 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 msgr2=0x7fd294140dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:06.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd28400d890 tx=0x7fd284008040 comp rx=0 tx=0).stop 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 shutdown_connections 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd2740386f0 0x7fd27403aba0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd29412fef0 0x7fd29412eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd294130300 0x7fd294096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 --2- 172.21.15.67:0/445221277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd29409b840 0x7fd294140dc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 >> 172.21.15.67:0/445221277 conn(0x7fd2940a2110 msgr2=0x7fd2940a0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 shutdown_connections 2024-09-15T01:58:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.258+0000 7fd298a99640 1 -- 172.21.15.67:0/445221277 wait complete. 2024-09-15T01:58:06.270 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.345+0000 7f50f631d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.345+0000 7f50f631d640 10 monclient: build_initial_monmap 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.345+0000 7f50f631d640 10 monclient: monmap: 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:06.348 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:06.347057+0000 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:06.347057+0000 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:06.349 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.346+0000 7f50f631d640 1 Processor -- start 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.346+0000 7f50f631d640 1 -- start start 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.346+0000 7f50f631d640 10 monclient: init 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T01:58:06.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-b con 0x7f50f00a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:06.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50f006bef0 0x7f50f0099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-c con 0x7f50f006bef0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50f009db10 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f00696a0 con 0x7f50f009db10 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f00a3c90 con 0x7f50f006bef0 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0069940 con 0x7f50f00a3830 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f631d640 10 monclient(hunting): authenticate will time out at 1118.816183s 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50f006bef0 0x7f50f0099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f48ba640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50f009db10 0x7f50f009df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47538/0 (socket says 172.21.15.67:47538) 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50f48ba640 1 -- 172.21.15.67:0/271313783 learned_addr learned my addr 172.21.15.67:0/271313783 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): get_auth_request con 0x7f50f006bef0 auth_method 0 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_done global_id 6112 payload 274 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50effff640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 -- 172.21.15.67:0/271313783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 msgr2=0x7f50f006a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f006a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 -- 172.21.15.67:0/271313783 >> v1:172.21.15.67:6789/0 conn(0x7f50f009db10 legacy=0x7f50f009df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 monclient: found mon.noname-c 2024-09-15T01:58:06.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:06.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50ef7fe640 1 -- 172.21.15.67:0/271313783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0069c20 con 0x7f50f006bef0 2024-09-15T01:58:06.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50effff640 10 monclient: get_auth_request con 0x7f50f00a3830 auth_method 0 2024-09-15T01:58:06.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.347+0000 7f50effff640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f006a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50ef7fe640 1 -- 172.21.15.67:0/271313783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f006bef0 msgr2=0x7f50f0099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50ef7fe640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f006bef0 0x7f50f0099aa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009850 tx=0x7f50e40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b701822eac10777f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 1 -- 172.21.15.67:0/271313783 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4032440 con 0x7f50f006bef0 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: dump: 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50f631d640 5 monclient: authenticate success, global_id 6112 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50f631d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:06.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 1 -- 172.21.15.67:0/271313783 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e40325a0 con 0x7f50f006bef0 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50f631d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50f631d640 4 set_mon_vals no callback set 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 1 -- 172.21.15.67:0/271313783 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4032940 con 0x7f50f006bef0 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.350+0000 7f50eeffd640 10 monclient: dump: 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.366 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 10 monclient: shutdown 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 -- 172.21.15.67:0/271313783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f006bef0 msgr2=0x7f50f0099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f006bef0 0x7f50f0099aa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009850 tx=0x7f50e40090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 -- 172.21.15.67:0/271313783 shutdown_connections 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f006bef0 0x7f50f0099aa0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 --2- 172.21.15.67:0/271313783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f006a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 -- 172.21.15.67:0/271313783 >> 172.21.15.67:0/271313783 conn(0x7f50f0070fb0 msgr2=0x7f50f00a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 -- 172.21.15.67:0/271313783 shutdown_connections 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 1 -- 172.21.15.67:0/271313783 wait complete. 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 10 monclient: build_initial_monmap 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.351+0000 7f50f631d640 10 monclient: monmap: 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:06.368 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 1 Processor -- start 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 1 -- start start 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 10 monclient: init 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 0x7f50f0133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.352+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-a con 0x7f50f006bef0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-b con 0x7f50f009db10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f0141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): picked mon.noname-c con 0x7f50f00a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f006ea50 con 0x7f50f006bef0 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0121fe0 con 0x7f50f009db10 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f01222c0 con 0x7f50f00a3830 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f631d640 10 monclient(hunting): authenticate will time out at 1118.822071s 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50effff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 msgr2=0x7f50f0133eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39142/0 (socket says 172.21.15.67:39142) 2024-09-15T01:58:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 -- 172.21.15.67:0/3429100044 learned_addr learned my addr 172.21.15.67:0/3429100044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): get_auth_request con 0x7f50f009db10 auth_method 0 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50effff640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 0x7f50f0133eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient(hunting): handle_auth_done global_id 6115 payload 824 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f48ba640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f0141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 -- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 msgr2=0x7f50f0141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f0141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 -- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 msgr2=0x7f50f0133eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 0x7f50f0133eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 monclient: found mon.noname-b 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50ef7fe640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4009390 con 0x7f50f009db10 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f48ba640 10 monclient: get_auth_request con 0x7f50f00a3830 auth_method 0 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.353+0000 7f50f48ba640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f0141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50ef7fe640 1 -- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 msgr2=0x7f50f0141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50ef7fe640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009850 tx=0x7f50e4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4031c70 con 0x7f50f009db10 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 10 monclient: dump: 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 10 monclient: _finish_auth 0 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e4031dd0 con 0x7f50f009db10 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50f631d640 5 monclient: authenticate success, global_id 6115 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50f631d640 10 monclient: _renew_subs 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50f631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f01225a0 con 0x7f50f009db10 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e403d690 con 0x7f50f009db10 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:06.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50d7fff640 10 monclient: dump: 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50f631d640 10 monclient: _renew_subs 2024-09-15T01:58:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50f631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.356+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f0121290 con 0x7f50f009db10 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.355+0000 7f50ee7fc640 4 set_mon_vals no callback set 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50f631d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50f631d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50f631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f004bf60 con 0x7f50f009db10 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f50e4039040 con 0x7f50f009db10 2024-09-15T01:58:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50d7fff640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 0x7f50dc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 140..169) v4 ==== 8485+0+0 (secure 0 0 0) 0x7f50e406c620 con 0x7f50f009db10 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50effff640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 0x7f50dc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.357+0000 7f50effff640 10 monclient: get_auth_request con 0x7f50dc0386f0 auth_method 0 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.360+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.359+0000 7f50effff640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 0x7f50dc03aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50e0006ca0 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.362+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f50e4044ab0 con 0x7f50f009db10 2024-09-15T01:58:06.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.362+0000 7f50d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:06.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.362+0000 7f50d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:06.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.525+0000 7f50f631d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.525+0000 7f50f631d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:06.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.525+0000 7f50f631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:06.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:06.525+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f50f0067120 con 0x7f50f009db10 2024-09-15T01:58:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.160+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v170) v1 ==== 148+0+0 (secure 0 0 0) 0x7f50e4037070 con 0x7f50f009db10 2024-09-15T01:58:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.160+0000 7f50d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.160+0000 7f50d7fff640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50f631d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50f631d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50f631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f50f0122400 con 0x7f50f009db10 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50d7fff640 1 -- 172.21.15.67:0/3429100044 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v170) v1 ==== 155+0+0 (secure 0 0 0) 0x7f50f0122400 con 0x7f50f009db10 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50d7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.260+0000 7f50d7fff640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-15T01:58:07.263 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'cold' does not exist 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 msgr2=0x7f50dc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 0x7f50dc03aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50e0006ca0 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 10 monclient: shutdown 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 msgr2=0x7f50f0141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009850 tx=0x7f50e4032000 comp rx=0 tx=0).stop 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 shutdown_connections 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50dc0386f0 0x7f50dc03aba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50f00a3830 0x7f50f0141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50f009db10 0x7f50f0141040 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 --2- 172.21.15.67:0/3429100044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50f006bef0 0x7f50f0133eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 >> 172.21.15.67:0/3429100044 conn(0x7f50f0070fb0 msgr2=0x7f50f0098a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 shutdown_connections 2024-09-15T01:58:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.262+0000 7f50f631d640 1 -- 172.21.15.67:0/3429100044 wait complete. 2024-09-15T01:58:07.274 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T01:58:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.348+0000 7f8b848d4640 10 monclient: get_monmap_and_config 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.348+0000 7f8b848d4640 10 monclient: build_initial_monmap 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.348+0000 7f8b848d4640 10 monclient: monmap: 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:07.349572+0000 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:07.349572+0000 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:07.351 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:07.352 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.348+0000 7f8b848d4640 1 Processor -- start 2024-09-15T01:58:07.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 -- start start 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient: init 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b8006e210 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b8006e5e0 0x7f8b80136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8006e5e0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 0x7f8b80139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-a con 0x7f8b80137240 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8006aa60 con 0x7f8b8006e210 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8006a190 con 0x7f8b80137240 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b80069bd0 con 0x7f8b8006e5e0 2024-09-15T01:58:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b848d4640 10 monclient(hunting): authenticate will time out at 1119.818663s 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b8006e5e0 0x7f8b80136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b8006e210 0x7f8b8006b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47546/0 (socket says 172.21.15.67:47546) 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7e59c640 1 -- 172.21.15.67:0/3466736289 learned_addr learned my addr 172.21.15.67:0/3466736289 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): get_auth_request con 0x7f8b8006e5e0 auth_method 0 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7ed9d640 1 -- 172.21.15.67:0/3466736289 >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 msgr2=0x7f8b80139620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7ed9d640 1 --2- 172.21.15.67:0/3466736289 >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 0x7f8b80139620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.349+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_done global_id 6118 payload 274 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/3466736289 >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 msgr2=0x7f8b80139620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 1 --2- 172.21.15.67:0/3466736289 >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 0x7f8b80139620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/3466736289 >> v1:172.21.15.67:6789/0 conn(0x7f8b8006e210 legacy=0x7f8b8006b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 1 monclient: found mon.noname-c 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.350+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/3466736289 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b800698f0 con 0x7f8b8006e5e0 2024-09-15T01:58:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.352+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/3466736289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 msgr2=0x7f8b80136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.352+0000 7f8b7dd9b640 1 --2- 172.21.15.67:0/3466736289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80136d00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8b800a0ce0 tx=0x7f8b740090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5a12d2f3aecd1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.352+0000 7f8b7d59a640 1 -- 172.21.15.67:0/3466736289 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74032440 con 0x7f8b8006e5e0 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: dump: 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:07.363 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: _finish_auth 0 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 5 monclient: authenticate success, global_id 6118 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 1 -- 172.21.15.67:0/3466736289 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b740325a0 con 0x7f8b8006e5e0 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 1 -- 172.21.15.67:0/3466736289 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74032940 con 0x7f8b8006e5e0 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 4 set_mon_vals no callback set 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b7d59a640 10 monclient: dump: 2024-09-15T01:58:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 10 monclient: shutdown 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 -- 172.21.15.67:0/3466736289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 msgr2=0x7f8b80136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 --2- 172.21.15.67:0/3466736289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80136d00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8b800a0ce0 tx=0x7f8b740090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.365 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 -- 172.21.15.67:0/3466736289 shutdown_connections 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 --2- 172.21.15.67:0/3466736289 >> v2:172.21.15.17:3300/0 conn(0x7f8b80137240 0x7f8b80139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 --2- 172.21.15.67:0/3466736289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80136d00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 -- 172.21.15.67:0/3466736289 >> 172.21.15.67:0/3466736289 conn(0x7f8b80070ff0 msgr2=0x7f8b800a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 -- 172.21.15.67:0/3466736289 shutdown_connections 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.353+0000 7f8b848d4640 1 -- 172.21.15.67:0/3466736289 wait complete. 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient: build_initial_monmap 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient: monmap: 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 Processor -- start 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 -- start start 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient: init 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 0x7f8b80141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8006e210 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8006e5e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8b80137240 0x7f8b8011db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): picked mon.noname-c con 0x7f8b80137240 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80099ee0 con 0x7f8b8006e210 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b800a2930 con 0x7f8b8006e5e0 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8011f640 con 0x7f8b80137240 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b848d4640 10 monclient(hunting): authenticate will time out at 1119.823892s 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39166/0 (socket says 172.21.15.67:39166) 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/651547149 learned_addr learned my addr 172.21.15.67:0/651547149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 10 monclient(hunting): get_auth_request con 0x7f8b8006e5e0 auth_method 0 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.354+0000 7f8b7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7e59c640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 msgr2=0x7f8b80141720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7e59c640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 0x7f8b80141720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7ed9d640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8b80137240 0x7f8b8011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7ed9d640 10 monclient(hunting): get_auth_request con 0x7f8b80137240 auth_method 0 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:07.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient(hunting): handle_auth_done global_id 6121 payload 824 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8b80137240 msgr2=0x7f8b8011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8b80137240 0x7f8b8011db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 msgr2=0x7f8b80141720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 0x7f8b80141720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 monclient: found mon.noname-b 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.355+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74009390 con 0x7f8b8006e5e0 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b7dd9b640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 msgr2=0x7f8b80134f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b7dd9b640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b74035040 tx=0x7f8b74032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74031990 con 0x7f8b8006e5e0 2024-09-15T01:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: dump: 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74031af0 con 0x7f8b8006e5e0 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 5 monclient: authenticate success, global_id 6121 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 10 monclient: _renew_subs 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b80121090 con 0x7f8b8006e5e0 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74031e70 con 0x7f8b8006e5e0 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b6e7fc640 10 monclient: dump: 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b7cd99640 4 set_mon_vals no callback set 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 10 monclient: _renew_subs 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.357+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b80141c60 con 0x7f8b8006e5e0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8b74039040 con 0x7f8b8006e5e0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b848d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b848d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b848d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b6e7fc640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 0x7f8b6003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8004bf60 con 0x7f8b8006e5e0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(170..170 src has 140..170) v4 ==== 7678+0+0 (secure 0 0 0) 0x7f8b7406c360 con 0x7f8b8006e5e0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b7e59c640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 0x7f8b6003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.358+0000 7f8b7e59c640 10 monclient: get_auth_request con 0x7f8b600386f0 auth_method 0 2024-09-15T01:58:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.361+0000 7f8b7e59c640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 0x7f8b6003aba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8b700047d0 tx=0x7f8b70007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.363+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8b74042960 con 0x7f8b8006e5e0 2024-09-15T01:58:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.363+0000 7f8b6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.363+0000 7f8b6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:07.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.536+0000 7f8b848d4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:07.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.536+0000 7f8b848d4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:07.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.536+0000 7f8b848d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:07.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:07.536+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f8b80069eb0 con 0x7f8b8006e5e0 2024-09-15T01:58:08.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:08.911+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v171) v1 ==== 78+0+0 (secure 0 0 0) 0x7f8b7403f030 con 0x7f8b8006e5e0 2024-09-15T01:58:08.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:08.911+0000 7f8b6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:08.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:08.911+0000 7f8b6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:09.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.027+0000 7f8b848d4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:09.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.027+0000 7f8b848d4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:09.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.027+0000 7f8b848d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:09.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.027+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f8b8009f980 con 0x7f8b8006e5e0 2024-09-15T01:58:09.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.914+0000 7f8b6e7fc640 1 -- 172.21.15.67:0/651547149 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v172) v1 ==== 78+0+0 (secure 0 0 0) 0x7f8b7403f020 con 0x7f8b8006e5e0 2024-09-15T01:58:09.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.914+0000 7f8b6e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:09.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.914+0000 7f8b6e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:09.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 msgr2=0x7f8b6003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:09.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 0x7f8b6003aba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8b700047d0 tx=0x7f8b70007b80 comp rx=0 tx=0).stop 2024-09-15T01:58:09.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 10 monclient: shutdown 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 msgr2=0x7f8b80134f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b74035040 tx=0x7f8b74032000 comp rx=0 tx=0).stop 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 shutdown_connections 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8b600386f0 0x7f8b6003aba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8b80137240 0x7f8b8011db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8006e5e0 0x7f8b80134f80 secure :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b74035040 tx=0x7f8b74032000 comp rx=0 tx=0).stop 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 --2- 172.21.15.67:0/651547149 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8b8006e210 0x7f8b80141720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:09.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 >> 172.21.15.67:0/651547149 conn(0x7f8b80070ff0 msgr2=0x7f8b800a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:09.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.921+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 shutdown_connections 2024-09-15T01:58:09.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:09.922+0000 7f8b848d4640 1 -- 172.21.15.67:0/651547149 wait complete. 2024-09-15T01:58:09.940 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T01:58:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 10 monclient: get_monmap_and_config 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 10 monclient: build_initial_monmap 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 10 monclient: monmap: 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:10.020236+0000 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:10.020236+0000 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:10.022 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 1 Processor -- start 2024-09-15T01:58:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 1 -- start start 2024-09-15T01:58:10.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.019+0000 7f072cc34640 10 monclient: init 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0728098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07280a0370 0x7f0728098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-c con 0x7f07280a0370 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0728098a50 0x7f07281390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-a con 0x7f0728098a50 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0728098ef0 con 0x7f0728098140 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07280a0b80 con 0x7f0728098a50 2024-09-15T01:58:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07281328f0 con 0x7f07280a0370 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072cc34640 10 monclient(hunting): authenticate will time out at 1122.489241s 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07280a0370 0x7f0728098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f0726d9d640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f0728098a50 msgr2=0x7f07281390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0728098140 0x7f07281321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47560/0 (socket says 172.21.15.67:47560) 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f072659c640 1 -- 172.21.15.67:0/1076892591 learned_addr learned my addr 172.21.15.67:0/1076892591 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f0726d9d640 1 --2- 172.21.15.67:0/1076892591 >> v2:172.21.15.17:3300/0 conn(0x7f0728098a50 0x7f07281390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): get_auth_request con 0x7f07280a0370 auth_method 0 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient(hunting): handle_auth_done global_id 6124 payload 274 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> v2:172.21.15.17:3300/0 conn(0x7f0728098a50 msgr2=0x7f07281390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1076892591 >> v2:172.21.15.17:3300/0 conn(0x7f0728098a50 0x7f07281390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> v1:172.21.15.67:6789/0 conn(0x7f0728098140 legacy=0x7f07281321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 monclient: found mon.noname-c 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.020+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280a00b0 con 0x7f07280a0370 2024-09-15T01:58:10.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 msgr2=0x7f0728098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:10.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 0x7f0728098510 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f07280717a0 tx=0x7f07100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c6085e2b86b5262 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:10.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 msgr2=0x7f0728098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 msgr2=0x7f0728098510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 0x7f0728098510 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f07280717a0 tx=0x7f07100090f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 0x7f0728098510 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f07280717a0 tx=0x7f07100090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 msgr2=0x7f0728098510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 0x7f0728098510 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient: _add_conns ranks=[2,0] 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 0x7f0714005900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient(hunting): picked mon.noname-b con 0x7f07140034c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07140061d0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071400a130 con 0x7f07140061d0 2024-09-15T01:58:10.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071400b2f0 con 0x7f07140034c0 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f0725d9b640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f071dd9b640 1 -- 172.21.15.67:0/1076892591 >> v1:172.21.15.17:6789/0 conn(0x7f07140061d0 legacy=0x7f0714008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.023+0000 7f072659c640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 0x7f0714005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): get_auth_request con 0x7f07140034c0 auth_method 0 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient(hunting): handle_auth_done global_id 6124 payload 293 2024-09-15T01:58:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 1 -- 172.21.15.67:0/1076892591 >> v1:172.21.15.17:6789/0 conn(0x7f07140061d0 legacy=0x7f0714008680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 1 monclient: found mon.noname-b 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.024+0000 7f072659c640 1 -- 172.21.15.67:0/1076892591 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071400b570 con 0x7f07140034c0 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f072659c640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 msgr2=0x7f0714005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f072659c640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 0x7f0714005900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0718005120 tx=0x7f0718007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1484f0e3f5e08bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180072e0 con 0x7f07140034c0 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: dump: 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:10.038 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718007440 con 0x7f07140034c0 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 1 -- 172.21.15.67:0/1076892591 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718005710 con 0x7f07140034c0 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f072cc34640 5 monclient: authenticate success, global_id 6124 2024-09-15T01:58:10.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f072cc34640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f072cc34640 4 set_mon_vals no callback set 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.026+0000 7f0725d9b640 10 monclient: dump: 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:10.040 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 10 monclient: shutdown 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 -- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 msgr2=0x7f0714005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07140034c0 0x7f0714005900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0718005120 tx=0x7f0718007d90 comp rx=0 tx=0).stop 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f0726d9d640 1 -- 172.21.15.67:0/1076892591 reap_dead start 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 -- 172.21.15.67:0/1076892591 shutdown_connections 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 --2- 172.21.15.67:0/1076892591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07280a0370 0x7f0728098510 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 -- 172.21.15.67:0/1076892591 >> 172.21.15.67:0/1076892591 conn(0x7f07280a5780 msgr2=0x7f07280a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 -- 172.21.15.67:0/1076892591 shutdown_connections 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.027+0000 7f072cc34640 1 -- 172.21.15.67:0/1076892591 wait complete. 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient: build_initial_monmap 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient: monmap: 2024-09-15T01:58:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 1 Processor -- start 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 1 -- start start 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient: init 2024-09-15T01:58:10.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07280a0370 0x7f0728138900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-c con 0x7f07280a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.028+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-b con 0x7f07140034c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 0x7f0728130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): picked mon.noname-a con 0x7f0728130410 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07281313e0 con 0x7f0728130410 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07281342e0 con 0x7f07140034c0 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0728133d20 con 0x7f07280a0370 2024-09-15T01:58:10.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072cc34640 10 monclient(hunting): authenticate will time out at 1122.498045s 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f0726d9d640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 msgr2=0x7f0728130880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f0726d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 0x7f0728130880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39190/0 (socket says 172.21.15.67:39190) 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 -- 172.21.15.67:0/1890517034 learned_addr learned my addr 172.21.15.67:0/1890517034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): get_auth_request con 0x7f07140034c0 auth_method 0 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07280a0370 0x7f0728138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): get_auth_request con 0x7f07280a0370 auth_method 0 2024-09-15T01:58:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f072659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient(hunting): handle_auth_done global_id 6127 payload 824 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 -- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07280a0370 msgr2=0x7f0728138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07280a0370 0x7f0728138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 -- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 msgr2=0x7f0728130880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 0x7f0728130880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 monclient: found mon.noname-b 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.029+0000 7f071dd9b640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728099090 con 0x7f07140034c0 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071dd9b640 1 -- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 msgr2=0x7f072812fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071dd9b640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0710009360 tx=0x7f07100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710033e10 con 0x7f07140034c0 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: dump: 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: _finish_auth 0 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0710031c80 con 0x7f07140034c0 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 5 monclient: authenticate success, global_id 6127 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 10 monclient: _renew_subs 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0728133a40 con 0x7f07140034c0 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071003d3f0 con 0x7f07140034c0 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f071effd640 10 monclient: dump: 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 10 monclient: _renew_subs 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07281325d0 con 0x7f07140034c0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.031+0000 7f072559a640 4 set_mon_vals no callback set 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f07100453d0 con 0x7f07140034c0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f071effd640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 0x7f070803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f072cc34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f072cc34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f072cc34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(172..172 src has 140..172) v4 ==== 7255+0+0 (secure 0 0 0) 0x7f071006c510 con 0x7f07140034c0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.032+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.033+0000 7f072659c640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 0x7f070803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:10.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.033+0000 7f072659c640 10 monclient: get_auth_request con 0x7f07080386f0 auth_method 0 2024-09-15T01:58:10.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.035+0000 7f072659c640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 0x7f070803aba0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0718004c10 tx=0x7f0718015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:10.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.037+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:10.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.037+0000 7f071effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:10.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.037+0000 7f071effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:10.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.206+0000 7f072cc34640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:10.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.206+0000 7f072cc34640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:10.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.206+0000 7f072cc34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:10.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.206+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:10.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.916+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v173) v1 ==== 82+0+0 (secure 0 0 0) 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:10.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.916+0000 7f071effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:10.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:10.916+0000 7f071effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:11.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f072cc34640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:11.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f072cc34640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:11.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f072cc34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:11.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f071effd640 1 -- 172.21.15.67:0/1890517034 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v173) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0728134420 con 0x7f07140034c0 2024-09-15T01:58:11.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f071effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:11.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.028+0000 7f071effd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T01:58:11.031 INFO:tasks.workunit.client.0.smithi067.stderr:compat weight-set already created 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 msgr2=0x7f070803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 0x7f070803aba0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0718004c10 tx=0x7f0718015000 comp rx=0 tx=0).stop 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 10 monclient: shutdown 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 msgr2=0x7f072812fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0710009360 tx=0x7f07100090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 shutdown_connections 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f07080386f0 0x7f070803aba0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0728130410 0x7f0728130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07140034c0 0x7f072812fed0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 --2- 172.21.15.67:0/1890517034 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f07280a0370 0x7f0728138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 >> 172.21.15.67:0/1890517034 conn(0x7f07280a5780 msgr2=0x7f072812dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:11.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 shutdown_connections 2024-09-15T01:58:11.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.030+0000 7f072cc34640 1 -- 172.21.15.67:0/1890517034 wait complete. 2024-09-15T01:58:11.042 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket fooo host 2024-09-15T01:58:11.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.116+0000 7f781579d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.116+0000 7f781579d640 10 monclient: build_initial_monmap 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.116+0000 7f781579d640 10 monclient: monmap: 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:11.117491+0000 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:11.117491+0000 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:11.119 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:11.120 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.116+0000 7f781579d640 1 Processor -- start 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 -- start start 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient: init 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 0x7f781006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): picked mon.noname-a con 0x7f781006e210 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:11.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 0x7f781009db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): picked mon.noname-b con 0x7f781006e5e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:11.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f781009e050 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:11.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781006aa60 con 0x7f781009e050 2024-09-15T01:58:11.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:11.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781006a190 con 0x7f781006e210 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7810069bd0 con 0x7f781006e5e0 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f781579d640 10 monclient(hunting): authenticate will time out at 1123.586645s 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780effd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 msgr2=0x7f781006b9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 0x7f781006b9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f781009e050 0x7f7810138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49702/0 (socket says 172.21.15.67:49702) 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780f7fe640 1 -- 172.21.15.67:0/1087746725 learned_addr learned my addr 172.21.15.67:0/1087746725 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780e7fc640 1 --2- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 0x7f781009db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780e7fc640 10 monclient(hunting): get_auth_request con 0x7f781006e5e0 auth_method 0 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:11.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.117+0000 7f780e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient(hunting): handle_auth_done global_id 94223 payload 274 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 1 -- 172.21.15.67:0/1087746725 >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 msgr2=0x7f781006b9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 1 --2- 172.21.15.67:0/1087746725 >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 0x7f781006b9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 1 -- 172.21.15.67:0/1087746725 >> v1:172.21.15.67:6789/0 conn(0x7f781009e050 legacy=0x7f7810138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:11.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:58:11.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 1 monclient: found mon.noname-b 2024-09-15T01:58:11.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:11.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:11.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.118+0000 7f780e7fc640 1 -- 172.21.15.67:0/1087746725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78100698f0 con 0x7f781006e5e0 2024-09-15T01:58:11.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780e7fc640 1 -- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 msgr2=0x7f781009db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780e7fc640 1 --2- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 0x7f781009db10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7804009850 tx=0x7f78040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd078519fd6890e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 1 -- 172.21.15.67:0/1087746725 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780402ae00 con 0x7f781006e5e0 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: dump: 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 5 monclient: authenticate success, global_id 94223 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 1 -- 172.21.15.67:0/1087746725 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780402f4b0 con 0x7f781006e5e0 2024-09-15T01:58:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 4 set_mon_vals no callback set 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 1 -- 172.21.15.67:0/1087746725 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780402f850 con 0x7f781006e5e0 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f780dffb640 10 monclient: dump: 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:11.134 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 10 monclient: shutdown 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 -- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 msgr2=0x7f781009db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 --2- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 0x7f781009db10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7804009850 tx=0x7f78040090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 -- 172.21.15.67:0/1087746725 shutdown_connections 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 --2- 172.21.15.67:0/1087746725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781006e5e0 0x7f781009db10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 --2- 172.21.15.67:0/1087746725 >> v2:172.21.15.17:3300/0 conn(0x7f781006e210 0x7f781006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 -- 172.21.15.67:0/1087746725 >> 172.21.15.67:0/1087746725 conn(0x7f7810070ff0 msgr2=0x7f78100a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 -- 172.21.15.67:0/1087746725 shutdown_connections 2024-09-15T01:58:11.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.121+0000 7f781579d640 1 -- 172.21.15.67:0/1087746725 wait complete. 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.122+0000 7f781579d640 10 monclient: build_initial_monmap 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.122+0000 7f781579d640 10 monclient: monmap: 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.136 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.122+0000 7f781579d640 1 Processor -- start 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.122+0000 7f781579d640 1 -- start start 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.122+0000 7f781579d640 10 monclient: init 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:11.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 0x7f78101417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): picked mon.noname-a con 0x7f781006e210 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): picked mon.noname-b con 0x7f781006e5e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781009e050 0x7f781009b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): picked mon.noname-c con 0x7f781009e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:11.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810099ee0 con 0x7f781006e210 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100a2c10 con 0x7f781006e5e0 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781009d440 con 0x7f781009e050 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f781579d640 10 monclient(hunting): authenticate will time out at 1123.592305s 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780effd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 msgr2=0x7f78101417d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 0x7f78101417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55738/0 (socket says 172.21.15.67:55738) 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 -- 172.21.15.67:0/2627753818 learned_addr learned my addr 172.21.15.67:0/2627753818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:11.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): get_auth_request con 0x7f781006e5e0 auth_method 0 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781009e050 0x7f781009b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): get_auth_request con 0x7f781009e050 auth_method 0 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:11.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient(hunting): handle_auth_done global_id 6130 payload 824 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 -- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781009e050 msgr2=0x7f781009b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781009e050 0x7f781009b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 -- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 msgr2=0x7f78101417d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 0x7f78101417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 monclient: found mon.noname-b 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.123+0000 7f780e7fc640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804009390 con 0x7f781006e5e0 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.125+0000 7f780e7fc640 1 -- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 msgr2=0x7f78101322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:11.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.125+0000 7f780e7fc640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7804009360 tx=0x7f7804032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804032ab0 con 0x7f781006e5e0 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: dump: 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:11.142 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804032c10 con 0x7f781006e5e0 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 5 monclient: authenticate success, global_id 6130 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 10 monclient: _renew_subs 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781011f3d0 con 0x7f781006e5e0 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804044410 con 0x7f781006e5e0 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:11.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f77f6ffd640 10 monclient: dump: 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 10 monclient: _renew_subs 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781011e470 con 0x7f781006e5e0 2024-09-15T01:58:11.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.126+0000 7f780d7fa640 4 set_mon_vals no callback set 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7804039040 con 0x7f781006e5e0 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f781579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f781579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f781579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781011f8d0 con 0x7f781006e5e0 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f77f6ffd640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 0x7f77fc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 148..173) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f780406c500 con 0x7f781006e5e0 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f780effd640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 0x7f77fc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:11.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.127+0000 7f780effd640 10 monclient: get_auth_request con 0x7f77fc0386f0 auth_method 0 2024-09-15T01:58:11.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.129+0000 7f780effd640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 0x7f77fc03aba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7800006fd0 tx=0x7f7800006d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:11.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.131+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f780403ad50 con 0x7f781006e5e0 2024-09-15T01:58:11.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.131+0000 7f77f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:11.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.131+0000 7f77f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.299+0000 7f781579d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.299+0000 7f781579d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.299+0000 7f781579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:11.299+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f781004bf60 con 0x7f781006e5e0 2024-09-15T01:58:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.103+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v174) v1 ==== 140+0+0 (secure 0 0 0) 0x7f78040370a0 con 0x7f781006e5e0 2024-09-15T01:58:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.103+0000 7f77f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.103+0000 7f77f6ffd640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-15T01:58:12.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f781579d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f781579d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f781579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f781004bf60 con 0x7f781006e5e0 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f77f6ffd640 1 -- 172.21.15.67:0/2627753818 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v174) v1 ==== 128+0+0 (secure 0 0 0) 0x7f780403f030 con 0x7f781006e5e0 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f77f6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.209+0000 7f77f6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-15T01:58:12.212 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'fooo' already exists 2024-09-15T01:58:12.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 msgr2=0x7f77fc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:12.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 0x7f77fc03aba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7800006fd0 tx=0x7f7800006d70 comp rx=0 tx=0).stop 2024-09-15T01:58:12.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 10 monclient: shutdown 2024-09-15T01:58:12.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 msgr2=0x7f78101322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7804009360 tx=0x7f7804032000 comp rx=0 tx=0).stop 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 shutdown_connections 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f77fc0386f0 0x7f77fc03aba0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f781009e050 0x7f781009b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781006e5e0 0x7f78101322b0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 --2- 172.21.15.67:0/2627753818 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f781006e210 0x7f78101417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 >> 172.21.15.67:0/2627753818 conn(0x7f7810070ff0 msgr2=0x7f78100a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 shutdown_connections 2024-09-15T01:58:12.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.212+0000 7f781579d640 1 -- 172.21.15.67:0/2627753818 wait complete. 2024-09-15T01:58:12.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush move fooo root=default 2024-09-15T01:58:12.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.300+0000 7fb548a8b640 10 monclient: get_monmap_and_config 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.300+0000 7fb548a8b640 10 monclient: build_initial_monmap 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.300+0000 7fb548a8b640 10 monclient: monmap: 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:12.301375+0000 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:12.301375+0000 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:12.303 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:12.304 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:12.304 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.300+0000 7fb548a8b640 1 Processor -- start 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 -- start start 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient: init 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 0x7fb54409aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-a con 0x7fb5440a1a00 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb54409d5a0 0x7fb54409b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-c con 0x7fb54409d5a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb544096a70 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54409cc40 con 0x7fb544096a70 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54409cf80 con 0x7fb5440a1a00 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5440a1dd0 con 0x7fb54409d5a0 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb548a8b640 10 monclient(hunting): authenticate will time out at 1124.770519s 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb54409d5a0 0x7fb54409b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb54259c640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 msgr2=0x7fb54409aae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb54259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 0x7fb54409aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb54409d5a0 0x7fb54409b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55754/0 (socket says 172.21.15.67:55754) 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb542d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb544096a70 0x7fb544096ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49708/0 (socket says 172.21.15.67:49708) 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 -- 172.21.15.67:0/3890612478 learned_addr learned my addr 172.21.15.67:0/3890612478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:12.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): get_auth_request con 0x7fb54409d5a0 auth_method 0 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient(hunting): handle_auth_done global_id 6133 payload 274 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 -- 172.21.15.67:0/3890612478 >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 msgr2=0x7fb54409aae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 --2- 172.21.15.67:0/3890612478 >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 0x7fb54409aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 -- 172.21.15.67:0/3890612478 >> v1:172.21.15.67:6789/0 conn(0x7fb544096a70 legacy=0x7fb544096ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 monclient: found mon.noname-c 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.301+0000 7fb53bfff640 1 -- 172.21.15.67:0/3890612478 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54412efa0 con 0x7fb54409d5a0 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb53bfff640 1 -- 172.21.15.67:0/3890612478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb54409d5a0 msgr2=0x7fb54409b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb53bfff640 1 --2- 172.21.15.67:0/3890612478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb54409d5a0 0x7fb54409b020 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a2080 tx=0x7fb52c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae74c2c02d6d58e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 1 -- 172.21.15.67:0/3890612478 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c032440 con 0x7fb54409d5a0 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: dump: 2024-09-15T01:58:12.311 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 1 -- 172.21.15.67:0/3890612478 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c0325a0 con 0x7fb54409d5a0 2024-09-15T01:58:12.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb548a8b640 5 monclient: authenticate success, global_id 6133 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb548a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb548a8b640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 1 -- 172.21.15.67:0/3890612478 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c032940 con 0x7fb54409d5a0 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb548a8b640 4 set_mon_vals no callback set 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.304+0000 7fb541d9b640 10 monclient: dump: 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.313 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 10 monclient: shutdown 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 -- 172.21.15.67:0/3890612478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb54409d5a0 msgr2=0x7fb54409b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 --2- 172.21.15.67:0/3890612478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb54409d5a0 0x7fb54409b020 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a2080 tx=0x7fb52c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 -- 172.21.15.67:0/3890612478 shutdown_connections 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 --2- 172.21.15.67:0/3890612478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb54409d5a0 0x7fb54409b020 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 --2- 172.21.15.67:0/3890612478 >> v2:172.21.15.17:3300/0 conn(0x7fb5440a1a00 0x7fb54409aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 -- 172.21.15.67:0/3890612478 >> 172.21.15.67:0/3890612478 conn(0x7fb544071340 msgr2=0x7fb5440a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 -- 172.21.15.67:0/3890612478 shutdown_connections 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 1 -- 172.21.15.67:0/3890612478 wait complete. 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 10 monclient: build_initial_monmap 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.305+0000 7fb548a8b640 10 monclient: monmap: 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:12.315 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 Processor -- start 2024-09-15T01:58:12.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 -- start start 2024-09-15T01:58:12.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient: init 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-b con 0x7fb544096a70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 0x7fb54412ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-c con 0x7fb54409d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 0x7fb54411ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): picked mon.noname-a con 0x7fb5440a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440985b0 con 0x7fb5440a1a00 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54412e920 con 0x7fb544096a70 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54411d410 con 0x7fb54409d5a0 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb548a8b640 10 monclient(hunting): authenticate will time out at 1124.775838s 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55756/0 (socket says 172.21.15.67:55756) 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 1 -- 172.21.15.67:0/1473108006 learned_addr learned my addr 172.21.15.67:0/1473108006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): get_auth_request con 0x7fb544096a70 auth_method 0 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb542d9d640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 msgr2=0x7fb54411ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb542d9d640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 0x7fb54411ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.306+0000 7fb54259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 10 monclient(hunting): handle_auth_done global_id 6136 payload 824 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb53bfff640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 0x7fb54412ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:12.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 msgr2=0x7fb54412ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 0x7fb54412ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 msgr2=0x7fb54411ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 0x7fb54411ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 monclient: found mon.noname-b 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb54259c640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c009450 con 0x7fb544096a70 2024-09-15T01:58:12.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb53bfff640 10 monclient: get_auth_request con 0x7fb54409d5a0 auth_method 0 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.307+0000 7fb53bfff640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 0x7fb54412ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb54259c640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 msgr2=0x7fb544141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb54259c640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fb53400b600 tx=0x7fb5340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534015070 con 0x7fb544096a70 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: dump: 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.325 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534003d90 con 0x7fb544096a70 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 5 monclient: authenticate success, global_id 6136 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 10 monclient: _renew_subs 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54411ef40 con 0x7fb544096a70 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534006380 con 0x7fb544096a70 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb53a7fc640 10 monclient: dump: 2024-09-15T01:58:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:12.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 10 monclient: _renew_subs 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb54159a640 4 set_mon_vals no callback set 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.309+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54411f2d0 con 0x7fb544096a70 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb534010440 con 0x7fb544096a70 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb548a8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb548a8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb548a8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54404bf60 con 0x7fb544096a70 2024-09-15T01:58:12.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb53a7fc640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 0x7fb52403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(174..174 src has 148..174) v4 ==== 7611+0+0 (secure 0 0 0) 0x7fb53404c170 con 0x7fb544096a70 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb53bfff640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 0x7fb52403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.310+0000 7fb53bfff640 10 monclient: get_auth_request con 0x7fb5240386f0 auth_method 0 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.313+0000 7fb53bfff640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 0x7fb52403aba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb544098030 tx=0x7fb52c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.315+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb534018ab0 con 0x7fb544096a70 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.315+0000 7fb53a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:12.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.315+0000 7fb53a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.485+0000 7fb548a8b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.485+0000 7fb548a8b640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.485+0000 7fb548a8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:12.485+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fb54411f0d0 con 0x7fb544096a70 2024-09-15T01:58:13.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.108+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v175) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb534018690 con 0x7fb544096a70 2024-09-15T01:58:13.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.108+0000 7fb53a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:13.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.108+0000 7fb53a7fc640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T01:58:13.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.217+0000 7fb548a8b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:13.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.217+0000 7fb548a8b640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:13.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.217+0000 7fb548a8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.217+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fb544141fb0 con 0x7fb544096a70 2024-09-15T01:58:13.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.218+0000 7fb53a7fc640 1 -- 172.21.15.67:0/1473108006 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v175) v1 ==== 183+0+0 (secure 0 0 0) 0x7fb534018ab0 con 0x7fb544096a70 2024-09-15T01:58:13.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.218+0000 7fb53a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T01:58:13.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.218+0000 7fb53a7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T01:58:13.221 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 msgr2=0x7fb52403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 0x7fb52403aba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb544098030 tx=0x7fb52c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 10 monclient: shutdown 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 msgr2=0x7fb544141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fb53400b600 tx=0x7fb5340091a0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 shutdown_connections 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb5240386f0 0x7fb52403aba0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb5440a1a00 0x7fb54411ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb54409d5a0 0x7fb54412ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 --2- 172.21.15.67:0/1473108006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb544096a70 0x7fb544141150 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 >> 172.21.15.67:0/1473108006 conn(0x7fb544071340 msgr2=0x7fb5440a08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:13.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 shutdown_connections 2024-09-15T01:58:13.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.220+0000 7fb548a8b640 1 -- 172.21.15.67:0/1473108006 wait complete. 2024-09-15T01:58:13.232 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket barr rack 2024-09-15T01:58:13.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.307+0000 7fa80f634640 10 monclient: get_monmap_and_config 2024-09-15T01:58:13.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.307+0000 7fa80f634640 10 monclient: build_initial_monmap 2024-09-15T01:58:13.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.307+0000 7fa80f634640 10 monclient: monmap: 2024-09-15T01:58:13.310 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:13.310 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:13.308767+0000 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:13.308767+0000 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:13.312 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:13.312 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 Processor -- start 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 -- start start 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient: init 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8080a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:13.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa80809eea0 0x7fa80809f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-c con 0x7fa80809eea0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 0x7fa808139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-a con 0x7fa808136e40 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080a61a0 con 0x7fa8080a0b50 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80809f800 con 0x7fa808136e40 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080a1610 con 0x7fa80809eea0 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.308+0000 7fa80f634640 10 monclient(hunting): authenticate will time out at 1125.777895s 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa80809eea0 0x7fa80809f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80d3d0640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa8080a0b50 0x7fa808069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49714/0 (socket says 172.21.15.67:49714) 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80d3d0640 1 -- 172.21.15.67:0/2786182622 learned_addr learned my addr 172.21.15.67:0/2786182622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80dbd1640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 msgr2=0x7fa808139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 --2- 172.21.15.67:0/2786182622 >> v2:172.21.15.67:3300/0 conn(0x7fa80809eea0 0x7fa80809f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55758/0 (socket says 172.21.15.67:55758) 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): get_auth_request con 0x7fa80809eea0 auth_method 0 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80dbd1640 1 --2- 172.21.15.67:0/2786182622 >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 0x7fa808139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:13.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient(hunting): handle_auth_done global_id 6139 payload 274 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 -- 172.21.15.67:0/2786182622 >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 msgr2=0x7fa808139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 --2- 172.21.15.67:0/2786182622 >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 0x7fa808139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 -- 172.21.15.67:0/2786182622 >> v1:172.21.15.67:6789/0 conn(0x7fa8080a0b50 legacy=0x7fa808069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient: _finish_hunting 0 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 monclient: found mon.noname-c 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.309+0000 7fa80cbcf640 1 -- 172.21.15.67:0/2786182622 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080a04c0 con 0x7fa80809eea0 2024-09-15T01:58:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80cbcf640 1 -- 172.21.15.67:0/2786182622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa80809eea0 msgr2=0x7fa80809f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80cbcf640 1 --2- 172.21.15.67:0/2786182622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa80809eea0 0x7fa80809f2c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa808066ac0 tx=0x7fa7f80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=764771c13fdea801 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 1 -- 172.21.15.67:0/2786182622 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8032440 con 0x7fa80809eea0 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: dump: 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: _finish_auth 0 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 5 monclient: authenticate success, global_id 6139 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 1 -- 172.21.15.67:0/2786182622 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f80325a0 con 0x7fa80809eea0 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:13.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 4 set_mon_vals no callback set 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 1 -- 172.21.15.67:0/2786182622 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8032940 con 0x7fa80809eea0 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa7fffff640 10 monclient: dump: 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:13.334 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 10 monclient: shutdown 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 -- 172.21.15.67:0/2786182622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa80809eea0 msgr2=0x7fa80809f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 --2- 172.21.15.67:0/2786182622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa80809eea0 0x7fa80809f2c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa808066ac0 tx=0x7fa7f80090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 -- 172.21.15.67:0/2786182622 shutdown_connections 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 --2- 172.21.15.67:0/2786182622 >> v2:172.21.15.17:3300/0 conn(0x7fa808136e40 0x7fa808139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 --2- 172.21.15.67:0/2786182622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa80809eea0 0x7fa80809f2c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 -- 172.21.15.67:0/2786182622 >> 172.21.15.67:0/2786182622 conn(0x7fa808071330 msgr2=0x7fa8080a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 -- 172.21.15.67:0/2786182622 shutdown_connections 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.312+0000 7fa80f634640 1 -- 172.21.15.67:0/2786182622 wait complete. 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.313+0000 7fa80f634640 10 monclient: build_initial_monmap 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.313+0000 7fa80f634640 10 monclient: monmap: 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:13.336 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.313+0000 7fa80f634640 1 Processor -- start 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.313+0000 7fa80f634640 1 -- start start 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.313+0000 7fa80f634640 10 monclient: init 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 0x7fa808130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-a con 0x7fa80809eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8080a0b50 0x7fa808131130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-c con 0x7fa8080a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): picked mon.noname-b con 0x7fa808136e40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:13.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808142110 con 0x7fa80809eea0 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80811cd00 con 0x7fa808136e40 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80811cfe0 con 0x7fa8080a0b50 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80f634640 10 monclient(hunting): authenticate will time out at 1125.783309s 2024-09-15T01:58:13.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80d3d0640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 msgr2=0x7fa808130bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80d3d0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 0x7fa808130bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55764/0 (socket says 172.21.15.67:55764) 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 -- 172.21.15.67:0/3272113344 learned_addr learned my addr 172.21.15.67:0/3272113344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): get_auth_request con 0x7fa808136e40 auth_method 0 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:13.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80cbcf640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8080a0b50 0x7fa808131130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80cbcf640 10 monclient(hunting): get_auth_request con 0x7fa8080a0b50 auth_method 0 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient(hunting): handle_auth_done global_id 6142 payload 824 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 -- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8080a0b50 msgr2=0x7fa808131130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8080a0b50 0x7fa808131130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 -- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 msgr2=0x7fa808130bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 0x7fa808130bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient: _finish_hunting 0 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 monclient: found mon.noname-b 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:13.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.314+0000 7fa80dbd1640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8009390 con 0x7fa808136e40 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.316+0000 7fa80dbd1640 1 -- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 msgr2=0x7fa808099870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.316+0000 7fa80dbd1640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b490 tx=0x7fa800009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80dbd1640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b490 tx=0x7fa800009060 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fa800003d40 mon_map magic: 0 v1 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800003d40 con 0x7fa808136e40 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 10 monclient: dump: 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:13.342 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800006150 con 0x7fa808136e40 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 5 monclient: authenticate success, global_id 6142 2024-09-15T01:58:13.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 10 monclient: _renew_subs 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80809b320 con 0x7fa808136e40 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa7ff7fe640 4 set_mon_vals no callback set 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 10 monclient: _renew_subs 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.317+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8081376e0 con 0x7fa808136e40 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800006510 con 0x7fa808136e40 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 10 monclient: dump: 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:13.344 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa800010570 con 0x7fa808136e40 2024-09-15T01:58:13.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80f634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80f634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80f634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80804bf60 con 0x7fa808136e40 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa7fd7fa640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 0x7fa7ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80d3d0640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 0x7fa7ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.318+0000 7fa80d3d0640 10 monclient: get_auth_request con 0x7fa7ec038740 auth_method 0 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.320+0000 7fa80d3d0640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 0x7fa7ec03abf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa804004b50 tx=0x7fa804005100 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.322+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(175..175 src has 148..175) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fa80004bc60 con 0x7fa808136e40 2024-09-15T01:58:13.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.940+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa800006770 con 0x7fa808136e40 2024-09-15T01:58:13.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.940+0000 7fa7fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:13.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:13.940+0000 7fa7fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:14.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:14.109+0000 7fa80f634640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:14.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:14.109+0000 7fa80f634640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:14.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:14.109+0000 7fa80f634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:14.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:14.109+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fa80811d1a0 con 0x7fa808136e40 2024-09-15T01:58:15.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.116+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v176) v1 ==== 140+0+0 (secure 0 0 0) 0x7fa800017030 con 0x7fa808136e40 2024-09-15T01:58:15.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.116+0000 7fa7fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:15.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.116+0000 7fa7fd7fa640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.222+0000 7fa80f634640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.222+0000 7fa80f634640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.222+0000 7fa80f634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.222+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fa8080008d0 con 0x7fa808136e40 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.223+0000 7fa7fd7fa640 1 -- 172.21.15.67:0/3272113344 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v176) v1 ==== 128+0+0 (secure 0 0 0) 0x7fa800017020 con 0x7fa808136e40 2024-09-15T01:58:15.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.223+0000 7fa7fd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T01:58:15.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.223+0000 7fa7fd7fa640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-15T01:58:15.226 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'barr' already exists 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 msgr2=0x7fa7ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 0x7fa7ec03abf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa804004b50 tx=0x7fa804005100 comp rx=0 tx=0).stop 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 10 monclient: shutdown 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 msgr2=0x7fa808099870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b490 tx=0x7fa800009060 comp rx=0 tx=0).stop 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 shutdown_connections 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa7ec038740 0x7fa7ec03abf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808136e40 0x7fa808099870 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa8080a0b50 0x7fa808131130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 --2- 172.21.15.67:0/3272113344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa80809eea0 0x7fa808130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 >> 172.21.15.67:0/3272113344 conn(0x7fa808071330 msgr2=0x7fa808131cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:15.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 shutdown_connections 2024-09-15T01:58:15.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.225+0000 7fa80f634640 1 -- 172.21.15.67:0/3272113344 wait complete. 2024-09-15T01:58:15.237 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush move barr root=default 2024-09-15T01:58:15.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.312+0000 7f3a6b11d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.312+0000 7f3a6b11d640 10 monclient: build_initial_monmap 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.312+0000 7f3a6b11d640 10 monclient: monmap: 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:15.313454+0000 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:15.313454+0000 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:15.316 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:15.316 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 Processor -- start 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 -- start start 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient: init 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a64068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a64069560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a64069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a64137150 0x7f3a64139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a64137150 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a64096c10 con 0x7f3a64068dc0 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a640a14d0 con 0x7f3a64137150 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64132960 con 0x7f3a64069190 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a6b11d640 10 monclient(hunting): authenticate will time out at 1127.782601s 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a68eb9640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f3a64068dc0 legacy=0x7f3a64132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a64137150 0x7f3a64139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a64137150 0x7f3a64139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55772/0 (socket says 172.21.15.67:55772) 2024-09-15T01:58:15.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 1 -- 172.21.15.67:0/2513877601 learned_addr learned my addr 172.21.15.67:0/2513877601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): get_auth_request con 0x7f3a64137150 auth_method 0 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a696ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a63fff640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a64069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a63fff640 10 monclient(hunting): get_auth_request con 0x7f3a64069190 auth_method 0 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.313+0000 7f3a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:15.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 10 monclient(hunting): handle_auth_done global_id 6145 payload 274 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 1 -- 172.21.15.67:0/2513877601 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 msgr2=0x7f3a64069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a64069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 1 -- 172.21.15.67:0/2513877601 >> v1:172.21.15.17:6789/0 conn(0x7f3a64068dc0 legacy=0x7f3a64132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 10 monclient: _finish_hunting 0 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 1 monclient: found mon.noname-c 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a696ba640 1 -- 172.21.15.67:0/2513877601 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6409ff20 con 0x7f3a64137150 2024-09-15T01:58:15.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.314+0000 7f3a63fff640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a64069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:58:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.316+0000 7f3a696ba640 1 -- 172.21.15.67:0/2513877601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64137150 msgr2=0x7f3a64139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.316+0000 7f3a696ba640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64137150 0x7f3a64139530 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009f20 tx=0x7f3a5c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9619d39e9afe24b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 1 -- 172.21.15.67:0/2513877601 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c007610 con 0x7f3a64137150 2024-09-15T01:58:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: dump: 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:15.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 1 -- 172.21.15.67:0/2513877601 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c02d050 con 0x7f3a64137150 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 5 monclient: authenticate success, global_id 6145 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 1 -- 172.21.15.67:0/2513877601 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0079a0 con 0x7f3a64137150 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 4 set_mon_vals no callback set 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:15.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a637fe640 10 monclient: dump: 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 10 monclient: shutdown 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2513877601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64137150 msgr2=0x7f3a64139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64137150 0x7f3a64139530 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009f20 tx=0x7f3a5c009840 comp rx=0 tx=0).stop 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2513877601 shutdown_connections 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64137150 0x7f3a64139530 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2513877601 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a64069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2513877601 >> 172.21.15.67:0/2513877601 conn(0x7f3a640711b0 msgr2=0x7f3a640a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:15.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2513877601 shutdown_connections 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.317+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2513877601 wait complete. 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 10 monclient: build_initial_monmap 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 10 monclient: monmap: 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:15.332 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 1 Processor -- start 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 1 -- start start 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 10 monclient: init 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.318+0000 7f3a6b11d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a64068dc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:15.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a640981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a64069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 0x7f3a64098710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a64137150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64131830 con 0x7f3a64137150 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6409b3b0 con 0x7f3a64068dc0 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640995a0 con 0x7f3a64069190 2024-09-15T01:58:15.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a6b11d640 10 monclient(hunting): authenticate will time out at 1127.788061s 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55778/0 (socket says 172.21.15.67:55778) 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 -- 172.21.15.67:0/2997170388 learned_addr learned my addr 172.21.15.67:0/2997170388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): get_auth_request con 0x7f3a64068dc0 auth_method 0 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a696ba640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 msgr2=0x7f3a64098710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:15.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a696ba640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 0x7f3a64098710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient(hunting): handle_auth_done global_id 6148 payload 824 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a63fff640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a640981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 msgr2=0x7f3a640981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a640981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 msgr2=0x7f3a64098710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:15.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 0x7f3a64098710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient: _finish_hunting 0 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 monclient: found mon.noname-b 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.319+0000 7f3a68eb9640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c009ae0 con 0x7f3a64068dc0 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a68eb9640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 msgr2=0x7f3a64097c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a68eb9640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b630 tx=0x7f3a580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58015070 con 0x7f3a64068dc0 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: dump: 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:15.337 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58003d90 con 0x7f3a64068dc0 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 5 monclient: authenticate success, global_id 6148 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 10 monclient: _renew_subs 2024-09-15T01:58:15.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6409b0d0 con 0x7f3a64068dc0 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58006380 con 0x7f3a64068dc0 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a60ff9640 10 monclient: dump: 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:15.339 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a62ffd640 4 set_mon_vals no callback set 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 10 monclient: _renew_subs 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.321+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6409a370 con 0x7f3a64068dc0 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a58010440 con 0x7f3a64068dc0 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a6b11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a6b11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a6b11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6404bf60 con 0x7f3a64068dc0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.322+0000 7f3a60ff9640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 0x7f3a4803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.323+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(176..176 src has 148..176) v4 ==== 7827+0+0 (secure 0 0 0) 0x7f3a5804c090 con 0x7f3a64068dc0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.323+0000 7f3a63fff640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 0x7f3a4803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.323+0000 7f3a63fff640 10 monclient: get_auth_request con 0x7f3a480386f0 auth_method 0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.325+0000 7f3a63fff640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 0x7f3a4803aba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3a50006e70 tx=0x7f3a50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.327+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3a58018910 con 0x7f3a64068dc0 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.327+0000 7f3a60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.327+0000 7f3a60ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.495+0000 7f3a6b11d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.495+0000 7f3a6b11d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.495+0000 7f3a6b11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:15.495+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f3a6409a370 con 0x7f3a64068dc0 2024-09-15T01:58:16.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.696+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v177) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3a6409a370 con 0x7f3a64068dc0 2024-09-15T01:58:16.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.696+0000 7f3a60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:16.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.696+0000 7f3a60ff9640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.803+0000 7f3a6b11d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.803+0000 7f3a6b11d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.803+0000 7f3a6b11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.803+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f3a6409a370 con 0x7f3a64068dc0 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.804+0000 7f3a60ff9640 1 -- 172.21.15.67:0/2997170388 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v177) v1 ==== 183+0+0 (secure 0 0 0) 0x7f3a6409a370 con 0x7f3a64068dc0 2024-09-15T01:58:16.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.804+0000 7f3a60ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T01:58:16.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.804+0000 7f3a60ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T01:58:16.807 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 msgr2=0x7f3a4803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 0x7f3a4803aba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3a50006e70 tx=0x7f3a50008040 comp rx=0 tx=0).stop 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 10 monclient: shutdown 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 msgr2=0x7f3a64097c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b630 tx=0x7f3a580091a0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 shutdown_connections 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a480386f0 0x7f3a4803aba0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3a64137150 0x7f3a64098710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3a64069190 0x7f3a640981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 --2- 172.21.15.67:0/2997170388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64068dc0 0x7f3a64097c90 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 >> 172.21.15.67:0/2997170388 conn(0x7f3a640711b0 msgr2=0x7f3a6409ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 shutdown_connections 2024-09-15T01:58:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.806+0000 7f3a6b11d640 1 -- 172.21.15.67:0/2997170388 wait complete. 2024-09-15T01:58:16.818 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush move fooo rack=barr 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.894+0000 7fbfdc11d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.894+0000 7fbfdc11d640 10 monclient: build_initial_monmap 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.894+0000 7fbfdc11d640 10 monclient: monmap: 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:16.895425+0000 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:16.895425+0000 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.897 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.894+0000 7fbfdc11d640 1 Processor -- start 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 -- start start 2024-09-15T01:58:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient: init 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfd4098140 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd40a0370 0x7fbfd4098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd40a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfd4098a50 0x7fbfd41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd4098a50 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4098ef0 con 0x7fbfd4098140 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40a1730 con 0x7fbfd4098a50 2024-09-15T01:58:16.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40a0b80 con 0x7fbfd40a0370 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfdc11d640 10 monclient(hunting): authenticate will time out at 1129.364542s 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfd4098a50 0x7fbfd41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd9eb9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbfd4098140 0x7fbfd41321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49722/0 (socket says 172.21.15.67:49722) 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfd4098a50 0x7fbfd41390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55784/0 (socket says 172.21.15.67:55784) 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd9eb9640 1 -- 172.21.15.67:0/4022505152 learned_addr learned my addr 172.21.15.67:0/4022505152 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request con 0x7fbfd4098a50 auth_method 0 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd96b8640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd40a0370 0x7fbfd4098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd96b8640 10 monclient(hunting): get_auth_request con 0x7fbfd40a0370 auth_method 0 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd96b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd96b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfd96b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_done global_id 6151 payload 274 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 -- 172.21.15.67:0/4022505152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd40a0370 msgr2=0x7fbfd4098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd40a0370 0x7fbfd4098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 -- 172.21.15.67:0/4022505152 >> v1:172.21.15.67:6789/0 conn(0x7fbfd4098140 legacy=0x7fbfd41321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient: _finish_hunting 0 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 monclient: found mon.noname-c 2024-09-15T01:58:16.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.895+0000 7fbfda6ba640 1 -- 172.21.15.67:0/4022505152 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd41328f0 con 0x7fbfd4098a50 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfda6ba640 1 -- 172.21.15.67:0/4022505152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd4098a50 msgr2=0x7fbfd41390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfda6ba640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd4098a50 0x7fbfd41390c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40a1660 tx=0x7fbfcc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3892ea9388820677 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 1 -- 172.21.15.67:0/4022505152 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc007760 con 0x7fbfd4098a50 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:16.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: dump: 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: _finish_auth 0 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 1 -- 172.21.15.67:0/4022505152 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc02d050 con 0x7fbfd4098a50 2024-09-15T01:58:16.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfdc11d640 5 monclient: authenticate success, global_id 6151 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfdc11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 1 -- 172.21.15.67:0/4022505152 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc007b30 con 0x7fbfd4098a50 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfdc11d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfdc11d640 4 set_mon_vals no callback set 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.898+0000 7fbfd8eb7640 10 monclient: dump: 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.907 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 10 monclient: shutdown 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 -- 172.21.15.67:0/4022505152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd4098a50 msgr2=0x7fbfd41390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd4098a50 0x7fbfd41390c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40a1660 tx=0x7fbfcc009780 comp rx=0 tx=0).stop 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 -- 172.21.15.67:0/4022505152 shutdown_connections 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd4098a50 0x7fbfd41390c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/4022505152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd40a0370 0x7fbfd4098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 -- 172.21.15.67:0/4022505152 >> 172.21.15.67:0/4022505152 conn(0x7fbfd40a5780 msgr2=0x7fbfd40a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 -- 172.21.15.67:0/4022505152 shutdown_connections 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.899+0000 7fbfdc11d640 1 -- 172.21.15.67:0/4022505152 wait complete. 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient: build_initial_monmap 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient: monmap: 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:16.909 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 Processor -- start 2024-09-15T01:58:16.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 -- start start 2024-09-15T01:58:16.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient: init 2024-09-15T01:58:16.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd4098140 0x7fbfd4132210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd4098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 0x7fbfd412f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd4098a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd40a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd409beb0 con 0x7fbfd4098a50 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd409b6f0 con 0x7fbfd40a0370 2024-09-15T01:58:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41314e0 con 0x7fbfd4098140 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfdc11d640 10 monclient(hunting): authenticate will time out at 1129.369828s 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfd96b8640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 msgr2=0x7fbfd412f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfd96b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 0x7fbfd412f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55798/0 (socket says 172.21.15.67:55798) 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 1 -- 172.21.15.67:0/1241003273 learned_addr learned my addr 172.21.15.67:0/1241003273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request con 0x7fbfd40a0370 auth_method 0 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfda6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:16.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.900+0000 7fbfd9eb9640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd4098140 0x7fbfd4132210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): get_auth_request con 0x7fbfd4098140 auth_method 0 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_done global_id 6154 payload 824 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 -- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd4098140 msgr2=0x7fbfd4132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd4098140 0x7fbfd4132210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 -- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 msgr2=0x7fbfd412f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 0x7fbfd412f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient: _finish_hunting 0 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 monclient: found mon.noname-b 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.901+0000 7fbfda6ba640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc009a20 con 0x7fbfd40a0370 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfda6ba640 1 -- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 msgr2=0x7fbfd412f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfda6ba640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0097c0 tx=0x7fbfcc037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc031c10 con 0x7fbfd40a0370 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: dump: 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:16.918 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc040070 con 0x7fbfd40a0370 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfdc11d640 5 monclient: authenticate success, global_id 6154 2024-09-15T01:58:16.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfdc11d640 10 monclient: _renew_subs 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd41317c0 con 0x7fbfd40a0370 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc045480 con 0x7fbfd40a0370 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfc9ffb640 10 monclient: dump: 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:16.920 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.904+0000 7fbfdc11d640 10 monclient: _renew_subs 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.903+0000 7fbfcbfff640 4 set_mon_vals no callback set 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.904+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.904+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd41302e0 con 0x7fbfd40a0370 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbfcc042030 con 0x7fbfd40a0370 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfdc11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd404bf60 con 0x7fbfd40a0370 2024-09-15T01:58:16.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfc9ffb640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 0x7fbfb803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(177..177 src has 148..177) v4 ==== 8015+0+0 (secure 0 0 0) 0x7fbfcc06dce0 con 0x7fbfd40a0370 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfd9eb9640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 0x7fbfb803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.905+0000 7fbfd9eb9640 10 monclient: get_auth_request con 0x7fbfb8038720 auth_method 0 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.907+0000 7fbfd9eb9640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 0x7fbfb803abd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4007fd0 tx=0x7fbfc4007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.909+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbfcc073020 con 0x7fbfd40a0370 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.909+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:16.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:16.909+0000 7fbfc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:17.079+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:17.079+0000 7fbfdc11d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:17.079+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:17.079+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fbfd4098ef0 con 0x7fbfd40a0370 2024-09-15T01:58:17.688 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T01:58:17.688 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T01:58:17.688 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T01:58:17.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T01:58:17.692 INFO:tasks.ceph.mon.a:Started 2024-09-15T01:58:17.692 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T01:58:17.692 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T01:58:17.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T01:58:19.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:19.901+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:19.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:19.901+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:19.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:19.901+0000 7fbfc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:20.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:20.901+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:20.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:20.901+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:21.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:21.901+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:21.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:21.901+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:22.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:22.901+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:22.901+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:23.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:23.901+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:23.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:23.901+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:24.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:24.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:24.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:24.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:25.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:25.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:25.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:25.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:26.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:26.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:26.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:26.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:27.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:27.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:27.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:27.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:28.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:28.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:28.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:28.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:29.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:29.903+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:29.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:29.903+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:29.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:29.903+0000 7fbfc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:30.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:30.902+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:30.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:30.902+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:31.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:31.903+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:31.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:31.903+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:32.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:32.903+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:32.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:32.903+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:33.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:33.903+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:33.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:33.903+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:34.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:34.903+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:34.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:34.903+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:35.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:35.904+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:35.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:35.904+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:36.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:36.904+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:36.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:36.904+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:37.693 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T01:58:37.693 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T01:58:37.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:58:37.694 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 10 monclient: get_monmap_and_config 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 10 monclient: build_initial_monmap 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 10 monclient: monmap: 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:37.801014+0000 2024-09-15T01:58:37.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:37.801014+0000 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:37.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:37.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 1 Processor -- start 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 1 -- start start 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.800+0000 7f1734e78640 10 monclient: init 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17300a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1730132040 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-c con 0x7f17300987b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17300a1580 con 0x7f1730132040 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17300a5c80 con 0x7f17300a58b0 2024-09-15T01:58:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1730132440 con 0x7f17300987b0 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1734e78640 10 monclient(hunting): authenticate will time out at 1177.797518s 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1727fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1730132040 0x7f1730136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48846/0 (socket says 172.21.15.17:48846) 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 learned_addr learned my addr 172.21.15.17:0/3503260016 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f172ed9d640 10 monclient(hunting): get_auth_request con 0x7f17300987b0 auth_method 0 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f172ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f172ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.801+0000 7f172ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.802+0000 7f172ed9d640 1 -- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 msgr2=0x7f173013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.802+0000 7f172ed9d640 1 -- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 msgr2=0x7f173013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:37.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.802+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:37.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.802+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:37.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.803+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 >> v1:172.21.15.17:6789/0 conn(0x7f1730132040 legacy=0x7f1730136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:37.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:37.804+0000 7f172e59c640 1 -- 172.21.15.17:0/3503260016 >> v1:172.21.15.67:6789/0 conn(0x7f17300a58b0 legacy=0x7f17300980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:37.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:37.904+0000 7fbfc8ff9640 10 monclient: tick 2024-09-15T01:58:37.906 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:37.904+0000 7fbfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 10 monclient(hunting): get_auth_request con 0x7f17300987b0 auth_method 0 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 1 -- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 msgr2=0x7f173013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 1 -- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 msgr2=0x7f173013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:38.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.003+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153135831 0 0) 0x7f17300a1580 con 0x7f1730132040 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 10 monclient(hunting): my global_id is 104097 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171c0034e0 con 0x7f1730132040 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725145506 0 0) 0x7f171c0034e0 con 0x7f1730132040 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.116+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 msgr2=0x7f173013a500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:38.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 --2- 172.21.15.17:0/3503260016 >> v2:172.21.15.67:3300/0 conn(0x7f17300987b0 0x7f173013a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 >> v1:172.21.15.67:6789/0 conn(0x7f17300a58b0 legacy=0x7f17300980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730098bb0 con 0x7f1730132040 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f1718001710 con 0x7f1730132040 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1734e78640 5 monclient: authenticate success, global_id 104097 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1734e78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: dump: 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:58:38.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 >> v1:172.21.15.17:6789/0 conn(0x7f1730132040 legacy=0x7f1730136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f171c003b20 0x7f171c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): picked mon.a con 0x7f171c003b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f171c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): picked mon.c con 0x7f171c004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171c0053b0 0x7f171c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): picked mon.b con 0x7f171c0053b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730132440 con 0x7f171c003b20 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c00c380 con 0x7f171c0053b0 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c00c540 con 0x7f171c004780 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1734e78640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f171c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): get_auth_request con 0x7f171c004780 auth_method 0 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171c0053b0 0x7f171c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): get_auth_request con 0x7f171c0053b0 auth_method 0 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient(hunting): handle_auth_done global_id 104097 payload 293 2024-09-15T01:58:38.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171c0053b0 msgr2=0x7f171c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f171c0053b0 0x7f171c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f171c003b20 msgr2=0x7f171c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f171c003b20 0x7f171c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 monclient: found mon.c 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c0034e0 con 0x7f171c004780 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f172ed9d640 1 -- 172.21.15.17:0/3503260016 reap_dead start 2024-09-15T01:58:38.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.117+0000 7f1727fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.120+0000 7f1727fff640 1 -- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 msgr2=0x7f171c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.120+0000 7f1727fff640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f171c004bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f17180018f0 tx=0x7f17300a1860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=806299d091387701 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718005890 con 0x7f171c004780 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: dump: 2024-09-15T01:58:38.133 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1718005a30 con 0x7f171c004780 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 4 set_mon_vals no callback set 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 1 -- 172.21.15.17:0/3503260016 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718005df0 con 0x7f171c004780 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f172dd9b640 10 monclient: dump: 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.135 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 10 monclient: shutdown 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 1 -- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 msgr2=0x7f171c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.121+0000 7f1734e78640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f171c004bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f17180018f0 tx=0x7f17300a1860 comp rx=0 tx=0).stop 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 1 -- 172.21.15.17:0/3503260016 shutdown_connections 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 1 --2- 172.21.15.17:0/3503260016 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f171c004bf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 1 -- 172.21.15.17:0/3503260016 >> 172.21.15.17:0/3503260016 conn(0x7f1730070ed0 msgr2=0x7f17300a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 1 -- 172.21.15.17:0/3503260016 shutdown_connections 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 1 -- 172.21.15.17:0/3503260016 wait complete. 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.122+0000 7f1734e78640 10 monclient: monmap: 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 1 Processor -- start 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 1 -- start start 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 10 monclient: init 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f173009b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-c con 0x7f171c004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-a con 0x7f1730130c60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.123+0000 7f1734e78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 0x7f173011bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): picked mon.noname-b con 0x7f173011b7d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730142100 con 0x7f1730130c60 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173011e060 con 0x7f173011b7d0 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173011cc50 con 0x7f171c004780 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1734e78640 10 monclient(hunting): authenticate will time out at 1178.120037s 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38432/0 (socket says 172.21.15.17:38432) 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 -- 172.21.15.17:0/2722203732 learned_addr learned my addr 172.21.15.17:0/2722203732 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): get_auth_request con 0x7f1730130c60 auth_method 0 2024-09-15T01:58:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f173009b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): get_auth_request con 0x7f171c004780 auth_method 0 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 0x7f173011bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): get_auth_request con 0x7f173011b7d0 auth_method 0 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient(hunting): handle_auth_done global_id 104100 payload 824 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 -- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 msgr2=0x7f173009b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f173009b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 -- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 msgr2=0x7f173011bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 0x7f173011bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 monclient: found mon.noname-a 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f1727fff640 1 -- 172.21.15.17:0/2722203732 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730098dc0 con 0x7f1730130c60 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172ed9d640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 0x7f173011bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.124+0000 7f172e59c640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f173009b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1727fff640 1 -- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 msgr2=0x7f1730131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1727fff640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f171802adf0 tx=0x7f1718034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718007510 con 0x7f1730130c60 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: dump: 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 5 monclient: authenticate success, global_id 104100 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 10 monclient: _renew_subs 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173009e910 con 0x7f1730130c60 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 10 monclient: _renew_subs 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173011dd80 con 0x7f1730130c60 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1718007880 con 0x7f1730130c60 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718005050 con 0x7f1730130c60 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f17267fc640 10 monclient: dump: 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.145 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.127+0000 7f172d59a640 4 set_mon_vals no callback set 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.128+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f17180051b0 con 0x7f1730130c60 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.128+0000 7f17267fc640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 0x7f171003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.128+0000 7f1734e78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.128+0000 7f1734e78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.129+0000 7f1734e78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.129+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(177..177 src has 148..177) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f171806cda0 con 0x7f1730130c60 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.129+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1714005010 con 0x7f1730130c60 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.129+0000 7f172e59c640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 0x7f171003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.129+0000 7f172e59c640 10 monclient: get_auth_request con 0x7f1710038770 auth_method 0 2024-09-15T01:58:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.131+0000 7f172e59c640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 0x7f171003ac20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1720007fd0 tx=0x7f1720009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.134+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f171803c5c0 con 0x7f1730130c60 2024-09-15T01:58:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.134+0000 7f17267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.134+0000 7f17267fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:38.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.176+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v178) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbfd4098ef0 con 0x7fbfd40a0370 2024-09-15T01:58:38.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.176+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:38.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.176+0000 7fbfc9ffb640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T01:58:38.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.281+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:38.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.281+0000 7fbfdc11d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:38.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.281+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.281+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fbfd4098ef0 con 0x7fbfd40a0370 2024-09-15T01:58:38.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.283+0000 7fbfc9ffb640 1 -- 172.21.15.67:0/1241003273 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v178) v1 ==== 177+0+0 (secure 0 0 0) 0x7fbfd4098ef0 con 0x7fbfd40a0370 2024-09-15T01:58:38.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.283+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T01:58:38.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.283+0000 7fbfc9ffb640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T01:58:38.285 INFO:tasks.workunit.client.0.smithi067.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T01:58:38.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 msgr2=0x7fbfb803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 0x7fbfb803abd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4007fd0 tx=0x7fbfc4007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:38.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 10 monclient: shutdown 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 msgr2=0x7fbfd412f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.284+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0097c0 tx=0x7fbfcc037000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 shutdown_connections 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfb8038720 0x7fbfb803abd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbfd4098a50 0x7fbfd412f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40a0370 0x7fbfd412f580 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 --2- 172.21.15.67:0/1241003273 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbfd4098140 0x7fbfd4132210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 >> 172.21.15.67:0/1241003273 conn(0x7fbfd40a5780 msgr2=0x7fbfd40a38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 shutdown_connections 2024-09-15T01:58:38.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.285+0000 7fbfdc11d640 1 -- 172.21.15.67:0/1241003273 wait complete. 2024-09-15T01:58:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm fooo 2024-09-15T01:58:38.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.368+0000 7f1734e78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.368+0000 7f1734e78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.368+0000 7f1734e78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.368+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1714005840 con 0x7f1730130c60 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.369+0000 7f17267fc640 1 -- 172.21.15.17:0/2722203732 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f1718030ba0 con 0x7f1730130c60 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.369+0000 7f17267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.369+0000 7f17267fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:38.371 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 msgr2=0x7f171003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 0x7f171003ac20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1720007fd0 tx=0x7f1720009040 comp rx=0 tx=0).stop 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 10 monclient: shutdown 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 msgr2=0x7f1730131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f171802adf0 tx=0x7f1718034000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 shutdown_connections 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1710038770 0x7f171003ac20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1730130c60 0x7f1730131090 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173011b7d0 0x7f173011bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 --2- 172.21.15.17:0/2722203732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f171c004780 0x7f173009b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 >> 172.21.15.17:0/2722203732 conn(0x7f1730070ed0 msgr2=0x7f1730096e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 shutdown_connections 2024-09-15T01:58:38.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.371+0000 7f1734e78640 1 -- 172.21.15.17:0/2722203732 wait complete. 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.370+0000 7f1b5626f640 10 monclient: get_monmap_and_config 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.370+0000 7f1b5626f640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.370+0000 7f1b5626f640 10 monclient: monmap: 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:38.372079+0000 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:38.372079+0000 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.377 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.371+0000 7f1b5626f640 1 Processor -- start 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.371+0000 7f1b5626f640 1 -- start start 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.371+0000 7f1b5626f640 10 monclient: init 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b500a3830 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b5006bef0 0x7f1b5006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b5006bef0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b50136d00 0x7f1b501390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b50136d00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500a3c00 con 0x7f1b500a3830 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b500696a0 con 0x7f1b5006bef0 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50069940 con 0x7f1b50136d00 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5626f640 10 monclient(hunting): authenticate will time out at 1150.841372s 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b500a3830 0x7f1b50099aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60574/0 (socket says 172.21.15.67:60574) 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4ffff640 1 -- 172.21.15.67:0/2000406860 learned_addr learned my addr 172.21.15.67:0/2000406860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 --2- 172.21.15.67:0/2000406860 >> v2:172.21.15.67:3300/0 conn(0x7f1b5006bef0 0x7f1b5006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request con 0x7f1b5006bef0 auth_method 0 2024-09-15T01:58:38.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b50136d00 0x7f1b501390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): get_auth_request con 0x7f1b50136d00 auth_method 0 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_done global_id 6160 payload 274 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 -- 172.21.15.67:0/2000406860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b50136d00 msgr2=0x7f1b501390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b50136d00 0x7f1b501390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 -- 172.21.15.67:0/2000406860 >> v1:172.21.15.67:6789/0 conn(0x7f1b500a3830 legacy=0x7f1b50099aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 monclient: found mon.noname-c 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.372+0000 7f1b4f7fe640 1 -- 172.21.15.67:0/2000406860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50069c20 con 0x7f1b5006bef0 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4f7fe640 1 -- 172.21.15.67:0/2000406860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5006bef0 msgr2=0x7f1b5006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4f7fe640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5006bef0 0x7f1b5006a600 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009850 tx=0x7f1b440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb75cdb7f8567342 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 1 -- 172.21.15.67:0/2000406860 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4402adf0 con 0x7f1b5006bef0 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: dump: 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.388 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b5626f640 5 monclient: authenticate success, global_id 6160 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b5626f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 1 -- 172.21.15.67:0/2000406860 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44032440 con 0x7f1b5006bef0 2024-09-15T01:58:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b5626f640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b5626f640 4 set_mon_vals no callback set 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 1 -- 172.21.15.67:0/2000406860 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440327e0 con 0x7f1b5006bef0 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.375+0000 7f1b4effd640 10 monclient: dump: 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.390 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 10 monclient: shutdown 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 -- 172.21.15.67:0/2000406860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5006bef0 msgr2=0x7f1b5006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5006bef0 0x7f1b5006a600 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009850 tx=0x7f1b440090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 -- 172.21.15.67:0/2000406860 shutdown_connections 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b50136d00 0x7f1b501390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 --2- 172.21.15.67:0/2000406860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5006bef0 0x7f1b5006a600 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 -- 172.21.15.67:0/2000406860 >> 172.21.15.67:0/2000406860 conn(0x7f1b50070fb0 msgr2=0x7f1b500a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 -- 172.21.15.67:0/2000406860 shutdown_connections 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 1 -- 172.21.15.67:0/2000406860 wait complete. 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.376+0000 7f1b5626f640 10 monclient: monmap: 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:38.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 Processor -- start 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 -- start start 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient: init 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b5006bef0 0x7f1b5009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-a con 0x7f1b5006bef0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b500a3830 0x7f1b5009c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b500a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b50136d00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5006ea50 con 0x7f1b5006bef0 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500a33f0 con 0x7f1b50136d00 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5011fc50 con 0x7f1b500a3830 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.377+0000 7f1b5626f640 10 monclient(hunting): authenticate will time out at 1150.846969s 2024-09-15T01:58:38.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51166/0 (socket says 172.21.15.67:51166) 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 -- 172.21.15.67:0/940217288 learned_addr learned my addr 172.21.15.67:0/940217288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): get_auth_request con 0x7f1b50136d00 auth_method 0 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b500a3830 0x7f1b5009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b5006bef0 0x7f1b5009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request con 0x7f1b500a3830 auth_method 0 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request con 0x7f1b5006bef0 auth_method 0 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient(hunting): handle_auth_done global_id 6163 payload 824 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 -- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b500a3830 msgr2=0x7f1b5009c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b500a3830 0x7f1b5009c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 -- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b5006bef0 msgr2=0x7f1b5009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b5006bef0 0x7f1b5009bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 monclient: found mon.noname-b 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.378+0000 7f1b5480c640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44009390 con 0x7f1b50136d00 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5480c640 1 -- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 msgr2=0x7f1b5009c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5480c640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009ef0 tx=0x7f1b4800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800ce10 con 0x7f1b50136d00 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: dump: 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.398 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4800f040 con 0x7f1b50136d00 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 5 monclient: authenticate success, global_id 6163 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480148e0 con 0x7f1b50136d00 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 10 monclient: _renew_subs 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5011eb80 con 0x7f1b50136d00 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b37fff640 10 monclient: dump: 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b4e7fc640 4 set_mon_vals no callback set 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 10 monclient: _renew_subs 2024-09-15T01:58:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.380+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b50136a00 con 0x7f1b50136d00 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.381+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1b48014c90 con 0x7f1b50136d00 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.381+0000 7f1b5626f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.381+0000 7f1b5626f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.381+0000 7f1b5626f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.381+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5004bf60 con 0x7f1b50136d00 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.382+0000 7f1b37fff640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 0x7f1b3c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.382+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 148..178) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f1b4804d540 con 0x7f1b50136d00 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.382+0000 7f1b4ffff640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 0x7f1b3c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.382+0000 7f1b4ffff640 10 monclient: get_auth_request con 0x7f1b3c038740 auth_method 0 2024-09-15T01:58:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.384+0000 7f1b4ffff640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 0x7f1b3c03abf0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1b40007fd0 tx=0x7f1b40007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.386+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1b48012b30 con 0x7f1b50136d00 2024-09-15T01:58:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.386+0000 7f1b37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.386+0000 7f1b37fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:38.403 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:58:38.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:58:38.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:58:38.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.498+0000 7f315c95d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.498+0000 7f315c95d640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.498+0000 7f315c95d640 10 monclient: monmap: 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:38.499658+0000 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:38.499658+0000 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:38.501 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:38.502 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:38.502 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.499+0000 7f315c95d640 1 Processor -- start 2024-09-15T01:58:38.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.499+0000 7f315c95d640 1 -- start start 2024-09-15T01:58:38.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.499+0000 7f315c95d640 10 monclient: init 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3158068f20 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-b con 0x7f31580692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3158136fe0 0x7f31581393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-c con 0x7f3158136fe0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31580a1950 con 0x7f3158068f20 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31580a1670 con 0x7f3158136fe0 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31581328d0 con 0x7f31580692f0 2024-09-15T01:58:38.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f315c95d640 10 monclient(hunting): authenticate will time out at 1178.496112s 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33802/0 (socket says 172.21.15.17:33802) 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 -- 172.21.15.17:0/688385485 learned_addr learned my addr 172.21.15.17:0/688385485 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): get_auth_request con 0x7f31580692f0 auth_method 0 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3156d9d640 1 --2- 172.21.15.17:0/688385485 >> v2:172.21.15.67:3300/0 conn(0x7f3158136fe0 0x7f31581393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3156d9d640 10 monclient(hunting): get_auth_request con 0x7f3158136fe0 auth_method 0 2024-09-15T01:58:38.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3156d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3156d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3156d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient(hunting): handle_auth_done global_id 94250 payload 274 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 -- 172.21.15.17:0/688385485 >> v2:172.21.15.67:3300/0 conn(0x7f3158136fe0 msgr2=0x7f31581393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 --2- 172.21.15.17:0/688385485 >> v2:172.21.15.67:3300/0 conn(0x7f3158136fe0 0x7f31581393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 -- 172.21.15.17:0/688385485 >> v1:172.21.15.67:6789/0 conn(0x7f3158068f20 legacy=0x7f31581321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 monclient: found mon.noname-b 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.500+0000 7f3155d9b640 1 -- 172.21.15.17:0/688385485 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31580a0020 con 0x7f31580692f0 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f3155d9b640 1 -- 172.21.15.17:0/688385485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 msgr2=0x7f31580696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f3155d9b640 1 --2- 172.21.15.17:0/688385485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f314c009910 tx=0x7f314c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a3adfc1d58667a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 1 -- 172.21.15.17:0/688385485 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c032440 con 0x7f31580692f0 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: dump: 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.515 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315c95d640 5 monclient: authenticate success, global_id 94250 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 1 -- 172.21.15.17:0/688385485 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c0325a0 con 0x7f31580692f0 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315c95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315c95d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315c95d640 4 set_mon_vals no callback set 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 1 -- 172.21.15.17:0/688385485 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c032940 con 0x7f31580692f0 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.503+0000 7f315559a640 10 monclient: dump: 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.517 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 10 monclient: shutdown 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 -- 172.21.15.17:0/688385485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 msgr2=0x7f31580696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 --2- 172.21.15.17:0/688385485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f314c009910 tx=0x7f314c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 -- 172.21.15.17:0/688385485 shutdown_connections 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 --2- 172.21.15.17:0/688385485 >> v2:172.21.15.67:3300/0 conn(0x7f3158136fe0 0x7f31581393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 --2- 172.21.15.17:0/688385485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31580692f0 0x7f31580696c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 -- 172.21.15.17:0/688385485 >> 172.21.15.17:0/688385485 conn(0x7f3158070fb0 msgr2=0x7f31580a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 -- 172.21.15.17:0/688385485 shutdown_connections 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 -- 172.21.15.17:0/688385485 wait complete. 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 10 monclient: monmap: 2024-09-15T01:58:38.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.504+0000 7f315c95d640 1 Processor -- start 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 -- start start 2024-09-15T01:58:38.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient: init 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 0x7f3158141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-a con 0x7f3158068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31580692f0 0x7f3158141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-b con 0x7f31580692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3158136fe0 0x7f3158099bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): picked mon.noname-c con 0x7f3158136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315809c9b0 con 0x7f3158068f20 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315811cd20 con 0x7f31580692f0 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31580a0750 con 0x7f3158136fe0 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315c95d640 10 monclient(hunting): authenticate will time out at 1178.501558s 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3158136fe0 0x7f3158099bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 0x7f3158141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 0x7f3158141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38448/0 (socket says 172.21.15.17:38448) 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 1 -- 172.21.15.17:0/1356685665 learned_addr learned my addr 172.21.15.17:0/1356685665 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): get_auth_request con 0x7f3158068f20 auth_method 0 2024-09-15T01:58:38.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): get_auth_request con 0x7f3158136fe0 auth_method 0 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31580692f0 0x7f3158141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): get_auth_request con 0x7f31580692f0 auth_method 0 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3156d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 10 monclient(hunting): handle_auth_done global_id 104106 payload 824 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3158136fe0 msgr2=0x7f3158099bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.505+0000 7f315659c640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3158136fe0 0x7f3158099bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31580692f0 msgr2=0x7f3158141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31580692f0 0x7f3158141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 1 monclient: found mon.noname-a 2024-09-15T01:58:38.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.506+0000 7f315659c640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c009450 con 0x7f3158068f20 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315659c640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 msgr2=0x7f3158141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315659c640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 0x7f3158141680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f314800c5e0 tx=0x7f3148007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148012840 con 0x7f3158068f20 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: dump: 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.524 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31480129c0 con 0x7f3158068f20 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 5 monclient: authenticate success, global_id 104106 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148012d80 con 0x7f3158068f20 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 10 monclient: _renew_subs 2024-09-15T01:58:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315811d250 con 0x7f3158068f20 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f3154d99640 4 set_mon_vals no callback set 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f31467fc640 10 monclient: dump: 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.526 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 10 monclient: _renew_subs 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.508+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315811c9b0 con 0x7f3158068f20 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3148011590 con 0x7f3158068f20 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f315c95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f315c95d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 0x7f315809ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3158063b90 con 0x7f315809a8b0 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 0x7f315809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient: get_auth_request con 0x7f315809a8b0 auth_method 0 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f31467fc640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3138038770 0x7f313803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(178..178 src has 148..178) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f314804c8f0 con 0x7f3158068f20 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3156d9d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3138038770 0x7f313803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3156d9d640 10 monclient: get_auth_request con 0x7f3138038770 auth_method 0 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 10 monclient(hunting): handle_auth_done global_id 104109 payload 824 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.509+0000 7f3155d9b640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f315811ba80 con 0x7f315809a8b0 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.511+0000 7f3155d9b640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 msgr2=0x7f315809ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.511+0000 7f3155d9b640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 0x7f315809ac80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31580687a0 tx=0x7f314c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c031c00 con 0x7f315809a8b0 2024-09-15T01:58:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 10 monclient: dump: 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f3156d9d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3138038770 0x7f313803ac20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3150007640 tx=0x7f3150009850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f314c031d60 con 0x7f315809a8b0 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 msgr2=0x7f315809ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 0x7f315809ac80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31580687a0 tx=0x7f314c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 msgr2=0x7f315809ac80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.512+0000 7f31467fc640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f315809a8b0 0x7f315809ac80 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31580687a0 tx=0x7f314c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f315c95d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f315c95d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 0x7f3158029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f315c95d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31580724a0 con 0x7f31580298c0 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 0x7f3158029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient: get_auth_request con 0x7f31580298c0 auth_method 0 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 10 monclient(hunting): handle_auth_done global_id 104112 payload 824 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.518+0000 7f3155d9b640 1 -- 172.21.15.17:0/1356685665 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3158136180 con 0x7f31580298c0 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.520+0000 7f3155d9b640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 msgr2=0x7f3158029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.520+0000 7f3155d9b640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 0x7f3158029cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3158063930 tx=0x7f314c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c031ed0 con 0x7f31580298c0 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 10 monclient: dump: 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.533 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7f314c047e90 con 0x7f31580298c0 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 msgr2=0x7f3158029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 0x7f3158029cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3158063930 tx=0x7f314c03e000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 msgr2=0x7f3158029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f31467fc640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f31580298c0 0x7f3158029cc0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3158063930 tx=0x7f314c03e000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.535 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.536 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:38.538 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.539 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:38.542 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.543 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.544 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:38.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:38.546 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:38.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3138038770 msgr2=0x7f313803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3138038770 0x7f313803ac20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3150007640 tx=0x7f3150009850 comp rx=0 tx=0).stop 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 10 monclient: shutdown 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 msgr2=0x7f3158141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 --2- 172.21.15.17:0/1356685665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3158068f20 0x7f3158141680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f314800c5e0 tx=0x7f3148007ed0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f3156d9d640 1 -- 172.21.15.17:0/1356685665 reap_dead start 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f3156d9d640 1 -- 172.21.15.17:0/1356685665 reap_dead start 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 shutdown_connections 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 >> 172.21.15.17:0/1356685665 conn(0x7f3158070fb0 msgr2=0x7f315809e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 shutdown_connections 2024-09-15T01:58:38.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.521+0000 7f315c95d640 1 -- 172.21.15.17:0/1356685665 wait complete. 2024-09-15T01:58:38.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:38.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:58:38.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:58:38.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.552+0000 7f1b5626f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:38.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.552+0000 7f1b5626f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:38.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.552+0000 7f1b5626f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:38.555 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:38.552+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f1b5011eb80 con 0x7f1b50136d00 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 10 monclient: monmap: 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:38.645017+0000 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:38.645017+0000 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.646 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:38.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 1 Processor -- start 2024-09-15T01:58:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 1 -- start start 2024-09-15T01:58:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.644+0000 7f110bf1d640 10 monclient: init 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1104067090 0x7f11041321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1104067090 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1104067460 0x7f1104067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1104067460 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f110412e6a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110409cbc0 con 0x7f1104067090 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110409e550 con 0x7f1104067460 2024-09-15T01:58:38.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110409e1e0 con 0x7f110412e6a0 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110bf1d640 10 monclient(hunting): authenticate will time out at 1178.641493s 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1104067090 0x7f11041321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1104067090 0x7f11041321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38478/0 (socket says 172.21.15.17:38478) 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33824/0 (socket says 172.21.15.17:33824) 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 -- 172.21.15.17:0/833046193 learned_addr learned my addr 172.21.15.17:0/833046193 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): get_auth_request con 0x7f1104067090 auth_method 0 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): get_auth_request con 0x7f110412e6a0 auth_method 0 2024-09-15T01:58:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 1 --2- 172.21.15.17:0/833046193 >> v2:172.21.15.67:3300/0 conn(0x7f1104067460 0x7f1104067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): get_auth_request con 0x7f1104067460 auth_method 0 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient(hunting): handle_auth_done global_id 104115 payload 274 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 -- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 msgr2=0x7f110412eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 -- 172.21.15.17:0/833046193 >> v2:172.21.15.67:3300/0 conn(0x7f1104067460 msgr2=0x7f1104067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 --2- 172.21.15.17:0/833046193 >> v2:172.21.15.67:3300/0 conn(0x7f1104067460 0x7f1104067830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 monclient: found mon.noname-a 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f1109cb9640 1 -- 172.21.15.17:0/833046193 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110409cfa0 con 0x7f1104067090 2024-09-15T01:58:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.645+0000 7f110a4ba640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1109cb9640 1 -- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1104067090 msgr2=0x7f11041321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1109cb9640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1104067090 0x7f11041321a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f110409c510 tx=0x7f10f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ade8a52c536043 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 1 -- 172.21.15.17:0/833046193 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4007610 con 0x7f1104067090 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: dump: 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 1 -- 172.21.15.17:0/833046193 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f402d050 con 0x7f1104067090 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f110bf1d640 5 monclient: authenticate success, global_id 104115 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f110bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f110bf1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 1 -- 172.21.15.17:0/833046193 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40079a0 con 0x7f1104067090 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f110bf1d640 4 set_mon_vals no callback set 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.648+0000 7f1108cb7640 10 monclient: dump: 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 10 monclient: shutdown 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 -- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1104067090 msgr2=0x7f11041321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1104067090 0x7f11041321a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f110409c510 tx=0x7f10f4009840 comp rx=0 tx=0).stop 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 -- 172.21.15.17:0/833046193 shutdown_connections 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 --2- 172.21.15.17:0/833046193 >> v2:172.21.15.67:3300/0 conn(0x7f1104067460 0x7f1104067830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412e6a0 0x7f110412eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 --2- 172.21.15.17:0/833046193 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1104067090 0x7f11041321a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 -- 172.21.15.17:0/833046193 >> 172.21.15.17:0/833046193 conn(0x7f11040a1ee0 msgr2=0x7f11040a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 -- 172.21.15.17:0/833046193 shutdown_connections 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.649+0000 7f110bf1d640 1 -- 172.21.15.17:0/833046193 wait complete. 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient: monmap: 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 Processor -- start 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 -- start start 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient: init 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1104067090 0x7f110409b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1104067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 0x7f110412da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1104067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f110412e6a0 0x7f1104118540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f110412e6a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110409d5a0 con 0x7f110412e6a0 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1104098ae0 con 0x7f1104067090 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1104119c50 con 0x7f1104067460 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110bf1d640 10 monclient(hunting): authenticate will time out at 1178.646848s 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 0x7f110412da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110a4ba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f110412e6a0 0x7f1104118540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 0x7f110412da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33826/0 (socket says 172.21.15.17:33826) 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 1 -- 172.21.15.17:0/1714325179 learned_addr learned my addr 172.21.15.17:0/1714325179 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 10 monclient(hunting): get_auth_request con 0x7f1104067460 auth_method 0 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f11094b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110a4ba640 10 monclient(hunting): get_auth_request con 0x7f110412e6a0 auth_method 0 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.650+0000 7f110a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1104067090 0x7f110409b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): get_auth_request con 0x7f1104067090 auth_method 0 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f110a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient(hunting): handle_auth_done global_id 94259 payload 824 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1104067090 msgr2=0x7f110409b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1104067090 0x7f110409b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f110412e6a0 msgr2=0x7f1104118540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f110412e6a0 0x7f1104118540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 monclient: found mon.noname-c 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f11094b8640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc009390 con 0x7f1104067460 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.651+0000 7f110a4ba640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f110412e6a0 0x7f1104118540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f11094b8640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 msgr2=0x7f110412da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f11094b8640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 0x7f110412da90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f10f800ea20 tx=0x7f10f80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8009600 con 0x7f1104067460 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: dump: 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f800cb50 con 0x7f1104067460 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 5 monclient: authenticate success, global_id 94259 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8013470 con 0x7f1104067460 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 10 monclient: _renew_subs 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f4009ae0 con 0x7f1104067460 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1103fff640 4 set_mon_vals no callback set 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f1101ffb640 10 monclient: dump: 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.672 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 10 monclient: _renew_subs 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.653+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110411ac60 con 0x7f1104067460 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f10f8013860 con 0x7f1104067460 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f110bf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f110bf1d640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 0x7f11040298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1104063870 con 0x7f110412dfd0 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 0x7f11040298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient: get_auth_request con 0x7f110412dfd0 auth_method 0 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.654+0000 7f1101ffb640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10f0038740 0x7f10f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(178..178 src has 148..178) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f10f804ca40 con 0x7f1104067460 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f1109cb9640 10 monclient(hunting): handle_auth_done global_id 94262 payload 824 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f1109cb9640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1104141f40 con 0x7f110412dfd0 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f110a4ba640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10f0038740 0x7f10f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.655+0000 7f110a4ba640 10 monclient: get_auth_request con 0x7f10f0038740 auth_method 0 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1109cb9640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 msgr2=0x7f11040298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1109cb9640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 0x7f11040298c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f10f4033000 tx=0x7f10f4036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4031610 con 0x7f110412dfd0 2024-09-15T01:58:38.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 10 monclient: dump: 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f110a4ba640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10f0038740 0x7f10f003abf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f10fc011580 tx=0x7f10fc0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f10f4040070 con 0x7f110412dfd0 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 msgr2=0x7f11040298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 0x7f11040298c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f10f4033000 tx=0x7f10f4036000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 msgr2=0x7f11040298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.657+0000 7f1101ffb640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f110412dfd0 0x7f11040298c0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f10f4033000 tx=0x7f10f4036000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f110bf1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f110bf1d640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 0x7f110402a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f110bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110412e3a0 con 0x7f1104029e00 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 0x7f110402a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient: get_auth_request con 0x7f1104029e00 auth_method 0 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.663+0000 7f1109cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.664+0000 7f1109cb9640 10 monclient(hunting): handle_auth_done global_id 94265 payload 824 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.664+0000 7f1109cb9640 1 -- 172.21.15.17:0/1714325179 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f11040724a0 con 0x7f1104029e00 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1109cb9640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 msgr2=0x7f110402a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1109cb9640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 0x7f110402a200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009880 tx=0x7f10f403e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4052450 con 0x7f1104029e00 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 10 monclient: dump: 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f10f403c3f0 con 0x7f1104029e00 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 msgr2=0x7f110402a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 0x7f110402a200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009880 tx=0x7f10f403e070 comp rx=0 tx=0).stop 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 msgr2=0x7f110402a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f1101ffb640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104029e00 0x7f110402a200 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009880 tx=0x7f10f403e070 comp rx=0 tx=0).stop 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:38.681 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.682 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.683 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:38.684 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.685 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:38.688 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.690 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.691 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:58:38.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.694 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:38.694 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.666+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10f0038740 msgr2=0x7f10f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f10f0038740 0x7f10f003abf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f10fc011580 tx=0x7f10fc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 10 monclient: shutdown 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 msgr2=0x7f110412da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 --2- 172.21.15.17:0/1714325179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1104067460 0x7f110412da90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f10f800ea20 tx=0x7f10f80090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110a4ba640 1 -- 172.21.15.17:0/1714325179 reap_dead start 2024-09-15T01:58:38.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110a4ba640 1 -- 172.21.15.17:0/1714325179 reap_dead start 2024-09-15T01:58:38.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 shutdown_connections 2024-09-15T01:58:38.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 >> 172.21.15.17:0/1714325179 conn(0x7f11040a1ee0 msgr2=0x7f11040978b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 shutdown_connections 2024-09-15T01:58:38.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.667+0000 7f110bf1d640 1 -- 172.21.15.17:0/1714325179 wait complete. 2024-09-15T01:58:38.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:38.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:58:38.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.791+0000 7f4230ed4640 10 monclient: get_monmap_and_config 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.791+0000 7f4230ed4640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.791+0000 7f4230ed4640 10 monclient: monmap: 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:38.792310+0000 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:38.792310+0000 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:38.794 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 Processor -- start 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 -- start start 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient: init 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f422c0a5780 0x7f422c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f422c0a5780 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f422c09ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f422c069c90 0x7f422c139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f422c069c90 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0a5b50 con 0x7f422c09ccf0 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0a18b0 con 0x7f422c0a5780 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c09c5f0 con 0x7f422c069c90 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f4230ed4640 10 monclient(hunting): authenticate will time out at 1178.788746s 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f422c0a5780 0x7f422c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f422c0a5780 0x7f422c09bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38502/0 (socket says 172.21.15.17:38502) 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 1 -- 172.21.15.17:0/1862353107 learned_addr learned my addr 172.21.15.17:0/1862353107 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 10 monclient(hunting): get_auth_request con 0x7f422c0a5780 auth_method 0 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.792+0000 7f422ad9d640 1 --2- 172.21.15.17:0/1862353107 >> v2:172.21.15.67:3300/0 conn(0x7f422c069c90 0x7f422c139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): get_auth_request con 0x7f422c069c90 auth_method 0 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient(hunting): handle_auth_done global_id 104121 payload 274 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 1 -- 172.21.15.17:0/1862353107 >> v2:172.21.15.67:3300/0 conn(0x7f422c069c90 msgr2=0x7f422c139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 1 --2- 172.21.15.17:0/1862353107 >> v2:172.21.15.67:3300/0 conn(0x7f422c069c90 0x7f422c139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 1 -- 172.21.15.17:0/1862353107 >> v1:172.21.15.67:6789/0 conn(0x7f422c09ccf0 legacy=0x7f422c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 1 monclient: found mon.noname-a 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.793+0000 7f422a59c640 1 -- 172.21.15.17:0/1862353107 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c09d210 con 0x7f422c0a5780 2024-09-15T01:58:38.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.795+0000 7f422a59c640 1 -- 172.21.15.17:0/1862353107 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c0a5780 msgr2=0x7f422c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.795+0000 7f422a59c640 1 --2- 172.21.15.17:0/1862353107 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c0a5780 0x7f422c09bc90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f422c06f800 tx=0x7f421c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d3cc160ea9a22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 1 -- 172.21.15.17:0/1862353107 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c032440 con 0x7f422c0a5780 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: dump: 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.807 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 1 -- 172.21.15.17:0/1862353107 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c0325a0 con 0x7f422c0a5780 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 5 monclient: authenticate success, global_id 104121 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 4 set_mon_vals no callback set 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 1 -- 172.21.15.17:0/1862353107 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c032940 con 0x7f422c0a5780 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f422959a640 10 monclient: dump: 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 10 monclient: shutdown 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 -- 172.21.15.17:0/1862353107 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c0a5780 msgr2=0x7f422c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1862353107 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c0a5780 0x7f422c09bc90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f422c06f800 tx=0x7f421c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 -- 172.21.15.17:0/1862353107 shutdown_connections 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1862353107 >> v2:172.21.15.67:3300/0 conn(0x7f422c069c90 0x7f422c139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1862353107 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c0a5780 0x7f422c09bc90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 -- 172.21.15.17:0/1862353107 >> 172.21.15.17:0/1862353107 conn(0x7f422c070e10 msgr2=0x7f422c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 -- 172.21.15.17:0/1862353107 shutdown_connections 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.796+0000 7f4230ed4640 1 -- 172.21.15.17:0/1862353107 wait complete. 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 10 monclient: monmap: 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:38.810 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 1 Processor -- start 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 1 -- start start 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 10 monclient: init 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.797+0000 7f4230ed4640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 0x7f422c13ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f422c069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c09ccf0 0x7f422c13d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f422c09ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f422c0a5780 0x7f422c141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f422c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c122dc0 con 0x7f422c069c90 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c122800 con 0x7f422c09ccf0 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c120290 con 0x7f422c0a5780 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4230ed4640 10 monclient(hunting): authenticate will time out at 1178.794091s 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 0x7f422c13ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 0x7f422c13ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38504/0 (socket says 172.21.15.17:38504) 2024-09-15T01:58:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 -- 172.21.15.17:0/1883733316 learned_addr learned my addr 172.21.15.17:0/1883733316 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): get_auth_request con 0x7f422c069c90 auth_method 0 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f422c0a5780 0x7f422c141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): get_auth_request con 0x7f422c0a5780 auth_method 0 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c09ccf0 0x7f422c13d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 10 monclient(hunting): get_auth_request con 0x7f422c09ccf0 auth_method 0 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient(hunting): handle_auth_done global_id 104124 payload 824 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f422c0a5780 msgr2=0x7f422c141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f422c0a5780 0x7f422c141340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c09ccf0 msgr2=0x7f422c13d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c09ccf0 0x7f422c13d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 monclient: found mon.noname-a 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f422a59c640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c009450 con 0x7f422c069c90 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.798+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c09ccf0 0x7f422c13d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f422a59c640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 msgr2=0x7f422c13ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f422a59c640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 0x7f422c13ddd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f421c035040 tx=0x7f421c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0319c0 con 0x7f422c069c90 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: dump: 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c031b40 con 0x7f422c069c90 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 5 monclient: authenticate success, global_id 104124 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 10 monclient: _renew_subs 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c122520 con 0x7f422c069c90 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c03d440 con 0x7f422c069c90 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f421a7fc640 10 monclient: dump: 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.817 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4228d99640 4 set_mon_vals no callback set 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 10 monclient: _renew_subs 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.800+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c122b90 con 0x7f422c069c90 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f421c039040 con 0x7f422c069c90 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f4230ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 0x7f422c141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c063b20 con 0x7f422c121650 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f421a7fc640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f420c038740 0x7f420c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.801+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(178..178 src has 148..178) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f421c06c200 con 0x7f422c069c90 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f422ad9d640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f420c038740 0x7f420c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 0x7f422c141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f422ad9d640 10 monclient: get_auth_request con 0x7f420c038740 auth_method 0 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient: get_auth_request con 0x7f422c121650 auth_method 0 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 10 monclient(hunting): handle_auth_done global_id 6184 payload 824 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.802+0000 7f4229d9b640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c122f30 con 0x7f422c121650 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f422ad9d640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f420c038740 0x7f420c03abf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4224009fd0 tx=0x7f422400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f4229d9b640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 msgr2=0x7f422c141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 0x7f422c141880 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f422c13ee40 tx=0x7f4220009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220007030 con 0x7f422c121650 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 10 monclient: dump: 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4220007190 con 0x7f422c121650 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 msgr2=0x7f422c141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.804+0000 7f421a7fc640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 0x7f422c141880 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f422c13ee40 tx=0x7f4220009040 comp rx=0 tx=0).stop 2024-09-15T01:58:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.805+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 msgr2=0x7f422c141880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.805+0000 7f421a7fc640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c121650 0x7f422c141880 secure :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f422c13ee40 tx=0x7f4220009040 comp rx=0 tx=0).stop 2024-09-15T01:58:38.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.810+0000 7f4230ed4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:38.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.810+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 0x7f422c141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.810+0000 7f4230ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.810+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13dbc0 con 0x7f422c13e750 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 0x7f422c141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient: get_auth_request con 0x7f422c13e750 auth_method 0 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 10 monclient(hunting): handle_auth_done global_id 6187 payload 824 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.811+0000 7f4229d9b640 1 -- 172.21.15.17:0/1883733316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f422c0724a0 con 0x7f422c13e750 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f4229d9b640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 msgr2=0x7f422c141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f4229d9b640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 0x7f422c141dc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4220018000 tx=0x7f4220019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220006c90 con 0x7f422c13e750 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f421a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f421a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.813+0000 7f421a7fc640 10 monclient: dump: 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.825 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f4220006df0 con 0x7f422c13e750 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 msgr2=0x7f422c141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 0x7f422c141dc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4220018000 tx=0x7f4220019000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 msgr2=0x7f422c141dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f421a7fc640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422c13e750 0x7f422c141dc0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:38.839 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:38.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f420c038740 msgr2=0x7f420c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f420c038740 0x7f420c03abf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4224009fd0 tx=0x7f422400b040 comp rx=0 tx=0).stop 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 10 monclient: shutdown 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 msgr2=0x7f422c13ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 --2- 172.21.15.17:0/1883733316 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f422c069c90 0x7f422c13ddd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f421c035040 tx=0x7f421c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f422ad9d640 1 -- 172.21.15.17:0/1883733316 reap_dead start 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f422ad9d640 1 -- 172.21.15.17:0/1883733316 reap_dead start 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 shutdown_connections 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 >> 172.21.15.17:0/1883733316 conn(0x7f422c070e10 msgr2=0x7f422c09dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 shutdown_connections 2024-09-15T01:58:38.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.814+0000 7f4230ed4640 1 -- 172.21.15.17:0/1883733316 wait complete. 2024-09-15T01:58:38.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:38.842 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:58:38.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:58:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.936+0000 7f6c31add640 10 monclient: get_monmap_and_config 2024-09-15T01:58:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.936+0000 7f6c31add640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.936+0000 7f6c31add640 10 monclient: monmap: 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:38.937491+0000 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:38.937491+0000 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:38.939 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 Processor -- start 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 -- start start 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient: init 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c2c068f20 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c2c0692f0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 0x7f6c2c13a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-b con 0x7f6c2c137ff0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c0a1950 con 0x7f6c2c068f20 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c0a1670 con 0x7f6c2c0692f0 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c1328d0 con 0x7f6c2c137ff0 2024-09-15T01:58:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.937+0000 7f6c31add640 10 monclient(hunting): authenticate will time out at 1178.933969s 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 0x7f6c2c13a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6c2c068f20 0x7f6c2c1321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48848/0 (socket says 172.21.15.17:48848) 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2b7fe640 1 -- 172.21.15.17:0/1127349890 learned_addr learned my addr 172.21.15.17:0/1127349890 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): get_auth_request con 0x7f6c2c137ff0 auth_method 0 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_done global_id 94271 payload 274 2024-09-15T01:58:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 1 -- 172.21.15.17:0/1127349890 >> v1:172.21.15.67:6789/0 conn(0x7f6c2c0692f0 legacy=0x7f6c2c0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 1 -- 172.21.15.17:0/1127349890 >> v1:172.21.15.17:6789/0 conn(0x7f6c2c068f20 legacy=0x7f6c2c1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 1 monclient: found mon.noname-b 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.938+0000 7f6c2bfff640 1 -- 172.21.15.17:0/1127349890 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c0a0020 con 0x7f6c2c137ff0 2024-09-15T01:58:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2bfff640 1 -- 172.21.15.17:0/1127349890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 msgr2=0x7f6c2c13a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2bfff640 1 --2- 172.21.15.17:0/1127349890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 0x7f6c2c13a3d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009ec0 tx=0x7f6c24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9c9d223db0e42f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 1 -- 172.21.15.17:0/1127349890 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24007840 con 0x7f6c2c137ff0 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: dump: 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.950 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 1 -- 172.21.15.17:0/1127349890 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2402d050 con 0x7f6c2c137ff0 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 5 monclient: authenticate success, global_id 94271 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 1 -- 172.21.15.17:0/1127349890 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24007bd0 con 0x7f6c2c137ff0 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 4 set_mon_vals no callback set 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c2a7fc640 10 monclient: dump: 2024-09-15T01:58:38.951 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 10 monclient: shutdown 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 -- 172.21.15.17:0/1127349890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 msgr2=0x7f6c2c13a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 --2- 172.21.15.17:0/1127349890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 0x7f6c2c13a3d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009ec0 tx=0x7f6c24009780 comp rx=0 tx=0).stop 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 -- 172.21.15.17:0/1127349890 shutdown_connections 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 --2- 172.21.15.17:0/1127349890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c137ff0 0x7f6c2c13a3d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 -- 172.21.15.17:0/1127349890 >> 172.21.15.17:0/1127349890 conn(0x7f6c2c070fb0 msgr2=0x7f6c2c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 -- 172.21.15.17:0/1127349890 shutdown_connections 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.941+0000 7f6c31add640 1 -- 172.21.15.17:0/1127349890 wait complete. 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.942+0000 7f6c31add640 10 monclient: build_initial_monmap 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.942+0000 7f6c31add640 10 monclient: monmap: 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.942+0000 7f6c31add640 1 Processor -- start 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.942+0000 7f6c31add640 1 -- start start 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.942+0000 7f6c31add640 10 monclient: init 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2c068f20 0x7f6c2c141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-b con 0x7f6c2c068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-c con 0x7f6c2c0692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): picked mon.noname-a con 0x7f6c2c137ff0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c11be50 con 0x7f6c2c137ff0 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c0a0750 con 0x7f6c2c068f20 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c09c550 con 0x7f6c2c0692f0 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c31add640 10 monclient(hunting): authenticate will time out at 1178.939358s 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38508/0 (socket says 172.21.15.17:38508) 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 1 -- 172.21.15.17:0/79706178 learned_addr learned my addr 172.21.15.17:0/79706178 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): get_auth_request con 0x7f6c2c137ff0 auth_method 0 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33876/0 (socket says 172.21.15.17:33876) 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2c068f20 0x7f6c2c141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): get_auth_request con 0x7f6c2c0692f0 auth_method 0 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): get_auth_request con 0x7f6c2c068f20 auth_method 0 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient(hunting): handle_auth_done global_id 94274 payload 824 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 -- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2c068f20 msgr2=0x7f6c2c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2c068f20 0x7f6c2c141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 -- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 msgr2=0x7f6c2c098990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient: _finish_hunting 0 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 monclient: found mon.noname-c 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2affd640 1 -- 172.21.15.17:0/79706178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24009a20 con 0x7f6c2c0692f0 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.943+0000 7f6c2bfff640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c2affd640 1 -- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 msgr2=0x7f6c2c141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c2affd640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c510 tx=0x7f6c20007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200117a0 con 0x7f6c2c0692f0 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: dump: 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20011920 con 0x7f6c2c0692f0 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20011ce0 con 0x7f6c2c0692f0 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 5 monclient: authenticate success, global_id 94274 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 10 monclient: _renew_subs 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c09aae0 con 0x7f6c2c0692f0 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c137fe640 10 monclient: dump: 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c29ffb640 4 set_mon_vals no callback set 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 10 monclient: _renew_subs 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.946+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c09ada0 con 0x7f6c2c0692f0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6c20012630 con 0x7f6c2c0692f0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c31add640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c31add640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c31add640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c11ca20 con 0x7f6c2c0692f0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c137fe640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 0x7f6c1803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(178..178 src has 148..178) v4 ==== 8015+0+0 (secure 0 0 0) 0x7f6c2004c250 con 0x7f6c2c0692f0 2024-09-15T01:58:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c2b7fe640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 0x7f6c1803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.947+0000 7f6c2b7fe640 10 monclient: get_auth_request con 0x7f6c18038790 auth_method 0 2024-09-15T01:58:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.950+0000 7f6c2b7fe640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 0x7f6c1803ac40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c007fd0 tx=0x7f6c1c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.952+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6c20010ac0 con 0x7f6c2c0692f0 2024-09-15T01:58:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.952+0000 7f6c137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:38.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:38.952+0000 7f6c137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:39.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.180+0000 7f6c31add640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:58:39.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.180+0000 7f6c31add640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:58:39.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.180+0000 7f6c31add640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:39.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.180+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6c2c09aae0 con 0x7f6c2c0692f0 2024-09-15T01:58:39.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.183+0000 7f6c137fe640 1 -- 172.21.15.17:0/79706178 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6c2c09aae0 con 0x7f6c2c0692f0 2024-09-15T01:58:39.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.183+0000 7f6c137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:58:39.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.183+0000 7f6c137fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.183+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v179) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1b5011eb80 con 0x7f1b50136d00 2024-09-15T01:58:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.183+0000 7f1b37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.183+0000 7f1b37fff640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 msgr2=0x7f6c1803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 0x7f6c1803ac40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c007fd0 tx=0x7f6c1c007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 10 monclient: shutdown 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 msgr2=0x7f6c2c141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c510 tx=0x7f6c20007e10 comp rx=0 tx=0).stop 2024-09-15T01:58:39.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 shutdown_connections 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6c18038790 0x7f6c1803ac40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6c2c137ff0 0x7f6c2c098990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6c2c0692f0 0x7f6c2c141bc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 --2- 172.21.15.17:0/79706178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2c068f20 0x7f6c2c141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 >> 172.21.15.17:0/79706178 conn(0x7f6c2c070fb0 msgr2=0x7f6c2c09e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 shutdown_connections 2024-09-15T01:58:39.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f6c31add640 1 -- 172.21.15.17:0/79706178 wait complete. 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=3979843502,config=3140514009,health=224038855,kv=1529054990,logm=3374680933}) 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=3561961565,config=3140514010,health=2313241988,kv=1529054990,logm=3294222091}) 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=3979843502,config=3140514009,health=224038855,kv=1529054990,logm=3374680933}) 2024-09-15T01:58:39.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=8,logm=61} crc {auth=3751867264,config=3140514009,health=1555756363,kv=1529054990,logm=583236891}) 2024-09-15T01:58:39.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:58:39.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.285+0000 7f1b5626f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:39.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.285+0000 7f1b5626f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:39.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.285+0000 7f1b5626f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.285+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f1b5011eb80 con 0x7f1b50136d00 2024-09-15T01:58:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.286+0000 7f1b37fff640 1 -- 172.21.15.67:0/940217288 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v179) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1b5011eb80 con 0x7f1b50136d00 2024-09-15T01:58:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.286+0000 7f1b37fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T01:58:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.286+0000 7f1b37fff640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-15T01:58:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:device 'fooo' does not appear in the crush map 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 msgr2=0x7f1b3c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 0x7f1b3c03abf0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1b40007fd0 tx=0x7f1b40007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 10 monclient: shutdown 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 msgr2=0x7f1b5009c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009ef0 tx=0x7f1b4800b040 comp rx=0 tx=0).stop 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 shutdown_connections 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b3c038740 0x7f1b3c03abf0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b50136d00 0x7f1b5009c950 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b500a3830 0x7f1b5009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 --2- 172.21.15.67:0/940217288 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b5006bef0 0x7f1b5009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 >> 172.21.15.67:0/940217288 conn(0x7f1b50070fb0 msgr2=0x7f1b50137f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 shutdown_connections 2024-09-15T01:58:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.288+0000 7f1b5626f640 1 -- 172.21.15.67:0/940217288 wait complete. 2024-09-15T01:58:39.299 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm barr 2024-09-15T01:58:39.375 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.373+0000 7f32bd4c0640 10 monclient: get_monmap_and_config 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.373+0000 7f32bd4c0640 10 monclient: build_initial_monmap 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.373+0000 7f32bd4c0640 10 monclient: monmap: 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:39.374600+0000 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:39.374600+0000 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:39.377 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 Processor -- start 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 -- start start 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient: init 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32b80a1a00 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 0x7f32b8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f32b81317a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32b81340c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b809d740 con 0x7f32b80a1a00 2024-09-15T01:58:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8097d40 con 0x7f32b81340c0 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8098da0 con 0x7f32b81317a0 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32bd4c0640 10 monclient(hunting): authenticate will time out at 1151.843726s 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32b77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32b81340c0 0x7f32b81364a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60580/0 (socket says 172.21.15.67:60580) 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.374+0000 7f32b77fe640 1 -- 172.21.15.67:0/4258211682 learned_addr learned my addr 172.21.15.67:0/4258211682 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 1 --2- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 0x7f32b8133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): get_auth_request con 0x7f32b81317a0 auth_method 0 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient(hunting): handle_auth_done global_id 94277 payload 274 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 1 -- 172.21.15.67:0/4258211682 >> v1:172.21.15.67:6789/0 conn(0x7f32b81340c0 legacy=0x7f32b81364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 1 -- 172.21.15.67:0/4258211682 >> v1:172.21.15.17:6789/0 conn(0x7f32b80a1a00 legacy=0x7f32b8096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 1 monclient: found mon.noname-b 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:39.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.375+0000 7f32affff640 1 -- 172.21.15.67:0/4258211682 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b8097ae0 con 0x7f32b81317a0 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32affff640 1 -- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 msgr2=0x7f32b8133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32affff640 1 --2- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 0x7f32b8133b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009880 tx=0x7f32a00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c59ae23e608e7b1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 1 -- 172.21.15.67:0/4258211682 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0032440 con 0x7f32b81317a0 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: dump: 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.386 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 1 -- 172.21.15.67:0/4258211682 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a00325a0 con 0x7f32b81317a0 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 1 -- 172.21.15.67:0/4258211682 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0032940 con 0x7f32b81317a0 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:39.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32b67fc640 10 monclient: dump: 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32bd4c0640 5 monclient: authenticate success, global_id 94277 2024-09-15T01:58:39.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32bd4c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:39.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.378+0000 7f32bd4c0640 4 set_mon_vals no callback set 2024-09-15T01:58:39.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 10 monclient: shutdown 2024-09-15T01:58:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 -- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 msgr2=0x7f32b8133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 0x7f32b8133b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009880 tx=0x7f32a00090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 -- 172.21.15.67:0/4258211682 shutdown_connections 2024-09-15T01:58:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/4258211682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81317a0 0x7f32b8133b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 -- 172.21.15.67:0/4258211682 >> 172.21.15.67:0/4258211682 conn(0x7f32b8071340 msgr2=0x7f32b80a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 -- 172.21.15.67:0/4258211682 shutdown_connections 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 -- 172.21.15.67:0/4258211682 wait complete. 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 10 monclient: build_initial_monmap 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 10 monclient: monmap: 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.379+0000 7f32bd4c0640 1 Processor -- start 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 -- start start 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient: init 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f32b80a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 0x7f32b8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f32b81317a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81340c0 0x7f32b8118f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f32b81340c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b809cd10 con 0x7f32b81317a0 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a0cb0 con 0x7f32b80a1a00 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b811a150 con 0x7f32b81340c0 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32bd4c0640 10 monclient(hunting): authenticate will time out at 1151.849498s 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51176/0 (socket says 172.21.15.67:51176) 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 -- 172.21.15.67:0/1471989243 learned_addr learned my addr 172.21.15.67:0/1471989243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): get_auth_request con 0x7f32b80a1a00 auth_method 0 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:39.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 0x7f32b8141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81340c0 0x7f32b8118f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): get_auth_request con 0x7f32b81317a0 auth_method 0 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): get_auth_request con 0x7f32b81340c0 auth_method 0 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:39.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient(hunting): handle_auth_done global_id 6193 payload 824 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 -- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81340c0 msgr2=0x7f32b8118f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81340c0 0x7f32b8118f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 -- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 msgr2=0x7f32b8141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 0x7f32b8141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:39.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 1 monclient: found mon.noname-b 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.380+0000 7f32b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.381+0000 7f32b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.381+0000 7f32b6ffd640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009390 con 0x7f32b80a1a00 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.381+0000 7f32affff640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 0x7f32b8141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32b6ffd640 1 -- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 msgr2=0x7f32b8099690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32b6ffd640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c610 tx=0x7f32a8007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8012840 con 0x7f32b80a1a00 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: dump: 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:39.399 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a80129e0 con 0x7f32b80a1a00 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 5 monclient: authenticate success, global_id 6193 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 10 monclient: _renew_subs 2024-09-15T01:58:39.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b811fb40 con 0x7f32b80a1a00 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8012da0 con 0x7f32b80a1a00 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32aeffd640 10 monclient: dump: 2024-09-15T01:58:39.401 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32b5ffb640 4 set_mon_vals no callback set 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 10 monclient: _renew_subs 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.383+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b811a530 con 0x7f32b80a1a00 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f32a8011590 con 0x7f32b80a1a00 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32bd4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32bd4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32bd4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b811fcd0 con 0x7f32b80a1a00 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32aeffd640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 0x7f329803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(179..179 src has 148..179) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f32a804c200 con 0x7f32b80a1a00 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.384+0000 7f32affff640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 0x7f329803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:39.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.385+0000 7f32affff640 10 monclient: get_auth_request con 0x7f3298038740 auth_method 0 2024-09-15T01:58:39.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.387+0000 7f32affff640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 0x7f329803abf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f32b8068bc0 tx=0x7f32a0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:39.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.389+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f32a80198e0 con 0x7f32b80a1a00 2024-09-15T01:58:39.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.389+0000 7f32aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:39.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.389+0000 7f32aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.555+0000 7f32bd4c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.555+0000 7f32bd4c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.555+0000 7f32bd4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:39.555+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f32b804bf60 con 0x7f32b80a1a00 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116155578}) 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1704340425}) 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116155578}) 2024-09-15T01:58:39.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347194253}) 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345050580}) 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615283444}) 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345050580}) 2024-09-15T01:58:39.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998092825}) 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169248545}) 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987387857}) 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169248545}) 2024-09-15T01:58:39.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744115317}) 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709092421}) 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669077729}) 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709092421}) 2024-09-15T01:58:39.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263917765}) 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827076436}) 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=905492642}) 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827076436}) 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=528735786}) 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636628770}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708463229}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636628770}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376878399}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528038719}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170207252}) 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.623 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528038719}) 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007843097}) 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583157344}) 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3653415822}) 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583157344}) 2024-09-15T01:58:39.624 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202229634}) 2024-09-15T01:58:39.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252035294}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321586603}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252035294}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869647488}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703074146}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747855982}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703074146}) 2024-09-15T01:58:39.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4101205932}) 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257629714}) 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816860586}) 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257629714}) 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774214213}) 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.627 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900073918}) 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143337759}) 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900073918}) 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995723061}) 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183590216}) 2024-09-15T01:58:39.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=544957016}) 2024-09-15T01:58:39.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183590216}) 2024-09-15T01:58:39.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=745026399}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413876905}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565940581}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413876905}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357795377}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318831695}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=279039012}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318831695}) 2024-09-15T01:58:39.630 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424538070}) 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438849462}) 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315914802}) 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438849462}) 2024-09-15T01:58:39.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672931967}) 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673842901}) 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2036467245}) 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673842901}) 2024-09-15T01:58:39.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373232049}) 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031114216}) 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962756967}) 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031114216}) 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719187208}) 2024-09-15T01:58:39.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033386646}) 2024-09-15T01:58:39.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2843290596}) 2024-09-15T01:58:39.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033386646}) 2024-09-15T01:58:39.634 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.631+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111941151}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547952042}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1089697750}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547952042}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.632+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196861382}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821210971}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194450667}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821210971}) 2024-09-15T01:58:39.635 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288540886}) 2024-09-15T01:58:39.636 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.636 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677442277}) 2024-09-15T01:58:39.636 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599285072}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677442277}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.633+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183408100}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264357113}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819163456}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264357113}) 2024-09-15T01:58:39.637 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.634+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995859936}) 2024-09-15T01:58:39.638 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615771584}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327617574}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615771584}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.635+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4133718021}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642894844}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452340516}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642894844}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083822059}) 2024-09-15T01:58:39.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146370674}) 2024-09-15T01:58:39.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59645236}) 2024-09-15T01:58:39.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146370674}) 2024-09-15T01:58:39.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588197639}) 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786447642}) 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160618689}) 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786447642}) 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.637+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2221493020}) 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.641 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631146760}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503978282}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631146760}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.638+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1119841932}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152793605}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371910988}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152793605}) 2024-09-15T01:58:39.642 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.639+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=171236820}) 2024-09-15T01:58:39.643 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.643 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092937518}) 2024-09-15T01:58:39.643 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136692255}) 2024-09-15T01:58:39.643 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092937518}) 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.640+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890852694}) 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=1354465401,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2204936280}) 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=2033841196,mdsmap=1531151339,mgr=2670634280,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2204936280,mon_config_key=1386283410}) 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=1354465401,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2204936280}) 2024-09-15T01:58:39.644 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.641+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3} crc {logm=124529743,mdsmap=3585761405,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=159652000,mon_config_key=1583035879,monmap=3178194974}) 2024-09-15T01:58:39.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=64} crc {mon_config_key=1583035879,monmap=3178194974,osd_metadata=611002287,osd_pg_creating=3060708514,osd_snap=1063953654,osdmap=4153578394}) 2024-09-15T01:58:39.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=60} crc {mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=3060708514,osd_snap=1063953654,osdmap=4084652450}) 2024-09-15T01:58:39.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:58:39.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=64} crc {mon_config_key=1583035879,monmap=3178194974,osd_metadata=611002287,osd_pg_creating=3060708514,osd_snap=1063953654,osdmap=4153578394}) 2024-09-15T01:58:39.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:58:39.645+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=61} crc {mon_config_key=1583035879,monmap=3178194974,osd_metadata=1079644522,osd_pg_creating=3060708514,osd_snap=3718248,osdmap=2650028548}) 2024-09-15T01:58:40.199 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:58:40.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:40.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.196+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v180) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32a8019310 con 0x7f32b80a1a00 2024-09-15T01:58:40.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.197+0000 7f32aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:40.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.197+0000 7f32aeffd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-15T01:58:40.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.300+0000 7f39f1b1d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:40.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.300+0000 7f39f1b1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.300+0000 7f39f1b1d640 10 monclient: monmap: 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:40.301443+0000 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:40.301443+0000 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:40.303 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.300+0000 7f32bd4c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.300+0000 7f32bd4c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.300+0000 7f32bd4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.300+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f32b80008d0 con 0x7f32b80a1a00 2024-09-15T01:58:40.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.301+0000 7f32aeffd640 1 -- 172.21.15.67:0/1471989243 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v180) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32a80194f0 con 0x7f32b80a1a00 2024-09-15T01:58:40.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.301+0000 7f32aeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T01:58:40.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.301+0000 7f32aeffd640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-15T01:58:40.305 INFO:tasks.workunit.client.0.smithi067.stderr:device 'barr' does not appear in the crush map 2024-09-15T01:58:40.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 1 Processor -- start 2024-09-15T01:58:40.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 1 -- start start 2024-09-15T01:58:40.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 10 monclient: init 2024-09-15T01:58:40.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.301+0000 7f39f1b1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f39ec0708a0 0x7f39ec097870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f39ec0708a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 0x7f39ec06a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f39ec06a1c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39ec06aad0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec070c70 con 0x7f39ec06aad0 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec06d7d0 con 0x7f39ec0708a0 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec096b40 con 0x7f39ec06a1c0 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39f1b1d640 10 monclient(hunting): authenticate will time out at 1180.298095s 2024-09-15T01:58:40.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f39ec0708a0 0x7f39ec097870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f39ec0708a0 0x7f39ec097870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38512/0 (socket says 172.21.15.17:38512) 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 -- 172.21.15.17:0/1038676545 learned_addr learned my addr 172.21.15.17:0/1038676545 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): get_auth_request con 0x7f39ec0708a0 auth_method 0 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39ebfff640 1 --1- 172.21.15.17:0/1038676545 >> v1:172.21.15.17:6789/0 conn(0x7f39ec06aad0 0x7f39ec12fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48860/0 (socket says 172.21.15.17:48860) 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 0x7f39ec06a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): get_auth_request con 0x7f39ec06a1c0 auth_method 0 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_done global_id 104133 payload 274 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 -- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 msgr2=0x7f39ec06a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 0x7f39ec06a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 -- 172.21.15.17:0/1038676545 >> v1:172.21.15.17:6789/0 conn(0x7f39ec06aad0 legacy=0x7f39ec12fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 monclient: found mon.noname-a 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eb7fe640 1 -- 172.21.15.17:0/1038676545 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec06e5a0 con 0x7f39ec0708a0 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.302+0000 7f39eaffd640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 0x7f39ec06a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39eb7fe640 1 -- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec0708a0 msgr2=0x7f39ec097870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39eb7fe640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec0708a0 0x7f39ec097870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009910 tx=0x7f39dc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf75a0dfe98f533 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 1 -- 172.21.15.17:0/1038676545 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc02aea0 con 0x7f39ec0708a0 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: dump: 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 1 -- 172.21.15.17:0/1038676545 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc032440 con 0x7f39ec0708a0 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39f1b1d640 5 monclient: authenticate success, global_id 104133 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39f1b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39f1b1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 1 -- 172.21.15.17:0/1038676545 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc0327e0 con 0x7f39ec0708a0 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39f1b1d640 4 set_mon_vals no callback set 2024-09-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.305+0000 7f39ea7fc640 10 monclient: dump: 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 10 monclient: shutdown 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 -- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec0708a0 msgr2=0x7f39ec097870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec0708a0 0x7f39ec097870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009910 tx=0x7f39dc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 -- 172.21.15.17:0/1038676545 shutdown_connections 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06a1c0 0x7f39ec06a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/1038676545 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec0708a0 0x7f39ec097870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 -- 172.21.15.17:0/1038676545 >> 172.21.15.17:0/1038676545 conn(0x7f39ec067ae0 msgr2=0x7f39ec0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 -- 172.21.15.17:0/1038676545 shutdown_connections 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 1 -- 172.21.15.17:0/1038676545 wait complete. 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.306+0000 7f39f1b1d640 10 monclient: monmap: 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.315 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 msgr2=0x7f329803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 0x7f329803abf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f32b8068bc0 tx=0x7f32a0032000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 10 monclient: shutdown 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 msgr2=0x7f32b8099690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c610 tx=0x7f32a8007ed0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 shutdown_connections 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3298038740 0x7f329803abf0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32b81317a0 0x7f32b8141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32b81340c0 0x7f32b8118f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 --2- 172.21.15.67:0/1471989243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32b80a1a00 0x7f32b8099690 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.303+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 >> 172.21.15.67:0/1471989243 conn(0x7f32b8071340 msgr2=0x7f32b80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.304+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 shutdown_connections 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.304+0000 7f32bd4c0640 1 -- 172.21.15.67:0/1471989243 wait complete. 2024-09-15T01:58:40.319 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 Processor -- start 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 -- start start 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient: init 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f39ec06a1c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f39ec06aad0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ec0708a0 0x7f39ec06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f39ec0708a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:40.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec06e8d0 con 0x7f39ec06a1c0 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec098230 con 0x7f39ec0708a0 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec142010 con 0x7f39ec06aad0 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39f1b1d640 10 monclient(hunting): authenticate will time out at 1180.303850s 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33902/0 (socket says 172.21.15.17:33902) 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38518/0 (socket says 172.21.15.17:38518) 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 1 -- 172.21.15.17:0/4048212210 learned_addr learned my addr 172.21.15.17:0/4048212210 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 10 monclient(hunting): get_auth_request con 0x7f39ec06aad0 auth_method 0 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 10 monclient(hunting): get_auth_request con 0x7f39ec06a1c0 auth_method 0 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.307+0000 7f39eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ec0708a0 0x7f39ec06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): get_auth_request con 0x7f39ec0708a0 auth_method 0 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient(hunting): handle_auth_done global_id 104136 payload 824 2024-09-15T01:58:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 -- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 msgr2=0x7f39ec06b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 -- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ec0708a0 msgr2=0x7f39ec06b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ec0708a0 0x7f39ec06b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 monclient: found mon.noname-a 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eb7fe640 1 -- 172.21.15.17:0/4048212210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009390 con 0x7f39ec06a1c0 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.308+0000 7f39eaffd640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39eb7fe640 1 -- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 msgr2=0x7f39ec132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39eb7fe640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39dc031000 tx=0x7f39dc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc02aea0 con 0x7f39ec06a1c0 2024-09-15T01:58:40.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: dump: 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.328 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc031c70 con 0x7f39ec06a1c0 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 5 monclient: authenticate success, global_id 104136 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 10 monclient: _renew_subs 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc009450 con 0x7f39ec06a1c0 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc03d440 con 0x7f39ec06a1c0 2024-09-15T01:58:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39d37fe640 10 monclient: dump: 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.330 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39e9ffb640 4 set_mon_vals no callback set 2024-09-15T01:58:40.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 10 monclient: _renew_subs 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.310+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec122110 con 0x7f39ec06a1c0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f39dc039040 con 0x7f39ec06a1c0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39f1b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39f1b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39f1b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec0724a0 con 0x7f39ec06a1c0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39d37fe640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 0x7f39d803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.311+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f39dc06b630 con 0x7f39ec06a1c0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.312+0000 7f39eaffd640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 0x7f39d803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.312+0000 7f39eaffd640 10 monclient: get_auth_request con 0x7f39d80387c0 auth_method 0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.314+0000 7f39eaffd640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 0x7f39d803ac70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f39ec122040 tx=0x7f39e00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.316+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f39dc038860 con 0x7f39ec06a1c0 2024-09-15T01:58:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.316+0000 7f39d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.316+0000 7f39d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:40.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.388+0000 7f0ed8b5d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.388+0000 7f0ed8b5d640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.388+0000 7f0ed8b5d640 10 monclient: monmap: 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:40.389557+0000 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:40.389557+0000 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:40.391 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.388+0000 7f0ed8b5d640 1 Processor -- start 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 -- start start 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient: init 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ed40a3850 0x7f0ed406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40a3850 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0ed406a6e0 0x7f0ed409db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed406a6e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ed409e050 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed409f9a0 con 0x7f0ed409e050 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed4069c20 con 0x7f0ed406a6e0 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40696c0 con 0x7f0ed40a3850 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed8b5d640 10 monclient(hunting): authenticate will time out at 1152.858673s 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ed40a3850 0x7f0ed406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed2d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0ed409e050 0x7f0ed4138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43096/0 (socket says 172.21.15.67:43096) 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/3838725572 learned_addr learned my addr 172.21.15.67:0/3838725572 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): get_auth_request con 0x7f0ed40a3850 auth_method 0 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed1d9b640 1 --2- 172.21.15.67:0/3838725572 >> v2:172.21.15.17:3300/0 conn(0x7f0ed406a6e0 0x7f0ed409db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ed406a6e0 auth_method 0 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.389+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 10 monclient(hunting): handle_auth_done global_id 6199 payload 274 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 1 -- 172.21.15.67:0/3838725572 >> v2:172.21.15.17:3300/0 conn(0x7f0ed406a6e0 msgr2=0x7f0ed409db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 1 --2- 172.21.15.67:0/3838725572 >> v2:172.21.15.17:3300/0 conn(0x7f0ed406a6e0 0x7f0ed409db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 1 -- 172.21.15.67:0/3838725572 >> v1:172.21.15.67:6789/0 conn(0x7f0ed409e050 legacy=0x7f0ed4138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 1 monclient: found mon.noname-c 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.390+0000 7f0ed259c640 1 -- 172.21.15.67:0/3838725572 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40a3c20 con 0x7f0ed40a3850 2024-09-15T01:58:40.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.392+0000 7f0ed259c640 1 -- 172.21.15.67:0/3838725572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.392+0000 7f0ed259c640 1 --2- 172.21.15.67:0/3838725572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed406e1a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40a4020 tx=0x7f0ec40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2f6c75be3992fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 1 -- 172.21.15.67:0/3838725572 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec402ae90 con 0x7f0ed40a3850 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: dump: 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 1 -- 172.21.15.67:0/3838725572 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec4032440 con 0x7f0ed40a3850 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 5 monclient: authenticate success, global_id 6199 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 4 set_mon_vals no callback set 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 1 -- 172.21.15.67:0/3838725572 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4032820 con 0x7f0ed40a3850 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed159a640 10 monclient: dump: 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 10 monclient: shutdown 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/3838725572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/3838725572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed406e1a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40a4020 tx=0x7f0ec40090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/3838725572 shutdown_connections 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/3838725572 >> v2:172.21.15.17:3300/0 conn(0x7f0ed406a6e0 0x7f0ed409db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/3838725572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed406e1a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/3838725572 >> 172.21.15.67:0/3838725572 conn(0x7f0ed4070ff0 msgr2=0x7f0ed40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/3838725572 shutdown_connections 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.393+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/3838725572 wait complete. 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient: monmap: 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 Processor -- start 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 -- start start 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient: init 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0ed4133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed406a6e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0ed411db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed409e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40a3850 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4121180 con 0x7f0ed406a6e0 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4141ff0 con 0x7f0ed40a3850 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4136600 con 0x7f0ed409e050 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.394+0000 7f0ed8b5d640 10 monclient(hunting): authenticate will time out at 1152.863973s 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0ed4133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50268/0 (socket says 172.21.15.67:50268) 2024-09-15T01:58:40.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 learned_addr learned my addr 172.21.15.67:0/296701874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): get_auth_request con 0x7f0ed40a3850 auth_method 0 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): get_auth_request con 0x7f0ed406a6e0 auth_method 0 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0ed411db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ed409e050 auth_method 0 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_done global_id 6202 payload 824 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 msgr2=0x7f0ed411db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0ed411db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 msgr2=0x7f0ed4133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0ed4133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 monclient: found mon.noname-b 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4009390 con 0x7f0ed40a3850 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.395+0000 7f0ed259c640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0ed4133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed411e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ccc0 tx=0x7f0ecc005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00ce80 con 0x7f0ed40a3850 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: dump: 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.411 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc00aa60 con 0x7f0ed40a3850 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00ae20 con 0x7f0ed40a3850 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 5 monclient: authenticate success, global_id 6202 2024-09-15T01:58:40.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ec27fc640 10 monclient: dump: 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 10 monclient: _renew_subs 2024-09-15T01:58:40.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed0d99640 4 set_mon_vals no callback set 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed411ff40 con 0x7f0ed40a3850 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 10 monclient: _renew_subs 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.397+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed4121460 con 0x7f0ed40a3850 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ccc0 tx=0x7f0ecc005370 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f0ecc04c6f0 osd_map(180..180 src has 148..180) v4 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ecc007300 con 0x7f0ed40a3850 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ed8b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ed8b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ed8b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed404bf60 con 0x7f0ed40a3850 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.398+0000 7f0ec27fc640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 0x7f0eb403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.399+0000 7f0ed259c640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 0x7f0eb403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.399+0000 7f0ed259c640 10 monclient: get_auth_request con 0x7f0eb4038740 auth_method 0 2024-09-15T01:58:40.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.401+0000 7f0ed259c640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 0x7f0eb403abf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4034000 tx=0x7f0ec4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:40.403+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f0ecc04c6f0 con 0x7f0ed40a3850 2024-09-15T01:58:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.542+0000 7f39f1b1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.542+0000 7f39f1b1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.542+0000 7f39f1b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.542+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39ec134630 con 0x7f39ec06a1c0 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.543+0000 7f39d37fe640 1 -- 172.21.15.17:0/4048212210 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f39dc037080 con 0x7f39ec06a1c0 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.543+0000 7f39d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.543+0000 7f39d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:40.545 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:40.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 msgr2=0x7f39d803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 0x7f39d803ac70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f39ec122040 tx=0x7f39e00090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 10 monclient: shutdown 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 msgr2=0x7f39ec132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39dc031000 tx=0x7f39dc032000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 shutdown_connections 2024-09-15T01:58:40.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f39d80387c0 0x7f39d803ac70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ec0708a0 0x7f39ec06b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f39ec06aad0 0x7f39ec06b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 --2- 172.21.15.17:0/4048212210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f39ec06a1c0 0x7f39ec132fd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 >> 172.21.15.17:0/4048212210 conn(0x7f39ec067ae0 msgr2=0x7f39ec09bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 shutdown_connections 2024-09-15T01:58:40.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.545+0000 7f39f1b1d640 1 -- 172.21.15.17:0/4048212210 wait complete. 2024-09-15T01:58:40.557 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:58:40.558 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:58:40.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:58:40.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.654+0000 7fe48f299640 10 monclient: get_monmap_and_config 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.654+0000 7fe48f299640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.654+0000 7fe48f299640 10 monclient: monmap: 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:40.655520+0000 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:40.655520+0000 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 Processor -- start 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 -- start start 2024-09-15T01:58:40.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient: init 2024-09-15T01:58:40.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T01:58:40.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe48806eb30 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:40.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-b con 0x7fe48809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 0x7fe488139b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-c con 0x7fe48809c4d0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48809f6a0 con 0x7fe48806eb30 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe488069f40 con 0x7fe48809c4d0 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48809ed40 con 0x7fe48809e970 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.655+0000 7fe48f299640 10 monclient(hunting): authenticate will time out at 1180.651995s 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33914/0 (socket says 172.21.15.17:33914) 2024-09-15T01:58:40.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 -- 172.21.15.17:0/549322410 learned_addr learned my addr 172.21.15.17:0/549322410 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:40.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): get_auth_request con 0x7fe48809e970 auth_method 0 2024-09-15T01:58:40.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48d836640 1 --2- 172.21.15.17:0/549322410 >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 0x7fe488139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient(hunting): handle_auth_done global_id 94292 payload 274 2024-09-15T01:58:40.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 -- 172.21.15.17:0/549322410 >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 msgr2=0x7fe488139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 --2- 172.21.15.17:0/549322410 >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 0x7fe488139b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 -- 172.21.15.17:0/549322410 >> v1:172.21.15.67:6789/0 conn(0x7fe48806eb30 legacy=0x7fe48806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 monclient: found mon.noname-b 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:40.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48c834640 1 -- 172.21.15.17:0/549322410 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe488069950 con 0x7fe48809e970 2024-09-15T01:58:40.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48d836640 10 monclient: get_auth_request con 0x7fe48809c4d0 auth_method 0 2024-09-15T01:58:40.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.656+0000 7fe48d836640 1 --2- 172.21.15.17:0/549322410 >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 0x7fe488139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:58:40.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48c834640 1 -- 172.21.15.17:0/549322410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 msgr2=0x7fe48809bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48c834640 1 --2- 172.21.15.17:0/549322410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe484009910 tx=0x7fe4840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0665d8fff7b1178 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 1 -- 172.21.15.17:0/549322410 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484032440 con 0x7fe48809e970 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: dump: 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 5 monclient: authenticate success, global_id 94292 2024-09-15T01:58:40.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:40.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 1 -- 172.21.15.17:0/549322410 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4840325a0 con 0x7fe48809e970 2024-09-15T01:58:40.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:40.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 1 -- 172.21.15.17:0/549322410 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484032940 con 0x7fe48809e970 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 4 set_mon_vals no callback set 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe47ffff640 10 monclient: dump: 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.680 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.681 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.681 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.681 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.681 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.682 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.682 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.682 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.682 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 10 monclient: shutdown 2024-09-15T01:58:40.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 -- 172.21.15.17:0/549322410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 msgr2=0x7fe48809bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 --2- 172.21.15.17:0/549322410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe484009910 tx=0x7fe4840091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 -- 172.21.15.17:0/549322410 shutdown_connections 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 --2- 172.21.15.17:0/549322410 >> v2:172.21.15.67:3300/0 conn(0x7fe48809c4d0 0x7fe488139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 --2- 172.21.15.17:0/549322410 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809e970 0x7fe48809bf90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.659+0000 7fe48f299640 1 -- 172.21.15.17:0/549322410 >> 172.21.15.17:0/549322410 conn(0x7fe4880712f0 msgr2=0x7fe4880a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 1 -- 172.21.15.17:0/549322410 shutdown_connections 2024-09-15T01:58:40.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 1 -- 172.21.15.17:0/549322410 wait complete. 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 10 monclient: monmap: 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:40.685 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 1 Processor -- start 2024-09-15T01:58:40.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 1 -- start start 2024-09-15T01:58:40.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.660+0000 7fe48f299640 10 monclient: init 2024-09-15T01:58:40.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:40.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48806eb30 0x7fe488133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-a con 0x7fe48806eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 0x7fe48811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-c con 0x7fe48809c4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48809e970 0x7fe48811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): picked mon.noname-b con 0x7fe48809e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488121000 con 0x7fe48806eb30 2024-09-15T01:58:40.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488141fa0 con 0x7fe48809e970 2024-09-15T01:58:40.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4881349a0 con 0x7fe48809c4d0 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48f299640 10 monclient(hunting): authenticate will time out at 1180.657483s 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 0x7fe48811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48806eb30 0x7fe488133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 0x7fe48811d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33928/0 (socket says 172.21.15.17:33928) 2024-09-15T01:58:40.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 -- 172.21.15.17:0/3954369571 learned_addr learned my addr 172.21.15.17:0/3954369571 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:40.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): get_auth_request con 0x7fe48809c4d0 auth_method 0 2024-09-15T01:58:40.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): get_auth_request con 0x7fe48806eb30 auth_method 0 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48809e970 0x7fe48811df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 10 monclient(hunting): get_auth_request con 0x7fe48809e970 auth_method 0 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient(hunting): handle_auth_done global_id 94295 payload 824 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48809e970 msgr2=0x7fe48811df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48809e970 0x7fe48811df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48806eb30 msgr2=0x7fe488133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48806eb30 0x7fe488133c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 monclient: found mon.noname-c 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:40.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48c834640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009450 con 0x7fe48809c4d0 2024-09-15T01:58:40.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.661+0000 7fe48d836640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48809e970 0x7fe48811df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:58:40.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48c834640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 msgr2=0x7fe48811d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48c834640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 0x7fe48811d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe484009970 tx=0x7fe484032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840319c0 con 0x7fe48809c4d0 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: dump: 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.698 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484031b20 con 0x7fe48809c4d0 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 5 monclient: authenticate success, global_id 94295 2024-09-15T01:58:40.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 10 monclient: _renew_subs 2024-09-15T01:58:40.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:40.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48811fe30 con 0x7fe48809c4d0 2024-09-15T01:58:40.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48403d440 con 0x7fe48809c4d0 2024-09-15T01:58:40.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47d7fa640 10 monclient: dump: 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe47f7fe640 4 set_mon_vals no callback set 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 10 monclient: _renew_subs 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.664+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48811f260 con 0x7fe48809c4d0 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48f299640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48f299640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 0x7fe48811e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488120ec0 con 0x7fe488134140 2024-09-15T01:58:40.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 0x7fe48811e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient: get_auth_request con 0x7fe488134140 auth_method 0 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe484039040 con 0x7fe48809c4d0 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 10 monclient(hunting): handle_auth_done global_id 104148 payload 824 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d035640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4880724a0 con 0x7fe488134140 2024-09-15T01:58:40.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe47d7fa640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe46c038740 0x7fe46c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fe48406c020 con 0x7fe48809c4d0 2024-09-15T01:58:40.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.665+0000 7fe48d836640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe46c038740 0x7fe46c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.666+0000 7fe48d836640 10 monclient: get_auth_request con 0x7fe46c038740 auth_method 0 2024-09-15T01:58:40.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe48d035640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 msgr2=0x7fe48811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe48d035640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 0x7fe48811e460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe47800e7b0 tx=0x7fe478006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800cac0 con 0x7fe488134140 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe47d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe47d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.667+0000 7fe47d7fa640 10 monclient: dump: 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe47800cc20 con 0x7fe488134140 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 msgr2=0x7fe48811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 0x7fe48811e460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe47800e7b0 tx=0x7fe478006f50 comp rx=0 tx=0).stop 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 msgr2=0x7fe48811e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe47d7fa640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe488134140 0x7fe48811e460 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe47800e7b0 tx=0x7fe478006f50 comp rx=0 tx=0).stop 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.668+0000 7fe48d836640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe46c038740 0x7fe46c03abf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e4c0 tx=0x7fe4800076c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.673+0000 7fe48f299640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48f299640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 0x7fe4880a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48f299640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488121470 con 0x7fe48811e9a0 2024-09-15T01:58:40.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 0x7fe4880a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient: get_auth_request con 0x7fe48811e9a0 auth_method 0 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 10 monclient(hunting): handle_auth_done global_id 104151 payload 824 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.674+0000 7fe48d035640 1 -- 172.21.15.17:0/3954369571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe48811eda0 con 0x7fe48811e9a0 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe48d035640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 msgr2=0x7fe4880a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe48d035640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 0x7fe4880a1410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe478009fd0 tx=0x7fe478015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800ce00 con 0x7fe48811e9a0 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe47d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe47d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.676+0000 7fe47d7fa640 10 monclient: dump: 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.712 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7fe47801cc50 con 0x7fe48811e9a0 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 msgr2=0x7fe4880a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 0x7fe4880a1410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe478009fd0 tx=0x7fe478015000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 msgr2=0x7fe4880a1410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:40.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe47d7fa640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe48811e9a0 0x7fe4880a1410 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe478009fd0 tx=0x7fe478015000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.714 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.716 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.717 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:40.719 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:40.720 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:40.721 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:40.721 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:40.721 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:40.721 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:40.721 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.722 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:40.723 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.724 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:40.726 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.727 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:40.728 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:40.728 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.728 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.728 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.729 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:40.729 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.729 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.730 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.732 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:40.732 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:40.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.732 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:40.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.735 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:40.735 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe46c038740 msgr2=0x7fe46c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe46c038740 0x7fe46c03abf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e4c0 tx=0x7fe4800076c0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 10 monclient: shutdown 2024-09-15T01:58:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 msgr2=0x7fe48811d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 --2- 172.21.15.17:0/3954369571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe48809c4d0 0x7fe48811d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe484009970 tx=0x7fe484032000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48d836640 1 -- 172.21.15.17:0/3954369571 reap_dead start 2024-09-15T01:58:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48d836640 1 -- 172.21.15.17:0/3954369571 reap_dead start 2024-09-15T01:58:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 shutdown_connections 2024-09-15T01:58:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 >> 172.21.15.17:0/3954369571 conn(0x7fe4880712f0 msgr2=0x7fe488134e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 shutdown_connections 2024-09-15T01:58:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.677+0000 7fe48f299640 1 -- 172.21.15.17:0/3954369571 wait complete. 2024-09-15T01:58:40.745 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:40.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:58:40.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.841+0000 7f8424c94640 10 monclient: get_monmap_and_config 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.841+0000 7f8424c94640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.841+0000 7f8424c94640 10 monclient: monmap: 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:40.842332+0000 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:40.842332+0000 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:40.844 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 Processor -- start 2024-09-15T01:58:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 -- start start 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient: init 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84201319a0 0x7f8420131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-c con 0x7f84201319a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84201322b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f842009f1d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200a1530 con 0x7f84201322b0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842009fca0 con 0x7f842009f1d0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84201326b0 con 0x7f84201319a0 2024-09-15T01:58:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8424c94640 10 monclient(hunting): authenticate will time out at 1180.838777s 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8415d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f84201322b0 0x7f842009eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48866/0 (socket says 172.21.15.17:48866) 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.842+0000 7f8415d9b640 1 -- 172.21.15.17:0/2781284886 learned_addr learned my addr 172.21.15.17:0/2781284886 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 1 --2- 172.21.15.17:0/2781284886 >> v2:172.21.15.67:3300/0 conn(0x7f84201319a0 0x7f8420131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): get_auth_request con 0x7f84201319a0 auth_method 0 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient(hunting): handle_auth_done global_id 6208 payload 274 2024-09-15T01:58:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 1 -- 172.21.15.17:0/2781284886 >> v1:172.21.15.67:6789/0 conn(0x7f842009f1d0 legacy=0x7f8420138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 1 -- 172.21.15.17:0/2781284886 >> v1:172.21.15.17:6789/0 conn(0x7f84201322b0 legacy=0x7f842009eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 1 monclient: found mon.noname-c 2024-09-15T01:58:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:58:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.843+0000 7f841e59c640 1 -- 172.21.15.17:0/2781284886 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200a00e0 con 0x7f84201319a0 2024-09-15T01:58:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841e59c640 1 -- 172.21.15.17:0/2781284886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201319a0 msgr2=0x7f8420131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841e59c640 1 --2- 172.21.15.17:0/2781284886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201319a0 0x7f8420131d70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8410009850 tx=0x7f84100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ded853db1d79e46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 1 -- 172.21.15.17:0/2781284886 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410032440 con 0x7f84201319a0 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: dump: 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.858 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 1 -- 172.21.15.17:0/2781284886 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84100325a0 con 0x7f84201319a0 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f8424c94640 5 monclient: authenticate success, global_id 6208 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f8424c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 1 -- 172.21.15.17:0/2781284886 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410032940 con 0x7f84201319a0 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:40.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f841dd9b640 10 monclient: dump: 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.860 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f8424c94640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.846+0000 7f8424c94640 4 set_mon_vals no callback set 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 10 monclient: shutdown 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 -- 172.21.15.17:0/2781284886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201319a0 msgr2=0x7f8420131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 --2- 172.21.15.17:0/2781284886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201319a0 0x7f8420131d70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8410009850 tx=0x7f84100090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 -- 172.21.15.17:0/2781284886 shutdown_connections 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 --2- 172.21.15.17:0/2781284886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201319a0 0x7f8420131d70 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 -- 172.21.15.17:0/2781284886 >> 172.21.15.17:0/2781284886 conn(0x7f84200a5000 msgr2=0x7f84200a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 -- 172.21.15.17:0/2781284886 shutdown_connections 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 1 -- 172.21.15.17:0/2781284886 wait complete. 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.847+0000 7f8424c94640 10 monclient: monmap: 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:40.862 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 Processor -- start 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 -- start start 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient: init 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 0x7f8420138560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-a con 0x7f842009f1d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f84201319a0 0x7f8420134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-c con 0x7f84201319a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201322b0 0x7f8420134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): picked mon.noname-b con 0x7f84201322b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:40.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842009bf50 con 0x7f842009f1d0 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200a4ca0 con 0x7f84201322b0 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8420135de0 con 0x7f84201319a0 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8424c94640 10 monclient(hunting): authenticate will time out at 1180.844836s 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 0x7f8420138560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 0x7f8420138560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38544/0 (socket says 172.21.15.17:38544) 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 1 -- 172.21.15.17:0/1950563922 learned_addr learned my addr 172.21.15.17:0/1950563922 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): get_auth_request con 0x7f842009f1d0 auth_method 0 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f84201319a0 0x7f8420134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8415d9b640 10 monclient(hunting): get_auth_request con 0x7f84201319a0 auth_method 0 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8415d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8415d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f8415d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.848+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201322b0 0x7f8420134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 10 monclient(hunting): get_auth_request con 0x7f84201322b0 auth_method 0 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 10 monclient(hunting): handle_auth_done global_id 104154 payload 824 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f84201319a0 msgr2=0x7f8420134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f84201319a0 0x7f8420134200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201322b0 msgr2=0x7f8420134740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201322b0 0x7f8420134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 10 monclient: _finish_hunting 0 2024-09-15T01:58:40.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 monclient: found mon.noname-a 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841e59c640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410009390 con 0x7f842009f1d0 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f84201319a0 0x7f8420134200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.849+0000 7f841ed9d640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84201322b0 0x7f8420134740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f841e59c640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 msgr2=0x7f8420138560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f841e59c640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 0x7f8420138560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8410009850 tx=0x7f8410032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410031c70 con 0x7f842009f1d0 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: dump: 2024-09-15T01:58:40.868 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.869 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8424c94640 5 monclient: authenticate success, global_id 104154 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410031dd0 con 0x7f842009f1d0 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8424c94640 10 monclient: _renew_subs 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841003d670 con 0x7f842009f1d0 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8424c94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84201378a0 con 0x7f842009f1d0 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f841d59a640 4 set_mon_vals no callback set 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8416ffd640 10 monclient: dump: 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.851+0000 7f8424c94640 10 monclient: _renew_subs 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8420138aa0 con 0x7f842009f1d0 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 0x7f8420134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8420063b00 con 0x7f842002aa40 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8410039040 con 0x7f842009f1d0 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 0x7f8420134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient: get_auth_request con 0x7f842002aa40 auth_method 0 2024-09-15T01:58:40.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.852+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f8416ffd640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8400038740 0x7f840003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f8415d9b640 10 monclient(hunting): handle_auth_done global_id 94301 payload 824 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f8415d9b640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842002ae10 con 0x7f842002aa40 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f841006c210 con 0x7f842009f1d0 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f841ed9d640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8400038740 0x7f840003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.853+0000 7f841ed9d640 10 monclient: get_auth_request con 0x7f8400038740 auth_method 0 2024-09-15T01:58:40.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8415d9b640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 msgr2=0x7f8420134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 0x7f8420134c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800fd50 con 0x7f842002aa40 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 10 monclient: dump: 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.874 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f840800feb0 con 0x7f842002aa40 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 msgr2=0x7f8420134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 0x7f8420134c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 msgr2=0x7f8420134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f8416ffd640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842002aa40 0x7f8420134c80 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:40.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.855+0000 7f841ed9d640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8400038740 0x7f840003abf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8418002ee0 tx=0x7f8418009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8424c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:40.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8424c94640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 0x7f842005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8424c94640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8420138cf0 con 0x7f842005e310 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 0x7f842005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient: get_auth_request con 0x7f842005e310 auth_method 0 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:40.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 10 monclient(hunting): handle_auth_done global_id 94304 payload 824 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.861+0000 7f8415d9b640 1 -- 172.21.15.17:0/1950563922 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f84200724a0 con 0x7f842005e310 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8415d9b640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 msgr2=0x7f842005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8415d9b640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 0x7f842005e710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408021950 con 0x7f842005e310 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 10 monclient: dump: 2024-09-15T01:58:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f8408021ab0 con 0x7f842005e310 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:40.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:40.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 msgr2=0x7f842005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 0x7f842005e710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408018000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 msgr2=0x7f842005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:40.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8416ffd640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f842005e310 0x7f842005e710 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8408005140 tx=0x7f8408018000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:58:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:40.896 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8400038740 msgr2=0x7f840003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8400038740 0x7f840003abf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8418002ee0 tx=0x7f8418009040 comp rx=0 tx=0).stop 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 10 monclient: shutdown 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 msgr2=0x7f8420138560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:40.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 --2- 172.21.15.17:0/1950563922 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f842009f1d0 0x7f8420138560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8410009850 tx=0x7f8410032000 comp rx=0 tx=0).stop 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f841ed9d640 1 -- 172.21.15.17:0/1950563922 reap_dead start 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f841ed9d640 1 -- 172.21.15.17:0/1950563922 reap_dead start 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 shutdown_connections 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 >> 172.21.15.17:0/1950563922 conn(0x7f84200a5000 msgr2=0x7f8420139ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 shutdown_connections 2024-09-15T01:58:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.864+0000 7f8424c94640 1 -- 172.21.15.17:0/1950563922 wait complete. 2024-09-15T01:58:40.898 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:40.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:58:40.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:58:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.993+0000 7f997359d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.993+0000 7f997359d640 10 monclient: build_initial_monmap 2024-09-15T01:58:40.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.993+0000 7f997359d640 10 monclient: monmap: 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:40.994349+0000 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:40.994349+0000 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:40.996 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:40.997 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:40.997 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 Processor -- start 2024-09-15T01:58:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 -- start start 2024-09-15T01:58:40.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient: init 2024-09-15T01:58:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:58:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f996c068f20 0x7f996c1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): picked mon.noname-c con 0x7f996c068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f996c0692f0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f996c136fe0 0x7f996c1393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): picked mon.noname-a con 0x7f996c136fe0 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996c0a1950 con 0x7f996c0692f0 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c0a1670 con 0x7f996c136fe0 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c0a0020 con 0x7f996c068f20 2024-09-15T01:58:41.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f997359d640 10 monclient(hunting): authenticate will time out at 1180.990830s 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f9971b3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f996c136fe0 0x7f996c1393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.994+0000 7f9971b3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f996c136fe0 0x7f996c1393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38552/0 (socket says 172.21.15.17:38552) 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 -- 172.21.15.17:0/1379524461 learned_addr learned my addr 172.21.15.17:0/1379524461 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): get_auth_request con 0x7f996c136fe0 auth_method 0 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971339640 1 --2- 172.21.15.17:0/1379524461 >> v2:172.21.15.67:3300/0 conn(0x7f996c068f20 0x7f996c1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient(hunting): handle_auth_done global_id 104157 payload 274 2024-09-15T01:58:41.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 -- 172.21.15.17:0/1379524461 >> v2:172.21.15.67:3300/0 conn(0x7f996c068f20 msgr2=0x7f996c1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:41.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 --2- 172.21.15.17:0/1379524461 >> v2:172.21.15.67:3300/0 conn(0x7f996c068f20 0x7f996c1321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 -- 172.21.15.17:0/1379524461 >> v1:172.21.15.67:6789/0 conn(0x7f996c0692f0 legacy=0x7f996c0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:41.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:58:41.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 monclient: found mon.noname-a 2024-09-15T01:58:41.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:41.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:41.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.995+0000 7f9971b3a640 1 -- 172.21.15.17:0/1379524461 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c0a1390 con 0x7f996c136fe0 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.998+0000 7f9971b3a640 1 -- 172.21.15.17:0/1379524461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 msgr2=0x7f996c1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:40.998+0000 7f9971b3a640 1 --2- 172.21.15.17:0/1379524461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c1393c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9964009f50 tx=0x7f9964009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8786c0c12198d159 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 1 -- 172.21.15.17:0/1379524461 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964007610 con 0x7f996c136fe0 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: dump: 2024-09-15T01:58:41.014 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: _finish_auth 0 2024-09-15T01:58:41.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 5 monclient: authenticate success, global_id 104157 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 1 -- 172.21.15.17:0/1379524461 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996402d050 con 0x7f996c136fe0 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 4 set_mon_vals no callback set 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 1 -- 172.21.15.17:0/1379524461 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99640079a0 con 0x7f996c136fe0 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f9963fff640 10 monclient: dump: 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.016 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.017 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 10 monclient: shutdown 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 1 -- 172.21.15.17:0/1379524461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 msgr2=0x7f996c1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 1 --2- 172.21.15.17:0/1379524461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c1393c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9964009f50 tx=0x7f9964009840 comp rx=0 tx=0).stop 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 1 -- 172.21.15.17:0/1379524461 shutdown_connections 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 1 --2- 172.21.15.17:0/1379524461 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c1393c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.000+0000 7f997359d640 1 --2- 172.21.15.17:0/1379524461 >> v2:172.21.15.67:3300/0 conn(0x7f996c068f20 0x7f996c1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 1 -- 172.21.15.17:0/1379524461 >> 172.21.15.17:0/1379524461 conn(0x7f996c070fb0 msgr2=0x7f996c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 1 -- 172.21.15.17:0/1379524461 shutdown_connections 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 1 -- 172.21.15.17:0/1379524461 wait complete. 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 10 monclient: build_initial_monmap 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 10 monclient: monmap: 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:41.019 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 1 Processor -- start 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 1 -- start start 2024-09-15T01:58:41.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.001+0000 7f997359d640 10 monclient: init 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 0x7f996c099db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): picked mon.noname-c con 0x7f996c068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c0692f0 0x7f996c09a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): picked mon.noname-b con 0x7f996c0692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c09a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): picked mon.noname-a con 0x7f996c136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0a0750 con 0x7f996c136fe0 2024-09-15T01:58:41.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c09c550 con 0x7f996c0692f0 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c11b9d0 con 0x7f996c068f20 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f997359d640 10 monclient(hunting): authenticate will time out at 1180.998476s 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 0x7f996c099db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c09a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 0x7f996c099db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33966/0 (socket says 172.21.15.17:33966) 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c09a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38560/0 (socket says 172.21.15.17:38560) 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 1 -- 172.21.15.17:0/359953067 learned_addr learned my addr 172.21.15.17:0/359953067 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): get_auth_request con 0x7f996c068f20 auth_method 0 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): get_auth_request con 0x7f996c136fe0 auth_method 0 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c0692f0 0x7f996c09a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): get_auth_request con 0x7f996c0692f0 auth_method 0 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9970b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient(hunting): handle_auth_done global_id 104160 payload 824 2024-09-15T01:58:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 msgr2=0x7f996c099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 0x7f996c099db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c0692f0 msgr2=0x7f996c09a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c0692f0 0x7f996c09a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient: _finish_hunting 0 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 monclient: found mon.noname-a 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:41.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964009ae0 con 0x7f996c136fe0 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.002+0000 7f9971339640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f996c068f20 0x7f996c099db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 msgr2=0x7f996c09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f9971b3a640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c09a830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9964033000 tx=0x7f9964036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964009520 con 0x7f996c136fe0 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: dump: 2024-09-15T01:58:41.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: _finish_auth 0 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9964032ce0 con 0x7f996c136fe0 2024-09-15T01:58:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996403b660 con 0x7f996c136fe0 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 5 monclient: authenticate success, global_id 104160 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 10 monclient: _renew_subs 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c09bca0 con 0x7f996c136fe0 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99637fe640 4 set_mon_vals no callback set 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f99617fa640 10 monclient: dump: 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 10 monclient: _renew_subs 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.005+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c11c970 con 0x7f996c136fe0 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f997359d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f997359d640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 0x7f996c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c063b90 con 0x7f996c09ad70 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9964038050 con 0x7f996c136fe0 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 0x7f996c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient: get_auth_request con 0x7f996c09ad70 auth_method 0 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f99617fa640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9950038740 0x7f995003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f996406d560 con 0x7f996c136fe0 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 10 monclient(hunting): handle_auth_done global_id 6217 payload 824 2024-09-15T01:58:41.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9971339640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f996c11cc30 con 0x7f996c09ad70 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9970b38640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9950038740 0x7f995003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.006+0000 7f9970b38640 10 monclient: get_auth_request con 0x7f9950038740 auth_method 0 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f9971339640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 msgr2=0x7f996c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f9971339640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 0x7f996c02aa40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f9968008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f9970b38640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9950038740 0x7f995003abf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f995c007fd0 tx=0x7f995c007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996800dee0 con 0x7f996c09ad70 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 10 monclient: dump: 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.033 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f996800c040 con 0x7f996c09ad70 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 msgr2=0x7f996c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 0x7f996c02aa40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f9968008040 comp rx=0 tx=0).stop 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 msgr2=0x7f996c02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.009+0000 7f99617fa640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c09ad70 0x7f996c02aa40 secure :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f9968008040 comp rx=0 tx=0).stop 2024-09-15T01:58:41.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.015+0000 7f997359d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:41.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.015+0000 7f997359d640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 0x7f996c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.015+0000 7f997359d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.015+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c11bb10 con 0x7f996c05e310 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 0x7f996c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient: get_auth_request con 0x7f996c05e310 auth_method 0 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 10 monclient(hunting): handle_auth_done global_id 6220 payload 824 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.016+0000 7f9971339640 1 -- 172.21.15.17:0/359953067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f996c0724a0 con 0x7f996c05e310 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f9971339640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 msgr2=0x7f996c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f9971339640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 0x7f996c05e710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f996801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996801be90 con 0x7f996c05e310 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f99617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f99617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.018+0000 7f99617fa640 10 monclient: dump: 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f9968005970 con 0x7f996c05e310 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 msgr2=0x7f996c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 0x7f996c05e710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f996801c000 comp rx=0 tx=0).stop 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 msgr2=0x7f996c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f99617fa640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f996c05e310 0x7f996c05e710 secure :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f996800fd40 tx=0x7f996801c000 comp rx=0 tx=0).stop 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:41.039 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 90, 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:41.040 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:41.041 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:41.042 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:41.043 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:41.044 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.045 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:41.046 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:41.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:41.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.050 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:58:41.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:41.052 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9950038740 msgr2=0x7f995003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9950038740 0x7f995003abf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f995c007fd0 tx=0x7f995c007e10 comp rx=0 tx=0).stop 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 10 monclient: shutdown 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 msgr2=0x7f996c09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 --2- 172.21.15.17:0/359953067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f996c136fe0 0x7f996c09a830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9964033000 tx=0x7f9964036000 comp rx=0 tx=0).stop 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 reap_dead start 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f9971b3a640 1 -- 172.21.15.17:0/359953067 reap_dead start 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 shutdown_connections 2024-09-15T01:58:41.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 >> 172.21.15.17:0/359953067 conn(0x7f996c070fb0 msgr2=0x7f996c09e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:41.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 shutdown_connections 2024-09-15T01:58:41.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.019+0000 7f997359d640 1 -- 172.21.15.17:0/359953067 wait complete. 2024-09-15T01:58:41.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:58:41.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T01:58:41.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:58:41.055 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T01:58:41.055 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T01:58:41.056 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T01:58:41.056 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:58:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed411e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:58:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed411e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:58:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ccc0 tx=0x7f0ecc005370 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ccc0 tx=0x7f0ecc005370 comp rx=0 tx=0).stop 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 msgr2=0x7f0ed411e0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ed40a3850 0x7f0ed411e0d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 0x7f0eb4043860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): picked mon.b con 0x7f0eb4041440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0eb4046290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed1d9b640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 msgr2=0x7f0eb4043860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed1d9b640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 0x7f0eb4043860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): picked mon.c con 0x7f0ed409e050 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0eb4048c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): picked mon.a con 0x7f0ed406a6e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4049610 con 0x7f0ed406a6e0 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4049750 con 0x7f0eb4041440 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4049910 con 0x7f0ed409e050 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (unknown 0 0 0) 0x7f0ecc01a360 con 0x7f0ed40a3850 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ec27fc640 10 monclient(hunting): discarding stray monitor message mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0eb4046290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): get_auth_request con 0x7f0ed409e050 auth_method 0 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient(hunting): handle_auth_done global_id 6202 payload 293 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 msgr2=0x7f0eb4043860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 0x7f0eb4043860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 msgr2=0x7f0eb4048c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0eb4048c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:58:41.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 monclient: found mon.c 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f0eb4049b90 con 0x7f0ed409e050 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed404bf60 con 0x7f0ed409e050 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 10 monclient: _finish_auth 0 2024-09-15T01:58:41.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.061+0000 7f0ed2d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.063+0000 7f0ed2d9d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 msgr2=0x7f0eb4046290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.063+0000 7f0ed2d9d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0eb4046290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012c80 tx=0x7f0ecc04a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc012cb0 con 0x7f0ed409e050 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 10 monclient: dump: 2024-09-15T01:58:41.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc012e10 con 0x7f0ed409e050 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec3fff640 4 set_mon_vals no callback set 2024-09-15T01:58:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.064+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ecc0104e0 con 0x7f0ed409e050 2024-09-15T01:58:41.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.069+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0ecc010ae0 con 0x7f0ed409e050 2024-09-15T01:58:41.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.069+0000 7f0ec27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.069+0000 7f0ec27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:41.072 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T01:58:41.072 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:58:41.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.165+0000 7f6e8859d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.165+0000 7f6e8859d640 10 monclient: build_initial_monmap 2024-09-15T01:58:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.165+0000 7f6e8859d640 10 monclient: monmap: 2024-09-15T01:58:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:41.167 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:41.166320+0000 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:41.166320+0000 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:41.168 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 Processor -- start 2024-09-15T01:58:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 -- start start 2024-09-15T01:58:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient: init 2024-09-15T01:58:41.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e8006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e8006f440 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 0x7f6e8009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8009e970 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8009ed90 con 0x7f6e8006f440 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8009f6a0 con 0x7f6e8009e970 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80069f40 con 0x7f6e8006eb30 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:41.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e8859d640 10 monclient(hunting): authenticate will time out at 1181.162766s 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86b3a640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 msgr2=0x7f6e8009c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86b3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 0x7f6e8009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e8006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e85b38640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6e8006f440 0x7f6e8009bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48878/0 (socket says 172.21.15.17:48878) 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e85b38640 1 -- 172.21.15.17:0/1141206906 learned_addr learned my addr 172.21.15.17:0/1141206906 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86339640 10 monclient(hunting): get_auth_request con 0x7f6e8006eb30 auth_method 0 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.166+0000 7f6e86339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient(hunting): handle_auth_done global_id 94310 payload 274 2024-09-15T01:58:41.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 1 -- 172.21.15.17:0/1141206906 >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 msgr2=0x7f6e8009c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 1 --2- 172.21.15.17:0/1141206906 >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 0x7f6e8009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 1 -- 172.21.15.17:0/1141206906 >> v1:172.21.15.17:6789/0 conn(0x7f6e8006f440 legacy=0x7f6e8009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient: _finish_hunting 0 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 1 monclient: found mon.noname-b 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.167+0000 7f6e86339640 1 -- 172.21.15.17:0/1141206906 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80069950 con 0x7f6e8006eb30 2024-09-15T01:58:41.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-15T01:58:41.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.169+0000 7f6e86339640 1 -- 172.21.15.17:0/1141206906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 msgr2=0x7f6e8006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.169+0000 7f6e86339640 1 --2- 172.21.15.17:0/1141206906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e8006ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e60 tx=0x7f6e70009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e85dd4a3aef2e23f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 1 -- 172.21.15.17:0/1141206906 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70007840 con 0x7f6e8006eb30 2024-09-15T01:58:41.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: dump: 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: _finish_auth 0 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 5 monclient: authenticate success, global_id 94310 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 1 -- 172.21.15.17:0/1141206906 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7002d050 con 0x7f6e8006eb30 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 4 set_mon_vals no callback set 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 1 -- 172.21.15.17:0/1141206906 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70007c10 con 0x7f6e8006eb30 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e85337640 10 monclient: dump: 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.181 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 10 monclient: shutdown 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 -- 172.21.15.17:0/1141206906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 msgr2=0x7f6e8006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1141206906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e8006ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e60 tx=0x7f6e70009780 comp rx=0 tx=0).stop 2024-09-15T01:58:41.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 -- 172.21.15.17:0/1141206906 shutdown_connections 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1141206906 >> v2:172.21.15.67:3300/0 conn(0x7f6e8009e970 0x7f6e8009c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1141206906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e8006ef00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 -- 172.21.15.17:0/1141206906 >> 172.21.15.17:0/1141206906 conn(0x7f6e800712f0 msgr2=0x7f6e800a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 -- 172.21.15.17:0/1141206906 shutdown_connections 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.170+0000 7f6e8859d640 1 -- 172.21.15.17:0/1141206906 wait complete. 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.171+0000 7f6e8859d640 10 monclient: build_initial_monmap 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.171+0000 7f6e8859d640 10 monclient: monmap: 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.183 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.171+0000 7f6e8859d640 1 Processor -- start 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.171+0000 7f6e8859d640 1 -- start start 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.171+0000 7f6e8859d640 10 monclient: init 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:41.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 0x7f6e801410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8006f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e8009e970 0x7f6e80141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8009e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80120570 con 0x7f6e8009e970 2024-09-15T01:58:41.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80120a60 con 0x7f6e8006f440 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80122280 con 0x7f6e8006eb30 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e8859d640 10 monclient(hunting): authenticate will time out at 1181.168251s 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e85b38640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 msgr2=0x7f6e801410f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e85b38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 0x7f6e801410f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33972/0 (socket says 172.21.15.17:33972) 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 -- 172.21.15.17:0/1358307110 learned_addr learned my addr 172.21.15.17:0/1358307110 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): get_auth_request con 0x7f6e8006eb30 auth_method 0 2024-09-15T01:58:41.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e8009e970 0x7f6e80141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): get_auth_request con 0x7f6e8009e970 auth_method 0 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient(hunting): handle_auth_done global_id 94313 payload 824 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 -- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 msgr2=0x7f6e801410f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 0x7f6e801410f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 -- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e8009e970 msgr2=0x7f6e80141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e8009e970 0x7f6e80141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient: _finish_hunting 0 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 monclient: found mon.noname-c 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:41.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.172+0000 7f6e86339640 1 -- 172.21.15.17:0/1358307110 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009a20 con 0x7f6e8006eb30 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.174+0000 7f6e86339640 1 -- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 msgr2=0x7f6e80133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.174+0000 7f6e86339640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6e700099f0 tx=0x7f6e70037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70031aa0 con 0x7f6e8006eb30 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: dump: 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70040070 con 0x7f6e8006eb30 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 5 monclient: authenticate success, global_id 94313 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 10 monclient: _renew_subs 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8009cd10 con 0x7f6e8006eb30 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70031e10 con 0x7f6e8006eb30 2024-09-15T01:58:41.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 10 monclient: dump: 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 10 monclient: _renew_subs 2024-09-15T01:58:41.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e84b36640 4 set_mon_vals no callback set 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e801343b0 con 0x7f6e8006eb30 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6e70042030 con 0x7f6e8006eb30 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.175+0000 7f6e8859d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e8859d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e8859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e80141b70 con 0x7f6e8006eb30 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e7e7fc640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 0x7f6e6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f6e7006dd40 con 0x7f6e8006eb30 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e85b38640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 0x7f6e6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:41.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.176+0000 7f6e85b38640 10 monclient: get_auth_request con 0x7f6e6c038740 auth_method 0 2024-09-15T01:58:41.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.178+0000 7f6e85b38640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 0x7f6e6c03abf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6e740047d0 tx=0x7f6e74007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:41.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.180+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6e7003c420 con 0x7f6e8006eb30 2024-09-15T01:58:41.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.180+0000 7f6e7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:41.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.180+0000 7f6e7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:41.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.243+0000 7f0ed8b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:41.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.243+0000 7f0ed8b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:41.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.243+0000 7f0ed8b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:41.243+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0ed4069940 con 0x7f0ed409e050 2024-09-15T01:58:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.407+0000 7f6e8859d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.407+0000 7f6e8859d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.407+0000 7f6e8859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.407+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e800724a0 con 0x7f6e8006eb30 2024-09-15T01:58:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.407+0000 7f6e7e7fc640 1 -- 172.21.15.17:0/1358307110 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f6e7002b0a0 con 0x7f6e8006eb30 2024-09-15T01:58:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.408+0000 7f6e7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.408+0000 7f6e7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:41.410 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 msgr2=0x7f6e6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 0x7f6e6c03abf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6e740047d0 tx=0x7f6e74007b80 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 10 monclient: shutdown 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 msgr2=0x7f6e80133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6e700099f0 tx=0x7f6e70037000 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 shutdown_connections 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e6c038740 0x7f6e6c03abf0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e8009e970 0x7f6e80141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8006f440 0x7f6e801410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 --2- 172.21.15.17:0/1358307110 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e8006eb30 0x7f6e80133e70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 >> 172.21.15.17:0/1358307110 conn(0x7f6e800712f0 msgr2=0x7f6e8009c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 shutdown_connections 2024-09-15T01:58:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:41.409+0000 7f6e8859d640 1 -- 172.21.15.17:0/1358307110 wait complete. 2024-09-15T01:58:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:43.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:43.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:43.396+0000 7f0ec17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:44.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:44.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:44.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:44.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:44.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.530+0000 7f3f2ab9d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.530+0000 7f3f2ab9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.530+0000 7f3f2ab9d640 10 monclient: monmap: 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:44.531369+0000 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:44.531369+0000 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:44.534 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 1 Processor -- start 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 1 -- start start 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 10 monclient: init 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 0x7f3f2409c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f2409c620 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.531+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f2406ed40 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f241347d0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2409bbd0 con 0x7f3f2406ed40 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2409b570 con 0x7f3f241347d0 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2409f750 con 0x7f3f2409c620 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2ab9d640 10 monclient(hunting): authenticate will time out at 1184.528071s 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f2913a640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f3f241347d0 legacy=0x7f3f24136bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f23fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3f2406ed40 0x7f3f2406f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:49734/0 (socket says 172.21.15.17:49734) 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 0x7f3f2409c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f23fff640 1 -- 172.21.15.17:0/881575162 learned_addr learned my addr 172.21.15.17:0/881575162 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): get_auth_request con 0x7f3f2409c620 auth_method 0 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient(hunting): handle_auth_done global_id 94316 payload 274 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 1 -- 172.21.15.17:0/881575162 >> v1:172.21.15.67:6789/0 conn(0x7f3f241347d0 legacy=0x7f3f24136bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 1 -- 172.21.15.17:0/881575162 >> v1:172.21.15.17:6789/0 conn(0x7f3f2406ed40 legacy=0x7f3f2406f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient: _finish_hunting 0 2024-09-15T01:58:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 1 monclient: found mon.noname-b 2024-09-15T01:58:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.532+0000 7f3f28939640 1 -- 172.21.15.17:0/881575162 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2409a680 con 0x7f3f2409c620 2024-09-15T01:58:44.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f28939640 1 -- 172.21.15.17:0/881575162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 msgr2=0x7f3f2409c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:44.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f28939640 1 --2- 172.21.15.17:0/881575162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 0x7f3f2409c9f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009880 tx=0x7f3f180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78fa1638f1ab5bd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:44.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 1 -- 172.21.15.17:0/881575162 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18032440 con 0x7f3f2409c620 2024-09-15T01:58:44.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:44.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: dump: 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:44.546 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 1 -- 172.21.15.17:0/881575162 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f180325a0 con 0x7f3f2409c620 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 5 monclient: authenticate success, global_id 94316 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 1 -- 172.21.15.17:0/881575162 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18032940 con 0x7f3f2409c620 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 4 set_mon_vals no callback set 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f237fe640 10 monclient: dump: 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:44.548 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 10 monclient: shutdown 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/881575162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 msgr2=0x7f3f2409c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.535+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/881575162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 0x7f3f2409c9f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009880 tx=0x7f3f180090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/881575162 shutdown_connections 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/881575162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2409c620 0x7f3f2409c9f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/881575162 >> 172.21.15.17:0/881575162 conn(0x7f3f24071300 msgr2=0x7f3f240a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/881575162 shutdown_connections 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/881575162 wait complete. 2024-09-15T01:58:44.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 10 monclient: monmap: 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:44.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.536+0000 7f3f2ab9d640 1 Processor -- start 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 -- start start 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient: init 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f2406ed40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 0x7f3f24119b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f2409c620 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:44.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 0x7f3f2411a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f241347d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2411c450 con 0x7f3f2409c620 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2409cf90 con 0x7f3f241347d0 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240a1900 con 0x7f3f2406ed40 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2ab9d640 10 monclient(hunting): authenticate will time out at 1184.533644s 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2913a640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 msgr2=0x7f3f2411a0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:44.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 0x7f3f24119b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f2913a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 0x7f3f2411a0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41230/0 (socket says 172.21.15.17:41230) 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 1 -- 172.21.15.17:0/1645177334 learned_addr learned my addr 172.21.15.17:0/1645177334 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): get_auth_request con 0x7f3f2406ed40 auth_method 0 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): get_auth_request con 0x7f3f2409c620 auth_method 0 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:44.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.537+0000 7f3f28939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 10 monclient(hunting): handle_auth_done global_id 104166 payload 824 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 -- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 msgr2=0x7f3f24119640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 -- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 msgr2=0x7f3f2411a0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:44.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 0x7f3f2411a0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 monclient: found mon.noname-a 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f23fff640 1 -- 172.21.15.17:0/1645177334 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18009390 con 0x7f3f2409c620 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.538+0000 7f3f28939640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f23fff640 1 -- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 msgr2=0x7f3f24119b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f23fff640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 0x7f3f24119b80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000c4c0 tx=0x7f3f10007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10005cb0 con 0x7f3f2409c620 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: dump: 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:44.556 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10005e30 con 0x7f3f2409c620 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10007320 con 0x7f3f2409c620 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 5 monclient: authenticate success, global_id 104166 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 10 monclient: _renew_subs 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2411b730 con 0x7f3f2409c620 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f22ffd640 4 set_mon_vals no callback set 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:44.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f20ff9640 10 monclient: dump: 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 10 monclient: _renew_subs 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.540+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2411ccf0 con 0x7f3f2409c620 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.541+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f1001b3b0 con 0x7f3f2409c620 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.541+0000 7f3f2ab9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.541+0000 7f3f2ab9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.541+0000 7f3f2ab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.541+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f241345c0 con 0x7f3f2409c620 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.542+0000 7f3f20ff9640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 0x7f3f0803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.542+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f3f1004bae0 con 0x7f3f2409c620 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.542+0000 7f3f28939640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 0x7f3f0803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:44.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.542+0000 7f3f28939640 10 monclient: get_auth_request con 0x7f3f080386f0 auth_method 0 2024-09-15T01:58:44.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.545+0000 7f3f28939640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 0x7f3f0803aba0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3f18035040 tx=0x7f3f18032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:44.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.546+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f1004de60 con 0x7f3f2409c620 2024-09-15T01:58:44.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.546+0000 7f3f20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:44.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.546+0000 7f3f20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:44.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f2ab9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:44.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f2ab9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:44.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f2ab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:44.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f240724a0 con 0x7f3f2409c620 2024-09-15T01:58:44.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f20ff9640 1 -- 172.21.15.17:0/1645177334 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f3f10017080 con 0x7f3f2409c620 2024-09-15T01:58:44.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:44.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.772+0000 7f3f20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:44.775 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:44.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 msgr2=0x7f3f0803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:44.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 0x7f3f0803aba0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3f18035040 tx=0x7f3f18032000 comp rx=0 tx=0).stop 2024-09-15T01:58:44.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 10 monclient: shutdown 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 msgr2=0x7f3f24119b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 0x7f3f24119b80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000c4c0 tx=0x7f3f10007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 shutdown_connections 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f080386f0 0x7f3f0803aba0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f2409c620 0x7f3f24119b80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f241347d0 0x7f3f2411a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 --2- 172.21.15.17:0/1645177334 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f2406ed40 0x7f3f24119640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 >> 172.21.15.17:0/1645177334 conn(0x7f3f24071300 msgr2=0x7f3f24098e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 shutdown_connections 2024-09-15T01:58:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:44.774+0000 7f3f2ab9d640 1 -- 172.21.15.17:0/1645177334 wait complete. 2024-09-15T01:58:45.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:45.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:45.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:45.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:46.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:46.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:46.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:46.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:47.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-15T01:58:47.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:47.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:47.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:47.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:47.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 10 monclient: get_monmap_and_config 2024-09-15T01:58:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 10 monclient: build_initial_monmap 2024-09-15T01:58:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 10 monclient: monmap: 2024-09-15T01:58:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:47.895016+0000 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:47.895016+0000 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:47.897 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 1 Processor -- start 2024-09-15T01:58:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 1 -- start start 2024-09-15T01:58:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.894+0000 7f922d894640 10 monclient: init 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 0x7f9228132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): picked mon.noname-c con 0x7f92280a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): picked mon.noname-b con 0x7f92280980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): picked mon.noname-a con 0x7f92280989e0 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280a01e0 con 0x7f92280989e0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280a1580 con 0x7f92280a58b0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92280a5c80 con 0x7f92280980a0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f922d894640 10 monclient(hunting): authenticate will time out at 1187.891388s 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f9226ffd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 msgr2=0x7f9228132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f9226ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 0x7f9228132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41236/0 (socket says 172.21.15.17:41236) 2024-09-15T01:58:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 -- 172.21.15.17:0/1794084204 learned_addr learned my addr 172.21.15.17:0/1794084204 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): get_auth_request con 0x7f92280980a0 auth_method 0 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57276/0 (socket says 172.21.15.17:57276) 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): get_auth_request con 0x7f92280989e0 auth_method 0 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient(hunting): handle_auth_done global_id 94322 payload 274 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 -- 172.21.15.17:0/1794084204 >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 msgr2=0x7f9228132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 0x7f9228132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 -- 172.21.15.17:0/1794084204 >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 msgr2=0x7f9228068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient: _finish_hunting 0 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 monclient: found mon.noname-b 2024-09-15T01:58:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92267fc640 1 -- 172.21.15.17:0/1794084204 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92281326c0 con 0x7f92280980a0 2024-09-15T01:58:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.895+0000 7f92277fe640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:47.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f92267fc640 1 -- 172.21.15.17:0/1794084204 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 msgr2=0x7f92280984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:47.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f92267fc640 1 --2- 172.21.15.17:0/1794084204 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f921c009910 tx=0x7f921c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec52da92b5e053f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:47.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 1 -- 172.21.15.17:0/1794084204 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c032440 con 0x7f92280980a0 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 10 monclient: dump: 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:47.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.898+0000 7f9225ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 5 monclient: authenticate success, global_id 94322 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 1 -- 172.21.15.17:0/1794084204 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c0325a0 con 0x7f92280980a0 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 1 -- 172.21.15.17:0/1794084204 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c032940 con 0x7f92280980a0 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 4 set_mon_vals no callback set 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:47.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f9225ffb640 10 monclient: dump: 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 10 monclient: shutdown 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:47.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 -- 172.21.15.17:0/1794084204 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 msgr2=0x7f92280984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 --2- 172.21.15.17:0/1794084204 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f921c009910 tx=0x7f921c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 -- 172.21.15.17:0/1794084204 shutdown_connections 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.17:3300/0 conn(0x7f92280989e0 0x7f9228068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 --2- 172.21.15.17:0/1794084204 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280980a0 0x7f92280984a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 --2- 172.21.15.17:0/1794084204 >> v2:172.21.15.67:3300/0 conn(0x7f92280a58b0 0x7f9228132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 -- 172.21.15.17:0/1794084204 >> 172.21.15.17:0/1794084204 conn(0x7f9228070ed0 msgr2=0x7f92280a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 -- 172.21.15.17:0/1794084204 shutdown_connections 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.899+0000 7f922d894640 1 -- 172.21.15.17:0/1794084204 wait complete. 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.900+0000 7f922d894640 10 monclient: build_initial_monmap 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.900+0000 7f922d894640 10 monclient: monmap: 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.900+0000 7f922d894640 1 Processor -- start 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.900+0000 7f922d894640 1 -- start start 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.900+0000 7f922d894640 10 monclient: init 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:47.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 0x7f9228134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): picked mon.noname-b con 0x7f92280980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): picked mon.noname-a con 0x7f92280a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f922811bcb0 0x7f9228141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): picked mon.noname-c con 0x7f922811bcb0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922809eaa0 con 0x7f92280a58b0 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922811dd50 con 0x7f92280980a0 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922811e030 con 0x7f922811bcb0 2024-09-15T01:58:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f922d894640 10 monclient(hunting): authenticate will time out at 1187.897253s 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f9226ffd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 msgr2=0x7f9228134c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57288/0 (socket says 172.21.15.17:57288) 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f9226ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 0x7f9228134c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f922811bcb0 0x7f9228141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 -- 172.21.15.17:0/3740108538 learned_addr learned my addr 172.21.15.17:0/3740108538 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): get_auth_request con 0x7f92280a58b0 auth_method 0 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): get_auth_request con 0x7f922811bcb0 auth_method 0 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient(hunting): handle_auth_done global_id 104172 payload 824 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 -- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f922811bcb0 msgr2=0x7f9228141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f922811bcb0 0x7f9228141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 -- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 msgr2=0x7f9228134c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 0x7f9228134c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient: _finish_hunting 0 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 monclient: found mon.noname-a 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.901+0000 7f92267fc640 1 -- 172.21.15.17:0/3740108538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220009390 con 0x7f92280a58b0 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f92267fc640 1 -- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 msgr2=0x7f9228130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f92267fc640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f921c035040 tx=0x7f921c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c03b820 con 0x7f92280a58b0 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: dump: 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:47.918 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: _finish_auth 0 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 5 monclient: authenticate success, global_id 104172 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 10 monclient: _renew_subs 2024-09-15T01:58:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c009450 con 0x7f92280a58b0 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c03b9a0 con 0x7f92280a58b0 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c03bd60 con 0x7f92280a58b0 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f920effd640 10 monclient: dump: 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 10 monclient: _renew_subs 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922811cdf0 con 0x7f92280a58b0 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.904+0000 7f92257fa640 4 set_mon_vals no callback set 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f921c039040 con 0x7f92280a58b0 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f922d894640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f922d894640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:47.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f922d894640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:47.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f920effd640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 0x7f921403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:47.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92280724a0 con 0x7f92280a58b0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f921c06c0d0 con 0x7f92280a58b0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f9226ffd640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 0x7f921403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.905+0000 7f9226ffd640 10 monclient: get_auth_request con 0x7f92140386f0 auth_method 0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.908+0000 7f9226ffd640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 0x7f921403aba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9218003c70 tx=0x7f9218009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.910+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f921c039b20 con 0x7f92280a58b0 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.910+0000 7f920effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:47.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:47.910+0000 7f920effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.136+0000 7f922d894640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.136+0000 7f922d894640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.136+0000 7f922d894640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.136+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f922809fc90 con 0x7f92280a58b0 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.137+0000 7f920effd640 1 -- 172.21.15.17:0/3740108538 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f921c03f020 con 0x7f92280a58b0 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.137+0000 7f920effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:48.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.137+0000 7f920effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:48.140 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 msgr2=0x7f921403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 0x7f921403aba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9218003c70 tx=0x7f9218009310 comp rx=0 tx=0).stop 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 10 monclient: shutdown 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 msgr2=0x7f9228130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f921c035040 tx=0x7f921c032000 comp rx=0 tx=0).stop 2024-09-15T01:58:48.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 shutdown_connections 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f92140386f0 0x7f921403aba0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92280a58b0 0x7f9228130c30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f922811bcb0 0x7f9228141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 --2- 172.21.15.17:0/3740108538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92280980a0 0x7f9228134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 >> 172.21.15.17:0/3740108538 conn(0x7f9228070ed0 msgr2=0x7f922809eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 shutdown_connections 2024-09-15T01:58:48.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:48.139+0000 7f922d894640 1 -- 172.21.15.17:0/3740108538 wait complete. 2024-09-15T01:58:48.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:48.395+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:48.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:48.395+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:49.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:49.395+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:49.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:49.395+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:50.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:50.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:50.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:50.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:51.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.258+0000 7f4449d14640 10 monclient: get_monmap_and_config 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.258+0000 7f4449d14640 10 monclient: build_initial_monmap 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.258+0000 7f4449d14640 10 monclient: monmap: 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:51.259179+0000 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:51.259179+0000 2024-09-15T01:58:51.260 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:51.261 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 Processor -- start 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 -- start start 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient: init 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-b con 0x7f44440a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f444409ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4444069c90 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:51.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44440a5b50 con 0x7f444409ccf0 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444409c5f0 con 0x7f4444069c90 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44440a18b0 con 0x7f44440a5780 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.259+0000 7f4449d14640 10 monclient(hunting): authenticate will time out at 1191.255965s 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f4443fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f4444069c90 legacy=0x7f4444139880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f443bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f444409ccf0 0x7f4444069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:49748/0 (socket says 172.21.15.17:49748) 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 learned_addr learned my addr 172.21.15.17:0/898208896 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f4442ffd640 1 -- 172.21.15.17:0/898208896 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f44440a5780 auth_method 0 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:51.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.260+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:51.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.262+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.17:6789/0 conn(0x7f444409ccf0 legacy=0x7f4444069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:51.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:51.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:51.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:51.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f4443fff640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.67:6789/0 conn(0x7f4444069c90 legacy=0x7f4444139880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f44440a5780 auth_method 0 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:51.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:58:51.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:51.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:51.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:58:51.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f4443fff640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.67:6789/0 conn(0x7f4444069c90 legacy=0x7f4444139880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:51.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:51.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f44440a5780 auth_method 0 2024-09-15T01:58:51.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:51.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:58:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:51.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:58:52.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:52.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:52.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:52.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:52.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:52.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f4443fff640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.67:6789/0 conn(0x7f4444069c90 legacy=0x7f4444139880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:52.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f44440a5780 auth_method 0 2024-09-15T01:58:52.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:52.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:52.663+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:58:53.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:53.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:53.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:53.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:53.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:53.396+0000 7f0ec17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): tick 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 msgr2=0x7f444409bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f44440a5780 0x7f444409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.67:6789/0 conn(0x7f4444069c90 legacy=0x7f4444139880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 -- 172.21.15.17:0/898208896 >> v1:172.21.15.17:6789/0 conn(0x7f444409ccf0 legacy=0x7f4444069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4424003710 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f4424006270 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 -- 172.21.15.17:0/898208896 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44440a18b0 con 0x7f4424003710 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 1 -- 172.21.15.17:0/898208896 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4424008de0 con 0x7f4424006270 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f4441ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f4424003710 auth_method 0 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 msgr2=0x7f4424008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f443bfff640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:54.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:58:54.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:54.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:54.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.260+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:54.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:54.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:54.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:54.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:54.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 msgr2=0x7f4424008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:54.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f443bfff640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:58:54.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:54.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f4424003710 auth_method 0 2024-09-15T01:58:54.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:54.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.461+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:58:54.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 msgr2=0x7f4424008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:54.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:54.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f443bfff640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:58:54.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f4424003710 auth_method 0 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 msgr2=0x7f4424005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:54.862+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:58:55.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:55.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:55.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:55.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f443bfff640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 msgr2=0x7f4424008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f443bfff640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.17:3300/0 conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): get_auth_request con 0x7f4424003710 auth_method 0 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient(hunting): handle_auth_done global_id 104175 payload 274 2024-09-15T01:58:55.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 msgr2=0x7f4424008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> v2:172.21.15.67:3300/0 conn(0x7f4424006270 0x7f4424008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 1 monclient: found mon.noname-a 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:55.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.662+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444409d210 con 0x7f4424003710 2024-09-15T01:58:55.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.665+0000 7f44437fe640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4424003710 msgr2=0x7f4424005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:55.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.665+0000 7f44437fe640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4424003710 0x7f4424005b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4424006240 tx=0x7f4434008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607394068629c88f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 1 -- 172.21.15.17:0/898208896 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443402cb70 con 0x7f4424003710 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: dump: 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 1 -- 172.21.15.17:0/898208896 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434030070 con 0x7f4424003710 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 1 -- 172.21.15.17:0/898208896 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434036400 con 0x7f4424003710 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 5 monclient: authenticate success, global_id 104175 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 4 set_mon_vals no callback set 2024-09-15T01:58:55.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4442ffd640 10 monclient: dump: 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:55.671 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:55.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 10 monclient: shutdown 2024-09-15T01:58:55.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:55.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 1 -- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4424003710 msgr2=0x7f4424005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:55.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4424003710 0x7f4424005b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4424006240 tx=0x7f4434008040 comp rx=0 tx=0).stop 2024-09-15T01:58:55.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4443fff640 1 -- 172.21.15.17:0/898208896 reap_dead start 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 1 -- 172.21.15.17:0/898208896 shutdown_connections 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 1 --2- 172.21.15.17:0/898208896 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4424003710 0x7f4424005b80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.666+0000 7f4449d14640 1 -- 172.21.15.17:0/898208896 >> 172.21.15.17:0/898208896 conn(0x7f4444070e10 msgr2=0x7f44440a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.667+0000 7f4449d14640 1 -- 172.21.15.17:0/898208896 shutdown_connections 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.667+0000 7f4449d14640 1 -- 172.21.15.17:0/898208896 wait complete. 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.667+0000 7f4449d14640 10 monclient: build_initial_monmap 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.667+0000 7f4449d14640 10 monclient: monmap: 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:55.677 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 Processor -- start 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 -- start start 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient: init 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:58:55.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 0x7f444413dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-b con 0x7f4424003710 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-a con 0x7f4444121550 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 0x7f4444141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): picked mon.noname-c con 0x7f444413e650 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4444122cc0 con 0x7f4444121550 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4444122700 con 0x7f4424003710 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44441201e0 con 0x7f444413e650 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4449d14640 10 monclient(hunting): authenticate will time out at 1195.665769s 2024-09-15T01:58:55.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f44437fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 msgr2=0x7f444413dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f44437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 0x7f444413dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443b7fe640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51052/0 (socket says 172.21.15.17:51052) 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4443fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 0x7f4444141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 1 -- 172.21.15.17:0/4185159044 learned_addr learned my addr 172.21.15.17:0/4185159044 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 10 monclient(hunting): get_auth_request con 0x7f4444121550 auth_method 0 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f443bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4443fff640 10 monclient(hunting): get_auth_request con 0x7f444413e650 auth_method 0 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4443fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:55.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4443fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.668+0000 7f4443fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f443bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f443bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f443bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient(hunting): handle_auth_done global_id 94328 payload 824 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 -- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 msgr2=0x7f444413dcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 0x7f444413dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 -- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 msgr2=0x7f444413d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 monclient: found mon.noname-c 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f4443fff640 1 -- 172.21.15.17:0/4185159044 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443400e040 con 0x7f444413e650 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.669+0000 7f443bfff640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.671+0000 7f4443fff640 1 -- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 msgr2=0x7f4444141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.671+0000 7f4443fff640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 0x7f4444141240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f444406a5b0 tx=0x7f443c0093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00bec0 con 0x7f444413e650 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: dump: 2024-09-15T01:58:55.682 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c00c040 con 0x7f444413e650 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 5 monclient: authenticate success, global_id 94328 2024-09-15T01:58:55.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 10 monclient: _renew_subs 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4444122420 con 0x7f444413e650 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00d680 con 0x7f444413e650 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f443b7fe640 10 monclient: dump: 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4441ffb640 4 set_mon_vals no callback set 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 10 monclient: _renew_subs 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.672+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444413d8c0 con 0x7f444413e650 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f443c00dab0 con 0x7f444413e650 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f443b7fe640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 0x7f442403b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f443c04bab0 con 0x7f444413e650 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f4449d14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f4449d14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:55.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f4449d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44440724a0 con 0x7f444413e650 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f44437fe640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 0x7f442403b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.673+0000 7f44437fe640 10 monclient: get_auth_request con 0x7f4424039450 auth_method 0 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.676+0000 7f44437fe640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 0x7f442403b900 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4434034000 tx=0x7f4434038000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.678+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f443c0103c0 con 0x7f444413e650 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.678+0000 7f443b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:55.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.678+0000 7f443b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f4449d14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:58:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f4449d14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f4449d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f444406f4f0 con 0x7f444413e650 2024-09-15T01:58:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f443b7fe640 1 -- 172.21.15.17:0/4185159044 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1521 (secure 0 0 0) 0x7f443c00f320 con 0x7f444413e650 2024-09-15T01:58:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f443b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:58:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.911+0000 7f443b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:55.915 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 msgr2=0x7f442403b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 0x7f442403b900 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4434034000 tx=0x7f4434038000 comp rx=0 tx=0).stop 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 10 monclient: shutdown 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 msgr2=0x7f4444141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 0x7f4444141240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f444406a5b0 tx=0x7f443c0093f0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 shutdown_connections 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4424039450 0x7f442403b900 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4444121550 0x7f444413d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f444413e650 0x7f4444141240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 --2- 172.21.15.17:0/4185159044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4424003710 0x7f444413dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.913+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 >> 172.21.15.17:0/4185159044 conn(0x7f4444070e10 msgr2=0x7f444409ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.914+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 shutdown_connections 2024-09-15T01:58:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:55.914+0000 7f4449d14640 1 -- 172.21.15.17:0/4185159044 wait complete. 2024-09-15T01:58:55.927 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:58:55.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:58:55.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:58:56.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.022+0000 7f81c6eef640 10 monclient: get_monmap_and_config 2024-09-15T01:58:56.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.022+0000 7f81c6eef640 10 monclient: build_initial_monmap 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.022+0000 7f81c6eef640 10 monclient: monmap: 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:56.024465+0000 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:56.024465+0000 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:56.026 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 Processor -- start 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 -- start start 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient: init 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 0x7f81c009c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-c con 0x7f81c009c300 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:56.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f81c009a920 0x7f81c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-a con 0x7f81c009a920 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-b con 0x7f81c006f070 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c006a300 con 0x7f81c009a920 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c0099f30 con 0x7f81c009c300 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c006a5c0 con 0x7f81c006f070 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.023+0000 7f81c6eef640 10 monclient(hunting): authenticate will time out at 1196.020966s 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c4c8b640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 msgr2=0x7f81c009c6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c4c8b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 0x7f81c009c6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:56.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f81c009a920 0x7f81c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f81c009a920 0x7f81c006eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51058/0 (socket says 172.21.15.17:51058) 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33054/0 (socket says 172.21.15.17:33054) 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 -- 172.21.15.17:0/2103730494 learned_addr learned my addr 172.21.15.17:0/2103730494 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): get_auth_request con 0x7f81c009a920 auth_method 0 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): get_auth_request con 0x7f81c006f070 auth_method 0 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient(hunting): handle_auth_done global_id 104181 payload 274 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 -- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 msgr2=0x7f81c009d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 -- 172.21.15.17:0/2103730494 >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 msgr2=0x7f81c009c6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 --2- 172.21.15.17:0/2103730494 >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 0x7f81c009c6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient: _finish_hunting 0 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 monclient: found mon.noname-a 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:56.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81bffff640 1 -- 172.21.15.17:0/2103730494 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c009cc10 con 0x7f81c009a920 2024-09-15T01:58:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.024+0000 7f81c548c640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bffff640 1 -- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c009a920 msgr2=0x7f81c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bffff640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c009a920 0x7f81c006eb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81c0060cf0 tx=0x7f81ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a520a85774c2c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 1 -- 172.21.15.17:0/2103730494 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac032440 con 0x7f81c009a920 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: dump: 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 5 monclient: authenticate success, global_id 104181 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 1 -- 172.21.15.17:0/2103730494 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac0325a0 con 0x7f81c009a920 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 1 -- 172.21.15.17:0/2103730494 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac032940 con 0x7f81c009a920 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 4 set_mon_vals no callback set 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81bf7fe640 10 monclient: dump: 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 10 monclient: shutdown 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 1 -- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c009a920 msgr2=0x7f81c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.027+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c009a920 0x7f81c006eb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81c0060cf0 tx=0x7f81ac0091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 -- 172.21.15.17:0/2103730494 shutdown_connections 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c006f070 0x7f81c009d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2103730494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c009a920 0x7f81c006eb30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2103730494 >> v2:172.21.15.67:3300/0 conn(0x7f81c009c300 0x7f81c009c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 -- 172.21.15.17:0/2103730494 >> 172.21.15.17:0/2103730494 conn(0x7f81c00a2f70 msgr2=0x7f81c00a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:56.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 -- 172.21.15.17:0/2103730494 shutdown_connections 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 -- 172.21.15.17:0/2103730494 wait complete. 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 10 monclient: build_initial_monmap 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 10 monclient: monmap: 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.028+0000 7f81c6eef640 1 Processor -- start 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 -- start start 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient: init 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 0x7f81c0141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-a con 0x7f81c006f070 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 0x7f81c0131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-c con 0x7f81c009a920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c009c300 0x7f81c006c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): picked mon.noname-b con 0x7f81c009c300 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c011f880 con 0x7f81c006f070 2024-09-15T01:58:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 0x7f81c0131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 0x7f81c0141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 0x7f81c0131e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33064/0 (socket says 172.21.15.17:33064) 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 learned_addr learned my addr 172.21.15.17:0/2998545559 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c009cdf0 con 0x7f81c009c300 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00a2ba0 con 0x7f81c009a920 2024-09-15T01:58:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c6eef640 10 monclient(hunting): authenticate will time out at 1196.026729s 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): get_auth_request con 0x7f81c009a920 auth_method 0 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): get_auth_request con 0x7f81c006f070 auth_method 0 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c548c640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c009c300 msgr2=0x7f81c006c960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c548c640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c009c300 0x7f81c006c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.029+0000 7f81c4c8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 10 monclient(hunting): handle_auth_done global_id 104184 payload 824 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 msgr2=0x7f81c0131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 0x7f81c0131e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c009c300 msgr2=0x7f81c006c960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c009c300 0x7f81c006c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 10 monclient: _finish_hunting 0 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 monclient: found mon.noname-a 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81c4c8b640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8009a20 con 0x7f81c006f070 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.030+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81c009a920 0x7f81c0131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c4c8b640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 msgr2=0x7f81c0141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c4c8b640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 0x7f81c0141260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f81b400b5e0 tx=0x7f81b40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4003d90 con 0x7f81c006f070 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: dump: 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.047 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b4006150 con 0x7f81c006f070 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 5 monclient: authenticate success, global_id 104184 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 10 monclient: _renew_subs 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac009450 con 0x7f81c006f070 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:56.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4006510 con 0x7f81c006f070 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81bcff9640 10 monclient: dump: 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 10 monclient: _renew_subs 2024-09-15T01:58:56.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81beffd640 4 set_mon_vals no callback set 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.032+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c006e540 con 0x7f81c006f070 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c6eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81b40103e0 con 0x7f81c006f070 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 0x7f81c006cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0063b90 con 0x7f81c0132370 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 0x7f81c006cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient: get_auth_request con 0x7f81c0132370 auth_method 0 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bcff9640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81a40386f0 0x7f81a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(180..180 src has 148..180) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f81b404b880 con 0x7f81c006f070 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 10 monclient(hunting): handle_auth_done global_id 104187 payload 824 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81c011f630 con 0x7f81c0132370 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c548c640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81a40386f0 0x7f81a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.033+0000 7f81c548c640 10 monclient: get_auth_request con 0x7f81a40386f0 auth_method 0 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.035+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 msgr2=0x7f81c006cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.035+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 0x7f81c006cea0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009420 tx=0x7f81ac032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac032440 con 0x7f81c0132370 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 10 monclient: dump: 2024-09-15T01:58:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f81ac031e70 con 0x7f81c0132370 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 msgr2=0x7f81c006cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 0x7f81c006cea0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009420 tx=0x7f81ac032000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 msgr2=0x7f81c006cea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81bcff9640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c0132370 0x7f81c006cea0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009420 tx=0x7f81ac032000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.036+0000 7f81c548c640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81a40386f0 0x7f81a403aba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f81c011ddf0 tx=0x7f81b800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81c6eef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81c6eef640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0132740 con 0x7f81c006d3e0 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 10 monclient: get_auth_request con 0x7f81c006d3e0 auth_method 0 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 msgr2=0x7f81c0096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:58:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 msgr2=0x7f81c0096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:58:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:58:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.042+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:56.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.206+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 5 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v181) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0ecc034050 con 0x7f0ed409e050 2024-09-15T01:58:56.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.206+0000 7f0ec27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:56.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.206+0000 7f0ec27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient: get_auth_request con 0x7f81c006d3e0 auth_method 0 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 10 monclient(hunting): handle_auth_done global_id 104193 payload 824 2024-09-15T01:58:56.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.243+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81c011e060 con 0x7f81c006d3e0 2024-09-15T01:58:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bffff640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 msgr2=0x7f81c0096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bffff640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81ac043000 tx=0x7f81ac03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac046c10 con 0x7f81c006d3e0 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 10 monclient: dump: 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.249 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4386 (secure 0 0 0) 0x7f81ac046d70 con 0x7f81c006d3e0 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 msgr2=0x7f81c0096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81ac043000 tx=0x7f81ac03d000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 msgr2=0x7f81c0096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.245+0000 7f81bcff9640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006d3e0 0x7f81c0096a10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.250 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 94, 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:56.251 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.252 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:56.254 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.255 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:56.256 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.258 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T01:58:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:56.264 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:56.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81a40386f0 msgr2=0x7f81a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81a40386f0 0x7f81a403aba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f81c011ddf0 tx=0x7f81b800b040 comp rx=0 tx=0).stop 2024-09-15T01:58:56.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 10 monclient: shutdown 2024-09-15T01:58:56.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 msgr2=0x7f81c0141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 --2- 172.21.15.17:0/2998545559 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81c006f070 0x7f81c0141260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f81b400b5e0 tx=0x7f81b40090f0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c548c640 1 -- 172.21.15.17:0/2998545559 reap_dead start 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c548c640 1 -- 172.21.15.17:0/2998545559 reap_dead start 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 shutdown_connections 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 >> 172.21.15.17:0/2998545559 conn(0x7f81c00a2f70 msgr2=0x7f81c0134cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 shutdown_connections 2024-09-15T01:58:56.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.246+0000 7f81c6eef640 1 -- 172.21.15.17:0/2998545559 wait complete. 2024-09-15T01:58:56.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:58:56.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T01:58:56.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:58:56.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.313+0000 7f0ed8b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:56.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.313+0000 7f0ed8b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:56.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.313+0000 7f0ed8b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:56.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.313+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0ed40008d0 con 0x7f0ed409e050 2024-09-15T01:58:56.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.361+0000 7f646ed1d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:56.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.361+0000 7f646ed1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:56.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.361+0000 7f646ed1d640 10 monclient: monmap: 2024-09-15T01:58:56.365 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:56.365 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:58:56.363625+0000 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:58:56.363625+0000 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:56.366 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:56.367 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:56.367 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:56.367 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:56.367 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.362+0000 7f646ed1d640 1 Processor -- start 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.362+0000 7f646ed1d640 1 -- start start 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.362+0000 7f646ed1d640 10 monclient: init 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 0x7f646809ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6468070a40 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f646806a550 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 0x7f6468139510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6468137130 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646806eb80 con 0x7f646806a550 2024-09-15T01:58:56.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646809dd00 con 0x7f6468070a40 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468070f30 con 0x7f6468137130 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646ed1d640 10 monclient(hunting): authenticate will time out at 1196.360100s 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646cab9640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 msgr2=0x7f646809ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 0x7f6468139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f6467fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f646806a550 0x7f646809b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58468/0 (socket says 172.21.15.17:58468) 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f6467fff640 1 -- 172.21.15.17:0/273965664 learned_addr learned my addr 172.21.15.17:0/273965664 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646cab9640 1 --2- 172.21.15.17:0/273965664 >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 0x7f646809ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): get_auth_request con 0x7f6468137130 auth_method 0 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient(hunting): handle_auth_done global_id 94337 payload 274 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 -- 172.21.15.17:0/273965664 >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 msgr2=0x7f646809ae20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 --2- 172.21.15.17:0/273965664 >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 0x7f646809ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 -- 172.21.15.17:0/273965664 >> v1:172.21.15.17:6789/0 conn(0x7f646806a550 legacy=0x7f646809b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient: _finish_hunting 0 2024-09-15T01:58:56.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 monclient: found mon.noname-b 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.363+0000 7f646d2ba640 1 -- 172.21.15.17:0/273965664 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646809c470 con 0x7f6468137130 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646d2ba640 1 -- 172.21.15.17:0/273965664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 msgr2=0x7f6468139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646d2ba640 1 --2- 172.21.15.17:0/273965664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 0x7f6468139510 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6460009e60 tx=0x7f6460009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6259fbbc79fd650 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 1 -- 172.21.15.17:0/273965664 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460007840 con 0x7f6468137130 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: dump: 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.384 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 1 -- 172.21.15.17:0/273965664 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646002d050 con 0x7f6468137130 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646ed1d640 5 monclient: authenticate success, global_id 94337 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646ed1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 1 -- 172.21.15.17:0/273965664 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460007bd0 con 0x7f6468137130 2024-09-15T01:58:56.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f64677fe640 10 monclient: dump: 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646ed1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.366+0000 7f646ed1d640 4 set_mon_vals no callback set 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 10 monclient: shutdown 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 -- 172.21.15.17:0/273965664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 msgr2=0x7f6468139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 --2- 172.21.15.17:0/273965664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 0x7f6468139510 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6460009e60 tx=0x7f6460009780 comp rx=0 tx=0).stop 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 -- 172.21.15.17:0/273965664 shutdown_connections 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 --2- 172.21.15.17:0/273965664 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6468137130 0x7f6468139510 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 --2- 172.21.15.17:0/273965664 >> v2:172.21.15.67:3300/0 conn(0x7f6468070a40 0x7f646809ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 -- 172.21.15.17:0/273965664 >> 172.21.15.17:0/273965664 conn(0x7f6468067ae0 msgr2=0x7f64680a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 -- 172.21.15.17:0/273965664 shutdown_connections 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 1 -- 172.21.15.17:0/273965664 wait complete. 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 10 monclient: build_initial_monmap 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.367+0000 7f646ed1d640 10 monclient: monmap: 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:56.389 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 Processor -- start 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 -- start start 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient: init 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 0x7f646811e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-c con 0x7f646806a550 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6468070a40 0x7f646811ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6468070a40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6468137130 0x7f646811efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6468137130 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6468136c30 con 0x7f6468137130 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646806deb0 con 0x7f6468070a40 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646806d020 con 0x7f646806a550 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646ed1d640 10 monclient(hunting): authenticate will time out at 1196.365848s 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 0x7f646811e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 0x7f646811e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33082/0 (socket says 172.21.15.17:33082) 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 1 -- 172.21.15.17:0/2209967898 learned_addr learned my addr 172.21.15.17:0/2209967898 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f6467fff640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6468070a40 msgr2=0x7f646811ea90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:56.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646d2ba640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6468137130 0x7f646811efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 10 monclient(hunting): get_auth_request con 0x7f646806a550 auth_method 0 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646cab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f6467fff640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6468070a40 0x7f646811ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646d2ba640 10 monclient(hunting): get_auth_request con 0x7f6468137130 auth_method 0 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.368+0000 7f646d2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient(hunting): handle_auth_done global_id 94340 payload 824 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6468070a40 msgr2=0x7f646811ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6468070a40 0x7f646811ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6468137130 msgr2=0x7f646811efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6468137130 0x7f646811efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient: _finish_hunting 0 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 monclient: found mon.noname-c 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:56.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646cab9640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460009a20 con 0x7f646806a550 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.369+0000 7f646d2ba640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6468137130 0x7f646811efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646cab9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 msgr2=0x7f646811e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646cab9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 0x7f646811e550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f645c00b600 tx=0x7f645c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c015070 con 0x7f646806a550 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: dump: 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.394 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: _finish_auth 0 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:56.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c003d90 con 0x7f646806a550 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 5 monclient: authenticate success, global_id 94340 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c006380 con 0x7f646806a550 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 10 monclient: _renew_subs 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6468136220 con 0x7f646806a550 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6464ff9640 10 monclient: dump: 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f6466ffd640 4 set_mon_vals no callback set 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 10 monclient: _renew_subs 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.371+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6468120c60 con 0x7f646806a550 2024-09-15T01:58:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f646ed1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f646ed1d640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 0x7f646811ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468120f40 con 0x7f646811f510 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 0x7f646811ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f645c010440 con 0x7f646806a550 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient: get_auth_request con 0x7f646811f510 auth_method 0 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 10 monclient(hunting): handle_auth_done global_id 94343 payload 824 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6467fff640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64681220e0 con 0x7f646811f510 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6464ff9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f644c038720 0x7f644c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.372+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(181..181 src has 148..181) v4 ==== 7255+0+0 (secure 0 0 0) 0x7f645c04bed0 con 0x7f646806a550 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.373+0000 7f646d2ba640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f644c038720 0x7f644c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.373+0000 7f646d2ba640 10 monclient: get_auth_request con 0x7f644c038720 auth_method 0 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6467fff640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 msgr2=0x7f646811ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6467fff640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 0x7f646811ad70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64540047d0 tx=0x7f6454007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454005d50 con 0x7f646811f510 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 10 monclient: dump: 2024-09-15T01:58:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f64540070f0 con 0x7f646811f510 2024-09-15T01:58:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 msgr2=0x7f646811ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 0x7f646811ad70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64540047d0 tx=0x7f6454007f00 comp rx=0 tx=0).stop 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 msgr2=0x7f646811ad70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f6464ff9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811f510 0x7f646811ad70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.375+0000 7f646d2ba640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f644c038720 0x7f644c03abd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f646806a010 tx=0x7f6460037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.396+0000 7f0ec17fa640 10 monclient: tick 2024-09-15T01:58:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:56.396+0000 7f0ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f646ed1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:58:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f646ed1d640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 0x7f646811b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f646ed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468122310 con 0x7f646811b2b0 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 0x7f646811b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient: get_auth_request con 0x7f646811b2b0 auth_method 0 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 10 monclient(hunting): handle_auth_done global_id 94346 payload 824 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.381+0000 7f6467fff640 1 -- 172.21.15.17:0/2209967898 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f64680724a0 con 0x7f646811b2b0 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6467fff640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 msgr2=0x7f646811b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6467fff640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 0x7f646811b6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6454007fd0 tx=0x7f6454016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454021910 con 0x7f646811b2b0 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.383+0000 7f6464ff9640 10 monclient: dump: 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4724 (secure 0 0 0) 0x7f6454005b40 con 0x7f646811b2b0 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 msgr2=0x7f646811b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 0x7f646811b6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6454007fd0 tx=0x7f6454016000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 msgr2=0x7f646811b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f6464ff9640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646811b2b0 0x7f646811b6b0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6454007fd0 tx=0x7f6454016000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:58:56.406 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 94, 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:58:56.407 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.408 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:58:56.409 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:58:56.410 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:58:56.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:58:56.412 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.413 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:58:56.414 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:58:56.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.417 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:58:56.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:58:56.420 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:58:56.420 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f644c038720 msgr2=0x7f644c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f644c038720 0x7f644c03abd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f646806a010 tx=0x7f6460037000 comp rx=0 tx=0).stop 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 10 monclient: shutdown 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 msgr2=0x7f646811e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 --2- 172.21.15.17:0/2209967898 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f646806a550 0x7f646811e550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f645c00b600 tx=0x7f645c0091a0 comp rx=0 tx=0).stop 2024-09-15T01:58:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646d2ba640 1 -- 172.21.15.17:0/2209967898 reap_dead start 2024-09-15T01:58:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646d2ba640 1 -- 172.21.15.17:0/2209967898 reap_dead start 2024-09-15T01:58:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 shutdown_connections 2024-09-15T01:58:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 >> 172.21.15.17:0/2209967898 conn(0x7f6468067ae0 msgr2=0x7f646806b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 shutdown_connections 2024-09-15T01:58:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:58:56.384+0000 7f646ed1d640 1 -- 172.21.15.17:0/2209967898 wait complete. 2024-09-15T01:58:56.422 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:58:56.422 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:58:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.207+0000 7f0ec27fc640 1 -- 172.21.15.67:0/296701874 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v182) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0ecc02b090 con 0x7f0ed409e050 2024-09-15T01:58:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.207+0000 7f0ec27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T01:58:57.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.207+0000 7f0ec27fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 msgr2=0x7f0eb403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 0x7f0eb403abf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4034000 tx=0x7f0ec4032000 comp rx=0 tx=0).stop 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 10 monclient: shutdown 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 msgr2=0x7f0eb4046290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0eb4046290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012c80 tx=0x7f0ecc04a440 comp rx=0 tx=0).stop 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 shutdown_connections 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0ed406a6e0 0x7f0eb4048c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0eb4038740 0x7f0eb403abf0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0ed409e050 0x7f0eb4046290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 --2- 172.21.15.67:0/296701874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0eb4041440 0x7f0eb4043860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 >> 172.21.15.67:0/296701874 conn(0x7f0ed4070ff0 msgr2=0x7f0ed4134f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 shutdown_connections 2024-09-15T01:58:57.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.210+0000 7f0ed8b5d640 1 -- 172.21.15.67:0/296701874 wait complete. 2024-09-15T01:58:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.303+0000 7f571419d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.303+0000 7f571419d640 10 monclient: build_initial_monmap 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.304+0000 7f571419d640 10 monclient: monmap: 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:57.305158+0000 2024-09-15T01:58:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:57.305158+0000 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:57.307 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.304+0000 7f571419d640 1 Processor -- start 2024-09-15T01:58:57.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.304+0000 7f571419d640 1 -- start start 2024-09-15T01:58:57.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.304+0000 7f571419d640 10 monclient: init 2024-09-15T01:58:57.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:57.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 0x7f570c09aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): picked mon.noname-c con 0x7f570c0a1a00 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f570c09d5a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f570c09b020 0x7f570c097180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): picked mon.noname-a con 0x7f570c09b020 addr v2:172.21.15.17:3300/0 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c09cc40 con 0x7f570c09d5a0 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c09cf80 con 0x7f570c09b020 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c09b450 con 0x7f570c0a1a00 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571419d640 10 monclient(hunting): authenticate will time out at 1169.774402s 2024-09-15T01:58:57.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f5711f39640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 msgr2=0x7f570c09aae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f5711f39640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 0x7f570c09aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f570c09b020 0x7f570c097180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f570c09b020 0x7f570c097180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:56624/0 (socket says 172.21.15.67:56624) 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 1 -- 172.21.15.67:0/1785846618 learned_addr learned my addr 172.21.15.67:0/1785846618 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f5711738640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f570c09d5a0 0x7f570c096a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:52238/0 (socket says 172.21.15.67:52238) 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): get_auth_request con 0x7f570c09b020 auth_method 0 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.305+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 10 monclient(hunting): handle_auth_done global_id 104199 payload 274 2024-09-15T01:58:57.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 1 -- 172.21.15.67:0/1785846618 >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 msgr2=0x7f570c09aae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 1 --2- 172.21.15.67:0/1785846618 >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 0x7f570c09aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 1 -- 172.21.15.67:0/1785846618 >> v1:172.21.15.17:6789/0 conn(0x7f570c09d5a0 legacy=0x7f570c096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 10 monclient: _finish_hunting 0 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 1 monclient: found mon.noname-a 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.306+0000 7f571273a640 1 -- 172.21.15.67:0/1785846618 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0a1dd0 con 0x7f570c09b020 2024-09-15T01:58:57.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f571273a640 1 -- 172.21.15.67:0/1785846618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c09b020 msgr2=0x7f570c097180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f571273a640 1 --2- 172.21.15.67:0/1785846618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c09b020 0x7f570c097180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f570c068b40 tx=0x7f5704009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468821658c71e6c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 1 -- 172.21.15.67:0/1785846618 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570402aea0 con 0x7f570c09b020 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: dump: 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.316 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: _finish_auth 0 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 1 -- 172.21.15.67:0/1785846618 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570402d050 con 0x7f570c09b020 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 1 -- 172.21.15.67:0/1785846618 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704007840 con 0x7f570c09b020 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f571419d640 5 monclient: authenticate success, global_id 104199 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f571419d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f571419d640 4 set_mon_vals no callback set 2024-09-15T01:58:57.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.309+0000 7f5710f37640 10 monclient: dump: 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 10 monclient: shutdown 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 -- 172.21.15.67:0/1785846618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c09b020 msgr2=0x7f570c097180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 --2- 172.21.15.67:0/1785846618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c09b020 0x7f570c097180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f570c068b40 tx=0x7f5704009840 comp rx=0 tx=0).stop 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 -- 172.21.15.67:0/1785846618 shutdown_connections 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 --2- 172.21.15.67:0/1785846618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c09b020 0x7f570c097180 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 --2- 172.21.15.67:0/1785846618 >> v2:172.21.15.67:3300/0 conn(0x7f570c0a1a00 0x7f570c09aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 -- 172.21.15.67:0/1785846618 >> 172.21.15.67:0/1785846618 conn(0x7f570c071340 msgr2=0x7f570c0a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 -- 172.21.15.67:0/1785846618 shutdown_connections 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 1 -- 172.21.15.67:0/1785846618 wait complete. 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 10 monclient: build_initial_monmap 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.310+0000 7f571419d640 10 monclient: monmap: 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:57.319 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 Processor -- start 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 -- start start 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient: init 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.320 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): picked mon.noname-c con 0x7f570c09b020 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 0x7f570c069840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): picked mon.noname-b con 0x7f570c09d5a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 0x7f570c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): picked mon.noname-a con 0x7f570c0a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c12ea40 con 0x7f570c0a1a00 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c121880 con 0x7f570c09d5a0 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1215a0 con 0x7f570c09b020 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f571419d640 10 monclient(hunting): authenticate will time out at 1169.780804s 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f5711738640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 msgr2=0x7f570c069840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:57.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.311+0000 7f5711738640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 0x7f570c069840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 0x7f570c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:45388/0 (socket says 172.21.15.67:45388) 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 1 -- 172.21.15.67:0/3974392210 learned_addr learned my addr 172.21.15.67:0/3974392210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): get_auth_request con 0x7f570c0a1a00 auth_method 0 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): get_auth_request con 0x7f570c09b020 auth_method 0 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:57.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient(hunting): handle_auth_done global_id 104202 payload 824 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 -- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 msgr2=0x7f570c12e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 -- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 msgr2=0x7f570c069840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 0x7f570c069840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient: _finish_hunting 0 2024-09-15T01:58:57.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 monclient: found mon.noname-a 2024-09-15T01:58:57.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:57.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f571273a640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704009ae0 con 0x7f570c0a1a00 2024-09-15T01:58:57.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.312+0000 7f5711f39640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f571273a640 1 -- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 msgr2=0x7f570c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f571273a640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 0x7f570c069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5704033000 tx=0x7f5704036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704031cd0 con 0x7f570c0a1a00 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f5701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f5701ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f5701ffb640 10 monclient: dump: 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.326 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.314+0000 7f5701ffb640 10 monclient: _finish_auth 0 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5704031e30 con 0x7f570c0a1a00 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 5 monclient: authenticate success, global_id 104202 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 10 monclient: _renew_subs 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c1212c0 con 0x7f570c0a1a00 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:57.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570403c650 con 0x7f570c0a1a00 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5701ffb640 10 monclient: dump: 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:57.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f5703fff640 4 set_mon_vals no callback set 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 10 monclient: _renew_subs 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.315+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c11fe00 con 0x7f570c0a1a00 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f571419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f571419d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f571419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f570403ca80 con 0x7f570c0a1a00 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f5701ffb640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 0x7f56f003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(182..182 src has 148..182) v4 ==== 7255+0+0 (secure 0 0 0) 0x7f570406d710 con 0x7f570c0a1a00 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f5711f39640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 0x7f56f003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.316+0000 7f5711f39640 10 monclient: get_auth_request con 0x7f56f00386f0 auth_method 0 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.319+0000 7f5711f39640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 0x7f56f003aba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5708003850 tx=0x7f5708009120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.321+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:57.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.321+0000 7f5701ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:57.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.321+0000 7f5701ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.493+0000 7f571419d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.493+0000 7f571419d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.493+0000 7f571419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:57.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:57.493+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:58.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.214+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v183) v1 ==== 82+0+0 (secure 0 0 0) 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:58.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.214+0000 7f5701ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:58.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.214+0000 7f5701ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:58:58.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.321+0000 7f571419d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:58.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.321+0000 7f571419d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:58.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.321+0000 7f571419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.321+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:58.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.322+0000 7f5701ffb640 1 -- 172.21.15.67:0/3974392210 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v183) v1 ==== 115+0+0 (secure 0 0 0) 0x7f570c1216e0 con 0x7f570c0a1a00 2024-09-15T01:58:58.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.322+0000 7f5701ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T01:58:58.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.322+0000 7f5701ffb640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T01:58:58.325 INFO:tasks.workunit.client.0.smithi067.stderr:compat weight-set already created 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 msgr2=0x7f56f003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 0x7f56f003aba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5708003850 tx=0x7f5708009120 comp rx=0 tx=0).stop 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 10 monclient: shutdown 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 msgr2=0x7f570c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 0x7f570c069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5704033000 tx=0x7f5704036000 comp rx=0 tx=0).stop 2024-09-15T01:58:58.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 shutdown_connections 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f56f00386f0 0x7f56f003aba0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f570c0a1a00 0x7f570c069d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f570c09d5a0 0x7f570c069840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 --2- 172.21.15.67:0/3974392210 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f570c09b020 0x7f570c12e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 >> 172.21.15.67:0/3974392210 conn(0x7f570c071340 msgr2=0x7f570c09a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 shutdown_connections 2024-09-15T01:58:58.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.324+0000 7f571419d640 1 -- 172.21.15.67:0/3974392210 wait complete. 2024-09-15T01:58:58.336 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.411+0000 7fbff5094640 10 monclient: get_monmap_and_config 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.411+0000 7fbff5094640 10 monclient: build_initial_monmap 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.411+0000 7fbff5094640 10 monclient: monmap: 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:58.412680+0000 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:58.412680+0000 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:58.415 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:58.415 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:58.415 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:58.415 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:58.415 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 Processor -- start 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 -- start start 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient: init 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-b con 0x7fbff00a1a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:58.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbff009d5a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 0x7fbff0097180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-c con 0x7fbff009b020 addr v2:172.21.15.67:3300/0 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff009b480 con 0x7fbff009d5a0 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff009cf80 con 0x7fbff009b020 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff009cc40 con 0x7fbff00a1a00 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbff5094640 10 monclient(hunting): authenticate will time out at 1170.881646s 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfef59e640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 msgr2=0x7fbff0097180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfee59c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fbff009d5a0 legacy=0x7fbff0096a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:58.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfef59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 0x7fbff0097180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:45398/0 (socket says 172.21.15.67:45398) 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 1 -- 172.21.15.67:0/3245029427 learned_addr learned my addr 172.21.15.67:0/3245029427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request con 0x7fbff00a1a00 auth_method 0 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.412+0000 7fbfeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:58.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_done global_id 94352 payload 274 2024-09-15T01:58:58.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 1 -- 172.21.15.67:0/3245029427 >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 msgr2=0x7fbff0097180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:58.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 1 --2- 172.21.15.67:0/3245029427 >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 0x7fbff0097180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 1 -- 172.21.15.67:0/3245029427 >> v1:172.21.15.67:6789/0 conn(0x7fbff009d5a0 legacy=0x7fbff0096a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:58.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient: _finish_hunting 0 2024-09-15T01:58:58.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 1 monclient: found mon.noname-b 2024-09-15T01:58:58.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:58.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:58.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.413+0000 7fbfeed9d640 1 -- 172.21.15.67:0/3245029427 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00a1dd0 con 0x7fbff00a1a00 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfeed9d640 1 -- 172.21.15.67:0/3245029427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 msgr2=0x7fbff009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfeed9d640 1 --2- 172.21.15.67:0/3245029427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009910 tx=0x7fbfe00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=322637a80d87c54d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 1 -- 172.21.15.67:0/3245029427 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0032440 con 0x7fbff00a1a00 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:58.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: dump: 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: _finish_auth 0 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:58.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 1 -- 172.21.15.67:0/3245029427 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe00325a0 con 0x7fbff00a1a00 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbff5094640 5 monclient: authenticate success, global_id 94352 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbff5094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbff5094640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 1 -- 172.21.15.67:0/3245029427 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0032940 con 0x7fbff00a1a00 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbff5094640 4 set_mon_vals no callback set 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.416+0000 7fbfedd9b640 10 monclient: dump: 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:58.428 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 10 monclient: shutdown 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 -- 172.21.15.67:0/3245029427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 msgr2=0x7fbff009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 --2- 172.21.15.67:0/3245029427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009910 tx=0x7fbfe00091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 -- 172.21.15.67:0/3245029427 shutdown_connections 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 --2- 172.21.15.67:0/3245029427 >> v2:172.21.15.67:3300/0 conn(0x7fbff009b020 0x7fbff0097180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 --2- 172.21.15.67:0/3245029427 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff00a1a00 0x7fbff009aae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 -- 172.21.15.67:0/3245029427 >> 172.21.15.67:0/3245029427 conn(0x7fbff0071340 msgr2=0x7fbff00a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 -- 172.21.15.67:0/3245029427 shutdown_connections 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 1 -- 172.21.15.67:0/3245029427 wait complete. 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 10 monclient: build_initial_monmap 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.417+0000 7fbff5094640 10 monclient: monmap: 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:58.430 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 Processor -- start 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 -- start start 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient: init 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 0x7fbff011d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-b con 0x7fbff009b020 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-c con 0x7fbff009d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 0x7fbff011db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): picked mon.noname-a con 0x7fbff00a1a00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff00985b0 con 0x7fbff00a1a00 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff012ea40 con 0x7fbff009b020 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0120720 con 0x7fbff009d5a0 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbff5094640 10 monclient(hunting): authenticate will time out at 1170.887635s 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbfeed9d640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 msgr2=0x7fbff011d0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:58.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbfeed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 0x7fbff011d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbfee59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.418+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 0x7fbff011db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:45412/0 (socket says 172.21.15.67:45412) 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 1 -- 172.21.15.67:0/878407822 learned_addr learned my addr 172.21.15.67:0/878407822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): get_auth_request con 0x7fbff009d5a0 auth_method 0 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): get_auth_request con 0x7fbff00a1a00 auth_method 0 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:58.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient(hunting): handle_auth_done global_id 104205 payload 824 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 -- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 msgr2=0x7fbff011d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 -- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 msgr2=0x7fbff011d0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 0x7fbff011d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient: _finish_hunting 0 2024-09-15T01:58:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 monclient: found mon.noname-a 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfef59e640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0009450 con 0x7fbff00a1a00 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.419+0000 7fbfee59c640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfef59e640 1 -- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 msgr2=0x7fbff011db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfef59e640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 0x7fbff011db30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800bdf0 tx=0x7fbfe8009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8006570 con 0x7fbff00a1a00 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: dump: 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe800f070 con 0x7fbff00a1a00 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8006750 con 0x7fbff00a1a00 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfd6ffd640 10 monclient: dump: 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:58.437 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbff5094640 5 monclient: authenticate success, global_id 104205 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbff5094640 10 monclient: _renew_subs 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbff5094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0120a00 con 0x7fbff00a1a00 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.421+0000 7fbfed59a640 4 set_mon_vals no callback set 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 10 monclient: _renew_subs 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff0121580 con 0x7fbff00a1a00 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff004bf60 con 0x7fbff00a1a00 2024-09-15T01:58:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.422+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbfe801e3a0 con 0x7fbff00a1a00 2024-09-15T01:58:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.423+0000 7fbfd6ffd640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 0x7fbfdc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.423+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(183..183 src has 148..183) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fbfe804cfd0 con 0x7fbff00a1a00 2024-09-15T01:58:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.423+0000 7fbfeed9d640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 0x7fbfdc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.423+0000 7fbfeed9d640 10 monclient: get_auth_request con 0x7fbfdc0386f0 auth_method 0 2024-09-15T01:58:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.425+0000 7fbfeed9d640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 0x7fbfdc03aba0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0031000 tx=0x7fbfe0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:58.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.425+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:58:58.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.428+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbfe8011ac0 con 0x7fbff00a1a00 2024-09-15T01:58:58.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.428+0000 7fbfd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:58.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.428+0000 7fbfd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:58.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.597+0000 7fbff5094640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:58.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.597+0000 7fbff5094640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:58.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.597+0000 7fbff5094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:58.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:58.597+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7fbff012efa0 con 0x7fbff00a1a00 2024-09-15T01:58:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.217+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v184) v1 ==== 176+0+0 (secure 0 0 0) 0x7fbfe80112e0 con 0x7fbff00a1a00 2024-09-15T01:58:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.217+0000 7fbfd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.217+0000 7fbfd6ffd640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-15T01:58:59.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.319+0000 7fbff5094640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.319+0000 7fbff5094640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.319+0000 7fbff5094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:58:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.319+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7fbff006cf20 con 0x7fbff00a1a00 2024-09-15T01:58:59.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.320+0000 7fbfd6ffd640 1 -- 172.21.15.67:0/878407822 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v184) v1 ==== 150+0+0 (secure 0 0 0) 0x7fbfe80240c0 con 0x7fbff00a1a00 2024-09-15T01:58:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.320+0000 7fbfd6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T01:58:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.320+0000 7fbfd6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-15T01:58:59.323 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'r1' already exists 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 msgr2=0x7fbfdc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 0x7fbfdc03aba0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0031000 tx=0x7fbfe0032000 comp rx=0 tx=0).stop 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 10 monclient: shutdown 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 msgr2=0x7fbff011db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 0x7fbff011db30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800bdf0 tx=0x7fbfe8009650 comp rx=0 tx=0).stop 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 shutdown_connections 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbfdc0386f0 0x7fbfdc03aba0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbff00a1a00 0x7fbff011db30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbff009d5a0 0x7fbff011d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 --2- 172.21.15.67:0/878407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff009b020 0x7fbff011d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 >> 172.21.15.67:0/878407822 conn(0x7fbff0071340 msgr2=0x7fbff012df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 shutdown_connections 2024-09-15T01:58:59.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.322+0000 7fbff5094640 1 -- 172.21.15.67:0/878407822 wait complete. 2024-09-15T01:58:59.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ seq 1 32 2024-09-15T01:58:59.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:58:59.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-15T01:58:59.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 10 monclient: get_monmap_and_config 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 10 monclient: monmap: 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:58:59.421226+0000 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:58:59.421226+0000 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 1 Processor -- start 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 1 -- start start 2024-09-15T01:58:59.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.420+0000 7f8dcdb9d640 10 monclient: init 2024-09-15T01:58:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:58:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc8131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:59.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dc8098420 addr v1:172.21.15.17:6789/0 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8dc81371a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80a13e0 con 0x7f8dc8098420 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80a0440 con 0x7f8dc81371a0 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80a01e0 con 0x7f8dc8131e70 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dcdb9d640 10 monclient(hunting): authenticate will time out at 1171.890363s 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc7fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f8dc81371a0 legacy=0x7f8dc8139580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:58:59.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:45424/0 (socket says 172.21.15.67:45424) 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 1 -- 172.21.15.67:0/3808636365 learned_addr learned my addr 172.21.15.67:0/3808636365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request con 0x7f8dc8131e70 auth_method 0 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc6ffd640 1 --1- 172.21.15.67:0/3808636365 >> v1:172.21.15.17:6789/0 conn(0x7f8dc8098420 0x7f8dc8098850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:52252/0 (socket says 172.21.15.67:52252) 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.421+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_done global_id 94358 payload 274 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 1 -- 172.21.15.67:0/3808636365 >> v1:172.21.15.67:6789/0 conn(0x7f8dc81371a0 legacy=0x7f8dc8139580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:58:59.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 1 -- 172.21.15.67:0/3808636365 >> v1:172.21.15.17:6789/0 conn(0x7f8dc8098420 legacy=0x7f8dc8098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:58:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 1 monclient: found mon.noname-b 2024-09-15T01:58:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.422+0000 7f8dc77fe640 1 -- 172.21.15.67:0/3808636365 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80a19b0 con 0x7f8dc8131e70 2024-09-15T01:58:59.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.424+0000 7f8dc77fe640 1 -- 172.21.15.67:0/3808636365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 msgr2=0x7f8dc8132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:59.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.424+0000 7f8dc77fe640 1 --2- 172.21.15.67:0/3808636365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009940 tx=0x7f8db80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25996401dc8456cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:59.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 1 -- 172.21.15.67:0/3808636365 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8032440 con 0x7f8dc8131e70 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: dump: 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: _finish_auth 0 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 1 -- 172.21.15.67:0/3808636365 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80325a0 con 0x7f8dc8131e70 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 1 -- 172.21.15.67:0/3808636365 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8032940 con 0x7f8dc8131e70 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dc67fc640 10 monclient: dump: 2024-09-15T01:58:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 5 monclient: authenticate success, global_id 94358 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 4 set_mon_vals no callback set 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 10 monclient: shutdown 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/3808636365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 msgr2=0x7f8dc8132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/3808636365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009940 tx=0x7f8db80091d0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/3808636365 shutdown_connections 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/3808636365 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8131e70 0x7f8dc8132240 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/3808636365 >> 172.21.15.67:0/3808636365 conn(0x7f8dc80a5c10 msgr2=0x7f8dc80a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.425+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/3808636365 shutdown_connections 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/3808636365 wait complete. 2024-09-15T01:58:59.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 10 monclient: build_initial_monmap 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 10 monclient: monmap: 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 1 Processor -- start 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 1 -- start start 2024-09-15T01:58:59.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.426+0000 7f8dcdb9d640 10 monclient: init 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc8098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dc8131e70 0x7f8dc8099f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc8131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 0x7f8dc809a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc81371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc813aa70 con 0x7f8dc8131e70 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:59.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81344c0 con 0x7f8dc81371a0 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8133010 con 0x7f8dc8098420 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dcdb9d640 10 monclient(hunting): authenticate will time out at 1171.896387s 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc7fff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 msgr2=0x7f8dc809a480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 0x7f8dc809a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dc8131e70 0x7f8dc8099f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:45432/0 (socket says 172.21.15.67:45432) 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 1 -- 172.21.15.67:0/1233664345 learned_addr learned my addr 172.21.15.67:0/1233664345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request con 0x7f8dc8098420 auth_method 0 2024-09-15T01:58:59.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): get_auth_request con 0x7f8dc8131e70 auth_method 0 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.427+0000 7f8dc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:58:59.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_done global_id 94361 payload 824 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 -- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 msgr2=0x7f8dc809a480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 0x7f8dc809a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 -- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dc8131e70 msgr2=0x7f8dc8099f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dc8131e70 0x7f8dc8099f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 monclient: found mon.noname-c 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.428+0000 7f8dc77fe640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db8009450 con 0x7f8dc8098420 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dc77fe640 1 -- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 msgr2=0x7f8dc8099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dc77fe640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8db80099a0 tx=0x7f8db8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80318c0 con 0x7f8dc8098420 2024-09-15T01:58:59.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: dump: 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:59.444 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: _finish_auth 0 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8031a20 con 0x7f8dc8098420 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8031de0 con 0x7f8dc8098420 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 5 monclient: authenticate success, global_id 94361 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 10 monclient: _renew_subs 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8133f00 con 0x7f8dc8098420 2024-09-15T01:58:59.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dc5ffb640 4 set_mon_vals no callback set 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8daf7fe640 10 monclient: dump: 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:58:59.446 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 10 monclient: _renew_subs 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.430+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81339a0 con 0x7f8dc8098420 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8db8039070 con 0x7f8dc8098420 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dcdb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dcdb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dcdb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc80639e0 con 0x7f8dc8098420 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8daf7fe640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 0x7f8db403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(184..184 src has 148..184) v4 ==== 7793+0+0 (secure 0 0 0) 0x7f8db806c500 con 0x7f8dc8098420 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dc6ffd640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 0x7f8db403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:58:59.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.431+0000 7f8dc6ffd640 10 monclient: get_auth_request con 0x7f8db4038720 auth_method 0 2024-09-15T01:58:59.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.434+0000 7f8dc6ffd640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 0x7f8db403abd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809bcf0 tx=0x7f8dbc006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:58:59.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.436+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8db803bba0 con 0x7f8dc8098420 2024-09-15T01:58:59.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.436+0000 7f8daf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:58:59.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.436+0000 7f8daf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:58:59.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.604+0000 7f8dcdb9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:58:59.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.604+0000 7f8dcdb9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:58:59.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.604+0000 7f8dcdb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:58:59.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:58:59.604+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8dc80639e0 con 0x7f8dc8098420 2024-09-15T01:59:00.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.224+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v185) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8db8037020 con 0x7f8dc8098420 2024-09-15T01:59:00.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.224+0000 7f8daf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.224+0000 7f8daf7fe640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-15T01:59:00.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.327+0000 7f8dcdb9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.328+0000 7f8dcdb9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.328+0000 7f8dcdb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.328+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8dc80639e0 con 0x7f8dc8098420 2024-09-15T01:59:00.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.329+0000 7f8daf7fe640 1 -- 172.21.15.67:0/1233664345 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v185) v1 ==== 145+0+0 (secure 0 0 0) 0x7f8db8040020 con 0x7f8dc8098420 2024-09-15T01:59:00.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.329+0000 7f8daf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.331 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.329+0000 7f8daf7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-15T01:59:00.331 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h1' already exists 2024-09-15T01:59:00.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 msgr2=0x7f8db403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:00.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 0x7f8db403abd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809bcf0 tx=0x7f8dbc006f50 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 10 monclient: shutdown 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 msgr2=0x7f8dc8099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8db80099a0 tx=0x7f8db8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 shutdown_connections 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8db4038720 0x7f8db403abd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dc8131e70 0x7f8dc8099f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc81371a0 0x7f8dc809a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 --2- 172.21.15.67:0/1233664345 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8dc8098420 0x7f8dc8099a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 >> 172.21.15.67:0/1233664345 conn(0x7f8dc80a5c10 msgr2=0x7f8dc8131580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 shutdown_connections 2024-09-15T01:59:00.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.331+0000 7f8dcdb9d640 1 -- 172.21.15.67:0/1233664345 wait complete. 2024-09-15T01:59:00.342 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:00.343 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-15T01:59:00.420 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.418+0000 7f63de11d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.418+0000 7f63de11d640 10 monclient: build_initial_monmap 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.418+0000 7f63de11d640 10 monclient: monmap: 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:00.419355+0000 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:00.419355+0000 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:00.421 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:00.422 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.418+0000 7f63de11d640 1 Processor -- start 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 -- start start 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient: init 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 0x7f63d8132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-a con 0x7f63d8131e70 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-b con 0x7f63d8098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63d8068e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80a13e0 con 0x7f63d8068e70 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d80a0440 con 0x7f63d8131e70 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80a01e0 con 0x7f63d8098420 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63de11d640 10 monclient(hunting): authenticate will time out at 1172.888522s 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d7fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f63d8068e70 legacy=0x7f63d80692f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 0x7f63d8132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:00.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44428/0 (socket says 172.21.15.67:44428) 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 1 -- 172.21.15.67:0/460603266 learned_addr learned my addr 172.21.15.67:0/460603266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): get_auth_request con 0x7f63d8098420 auth_method 0 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): get_auth_request con 0x7f63d8131e70 auth_method 0 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.419+0000 7f63d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_done global_id 94364 payload 274 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 1 -- 172.21.15.67:0/460603266 >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 msgr2=0x7f63d8132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 1 --2- 172.21.15.67:0/460603266 >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 0x7f63d8132240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 1 -- 172.21.15.67:0/460603266 >> v1:172.21.15.67:6789/0 conn(0x7f63d8068e70 legacy=0x7f63d80692f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 1 monclient: found mon.noname-b 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d6ffd640 1 -- 172.21.15.67:0/460603266 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80a19b0 con 0x7f63d8098420 2024-09-15T01:59:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.420+0000 7f63d77fe640 1 --2- 172.21.15.67:0/460603266 >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 0x7f63d8132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:00.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.422+0000 7f63d6ffd640 1 -- 172.21.15.67:0/460603266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 msgr2=0x7f63d8098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:00.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.422+0000 7f63d6ffd640 1 --2- 172.21.15.67:0/460603266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009910 tx=0x7f63c00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a99a13c2e129d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 1 -- 172.21.15.67:0/460603266 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c002aea0 con 0x7f63d8098420 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: dump: 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:00.433 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 1 -- 172.21.15.67:0/460603266 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0032440 con 0x7f63d8098420 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 5 monclient: authenticate success, global_id 94364 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 1 -- 172.21.15.67:0/460603266 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0032820 con 0x7f63d8098420 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 4 set_mon_vals no callback set 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63d67fc640 10 monclient: dump: 2024-09-15T01:59:00.434 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 10 monclient: shutdown 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 -- 172.21.15.67:0/460603266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 msgr2=0x7f63d8098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 --2- 172.21.15.67:0/460603266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009910 tx=0x7f63c00091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 -- 172.21.15.67:0/460603266 shutdown_connections 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 --2- 172.21.15.67:0/460603266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d8098850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 --2- 172.21.15.67:0/460603266 >> v2:172.21.15.17:3300/0 conn(0x7f63d8131e70 0x7f63d8132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 -- 172.21.15.67:0/460603266 >> 172.21.15.67:0/460603266 conn(0x7f63d80a5c10 msgr2=0x7f63d80a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 -- 172.21.15.67:0/460603266 shutdown_connections 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.423+0000 7f63de11d640 1 -- 172.21.15.67:0/460603266 wait complete. 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.424+0000 7f63de11d640 10 monclient: build_initial_monmap 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.424+0000 7f63de11d640 10 monclient: monmap: 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:00.436 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.424+0000 7f63de11d640 1 Processor -- start 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.424+0000 7f63de11d640 1 -- start start 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.424+0000 7f63de11d640 10 monclient: init 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 0x7f63d8141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-b con 0x7f63d8068e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:00.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-c con 0x7f63d8098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): picked mon.noname-a con 0x7f63d8131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d809e450 con 0x7f63d8131e70 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d811bf20 con 0x7f63d8068e70 2024-09-15T01:59:00.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d8134f10 con 0x7f63d8098420 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63de11d640 10 monclient(hunting): authenticate will time out at 1172.894228s 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d77fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 msgr2=0x7f63d8141a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 0x7f63d8141a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44430/0 (socket says 172.21.15.67:44430) 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 -- 172.21.15.67:0/1500670068 learned_addr learned my addr 172.21.15.67:0/1500670068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47210/0 (socket says 172.21.15.67:47210) 2024-09-15T01:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): get_auth_request con 0x7f63d8098420 auth_method 0 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): get_auth_request con 0x7f63d8131e70 auth_method 0 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient(hunting): handle_auth_done global_id 94367 payload 824 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 -- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 msgr2=0x7f63d8141a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 0x7f63d8141a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 -- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 msgr2=0x7f63d809cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 monclient: found mon.noname-c 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d6ffd640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c8009390 con 0x7f63d8098420 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.425+0000 7f63d7fff640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63d6ffd640 1 -- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 msgr2=0x7f63d809c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63d6ffd640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f63c002a6d0 tx=0x7f63c0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c002aea0 con 0x7f63d8098420 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: dump: 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 5 monclient: authenticate success, global_id 94367 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 10 monclient: _renew_subs 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0000d10 con 0x7f63d8098420 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c0009450 con 0x7f63d8098420 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0031ea0 con 0x7f63d8098420 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63cf7fe640 10 monclient: dump: 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 10 monclient: _renew_subs 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d809de90 con 0x7f63d8098420 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.428+0000 7f63d5ffb640 4 set_mon_vals no callback set 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f63c003c610 con 0x7f63d8098420 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63de11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63de11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63de11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d80639e0 con 0x7f63d8098420 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63cf7fe640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 0x7f63b803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(185..185 src has 148..185) v4 ==== 8003+0+0 (secure 0 0 0) 0x7f63c006cdf0 con 0x7f63d8098420 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63d77fe640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 0x7f63b803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.429+0000 7f63d77fe640 10 monclient: get_auth_request con 0x7f63b8038720 auth_method 0 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.432+0000 7f63d77fe640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 0x7f63b803abd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009610 tx=0x7f63c80090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.434+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f63c0070030 con 0x7f63d8098420 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.434+0000 7f63cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.434+0000 7f63cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.598+0000 7f63de11d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.598+0000 7f63de11d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.598+0000 7f63de11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:00.598+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63d8067a50 con 0x7f63d8098420 2024-09-15T01:59:01.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.229+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v186) v1 ==== 166+0+0 (secure 0 0 0) 0x7f63c0037070 con 0x7f63d8098420 2024-09-15T01:59:01.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.229+0000 7f63cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.229+0000 7f63cf7fe640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-15T01:59:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.330+0000 7f63de11d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.330+0000 7f63de11d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.330+0000 7f63de11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.330+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63d8067a50 con 0x7f63d8098420 2024-09-15T01:59:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.331+0000 7f63cf7fe640 1 -- 172.21.15.67:0/1500670068 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v186) v1 ==== 145+0+0 (secure 0 0 0) 0x7f63c00382b0 con 0x7f63d8098420 2024-09-15T01:59:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.331+0000 7f63cf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.331+0000 7f63cf7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-15T01:59:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h2' already exists 2024-09-15T01:59:01.335 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 msgr2=0x7f63b803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 0x7f63b803abd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009610 tx=0x7f63c80090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 10 monclient: shutdown 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 msgr2=0x7f63d809c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f63c002a6d0 tx=0x7f63c0032000 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 shutdown_connections 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63b8038720 0x7f63b803abd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63d8131e70 0x7f63d809cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63d8098420 0x7f63d809c620 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 --2- 172.21.15.67:0/1500670068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d8068e70 0x7f63d8141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 >> 172.21.15.67:0/1500670068 conn(0x7f63d80a5c10 msgr2=0x7f63d8097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:01.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 shutdown_connections 2024-09-15T01:59:01.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.333+0000 7f63de11d640 1 -- 172.21.15.67:0/1500670068 wait complete. 2024-09-15T01:59:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 10 monclient: get_monmap_and_config 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 10 monclient: build_initial_monmap 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 10 monclient: monmap: 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:01.422430+0000 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:01.422430+0000 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 1 Processor -- start 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 1 -- start start 2024-09-15T01:59:01.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.421+0000 7f0b5cac9640 10 monclient: init 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b58131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f0b58098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b581371a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5659c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0b58131e70 legacy=0x7f0b58132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:01.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580a13e0 con 0x7f0b581371a0 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580a0440 con 0x7f0b58131e70 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580a01e0 con 0x7f0b58098420 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b5cac9640 10 monclient(hunting): authenticate will time out at 1173.891738s 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.422+0000 7f0b55d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b56d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f0b581371a0 0x7f0b58139580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:38132/0 (socket says 172.21.15.67:38132) 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44442/0 (socket says 172.21.15.67:44442) 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b56d9d640 1 -- 172.21.15.67:0/1760336206 learned_addr learned my addr 172.21.15.67:0/1760336206 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): get_auth_request con 0x7f0b58098420 auth_method 0 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:01.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_done global_id 94370 payload 274 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1760336206 >> v1:172.21.15.67:6789/0 conn(0x7f0b58131e70 legacy=0x7f0b58132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1760336206 >> v1:172.21.15.17:6789/0 conn(0x7f0b581371a0 legacy=0x7f0b58139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 1 monclient: found mon.noname-b 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:01.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.423+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1760336206 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580a19b0 con 0x7f0b58098420 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1760336206 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 msgr2=0x7f0b58098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b55d9b640 1 --2- 172.21.15.67:0/1760336206 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009880 tx=0x7f0b4c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7da013797279 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 1 -- 172.21.15.67:0/1760336206 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c032440 con 0x7f0b58098420 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:01.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: dump: 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: _finish_auth 0 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:01.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 1 -- 172.21.15.67:0/1760336206 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c0325a0 con 0x7f0b58098420 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 5 monclient: authenticate success, global_id 94370 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 1 -- 172.21.15.67:0/1760336206 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c032940 con 0x7f0b58098420 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 4 set_mon_vals no callback set 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5559a640 10 monclient: dump: 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.435 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 10 monclient: shutdown 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1760336206 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 msgr2=0x7f0b58098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.426+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1760336206 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009880 tx=0x7f0b4c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1760336206 shutdown_connections 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1760336206 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b58098420 0x7f0b58098850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1760336206 >> 172.21.15.67:0/1760336206 conn(0x7f0b580a5c10 msgr2=0x7f0b580a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:01.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1760336206 shutdown_connections 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1760336206 wait complete. 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 10 monclient: build_initial_monmap 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 10 monclient: monmap: 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:01.437 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.427+0000 7f0b5cac9640 1 Processor -- start 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 -- start start 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient: init 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 0x7f0b5812e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f0b58098420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-a con 0x7f0b58131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b581371a0 0x7f0b58132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5659c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 msgr2=0x7f0b5812e9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:01.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): picked mon.noname-c con 0x7f0b581371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580a0d80 con 0x7f0b58131e70 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580a3370 con 0x7f0b58098420 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 0x7f0b5812e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58134fb0 con 0x7f0b581371a0 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b5cac9640 10 monclient(hunting): authenticate will time out at 1173.897543s 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47220/0 (socket says 172.21.15.67:47220) 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b56d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b581371a0 0x7f0b58132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:01.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1277756519 learned_addr learned my addr 172.21.15.67:0/1277756519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): get_auth_request con 0x7f0b58131e70 auth_method 0 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b56d9d640 10 monclient(hunting): get_auth_request con 0x7f0b581371a0 auth_method 0 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.428+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:01.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 10 monclient(hunting): handle_auth_done global_id 104217 payload 824 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b581371a0 msgr2=0x7f0b58132ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b581371a0 0x7f0b58132ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 msgr2=0x7f0b5812e9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 0x7f0b5812e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 monclient: found mon.noname-a 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:01.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.429+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c009390 con 0x7f0b58131e70 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b55d9b640 1 -- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 msgr2=0x7f0b5812ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b55d9b640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009880 tx=0x7f0b4c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c031910 con 0x7f0b58131e70 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: dump: 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:01.444 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c031ab0 con 0x7f0b58131e70 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c031e70 con 0x7f0b58131e70 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b5cac9640 5 monclient: authenticate success, global_id 104217 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b5cac9640 10 monclient: _renew_subs 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b5cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:01.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58142470 con 0x7f0b58131e70 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b54d99640 4 set_mon_vals no callback set 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.431+0000 7f0b467fc640 10 monclient: dump: 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:01.446 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 10 monclient: _renew_subs 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b581347e0 con 0x7f0b58131e70 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b580639e0 con 0x7f0b58131e70 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.432+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0b4c039040 con 0x7f0b58131e70 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.433+0000 7f0b467fc640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 0x7f0b3803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.433+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(186..186 src has 148..186) v4 ==== 8213+0+0 (secure 0 0 0) 0x7f0b4c06ce90 con 0x7f0b58131e70 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.433+0000 7f0b5659c640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 0x7f0b3803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.433+0000 7f0b5659c640 10 monclient: get_auth_request con 0x7f0b380386f0 auth_method 0 2024-09-15T01:59:01.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.435+0000 7f0b5659c640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 0x7f0b3803aba0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48006d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.437+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0b4c044620 con 0x7f0b58131e70 2024-09-15T01:59:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.437+0000 7f0b467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.437+0000 7f0b467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:01.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.602+0000 7f0b5cac9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.602+0000 7f0b5cac9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:01.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.602+0000 7f0b5cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:01.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:01.602+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0b580639e0 con 0x7f0b58131e70 2024-09-15T01:59:02.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.233+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v187) v1 ==== 166+0+0 (secure 0 0 0) 0x7f0b4c049030 con 0x7f0b58131e70 2024-09-15T01:59:02.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.233+0000 7f0b467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.233+0000 7f0b467fc640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.334+0000 7f0b5cac9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.334+0000 7f0b5cac9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.334+0000 7f0b5cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.334+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0b580639e0 con 0x7f0b58131e70 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.335+0000 7f0b467fc640 1 -- 172.21.15.67:0/1277756519 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v187) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0b4c049030 con 0x7f0b58131e70 2024-09-15T01:59:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.335+0000 7f0b467fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.335+0000 7f0b467fc640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-15T01:59:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h3' already exists 2024-09-15T01:59:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 msgr2=0x7f0b3803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 0x7f0b3803aba0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48006d70 comp rx=0 tx=0).stop 2024-09-15T01:59:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 10 monclient: shutdown 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 msgr2=0x7f0b5812ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009880 tx=0x7f0b4c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 shutdown_connections 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0b380386f0 0x7f0b3803aba0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b58131e70 0x7f0b5812ef10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0b581371a0 0x7f0b58132ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 --2- 172.21.15.67:0/1277756519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58098420 0x7f0b5812e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 >> 172.21.15.67:0/1277756519 conn(0x7f0b580a5c10 msgr2=0x7f0b58068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 shutdown_connections 2024-09-15T01:59:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.337+0000 7f0b5cac9640 1 -- 172.21.15.67:0/1277756519 wait complete. 2024-09-15T01:59:02.348 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:02.348 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 10 monclient: get_monmap_and_config 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 10 monclient: build_initial_monmap 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 10 monclient: monmap: 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:02.425261+0000 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:02.425261+0000 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:02.427 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 1 Processor -- start 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 1 -- start start 2024-09-15T01:59:02.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.424+0000 7f76d9109640 10 monclient: init 2024-09-15T01:59:02.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76d4131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 0x7f76d4139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-c con 0x7f76d41371a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d2d9d640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f76d4131e70 legacy=0x7f76d4132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40a13e0 con 0x7f76d4131e70 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:02.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d40a0440 con 0x7f76d41371a0 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40a01e0 con 0x7f76d4098420 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d9109640 10 monclient(hunting): authenticate will time out at 1174.894328s 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d359e640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 msgr2=0x7f76d4139580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d359e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 0x7f76d4139580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44460/0 (socket says 172.21.15.67:44460) 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 -- 172.21.15.67:0/1759920619 learned_addr learned my addr 172.21.15.67:0/1759920619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:02.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): get_auth_request con 0x7f76d4098420 auth_method 0 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient(hunting): handle_auth_done global_id 94376 payload 274 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 -- 172.21.15.67:0/1759920619 >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 msgr2=0x7f76d4139580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 --2- 172.21.15.67:0/1759920619 >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 0x7f76d4139580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 -- 172.21.15.67:0/1759920619 >> v1:172.21.15.67:6789/0 conn(0x7f76d4131e70 legacy=0x7f76d4132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:02.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 monclient: found mon.noname-b 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.425+0000 7f76d259c640 1 -- 172.21.15.67:0/1759920619 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40a19b0 con 0x7f76d4098420 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.428+0000 7f76d259c640 1 -- 172.21.15.67:0/1759920619 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 msgr2=0x7f76d4098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.428+0000 7f76d259c640 1 --2- 172.21.15.67:0/1759920619 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009910 tx=0x7f76c80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7000c928cad9c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 1 -- 172.21.15.67:0/1759920619 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8032440 con 0x7f76d4098420 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: dump: 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:02.435 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: _finish_auth 0 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 5 monclient: authenticate success, global_id 94376 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 1 -- 172.21.15.67:0/1759920619 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c80325a0 con 0x7f76d4098420 2024-09-15T01:59:02.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 4 set_mon_vals no callback set 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 1 -- 172.21.15.67:0/1759920619 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8032940 con 0x7f76d4098420 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d1d9b640 10 monclient: dump: 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.437 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 10 monclient: shutdown 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 -- 172.21.15.67:0/1759920619 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 msgr2=0x7f76d4098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 --2- 172.21.15.67:0/1759920619 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009910 tx=0x7f76c80091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 -- 172.21.15.67:0/1759920619 shutdown_connections 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 --2- 172.21.15.67:0/1759920619 >> v2:172.21.15.67:3300/0 conn(0x7f76d41371a0 0x7f76d4139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 --2- 172.21.15.67:0/1759920619 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4098850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 -- 172.21.15.67:0/1759920619 >> 172.21.15.67:0/1759920619 conn(0x7f76d40a5c10 msgr2=0x7f76d40a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 -- 172.21.15.67:0/1759920619 shutdown_connections 2024-09-15T01:59:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.429+0000 7f76d9109640 1 -- 172.21.15.67:0/1759920619 wait complete. 2024-09-15T01:59:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 10 monclient: build_initial_monmap 2024-09-15T01:59:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 10 monclient: monmap: 2024-09-15T01:59:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 1 Processor -- start 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 1 -- start start 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 10 monclient: init 2024-09-15T01:59:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.430+0000 7f76d9109640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-c con 0x7f76d4098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76d4131e70 0x7f76d4132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-a con 0x7f76d4131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 0x7f76d4133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): picked mon.noname-b con 0x7f76d41371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40a3370 con 0x7f76d4131e70 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4135170 con 0x7f76d41371a0 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4134790 con 0x7f76d4098420 2024-09-15T01:59:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d9109640 10 monclient(hunting): authenticate will time out at 1174.900051s 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d359e640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 msgr2=0x7f76d4133430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 0x7f76d4133430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76d4131e70 0x7f76d4132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44470/0 (socket says 172.21.15.67:44470) 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 -- 172.21.15.67:0/3867884541 learned_addr learned my addr 172.21.15.67:0/3867884541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): get_auth_request con 0x7f76d4098420 auth_method 0 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): get_auth_request con 0x7f76d4131e70 auth_method 0 2024-09-15T01:59:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient(hunting): handle_auth_done global_id 94379 payload 824 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 -- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 msgr2=0x7f76d4133430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 0x7f76d4133430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 -- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76d4131e70 msgr2=0x7f76d4132ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76d4131e70 0x7f76d4132ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 monclient: found mon.noname-c 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.431+0000 7f76d2d9d640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009450 con 0x7f76d4098420 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.433+0000 7f76d2d9d640 1 -- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 msgr2=0x7f76d4141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.433+0000 7f76d2d9d640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bbb0 tx=0x7f76c40097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400cc80 con 0x7f76d4098420 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: dump: 2024-09-15T01:59:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c400cde0 con 0x7f76d4098420 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4016430 con 0x7f76d4098420 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 5 monclient: authenticate success, global_id 94379 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 10 monclient: _renew_subs 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d4134c80 con 0x7f76d4098420 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d159a640 4 set_mon_vals no callback set 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76baffd640 10 monclient: dump: 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 10 monclient: _renew_subs 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d41367f0 con 0x7f76d4098420 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.434+0000 7f76d9109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76d9109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76d9109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d40639e0 con 0x7f76d4098420 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f76c4012020 con 0x7f76d4098420 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76baffd640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 0x7f76c003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(187..187 src has 148..187) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f76c404c8b0 con 0x7f76d4098420 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76d259c640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 0x7f76c003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.435+0000 7f76d259c640 10 monclient: get_auth_request con 0x7f76c00386f0 auth_method 0 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.437+0000 7f76d259c640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 0x7f76c003aba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f76d40a6250 tx=0x7f76c8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.440+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f76c401d900 con 0x7f76d4098420 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.440+0000 7f76baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.440+0000 7f76baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.606+0000 7f76d9109640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.606+0000 7f76d9109640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.606+0000 7f76d9109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:02.606+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f76d4067a50 con 0x7f76d4098420 2024-09-15T01:59:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.235+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v188) v1 ==== 166+0+0 (secure 0 0 0) 0x7f76c401d300 con 0x7f76d4098420 2024-09-15T01:59:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.235+0000 7f76baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.235+0000 7f76baffd640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-15T01:59:03.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.339+0000 7f76d9109640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.339+0000 7f76d9109640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.339+0000 7f76d9109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.339+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f76d4135340 con 0x7f76d4098420 2024-09-15T01:59:03.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.340+0000 7f76baffd640 1 -- 172.21.15.67:0/3867884541 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v188) v1 ==== 145+0+0 (secure 0 0 0) 0x7f76c401d4e0 con 0x7f76d4098420 2024-09-15T01:59:03.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.340+0000 7f76baffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.340+0000 7f76baffd640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-15T01:59:03.343 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h4' already exists 2024-09-15T01:59:03.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 msgr2=0x7f76c003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:03.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 0x7f76c003aba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f76d40a6250 tx=0x7f76c8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:03.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 10 monclient: shutdown 2024-09-15T01:59:03.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 msgr2=0x7f76d4141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bbb0 tx=0x7f76c40097f0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 shutdown_connections 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f76c00386f0 0x7f76c003aba0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f76d4131e70 0x7f76d4132ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d41371a0 0x7f76d4133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.342+0000 7f76d9109640 1 --2- 172.21.15.67:0/3867884541 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f76d4098420 0x7f76d4141c00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.343+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 >> 172.21.15.67:0/3867884541 conn(0x7f76d40a5c10 msgr2=0x7f76d4097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.343+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 shutdown_connections 2024-09-15T01:59:03.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.343+0000 7f76d9109640 1 -- 172.21.15.67:0/3867884541 wait complete. 2024-09-15T01:59:03.354 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:03.354 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 10 monclient: get_monmap_and_config 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 10 monclient: build_initial_monmap 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 10 monclient: monmap: 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:03.432438+0000 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:03.432438+0000 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:03.434 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 1 Processor -- start 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 1 -- start start 2024-09-15T01:59:03.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.431+0000 7f6640c80640 10 monclient: init 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f663c131aa0 0x7f663c131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-a con 0x7f663c131aa0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f663c0a0a60 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 0x7f663c138de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-c con 0x7f663c09f490 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c132930 con 0x7f663c0a0a60 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c0a1b00 con 0x7f663c131aa0 2024-09-15T01:59:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c0a03d0 con 0x7f663c09f490 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6640c80640 10 monclient(hunting): authenticate will time out at 1175.901556s 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f6639d9b640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f663c0a0a60 legacy=0x7f663c09ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f663ad9d640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 msgr2=0x7f663c138de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f663ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 0x7f663c138de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f663a59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f663c131aa0 0x7f663c131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.432+0000 7f663a59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f663c131aa0 0x7f663c131e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47232/0 (socket says 172.21.15.67:47232) 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 -- 172.21.15.67:0/1916227305 learned_addr learned my addr 172.21.15.67:0/1916227305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): get_auth_request con 0x7f663c131aa0 auth_method 0 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient(hunting): handle_auth_done global_id 104223 payload 274 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 -- 172.21.15.67:0/1916227305 >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 msgr2=0x7f663c138de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 --2- 172.21.15.67:0/1916227305 >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 0x7f663c138de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 -- 172.21.15.67:0/1916227305 >> v1:172.21.15.67:6789/0 conn(0x7f663c0a0a60 legacy=0x7f663c09ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 monclient: found mon.noname-a 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:03.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.433+0000 7f663a59c640 1 -- 172.21.15.67:0/1916227305 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0a1540 con 0x7f663c131aa0 2024-09-15T01:59:03.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663a59c640 1 -- 172.21.15.67:0/1916227305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 msgr2=0x7f663c131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:03.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663a59c640 1 --2- 172.21.15.67:0/1916227305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c131e70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f663c09ccf0 tx=0x7f662c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62b34ae0478410 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:03.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 1 -- 172.21.15.67:0/1916227305 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c032440 con 0x7f663c131aa0 2024-09-15T01:59:03.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: dump: 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:03.449 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: _finish_auth 0 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 1 -- 172.21.15.67:0/1916227305 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c0325a0 con 0x7f663c131aa0 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 1 -- 172.21.15.67:0/1916227305 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c032940 con 0x7f663c131aa0 2024-09-15T01:59:03.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f6640c80640 5 monclient: authenticate success, global_id 104223 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f6640c80640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f6640c80640 4 set_mon_vals no callback set 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.436+0000 7f663959a640 10 monclient: dump: 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:03.451 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 10 monclient: shutdown 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 -- 172.21.15.67:0/1916227305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 msgr2=0x7f663c131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 --2- 172.21.15.67:0/1916227305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c131e70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f663c09ccf0 tx=0x7f662c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 -- 172.21.15.67:0/1916227305 shutdown_connections 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 --2- 172.21.15.67:0/1916227305 >> v2:172.21.15.67:3300/0 conn(0x7f663c09f490 0x7f663c138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 --2- 172.21.15.67:0/1916227305 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c131e70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 -- 172.21.15.67:0/1916227305 >> 172.21.15.67:0/1916227305 conn(0x7f663c071390 msgr2=0x7f663c0a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 -- 172.21.15.67:0/1916227305 shutdown_connections 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 1 -- 172.21.15.67:0/1916227305 wait complete. 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 10 monclient: build_initial_monmap 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.437+0000 7f6640c80640 10 monclient: monmap: 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:03.453 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 Processor -- start 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 -- start start 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient: init 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 0x7f663c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-b con 0x7f663c09f490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-c con 0x7f663c0a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): picked mon.noname-a con 0x7f663c131aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663a59c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 msgr2=0x7f663c141e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c11b940 con 0x7f663c131aa0 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c1369d0 con 0x7f663c09f490 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 0x7f663c141e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0a0f40 con 0x7f663c0a0a60 2024-09-15T01:59:03.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6640c80640 10 monclient(hunting): authenticate will time out at 1175.907613s 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44482/0 (socket says 172.21.15.67:44482) 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 1 -- 172.21.15.67:0/1989921151 learned_addr learned my addr 172.21.15.67:0/1989921151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): get_auth_request con 0x7f663c0a0a60 auth_method 0 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:03.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): get_auth_request con 0x7f663c131aa0 auth_method 0 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.438+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 10 monclient(hunting): handle_auth_done global_id 94382 payload 824 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 -- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 msgr2=0x7f663c141e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:03.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 0x7f663c141e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 -- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 msgr2=0x7f663c134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c134ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 monclient: found mon.noname-c 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.439+0000 7f6639d9b640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c009450 con 0x7f663c0a0a60 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6639d9b640 1 -- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 msgr2=0x7f663c1349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6639d9b640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f663000b630 tx=0x7f66300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630015070 con 0x7f663c0a0a60 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:03.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: dump: 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:03.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 5 monclient: authenticate success, global_id 94382 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 10 monclient: _renew_subs 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c0a4500 con 0x7f663c0a0a60 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630003d90 con 0x7f663c0a0a60 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630006380 con 0x7f663c0a0a60 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f662a7fc640 10 monclient: dump: 2024-09-15T01:59:03.460 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6638d99640 4 set_mon_vals no callback set 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 10 monclient: _renew_subs 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.441+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c131240 con 0x7f663c0a0a60 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f6640c80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f6640c80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f6640c80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6630010440 con 0x7f663c0a0a60 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c0639e0 con 0x7f663c0a0a60 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f662a7fc640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 0x7f661c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.442+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(188..188 src has 148..188) v4 ==== 8633+0+0 (secure 0 0 0) 0x7f663004c430 con 0x7f663c0a0a60 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.443+0000 7f663a59c640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 0x7f661c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.443+0000 7f663a59c640 10 monclient: get_auth_request con 0x7f661c0386f0 auth_method 0 2024-09-15T01:59:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.445+0000 7f663a59c640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 0x7f661c03aba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f663c0a3120 tx=0x7f662c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:03.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.447+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6630018a70 con 0x7f663c0a0a60 2024-09-15T01:59:03.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.447+0000 7f662a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:03.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.447+0000 7f662a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:03.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.615+0000 7f6640c80640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.615+0000 7f6640c80640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:03.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.615+0000 7f6640c80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:03.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:03.615+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f663c0a4160 con 0x7f663c0a0a60 2024-09-15T01:59:04.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.243+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v189) v1 ==== 166+0+0 (secure 0 0 0) 0x7f6630018a70 con 0x7f663c0a0a60 2024-09-15T01:59:04.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.243+0000 7f662a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.243+0000 7f662a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-15T01:59:04.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.350+0000 7f6640c80640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.350+0000 7f6640c80640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.350+0000 7f6640c80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.350+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f663c137fc0 con 0x7f663c0a0a60 2024-09-15T01:59:04.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.351+0000 7f662a7fc640 1 -- 172.21.15.67:0/1989921151 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v189) v1 ==== 145+0+0 (secure 0 0 0) 0x7f6630018a70 con 0x7f663c0a0a60 2024-09-15T01:59:04.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.351+0000 7f662a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.351+0000 7f662a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-15T01:59:04.354 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h5' already exists 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.353+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 msgr2=0x7f661c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.353+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 0x7f661c03aba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f663c0a3120 tx=0x7f662c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 10 monclient: shutdown 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 msgr2=0x7f663c1349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:04.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f663000b630 tx=0x7f66300091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 shutdown_connections 2024-09-15T01:59:04.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f661c0386f0 0x7f661c03aba0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f663c131aa0 0x7f663c134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663c0a0a60 0x7f663c1349a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 --2- 172.21.15.67:0/1989921151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f663c09f490 0x7f663c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 >> 172.21.15.67:0/1989921151 conn(0x7f663c071390 msgr2=0x7f663c09dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:04.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 shutdown_connections 2024-09-15T01:59:04.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.354+0000 7f6640c80640 1 -- 172.21.15.67:0/1989921151 wait complete. 2024-09-15T01:59:04.365 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-15T01:59:04.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:04.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 10 monclient: build_initial_monmap 2024-09-15T01:59:04.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 10 monclient: monmap: 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:04.443441+0000 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:04.443441+0000 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.450 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:04.450 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:04.450 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:04.450 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:04.450 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:04.451 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 1 Processor -- start 2024-09-15T01:59:04.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 1 -- start start 2024-09-15T01:59:04.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.442+0000 7ff5f839d640 10 monclient: init 2024-09-15T01:59:04.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:04.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T01:59:04.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 0x7ff5f009cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f0096a70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5f0096e40 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f0134f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f009ad30 con 0x7ff5f0096e40 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:04.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f009bf70 con 0x7ff5f0096a70 2024-09-15T01:59:04.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:04.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f009ba10 con 0x7ff5f0134f20 2024-09-15T01:59:04.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:04.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f839d640 10 monclient(hunting): authenticate will time out at 1176.912535s 2024-09-15T01:59:04.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f6139640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 msgr2=0x7ff5f009cee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f5938640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7ff5f0096e40 legacy=0x7ff5f0097240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f6139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 0x7ff5f009cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44496/0 (socket says 172.21.15.67:44496) 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 1 -- 172.21.15.67:0/23289922 learned_addr learned my addr 172.21.15.67:0/23289922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:04.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): get_auth_request con 0x7ff5f0134f20 auth_method 0 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.443+0000 7ff5f693a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:04.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 10 monclient(hunting): handle_auth_done global_id 94385 payload 274 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 1 -- 172.21.15.67:0/23289922 >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 msgr2=0x7ff5f009cee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 1 --2- 172.21.15.67:0/23289922 >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 0x7ff5f009cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 1 -- 172.21.15.67:0/23289922 >> v1:172.21.15.67:6789/0 conn(0x7ff5f0096e40 legacy=0x7ff5f0097240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:04.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 1 monclient: found mon.noname-b 2024-09-15T01:59:04.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:04.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.444+0000 7ff5f693a640 1 -- 172.21.15.67:0/23289922 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f009c220 con 0x7ff5f0134f20 2024-09-15T01:59:04.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f693a640 1 -- 172.21.15.67:0/23289922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 msgr2=0x7ff5f0137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f693a640 1 --2- 172.21.15.67:0/23289922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009f20 tx=0x7ff5e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e2b970da6e6450f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 1 -- 172.21.15.67:0/23289922 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8007610 con 0x7ff5f0134f20 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: dump: 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:04.470 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: _finish_auth 0 2024-09-15T01:59:04.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 1 -- 172.21.15.67:0/23289922 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e802d050 con 0x7ff5f0134f20 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f839d640 5 monclient: authenticate success, global_id 94385 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f839d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f839d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 1 -- 172.21.15.67:0/23289922 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80079a0 con 0x7ff5f0134f20 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f839d640 4 set_mon_vals no callback set 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.447+0000 7ff5f5137640 10 monclient: dump: 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 10 monclient: shutdown 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 -- 172.21.15.67:0/23289922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 msgr2=0x7ff5f0137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 --2- 172.21.15.67:0/23289922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009f20 tx=0x7ff5e8009840 comp rx=0 tx=0).stop 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 -- 172.21.15.67:0/23289922 shutdown_connections 2024-09-15T01:59:04.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 --2- 172.21.15.67:0/23289922 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0134f20 0x7ff5f0137300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 --2- 172.21.15.67:0/23289922 >> v2:172.21.15.67:3300/0 conn(0x7ff5f0096a70 0x7ff5f009cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 -- 172.21.15.67:0/23289922 >> 172.21.15.67:0/23289922 conn(0x7ff5f00a1020 msgr2=0x7ff5f00a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 -- 172.21.15.67:0/23289922 shutdown_connections 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 1 -- 172.21.15.67:0/23289922 wait complete. 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 10 monclient: build_initial_monmap 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.448+0000 7ff5f839d640 10 monclient: monmap: 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:04.474 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 Processor -- start 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 -- start start 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient: init 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5f0096a70 0x7ff5f011b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f0096a70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f0096e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 0x7ff5f011bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f0134f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:04.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0132c00 con 0x7ff5f0096a70 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f009bc40 con 0x7ff5f0134f20 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f006e450 con 0x7ff5f0096e40 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f839d640 10 monclient(hunting): authenticate will time out at 1176.918617s 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f693a640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 msgr2=0x7ff5f011bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f693a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 0x7ff5f011bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44510/0 (socket says 172.21.15.67:44510) 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 1 -- 172.21.15.67:0/2873226672 learned_addr learned my addr 172.21.15.67:0/2873226672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): get_auth_request con 0x7ff5f0096e40 auth_method 0 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f6139640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5f0096a70 0x7ff5f011b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f6139640 10 monclient(hunting): get_auth_request con 0x7ff5f0096a70 auth_method 0 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f6139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f6139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f6139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.449+0000 7ff5f5938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f6139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f6139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:04.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f6139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 10 monclient(hunting): handle_auth_done global_id 94388 payload 824 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 -- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 msgr2=0x7ff5f011bb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 0x7ff5f011bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 -- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5f0096a70 msgr2=0x7ff5f011b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5f0096a70 0x7ff5f011b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 10 monclient: _finish_hunting 0 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 monclient: found mon.noname-c 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.450+0000 7ff5f5938640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e8009ae0 con 0x7ff5f0096e40 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f5938640 1 -- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 msgr2=0x7ff5f011b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f5938640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00b5f0 tx=0x7ff5ec009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec003cc0 con 0x7ff5f0096e40 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: dump: 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec003e40 con 0x7ff5f0096e40 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec005340 con 0x7ff5f0096e40 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 5 monclient: authenticate success, global_id 94388 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 10 monclient: _renew_subs 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f011c9b0 con 0x7ff5f0096e40 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f4936640 4 set_mon_vals no callback set 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5e67fc640 10 monclient: dump: 2024-09-15T01:59:04.480 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 10 monclient: _renew_subs 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.452+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f011d580 con 0x7ff5f0096e40 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff5ec010480 con 0x7ff5f0096e40 2024-09-15T01:59:04.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5f839d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5f839d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5f839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f00639e0 con 0x7ff5f0096e40 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5e67fc640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 0x7ff5d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.453+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(189..189 src has 148..189) v4 ==== 8971+0+0 (secure 0 0 0) 0x7ff5ec04c900 con 0x7ff5f0096e40 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.454+0000 7ff5f6139640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 0x7ff5d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.454+0000 7ff5f6139640 10 monclient: get_auth_request con 0x7ff5d40386f0 auth_method 0 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.456+0000 7ff5f6139640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 0x7ff5d403aba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0003850 tx=0x7ff5e00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.458+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff5ec019b80 con 0x7ff5f0096e40 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.458+0000 7ff5e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:04.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.458+0000 7ff5e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.623+0000 7ff5f839d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.623+0000 7ff5f839d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.623+0000 7ff5f839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:04.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:04.623+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff5f0067290 con 0x7ff5f0096e40 2024-09-15T01:59:05.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.244+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v190) v1 ==== 166+0+0 (secure 0 0 0) 0x7ff5ec0193c0 con 0x7ff5f0096e40 2024-09-15T01:59:05.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.244+0000 7ff5e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.244+0000 7ff5e67fc640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-15T01:59:05.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.347+0000 7ff5f839d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.347+0000 7ff5f839d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.347+0000 7ff5f839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.347+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff5f011e4e0 con 0x7ff5f0096e40 2024-09-15T01:59:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.349+0000 7ff5e67fc640 1 -- 172.21.15.67:0/2873226672 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v190) v1 ==== 145+0+0 (secure 0 0 0) 0x7ff5ec0195a0 con 0x7ff5f0096e40 2024-09-15T01:59:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.349+0000 7ff5e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.349+0000 7ff5e67fc640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-15T01:59:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h6' already exists 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 msgr2=0x7ff5d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 0x7ff5d403aba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0003850 tx=0x7ff5e00090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 10 monclient: shutdown 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 msgr2=0x7ff5f011b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00b5f0 tx=0x7ff5ec009120 comp rx=0 tx=0).stop 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 shutdown_connections 2024-09-15T01:59:05.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5d40386f0 0x7ff5d403aba0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f0134f20 0x7ff5f011bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5f0096e40 0x7ff5f011b630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 --2- 172.21.15.67:0/2873226672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5f0096a70 0x7ff5f011b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 >> 172.21.15.67:0/2873226672 conn(0x7ff5f00a1020 msgr2=0x7ff5f00a0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 shutdown_connections 2024-09-15T01:59:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.350+0000 7ff5f839d640 1 -- 172.21.15.67:0/2873226672 wait complete. 2024-09-15T01:59:05.363 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:05.363 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-15T01:59:05.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 10 monclient: get_monmap_and_config 2024-09-15T01:59:05.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 10 monclient: build_initial_monmap 2024-09-15T01:59:05.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 10 monclient: monmap: 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:05.441283+0000 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:05.441283+0000 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 1 Processor -- start 2024-09-15T01:59:05.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 1 -- start start 2024-09-15T01:59:05.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.440+0000 7fb2a7ec9640 10 monclient: init 2024-09-15T01:59:05.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2a0131e70 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2a0098420 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb2a01381b0 0x7fb2a013a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a01381b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a00a13e0 con 0x7fb2a0131e70 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a00a0440 con 0x7fb2a0098420 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a00a01e0 con 0x7fb2a01381b0 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a7ec9640 10 monclient(hunting): authenticate will time out at 1177.910451s 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a5464640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fb2a0098420 legacy=0x7fb2a0098850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:05.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a5c65640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb2a0131e70 0x7fb2a0132240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:38140/0 (socket says 172.21.15.67:38140) 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a5c65640 1 -- 172.21.15.67:0/1419228474 learned_addr learned my addr 172.21.15.67:0/1419228474 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 1 --2- 172.21.15.67:0/1419228474 >> v2:172.21.15.17:3300/0 conn(0x7fb2a01381b0 0x7fb2a013a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): get_auth_request con 0x7fb2a01381b0 auth_method 0 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.441+0000 7fb2a6466640 10 monclient(hunting): handle_auth_done global_id 104232 payload 274 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 1 -- 172.21.15.67:0/1419228474 >> v1:172.21.15.67:6789/0 conn(0x7fb2a0098420 legacy=0x7fb2a0098850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 1 -- 172.21.15.67:0/1419228474 >> v1:172.21.15.17:6789/0 conn(0x7fb2a0131e70 legacy=0x7fb2a0132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:05.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 10 monclient: _finish_hunting 0 2024-09-15T01:59:05.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 1 monclient: found mon.noname-a 2024-09-15T01:59:05.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:05.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:05.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.442+0000 7fb2a6466640 1 -- 172.21.15.67:0/1419228474 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a00a19b0 con 0x7fb2a01381b0 2024-09-15T01:59:05.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.444+0000 7fb2a6466640 1 -- 172.21.15.67:0/1419228474 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a01381b0 msgr2=0x7fb2a013a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:05.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.444+0000 7fb2a6466640 1 --2- 172.21.15.67:0/1419228474 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a01381b0 0x7fb2a013a590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb290009e60 tx=0x7fb290009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9272b4330792930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:05.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 1 -- 172.21.15.67:0/1419228474 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290007630 con 0x7fb2a01381b0 2024-09-15T01:59:05.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: dump: 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: _finish_auth 0 2024-09-15T01:59:05.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 1 -- 172.21.15.67:0/1419228474 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29002d050 con 0x7fb2a01381b0 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 1 -- 172.21.15.67:0/1419228474 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900079c0 con 0x7fb2a01381b0 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a4c63640 10 monclient: dump: 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 5 monclient: authenticate success, global_id 104232 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 4 set_mon_vals no callback set 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 10 monclient: shutdown 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/1419228474 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a01381b0 msgr2=0x7fb2a013a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:05.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/1419228474 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a01381b0 0x7fb2a013a590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb290009e60 tx=0x7fb290009780 comp rx=0 tx=0).stop 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/1419228474 shutdown_connections 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/1419228474 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a01381b0 0x7fb2a013a590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/1419228474 >> 172.21.15.67:0/1419228474 conn(0x7fb2a00a5c10 msgr2=0x7fb2a00a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/1419228474 shutdown_connections 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.445+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/1419228474 wait complete. 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.446+0000 7fb2a7ec9640 10 monclient: build_initial_monmap 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.446+0000 7fb2a7ec9640 10 monclient: monmap: 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.446+0000 7fb2a7ec9640 1 Processor -- start 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.446+0000 7fb2a7ec9640 1 -- start start 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.446+0000 7fb2a7ec9640 10 monclient: init 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a0098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a0131e70 0x7fb2a009d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a0131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 0x7fb2a009d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a01381b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a009ed20 con 0x7fb2a0131e70 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0133b10 con 0x7fb2a01381b0 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a009e7a0 con 0x7fb2a0098420 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a7ec9640 10 monclient(hunting): authenticate will time out at 1177.916183s 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a6466640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 msgr2=0x7fb2a009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a6466640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 0x7fb2a009d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44522/0 (socket says 172.21.15.67:44522) 2024-09-15T01:59:05.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 -- 172.21.15.67:0/3933123616 learned_addr learned my addr 172.21.15.67:0/3933123616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:05.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): get_auth_request con 0x7fb2a0098420 auth_method 0 2024-09-15T01:59:05.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:05.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:05.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:05.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a0131e70 0x7fb2a009d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): get_auth_request con 0x7fb2a0131e70 auth_method 0 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient(hunting): handle_auth_done global_id 94391 payload 824 2024-09-15T01:59:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 -- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 msgr2=0x7fb2a009d6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 0x7fb2a009d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 -- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a0131e70 msgr2=0x7fb2a009d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a0131e70 0x7fb2a009d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient: _finish_hunting 0 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 monclient: found mon.noname-c 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.447+0000 7fb2a5c65640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290009a20 con 0x7fb2a0098420 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a5c65640 1 -- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 msgr2=0x7fb2a009cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a5c65640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29800c610 tx=0x7fb298004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298005cb0 con 0x7fb2a0098420 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:05.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: dump: 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.463 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb298005e30 con 0x7fb2a0098420 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 5 monclient: authenticate success, global_id 94391 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 10 monclient: _renew_subs 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a013a6b0 con 0x7fb2a0098420 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298007340 con 0x7fb2a0098420 2024-09-15T01:59:05.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29dffb640 10 monclient: dump: 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 10 monclient: _renew_subs 2024-09-15T01:59:05.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb29ffff640 4 set_mon_vals no callback set 2024-09-15T01:59:05.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.450+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a009f370 con 0x7fb2a0098420 2024-09-15T01:59:05.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb29801b400 con 0x7fb2a0098420 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb2a7ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb2a7ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb2a7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb29dffb640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 0x7fb28c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a00639e0 con 0x7fb2a0098420 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.451+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(190..190 src has 148..190) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb29804c860 con 0x7fb2a0098420 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.452+0000 7fb2a5464640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 0x7fb28c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:05.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.452+0000 7fb2a5464640 10 monclient: get_auth_request con 0x7fb28c0386f0 auth_method 0 2024-09-15T01:59:05.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.454+0000 7fb2a5464640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 0x7fb28c03aba0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb294003850 tx=0x7fb2940090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:05.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.456+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb298012760 con 0x7fb2a0098420 2024-09-15T01:59:05.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.456+0000 7fb29dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:05.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.456+0000 7fb29dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.622+0000 7fb2a7ec9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.622+0000 7fb2a7ec9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.622+0000 7fb2a7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:05.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:05.622+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb2a0132ae0 con 0x7fb2a0098420 2024-09-15T01:59:06.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.252+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v191) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb2980290b0 con 0x7fb2a0098420 2024-09-15T01:59:06.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.252+0000 7fb29dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.252+0000 7fb29dffb640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-15T01:59:06.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.356+0000 7fb2a7ec9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.356+0000 7fb2a7ec9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.356+0000 7fb2a7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.356+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb2a009c020 con 0x7fb2a0098420 2024-09-15T01:59:06.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.357+0000 7fb29dffb640 1 -- 172.21.15.67:0/3933123616 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v191) v1 ==== 145+0+0 (secure 0 0 0) 0x7fb29801a360 con 0x7fb2a0098420 2024-09-15T01:59:06.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.357+0000 7fb29dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.360 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.357+0000 7fb29dffb640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-15T01:59:06.360 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h7' already exists 2024-09-15T01:59:06.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 msgr2=0x7fb28c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 0x7fb28c03aba0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb294003850 tx=0x7fb2940090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 10 monclient: shutdown 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 msgr2=0x7fb2a009cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29800c610 tx=0x7fb298004960 comp rx=0 tx=0).stop 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 shutdown_connections 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb28c0386f0 0x7fb28c03aba0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2a0131e70 0x7fb2a009d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a01381b0 0x7fb2a009d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 --2- 172.21.15.67:0/3933123616 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2a0098420 0x7fb2a009cc70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 >> 172.21.15.67:0/3933123616 conn(0x7fb2a00a5c10 msgr2=0x7fb2a00a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.359+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 shutdown_connections 2024-09-15T01:59:06.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.360+0000 7fb2a7ec9640 1 -- 172.21.15.67:0/3933123616 wait complete. 2024-09-15T01:59:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 10 monclient: get_monmap_and_config 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 10 monclient: build_initial_monmap 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 10 monclient: monmap: 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:06.449233+0000 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:06.449233+0000 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:06.451 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 1 Processor -- start 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 1 -- start start 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.448+0000 7f9a45ed4640 10 monclient: init 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 0x7f9a40132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a40131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 0x7f9a40098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f9a40098420 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:06.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a40068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a400a13e0 con 0x7f9a40098420 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a400a0440 con 0x7f9a40131e70 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400a01e0 con 0x7f9a40068e70 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a45ed4640 10 monclient(hunting): authenticate will time out at 1178.918361s 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3f7fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 msgr2=0x7f9a40132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:06.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 0x7f9a40132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 0x7f9a40098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44530/0 (socket says 172.21.15.67:44530) 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 0x7f9a40098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47278/0 (socket says 172.21.15.67:47278) 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 1 -- 172.21.15.67:0/992635901 learned_addr learned my addr 172.21.15.67:0/992635901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request con 0x7f9a40068e70 auth_method 0 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): get_auth_request con 0x7f9a40098420 auth_method 0 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:06.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.449+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_done global_id 94394 payload 274 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 -- 172.21.15.67:0/992635901 >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 msgr2=0x7f9a40132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 --2- 172.21.15.67:0/992635901 >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 0x7f9a40132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 -- 172.21.15.67:0/992635901 >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 msgr2=0x7f9a40098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 --2- 172.21.15.67:0/992635901 >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 0x7f9a40098850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 monclient: found mon.noname-b 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.450+0000 7f9a3ffff640 1 -- 172.21.15.67:0/992635901 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a400a19b0 con 0x7f9a40068e70 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3ffff640 1 -- 172.21.15.67:0/992635901 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 msgr2=0x7f9a400692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3ffff640 1 --2- 172.21.15.67:0/992635901 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9a400a6250 tx=0x7f9a38009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8dcd6c2f1bea82e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 1 -- 172.21.15.67:0/992635901 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38007610 con 0x7f9a40068e70 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: dump: 2024-09-15T01:59:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 1 -- 172.21.15.67:0/992635901 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3802d050 con 0x7f9a40068e70 2024-09-15T01:59:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 1 -- 172.21.15.67:0/992635901 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380079a0 con 0x7f9a40068e70 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a3e7fc640 10 monclient: dump: 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:06.465 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a45ed4640 5 monclient: authenticate success, global_id 94394 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a45ed4640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.453+0000 7f9a45ed4640 4 set_mon_vals no callback set 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 10 monclient: shutdown 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 -- 172.21.15.67:0/992635901 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 msgr2=0x7f9a400692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/992635901 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9a400a6250 tx=0x7f9a38009840 comp rx=0 tx=0).stop 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 -- 172.21.15.67:0/992635901 shutdown_connections 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/992635901 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a400692f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/992635901 >> v2:172.21.15.17:3300/0 conn(0x7f9a40098420 0x7f9a40098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/992635901 >> v2:172.21.15.67:3300/0 conn(0x7f9a40131e70 0x7f9a40132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 -- 172.21.15.67:0/992635901 >> 172.21.15.67:0/992635901 conn(0x7f9a400a5c10 msgr2=0x7f9a400a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:06.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 -- 172.21.15.67:0/992635901 shutdown_connections 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 1 -- 172.21.15.67:0/992635901 wait complete. 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 10 monclient: build_initial_monmap 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.454+0000 7f9a45ed4640 10 monclient: monmap: 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 Processor -- start 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 -- start start 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient: init 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a40068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a40098420 0x7f9a4012ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f9a40098420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 0x7f9a4012f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a40131e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400a0d80 con 0x7f9a40098420 2024-09-15T01:59:06.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400a3740 con 0x7f9a40131e70 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4011f070 con 0x7f9a40068e70 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a45ed4640 10 monclient(hunting): authenticate will time out at 1178.924698s 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3ffff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 msgr2=0x7f9a4012f0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 0x7f9a4012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a40098420 0x7f9a4012ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44542/0 (socket says 172.21.15.67:44542) 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 1 -- 172.21.15.67:0/2617755940 learned_addr learned my addr 172.21.15.67:0/2617755940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): get_auth_request con 0x7f9a40068e70 auth_method 0 2024-09-15T01:59:06.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3effd640 10 monclient(hunting): get_auth_request con 0x7f9a40098420 auth_method 0 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.455+0000 7f9a3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 10 monclient(hunting): handle_auth_done global_id 94397 payload 824 2024-09-15T01:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 -- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 msgr2=0x7f9a4012f0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 0x7f9a4012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 -- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a40098420 msgr2=0x7f9a4012ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a40098420 0x7f9a4012ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 monclient: found mon.noname-c 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.456+0000 7f9a3f7fe640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38009ae0 con 0x7f9a40068e70 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a3f7fe640 1 -- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 msgr2=0x7f9a40141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a3f7fe640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000b660 tx=0x7f9a300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30014030 con 0x7f9a40068e70 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: dump: 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:06.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 5 monclient: authenticate success, global_id 94397 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 10 monclient: _renew_subs 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30003d90 con 0x7f9a40068e70 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a40141580 con 0x7f9a40068e70 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30006380 con 0x7f9a40068e70 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a277fe640 10 monclient: dump: 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a3dffb640 4 set_mon_vals no callback set 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 10 monclient: _renew_subs 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.458+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4011ed00 con 0x7f9a40068e70 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9a30017070 con 0x7f9a40068e70 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a45ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a45ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:06.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a45ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4011d230 con 0x7f9a40068e70 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a277fe640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 0x7f9a2c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(191..191 src has 148..191) v4 ==== 9391+0+0 (secure 0 0 0) 0x7f9a30015070 con 0x7f9a40068e70 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a3effd640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 0x7f9a2c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.459+0000 7f9a3effd640 10 monclient: get_auth_request con 0x7f9a2c0386f0 auth_method 0 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.462+0000 7f9a3effd640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 0x7f9a2c03aba0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9a40131df0 tx=0x7f9a340090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:06.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.464+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a30006770 con 0x7f9a40068e70 2024-09-15T01:59:06.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.464+0000 7f9a277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:06.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.464+0000 7f9a277fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:06.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.632+0000 7f9a45ed4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.632+0000 7f9a45ed4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:06.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.632+0000 7f9a45ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:06.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:06.632+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9a400639e0 con 0x7f9a40068e70 2024-09-15T01:59:07.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.253+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v192) v1 ==== 166+0+0 (secure 0 0 0) 0x7f9a300193d0 con 0x7f9a40068e70 2024-09-15T01:59:07.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.253+0000 7f9a277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.253+0000 7f9a277fe640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-15T01:59:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.360+0000 7f9a45ed4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.360+0000 7f9a45ed4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.360+0000 7f9a45ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.360+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9a4011f600 con 0x7f9a40068e70 2024-09-15T01:59:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.362+0000 7f9a277fe640 1 -- 172.21.15.67:0/2617755940 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v192) v1 ==== 145+0+0 (secure 0 0 0) 0x7f9a3000e360 con 0x7f9a40068e70 2024-09-15T01:59:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.362+0000 7f9a277fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.362+0000 7f9a277fe640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-15T01:59:07.364 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h8' already exists 2024-09-15T01:59:07.366 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 msgr2=0x7f9a2c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 0x7f9a2c03aba0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9a40131df0 tx=0x7f9a340090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 10 monclient: shutdown 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 msgr2=0x7f9a40141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000b660 tx=0x7f9a300091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 shutdown_connections 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a2c0386f0 0x7f9a2c03aba0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a40098420 0x7f9a4012ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40131e70 0x7f9a4012f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 --2- 172.21.15.67:0/2617755940 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9a40068e70 0x7f9a40141040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 >> 172.21.15.67:0/2617755940 conn(0x7f9a400a5c10 msgr2=0x7f9a40097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 shutdown_connections 2024-09-15T01:59:07.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.364+0000 7f9a45ed4640 1 -- 172.21.15.67:0/2617755940 wait complete. 2024-09-15T01:59:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:07.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.451+0000 7f9d8c39d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.451+0000 7f9d8c39d640 10 monclient: build_initial_monmap 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.451+0000 7f9d8c39d640 10 monclient: monmap: 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:07.452823+0000 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:07.452823+0000 2024-09-15T01:59:07.454 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 Processor -- start 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 -- start start 2024-09-15T01:59:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient: init 2024-09-15T01:59:07.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:07.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d84131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 0x7f9d84098630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d84098260 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d84098b70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d84096a70 con 0x7f9d84098b70 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d84098fd0 con 0x7f9d84098260 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8406a6b0 con 0x7f9d84131f90 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8c39d640 10 monclient(hunting): authenticate will time out at 1179.921905s 2024-09-15T01:59:07.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d89938640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 msgr2=0x7f9d84098630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d89938640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 0x7f9d84098630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.452+0000 7f9d8a93a640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f9d84098b70 legacy=0x7f9d84068e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44554/0 (socket says 172.21.15.67:44554) 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 -- 172.21.15.67:0/2389402113 learned_addr learned my addr 172.21.15.67:0/2389402113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): get_auth_request con 0x7f9d84131f90 auth_method 0 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient(hunting): handle_auth_done global_id 94400 payload 274 2024-09-15T01:59:07.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 -- 172.21.15.67:0/2389402113 >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 msgr2=0x7f9d84098630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 --2- 172.21.15.67:0/2389402113 >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 0x7f9d84098630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 -- 172.21.15.67:0/2389402113 >> v1:172.21.15.67:6789/0 conn(0x7f9d84098b70 legacy=0x7f9d84068e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient: _finish_hunting 0 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 monclient: found mon.noname-b 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.460 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.453+0000 7f9d8a139640 1 -- 172.21.15.67:0/2389402113 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d840a16e0 con 0x7f9d84131f90 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.456+0000 7f9d8a139640 1 -- 172.21.15.67:0/2389402113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 msgr2=0x7f9d84132410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.456+0000 7f9d8a139640 1 --2- 172.21.15.67:0/2389402113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d84064e70 tx=0x7f9d800091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196053da20c16858 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 1 -- 172.21.15.67:0/2389402113 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80032440 con 0x7f9d84131f90 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: dump: 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: _finish_auth 0 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 1 -- 172.21.15.67:0/2389402113 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d800325a0 con 0x7f9d84131f90 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 1 -- 172.21.15.67:0/2389402113 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80032940 con 0x7f9d84131f90 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d8c39d640 5 monclient: authenticate success, global_id 94400 2024-09-15T01:59:07.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d8c39d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d8c39d640 4 set_mon_vals no callback set 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.457+0000 7f9d89137640 10 monclient: dump: 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:07.466 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 10 monclient: shutdown 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 -- 172.21.15.67:0/2389402113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 msgr2=0x7f9d84132410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/2389402113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d84064e70 tx=0x7f9d800091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 -- 172.21.15.67:0/2389402113 shutdown_connections 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/2389402113 >> v2:172.21.15.67:3300/0 conn(0x7f9d84098260 0x7f9d84098630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/2389402113 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84132410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 -- 172.21.15.67:0/2389402113 >> 172.21.15.67:0/2389402113 conn(0x7f9d84071690 msgr2=0x7f9d840a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 -- 172.21.15.67:0/2389402113 shutdown_connections 2024-09-15T01:59:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 1 -- 172.21.15.67:0/2389402113 wait complete. 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 10 monclient: build_initial_monmap 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.458+0000 7f9d8c39d640 10 monclient: monmap: 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 Processor -- start 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 -- start start 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient: init 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 0x7f9d8413de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d84098260 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d84098b70 0x7f9d8412f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d84098b70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d84131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d840a0d80 con 0x7f9d84098b70 2024-09-15T01:59:07.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84130e20 con 0x7f9d84098260 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8411d6a0 con 0x7f9d84131f90 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8c39d640 10 monclient(hunting): authenticate will time out at 1179.928676s 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a139640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 msgr2=0x7f9d8413de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a139640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 0x7f9d8413de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d89938640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d84098b70 0x7f9d8412f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44558/0 (socket says 172.21.15.67:44558) 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 1 -- 172.21.15.67:0/4012323412 learned_addr learned my addr 172.21.15.67:0/4012323412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 10 monclient(hunting): get_auth_request con 0x7f9d84131f90 auth_method 0 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d8a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d89938640 10 monclient(hunting): get_auth_request con 0x7f9d84098b70 auth_method 0 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d89938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d89938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.459+0000 7f9d89938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d89938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:07.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d89938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d89938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient(hunting): handle_auth_done global_id 94403 payload 824 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 -- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 msgr2=0x7f9d8413de10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 0x7f9d8413de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 -- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d84098b70 msgr2=0x7f9d8412f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d84098b70 0x7f9d8412f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 monclient: found mon.noname-c 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.460+0000 7f9d8a93a640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80009450 con 0x7f9d84131f90 2024-09-15T01:59:07.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8a93a640 1 -- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 msgr2=0x7f9d84141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:07.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8a93a640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00beb0 tx=0x7f9d7c0096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:07.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c016430 con 0x7f9d84131f90 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: dump: 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.475 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c0165d0 con 0x7f9d84131f90 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8c39d640 5 monclient: authenticate success, global_id 94403 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8c39d640 10 monclient: _renew_subs 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c016780 con 0x7f9d84131f90 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8411f7f0 con 0x7f9d84131f90 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d7a7fc640 10 monclient: dump: 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.462+0000 7f9d88936640 4 set_mon_vals no callback set 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 10 monclient: _renew_subs 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8411eaa0 con 0x7f9d84131f90 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8411f9e0 con 0x7f9d84131f90 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.463+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d7c012020 con 0x7f9d84131f90 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.464+0000 7f9d7a7fc640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 0x7f9d6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.464+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(192..192 src has 148..192) v4 ==== 9601+0+0 (secure 0 0 0) 0x7f9d7c04d230 con 0x7f9d84131f90 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.464+0000 7f9d8a139640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 0x7f9d6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.464+0000 7f9d8a139640 10 monclient: get_auth_request con 0x7f9d68038740 auth_method 0 2024-09-15T01:59:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.466+0000 7f9d8a139640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 0x7f9d6803abf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d80031000 tx=0x7f9d80032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:07.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.468+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d7c012ab0 con 0x7f9d84131f90 2024-09-15T01:59:07.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.468+0000 7f9d7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:07.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.468+0000 7f9d7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:07.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.632+0000 7f9d8c39d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.632+0000 7f9d8c39d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:07.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.632+0000 7f9d8c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:07.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:07.632+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9d8411fbf0 con 0x7f9d84131f90 2024-09-15T01:59:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.097+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v193) v1 ==== 166+0+0 (secure 0 0 0) 0x7f9d7c0122d0 con 0x7f9d84131f90 2024-09-15T01:59:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.097+0000 7f9d7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.097+0000 7f9d7a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-15T01:59:08.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.197+0000 7f9d8c39d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.197+0000 7f9d8c39d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.197+0000 7f9d8c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.197+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9d840639e0 con 0x7f9d84131f90 2024-09-15T01:59:08.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.199+0000 7f9d7a7fc640 1 -- 172.21.15.67:0/4012323412 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v193) v1 ==== 145+0+0 (secure 0 0 0) 0x7f9d7c0124b0 con 0x7f9d84131f90 2024-09-15T01:59:08.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.199+0000 7f9d7a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.199+0000 7f9d7a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-15T01:59:08.201 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h9' already exists 2024-09-15T01:59:08.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 msgr2=0x7f9d6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:08.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 0x7f9d6803abf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d80031000 tx=0x7f9d80032000 comp rx=0 tx=0).stop 2024-09-15T01:59:08.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 10 monclient: shutdown 2024-09-15T01:59:08.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 msgr2=0x7f9d84141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00beb0 tx=0x7f9d7c0096e0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 shutdown_connections 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d68038740 0x7f9d6803abf0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d84098b70 0x7f9d8412f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d84131f90 0x7f9d84141360 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 --2- 172.21.15.67:0/4012323412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d84098260 0x7f9d8413de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.200+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 >> 172.21.15.67:0/4012323412 conn(0x7f9d84071690 msgr2=0x7f9d84130340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.201+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 shutdown_connections 2024-09-15T01:59:08.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.201+0000 7f9d8c39d640 1 -- 172.21.15.67:0/4012323412 wait complete. 2024-09-15T01:59:08.212 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:08.212 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.287+0000 7fcb5fb5d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.287+0000 7fcb5fb5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.287+0000 7fcb5fb5d640 10 monclient: monmap: 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:08.288612+0000 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:08.288612+0000 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.287+0000 7fcb5fb5d640 1 Processor -- start 2024-09-15T01:59:08.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 -- start start 2024-09-15T01:59:08.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient: init 2024-09-15T01:59:08.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:08.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T01:59:08.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 0x7fcb581321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb58098190 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb58098560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcb58068dc0 0x7fcb580691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb58068dc0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb580a16c0 con 0x7fcb58068dc0 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb580a0700 con 0x7fcb58098190 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580a13e0 con 0x7fcb58098560 2024-09-15T01:59:08.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5fb5d640 10 monclient(hunting): authenticate will time out at 1180.757731s 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5d8f9640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 msgr2=0x7fcb581321b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.288+0000 7fcb5d8f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 0x7fcb581321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44566/0 (socket says 172.21.15.67:44566) 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 -- 172.21.15.67:0/1794524433 learned_addr learned my addr 172.21.15.67:0/1794524433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): get_auth_request con 0x7fcb58098560 auth_method 0 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:08.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 1 --2- 172.21.15.67:0/1794524433 >> v2:172.21.15.17:3300/0 conn(0x7fcb58068dc0 0x7fcb580691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): get_auth_request con 0x7fcb58068dc0 auth_method 0 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient(hunting): handle_auth_done global_id 94406 payload 274 2024-09-15T01:59:08.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 -- 172.21.15.67:0/1794524433 >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 msgr2=0x7fcb581321b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 --2- 172.21.15.67:0/1794524433 >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 0x7fcb581321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 -- 172.21.15.67:0/1794524433 >> v2:172.21.15.17:3300/0 conn(0x7fcb58068dc0 msgr2=0x7fcb580691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 --2- 172.21.15.67:0/1794524433 >> v2:172.21.15.17:3300/0 conn(0x7fcb58068dc0 0x7fcb580691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 monclient: found mon.noname-b 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.289+0000 7fcb5d0f8640 1 -- 172.21.15.67:0/1794524433 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb580a19a0 con 0x7fcb58098560 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5d0f8640 1 -- 172.21.15.67:0/1794524433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 msgr2=0x7fcb58098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:08.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5d0f8640 1 --2- 172.21.15.67:0/1794524433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb5809c360 tx=0x7fcb480091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ef1d2041065feb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 1 -- 172.21.15.67:0/1794524433 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48032440 con 0x7fcb58098560 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: dump: 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: _finish_auth 0 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 1 -- 172.21.15.67:0/1794524433 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb480325a0 con 0x7fcb58098560 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 1 -- 172.21.15.67:0/1794524433 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48032940 con 0x7fcb58098560 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5fb5d640 5 monclient: authenticate success, global_id 94406 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5fb5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:08.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5fb5d640 4 set_mon_vals no callback set 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.292+0000 7fcb5c8f7640 10 monclient: dump: 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 10 monclient: shutdown 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/1794524433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 msgr2=0x7fcb58098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/1794524433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb5809c360 tx=0x7fcb480091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/1794524433 shutdown_connections 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/1794524433 >> v2:172.21.15.17:3300/0 conn(0x7fcb58068dc0 0x7fcb580691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/1794524433 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb58098960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/1794524433 >> v2:172.21.15.67:3300/0 conn(0x7fcb58098190 0x7fcb581321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/1794524433 >> 172.21.15.67:0/1794524433 conn(0x7fcb580711e0 msgr2=0x7fcb580a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/1794524433 shutdown_connections 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/1794524433 wait complete. 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.293+0000 7fcb5fb5d640 10 monclient: monmap: 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:08.307 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 Processor -- start 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 -- start start 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient: init 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:08.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb58068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 0x7fcb5811e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb58098190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb5811e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb58098560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581318c0 con 0x7fcb58068dc0 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58098f70 con 0x7fcb58098190 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581204a0 con 0x7fcb58098560 2024-09-15T01:59:08.309 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5fb5d640 10 monclient(hunting): authenticate will time out at 1180.763846s 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5d0f8640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 msgr2=0x7fcb5811e400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5d0f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 0x7fcb5811e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5d8f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.294+0000 7fcb5e0fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb5811e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:47320/0 (socket says 172.21.15.67:47320) 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 1 -- 172.21.15.67:0/3925108549 learned_addr learned my addr 172.21.15.67:0/3925108549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): get_auth_request con 0x7fcb58068dc0 auth_method 0 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): get_auth_request con 0x7fcb58098560 auth_method 0 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient(hunting): handle_auth_done global_id 94409 payload 824 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 -- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 msgr2=0x7fcb5811e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 0x7fcb5811e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 -- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 msgr2=0x7fcb5811dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient: _finish_hunting 0 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 monclient: found mon.noname-c 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5e0fa640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009450 con 0x7fcb58098560 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.295+0000 7fcb5d8f9640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb5e0fa640 1 -- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 msgr2=0x7fcb5811e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb5e0fa640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb5811e940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ecc0 tx=0x7fcb5000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5000cce0 con 0x7fcb58098560 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.297+0000 7fcb4dffb640 10 monclient: dump: 2024-09-15T01:59:08.312 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 5 monclient: authenticate success, global_id 94409 2024-09-15T01:59:08.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5000f040 con 0x7fcb58098560 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 10 monclient: _renew_subs 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5809ea00 con 0x7fcb58098560 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb500146d0 con 0x7fcb58098560 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4dffb640 10 monclient: dump: 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 10 monclient: _renew_subs 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb4ffff640 4 set_mon_vals no callback set 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.298+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb58121840 con 0x7fcb58098560 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5fb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5fb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5fb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580639e0 con 0x7fcb58098560 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb50014b00 con 0x7fcb58098560 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb4dffb640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 0x7fcb3c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(193..193 src has 148..193) v4 ==== 9811+0+0 (secure 0 0 0) 0x7fcb5004dc50 con 0x7fcb58098560 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5d8f9640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 0x7fcb3c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.299+0000 7fcb5d8f9640 10 monclient: get_auth_request con 0x7fcb3c0386f0 auth_method 0 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.302+0000 7fcb5d8f9640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 0x7fcb3c03aba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009a00 tx=0x7fcb540091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:08.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.304+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb5001d400 con 0x7fcb58098560 2024-09-15T01:59:08.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.304+0000 7fcb4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:08.317 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.304+0000 7fcb4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:08.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.468+0000 7fcb5fb5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.468+0000 7fcb5fb5d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.468+0000 7fcb5fb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:08.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:08.468+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcb5811faa0 con 0x7fcb58098560 2024-09-15T01:59:09.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.370+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v194) v1 ==== 168+0+0 (secure 0 0 0) 0x7fcb50018030 con 0x7fcb58098560 2024-09-15T01:59:09.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.370+0000 7fcb4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.370+0000 7fcb4dffb640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-15T01:59:09.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.473+0000 7fcb5fb5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.473+0000 7fcb5fb5d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.473+0000 7fcb5fb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.473+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcb580008d0 con 0x7fcb58098560 2024-09-15T01:59:09.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.474+0000 7fcb4dffb640 1 -- 172.21.15.67:0/3925108549 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v194) v1 ==== 147+0+0 (secure 0 0 0) 0x7fcb50018070 con 0x7fcb58098560 2024-09-15T01:59:09.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.474+0000 7fcb4dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.474+0000 7fcb4dffb640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-15T01:59:09.477 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h10' already exists 2024-09-15T01:59:09.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 msgr2=0x7fcb3c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 0x7fcb3c03aba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009a00 tx=0x7fcb540091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 10 monclient: shutdown 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 msgr2=0x7fcb5811e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb5811e940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ecc0 tx=0x7fcb5000b040 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 shutdown_connections 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb3c0386f0 0x7fcb3c03aba0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb58098560 0x7fcb5811e940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb58098190 0x7fcb5811e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 --2- 172.21.15.67:0/3925108549 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb58068dc0 0x7fcb5811dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 >> 172.21.15.67:0/3925108549 conn(0x7fcb580711e0 msgr2=0x7fcb5809dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 shutdown_connections 2024-09-15T01:59:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.477+0000 7fcb5fb5d640 1 -- 172.21.15.67:0/3925108549 wait complete. 2024-09-15T01:59:09.489 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:09.489 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.565+0000 7f359fd1d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.565+0000 7f359fd1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.565+0000 7f359fd1d640 10 monclient: monmap: 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:09.568 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:09.566978+0000 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:09.566978+0000 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:09.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.566+0000 7f359fd1d640 1 Processor -- start 2024-09-15T01:59:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.566+0000 7f359fd1d640 1 -- start start 2024-09-15T01:59:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.566+0000 7f359fd1d640 10 monclient: init 2024-09-15T01:59:09.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3598096a70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-b con 0x7f3598096e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3598134f20 0x7f3598137300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-a con 0x7f3598134f20 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359dab9640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f3598096a70 legacy=0x7f359809cee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359809ad30 con 0x7f3598096a70 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359809bf70 con 0x7f3598134f20 2024-09-15T01:59:09.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359809d5f0 con 0x7f3598096e40 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359fd1d640 10 monclient(hunting): authenticate will time out at 1182.036271s 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3598134f20 0x7f3598137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44578/0 (socket says 172.21.15.67:44578) 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 -- 172.21.15.67:0/1275091881 learned_addr learned my addr 172.21.15.67:0/1275091881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): get_auth_request con 0x7f3598096e40 auth_method 0 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): get_auth_request con 0x7f3598134f20 auth_method 0 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient(hunting): handle_auth_done global_id 94412 payload 274 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 -- 172.21.15.67:0/1275091881 >> v2:172.21.15.17:3300/0 conn(0x7f3598134f20 msgr2=0x7f3598137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 --2- 172.21.15.67:0/1275091881 >> v2:172.21.15.17:3300/0 conn(0x7f3598134f20 0x7f3598137300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 -- 172.21.15.67:0/1275091881 >> v1:172.21.15.67:6789/0 conn(0x7f3598096a70 legacy=0x7f359809cee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 monclient: found mon.noname-b 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.567+0000 7f359d2b8640 1 -- 172.21.15.67:0/1275091881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359809ba10 con 0x7f3598096e40 2024-09-15T01:59:09.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.570+0000 7f359d2b8640 1 -- 172.21.15.67:0/1275091881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 msgr2=0x7f3598097210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:09.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.570+0000 7f359d2b8640 1 --2- 172.21.15.67:0/1275091881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f358c009910 tx=0x7f358c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355d9b1e53702c5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:09.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 1 -- 172.21.15.67:0/1275091881 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c02aea0 con 0x7f3598096e40 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: dump: 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:09.583 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: _finish_auth 0 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 1 -- 172.21.15.67:0/1275091881 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c032440 con 0x7f3598096e40 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 5 monclient: authenticate success, global_id 94412 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 1 -- 172.21.15.67:0/1275091881 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c0327e0 con 0x7f3598096e40 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 4 set_mon_vals no callback set 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:09.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359cab7640 10 monclient: dump: 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 10 monclient: shutdown 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:09.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 -- 172.21.15.67:0/1275091881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 msgr2=0x7f3598097210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 --2- 172.21.15.67:0/1275091881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f358c009910 tx=0x7f358c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 -- 172.21.15.67:0/1275091881 shutdown_connections 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 --2- 172.21.15.67:0/1275091881 >> v2:172.21.15.17:3300/0 conn(0x7f3598134f20 0x7f3598137300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 --2- 172.21.15.67:0/1275091881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598097210 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 -- 172.21.15.67:0/1275091881 >> 172.21.15.67:0/1275091881 conn(0x7f35980a1020 msgr2=0x7f35980a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 -- 172.21.15.67:0/1275091881 shutdown_connections 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.571+0000 7f359fd1d640 1 -- 172.21.15.67:0/1275091881 wait complete. 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.572+0000 7f359fd1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.572+0000 7f359fd1d640 10 monclient: monmap: 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:09.586 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.572+0000 7f359fd1d640 1 Processor -- start 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.572+0000 7f359fd1d640 1 -- start start 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.572+0000 7f359fd1d640 10 monclient: init 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:09.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 0x7f3598141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-b con 0x7f3598096a70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-c con 0x7f3598096e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3598134f20 0x7f35980985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359dab9640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 msgr2=0x7f3598141520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): picked mon.noname-a con 0x7f3598134f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598069b90 con 0x7f3598134f20 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981423b0 con 0x7f3598096a70 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359dab9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 0x7f3598141520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:09.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359809a100 con 0x7f3598096e40 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359fd1d640 10 monclient(hunting): authenticate will time out at 1182.042192s 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:44590/0 (socket says 172.21.15.67:44590) 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 -- 172.21.15.67:0/2132024905 learned_addr learned my addr 172.21.15.67:0/2132024905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): get_auth_request con 0x7f3598096e40 auth_method 0 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3598134f20 0x7f35980985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): get_auth_request con 0x7f3598134f20 auth_method 0 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359e2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient(hunting): handle_auth_done global_id 94415 payload 824 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 -- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 msgr2=0x7f3598141520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:09.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 0x7f3598141520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 -- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3598134f20 msgr2=0x7f35980985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3598134f20 0x7f35980985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 monclient: found mon.noname-c 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.573+0000 7f359d2b8640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c009450 con 0x7f3598096e40 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359d2b8640 1 -- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 msgr2=0x7f3598132ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359d2b8640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f358c035040 tx=0x7f358c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c02aea0 con 0x7f3598096e40 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:09.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: dump: 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:09.592 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 5 monclient: authenticate success, global_id 94415 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c000d10 con 0x7f3598096e40 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 10 monclient: _renew_subs 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3598120bf0 con 0x7f3598096e40 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c03d440 con 0x7f3598096e40 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:09.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3595ffb640 10 monclient: dump: 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 10 monclient: _renew_subs 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598141a60 con 0x7f3598096e40 2024-09-15T01:59:09.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.576+0000 7f3597fff640 4 set_mon_vals no callback set 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f359fd1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f358c039040 con 0x7f3598096e40 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f359fd1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f359fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35980639e0 con 0x7f3598096e40 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.577+0000 7f3595ffb640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 0x7f358403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.578+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(194..194 src has 148..194) v4 ==== 10024+0+0 (secure 0 0 0) 0x7f358c06ca30 con 0x7f3598096e40 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.578+0000 7f359dab9640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 0x7f358403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.578+0000 7f359dab9640 10 monclient: get_auth_request con 0x7f35840386f0 auth_method 0 2024-09-15T01:59:09.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.580+0000 7f359dab9640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 0x7f358403aba0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3588003cc0 tx=0x7f3588009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:09.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.582+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f358c02f2a0 con 0x7f3598096e40 2024-09-15T01:59:09.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.582+0000 7f3595ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:09.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.582+0000 7f3595ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.748+0000 7f359fd1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.748+0000 7f359fd1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:09.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.748+0000 7f359fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:09.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:09.748+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f359809c220 con 0x7f3598096e40 2024-09-15T01:59:10.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.546+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v195) v1 ==== 168+0+0 (secure 0 0 0) 0x7f358c0420b0 con 0x7f3598096e40 2024-09-15T01:59:10.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.546+0000 7f3595ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.546+0000 7f3595ffb640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-15T01:59:10.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.652+0000 7f359fd1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.652+0000 7f359fd1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.652+0000 7f359fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.652+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f359809c220 con 0x7f3598096e40 2024-09-15T01:59:10.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.654+0000 7f3595ffb640 1 -- 172.21.15.67:0/2132024905 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v195) v1 ==== 147+0+0 (secure 0 0 0) 0x7f358c037020 con 0x7f3598096e40 2024-09-15T01:59:10.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.654+0000 7f3595ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.654+0000 7f3595ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-15T01:59:10.657 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h11' already exists 2024-09-15T01:59:10.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 msgr2=0x7f358403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:10.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 0x7f358403aba0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3588003cc0 tx=0x7f3588009310 comp rx=0 tx=0).stop 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 10 monclient: shutdown 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 msgr2=0x7f3598132ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f358c035040 tx=0x7f358c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 shutdown_connections 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f35840386f0 0x7f358403aba0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3598134f20 0x7f35980985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3598096e40 0x7f3598132ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 --2- 172.21.15.67:0/2132024905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3598096a70 0x7f3598141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 >> 172.21.15.67:0/2132024905 conn(0x7f35980a1020 msgr2=0x7f359806cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:10.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 shutdown_connections 2024-09-15T01:59:10.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.656+0000 7f359fd1d640 1 -- 172.21.15.67:0/2132024905 wait complete. 2024-09-15T01:59:10.668 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:10.668 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-15T01:59:10.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.745+0000 7f469fc17640 10 monclient: get_monmap_and_config 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.745+0000 7f469fc17640 10 monclient: build_initial_monmap 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.745+0000 7f469fc17640 10 monclient: monmap: 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:10.746397+0000 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:10.746397+0000 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:10.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.745+0000 7f469fc17640 1 Processor -- start 2024-09-15T01:59:10.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 -- start start 2024-09-15T01:59:10.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient: init 2024-09-15T01:59:10.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:10.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4698131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 0x7f4698098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-c con 0x7f4698098420 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-b con 0x7f46981371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46980a13e0 con 0x7f4698131e70 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46980a0440 con 0x7f4698098420 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46980a01e0 con 0x7f46981371a0 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469fc17640 10 monclient(hunting): authenticate will time out at 1183.215527s 2024-09-15T01:59:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469d9b3640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f4698131e70 legacy=0x7f4698132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469d1b2640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 msgr2=0x7f4698098850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469d1b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 0x7f4698098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38312/0 (socket says 172.21.15.67:38312) 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 1 -- 172.21.15.67:0/3481401983 learned_addr learned my addr 172.21.15.67:0/3481401983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 10 monclient(hunting): get_auth_request con 0x7f46981371a0 auth_method 0 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.746+0000 7f469e1b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient(hunting): handle_auth_done global_id 94418 payload 274 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 1 -- 172.21.15.67:0/3481401983 >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 msgr2=0x7f4698098850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 1 --2- 172.21.15.67:0/3481401983 >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 0x7f4698098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 1 -- 172.21.15.67:0/3481401983 >> v1:172.21.15.67:6789/0 conn(0x7f4698131e70 legacy=0x7f4698132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient: _finish_hunting 0 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 1 monclient: found mon.noname-b 2024-09-15T01:59:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.747+0000 7f469e1b4640 1 -- 172.21.15.67:0/3481401983 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46980a19b0 con 0x7f46981371a0 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469e1b4640 1 -- 172.21.15.67:0/3481401983 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 msgr2=0x7f4698139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469e1b4640 1 --2- 172.21.15.67:0/3481401983 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f20 tx=0x7f4690009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a2819cab504758 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 1 -- 172.21.15.67:0/3481401983 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690007610 con 0x7f46981371a0 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: dump: 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:10.761 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: _finish_auth 0 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 5 monclient: authenticate success, global_id 94418 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 1 -- 172.21.15.67:0/3481401983 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469002d050 con 0x7f46981371a0 2024-09-15T01:59:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 1 -- 172.21.15.67:0/3481401983 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46900079a0 con 0x7f46981371a0 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 4 set_mon_vals no callback set 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469c9b1640 10 monclient: dump: 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 10 monclient: shutdown 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 1 -- 172.21.15.67:0/3481401983 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 msgr2=0x7f4698139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.750+0000 7f469fc17640 1 --2- 172.21.15.67:0/3481401983 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f20 tx=0x7f4690009840 comp rx=0 tx=0).stop 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 -- 172.21.15.67:0/3481401983 shutdown_connections 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 --2- 172.21.15.67:0/3481401983 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f4698139580 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 --2- 172.21.15.67:0/3481401983 >> v2:172.21.15.67:3300/0 conn(0x7f4698098420 0x7f4698098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 -- 172.21.15.67:0/3481401983 >> 172.21.15.67:0/3481401983 conn(0x7f46980a5c10 msgr2=0x7f46980a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 -- 172.21.15.67:0/3481401983 shutdown_connections 2024-09-15T01:59:10.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 -- 172.21.15.67:0/3481401983 wait complete. 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 10 monclient: build_initial_monmap 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 10 monclient: monmap: 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 Processor -- start 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 1 -- start start 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.751+0000 7f469fc17640 10 monclient: init 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4698098420 0x7f469809bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-a con 0x7f4698098420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 0x7f4698132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-b con 0x7f4698131e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): picked mon.noname-c con 0x7f46981371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698136990 con 0x7f4698098420 2024-09-15T01:59:10.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46980a0d80 con 0x7f4698131e70 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4698069dd0 con 0x7f46981371a0 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469fc17640 10 monclient(hunting): authenticate will time out at 1183.221612s 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469d1b2640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 msgr2=0x7f4698132ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469d1b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 0x7f4698132ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38324/0 (socket says 172.21.15.67:38324) 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 1 -- 172.21.15.67:0/2073304063 learned_addr learned my addr 172.21.15.67:0/2073304063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 10 monclient(hunting): get_auth_request con 0x7f46981371a0 auth_method 0 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:10.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469e1b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.752+0000 7f469d9b3640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4698098420 0x7f469809bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): get_auth_request con 0x7f4698098420 auth_method 0 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:10.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469d9b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient(hunting): handle_auth_done global_id 94421 payload 824 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 -- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 msgr2=0x7f4698132ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 0x7f4698132ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 -- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4698098420 msgr2=0x7f469809bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4698098420 0x7f469809bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient: _finish_hunting 0 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 monclient: found mon.noname-c 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.753+0000 7f469e1b4640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4690009ae0 con 0x7f46981371a0 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469e1b4640 1 -- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 msgr2=0x7f46981333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469e1b4640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f20 tx=0x7f4690036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690007e10 con 0x7f46981371a0 2024-09-15T01:59:10.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: dump: 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4690031c50 con 0x7f46981371a0 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469fc17640 5 monclient: authenticate success, global_id 94421 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469fc17640 10 monclient: _renew_subs 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469fc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4698136ba0 con 0x7f46981371a0 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690045460 con 0x7f46981371a0 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468dffb640 10 monclient: dump: 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:10.771 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.756+0000 7f469fc17640 10 monclient: _renew_subs 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.755+0000 7f468ffff640 4 set_mon_vals no callback set 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.756+0000 7f469fc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.756+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469812f710 con 0x7f46981371a0 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469fc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469fc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469fc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4690038070 con 0x7f46981371a0 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46980639e0 con 0x7f46981371a0 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f468dffb640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 0x7f467c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(195..195 src has 148..195) v4 ==== 10237+0+0 (secure 0 0 0) 0x7f469006e5c0 con 0x7f46981371a0 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469d9b3640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 0x7f467c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:10.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.757+0000 7f469d9b3640 10 monclient: get_auth_request con 0x7f467c0386f0 auth_method 0 2024-09-15T01:59:10.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.759+0000 7f469d9b3640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 0x7f467c03aba0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4694003940 tx=0x7f46940090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:10.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.762+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f469002ae60 con 0x7f46981371a0 2024-09-15T01:59:10.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.762+0000 7f468dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:10.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.762+0000 7f468dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:10.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.929+0000 7f469fc17640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.929+0000 7f469fc17640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:10.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.929+0000 7f469fc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:10.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:10.929+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4698067a50 con 0x7f46981371a0 2024-09-15T01:59:11.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.817+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v196) v1 ==== 168+0+0 (secure 0 0 0) 0x7f4690040070 con 0x7f46981371a0 2024-09-15T01:59:11.819 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.817+0000 7f468dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:11.820 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.817+0000 7f468dffb640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-15T01:59:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.923+0000 7f469fc17640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.923+0000 7f469fc17640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.923+0000 7f469fc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.923+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4698067a50 con 0x7f46981371a0 2024-09-15T01:59:11.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.925+0000 7f468dffb640 1 -- 172.21.15.67:0/2073304063 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v196) v1 ==== 147+0+0 (secure 0 0 0) 0x7f4690030b50 con 0x7f46981371a0 2024-09-15T01:59:11.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.925+0000 7f468dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:11.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.925+0000 7f468dffb640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-15T01:59:11.927 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h12' already exists 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 msgr2=0x7f467c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 0x7f467c03aba0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4694003940 tx=0x7f46940090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 10 monclient: shutdown 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 msgr2=0x7f46981333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:11.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f20 tx=0x7f4690036000 comp rx=0 tx=0).stop 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 shutdown_connections 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f467c0386f0 0x7f467c03aba0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46981371a0 0x7f46981333e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4698131e70 0x7f4698132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 --2- 172.21.15.67:0/2073304063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4698098420 0x7f469809bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 >> 172.21.15.67:0/2073304063 conn(0x7f46980a5c10 msgr2=0x7f46981316f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.926+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 shutdown_connections 2024-09-15T01:59:11.930 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:11.927+0000 7f469fc17640 1 -- 172.21.15.67:0/2073304063 wait complete. 2024-09-15T01:59:11.939 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:11.939 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.015+0000 7f4712480640 10 monclient: get_monmap_and_config 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.015+0000 7f4712480640 10 monclient: build_initial_monmap 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.015+0000 7f4712480640 10 monclient: monmap: 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:12.016957+0000 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:12.016957+0000 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.018 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:12.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:12.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:12.019 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:12.019 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:12.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.016+0000 7f4712480640 1 Processor -- start 2024-09-15T01:59:12.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.016+0000 7f4712480640 1 -- start start 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.016+0000 7f4712480640 10 monclient: init 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): picked mon.noname-b con 0x7f470c131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f470c098260 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 0x7f470c139130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): picked mon.noname-c con 0x7f470c098d40 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c096a70 con 0x7f470c098260 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c06a6b0 con 0x7f470c098d40 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:12.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0a16e0 con 0x7f470c131f90 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4712480640 10 monclient(hunting): authenticate will time out at 1184.486131s 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470b7fe640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f470c098260 legacy=0x7f470c098630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4710a1d640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 msgr2=0x7f470c139130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f4710a1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 0x7f470c139130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38340/0 (socket says 172.21.15.67:38340) 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 -- 172.21.15.67:0/1102504063 learned_addr learned my addr 172.21.15.67:0/1102504063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): get_auth_request con 0x7f470c131f90 auth_method 0 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:12.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient(hunting): handle_auth_done global_id 94424 payload 274 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 -- 172.21.15.67:0/1102504063 >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 msgr2=0x7f470c139130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 --2- 172.21.15.67:0/1102504063 >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 0x7f470c139130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 -- 172.21.15.67:0/1102504063 >> v1:172.21.15.67:6789/0 conn(0x7f470c098260 legacy=0x7f470c098630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 monclient: found mon.noname-b 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:12.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.017+0000 7f470bfff640 1 -- 172.21.15.67:0/1102504063 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0a19c0 con 0x7f470c131f90 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470bfff640 1 -- 172.21.15.67:0/1102504063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 msgr2=0x7f470c132410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470bfff640 1 --2- 172.21.15.67:0/1102504063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009910 tx=0x7f46fc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ac762c0e557549 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 1 -- 172.21.15.67:0/1102504063 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc032440 con 0x7f470c131f90 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: dump: 2024-09-15T01:59:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: _finish_auth 0 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f4712480640 5 monclient: authenticate success, global_id 94424 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f4712480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:12.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 1 -- 172.21.15.67:0/1102504063 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc0325a0 con 0x7f470c131f90 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 1 -- 172.21.15.67:0/1102504063 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc032940 con 0x7f470c131f90 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f4712480640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f4712480640 4 set_mon_vals no callback set 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.020+0000 7f470affd640 10 monclient: dump: 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.032 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 10 monclient: shutdown 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 -- 172.21.15.67:0/1102504063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 msgr2=0x7f470c132410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 --2- 172.21.15.67:0/1102504063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009910 tx=0x7f46fc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 -- 172.21.15.67:0/1102504063 shutdown_connections 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 --2- 172.21.15.67:0/1102504063 >> v2:172.21.15.67:3300/0 conn(0x7f470c098d40 0x7f470c139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 --2- 172.21.15.67:0/1102504063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c132410 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 -- 172.21.15.67:0/1102504063 >> 172.21.15.67:0/1102504063 conn(0x7f470c071690 msgr2=0x7f470c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 -- 172.21.15.67:0/1102504063 shutdown_connections 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 1 -- 172.21.15.67:0/1102504063 wait complete. 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 10 monclient: build_initial_monmap 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.021+0000 7f4712480640 10 monclient: monmap: 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:12.034 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 Processor -- start 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 -- start start 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient: init 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 0x7f470c137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): picked mon.noname-b con 0x7f470c098260 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c098d40 0x7f470c133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): picked mon.noname-a con 0x7f470c098d40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): picked mon.noname-c con 0x7f470c131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c136e70 con 0x7f470c098d40 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f470bfff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 msgr2=0x7f470c137f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0a0d80 con 0x7f470c098260 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0a3140 con 0x7f470c131f90 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f4712480640 10 monclient(hunting): authenticate will time out at 1184.491952s 2024-09-15T01:59:12.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.022+0000 7f470bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 0x7f470c137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c098d40 0x7f470c133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38344/0 (socket says 172.21.15.67:38344) 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 -- 172.21.15.67:0/404575172 learned_addr learned my addr 172.21.15.67:0/404575172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): get_auth_request con 0x7f470c131f90 auth_method 0 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): get_auth_request con 0x7f470c098d40 auth_method 0 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f470b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient(hunting): handle_auth_done global_id 94427 payload 824 2024-09-15T01:59:12.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 -- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 msgr2=0x7f470c137f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 0x7f470c137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 -- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c098d40 msgr2=0x7f470c133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c098d40 0x7f470c133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient: _finish_hunting 0 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 monclient: found mon.noname-c 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.023+0000 7f4710a1d640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc009450 con 0x7f470c131f90 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.025+0000 7f4710a1d640 1 -- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 msgr2=0x7f470c1341d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.025+0000 7f4710a1d640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4704009fc0 tx=0x7f4704009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704016430 con 0x7f470c131f90 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:12.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: dump: 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:12.040 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 5 monclient: authenticate success, global_id 94427 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47040165d0 con 0x7f470c131f90 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 10 monclient: _renew_subs 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c1357e0 con 0x7f470c131f90 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704016780 con 0x7f470c131f90 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f46f3fff640 10 monclient: dump: 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 10 monclient: _renew_subs 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c1384a0 con 0x7f470c131f90 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.026+0000 7f470a7fc640 4 set_mon_vals no callback set 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f4712480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f4712480640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f4712480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c0639e0 con 0x7f470c131f90 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4704006150 con 0x7f470c131f90 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f46f3fff640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 0x7f46f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:12.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(196..196 src has 148..196) v4 ==== 10450+0+0 (secure 0 0 0) 0x7f470404da90 con 0x7f470c131f90 2024-09-15T01:59:12.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f470bfff640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 0x7f46f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:12.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.027+0000 7f470bfff640 10 monclient: get_auth_request con 0x7f46f80386f0 auth_method 0 2024-09-15T01:59:12.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.029+0000 7f470bfff640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 0x7f46f803aba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f470c064e70 tx=0x7f46fc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:12.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.032+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4704007ab0 con 0x7f470c131f90 2024-09-15T01:59:12.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.032+0000 7f46f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:12.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.032+0000 7f46f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:12.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.196+0000 7f4712480640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.196+0000 7f4712480640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.196+0000 7f4712480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.196+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f470c132990 con 0x7f470c131f90 2024-09-15T01:59:12.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.818+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v197) v1 ==== 168+0+0 (secure 0 0 0) 0x7f47040072f0 con 0x7f470c131f90 2024-09-15T01:59:12.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.818+0000 7f46f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.821 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.818+0000 7f46f3fff640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-15T01:59:12.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.920+0000 7f4712480640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.920+0000 7f4712480640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.920+0000 7f4712480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:12.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.920+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f470c067a20 con 0x7f470c131f90 2024-09-15T01:59:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.929+0000 7f46f3fff640 1 -- 172.21.15.67:0/404575172 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v197) v1 ==== 147+0+0 (secure 0 0 0) 0x7f47040074d0 con 0x7f470c131f90 2024-09-15T01:59:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.929+0000 7f46f3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.929+0000 7f46f3fff640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-15T01:59:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h13' already exists 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 msgr2=0x7f46f803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 0x7f46f803aba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f470c064e70 tx=0x7f46fc032000 comp rx=0 tx=0).stop 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 10 monclient: shutdown 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 msgr2=0x7f470c1341d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4704009fc0 tx=0x7f4704009730 comp rx=0 tx=0).stop 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 shutdown_connections 2024-09-15T01:59:12.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46f80386f0 0x7f46f803aba0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f470c098d40 0x7f470c133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f470c131f90 0x7f470c1341d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 --2- 172.21.15.67:0/404575172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f470c098260 0x7f470c137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 >> 172.21.15.67:0/404575172 conn(0x7f470c071690 msgr2=0x7f470c131660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 shutdown_connections 2024-09-15T01:59:12.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:12.930+0000 7f4712480640 1 -- 172.21.15.67:0/404575172 wait complete. 2024-09-15T01:59:12.942 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:12.942 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.018+0000 7f8c34620640 10 monclient: get_monmap_and_config 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.018+0000 7f8c34620640 10 monclient: build_initial_monmap 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.018+0000 7f8c34620640 10 monclient: monmap: 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:13.019670+0000 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:13.019670+0000 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:13.021 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:13.022 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 Processor -- start 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 -- start start 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient: init 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c2c097300 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 0x7f8c2c09c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2c09a710 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8c2c097d30 0x7f8c2c135830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2c097d30 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:13.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2c0987e0 con 0x7f8c2c097300 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2c0976f0 con 0x7f8c2c097d30 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2c097af0 con 0x7f8c2c09a710 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c323bc640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f8c2c097300 legacy=0x7f8c2c09ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c34620640 10 monclient(hunting): authenticate will time out at 1185.488803s 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c31bbb640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 msgr2=0x7f8c2c09c120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.019+0000 7f8c31bbb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 0x7f8c2c09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8c2c097d30 0x7f8c2c135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8c2c097d30 0x7f8c2c135830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:58634/0 (socket says 172.21.15.67:58634) 2024-09-15T01:59:13.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 -- 172.21.15.67:0/1865983319 learned_addr learned my addr 172.21.15.67:0/1865983319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): get_auth_request con 0x7f8c2c097d30 auth_method 0 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_done global_id 104265 payload 274 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 -- 172.21.15.67:0/1865983319 >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 msgr2=0x7f8c2c09c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 --2- 172.21.15.67:0/1865983319 >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 0x7f8c2c09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 -- 172.21.15.67:0/1865983319 >> v1:172.21.15.67:6789/0 conn(0x7f8c2c097300 legacy=0x7f8c2c09ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:13.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:13.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 monclient: found mon.noname-a 2024-09-15T01:59:13.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:13.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:13.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.020+0000 7f8c32bbd640 1 -- 172.21.15.67:0/1865983319 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2c06ee60 con 0x7f8c2c097d30 2024-09-15T01:59:13.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c32bbd640 1 -- 172.21.15.67:0/1865983319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c097d30 msgr2=0x7f8c2c135830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c32bbd640 1 --2- 172.21.15.67:0/1865983319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c097d30 0x7f8c2c135830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0a0f50 tx=0x7f8c24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a272bb452b0124c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 1 -- 172.21.15.67:0/1865983319 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2402aea0 con 0x7f8c2c097d30 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: dump: 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:13.034 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: _finish_auth 0 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c34620640 5 monclient: authenticate success, global_id 104265 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c34620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 1 -- 172.21.15.67:0/1865983319 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2402d050 con 0x7f8c2c097d30 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c34620640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:13.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c34620640 4 set_mon_vals no callback set 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 1 -- 172.21.15.67:0/1865983319 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24007840 con 0x7f8c2c097d30 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.023+0000 7f8c313ba640 10 monclient: dump: 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:13.036 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 10 monclient: shutdown 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 -- 172.21.15.67:0/1865983319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c097d30 msgr2=0x7f8c2c135830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 --2- 172.21.15.67:0/1865983319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c097d30 0x7f8c2c135830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0a0f50 tx=0x7f8c24009840 comp rx=0 tx=0).stop 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 -- 172.21.15.67:0/1865983319 shutdown_connections 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 --2- 172.21.15.67:0/1865983319 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c097d30 0x7f8c2c135830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 --2- 172.21.15.67:0/1865983319 >> v2:172.21.15.67:3300/0 conn(0x7f8c2c09a710 0x7f8c2c09c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 -- 172.21.15.67:0/1865983319 >> 172.21.15.67:0/1865983319 conn(0x7f8c2c0a1020 msgr2=0x7f8c2c0a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 -- 172.21.15.67:0/1865983319 shutdown_connections 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 -- 172.21.15.67:0/1865983319 wait complete. 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 10 monclient: build_initial_monmap 2024-09-15T01:59:13.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 10 monclient: monmap: 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:13.038 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.024+0000 7f8c34620640 1 Processor -- start 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 -- start start 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient: init 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2c097300 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 0x7f8c2c133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-b con 0x7f8c2c097d30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c09a710 0x7f8c2c133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2c09a710 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:13.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c117bb0 con 0x7f8c2c09a710 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c135b10 con 0x7f8c2c097d30 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c09cd10 con 0x7f8c2c097300 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c34620640 10 monclient(hunting): authenticate will time out at 1185.494492s 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c31bbb640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 msgr2=0x7f8c2c133780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c31bbb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 0x7f8c2c133780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c32bbd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c09a710 0x7f8c2c133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:13.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38360/0 (socket says 172.21.15.67:38360) 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 1 -- 172.21.15.67:0/1351782192 learned_addr learned my addr 172.21.15.67:0/1351782192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 10 monclient(hunting): get_auth_request con 0x7f8c2c097300 auth_method 0 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c323bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c32bbd640 10 monclient(hunting): get_auth_request con 0x7f8c2c09a710 auth_method 0 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c32bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c32bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.025+0000 7f8c32bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:13.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c32bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient(hunting): handle_auth_done global_id 94430 payload 824 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 -- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 msgr2=0x7f8c2c133780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 0x7f8c2c133780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 -- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c09a710 msgr2=0x7f8c2c133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c09a710 0x7f8c2c133cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 monclient: found mon.noname-c 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.026+0000 7f8c323bc640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24009ae0 con 0x7f8c2c097300 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c323bc640 1 -- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 msgr2=0x7f8c2c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c323bc640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800b600 tx=0x7f8c280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28015070 con 0x7f8c2c097300 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: dump: 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:13.043 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c28003d90 con 0x7f8c2c097300 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28006380 con 0x7f8c2c097300 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 5 monclient: authenticate success, global_id 94430 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 10 monclient: _renew_subs 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2c1307e0 con 0x7f8c2c097300 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c30bb9640 4 set_mon_vals no callback set 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c227fc640 10 monclient: dump: 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 10 monclient: _renew_subs 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.028+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2c135310 con 0x7f8c2c097300 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8c28010440 con 0x7f8c2c097300 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c34620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c34620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c34620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2c0639e0 con 0x7f8c2c097300 2024-09-15T01:59:13.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c227fc640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 0x7f8c1003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.029+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(197..197 src has 148..197) v4 ==== 10663+0+0 (secure 0 0 0) 0x7f8c2804cc30 con 0x7f8c2c097300 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.030+0000 7f8c31bbb640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 0x7f8c1003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.030+0000 7f8c31bbb640 10 monclient: get_auth_request con 0x7f8c10038740 auth_method 0 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.032+0000 7f8c31bbb640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 0x7f8c1003abf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c003940 tx=0x7f8c1c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.034+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8c28018ab0 con 0x7f8c2c097300 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.034+0000 7f8c227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:13.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.034+0000 7f8c227fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:13.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.199+0000 7f8c34620640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.199+0000 7f8c34620640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.199+0000 7f8c34620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.199+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8c2c09c6a0 con 0x7f8c2c097300 2024-09-15T01:59:13.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.826+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v198) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8c28018690 con 0x7f8c2c097300 2024-09-15T01:59:13.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.826+0000 7f8c227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.826+0000 7f8c227fc640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-15T01:59:13.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.931+0000 7f8c34620640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.931+0000 7f8c34620640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.931+0000 7f8c34620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:13.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.931+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8c2c0978d0 con 0x7f8c2c097300 2024-09-15T01:59:13.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.932+0000 7f8c227fc640 1 -- 172.21.15.67:0/1351782192 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v198) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8c28018ab0 con 0x7f8c2c097300 2024-09-15T01:59:13.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.932+0000 7f8c227fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:13.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.932+0000 7f8c227fc640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-15T01:59:13.935 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h14' already exists 2024-09-15T01:59:13.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 msgr2=0x7f8c1003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:13.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 0x7f8c1003abf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c003940 tx=0x7f8c1c006d80 comp rx=0 tx=0).stop 2024-09-15T01:59:13.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 10 monclient: shutdown 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 msgr2=0x7f8c2c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800b600 tx=0x7f8c280091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 shutdown_connections 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8c10038740 0x7f8c1003abf0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c2c09a710 0x7f8c2c133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c097d30 0x7f8c2c133780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 --2- 172.21.15.67:0/1351782192 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c2c097300 0x7f8c2c13e570 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:13.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.934+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 >> 172.21.15.67:0/1351782192 conn(0x7f8c2c0a1020 msgr2=0x7f8c2c09d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:13.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.935+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 shutdown_connections 2024-09-15T01:59:13.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:13.935+0000 7f8c34620640 1 -- 172.21.15.67:0/1351782192 wait complete. 2024-09-15T01:59:13.948 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:13.948 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-15T01:59:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 10 monclient: get_monmap_and_config 2024-09-15T01:59:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 10 monclient: build_initial_monmap 2024-09-15T01:59:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 10 monclient: monmap: 2024-09-15T01:59:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:14.024379+0000 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:14.024379+0000 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:14.026 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 1 Processor -- start 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 1 -- start start 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.023+0000 7f63e92c0640 10 monclient: init 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T01:59:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 0x7f63e4132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-c con 0x7f63e4131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-b con 0x7f63e4098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63e4068e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40a13e0 con 0x7f63e4068e70 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e40a0440 con 0x7f63e4131e70 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40a01e0 con 0x7f63e4098420 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e92c0640 10 monclient(hunting): authenticate will time out at 1186.493377s 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e2ffd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 msgr2=0x7f63e4132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e37fe640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f63e4068e70 legacy=0x7f63e40692f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 0x7f63e4132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38368/0 (socket says 172.21.15.67:38368) 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 1 -- 172.21.15.67:0/236776039 learned_addr learned my addr 172.21.15.67:0/236776039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): get_auth_request con 0x7f63e4098420 auth_method 0 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.024+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 10 monclient(hunting): handle_auth_done global_id 94433 payload 274 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 1 -- 172.21.15.67:0/236776039 >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 msgr2=0x7f63e4132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 1 --2- 172.21.15.67:0/236776039 >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 0x7f63e4132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 1 -- 172.21.15.67:0/236776039 >> v1:172.21.15.67:6789/0 conn(0x7f63e4068e70 legacy=0x7f63e40692f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 1 monclient: found mon.noname-b 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.025+0000 7f63e27fc640 1 -- 172.21.15.67:0/236776039 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40a19b0 con 0x7f63e4098420 2024-09-15T01:59:14.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e27fc640 1 -- 172.21.15.67:0/236776039 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 msgr2=0x7f63e4098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:14.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e27fc640 1 --2- 172.21.15.67:0/236776039 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009910 tx=0x7f63cc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701a0df63caea657 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:14.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 1 -- 172.21.15.67:0/236776039 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc032440 con 0x7f63e4098420 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: dump: 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:14.039 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 1 -- 172.21.15.67:0/236776039 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc0325a0 con 0x7f63e4098420 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 5 monclient: authenticate success, global_id 94433 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 1 -- 172.21.15.67:0/236776039 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc032940 con 0x7f63e4098420 2024-09-15T01:59:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 4 set_mon_vals no callback set 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e1ffb640 10 monclient: dump: 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 10 monclient: shutdown 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 -- 172.21.15.67:0/236776039 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 msgr2=0x7f63e4098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 --2- 172.21.15.67:0/236776039 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009910 tx=0x7f63cc0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 -- 172.21.15.67:0/236776039 shutdown_connections 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 --2- 172.21.15.67:0/236776039 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e4098850 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 --2- 172.21.15.67:0/236776039 >> v2:172.21.15.67:3300/0 conn(0x7f63e4131e70 0x7f63e4132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 -- 172.21.15.67:0/236776039 >> 172.21.15.67:0/236776039 conn(0x7f63e40a5c10 msgr2=0x7f63e40a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 -- 172.21.15.67:0/236776039 shutdown_connections 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.028+0000 7f63e92c0640 1 -- 172.21.15.67:0/236776039 wait complete. 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.029+0000 7f63e92c0640 10 monclient: build_initial_monmap 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.029+0000 7f63e92c0640 10 monclient: monmap: 2024-09-15T01:59:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.029+0000 7f63e92c0640 1 Processor -- start 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.029+0000 7f63e92c0640 1 -- start start 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.029+0000 7f63e92c0640 10 monclient: init 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 0x7f63e4141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-b con 0x7f63e4068e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-c con 0x7f63e4098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): picked mon.noname-a con 0x7f63e4131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e4142330 con 0x7f63e4131e70 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40a0d80 con 0x7f63e4068e70 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e411d6a0 con 0x7f63e4098420 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e92c0640 10 monclient(hunting): authenticate will time out at 1186.499469s 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e2ffd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 msgr2=0x7f63e4141400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 0x7f63e4141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:58654/0 (socket says 172.21.15.67:58654) 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38384/0 (socket says 172.21.15.67:38384) 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 1 -- 172.21.15.67:0/3705371607 learned_addr learned my addr 172.21.15.67:0/3705371607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): get_auth_request con 0x7f63e4131e70 auth_method 0 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): get_auth_request con 0x7f63e4098420 auth_method 0 2024-09-15T01:59:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.030+0000 7f63e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 10 monclient(hunting): handle_auth_done global_id 94436 payload 824 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 -- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 msgr2=0x7f63e4141400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 0x7f63e4141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 -- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 msgr2=0x7f63e412f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 monclient: found mon.noname-c 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e27fc640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc009450 con 0x7f63e4098420 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.031+0000 7f63e37fe640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e27fc640 1 -- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 msgr2=0x7f63e412efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e27fc640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63e40a0320 tx=0x7f63cc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc031a90 con 0x7f63e4098420 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: dump: 2024-09-15T01:59:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:14.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 5 monclient: authenticate success, global_id 94436 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 10 monclient: _renew_subs 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e4130ee0 con 0x7f63e4098420 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc031bf0 con 0x7f63e4098420 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc03b400 con 0x7f63e4098420 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63daffd640 10 monclient: dump: 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:14.049 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 10 monclient: _renew_subs 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e17fa640 4 set_mon_vals no callback set 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.033+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e411e200 con 0x7f63e4098420 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.034+0000 7f63e92c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:14.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.034+0000 7f63e92c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.034+0000 7f63e92c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.034+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e411f700 con 0x7f63e4098420 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.034+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f63cc039040 con 0x7f63e4098420 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.035+0000 7f63daffd640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 0x7f63c403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.035+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(198..198 src has 148..198) v4 ==== 10876+0+0 (secure 0 0 0) 0x7f63cc06ca60 con 0x7f63e4098420 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.035+0000 7f63e2ffd640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 0x7f63c403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:14.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.035+0000 7f63e2ffd640 10 monclient: get_auth_request con 0x7f63c40386f0 auth_method 0 2024-09-15T01:59:14.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.037+0000 7f63e2ffd640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 0x7f63c403aba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f63dc003940 tx=0x7f63dc006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:14.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.039+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f63cc030340 con 0x7f63e4098420 2024-09-15T01:59:14.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.039+0000 7f63daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:14.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.039+0000 7f63daffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:14.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.204+0000 7f63e92c0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.204+0000 7f63e92c0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.204+0000 7f63e92c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.204+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63e4130ee0 con 0x7f63e4098420 2024-09-15T01:59:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.830+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v199) v1 ==== 168+0+0 (secure 0 0 0) 0x7f63cc03f0b0 con 0x7f63e4098420 2024-09-15T01:59:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.830+0000 7f63daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.830+0000 7f63daffd640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-15T01:59:14.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.932+0000 7f63e92c0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.932+0000 7f63e92c0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.932+0000 7f63e92c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:14.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.932+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63e4130e80 con 0x7f63e4098420 2024-09-15T01:59:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.934+0000 7f63daffd640 1 -- 172.21.15.67:0/3705371607 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v199) v1 ==== 147+0+0 (secure 0 0 0) 0x7f63cc037070 con 0x7f63e4098420 2024-09-15T01:59:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.934+0000 7f63daffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.934+0000 7f63daffd640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-15T01:59:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h15' already exists 2024-09-15T01:59:14.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 msgr2=0x7f63c403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:14.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 0x7f63c403aba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f63dc003940 tx=0x7f63dc006d80 comp rx=0 tx=0).stop 2024-09-15T01:59:14.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 10 monclient: shutdown 2024-09-15T01:59:14.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 msgr2=0x7f63e412efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63e40a0320 tx=0x7f63cc032000 comp rx=0 tx=0).stop 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 shutdown_connections 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63c40386f0 0x7f63c403aba0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63e4131e70 0x7f63e412f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63e4098420 0x7f63e412efa0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 --2- 172.21.15.67:0/3705371607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63e4068e70 0x7f63e4141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 >> 172.21.15.67:0/3705371607 conn(0x7f63e40a5c10 msgr2=0x7f63e41302d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 shutdown_connections 2024-09-15T01:59:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:14.936+0000 7f63e92c0640 1 -- 172.21.15.67:0/3705371607 wait complete. 2024-09-15T01:59:14.948 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:14.948 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 10 monclient: build_initial_monmap 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 10 monclient: monmap: 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:15.025314+0000 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:15.025314+0000 2024-09-15T01:59:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 1 Processor -- start 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 1 -- start start 2024-09-15T01:59:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.024+0000 7fb64595d640 10 monclient: init 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb640131e70 0x7fb640132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-c con 0x7fb640131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb640098420 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6401371a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:15.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400a13e0 con 0x7fb6401371a0 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400a0440 con 0x7fb640098420 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6400a01e0 con 0x7fb640131e70 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb64595d640 10 monclient(hunting): authenticate will time out at 1187.494407s 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb637fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fb640098420 legacy=0x7fb640098850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb63effd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fb640131e70 msgr2=0x7fb640132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb63effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb640131e70 0x7fb640132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb63f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb6401371a0 0x7fb640139580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:34132/0 (socket says 172.21.15.67:34132) 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.025+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 learned_addr learned my addr 172.21.15.67:0/2172490470 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:15.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 >> v1:172.21.15.17:6789/0 conn(0x7fb6401371a0 legacy=0x7fb640139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865786328 0 0) 0x7fb6400a13e0 con 0x7fb6401371a0 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): my global_id is 104274 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c0034e0 con 0x7fb6401371a0 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090751501 0 0) 0x7fb62c0034e0 con 0x7fb6401371a0 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 >> v2:172.21.15.67:3300/0 conn(0x7fb640131e70 msgr2=0x7fb640132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 --2- 172.21.15.67:0/2172490470 >> v2:172.21.15.67:3300/0 conn(0x7fb640131e70 0x7fb640132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 >> v1:172.21.15.67:6789/0 conn(0x7fb640098420 legacy=0x7fb640098850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:15.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400a19b0 con 0x7fb6401371a0 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fb638001b60 con 0x7fb6401371a0 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: dump: 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:15.036 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 >> v1:172.21.15.17:6789/0 conn(0x7fb6401371a0 legacy=0x7fb640139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:15.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0039d0 0x7fb62c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): picked mon.b con 0x7fb62c0039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb62c004780 0x7fb62c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): picked mon.a con 0x7fb62c004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 0x7fb62c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): picked mon.c con 0x7fb62c005280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400a01e0 con 0x7fb62c004780 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00c240 con 0x7fb62c0039d0 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00c400 con 0x7fb62c005280 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63effd640 1 -- 172.21.15.67:0/2172490470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0039d0 msgr2=0x7fb62c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63effd640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0039d0 0x7fb62c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb62c004780 0x7fb62c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 0x7fb62c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): get_auth_request con 0x7fb62c004780 auth_method 0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): get_auth_request con 0x7fb62c005280 auth_method 0 2024-09-15T01:59:15.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.028+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_done global_id 104274 payload 293 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0039d0 msgr2=0x7fb62c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0039d0 0x7fb62c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb62c004780 msgr2=0x7fb62c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:15.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb62c004780 0x7fb62c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 monclient: found mon.c 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0034e0 con 0x7fb62c005280 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb64595d640 5 monclient: authenticate success, global_id 104274 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb64595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.029+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 reap_dead start 2024-09-15T01:59:15.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.031+0000 7fb63f7fe640 1 -- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 msgr2=0x7fb62c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:15.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.031+0000 7fb63f7fe640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 0x7fb62c00b9d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb63800acb0 tx=0x7fb6400a0320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5029a992817eb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63802b950 con 0x7fb62c005280 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: dump: 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63802bad0 con 0x7fb62c005280 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 1 -- 172.21.15.67:0/2172490470 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63802be90 con 0x7fb62c005280 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb63e7fc640 10 monclient: dump: 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.049 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 4 set_mon_vals no callback set 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 10 monclient: shutdown 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 -- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 msgr2=0x7fb62c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:15.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 0x7fb62c00b9d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb63800acb0 tx=0x7fb6400a0320 comp rx=0 tx=0).stop 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 -- 172.21.15.67:0/2172490470 shutdown_connections 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 --2- 172.21.15.67:0/2172490470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb62c005280 0x7fb62c00b9d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 -- 172.21.15.67:0/2172490470 >> 172.21.15.67:0/2172490470 conn(0x7fb6400a5c10 msgr2=0x7fb6400a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 -- 172.21.15.67:0/2172490470 shutdown_connections 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.032+0000 7fb64595d640 1 -- 172.21.15.67:0/2172490470 wait complete. 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.033+0000 7fb64595d640 10 monclient: build_initial_monmap 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.033+0000 7fb64595d640 10 monclient: monmap: 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.033+0000 7fb64595d640 1 Processor -- start 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.033+0000 7fb64595d640 1 -- start start 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.033+0000 7fb64595d640 10 monclient: init 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 0x7fb640134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-b con 0x7fb62c005280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-a con 0x7fb640134560 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 0x7fb640147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): picked mon.noname-c con 0x7fb640134ed0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640069d10 con 0x7fb640134560 2024-09-15T01:59:15.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400a0d80 con 0x7fb62c005280 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb640135ee0 con 0x7fb640134ed0 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63effd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 msgr2=0x7fb640134020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb64595d640 10 monclient(hunting): authenticate will time out at 1187.503137s 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 0x7fb640134020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 0x7fb640147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:58666/0 (socket says 172.21.15.67:58666) 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 -- 172.21.15.67:0/3886004108 learned_addr learned my addr 172.21.15.67:0/3886004108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:15.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): get_auth_request con 0x7fb640134560 auth_method 0 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): get_auth_request con 0x7fb640134ed0 auth_method 0 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:15.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient(hunting): handle_auth_done global_id 104277 payload 824 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 -- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 msgr2=0x7fb640147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 0x7fb640147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 -- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 msgr2=0x7fb640134020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 0x7fb640134020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 monclient: found mon.noname-a 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb637fff640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6401359d0 con 0x7fb640134560 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.034+0000 7fb63f7fe640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 0x7fb640147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb637fff640 1 -- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 msgr2=0x7fb640134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb637fff640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6280096d0 tx=0x7fb628009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62800e8a0 con 0x7fb640134560 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: dump: 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 5 monclient: authenticate success, global_id 104277 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62800ea20 con 0x7fb640134560 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 10 monclient: _renew_subs 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64011bcb0 con 0x7fb640134560 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62800ede0 con 0x7fb640134560 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb636ffd640 10 monclient: dump: 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:15.059 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 10 monclient: _renew_subs 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb640130db0 con 0x7fb640134560 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.037+0000 7fb63dffb640 4 set_mon_vals no callback set 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb64595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb64595d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb64595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6400639e0 con 0x7fb640134560 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb628012600 con 0x7fb640134560 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb636ffd640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 0x7fb62003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(199..199 src has 148..199) v4 ==== 11089+0+0 (secure 0 0 0) 0x7fb62804d0e0 con 0x7fb640134560 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb63effd640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 0x7fb62003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.038+0000 7fb63effd640 10 monclient: get_auth_request con 0x7fb6200386f0 auth_method 0 2024-09-15T01:59:15.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.041+0000 7fb63effd640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 0x7fb62003aba0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb630006fd0 tx=0x7fb630006dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:15.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.043+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb628022ad0 con 0x7fb640134560 2024-09-15T01:59:15.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.043+0000 7fb636ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:15.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.043+0000 7fb636ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:15.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.209+0000 7fb64595d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.209+0000 7fb64595d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.209+0000 7fb64595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.209+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb640132ae0 con 0x7fb640134560 2024-09-15T01:59:15.836 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.834+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v200) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb6280222f0 con 0x7fb640134560 2024-09-15T01:59:15.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.834+0000 7fb636ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.834+0000 7fb636ffd640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-15T01:59:15.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb64595d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb64595d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb64595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:15.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb640067a50 con 0x7fb640134560 2024-09-15T01:59:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb636ffd640 1 -- 172.21.15.67:0/3886004108 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v200) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb6280224d0 con 0x7fb640134560 2024-09-15T01:59:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb636ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.939+0000 7fb636ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-15T01:59:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h16' already exists 2024-09-15T01:59:15.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.941+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 msgr2=0x7fb62003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:15.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.941+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 0x7fb62003aba0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb630006fd0 tx=0x7fb630006dd0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 10 monclient: shutdown 2024-09-15T01:59:15.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:15.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 msgr2=0x7fb640134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6280096d0 tx=0x7fb628009060 comp rx=0 tx=0).stop 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 shutdown_connections 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6200386f0 0x7fb62003aba0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb640134560 0x7fb640134990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb640134ed0 0x7fb640147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 --2- 172.21.15.67:0/3886004108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c005280 0x7fb640134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 >> 172.21.15.67:0/3886004108 conn(0x7fb6400a5c10 msgr2=0x7fb640131600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 shutdown_connections 2024-09-15T01:59:15.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:15.942+0000 7fb64595d640 1 -- 172.21.15.67:0/3886004108 wait complete. 2024-09-15T01:59:15.954 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:15.954 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.029+0000 7fbd5139d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.029+0000 7fbd5139d640 10 monclient: build_initial_monmap 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.030+0000 7fbd5139d640 10 monclient: monmap: 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:16.031156+0000 2024-09-15T01:59:16.032 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:16.031156+0000 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:16.033 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.030+0000 7fbd5139d640 1 Processor -- start 2024-09-15T01:59:16.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.030+0000 7fbd5139d640 1 -- start start 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.030+0000 7fbd5139d640 10 monclient: init 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 0x7fbd4c132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-c con 0x7fbd4c131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-a con 0x7fbd4c098420 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 0x7fbd4c0692f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-b con 0x7fbd4c068e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a13e0 con 0x7fbd4c098420 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a0440 con 0x7fbd4c131e70 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a01e0 con 0x7fbd4c068e70 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4affd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 msgr2=0x7fbd4c132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd5139d640 10 monclient(hunting): authenticate will time out at 1188.500249s 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 0x7fbd4c132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 0x7fbd4c0692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:58674/0 (socket says 172.21.15.67:58674) 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 1 -- 172.21.15.67:0/1073619111 learned_addr learned my addr 172.21.15.67:0/1073619111 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): get_auth_request con 0x7fbd4c068e70 auth_method 0 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): get_auth_request con 0x7fbd4c098420 auth_method 0 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.031+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 10 monclient(hunting): handle_auth_done global_id 94442 payload 274 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 -- 172.21.15.67:0/1073619111 >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 msgr2=0x7fbd4c132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:16.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 --2- 172.21.15.67:0/1073619111 >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 0x7fbd4c132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 -- 172.21.15.67:0/1073619111 >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 msgr2=0x7fbd4c098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 --2- 172.21.15.67:0/1073619111 >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 monclient: found mon.noname-b 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4b7fe640 1 -- 172.21.15.67:0/1073619111 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c0a19b0 con 0x7fbd4c068e70 2024-09-15T01:59:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.032+0000 7fbd4a7fc640 1 --2- 172.21.15.67:0/1073619111 >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.034+0000 7fbd4b7fe640 1 -- 172.21.15.67:0/1073619111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 msgr2=0x7fbd4c0692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.034+0000 7fbd4b7fe640 1 --2- 172.21.15.67:0/1073619111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 0x7fbd4c0692f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0a6250 tx=0x7fbd3c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c882faf4fca06c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 1 -- 172.21.15.67:0/1073619111 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c007760 con 0x7fbd4c068e70 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: dump: 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 5 monclient: authenticate success, global_id 94442 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 1 -- 172.21.15.67:0/1073619111 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c02d050 con 0x7fbd4c068e70 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 1 -- 172.21.15.67:0/1073619111 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c007af0 con 0x7fbd4c068e70 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 4 set_mon_vals no callback set 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd49ffb640 10 monclient: dump: 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 10 monclient: shutdown 2024-09-15T01:59:16.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 -- 172.21.15.67:0/1073619111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 msgr2=0x7fbd4c0692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 --2- 172.21.15.67:0/1073619111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 0x7fbd4c0692f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0a6250 tx=0x7fbd3c009780 comp rx=0 tx=0).stop 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 -- 172.21.15.67:0/1073619111 shutdown_connections 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 --2- 172.21.15.67:0/1073619111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c068e70 0x7fbd4c0692f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 --2- 172.21.15.67:0/1073619111 >> v2:172.21.15.17:3300/0 conn(0x7fbd4c098420 0x7fbd4c098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 --2- 172.21.15.67:0/1073619111 >> v2:172.21.15.67:3300/0 conn(0x7fbd4c131e70 0x7fbd4c132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 -- 172.21.15.67:0/1073619111 >> 172.21.15.67:0/1073619111 conn(0x7fbd4c0a5c10 msgr2=0x7fbd4c0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 -- 172.21.15.67:0/1073619111 shutdown_connections 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.035+0000 7fbd5139d640 1 -- 172.21.15.67:0/1073619111 wait complete. 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.036+0000 7fbd5139d640 10 monclient: build_initial_monmap 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.036+0000 7fbd5139d640 10 monclient: monmap: 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:16.049 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.036+0000 7fbd5139d640 1 Processor -- start 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.036+0000 7fbd5139d640 1 -- start start 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.036+0000 7fbd5139d640 10 monclient: init 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:16.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 0x7fbd4c12eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-a con 0x7fbd4c068e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-c con 0x7fbd4c098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 0x7fbd4c11bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): picked mon.noname-b con 0x7fbd4c131e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a0d80 con 0x7fbd4c068e70 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0a3590 con 0x7fbd4c131e70 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c11d1b0 con 0x7fbd4c098420 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd5139d640 10 monclient(hunting): authenticate will time out at 1188.506476s 2024-09-15T01:59:16.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4b7fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 msgr2=0x7fbd4c11bde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 0x7fbd4c11bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 0x7fbd4c12eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:38426/0 (socket says 172.21.15.67:38426) 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 1 -- 172.21.15.67:0/2131727331 learned_addr learned my addr 172.21.15.67:0/2131727331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): get_auth_request con 0x7fbd4c098420 auth_method 0 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): get_auth_request con 0x7fbd4c068e70 auth_method 0 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient(hunting): handle_auth_done global_id 104283 payload 824 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 -- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 msgr2=0x7fbd4c11b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 -- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 msgr2=0x7fbd4c11bde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 0x7fbd4c11bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:16.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.037+0000 7fbd4affd640 1 monclient: found mon.noname-a 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.038+0000 7fbd4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.038+0000 7fbd4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.038+0000 7fbd4affd640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34009450 con 0x7fbd4c068e70 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.038+0000 7fbd4a7fc640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd4affd640 1 -- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 msgr2=0x7fbd4c12eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd4affd640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 0x7fbd4c12eb20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b5b0 tx=0x7fbd440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44003d90 con 0x7fbd4c068e70 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: dump: 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.054 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44006150 con 0x7fbd4c068e70 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 5 monclient: authenticate success, global_id 104283 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 10 monclient: _renew_subs 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3c009a20 con 0x7fbd4c068e70 2024-09-15T01:59:16.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44006510 con 0x7fbd4c068e70 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd42ffd640 10 monclient: dump: 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.056 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd497fa640 4 set_mon_vals no callback set 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 10 monclient: _renew_subs 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.040+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c11dd80 con 0x7fbd4c068e70 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.041+0000 7fbd5139d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.041+0000 7fbd5139d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.041+0000 7fbd5139d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.041+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c0639e0 con 0x7fbd4c068e70 2024-09-15T01:59:16.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.041+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbd440103e0 con 0x7fbd4c068e70 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.042+0000 7fbd42ffd640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 0x7fbd2c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.042+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(200..200 src has 148..200) v4 ==== 11558+0+0 (secure 0 0 0) 0x7fbd4404cd70 con 0x7fbd4c068e70 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.042+0000 7fbd4a7fc640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 0x7fbd2c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.042+0000 7fbd4a7fc640 10 monclient: get_auth_request con 0x7fbd2c0386f0 auth_method 0 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.044+0000 7fbd4a7fc640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 0x7fbd2c03aba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009fd0 tx=0x7fbd340091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.046+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbd4400ed00 con 0x7fbd4c068e70 2024-09-15T01:59:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.046+0000 7fbd42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.046+0000 7fbd42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:16.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.213+0000 7fbd5139d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.213+0000 7fbd5139d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.213+0000 7fbd5139d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.213+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbd4c067a50 con 0x7fbd4c068e70 2024-09-15T01:59:16.424 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T01:59:16.424 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T01:59:16.424 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T01:59:16.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T01:59:16.427 INFO:tasks.ceph.mon.b:Started 2024-09-15T01:59:16.427 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:59:16.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 10 monclient: get_monmap_and_config 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 10 monclient: build_initial_monmap 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 10 monclient: monmap: 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:16.535876+0000 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:16.535876+0000 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:16.537 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:16.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 1 Processor -- start 2024-09-15T01:59:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 1 -- start start 2024-09-15T01:59:16.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.535+0000 7f5c9ce57640 10 monclient: init 2024-09-15T01:59:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:59:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-b con 0x7f5c98097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:16.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 0x7f5c9812e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9809a590 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9809a960 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c980985d0 con 0x7f5c9809a590 2024-09-15T01:59:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9809d200 con 0x7f5c9809a960 2024-09-15T01:59:16.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98097c30 con 0x7f5c98097320 2024-09-15T01:59:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9ce57640 10 monclient(hunting): authenticate will time out at 1216.532520s 2024-09-15T01:59:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:53100/0 (socket says 172.21.15.17:53100) 2024-09-15T01:59:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 -- 172.21.15.17:0/32768906 learned_addr learned my addr 172.21.15.17:0/32768906 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): get_auth_request con 0x7f5c98097320 auth_method 0 2024-09-15T01:59:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 0x7f5c9812e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): get_auth_request con 0x7f5c9809a590 auth_method 0 2024-09-15T01:59:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 10 monclient(hunting): get_auth_request con 0x7f5c9809a960 auth_method 0 2024-09-15T01:59:16.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 1 -- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 msgr2=0x7f5c9812ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 1 -- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 msgr2=0x7f5c9812ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:16.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient(hunting): handle_auth_done global_id 94448 payload 274 2024-09-15T01:59:16.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c96d9d640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:16.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 -- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 msgr2=0x7f5c9812ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:16.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 -- 172.21.15.17:0/32768906 >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 msgr2=0x7f5c9812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:16.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 0x7f5c9812e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:16.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 monclient: found mon.noname-b 2024-09-15T01:59:16.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:16.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:16.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c9659c640 1 -- 172.21.15.17:0/32768906 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980992f0 con 0x7f5c98097320 2024-09-15T01:59:16.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.536+0000 7f5c95d9b640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 0x7f5c9812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:16.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.539+0000 7f5c9659c640 1 -- 172.21.15.17:0/32768906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 msgr2=0x7f5c980976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:16.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.539+0000 7f5c9659c640 1 --2- 172.21.15.17:0/32768906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5c98099a60 tx=0x7f5c880091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa806f98023dc605 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 1 -- 172.21.15.17:0/32768906 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88032440 con 0x7f5c98097320 2024-09-15T01:59:16.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:16.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: dump: 2024-09-15T01:59:16.564 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:16.564 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.564 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.565 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: _finish_auth 0 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 5 monclient: authenticate success, global_id 94448 2024-09-15T01:59:16.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 1 -- 172.21.15.17:0/32768906 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c880325a0 con 0x7f5c98097320 2024-09-15T01:59:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 1 -- 172.21.15.17:0/32768906 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88032940 con 0x7f5c98097320 2024-09-15T01:59:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 4 set_mon_vals no callback set 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9559a640 10 monclient: dump: 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.568 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.570 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 10 monclient: shutdown 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 -- 172.21.15.17:0/32768906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 msgr2=0x7f5c980976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/32768906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5c98099a60 tx=0x7f5c880091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 -- 172.21.15.17:0/32768906 shutdown_connections 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.67:3300/0 conn(0x7f5c9809a960 0x7f5c9812ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/32768906 >> v2:172.21.15.17:3300/0 conn(0x7f5c9809a590 0x7f5c9812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/32768906 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c98097320 0x7f5c980976f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 -- 172.21.15.17:0/32768906 >> 172.21.15.17:0/32768906 conn(0x7f5c980a1000 msgr2=0x7f5c980a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 -- 172.21.15.17:0/32768906 shutdown_connections 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.540+0000 7f5c9ce57640 1 -- 172.21.15.17:0/32768906 wait complete. 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.541+0000 7f5c9ce57640 10 monclient: build_initial_monmap 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.541+0000 7f5c9ce57640 10 monclient: monmap: 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.541+0000 7f5c9ce57640 1 Processor -- start 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.541+0000 7f5c9ce57640 1 -- start start 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.541+0000 7f5c9ce57640 10 monclient: init 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-a con 0x7f5c98097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9809a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 0x7f5c981301a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9809a960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:16.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9809ad50 con 0x7f5c98097320 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98131f00 con 0x7f5c9809a960 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98131c20 con 0x7f5c9809a590 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9ce57640 10 monclient(hunting): authenticate will time out at 1216.538122s 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52696/0 (socket says 172.21.15.17:52696) 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:53106/0 (socket says 172.21.15.17:53106) 2024-09-15T01:59:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 -- 172.21.15.17:0/472565670 learned_addr learned my addr 172.21.15.17:0/472565670 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): get_auth_request con 0x7f5c98097320 auth_method 0 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): get_auth_request con 0x7f5c9809a590 auth_method 0 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:16.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient(hunting): handle_auth_done global_id 104289 payload 824 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c96d9d640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 0x7f5c981301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 -- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 msgr2=0x7f5c9812fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 -- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 msgr2=0x7f5c981301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 0x7f5c981301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 monclient: found mon.noname-a 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c9659c640 1 -- 172.21.15.17:0/472565670 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c009390 con 0x7f5c98097320 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c96d9d640 10 monclient: get_auth_request con 0x7f5c9809a960 auth_method 0 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c95d9b640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.542+0000 7f5c96d9d640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 0x7f5c981301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9659c640 1 -- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 msgr2=0x7f5c9812f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9659c640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009210 tx=0x7f5c88032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880319c0 con 0x7f5c98097320 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 10 monclient: dump: 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.578 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9ce57640 5 monclient: authenticate success, global_id 104289 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88031b20 con 0x7f5c98097320 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8803d440 con 0x7f5c98097320 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9ce57640 10 monclient: _renew_subs 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9ce57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c88009450 con 0x7f5c98097320 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:16.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c867fc640 10 monclient: dump: 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:16.580 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.544+0000 7f5c94d99640 4 set_mon_vals no callback set 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c9ce57640 10 monclient: _renew_subs 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c9ce57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.545+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9806e980 con 0x7f5c98097320 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5c88039040 con 0x7f5c98097320 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c9ce57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c9ce57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c9ce57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c980724a0 con 0x7f5c98097320 2024-09-15T01:59:16.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c867fc640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 0x7f5c7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(200..200 src has 148..200) v4 ==== 11558+0+0 (secure 0 0 0) 0x7f5c8806d010 con 0x7f5c98097320 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c95d9b640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 0x7f5c7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.546+0000 7f5c95d9b640 10 monclient: get_auth_request con 0x7f5c78038740 auth_method 0 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.549+0000 7f5c95d9b640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 0x7f5c7803abf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c010820 tx=0x7f5c8c012040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.550+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5c8809ee40 con 0x7f5c98097320 2024-09-15T01:59:16.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.550+0000 7f5c867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:16.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.550+0000 7f5c867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:16.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c9ce57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c9ce57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c9ce57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c980a0c00 con 0x7f5c98097320 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c867fc640 1 -- 172.21.15.17:0/472565670 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f5c88042020 con 0x7f5c98097320 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.777+0000 7f5c867fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:16.780 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 msgr2=0x7f5c7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 0x7f5c7803abf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c010820 tx=0x7f5c8c012040 comp rx=0 tx=0).stop 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 10 monclient: shutdown 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 msgr2=0x7f5c9812f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009210 tx=0x7f5c88032000 comp rx=0 tx=0).stop 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 shutdown_connections 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5c78038740 0x7f5c7803abf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9809a960 0x7f5c981301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5c9809a590 0x7f5c9812fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 --2- 172.21.15.17:0/472565670 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5c98097320 0x7f5c9812f720 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 >> 172.21.15.17:0/472565670 conn(0x7f5c980a1000 msgr2=0x7f5c9812c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 shutdown_connections 2024-09-15T01:59:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:16.780+0000 7f5c9ce57640 1 -- 172.21.15.17:0/472565670 wait complete. 2024-09-15T01:59:16.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.834+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v201) v1 ==== 168+0+0 (secure 0 0 0) 0x7fbd440160b0 con 0x7fbd4c068e70 2024-09-15T01:59:16.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.834+0000 7fbd42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.834+0000 7fbd42ffd640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-15T01:59:16.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.937+0000 7fbd5139d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.937+0000 7fbd5139d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.937+0000 7fbd5139d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:16.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.937+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbd4c06f120 con 0x7fbd4c068e70 2024-09-15T01:59:16.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.938+0000 7fbd42ffd640 1 -- 172.21.15.67:0/2131727331 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v201) v1 ==== 147+0+0 (secure 0 0 0) 0x7fbd4400e700 con 0x7fbd4c068e70 2024-09-15T01:59:16.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.938+0000 7fbd42ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:16.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.938+0000 7fbd42ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-15T01:59:16.941 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h17' already exists 2024-09-15T01:59:16.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 msgr2=0x7fbd2c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 0x7fbd2c03aba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009fd0 tx=0x7fbd340091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 10 monclient: shutdown 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 msgr2=0x7fbd4c12eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 0x7fbd4c12eb20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b5b0 tx=0x7fbd440090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 shutdown_connections 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbd2c0386f0 0x7fbd2c03aba0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4c131e70 0x7fbd4c11bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbd4c098420 0x7fbd4c11b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 --2- 172.21.15.67:0/2131727331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd4c068e70 0x7fbd4c12eb20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 >> 172.21.15.67:0/2131727331 conn(0x7fbd4c0a5c10 msgr2=0x7fbd4c097170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:16.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 shutdown_connections 2024-09-15T01:59:16.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:16.940+0000 7fbd5139d640 1 -- 172.21.15.67:0/2131727331 wait complete. 2024-09-15T01:59:16.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:16.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-15T01:59:17.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.027+0000 7f59a5694640 10 monclient: get_monmap_and_config 2024-09-15T01:59:17.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.027+0000 7f59a5694640 10 monclient: build_initial_monmap 2024-09-15T01:59:17.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.027+0000 7f59a5694640 10 monclient: monmap: 2024-09-15T01:59:17.030 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:17.030 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:17.029072+0000 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:17.029072+0000 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:17.031 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.028+0000 7f59a5694640 1 Processor -- start 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.028+0000 7f59a5694640 1 -- start start 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.028+0000 7f59a5694640 10 monclient: init 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.028+0000 7f59a5694640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.028+0000 7f59a5694640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59a0131aa0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-b con 0x7f59a009f490 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59a00a0a60 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:17.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a00a1820 con 0x7f59a0131aa0 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a0132930 con 0x7f59a00a0a60 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00a1b00 con 0x7f59a009f490 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f59a5694640 10 monclient(hunting): authenticate will time out at 1189.498122s 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f59a00a0a60 0x7f59a0138de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59626/0 (socket says 172.21.15.67:59626) 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599f7fe640 1 -- 172.21.15.67:0/532761134 learned_addr learned my addr 172.21.15.67:0/532761134 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 1 --2- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): get_auth_request con 0x7f59a009f490 auth_method 0 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient(hunting): handle_auth_done global_id 94454 payload 274 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 1 -- 172.21.15.67:0/532761134 >> v1:172.21.15.67:6789/0 conn(0x7f59a00a0a60 legacy=0x7f59a0138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 1 -- 172.21.15.67:0/532761134 >> v1:172.21.15.17:6789/0 conn(0x7f59a0131aa0 legacy=0x7f59a009ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 1 monclient: found mon.noname-b 2024-09-15T01:59:17.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:17.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.029+0000 7f599e7fc640 1 -- 172.21.15.67:0/532761134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00a03d0 con 0x7f59a009f490 2024-09-15T01:59:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.032+0000 7f599e7fc640 1 -- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 msgr2=0x7f59a0131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.032+0000 7f599e7fc640 1 --2- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0131e70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5988009880 tx=0x7f59880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f08f5dd0667698a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 1 -- 172.21.15.67:0/532761134 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988032440 con 0x7f59a009f490 2024-09-15T01:59:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: dump: 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 5 monclient: authenticate success, global_id 94454 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 1 -- 172.21.15.67:0/532761134 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59880325a0 con 0x7f59a009f490 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 4 set_mon_vals no callback set 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 1 -- 172.21.15.67:0/532761134 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988032940 con 0x7f59a009f490 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f599dffb640 10 monclient: dump: 2024-09-15T01:59:17.045 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 10 monclient: shutdown 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 -- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 msgr2=0x7f59a0131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 --2- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0131e70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5988009880 tx=0x7f59880090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 -- 172.21.15.67:0/532761134 shutdown_connections 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 --2- 172.21.15.67:0/532761134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0131e70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 -- 172.21.15.67:0/532761134 >> 172.21.15.67:0/532761134 conn(0x7f59a0071390 msgr2=0x7f59a00a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 -- 172.21.15.67:0/532761134 shutdown_connections 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.033+0000 7f59a5694640 1 -- 172.21.15.67:0/532761134 wait complete. 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.034+0000 7f59a5694640 10 monclient: build_initial_monmap 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.034+0000 7f59a5694640 10 monclient: monmap: 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.034+0000 7f59a5694640 1 Processor -- start 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.034+0000 7f59a5694640 1 -- start start 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.034+0000 7f59a5694640 10 monclient: init 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-c con 0x7f59a009f490 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 0x7f59a0134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-a con 0x7f59a00a0a60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): picked mon.noname-b con 0x7f59a0131aa0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0136090 con 0x7f59a00a0a60 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00a0f40 con 0x7f59a0131aa0 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0137c10 con 0x7f59a009f490 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:17.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f59a5694640 10 monclient(hunting): authenticate will time out at 1189.504108s 2024-09-15T01:59:17.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36000/0 (socket says 172.21.15.67:36000) 2024-09-15T01:59:17.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 -- 172.21.15.67:0/1140388259 learned_addr learned my addr 172.21.15.67:0/1140388259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:17.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 10 monclient(hunting): get_auth_request con 0x7f59a0131aa0 auth_method 0 2024-09-15T01:59:17.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 msgr2=0x7f59a0134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 msgr2=0x7f59a0134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599f7fe640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 0x7f59a0134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): get_auth_request con 0x7f59a00a0a60 auth_method 0 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): get_auth_request con 0x7f59a009f490 auth_method 0 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient(hunting): handle_auth_done global_id 94457 payload 824 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 msgr2=0x7f59a0134ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:17.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 msgr2=0x7f59a0134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 0x7f59a0134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 monclient: found mon.noname-c 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599effd640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5988009390 con 0x7f59a009f490 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.035+0000 7f599e7fc640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 0x7f59a0134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f599effd640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 msgr2=0x7f59a0138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f599effd640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0138840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f599800c510 tx=0x7f5998007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980117a0 con 0x7f59a009f490 2024-09-15T01:59:17.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: dump: 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998011920 con 0x7f59a009f490 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011ce0 con 0x7f59a009f490 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f5996ffd640 10 monclient: dump: 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:17.057 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 5 monclient: authenticate success, global_id 94457 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 10 monclient: _renew_subs 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a0136560 con 0x7f59a009f490 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f599d7fa640 4 set_mon_vals no callback set 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 10 monclient: _renew_subs 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.038+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a0136c20 con 0x7f59a009f490 2024-09-15T01:59:17.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5998012630 con 0x7f59a009f490 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f5996ffd640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 0x7f598003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f59a5694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f59a5694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f59a5694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(201..201 src has 148..201) v4 ==== 11771+0+0 (secure 0 0 0) 0x7f599804d230 con 0x7f59a009f490 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a00639e0 con 0x7f59a009f490 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f599e7fc640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 0x7f598003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.039+0000 7f599e7fc640 10 monclient: get_auth_request con 0x7f5980038740 auth_method 0 2024-09-15T01:59:17.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.041+0000 7f599e7fc640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 0x7f598003abf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5988034000 tx=0x7f5988032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:17.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.044+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5998010ac0 con 0x7f59a009f490 2024-09-15T01:59:17.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.044+0000 7f5996ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:17.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.044+0000 7f5996ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:17.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.210+0000 7f59a5694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.210+0000 7f59a5694640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.210+0000 7f59a5694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.210+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f59a006f0b0 con 0x7f59a009f490 2024-09-15T01:59:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.793+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v202) v1 ==== 168+0+0 (secure 0 0 0) 0x7f59980102e0 con 0x7f59a009f490 2024-09-15T01:59:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.793+0000 7f5996ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.793+0000 7f5996ffd640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-15T01:59:17.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.896+0000 7f59a5694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.896+0000 7f59a5694640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.896+0000 7f59a5694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.896+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f59a0138de0 con 0x7f59a009f490 2024-09-15T01:59:17.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.897+0000 7f5996ffd640 1 -- 172.21.15.67:0/1140388259 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v202) v1 ==== 147+0+0 (secure 0 0 0) 0x7f59980104c0 con 0x7f59a009f490 2024-09-15T01:59:17.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.897+0000 7f5996ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:17.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.897+0000 7f5996ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-15T01:59:17.900 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h18' already exists 2024-09-15T01:59:17.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 msgr2=0x7f598003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:17.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 0x7f598003abf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5988034000 tx=0x7f5988032000 comp rx=0 tx=0).stop 2024-09-15T01:59:17.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 10 monclient: shutdown 2024-09-15T01:59:17.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 msgr2=0x7f59a0138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0138840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f599800c510 tx=0x7f5998007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 shutdown_connections 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5980038740 0x7f598003abf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f59a00a0a60 0x7f59a0134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0131aa0 0x7f59a0134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 --2- 172.21.15.67:0/1140388259 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f59a009f490 0x7f59a0138840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 >> 172.21.15.67:0/1140388259 conn(0x7f59a0071390 msgr2=0x7f59a009e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.899+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 shutdown_connections 2024-09-15T01:59:17.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.900+0000 7f59a5694640 1 -- 172.21.15.67:0/1140388259 wait complete. 2024-09-15T01:59:17.911 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:17.911 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-15T01:59:17.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:17.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 10 monclient: build_initial_monmap 2024-09-15T01:59:17.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 10 monclient: monmap: 2024-09-15T01:59:17.990 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:17.989351+0000 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:17.989351+0000 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:17.991 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 1 Processor -- start 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 1 -- start start 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.988+0000 7fe89891d640 10 monclient: init 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-c con 0x7fe894131aa0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 0x7fe89409ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8940a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe89409f2c0 0x7fe894068e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:17.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-a con 0x7fe89409f2c0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89409f6e0 con 0x7fe89409f2c0 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8940a1820 con 0x7fe894131aa0 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe894132930 con 0x7fe8940a0a60 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89891d640 10 monclient(hunting): authenticate will time out at 1190.458454s 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36002/0 (socket says 172.21.15.67:36002) 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 -- 172.21.15.67:0/2384466951 learned_addr learned my addr 172.21.15.67:0/2384466951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 10 monclient(hunting): get_auth_request con 0x7fe894131aa0 auth_method 0 2024-09-15T01:59:17.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 -- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 msgr2=0x7fe894131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 -- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 msgr2=0x7fe894131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe89259c640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 1 --2- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 0x7fe89409ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe892d9d640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.17:3300/0 conn(0x7fe89409f2c0 0x7fe894068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): get_auth_request con 0x7fe8940a0a60 auth_method 0 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe892d9d640 10 monclient(hunting): get_auth_request con 0x7fe89409f2c0 auth_method 0 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe892d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.989+0000 7fe88bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 10 monclient(hunting): handle_auth_done global_id 94460 payload 274 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 -- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 msgr2=0x7fe894131e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 -- 172.21.15.67:0/2384466951 >> v2:172.21.15.17:3300/0 conn(0x7fe89409f2c0 msgr2=0x7fe894068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.17:3300/0 conn(0x7fe89409f2c0 0x7fe894068e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 monclient: found mon.noname-b 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.990+0000 7fe88bfff640 1 -- 172.21.15.67:0/2384466951 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940a1b00 con 0x7fe8940a0a60 2024-09-15T01:59:18.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe88bfff640 1 -- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 msgr2=0x7fe89409ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:18.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe88bfff640 1 --2- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 0x7fe89409ed80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe89406a5e0 tx=0x7fe87c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=624f8e7fa0ec1c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:18.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 1 -- 172.21.15.67:0/2384466951 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c02adf0 con 0x7fe8940a0a60 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: dump: 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:18.004 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: _finish_auth 0 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 1 -- 172.21.15.67:0/2384466951 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c032440 con 0x7fe8940a0a60 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 1 -- 172.21.15.67:0/2384466951 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0327e0 con 0x7fe8940a0a60 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe891d9b640 10 monclient: dump: 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:18.005 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 5 monclient: authenticate success, global_id 94460 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 4 set_mon_vals no callback set 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 10 monclient: shutdown 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 -- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 msgr2=0x7fe89409ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 --2- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 0x7fe89409ed80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe89406a5e0 tx=0x7fe87c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 -- 172.21.15.67:0/2384466951 shutdown_connections 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.17:3300/0 conn(0x7fe89409f2c0 0x7fe894068e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 --2- 172.21.15.67:0/2384466951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8940a0a60 0x7fe89409ed80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 --2- 172.21.15.67:0/2384466951 >> v2:172.21.15.67:3300/0 conn(0x7fe894131aa0 0x7fe894131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.993+0000 7fe89891d640 1 -- 172.21.15.67:0/2384466951 >> 172.21.15.67:0/2384466951 conn(0x7fe894071390 msgr2=0x7fe8940a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 1 -- 172.21.15.67:0/2384466951 shutdown_connections 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 1 -- 172.21.15.67:0/2384466951 wait complete. 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 10 monclient: build_initial_monmap 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 10 monclient: monmap: 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:18.007 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 1 Processor -- start 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 1 -- start start 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.994+0000 7fe89891d640 10 monclient: init 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-c con 0x7fe89409f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8940a0a60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 0x7fe8941314a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:18.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): picked mon.noname-a con 0x7fe894131aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89411f370 con 0x7fe894131aa0 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89411da70 con 0x7fe8940a0a60 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89411f090 con 0x7fe89409f2c0 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89891d640 10 monclient(hunting): authenticate will time out at 1190.464263s 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36014/0 (socket says 172.21.15.67:36014) 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 -- 172.21.15.67:0/3091697160 learned_addr learned my addr 172.21.15.67:0/3091697160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 10 monclient(hunting): get_auth_request con 0x7fe8940a0a60 auth_method 0 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:18.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 msgr2=0x7fe894130f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 msgr2=0x7fe894130f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe88bfff640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): get_auth_request con 0x7fe89409f2c0 auth_method 0 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 0x7fe8941314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): get_auth_request con 0x7fe894131aa0 auth_method 0 2024-09-15T01:59:18.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient(hunting): handle_auth_done global_id 94463 payload 824 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 msgr2=0x7fe894130f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 msgr2=0x7fe8941314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 0x7fe8941314a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 monclient: found mon.noname-c 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe89259c640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009390 con 0x7fe89409f2c0 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.995+0000 7fe892d9d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 0x7fe8941314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89259c640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 msgr2=0x7fe8941411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89259c640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe884006fd0 tx=0x7fe884008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400c6a0 con 0x7fe89409f2c0 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: dump: 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:18.012 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 5 monclient: authenticate success, global_id 94463 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88400c800 con 0x7fe89409f2c0 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 10 monclient: _renew_subs 2024-09-15T01:59:18.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89411d040 con 0x7fe89409f2c0 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400cbc0 con 0x7fe89409f2c0 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe88a7fc640 10 monclient: dump: 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 10 monclient: _renew_subs 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89159a640 4 set_mon_vals no callback set 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.998+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89411d580 con 0x7fe89409f2c0 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe89891d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe89891d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe89891d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe89259c640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe884006fd0 tx=0x7fe884008040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fe884013410 mgrmap(e 5) v1 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89411f530 con 0x7fe89409f2c0 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe884013410 con 0x7fe89409f2c0 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe88a7fc640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 0x7fe87403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe88bfff640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 0x7fe87403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:18.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:17.999+0000 7fe88bfff640 10 monclient: get_auth_request con 0x7fe874038740 auth_method 0 2024-09-15T01:59:18.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.002+0000 7fe88bfff640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 0x7fe87403abf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe87c035040 tx=0x7fe87c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:18.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.004+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(202..202 src has 148..202) v4 ==== 11984+0+0 (secure 0 0 0) 0x7fe88404d7f0 con 0x7fe89409f2c0 2024-09-15T01:59:18.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.827+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe88401c400 con 0x7fe89409f2c0 2024-09-15T01:59:18.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.827+0000 7fe88a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:18.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.827+0000 7fe88a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:19.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.999+0000 7fe89891d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.999+0000 7fe89891d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.999+0000 7fe89891d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:18.999+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe8940639e0 con 0x7fe89409f2c0 2024-09-15T01:59:19.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:19.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.846+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v203) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe884017030 con 0x7fe89409f2c0 2024-09-15T01:59:19.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.846+0000 7fe88a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.846+0000 7fe88a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-15T01:59:19.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.901+0000 7f06f3c68640 10 monclient: get_monmap_and_config 2024-09-15T01:59:19.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.901+0000 7f06f3c68640 10 monclient: build_initial_monmap 2024-09-15T01:59:19.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.901+0000 7f06f3c68640 10 monclient: monmap: 2024-09-15T01:59:19.905 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:19.905 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:19.902763+0000 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:19.902763+0000 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.902+0000 7f06f3c68640 1 Processor -- start 2024-09-15T01:59:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.902+0000 7f06f3c68640 1 -- start start 2024-09-15T01:59:19.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.902+0000 7f06f3c68640 10 monclient: init 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06ec068f20 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f06ec0692f0 0x7f06ec0696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-a con 0x7f06ec0692f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-b con 0x7f06ec136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec0a1950 con 0x7f06ec068f20 2024-09-15T01:59:19.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec0a1670 con 0x7f06ec0692f0 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1328d0 con 0x7f06ec136fe0 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f3c68640 10 monclient(hunting): authenticate will time out at 1219.899359s 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:53112/0 (socket says 172.21.15.17:53112) 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 -- 172.21.15.17:0/2442984806 learned_addr learned my addr 172.21.15.17:0/2442984806 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:19.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): get_auth_request con 0x7f06ec136fe0 auth_method 0 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 1 --2- 172.21.15.17:0/2442984806 >> v2:172.21.15.17:3300/0 conn(0x7f06ec0692f0 0x7f06ec0696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): get_auth_request con 0x7f06ec0692f0 auth_method 0 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:19.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient(hunting): handle_auth_done global_id 94466 payload 274 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 -- 172.21.15.17:0/2442984806 >> v2:172.21.15.17:3300/0 conn(0x7f06ec0692f0 msgr2=0x7f06ec0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 --2- 172.21.15.17:0/2442984806 >> v2:172.21.15.17:3300/0 conn(0x7f06ec0692f0 0x7f06ec0696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 -- 172.21.15.17:0/2442984806 >> v1:172.21.15.67:6789/0 conn(0x7f06ec068f20 legacy=0x7f06ec1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient: _finish_hunting 0 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 monclient: found mon.noname-b 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.903+0000 7f06f2205640 1 -- 172.21.15.17:0/2442984806 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0a0020 con 0x7f06ec136fe0 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f2205640 1 -- 172.21.15.17:0/2442984806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 msgr2=0x7f06ec1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:19.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f2205640 1 --2- 172.21.15.17:0/2442984806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009f50 tx=0x7f06e4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167488d79fb62d7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 1 -- 172.21.15.17:0/2442984806 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4007610 con 0x7f06ec136fe0 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 10 monclient: dump: 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:19.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 10 monclient: _finish_auth 0 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.906+0000 7f06f0a02640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 5 monclient: authenticate success, global_id 94466 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 1 -- 172.21.15.17:0/2442984806 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e402d050 con 0x7f06ec136fe0 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 1 -- 172.21.15.17:0/2442984806 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e40079a0 con 0x7f06ec136fe0 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 4 set_mon_vals no callback set 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:19.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f0a02640 10 monclient: dump: 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 10 monclient: shutdown 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 -- 172.21.15.17:0/2442984806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 msgr2=0x7f06ec1393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 --2- 172.21.15.17:0/2442984806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009f50 tx=0x7f06e4009840 comp rx=0 tx=0).stop 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 -- 172.21.15.17:0/2442984806 shutdown_connections 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 --2- 172.21.15.17:0/2442984806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec136fe0 0x7f06ec1393c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 --2- 172.21.15.17:0/2442984806 >> v2:172.21.15.17:3300/0 conn(0x7f06ec0692f0 0x7f06ec0696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 -- 172.21.15.17:0/2442984806 >> 172.21.15.17:0/2442984806 conn(0x7f06ec070fb0 msgr2=0x7f06ec0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 -- 172.21.15.17:0/2442984806 shutdown_connections 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.907+0000 7f06f3c68640 1 -- 172.21.15.17:0/2442984806 wait complete. 2024-09-15T01:59:19.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.908+0000 7f06f3c68640 10 monclient: build_initial_monmap 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.908+0000 7f06f3c68640 10 monclient: monmap: 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.908+0000 7f06f3c68640 1 Processor -- start 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.908+0000 7f06f3c68640 1 -- start start 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.908+0000 7f06f3c68640 10 monclient: init 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 0x7f06ec099ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-a con 0x7f06ec068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-c con 0x7f06ec0692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): picked mon.noname-b con 0x7f06ec136fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec11c6c0 con 0x7f06ec068f20 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec09c550 con 0x7f06ec136fe0 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec11b870 con 0x7f06ec0692f0 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f3c68640 10 monclient(hunting): authenticate will time out at 1219.905201s 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 0x7f06ec099ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:53122/0 (socket says 172.21.15.17:53122) 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 -- 172.21.15.17:0/1930904611 learned_addr learned my addr 172.21.15.17:0/1930904611 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): get_auth_request con 0x7f06ec0692f0 auth_method 0 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:19.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): get_auth_request con 0x7f06ec068f20 auth_method 0 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:19.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 10 monclient(hunting): get_auth_request con 0x7f06ec136fe0 auth_method 0 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient(hunting): handle_auth_done global_id 94469 payload 824 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 msgr2=0x7f06ec09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 msgr2=0x7f06ec09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 msgr2=0x7f06ec09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:19.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f2205640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 msgr2=0x7f06ec099ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 0x7f06ec099ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient: _finish_hunting 0 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 monclient: found mon.noname-c 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1203640 1 -- 172.21.15.17:0/1930904611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4009ae0 con 0x7f06ec0692f0 2024-09-15T01:59:19.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.909+0000 7f06f1a04640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 0x7f06ec099ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.911+0000 7f06f1203640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 msgr2=0x7f06ec09a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:19.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.911+0000 7f06f1203640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00ea50 tx=0x7f06dc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc009600 con 0x7f06ec0692f0 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: dump: 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:19.929 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 5 monclient: authenticate success, global_id 94469 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc00cc20 con 0x7f06ec0692f0 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 10 monclient: _renew_subs 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec09bb40 con 0x7f06ec0692f0 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc013470 con 0x7f06ec0692f0 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:19.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e1ffb640 10 monclient: dump: 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 10 monclient: _renew_subs 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec1317c0 con 0x7f06ec0692f0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.912+0000 7f06e3fff640 4 set_mon_vals no callback set 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f06dc013860 con 0x7f06ec0692f0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f3c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f3c68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f3c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec0724a0 con 0x7f06ec0692f0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06e1ffb640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 0x7f06d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(203..203 src has 148..203) v4 ==== 12197+0+0 (secure 0 0 0) 0x7f06dc04db30 con 0x7f06ec0692f0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f1a04640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 0x7f06d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.913+0000 7f06f1a04640 10 monclient: get_auth_request con 0x7f06d0038740 auth_method 0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.916+0000 7f06f1a04640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 0x7f06d003abf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06e8007fd0 tx=0x7f06e8007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.917+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f06dc023ad0 con 0x7f06ec0692f0 2024-09-15T01:59:19.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.917+0000 7f06e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:19.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:19.917+0000 7f06e1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.950+0000 7fe89891d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.950+0000 7fe89891d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.950+0000 7fe89891d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:19.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.950+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe89406f0b0 con 0x7fe89409f2c0 2024-09-15T01:59:19.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.952+0000 7fe88a7fc640 1 -- 172.21.15.67:0/3091697160 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v203) v1 ==== 147+0+0 (secure 0 0 0) 0x7fe884017030 con 0x7fe89409f2c0 2024-09-15T01:59:19.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.952+0000 7fe88a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:19.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.952+0000 7fe88a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-15T01:59:19.954 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h19' already exists 2024-09-15T01:59:19.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 msgr2=0x7fe87403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:19.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 0x7fe87403abf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe87c035040 tx=0x7fe87c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:19.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 10 monclient: shutdown 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 msgr2=0x7fe8941411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.953+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe884006fd0 tx=0x7fe884008040 comp rx=0 tx=0).stop 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 shutdown_connections 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe874038740 0x7fe87403abf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe894131aa0 0x7fe8941314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8940a0a60 0x7fe894130f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 --2- 172.21.15.67:0/3091697160 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89409f2c0 0x7fe8941411e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 >> 172.21.15.67:0/3091697160 conn(0x7fe894071390 msgr2=0x7fe89409d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 shutdown_connections 2024-09-15T01:59:19.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:19.954+0000 7fe89891d640 1 -- 172.21.15.67:0/3091697160 wait complete. 2024-09-15T01:59:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-15T01:59:20.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.043+0000 7f5e4bb9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.043+0000 7f5e4bb9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.043+0000 7f5e4bb9d640 10 monclient: monmap: 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:20.044457+0000 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:20.044457+0000 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:20.046 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:20.047 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:20.047 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.043+0000 7f5e4bb9d640 1 Processor -- start 2024-09-15T01:59:20.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 -- start start 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient: init 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e44131e70 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e44098420 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 0x7f5e44139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e441371a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e440a13e0 con 0x7f5e44131e70 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:20.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e440a0440 con 0x7f5e44098420 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e440a01e0 con 0x7f5e441371a0 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4bb9d640 10 monclient(hunting): authenticate will time out at 1192.513683s 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36024/0 (socket says 172.21.15.67:36024) 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 -- 172.21.15.67:0/2248061067 learned_addr learned my addr 172.21.15.67:0/2248061067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 10 monclient(hunting): get_auth_request con 0x7f5e44098420 auth_method 0 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 1 --2- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 0x7f5e44139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): get_auth_request con 0x7f5e441371a0 auth_method 0 2024-09-15T01:59:20.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 -- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 msgr2=0x7f5e44098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 -- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 msgr2=0x7f5e44098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 --2- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e49138640 1 --2- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.044+0000 7f5e4a13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 10 monclient(hunting): handle_auth_done global_id 94472 payload 274 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 msgr2=0x7f5e44098850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 1 --2- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2248061067 >> v1:172.21.15.17:6789/0 conn(0x7f5e44131e70 legacy=0x7f5e44132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 1 monclient: found mon.noname-b 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.045+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2248061067 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440a19b0 con 0x7f5e441371a0 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 msgr2=0x7f5e44139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4a13a640 1 --2- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 0x7f5e44139580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009e60 tx=0x7f5e3c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e26039fdad81361e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 1 -- 172.21.15.67:0/2248061067 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c007760 con 0x7f5e441371a0 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: dump: 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:20.060 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: _finish_auth 0 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 1 -- 172.21.15.67:0/2248061067 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c02d050 con 0x7f5e441371a0 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:20.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 1 -- 172.21.15.67:0/2248061067 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c007af0 con 0x7f5e441371a0 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e48937640 10 monclient: dump: 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:20.062 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 5 monclient: authenticate success, global_id 94472 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 4 set_mon_vals no callback set 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 10 monclient: shutdown 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 msgr2=0x7f5e44139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 0x7f5e44139580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009e60 tx=0x7f5e3c009780 comp rx=0 tx=0).stop 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2248061067 shutdown_connections 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2248061067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e441371a0 0x7f5e44139580 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2248061067 >> v2:172.21.15.67:3300/0 conn(0x7f5e44098420 0x7f5e44098850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.048+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2248061067 >> 172.21.15.67:0/2248061067 conn(0x7f5e440a5c10 msgr2=0x7f5e440a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2248061067 shutdown_connections 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2248061067 wait complete. 2024-09-15T01:59:20.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 10 monclient: monmap: 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 1 Processor -- start 2024-09-15T01:59:20.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 1 -- start start 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.049+0000 7f5e4bb9d640 10 monclient: init 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 0x7f5e441330d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e44098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 0x7f5e44133610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e44131e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e441371a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44069d80 con 0x7f5e44131e70 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:20.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e441361f0 con 0x7f5e441371a0 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e441364d0 con 0x7f5e44098420 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4bb9d640 10 monclient(hunting): authenticate will time out at 1192.519340s 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 0x7f5e44133610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36026/0 (socket says 172.21.15.67:36026) 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2767230170 learned_addr learned my addr 172.21.15.67:0/2767230170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 10 monclient(hunting): get_auth_request con 0x7f5e441371a0 auth_method 0 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:20.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): get_auth_request con 0x7f5e44131e70 auth_method 0 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 0x7f5e441330d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): get_auth_request con 0x7f5e44098420 auth_method 0 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 msgr2=0x7f5e44133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 msgr2=0x7f5e44133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:20.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e4a13a640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient(hunting): handle_auth_done global_id 104307 payload 824 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 msgr2=0x7f5e441330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 0x7f5e441330d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 msgr2=0x7f5e44133b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient: _finish_hunting 0 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 monclient: found mon.noname-a 2024-09-15T01:59:20.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49138640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c009a20 con 0x7f5e44131e70 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.050+0000 7f5e49939640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 0x7f5e441330d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e49138640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 msgr2=0x7f5e44133610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e49138640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 0x7f5e44133610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e34006fd0 tx=0x7f5e34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3400c6a0 con 0x7f5e44131e70 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: dump: 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:20.069 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:20.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3400c800 con 0x7f5e44131e70 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3400cbc0 con 0x7f5e44131e70 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 5 monclient: authenticate success, global_id 104307 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e39ffb640 10 monclient: dump: 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 10 monclient: _renew_subs 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e3bfff640 4 set_mon_vals no callback set 2024-09-15T01:59:20.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e44134fb0 con 0x7f5e44131e70 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 10 monclient: _renew_subs 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.053+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e44134090 con 0x7f5e44131e70 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5e34013410 con 0x7f5e44131e70 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e4bb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e4bb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e4bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e39ffb640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 0x7f5e2803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e44134920 con 0x7f5e44131e70 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(203..203 src has 148..203) v4 ==== 12197+0+0 (secure 0 0 0) 0x7f5e3404d890 con 0x7f5e44131e70 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e49939640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 0x7f5e2803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:20.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.054+0000 7f5e49939640 10 monclient: get_auth_request con 0x7f5e28038740 auth_method 0 2024-09-15T01:59:20.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.056+0000 7f5e49939640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 0x7f5e2803abf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007fd0 tx=0x7f5e40007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:20.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.059+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5e3401c400 con 0x7f5e44131e70 2024-09-15T01:59:20.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.059+0000 7f5e39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:20.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.059+0000 7f5e39ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06f3c68640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06f3c68640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06f3c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f06ec136180 con 0x7f06ec0692f0 2024-09-15T01:59:20.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06e1ffb640 1 -- 172.21.15.17:0/1930904611 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f06dc0232f0 con 0x7f06ec0692f0 2024-09-15T01:59:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:20.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.144+0000 7f06e1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:20.147 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:20.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 msgr2=0x7f06d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 0x7f06d003abf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06e8007fd0 tx=0x7f06e8007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 10 monclient: shutdown 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 msgr2=0x7f06ec09a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00ea50 tx=0x7f06dc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 shutdown_connections 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f06d0038740 0x7f06d003abf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec136fe0 0x7f06ec09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f06ec0692f0 0x7f06ec09a020 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 --2- 172.21.15.17:0/1930904611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06ec068f20 0x7f06ec099ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 >> 172.21.15.17:0/1930904611 conn(0x7f06ec070fb0 msgr2=0x7f06ec09dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:20.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 shutdown_connections 2024-09-15T01:59:20.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:20.146+0000 7f06f3c68640 1 -- 172.21.15.17:0/1930904611 wait complete. 2024-09-15T01:59:20.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.224+0000 7f5e4bb9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.224+0000 7f5e4bb9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.224+0000 7f5e4bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.224+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5e440639e0 con 0x7f5e44131e70 2024-09-15T01:59:20.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.847+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v204) v1 ==== 168+0+0 (secure 0 0 0) 0x7f5e34017030 con 0x7f5e44131e70 2024-09-15T01:59:20.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.847+0000 7f5e39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.847+0000 7f5e39ffb640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-15T01:59:20.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.948+0000 7f5e4bb9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.948+0000 7f5e4bb9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.948+0000 7f5e4bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:20.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.948+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5e4409c020 con 0x7f5e44131e70 2024-09-15T01:59:20.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.949+0000 7f5e39ffb640 1 -- 172.21.15.67:0/2767230170 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v204) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5e34017030 con 0x7f5e44131e70 2024-09-15T01:59:20.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.949+0000 7f5e39ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:20.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.949+0000 7f5e39ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-15T01:59:20.951 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h20' already exists 2024-09-15T01:59:20.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 msgr2=0x7f5e2803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 0x7f5e2803abf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007fd0 tx=0x7f5e40007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 10 monclient: shutdown 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 msgr2=0x7f5e44133610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 0x7f5e44133610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e34006fd0 tx=0x7f5e34008040 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 shutdown_connections 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5e28038740 0x7f5e2803abf0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5e44131e70 0x7f5e44133610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e441371a0 0x7f5e44133b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 --2- 172.21.15.67:0/2767230170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5e44098420 0x7f5e441330d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 >> 172.21.15.67:0/2767230170 conn(0x7f5e440a5c10 msgr2=0x7f5e441314f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 shutdown_connections 2024-09-15T01:59:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:20.951+0000 7f5e4bb9d640 1 -- 172.21.15.67:0/2767230170 wait complete. 2024-09-15T01:59:20.965 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:20.965 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.037+0000 7f64ed51d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.037+0000 7f64ed51d640 10 monclient: build_initial_monmap 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.038+0000 7f64ed51d640 10 monclient: monmap: 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:21.040 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:21.039163+0000 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:21.039163+0000 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:21.041 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.038+0000 7f64ed51d640 1 Processor -- start 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.038+0000 7f64ed51d640 1 -- start start 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.038+0000 7f64ed51d640 10 monclient: init 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f64e8131aa0 0x7f64e8131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-a con 0x7f64e8131aa0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:21.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64e80a0a60 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 0x7f64e8138de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-b con 0x7f64e809f490 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80a1820 con 0x7f64e80a0a60 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e8132930 con 0x7f64e8131aa0 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80a1b00 con 0x7f64e809f490 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64ed51d640 10 monclient(hunting): authenticate will time out at 1193.508212s 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e67fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64e80a0a60 0x7f64e809ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60616/0 (socket says 172.21.15.67:60616) 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e67fc640 1 -- 172.21.15.67:0/951246781 learned_addr learned my addr 172.21.15.67:0/951246781 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 --2- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 0x7f64e8138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/951246781 >> v2:172.21.15.17:3300/0 conn(0x7f64e8131aa0 0x7f64e8131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): get_auth_request con 0x7f64e809f490 auth_method 0 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): get_auth_request con 0x7f64e8131aa0 auth_method 0 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:21.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient(hunting): handle_auth_done global_id 94478 payload 274 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 -- 172.21.15.67:0/951246781 >> v2:172.21.15.17:3300/0 conn(0x7f64e8131aa0 msgr2=0x7f64e8131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 --2- 172.21.15.67:0/951246781 >> v2:172.21.15.17:3300/0 conn(0x7f64e8131aa0 0x7f64e8131e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 -- 172.21.15.67:0/951246781 >> v1:172.21.15.67:6789/0 conn(0x7f64e80a0a60 legacy=0x7f64e809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 monclient: found mon.noname-b 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.039+0000 7f64e77fe640 1 -- 172.21.15.67:0/951246781 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80a03d0 con 0x7f64e809f490 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.042+0000 7f64e77fe640 1 -- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 msgr2=0x7f64e8138de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.042+0000 7f64e77fe640 1 --2- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 0x7f64e8138de0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009e60 tx=0x7f64e0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccef9c0eda222c0a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 1 -- 172.21.15.67:0/951246781 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0007760 con 0x7f64e809f490 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: dump: 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:21.054 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 5 monclient: authenticate success, global_id 94478 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:21.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 1 -- 172.21.15.67:0/951246781 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e002d050 con 0x7f64e809f490 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 1 -- 172.21.15.67:0/951246781 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0007af0 con 0x7f64e809f490 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64e5ffb640 10 monclient: dump: 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 4 set_mon_vals no callback set 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 10 monclient: shutdown 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 -- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 msgr2=0x7f64e8138de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 --2- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 0x7f64e8138de0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009e60 tx=0x7f64e0009780 comp rx=0 tx=0).stop 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 -- 172.21.15.67:0/951246781 shutdown_connections 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 --2- 172.21.15.67:0/951246781 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e809f490 0x7f64e8138de0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 --2- 172.21.15.67:0/951246781 >> v2:172.21.15.17:3300/0 conn(0x7f64e8131aa0 0x7f64e8131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 -- 172.21.15.67:0/951246781 >> 172.21.15.67:0/951246781 conn(0x7f64e8071390 msgr2=0x7f64e80a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 -- 172.21.15.67:0/951246781 shutdown_connections 2024-09-15T01:59:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.043+0000 7f64ed51d640 1 -- 172.21.15.67:0/951246781 wait complete. 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.044+0000 7f64ed51d640 10 monclient: build_initial_monmap 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.044+0000 7f64ed51d640 10 monclient: monmap: 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.044+0000 7f64ed51d640 1 Processor -- start 2024-09-15T01:59:21.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.044+0000 7f64ed51d640 1 -- start start 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.044+0000 7f64ed51d640 10 monclient: init 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-b con 0x7f64e809f490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 0x7f64e8096a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-c con 0x7f64e80a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 0x7f64e8096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): picked mon.noname-a con 0x7f64e8131aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:21.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80a2940 con 0x7f64e8131aa0 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e8099eb0 con 0x7f64e809f490 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80998d0 con 0x7f64e80a0a60 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64ed51d640 10 monclient(hunting): authenticate will time out at 1193.514296s 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59118/0 (socket says 172.21.15.67:59118) 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 -- 172.21.15.67:0/1751667208 learned_addr learned my addr 172.21.15.67:0/1751667208 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 10 monclient(hunting): get_auth_request con 0x7f64e809f490 auth_method 0 2024-09-15T01:59:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 msgr2=0x7f64e809a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 msgr2=0x7f64e809a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 0x7f64e8096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 0x7f64e8096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): get_auth_request con 0x7f64e80a0a60 auth_method 0 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): get_auth_request con 0x7f64e8131aa0 auth_method 0 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.045+0000 7f64e67fc640 10 monclient(hunting): handle_auth_done global_id 94481 payload 824 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 msgr2=0x7f64e809a890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 msgr2=0x7f64e8096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 0x7f64e8096fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 monclient: found mon.noname-c 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e67fc640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0009a20 con 0x7f64e80a0a60 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.046+0000 7f64e77fe640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 0x7f64e8096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64e67fc640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 msgr2=0x7f64e8096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64e67fc640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 0x7f64e8096a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c4c0 tx=0x7f64d8007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8005cb0 con 0x7f64e80a0a60 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: dump: 2024-09-15T01:59:21.063 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:21.064 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8005e30 con 0x7f64e80a0a60 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8007320 con 0x7f64e80a0a60 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64deffd640 10 monclient: dump: 2024-09-15T01:59:21.065 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64e57fa640 4 set_mon_vals no callback set 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 5 monclient: authenticate success, global_id 94481 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 10 monclient: _renew_subs 2024-09-15T01:59:21.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e8099bd0 con 0x7f64e80a0a60 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 10 monclient: _renew_subs 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.048+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e8098c70 con 0x7f64e80a0a60 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64ed51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64ed51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64ed51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e80639e0 con 0x7f64e80a0a60 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f64d801b3b0 con 0x7f64e80a0a60 2024-09-15T01:59:21.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.049+0000 7f64deffd640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 0x7f64c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:21.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.050+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(204..204 src has 148..204) v4 ==== 12410+0+0 (secure 0 0 0) 0x7f64d804cdc0 con 0x7f64e80a0a60 2024-09-15T01:59:21.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.050+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 0x7f64c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:21.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.050+0000 7f64e6ffd640 10 monclient: get_auth_request con 0x7f64c8038740 auth_method 0 2024-09-15T01:59:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.052+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 0x7f64c803abf0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009550 tx=0x7f64d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.054+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f64d8007710 con 0x7f64e80a0a60 2024-09-15T01:59:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.054+0000 7f64deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.054+0000 7f64deffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:21.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.221+0000 7f64ed51d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.221+0000 7f64ed51d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.221+0000 7f64ed51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.221+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f64e80a1540 con 0x7f64e80a0a60 2024-09-15T01:59:21.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.853+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v205) v1 ==== 168+0+0 (secure 0 0 0) 0x7f64d8017020 con 0x7f64e80a0a60 2024-09-15T01:59:21.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.853+0000 7f64deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.853+0000 7f64deffd640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-15T01:59:21.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.957+0000 7f64ed51d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.957+0000 7f64ed51d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.957+0000 7f64ed51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.957+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f64e809a020 con 0x7f64e80a0a60 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.959+0000 7f64deffd640 1 -- 172.21.15.67:0/1751667208 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v205) v1 ==== 147+0+0 (secure 0 0 0) 0x7f64d8018630 con 0x7f64e80a0a60 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.959+0000 7f64deffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.959+0000 7f64deffd640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-15T01:59:21.962 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h21' already exists 2024-09-15T01:59:21.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 msgr2=0x7f64c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:21.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 0x7f64c803abf0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009550 tx=0x7f64d0008040 comp rx=0 tx=0).stop 2024-09-15T01:59:21.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 10 monclient: shutdown 2024-09-15T01:59:21.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 msgr2=0x7f64e8096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 0x7f64e8096a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c4c0 tx=0x7f64d8007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 shutdown_connections 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f64c8038740 0x7f64c803abf0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64e8131aa0 0x7f64e8096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64e80a0a60 0x7f64e8096a70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 --2- 172.21.15.67:0/1751667208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e809f490 0x7f64e809a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 >> 172.21.15.67:0/1751667208 conn(0x7f64e8071390 msgr2=0x7f64e812cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 shutdown_connections 2024-09-15T01:59:21.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:21.961+0000 7f64ed51d640 1 -- 172.21.15.67:0/1751667208 wait complete. 2024-09-15T01:59:21.973 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:21.973 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-15T01:59:22.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:22.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 10 monclient: build_initial_monmap 2024-09-15T01:59:22.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 10 monclient: monmap: 2024-09-15T01:59:22.051 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:22.051 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:22.050316+0000 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:22.050316+0000 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:22.052 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 1 Processor -- start 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 1 -- start start 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.049+0000 7f865c99d640 10 monclient: init 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-b con 0x7f8658131aa0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86580a0a60 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 0x7f8658138de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-a con 0x7f865809f490 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580a1820 con 0x7f86580a0a60 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8658132930 con 0x7f865809f490 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580a1b00 con 0x7f8658131aa0 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865c99d640 10 monclient(hunting): authenticate will time out at 1194.519454s 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8655d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f86580a0a60 0x7f865809ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60630/0 (socket says 172.21.15.67:60630) 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8655d9b640 1 -- 172.21.15.67:0/3818983885 learned_addr learned my addr 172.21.15.67:0/3818983885 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 1 --2- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3818983885 >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 0x7f8658138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): get_auth_request con 0x7f8658131aa0 auth_method 0 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:22.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): get_auth_request con 0x7f865809f490 auth_method 0 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.050+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 10 monclient(hunting): handle_auth_done global_id 94484 payload 274 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 1 -- 172.21.15.67:0/3818983885 >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 msgr2=0x7f8658138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 1 --2- 172.21.15.67:0/3818983885 >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 0x7f8658138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 1 -- 172.21.15.67:0/3818983885 >> v1:172.21.15.67:6789/0 conn(0x7f86580a0a60 legacy=0x7f865809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 1 monclient: found mon.noname-b 2024-09-15T01:59:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f865659c640 1 -- 172.21.15.67:0/3818983885 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580a03d0 con 0x7f8658131aa0 2024-09-15T01:59:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.051+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3818983885 >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 0x7f8658138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.053+0000 7f865659c640 1 -- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 msgr2=0x7f8658131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.053+0000 7f865659c640 1 --2- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658131e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8648009880 tx=0x7f86480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8603d314b25733f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 1 -- 172.21.15.67:0/3818983885 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864802ae20 con 0x7f8658131aa0 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: dump: 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:22.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: _finish_auth 0 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 5 monclient: authenticate success, global_id 94484 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 1 -- 172.21.15.67:0/3818983885 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8648032440 con 0x7f8658131aa0 2024-09-15T01:59:22.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 1 -- 172.21.15.67:0/3818983885 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86480327e0 con 0x7f8658131aa0 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 4 set_mon_vals no callback set 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865559a640 10 monclient: dump: 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 10 monclient: shutdown 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 -- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 msgr2=0x7f8658131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 --2- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658131e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8648009880 tx=0x7f86480090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 -- 172.21.15.67:0/3818983885 shutdown_connections 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 --2- 172.21.15.67:0/3818983885 >> v2:172.21.15.17:3300/0 conn(0x7f865809f490 0x7f8658138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 --2- 172.21.15.67:0/3818983885 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658131e70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 -- 172.21.15.67:0/3818983885 >> 172.21.15.67:0/3818983885 conn(0x7f8658071390 msgr2=0x7f86580a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 -- 172.21.15.67:0/3818983885 shutdown_connections 2024-09-15T01:59:22.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.054+0000 7f865c99d640 1 -- 172.21.15.67:0/3818983885 wait complete. 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.055+0000 7f865c99d640 10 monclient: build_initial_monmap 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.055+0000 7f865c99d640 10 monclient: monmap: 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.055+0000 7f865c99d640 1 Processor -- start 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.055+0000 7f865c99d640 1 -- start start 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.055+0000 7f865c99d640 10 monclient: init 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 0x7f8658141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-a con 0x7f865809f490 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-b con 0x7f86580a0a60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): picked mon.noname-c con 0x7f8658131aa0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658137010 con 0x7f865809f490 2024-09-15T01:59:22.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658136740 con 0x7f86580a0a60 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580a0fc0 con 0x7f8658131aa0 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865c99d640 10 monclient(hunting): authenticate will time out at 1194.525229s 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59124/0 (socket says 172.21.15.67:59124) 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 -- 172.21.15.67:0/3422718435 learned_addr learned my addr 172.21.15.67:0/3422718435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 10 monclient(hunting): get_auth_request con 0x7f86580a0a60 auth_method 0 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 msgr2=0x7f8658141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 msgr2=0x7f8658141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8655d9b640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 0x7f8658141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): get_auth_request con 0x7f8658131aa0 auth_method 0 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): get_auth_request con 0x7f865809f490 auth_method 0 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:22.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient(hunting): handle_auth_done global_id 94487 payload 824 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 msgr2=0x7f8658141ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 msgr2=0x7f8658141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 0x7f8658141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:59:22.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 monclient: found mon.noname-c 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f8656d9d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650009a20 con 0x7f8658131aa0 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.056+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 0x7f8658141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f8656d9d640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 msgr2=0x7f8658134680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ee0 tx=0x7f865000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865000ce10 con 0x7f8658131aa0 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: dump: 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.073 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 5 monclient: authenticate success, global_id 94487 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 10 monclient: _renew_subs 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8648009390 con 0x7f8658131aa0 2024-09-15T01:59:22.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 10 monclient: _renew_subs 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8658136190 con 0x7f8658131aa0 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000f040 con 0x7f8658131aa0 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500148e0 con 0x7f8658131aa0 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f864e7fc640 10 monclient: dump: 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:22.075 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.059+0000 7f8654d99640 4 set_mon_vals no callback set 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865c99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865c99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86580639e0 con 0x7f8658131aa0 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f8656d9d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ee0 tx=0x7f865000b040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f865004e400 osd_map(205..205 src has 148..205) v4 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8650003020 con 0x7f8658131aa0 2024-09-15T01:59:22.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f864e7fc640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.060+0000 7f865659c640 10 monclient: get_auth_request con 0x7f8638038740 auth_method 0 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.061+0000 7f865659c640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 msgr2=0x7f863803abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.061+0000 7f865659c640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 msgr2=0x7f863803abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.061+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.061+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.065+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(205..205 src has 148..205) v4 ==== 12623+0+0 (secure 0 0 0) 0x7f865004e400 con 0x7f8658131aa0 2024-09-15T01:59:22.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.262+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:22.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.262+0000 7f865659c640 10 monclient: get_auth_request con 0x7f8638038740 auth_method 0 2024-09-15T01:59:22.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.264+0000 7f865659c640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f86580a5180 tx=0x7f8648032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:22.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.889+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8650006000 con 0x7f8658131aa0 2024-09-15T01:59:22.890 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.889+0000 7f864e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:22.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:22.889+0000 7f864e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:23.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:23.062+0000 7f865c99d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:23.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:23.062+0000 7f865c99d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:23.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:23.062+0000 7f865c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:23.062+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f86581378a0 con 0x7f8658131aa0 2024-09-15T01:59:23.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:23.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.266+0000 7f529ef9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.266+0000 7f529ef9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.266+0000 7f529ef9d640 10 monclient: monmap: 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:23.267459+0000 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:23.267459+0000 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.269 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:23.270 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:23.270 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:23.270 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:23.270 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:23.270 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.267+0000 7f529ef9d640 1 Processor -- start 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.267+0000 7f529ef9d640 1 -- start start 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.267+0000 7f529ef9d640 10 monclient: init 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.267+0000 7f529ef9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.267+0000 7f529ef9d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52980a5780 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:23.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f529809ccf0 0x7f529809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-c con 0x7f529809ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5298069c90 0x7f5298139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5298069c90 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52980a5b50 con 0x7f52980a5780 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52980a18b0 con 0x7f5298069c90 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529809c5f0 con 0x7f529809ccf0 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529ef9d640 10 monclient(hunting): authenticate will time out at 1223.264130s 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5298069c90 0x7f5298139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529cd39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f52980a5780 0x7f5298069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42620/0 (socket says 172.21.15.17:42620) 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529cd39640 1 -- 172.21.15.17:0/1011699908 learned_addr learned my addr 172.21.15.17:0/1011699908 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): get_auth_request con 0x7f5298069c90 auth_method 0 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f5297fff640 1 --2- 172.21.15.17:0/1011699908 >> v2:172.21.15.67:3300/0 conn(0x7f529809ccf0 0x7f529809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f5297fff640 10 monclient(hunting): get_auth_request con 0x7f529809ccf0 auth_method 0 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f5297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f5297fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f5297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient(hunting): handle_auth_done global_id 104322 payload 274 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 -- 172.21.15.17:0/1011699908 >> v2:172.21.15.67:3300/0 conn(0x7f529809ccf0 msgr2=0x7f529809bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 --2- 172.21.15.17:0/1011699908 >> v2:172.21.15.67:3300/0 conn(0x7f529809ccf0 0x7f529809bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 -- 172.21.15.17:0/1011699908 >> v1:172.21.15.17:6789/0 conn(0x7f52980a5780 legacy=0x7f5298069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 monclient: found mon.noname-a 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.268+0000 7f529d53a640 1 -- 172.21.15.17:0/1011699908 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529809d210 con 0x7f5298069c90 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f529d53a640 1 -- 172.21.15.17:0/1011699908 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5298069c90 msgr2=0x7f5298139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f529d53a640 1 --2- 172.21.15.17:0/1011699908 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5298069c90 0x7f5298139880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5298068e70 tx=0x7f5290009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a09f3ad9e9f677 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 1 -- 172.21.15.17:0/1011699908 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290007760 con 0x7f5298069c90 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: dump: 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.279 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 1 -- 172.21.15.17:0/1011699908 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529002d050 con 0x7f5298069c90 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 1 -- 172.21.15.17:0/1011699908 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290007af0 con 0x7f5298069c90 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f529ef9d640 5 monclient: authenticate success, global_id 104322 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f529ef9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f529ef9d640 4 set_mon_vals no callback set 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.271+0000 7f52977fe640 10 monclient: dump: 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 10 monclient: shutdown 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 -- 172.21.15.17:0/1011699908 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5298069c90 msgr2=0x7f5298139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 --2- 172.21.15.17:0/1011699908 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5298069c90 0x7f5298139880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5298068e70 tx=0x7f5290009780 comp rx=0 tx=0).stop 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 -- 172.21.15.17:0/1011699908 shutdown_connections 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 --2- 172.21.15.17:0/1011699908 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5298069c90 0x7f5298139880 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 --2- 172.21.15.17:0/1011699908 >> v2:172.21.15.67:3300/0 conn(0x7f529809ccf0 0x7f529809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 -- 172.21.15.17:0/1011699908 >> 172.21.15.17:0/1011699908 conn(0x7f5298070e10 msgr2=0x7f52980a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 -- 172.21.15.17:0/1011699908 shutdown_connections 2024-09-15T01:59:23.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 1 -- 172.21.15.17:0/1011699908 wait complete. 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.272+0000 7f529ef9d640 10 monclient: monmap: 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 Processor -- start 2024-09-15T01:59:23.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 -- start start 2024-09-15T01:59:23.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient: init 2024-09-15T01:59:23.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5298069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-b con 0x7f529809ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 0x7f529811b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): picked mon.noname-a con 0x7f52980a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529811f2d0 con 0x7f52980a5780 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52980a0920 con 0x7f529809ccf0 2024-09-15T01:59:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529809b4e0 con 0x7f5298069c90 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.273+0000 7f529ef9d640 10 monclient(hunting): authenticate will time out at 1223.269989s 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 0x7f529811b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36774/0 (socket says 172.21.15.17:36774) 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 1 -- 172.21.15.17:0/2514700924 learned_addr learned my addr 172.21.15.17:0/2514700924 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): get_auth_request con 0x7f5298069c90 auth_method 0 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): get_auth_request con 0x7f52980a5780 auth_method 0 2024-09-15T01:59:23.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 10 monclient(hunting): get_auth_request con 0x7f529809ccf0 auth_method 0 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 msgr2=0x7f5298131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 msgr2=0x7f5298131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f5297fff640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient(hunting): handle_auth_done global_id 104325 payload 824 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 msgr2=0x7f5298141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 msgr2=0x7f5298131b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 monclient: found mon.noname-a 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529d53a640 1 -- 172.21.15.17:0/2514700924 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290009a20 con 0x7f52980a5780 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.274+0000 7f529cd39640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f529d53a640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 msgr2=0x7f529811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f529d53a640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 0x7f529811b6d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f529806a800 tx=0x7f5290037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290031e20 con 0x7f52980a5780 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 10 monclient: dump: 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.276+0000 7f5294ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 5 monclient: authenticate success, global_id 104325 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 10 monclient: _renew_subs 2024-09-15T01:59:23.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529811d150 con 0x7f52980a5780 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5290045460 con 0x7f52980a5780 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290045820 con 0x7f52980a5780 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 10 monclient: dump: 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5296ffd640 4 set_mon_vals no callback set 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 10 monclient: _renew_subs 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298141fb0 con 0x7f52980a5780 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.277+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5290045c10 con 0x7f52980a5780 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529ef9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f5294ff9640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 0x7f527c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529ef9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529ef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52980724a0 con 0x7f52980a5780 2024-09-15T01:59:23.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(206..206 src has 148..206) v4 ==== 12623+0+0 (secure 0 0 0) 0x7f529006f600 con 0x7f52980a5780 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529cd39640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 0x7f527c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.278+0000 7f529cd39640 10 monclient: get_auth_request con 0x7f527c038740 auth_method 0 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.281+0000 7f529cd39640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 0x7f527c03abf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f528c006fd0 tx=0x7f528c004960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.283+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5290040030 con 0x7f52980a5780 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.283+0000 7f5294ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.283+0000 7f5294ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:23.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f529ef9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:23.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f529ef9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:23.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f529ef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f529806f4f0 con 0x7f52980a5780 2024-09-15T01:59:23.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f5294ff9640 1 -- 172.21.15.17:0/2514700924 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f5290040020 con 0x7f52980a5780 2024-09-15T01:59:23.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f5294ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:23.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.511+0000 7f5294ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:23.514 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:23.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 msgr2=0x7f527c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 0x7f527c03abf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f528c006fd0 tx=0x7f528c004960 comp rx=0 tx=0).stop 2024-09-15T01:59:23.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 10 monclient: shutdown 2024-09-15T01:59:23.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 msgr2=0x7f529811b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 0x7f529811b6d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f529806a800 tx=0x7f5290037000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 shutdown_connections 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f527c038740 0x7f527c03abf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52980a5780 0x7f529811b6d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f529809ccf0 0x7f5298131b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 --2- 172.21.15.17:0/2514700924 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5298069c90 0x7f5298141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 >> 172.21.15.17:0/2514700924 conn(0x7f5298070e10 msgr2=0x7f529809d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 shutdown_connections 2024-09-15T01:59:23.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.513+0000 7f529ef9d640 1 -- 172.21.15.17:0/2514700924 wait complete. 2024-09-15T01:59:23.526 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:59:23.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:59:23.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 10 monclient: get_monmap_and_config 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 10 monclient: monmap: 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:23.621905+0000 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:23.621905+0000 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.623 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:23.624 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:23.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:23.624 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:23.624 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:23.624 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 1 Processor -- start 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 1 -- start start 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.621+0000 7f80faee8640 10 monclient: init 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f80f40a58b0 0x7f80f4132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-a con 0x7f80f40a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 0x7f80f40984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-c con 0x7f80f40980a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 0x7f80f4068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-b con 0x7f80f40989e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f40a01e0 con 0x7f80f40a58b0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f40a1580 con 0x7f80f40980a0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40a5c80 con 0x7f80f40989e0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80faee8640 10 monclient(hunting): authenticate will time out at 1223.618450s 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f80f40a58b0 0x7f80f4132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 0x7f80f4068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f80f40a58b0 0x7f80f4132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51170/0 (socket says 172.21.15.17:51170) 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 -- 172.21.15.17:0/2433797853 learned_addr learned my addr 172.21.15.17:0/2433797853 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): get_auth_request con 0x7f80f40a58b0 auth_method 0 2024-09-15T01:59:23.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): get_auth_request con 0x7f80f40989e0 auth_method 0 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient(hunting): handle_auth_done global_id 104328 payload 274 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 -- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 msgr2=0x7f80f4068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 0x7f80f4068ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 -- 172.21.15.17:0/2433797853 >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 msgr2=0x7f80f40984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f3fff640 1 --2- 172.21.15.17:0/2433797853 >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 0x7f80f40984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 --2- 172.21.15.17:0/2433797853 >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 0x7f80f40984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 monclient: found mon.noname-a 2024-09-15T01:59:23.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f8c84640 1 -- 172.21.15.17:0/2433797853 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f41326c0 con 0x7f80f40a58b0 2024-09-15T01:59:23.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f3fff640 10 monclient: get_auth_request con 0x7f80f40980a0 auth_method 0 2024-09-15T01:59:23.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f9485640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 0x7f80f4068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:23.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.622+0000 7f80f3fff640 1 --2- 172.21.15.17:0/2433797853 >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 0x7f80f40984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T01:59:23.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f8c84640 1 -- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40a58b0 msgr2=0x7f80f4132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f8c84640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40a58b0 0x7f80f4132040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f80f41328b0 tx=0x7f80e40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164c2d1fed1cc3c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 1 -- 172.21.15.17:0/2433797853 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4032440 con 0x7f80f40a58b0 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: dump: 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80faee8640 5 monclient: authenticate success, global_id 104328 2024-09-15T01:59:23.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 1 -- 172.21.15.17:0/2433797853 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e40325a0 con 0x7f80f40a58b0 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 1 -- 172.21.15.17:0/2433797853 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4032940 con 0x7f80f40a58b0 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80faee8640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80faee8640 4 set_mon_vals no callback set 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.625+0000 7f80f37fe640 10 monclient: dump: 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.641 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 10 monclient: shutdown 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 -- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40a58b0 msgr2=0x7f80f4132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40a58b0 0x7f80f4132040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f80f41328b0 tx=0x7f80e40091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 -- 172.21.15.17:0/2433797853 shutdown_connections 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 --2- 172.21.15.17:0/2433797853 >> v2:172.21.15.67:3300/0 conn(0x7f80f40980a0 0x7f80f40984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40989e0 0x7f80f4068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 --2- 172.21.15.17:0/2433797853 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40a58b0 0x7f80f4132040 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 -- 172.21.15.17:0/2433797853 >> 172.21.15.17:0/2433797853 conn(0x7f80f4070ed0 msgr2=0x7f80f40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 -- 172.21.15.17:0/2433797853 shutdown_connections 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 1 -- 172.21.15.17:0/2433797853 wait complete. 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.626+0000 7f80faee8640 10 monclient: monmap: 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:23.643 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 1 Processor -- start 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 1 -- start start 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient: init 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 0x7f80f4135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-c con 0x7f80f40980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 0x7f80f411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-b con 0x7f80f40a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f411bd10 0x7f80f411c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.627+0000 7f80faee8640 10 monclient(hunting): picked mon.noname-a con 0x7f80f411bd10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f411cf40 con 0x7f80f411bd10 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f411e870 con 0x7f80f40a58b0 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f411cc90 con 0x7f80f40980a0 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80faee8640 10 monclient(hunting): authenticate will time out at 1223.624027s 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 0x7f80f4135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 0x7f80f4135100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36802/0 (socket says 172.21.15.17:36802) 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f411bd10 0x7f80f411c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 -- 172.21.15.17:0/1206856973 learned_addr learned my addr 172.21.15.17:0/1206856973 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): get_auth_request con 0x7f80f40980a0 auth_method 0 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): get_auth_request con 0x7f80f411bd10 auth_method 0 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 0x7f80f411b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 10 monclient(hunting): get_auth_request con 0x7f80f40a58b0 auth_method 0 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 msgr2=0x7f80f411b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 msgr2=0x7f80f411b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient(hunting): handle_auth_done global_id 94496 payload 824 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 0x7f80f411b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 0x7f80f411b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 msgr2=0x7f80f411b7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:23.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80f40a58b0 0x7f80f411b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f411bd10 msgr2=0x7f80f411c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f411bd10 0x7f80f411c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 monclient: found mon.noname-c 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f8c84640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec009a20 con 0x7f80f40980a0 2024-09-15T01:59:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.628+0000 7f80f9485640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f411bd10 0x7f80f411c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f8c84640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 msgr2=0x7f80f4135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f8c84640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 0x7f80f4135100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009210 tx=0x7f80e4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40319c0 con 0x7f80f40980a0 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: dump: 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4031b20 con 0x7f80f40980a0 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e403d440 con 0x7f80f40980a0 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 5 monclient: authenticate success, global_id 94496 2024-09-15T01:59:23.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 10 monclient: _renew_subs 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e4009450 con 0x7f80f40980a0 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f2ffd640 4 set_mon_vals no callback set 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80f0ff9640 10 monclient: dump: 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.651 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 10 monclient: _renew_subs 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.630+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f411e590 con 0x7f80f40980a0 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80faee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80faee8640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 0x7f80f4099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f4063b20 con 0x7f80f40989e0 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 0x7f80f4099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient: get_auth_request con 0x7f80f40989e0 auth_method 0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f80e4039040 con 0x7f80f40980a0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.631+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f3fff640 10 monclient(hunting): handle_auth_done global_id 104334 payload 824 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80f40724a0 con 0x7f80f40989e0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f0ff9640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80e0038740 0x7f80e003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(206..206 src has 148..206) v4 ==== 12623+0+0 (secure 0 0 0) 0x7f80e406d290 con 0x7f80f40980a0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f9485640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80e0038740 0x7f80e003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.632+0000 7f80f9485640 10 monclient: get_auth_request con 0x7f80e0038740 auth_method 0 2024-09-15T01:59:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 msgr2=0x7f80f4099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 0x7f80f4099fe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80e800f490 tx=0x7f80e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8014980 con 0x7f80f40989e0 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 10 monclient: dump: 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f80e800c030 con 0x7f80f40989e0 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 msgr2=0x7f80f4099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 0x7f80f4099fe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80e800f490 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 msgr2=0x7f80f4099fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.634+0000 7f80f0ff9640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f40989e0 0x7f80f4099fe0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80e800f490 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2024-09-15T01:59:23.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.635+0000 7f80f9485640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80e0038740 0x7f80e003abf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00eb00 tx=0x7f80ec00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80faee8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80faee8640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 0x7f80f409a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80faee8640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f406a120 con 0x7f80f409a520 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 0x7f80f409a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient: get_auth_request con 0x7f80f409a520 auth_method 0 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 10 monclient(hunting): handle_auth_done global_id 104337 payload 824 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.640+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f80f409fc90 con 0x7f80f409a520 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f3fff640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 msgr2=0x7f80f409a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f3fff640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 0x7f80f409a920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f80e8019000 tx=0x7f80e801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8014e80 con 0x7f80f409a520 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 10 monclient: dump: 2024-09-15T01:59:23.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7f80e8013800 con 0x7f80f409a520 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:23.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 msgr2=0x7f80f409a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 0x7f80f409a920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f80e8019000 tx=0x7f80e801c000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 msgr2=0x7f80f409a920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f0ff9640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80f409a520 0x7f80f409a920 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f80e8019000 tx=0x7f80e801c000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:23.660 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.661 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.662 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:23.663 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:23.664 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:23.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.668 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:23.669 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.671 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.673 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:23.673 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:23.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80e0038740 msgr2=0x7f80e003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80e0038740 0x7f80e003abf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00eb00 tx=0x7f80ec00b040 comp rx=0 tx=0).stop 2024-09-15T01:59:23.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 10 monclient: shutdown 2024-09-15T01:59:23.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 msgr2=0x7f80f4135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 --2- 172.21.15.17:0/1206856973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f80f40980a0 0x7f80f4135100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009210 tx=0x7f80e4032000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f9485640 1 -- 172.21.15.17:0/1206856973 reap_dead start 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80f9485640 1 -- 172.21.15.17:0/1206856973 reap_dead start 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 shutdown_connections 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.643+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 >> 172.21.15.17:0/1206856973 conn(0x7f80f4070ed0 msgr2=0x7f80f40a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.644+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 shutdown_connections 2024-09-15T01:59:23.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.644+0000 7f80faee8640 1 -- 172.21.15.17:0/1206856973 wait complete. 2024-09-15T01:59:23.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:23.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:59:23.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.771+0000 7f15ed71d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.771+0000 7f15ed71d640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.771+0000 7f15ed71d640 10 monclient: monmap: 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:23.772181+0000 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:23.772181+0000 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:23.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 Processor -- start 2024-09-15T01:59:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 -- start start 2024-09-15T01:59:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient: init 2024-09-15T01:59:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15e8067090 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15e8067460 0x7f15e8067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-a con 0x7f15e8067460 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-b con 0x7f15e8136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e809cbc0 con 0x7f15e8067090 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e809e550 con 0x7f15e8067460 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81328b0 con 0x7f15e8136f90 2024-09-15T01:59:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15ed71d640 10 monclient(hunting): authenticate will time out at 1223.768629s 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15e8067460 0x7f15e8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15e8067460 0x7f15e8067830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51208/0 (socket says 172.21.15.17:51208) 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 1 -- 172.21.15.17:0/1608240539 learned_addr learned my addr 172.21.15.17:0/1608240539 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): get_auth_request con 0x7f15e8067460 auth_method 0 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36816/0 (socket says 172.21.15.17:36816) 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): get_auth_request con 0x7f15e8136f90 auth_method 0 2024-09-15T01:59:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.772+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 10 monclient(hunting): handle_auth_done global_id 104340 payload 274 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 1 -- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 msgr2=0x7f15e8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 1 -- 172.21.15.17:0/1608240539 >> v1:172.21.15.67:6789/0 conn(0x7f15e8067090 legacy=0x7f15e81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 1 monclient: found mon.noname-a 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15dffff640 1 -- 172.21.15.17:0/1608240539 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e809e1e0 con 0x7f15e8067460 2024-09-15T01:59:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.773+0000 7f15e77fe640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:23.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.775+0000 7f15dffff640 1 -- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067460 msgr2=0x7f15e8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.775+0000 7f15dffff640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067460 0x7f15e8067830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009910 tx=0x7f15d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce72202013710d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.775+0000 7f15e67fc640 1 -- 172.21.15.17:0/1608240539 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0032440 con 0x7f15e8067460 2024-09-15T01:59:23.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: dump: 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.788 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 5 monclient: authenticate success, global_id 104340 2024-09-15T01:59:23.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 1 -- 172.21.15.17:0/1608240539 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d00325a0 con 0x7f15e8067460 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 4 set_mon_vals no callback set 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 1 -- 172.21.15.17:0/1608240539 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0032940 con 0x7f15e8067460 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15e67fc640 10 monclient: dump: 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.790 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 10 monclient: shutdown 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 -- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067460 msgr2=0x7f15e8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067460 0x7f15e8067830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009910 tx=0x7f15d00091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 -- 172.21.15.17:0/1608240539 shutdown_connections 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 --2- 172.21.15.17:0/1608240539 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067460 0x7f15e8067830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 -- 172.21.15.17:0/1608240539 >> 172.21.15.17:0/1608240539 conn(0x7f15e80a1ee0 msgr2=0x7f15e80a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 -- 172.21.15.17:0/1608240539 shutdown_connections 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.776+0000 7f15ed71d640 1 -- 172.21.15.17:0/1608240539 wait complete. 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient: monmap: 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.792 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 Processor -- start 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 -- start start 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient: init 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 0x7f15e8141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-a con 0x7f15e8067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 0x7f15e8141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-b con 0x7f15e8067460 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e81187e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): picked mon.noname-c con 0x7f15e8136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e809d530 con 0x7f15e8067090 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e809ff80 con 0x7f15e8067460 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e811ae50 con 0x7f15e8136f90 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15ed71d640 10 monclient(hunting): authenticate will time out at 1223.773911s 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.777+0000 7f15e6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 0x7f15e8141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e81187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 0x7f15e8141630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51216/0 (socket says 172.21.15.17:51216) 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): get_auth_request con 0x7f15e8136f90 auth_method 0 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 1 -- 172.21.15.17:0/2896894563 learned_addr learned my addr 172.21.15.17:0/2896894563 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): get_auth_request con 0x7f15e8067090 auth_method 0 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 0x7f15e8141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 10 monclient(hunting): get_auth_request con 0x7f15e8067460 auth_method 0 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient(hunting): handle_auth_done global_id 94502 payload 824 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 msgr2=0x7f15e8141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 msgr2=0x7f15e8141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 msgr2=0x7f15e8141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 0x7f15e8141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15dffff640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 0x7f15e8141b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:23.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8067460 0x7f15e8141b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 msgr2=0x7f15e8141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 0x7f15e8141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 monclient: found mon.noname-c 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e77fe640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e0009a20 con 0x7f15e8136f90 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.778+0000 7f15e6ffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e8067090 0x7f15e8141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15e77fe640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 msgr2=0x7f15e81187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15e77fe640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e81187e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009f40 tx=0x7f15e000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0014480 con 0x7f15e8136f90 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: dump: 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.801 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 5 monclient: authenticate success, global_id 94502 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 10 monclient: _renew_subs 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d0009450 con 0x7f15e8136f90 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e000f040 con 0x7f15e8136f90 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0014830 con 0x7f15e8136f90 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15deffd640 10 monclient: dump: 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15e5ffb640 4 set_mon_vals no callback set 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 10 monclient: _renew_subs 2024-09-15T01:59:23.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.780+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e811bdf0 con 0x7f15e8136f90 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15ed71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15ed71d640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 0x7f15e81190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8063870 con 0x7f15e8118d20 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f15e0003020 con 0x7f15e8136f90 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 0x7f15e81190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient: get_auth_request con 0x7f15e8118d20 auth_method 0 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.781+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15deffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15c8038740 0x7f15c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(206..206 src has 148..206) v4 ==== 12623+0+0 (secure 0 0 0) 0x7f15e004e530 con 0x7f15e8136f90 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_done global_id 94505 payload 824 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15e6ffd640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15e811a340 con 0x7f15e8118d20 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15dffff640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15c8038740 0x7f15c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.782+0000 7f15dffff640 10 monclient: get_auth_request con 0x7f15c8038740 auth_method 0 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15e6ffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 msgr2=0x7f15e81190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15e6ffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 0x7f15e81190f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f15d800fbb0 tx=0x7f15d8007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d800de70 con 0x7f15e8118d20 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 10 monclient: dump: 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f15d8014470 con 0x7f15e8118d20 2024-09-15T01:59:23.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 msgr2=0x7f15e81190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 0x7f15e81190f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f15d800fbb0 tx=0x7f15d8007ed0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 msgr2=0x7f15e81190f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15deffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8118d20 0x7f15e81190f0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f15d800fbb0 tx=0x7f15d8007ed0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.784+0000 7f15dffff640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15c8038740 0x7f15c803abf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f15e804b6a0 tx=0x7f15d0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15ed71d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:23.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15ed71d640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 0x7f15e8132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15ed71d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80724a0 con 0x7f15e8119630 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 0x7f15e8132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient: get_auth_request con 0x7f15e8119630 auth_method 0 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 10 monclient(hunting): handle_auth_done global_id 94508 payload 824 2024-09-15T01:59:23.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.790+0000 7f15e6ffd640 1 -- 172.21.15.17:0/2896894563 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f15e80689a0 con 0x7f15e8119630 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15e6ffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 msgr2=0x7f15e8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15e6ffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 0x7f15e8132ad0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15d8019000 tx=0x7f15d801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d801ee20 con 0x7f15e8119630 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 10 monclient: dump: 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7f15d8007020 con 0x7f15e8119630 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 msgr2=0x7f15e8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 0x7f15e8132ad0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15d8019000 tx=0x7f15d801a000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 msgr2=0x7f15e8132ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:23.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15deffd640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8119630 0x7f15e8132ad0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15d8019000 tx=0x7f15d801a000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:23.827 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15c8038740 msgr2=0x7f15c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15c8038740 0x7f15c803abf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f15e804b6a0 tx=0x7f15d0032000 comp rx=0 tx=0).stop 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 10 monclient: shutdown 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 msgr2=0x7f15e81187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e81187e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009f40 tx=0x7f15e000b040 comp rx=0 tx=0).stop 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15e77fe640 1 -- 172.21.15.17:0/2896894563 reap_dead start 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 shutdown_connections 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 --2- 172.21.15.17:0/2896894563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e8136f90 0x7f15e81187e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.793+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 >> 172.21.15.17:0/2896894563 conn(0x7f15e80a1ee0 msgr2=0x7f15e8097e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.794+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 shutdown_connections 2024-09-15T01:59:23.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.794+0000 7f15ed71d640 1 -- 172.21.15.17:0/2896894563 wait complete. 2024-09-15T01:59:23.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:23.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:59:23.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.925+0000 7fa6e044f640 10 monclient: get_monmap_and_config 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.925+0000 7fa6e044f640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.925+0000 7fa6e044f640 10 monclient: monmap: 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:23.926171+0000 2024-09-15T01:59:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:23.926171+0000 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:23.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.925+0000 7fa6e044f640 1 Processor -- start 2024-09-15T01:59:23.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 -- start start 2024-09-15T01:59:23.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient: init 2024-09-15T01:59:23.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6d8068f20 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 0x7fa6d80696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d80692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa6d8136fe0 0x7fa6d81393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d8136fe0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80a1950 con 0x7fa6d8068f20 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d80a1670 con 0x7fa6d8136fe0 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d81328d0 con 0x7fa6d80692f0 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6e044f640 10 monclient(hunting): authenticate will time out at 1223.922822s 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de1eb640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa6d8068f20 0x7fa6d81321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42632/0 (socket says 172.21.15.17:42632) 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de1eb640 1 -- 172.21.15.17:0/2992669025 learned_addr learned my addr 172.21.15.17:0/2992669025 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 0x7fa6d80696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de9ec640 1 --2- 172.21.15.17:0/2992669025 >> v2:172.21.15.17:3300/0 conn(0x7fa6d8136fe0 0x7fa6d81393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request con 0x7fa6d80692f0 auth_method 0 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de9ec640 10 monclient(hunting): get_auth_request con 0x7fa6d8136fe0 auth_method 0 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de9ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6de9ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.926+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_done global_id 94511 payload 274 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/2992669025 >> v2:172.21.15.17:3300/0 conn(0x7fa6d8136fe0 msgr2=0x7fa6d81393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/2992669025 >> v2:172.21.15.17:3300/0 conn(0x7fa6d8136fe0 0x7fa6d81393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/2992669025 >> v1:172.21.15.17:6789/0 conn(0x7fa6d8068f20 legacy=0x7fa6d81321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 1 monclient: found mon.noname-b 2024-09-15T01:59:23.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.927+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/2992669025 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80a0020 con 0x7fa6d80692f0 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.929+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 msgr2=0x7fa6d80696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.929+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 0x7fa6d80696c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009850 tx=0x7fa6c80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75afe8ce88fcb827 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 1 -- 172.21.15.17:0/2992669025 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c802ae80 con 0x7fa6d80692f0 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: dump: 2024-09-15T01:59:23.939 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 1 -- 172.21.15.17:0/2992669025 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8032440 con 0x7fa6d80692f0 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 5 monclient: authenticate success, global_id 94511 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 1 -- 172.21.15.17:0/2992669025 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80327c0 con 0x7fa6d80692f0 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 4 set_mon_vals no callback set 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6dd1e9640 10 monclient: dump: 2024-09-15T01:59:23.941 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 10 monclient: shutdown 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 -- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 msgr2=0x7fa6d80696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 --2- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 0x7fa6d80696c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009850 tx=0x7fa6c80090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 -- 172.21.15.17:0/2992669025 shutdown_connections 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 --2- 172.21.15.17:0/2992669025 >> v2:172.21.15.17:3300/0 conn(0x7fa6d8136fe0 0x7fa6d81393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 --2- 172.21.15.17:0/2992669025 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d80692f0 0x7fa6d80696c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 -- 172.21.15.17:0/2992669025 >> 172.21.15.17:0/2992669025 conn(0x7fa6d8070fb0 msgr2=0x7fa6d80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 -- 172.21.15.17:0/2992669025 shutdown_connections 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.930+0000 7fa6e044f640 1 -- 172.21.15.17:0/2992669025 wait complete. 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.931+0000 7fa6e044f640 10 monclient: build_initial_monmap 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.931+0000 7fa6e044f640 10 monclient: monmap: 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:23.943 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.931+0000 7fa6e044f640 1 Processor -- start 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.931+0000 7fa6e044f640 1 -- start start 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.931+0000 7fa6e044f640 10 monclient: init 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:23.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 0x7fa6d8141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d8068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80692f0 0x7fa6d8141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d80692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa6d8136fe0 0x7fa6d8099bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d8136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d809c9b0 con 0x7fa6d8136fe0 2024-09-15T01:59:23.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d811cd20 con 0x7fa6d80692f0 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80a0750 con 0x7fa6d8068f20 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6e044f640 10 monclient(hunting): authenticate will time out at 1223.928192s 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 0x7fa6d8141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa6d8136fe0 0x7fa6d8099bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 0x7fa6d8141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36884/0 (socket says 172.21.15.17:36884) 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 -- 172.21.15.17:0/1270032532 learned_addr learned my addr 172.21.15.17:0/1270032532 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): get_auth_request con 0x7fa6d8068f20 auth_method 0 2024-09-15T01:59:23.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): get_auth_request con 0x7fa6d8136fe0 auth_method 0 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:23.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80692f0 0x7fa6d8141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request con 0x7fa6d80692f0 auth_method 0 2024-09-15T01:59:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient(hunting): handle_auth_done global_id 94514 payload 824 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80692f0 msgr2=0x7fa6d8141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80692f0 0x7fa6d8141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa6d8136fe0 msgr2=0x7fa6d8099bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:23.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa6d8136fe0 0x7fa6d8099bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient: _finish_hunting 0 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 monclient: found mon.noname-c 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.932+0000 7fa6de1eb640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8009390 con 0x7fa6d8068f20 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6de1eb640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 msgr2=0x7fa6d8141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6de1eb640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 0x7fa6d8141680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c5e0 tx=0x7fa6d4004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4005cb0 con 0x7fa6d8068f20 2024-09-15T01:59:23.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: dump: 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.951 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4005e30 con 0x7fa6d8068f20 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6e044f640 5 monclient: authenticate success, global_id 94514 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6e044f640 10 monclient: _renew_subs 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6e044f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d811d250 con 0x7fa6d8068f20 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4007340 con 0x7fa6d8068f20 2024-09-15T01:59:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6ce7fc640 10 monclient: dump: 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:23.953 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 10 monclient: _renew_subs 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.934+0000 7fa6dc9e8640 4 set_mon_vals no callback set 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d811c9b0 con 0x7fa6d8068f20 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8063b90 con 0x7fa6d809a8b0 2024-09-15T01:59:23.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.935+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa6d401b400 con 0x7fa6d8068f20 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6ce7fc640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6bc038770 0x7fa6bc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(206..206 src has 148..206) v4 ==== 12623+0+0 (secure 0 0 0) 0x7fa6d404d470 con 0x7fa6d8068f20 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d809a8b0 auth_method 0 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:23.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6de9ec640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6bc038770 0x7fa6bc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:23.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:23.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.936+0000 7fa6de9ec640 10 monclient: get_auth_request con 0x7fa6bc038770 auth_method 0 2024-09-15T01:59:23.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:23.938+0000 7fa6de9ec640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6bc038770 0x7fa6bc03ac20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009fd0 tx=0x7fa6d000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:24.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d809a8b0 auth_method 0 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:24.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.137+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:59:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.517+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v207) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8650018020 con 0x7f8658131aa0 2024-09-15T01:59:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.517+0000 7f864e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.517+0000 7f864e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-15T01:59:24.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d809a8b0 auth_method 0 2024-09-15T01:59:24.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:24.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:24.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:24.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:24.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:24.538+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:59:24.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.622+0000 7f865c99d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.622+0000 7f865c99d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.622+0000 7f865c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:24.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.622+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f865806f0b0 con 0x7f8658131aa0 2024-09-15T01:59:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.625+0000 7f864e7fc640 1 -- 172.21.15.67:0/3422718435 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v207) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8650012340 con 0x7f8658131aa0 2024-09-15T01:59:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.625+0000 7f864e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.625+0000 7f864e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-15T01:59:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h22' already exists 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 msgr2=0x7f863803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f86580a5180 tx=0x7f8648032000 comp rx=0 tx=0).stop 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 10 monclient: shutdown 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 msgr2=0x7f8658134680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ee0 tx=0x7f865000b040 comp rx=0 tx=0).stop 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 shutdown_connections 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8638038740 0x7f863803abf0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8658131aa0 0x7f8658134680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86580a0a60 0x7f8658141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 --2- 172.21.15.67:0/3422718435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f865809f490 0x7f8658141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 >> 172.21.15.67:0/3422718435 conn(0x7f8658071390 msgr2=0x7f865809db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 shutdown_connections 2024-09-15T01:59:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.627+0000 7f865c99d640 1 -- 172.21.15.67:0/3422718435 wait complete. 2024-09-15T01:59:24.639 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:24.639 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-15T01:59:24.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:24.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 10 monclient: build_initial_monmap 2024-09-15T01:59:24.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 10 monclient: monmap: 2024-09-15T01:59:24.717 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:24.717 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:24.716264+0000 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:24.716264+0000 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:24.718 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 1 Processor -- start 2024-09-15T01:59:24.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 1 -- start start 2024-09-15T01:59:24.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.715+0000 7f6ef375d640 10 monclient: init 2024-09-15T01:59:24.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:24.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6eec131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6eec098420 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6eec1381b0 0x7f6eec13a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec1381b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0a13e0 con 0x7f6eec098420 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0a0440 con 0x7f6eec131e70 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec0a01e0 con 0x7f6eec1381b0 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef375d640 10 monclient(hunting): authenticate will time out at 1197.185395s 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef14f9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6eec131e70 0x7f6eec132240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60638/0 (socket says 172.21.15.67:60638) 2024-09-15T01:59:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef14f9640 1 -- 172.21.15.67:0/2927905679 learned_addr learned my addr 172.21.15.67:0/2927905679 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/2927905679 >> v2:172.21.15.17:3300/0 conn(0x7f6eec1381b0 0x7f6eec13a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): get_auth_request con 0x7f6eec1381b0 auth_method 0 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.716+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_done global_id 104352 payload 274 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/2927905679 >> v1:172.21.15.67:6789/0 conn(0x7f6eec131e70 legacy=0x7f6eec132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/2927905679 >> v1:172.21.15.17:6789/0 conn(0x7f6eec098420 legacy=0x7f6eec098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:24.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 10 monclient: _finish_hunting 0 2024-09-15T01:59:24.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 1 monclient: found mon.noname-a 2024-09-15T01:59:24.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:24.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.717+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/2927905679 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec0a19b0 con 0x7f6eec1381b0 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/2927905679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 msgr2=0x7f6eec13a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/2927905679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec13a590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e60 tx=0x7f6ee4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f8628632385e2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 1 -- 172.21.15.67:0/2927905679 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4007630 con 0x7f6eec1381b0 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: dump: 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:24.727 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 1 -- 172.21.15.67:0/2927905679 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee402d050 con 0x7f6eec1381b0 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 1 -- 172.21.15.67:0/2927905679 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40079c0 con 0x7f6eec1381b0 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ee3fff640 10 monclient: dump: 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:24.729 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 5 monclient: authenticate success, global_id 104352 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 4 set_mon_vals no callback set 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 10 monclient: shutdown 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 -- 172.21.15.67:0/2927905679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 msgr2=0x7f6eec13a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 --2- 172.21.15.67:0/2927905679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec13a590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e60 tx=0x7f6ee4009780 comp rx=0 tx=0).stop 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 -- 172.21.15.67:0/2927905679 shutdown_connections 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 --2- 172.21.15.67:0/2927905679 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec13a590 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 -- 172.21.15.67:0/2927905679 >> 172.21.15.67:0/2927905679 conn(0x7f6eec0a5c10 msgr2=0x7f6eec0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 -- 172.21.15.67:0/2927905679 shutdown_connections 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 1 -- 172.21.15.67:0/2927905679 wait complete. 2024-09-15T01:59:24.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 10 monclient: build_initial_monmap 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.721+0000 7f6ef375d640 10 monclient: monmap: 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 Processor -- start 2024-09-15T01:59:24.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 -- start start 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient: init 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 0x7f6eec137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-c con 0x7f6eec098420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-b con 0x7f6eec131e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec1381b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec133c10 con 0x7f6eec1381b0 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:24.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0a0d80 con 0x7f6eec131e70 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0a4eb0 con 0x7f6eec098420 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef375d640 10 monclient(hunting): authenticate will time out at 1197.191747s 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59142/0 (socket says 172.21.15.67:59142) 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 -- 172.21.15.67:0/3443238132 learned_addr learned my addr 172.21.15.67:0/3443238132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 10 monclient(hunting): get_auth_request con 0x7f6eec131e70 auth_method 0 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 msgr2=0x7f6eec099430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:24.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 msgr2=0x7f6eec099430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.722+0000 7f6ef0cf8640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 0x7f6eec137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): get_auth_request con 0x7f6eec098420 auth_method 0 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:24.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): get_auth_request con 0x7f6eec1381b0 auth_method 0 2024-09-15T01:59:24.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:24.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:24.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient(hunting): handle_auth_done global_id 104355 payload 824 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 msgr2=0x7f6eec137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 0x7f6eec137900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 msgr2=0x7f6eec099430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:24.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient: _finish_hunting 0 2024-09-15T01:59:24.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 monclient: found mon.noname-a 2024-09-15T01:59:24.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:24.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4009a20 con 0x7f6eec1381b0 2024-09-15T01:59:24.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.723+0000 7f6ef14f9640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 0x7f6eec137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.725+0000 7f6ef1cfa640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 msgr2=0x7f6eec099970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.725+0000 7f6ef1cfa640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec099970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4034000 tx=0x7f6ee4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4009460 con 0x7f6eec1381b0 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: dump: 2024-09-15T01:59:24.850 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:24.851 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 5 monclient: authenticate success, global_id 104355 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 10 monclient: _renew_subs 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec133930 con 0x7f6eec1381b0 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee4040070 con 0x7f6eec1381b0 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:24.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4031cf0 con 0x7f6eec1381b0 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee17fa640 10 monclient: dump: 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.853 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ee37fe640 4 set_mon_vals no callback set 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 10 monclient: _renew_subs 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.726+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec136ce0 con 0x7f6eec1381b0 2024-09-15T01:59:24.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6ee4042030 con 0x7f6eec1381b0 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef375d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef375d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec0639e0 con 0x7f6eec1381b0 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ee17fa640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 0x7f6ed003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(207..207 src has 148..207) v4 ==== 12836+0+0 (secure 0 0 0) 0x7f6ee406f320 con 0x7f6eec1381b0 2024-09-15T01:59:24.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef14f9640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 0x7f6ed003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:24.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.727+0000 7f6ef14f9640 10 monclient: get_auth_request con 0x7f6ed0038740 auth_method 0 2024-09-15T01:59:24.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.729+0000 7f6ef14f9640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 0x7f6ed003abf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8007fd0 tx=0x7f6ee8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:24.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.732+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6ee4045640 con 0x7f6eec1381b0 2024-09-15T01:59:24.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.732+0000 7f6ee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:24.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.732+0000 7f6ee17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:24.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.898+0000 7f6ef375d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.898+0000 7f6ef375d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:24.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.898+0000 7f6ef375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:24.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:24.898+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6eec0639e0 con 0x7f6eec1381b0 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.339+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.339+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d809a8b0 auth_method 0 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.339+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.339+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.339+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.340+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T01:59:25.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.340+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:25.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.340+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:25.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:25.340+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:59:25.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.523+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v208) v1 ==== 168+0+0 (secure 0 0 0) 0x7f6ee40500c0 con 0x7f6eec1381b0 2024-09-15T01:59:25.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.523+0000 7f6ee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.523+0000 7f6ee17fa640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-15T01:59:25.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ef375d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ef375d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ef375d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:25.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6eec0639e0 con 0x7f6eec1381b0 2024-09-15T01:59:25.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ee17fa640 1 -- 172.21.15.67:0/3443238132 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v208) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6ee4044840 con 0x7f6eec1381b0 2024-09-15T01:59:25.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ee17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.627+0000 7f6ee17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-15T01:59:25.629 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h23' already exists 2024-09-15T01:59:25.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 msgr2=0x7f6ed003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:25.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 0x7f6ed003abf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8007fd0 tx=0x7f6ee8007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:25.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 10 monclient: shutdown 2024-09-15T01:59:25.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 msgr2=0x7f6eec099970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.629+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec099970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4034000 tx=0x7f6ee4037000 comp rx=0 tx=0).stop 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 shutdown_connections 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6ed0038740 0x7f6ed003abf0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6eec1381b0 0x7f6eec099970 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eec131e70 0x7f6eec099430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 --2- 172.21.15.67:0/3443238132 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6eec098420 0x7f6eec137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 >> 172.21.15.67:0/3443238132 conn(0x7f6eec0a5c10 msgr2=0x7f6eec1314d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 shutdown_connections 2024-09-15T01:59:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.630+0000 7f6ef375d640 1 -- 172.21.15.67:0/3443238132 wait complete. 2024-09-15T01:59:25.640 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:25.640 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.717+0000 7f34eed9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.717+0000 7f34eed9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.717+0000 7f34eed9d640 10 monclient: monmap: 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:25.717919+0000 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:25.717919+0000 2024-09-15T01:59:25.719 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:25.720 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.718+0000 7f34eed9d640 1 Processor -- start 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.718+0000 7f34eed9d640 1 -- start start 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.718+0000 7f34eed9d640 10 monclient: init 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.718+0000 7f34eed9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.718+0000 7f34eed9d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 0x7f34e8099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f34e80a3800 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 0x7f34e809da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f34e809c780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f34e809cb80 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e806ee50 con 0x7f34e80a3800 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e809be00 con 0x7f34e809cb80 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e809f520 con 0x7f34e809c780 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34eed9d640 10 monclient(hunting): authenticate will time out at 1198.187082s 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59148/0 (socket says 172.21.15.67:59148) 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 -- 172.21.15.67:0/3144640928 learned_addr learned my addr 172.21.15.67:0/3144640928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 10 monclient(hunting): get_auth_request con 0x7f34e809cb80 auth_method 0 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 0x7f34e8099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 --2- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 0x7f34e809da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): get_auth_request con 0x7f34e80a3800 auth_method 0 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 -- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 msgr2=0x7f34e809df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:59:25.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 -- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 msgr2=0x7f34e809df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): get_auth_request con 0x7f34e809c780 auth_method 0 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ed33a640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient(hunting): handle_auth_done global_id 94520 payload 274 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 -- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 msgr2=0x7f34e809df70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 -- 172.21.15.67:0/3144640928 >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 msgr2=0x7f34e8099aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 0x7f34e8099aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 monclient: found mon.noname-b 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:25.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34dffff640 1 -- 172.21.15.67:0/3144640928 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80a3bd0 con 0x7f34e809c780 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.719+0000 7f34ecb39640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 0x7f34e8099aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34dffff640 1 -- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 msgr2=0x7f34e809da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34dffff640 1 --2- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 0x7f34e809da30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34e809f4d0 tx=0x7f34d80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bc07aea448b4e78 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 1 -- 172.21.15.67:0/3144640928 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d802adf0 con 0x7f34e809c780 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: dump: 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.733 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34eed9d640 5 monclient: authenticate success, global_id 94520 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34eed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 1 -- 172.21.15.67:0/3144640928 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8032440 con 0x7f34e809c780 2024-09-15T01:59:25.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 1 -- 172.21.15.67:0/3144640928 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80327e0 con 0x7f34e809c780 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34eed9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34eed9d640 4 set_mon_vals no callback set 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.722+0000 7f34df7fe640 10 monclient: dump: 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:25.735 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 10 monclient: shutdown 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 -- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 msgr2=0x7f34e809da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 --2- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 0x7f34e809da30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34e809f4d0 tx=0x7f34d80090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 -- 172.21.15.67:0/3144640928 shutdown_connections 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.67:3300/0 conn(0x7f34e809cb80 0x7f34e809df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 --2- 172.21.15.67:0/3144640928 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809c780 0x7f34e809da30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 --2- 172.21.15.67:0/3144640928 >> v2:172.21.15.17:3300/0 conn(0x7f34e80a3800 0x7f34e8099aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 -- 172.21.15.67:0/3144640928 >> 172.21.15.67:0/3144640928 conn(0x7f34e8071090 msgr2=0x7f34e80a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 -- 172.21.15.67:0/3144640928 shutdown_connections 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 1 -- 172.21.15.67:0/3144640928 wait complete. 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.723+0000 7f34eed9d640 10 monclient: monmap: 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:25.737 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 Processor -- start 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 -- start start 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient: init 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f34e809c780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:25.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 0x7f34e8131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f34e809cb80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 0x7f34e811b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f34e80a3800 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80a2bc0 con 0x7f34e80a3800 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e813fbb0 con 0x7f34e809c780 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e811eb90 con 0x7f34e809cb80 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34eed9d640 10 monclient(hunting): authenticate will time out at 1198.192868s 2024-09-15T01:59:25.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59150/0 (socket says 172.21.15.67:59150) 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 1 -- 172.21.15.67:0/2695196036 learned_addr learned my addr 172.21.15.67:0/2695196036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 10 monclient(hunting): get_auth_request con 0x7f34e809c780 auth_method 0 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.724+0000 7f34ecb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ecb39640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 msgr2=0x7f34e813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ecb39640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 msgr2=0x7f34e813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ecb39640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 0x7f34e8131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 0x7f34e811b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ecb39640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:25.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): get_auth_request con 0x7f34e809cb80 auth_method 0 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): get_auth_request con 0x7f34e80a3800 auth_method 0 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:25.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient(hunting): handle_auth_done global_id 94523 payload 824 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 msgr2=0x7f34e813ed40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 msgr2=0x7f34e811b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 0x7f34e811b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 monclient: found mon.noname-c 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34dffff640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4009390 con 0x7f34e809cb80 2024-09-15T01:59:25.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.725+0000 7f34ed33a640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 0x7f34e811b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.727+0000 7f34dffff640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 msgr2=0x7f34e8131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.727+0000 7f34dffff640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 0x7f34e8131a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f34e809f660 tx=0x7f34d8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8031b10 con 0x7f34e809cb80 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: dump: 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:25.746 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 5 monclient: authenticate success, global_id 94523 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 10 monclient: _renew_subs 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8009390 con 0x7f34e809cb80 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8031c70 con 0x7f34e809cb80 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:25.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d803d440 con 0x7f34e809cb80 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34dcff9640 10 monclient: dump: 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:25.748 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 10 monclient: _renew_subs 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34deffd640 4 set_mon_vals no callback set 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.728+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e811d060 con 0x7f34e809cb80 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34eed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34eed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e80639e0 con 0x7f34e809cb80 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f34d8039040 con 0x7f34e809cb80 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34dcff9640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 0x7f34d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(208..208 src has 148..208) v4 ==== 13049+0+0 (secure 0 0 0) 0x7f34d806d380 con 0x7f34e809cb80 2024-09-15T01:59:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34ecb39640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 0x7f34d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:25.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.729+0000 7f34ecb39640 10 monclient: get_auth_request con 0x7f34d4038740 auth_method 0 2024-09-15T01:59:25.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.731+0000 7f34ecb39640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 0x7f34d403abf0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009640 tx=0x7f34e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:25.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.734+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f34d802f4b0 con 0x7f34e809cb80 2024-09-15T01:59:25.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.734+0000 7f34dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:25.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.734+0000 7f34dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:25.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.898+0000 7f34eed9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.898+0000 7f34eed9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:25.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.898+0000 7f34eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:25.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:25.898+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f34e809b4f0 con 0x7f34e809cb80 2024-09-15T01:59:26.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.532+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v209) v1 ==== 168+0+0 (secure 0 0 0) 0x7f34d8037020 con 0x7f34e809cb80 2024-09-15T01:59:26.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.532+0000 7f34dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.532+0000 7f34dcff9640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-15T01:59:26.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.635+0000 7f34eed9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.635+0000 7f34eed9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.635+0000 7f34eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:26.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.635+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f34e811ecd0 con 0x7f34e809cb80 2024-09-15T01:59:26.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.637+0000 7f34dcff9640 1 -- 172.21.15.67:0/2695196036 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v209) v1 ==== 147+0+0 (secure 0 0 0) 0x7f34d8042070 con 0x7f34e809cb80 2024-09-15T01:59:26.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.637+0000 7f34dcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.637+0000 7f34dcff9640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-15T01:59:26.638 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h24' already exists 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 msgr2=0x7f34d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 0x7f34d403abf0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009640 tx=0x7f34e4008040 comp rx=0 tx=0).stop 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 10 monclient: shutdown 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 msgr2=0x7f34e8131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 0x7f34e8131a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f34e809f660 tx=0x7f34d8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:26.641 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 shutdown_connections 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f34d4038740 0x7f34d403abf0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34e80a3800 0x7f34e811b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f34e809cb80 0x7f34e8131a70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 --2- 172.21.15.67:0/2695196036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e809c780 0x7f34e813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 >> 172.21.15.67:0/2695196036 conn(0x7f34e8071090 msgr2=0x7f34e80a1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 shutdown_connections 2024-09-15T01:59:26.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.639+0000 7f34eed9d640 1 -- 172.21.15.67:0/2695196036 wait complete. 2024-09-15T01:59:26.650 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:26.650 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-15T01:59:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.727+0000 7f2ca9d1d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:26.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.727+0000 7f2ca9d1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.727+0000 7f2ca9d1d640 10 monclient: monmap: 2024-09-15T01:59:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:26.727824+0000 2024-09-15T01:59:26.730 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:26.727824+0000 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:26.731 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 Processor -- start 2024-09-15T01:59:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 -- start start 2024-09-15T01:59:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient: init 2024-09-15T01:59:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ca4131f90 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ca4098260 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ca4138130 0x7f2ca413a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca4138130 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca4096a70 con 0x7f2ca4098260 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca4132b20 con 0x7f2ca4131f90 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca406a6b0 con 0x7f2ca4138130 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.728+0000 7f2ca9d1d640 10 monclient(hunting): authenticate will time out at 1199.196957s 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ca4138130 0x7f2ca413a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2ca4131f90 0x7f2ca4132410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60652/0 (socket says 172.21.15.67:60652) 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): get_auth_request con 0x7f2ca4138130 auth_method 0 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca37fe640 1 -- 172.21.15.67:0/1886844956 learned_addr learned my addr 172.21.15.67:0/1886844956 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:26.747 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_done global_id 114100 payload 274 2024-09-15T01:59:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 1 -- 172.21.15.67:0/1886844956 >> v1:172.21.15.67:6789/0 conn(0x7f2ca4131f90 legacy=0x7f2ca4132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 1 -- 172.21.15.67:0/1886844956 >> v1:172.21.15.17:6789/0 conn(0x7f2ca4098260 legacy=0x7f2ca4098660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 1 monclient: found mon.noname-c 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.729+0000 7f2ca3fff640 1 -- 172.21.15.67:0/1886844956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40a16e0 con 0x7f2ca4138130 2024-09-15T01:59:26.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca3fff640 1 -- 172.21.15.67:0/1886844956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 msgr2=0x7f2ca413a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:26.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca3fff640 1 --2- 172.21.15.67:0/1886844956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca413a510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ca406aad0 tx=0x7f2c9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0faebee0f49150 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:26.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 1 -- 172.21.15.67:0/1886844956 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c007840 con 0x7f2ca4138130 2024-09-15T01:59:26.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: dump: 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.751 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 5 monclient: authenticate success, global_id 114100 2024-09-15T01:59:26.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 1 -- 172.21.15.67:0/1886844956 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c02d050 con 0x7f2ca4138130 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 4 set_mon_vals no callback set 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 1 -- 172.21.15.67:0/1886844956 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c007bd0 con 0x7f2ca4138130 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca27fc640 10 monclient: dump: 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.754 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 10 monclient: shutdown 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/1886844956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 msgr2=0x7f2ca413a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/1886844956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca413a510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ca406aad0 tx=0x7f2c9c009780 comp rx=0 tx=0).stop 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/1886844956 shutdown_connections 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/1886844956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca413a510 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/1886844956 >> 172.21.15.67:0/1886844956 conn(0x7f2ca4071690 msgr2=0x7f2ca40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/1886844956 shutdown_connections 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.732+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/1886844956 wait complete. 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.733+0000 7f2ca9d1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.733+0000 7f2ca9d1d640 10 monclient: monmap: 2024-09-15T01:59:26.756 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:26.757 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.733+0000 7f2ca9d1d640 1 Processor -- start 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.733+0000 7f2ca9d1d640 1 -- start start 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.733+0000 7f2ca9d1d640 10 monclient: init 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ca4098260 0x7f2ca40998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ca4098260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:26.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4131f90 0x7f2ca4099de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca4131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:26.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ca4138130 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40a0d80 con 0x7f2ca4098260 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40a54c0 con 0x7f2ca4138130 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca409b500 con 0x7f2ca4131f90 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca9d1d640 10 monclient(hunting): authenticate will time out at 1199.202366s 2024-09-15T01:59:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59180/0 (socket says 172.21.15.67:59180) 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 -- 172.21.15.67:0/857367062 learned_addr learned my addr 172.21.15.67:0/857367062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): get_auth_request con 0x7f2ca4138130 auth_method 0 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca37fe640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ca4098260 0x7f2ca40998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca2ffd640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4131f90 0x7f2ca4099de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca2ffd640 10 monclient(hunting): get_auth_request con 0x7f2ca4131f90 auth_method 0 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca37fe640 10 monclient(hunting): get_auth_request con 0x7f2ca4098260 auth_method 0 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient(hunting): handle_auth_done global_id 114103 payload 824 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 -- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4131f90 msgr2=0x7f2ca4099de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4131f90 0x7f2ca4099de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 -- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ca4098260 msgr2=0x7f2ca40998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ca4098260 0x7f2ca40998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:26.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 monclient: found mon.noname-b 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.734+0000 7f2ca3fff640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c009a20 con 0x7f2ca4138130 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca3fff640 1 -- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 msgr2=0x7f2ca409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca3fff640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009e90 tx=0x7f2c9c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c007ec0 con 0x7f2ca4138130 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: dump: 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.763 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 5 monclient: authenticate success, global_id 114103 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 10 monclient: _renew_subs 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c040070 con 0x7f2ca4138130 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca4133ad0 con 0x7f2ca4138130 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c045430 con 0x7f2ca4138130 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2c8b7fe640 10 monclient: dump: 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.765 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 10 monclient: _renew_subs 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca4133e60 con 0x7f2ca4138130 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.737+0000 7f2ca1ffb640 4 set_mon_vals no callback set 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2c9c042030 con 0x7f2ca4138130 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2c8b7fe640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 0x7f2c9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.766 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(209..209 src has 148..209) v4 ==== 13262+0+0 (secure 0 0 0) 0x7f2c9c06f7f0 con 0x7f2ca4138130 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca9d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca9d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca9d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca37fe640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 0x7f2c9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.738+0000 7f2ca37fe640 10 monclient: get_auth_request con 0x7f2c90038740 auth_method 0 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.741+0000 7f2ca37fe640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 0x7f2c9003abf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007fd0 tx=0x7f2c94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.743+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.743+0000 7f2c8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:26.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.743+0000 7f2c8b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:26.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.911+0000 7f2ca9d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.911+0000 7f2ca9d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:26.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.911+0000 7f2ca9d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:26.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:26.911+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:26.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.932+0000 7fa6cd7fa640 10 monclient: tick 2024-09-15T01:59:26.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.932+0000 7fa6cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:26.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.932+0000 7fa6cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:26.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d809a8b0 auth_method 0 2024-09-15T01:59:26.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_done global_id 114106 payload 824 2024-09-15T01:59:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.942+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6d80724a0 con 0x7fa6d809a8b0 2024-09-15T01:59:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8034000 tx=0x7fa6c8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8046410 con 0x7fa6d809a8b0 2024-09-15T01:59:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 10 monclient: dump: 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa6c8046570 con 0x7fa6d809a8b0 2024-09-15T01:59:26.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:26.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:26.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8034000 tx=0x7fa6c8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:26.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 msgr2=0x7fa6d809ac80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:26.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.945+0000 7fa6ce7fc640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d809a8b0 0x7fa6d809ac80 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8034000 tx=0x7fa6c8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6e044f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6e044f640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 0x7fa6d8029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6e044f640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d811ce60 con 0x7fa6d80298c0 2024-09-15T01:59:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 0x7fa6d8029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient: get_auth_request con 0x7fa6d80298c0 auth_method 0 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 10 monclient(hunting): handle_auth_done global_id 114109 payload 824 2024-09-15T01:59:26.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.952+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa6d8136180 con 0x7fa6d80298c0 2024-09-15T01:59:26.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6dd9ea640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 msgr2=0x7fa6d8029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:26.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6dd9ea640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 0x7fa6d8029cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009850 tx=0x7fa6c8040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:26.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c804d070 con 0x7fa6d80298c0 2024-09-15T01:59:26.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:26.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 10 monclient: dump: 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7fa6c803bdc0 con 0x7fa6d80298c0 2024-09-15T01:59:26.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 msgr2=0x7fa6d8029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 0x7fa6d8029cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009850 tx=0x7fa6c8040040 comp rx=0 tx=0).stop 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 msgr2=0x7fa6d8029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.955+0000 7fa6ce7fc640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d80298c0 0x7fa6d8029cc0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009850 tx=0x7fa6c8040040 comp rx=0 tx=0).stop 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:26.959 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:26.960 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:26.961 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.962 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:26.963 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:26.964 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:26.965 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:26.966 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:26.967 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:26.969 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.972 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:59:26.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:26.974 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:26.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6bc038770 msgr2=0x7fa6bc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa6bc038770 0x7fa6bc03ac20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009fd0 tx=0x7fa6d000b040 comp rx=0 tx=0).stop 2024-09-15T01:59:26.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 10 monclient: shutdown 2024-09-15T01:59:26.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:26.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 msgr2=0x7fa6d8141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 --2- 172.21.15.17:0/1270032532 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa6d8068f20 0x7fa6d8141680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c5e0 tx=0x7fa6d4004960 comp rx=0 tx=0).stop 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6de9ec640 1 -- 172.21.15.17:0/1270032532 reap_dead start 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6de9ec640 1 -- 172.21.15.17:0/1270032532 reap_dead start 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 shutdown_connections 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 >> 172.21.15.17:0/1270032532 conn(0x7fa6d8070fb0 msgr2=0x7fa6d809e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 shutdown_connections 2024-09-15T01:59:26.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:26.956+0000 7fa6e044f640 1 -- 172.21.15.17:0/1270032532 wait complete. 2024-09-15T01:59:26.978 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:26.978 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T01:59:26.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T01:59:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.082+0000 7f0a4335d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.082+0000 7f0a4335d640 10 monclient: build_initial_monmap 2024-09-15T01:59:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.082+0000 7f0a4335d640 10 monclient: monmap: 2024-09-15T01:59:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:27.083157+0000 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:27.083157+0000 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:27.085 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.082+0000 7f0a4335d640 1 Processor -- start 2024-09-15T01:59:27.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 -- start start 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient: init 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a3c0708a0 0x7f0a3c09ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c0708a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a3c06a510 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0a3c137130 0x7f0a3c139510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c137130 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c070c70 con 0x7f0a3c06a510 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3c09da40 con 0x7f0a3c137130 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3c070f30 con 0x7f0a3c0708a0 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a4335d640 10 monclient(hunting): authenticate will time out at 1227.079794s 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0a3c137130 0x7f0a3c139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0a3c137130 0x7f0a3c139510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51248/0 (socket says 172.21.15.17:51248) 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 1 -- 172.21.15.17:0/1945880068 learned_addr learned my addr 172.21.15.17:0/1945880068 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 10 monclient(hunting): get_auth_request con 0x7f0a3c137130 auth_method 0 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.083+0000 7f0a418fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 1 --2- 172.21.15.17:0/1945880068 >> v2:172.21.15.67:3300/0 conn(0x7f0a3c0708a0 0x7f0a3c09ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): get_auth_request con 0x7f0a3c0708a0 auth_method 0 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient(hunting): handle_auth_done global_id 104367 payload 274 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 1 -- 172.21.15.17:0/1945880068 >> v2:172.21.15.67:3300/0 conn(0x7f0a3c0708a0 msgr2=0x7f0a3c09ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 1 --2- 172.21.15.17:0/1945880068 >> v2:172.21.15.67:3300/0 conn(0x7f0a3c0708a0 0x7f0a3c09ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 1 -- 172.21.15.17:0/1945880068 >> v1:172.21.15.67:6789/0 conn(0x7f0a3c06a510 legacy=0x7f0a3c09b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient: _finish_hunting 0 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 1 monclient: found mon.noname-a 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.084+0000 7f0a418fa640 1 -- 172.21.15.17:0/1945880068 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c09c4b0 con 0x7f0a3c137130 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a418fa640 1 -- 172.21.15.17:0/1945880068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c137130 msgr2=0x7f0a3c139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a418fa640 1 --2- 172.21.15.17:0/1945880068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c137130 0x7f0a3c139510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c071bb0 tx=0x7f0a34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3e96302ef5c16a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 1 -- 172.21.15.17:0/1945880068 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34007610 con 0x7f0a3c137130 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: dump: 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.094 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 5 monclient: authenticate success, global_id 104367 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 1 -- 172.21.15.17:0/1945880068 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3402d050 con 0x7f0a3c137130 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:27.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 1 -- 172.21.15.17:0/1945880068 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a340079a0 con 0x7f0a3c137130 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 4 set_mon_vals no callback set 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a33fff640 10 monclient: dump: 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.096 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 10 monclient: shutdown 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 -- 172.21.15.17:0/1945880068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c137130 msgr2=0x7f0a3c139510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 --2- 172.21.15.17:0/1945880068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c137130 0x7f0a3c139510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c071bb0 tx=0x7f0a34009840 comp rx=0 tx=0).stop 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 -- 172.21.15.17:0/1945880068 shutdown_connections 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 --2- 172.21.15.17:0/1945880068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c137130 0x7f0a3c139510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 --2- 172.21.15.17:0/1945880068 >> v2:172.21.15.67:3300/0 conn(0x7f0a3c0708a0 0x7f0a3c09ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 -- 172.21.15.17:0/1945880068 >> 172.21.15.17:0/1945880068 conn(0x7f0a3c067ae0 msgr2=0x7f0a3c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:27.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.087+0000 7f0a4335d640 1 -- 172.21.15.17:0/1945880068 shutdown_connections 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 1 -- 172.21.15.17:0/1945880068 wait complete. 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 10 monclient: build_initial_monmap 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 10 monclient: monmap: 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:27.098 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 1 Processor -- start 2024-09-15T01:59:27.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 1 -- start start 2024-09-15T01:59:27.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.088+0000 7f0a4335d640 10 monclient: init 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c06a510 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a3c0708a0 0x7f0a3c11eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c0708a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:27.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3c137130 0x7f0a3c11eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3c137130 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c06deb0 con 0x7f0a3c06a510 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c06d590 con 0x7f0a3c137130 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c121ea0 con 0x7f0a3c0708a0 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a4335d640 10 monclient(hunting): authenticate will time out at 1227.085459s 2024-09-15T01:59:27.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51252/0 (socket says 172.21.15.17:51252) 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 -- 172.21.15.17:0/2517399368 learned_addr learned my addr 172.21.15.17:0/2517399368 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): get_auth_request con 0x7f0a3c06a510 auth_method 0 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3c137130 0x7f0a3c11eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a3c0708a0 0x7f0a3c11eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): get_auth_request con 0x7f0a3c137130 auth_method 0 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): get_auth_request con 0x7f0a3c0708a0 auth_method 0 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a418fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a408f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient(hunting): handle_auth_done global_id 104370 payload 824 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 -- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a3c0708a0 msgr2=0x7f0a3c11eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a3c0708a0 0x7f0a3c11eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 -- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3c137130 msgr2=0x7f0a3c11eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3c137130 0x7f0a3c11eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient: _finish_hunting 0 2024-09-15T01:59:27.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 monclient: found mon.noname-a 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.089+0000 7f0a410f9640 1 -- 172.21.15.17:0/2517399368 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a34009ae0 con 0x7f0a3c06a510 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a410f9640 1 -- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 msgr2=0x7f0a3c11e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a410f9640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800ea20 tx=0x7f0a380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38009600 con 0x7f0a3c06a510 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:27.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: dump: 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3800cc20 con 0x7f0a3c06a510 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 5 monclient: authenticate success, global_id 104370 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 10 monclient: _renew_subs 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3c136ca0 con 0x7f0a3c06a510 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:27.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38013470 con 0x7f0a3c06a510 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a317fa640 10 monclient: dump: 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.111 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 10 monclient: _renew_subs 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a337fe640 4 set_mon_vals no callback set 2024-09-15T01:59:27.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.092+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3c120cd0 con 0x7f0a3c06a510 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a38013860 con 0x7f0a3c06a510 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a4335d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a4335d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a4335d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3c0724a0 con 0x7f0a3c06a510 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a317fa640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 0x7f0a2003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(209..209 src has 148..209) v4 ==== 13262+0+0 (secure 0 0 0) 0x7f0a3804df40 con 0x7f0a3c06a510 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a408f8640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 0x7f0a2003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.093+0000 7f0a408f8640 10 monclient: get_auth_request con 0x7f0a20038770 auth_method 0 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.096+0000 7f0a408f8640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 0x7f0a2003ac20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c11fd00 tx=0x7f0a2c007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.098+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0a380236b0 con 0x7f0a3c06a510 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.098+0000 7f0a317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.098+0000 7f0a317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:27.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.324+0000 7f0a4335d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T01:59:27.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.324+0000 7f0a4335d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T01:59:27.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.324+0000 7f0a4335d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.324+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0a3c11b430 con 0x7f0a3c06a510 2024-09-15T01:59:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f0a317fa640 1 -- 172.21.15.17:0/2517399368 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0a380232f0 con 0x7f0a3c06a510 2024-09-15T01:59:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f0a317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T01:59:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f0a317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=8,logm=60} crc {auth=339885606,config=3140514009,health=810507239,kv=1529054990,logm=13758568}) 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=8,logm=61} crc {auth=3950517127,config=3140514009,health=3410050882,kv=1529054990,logm=321989831}) 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=8,logm=60} crc {auth=339885606,config=3140514009,health=810507239,kv=1529054990,logm=13758568}) 2024-09-15T01:59:27.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=3050959995,config=3140514010,health=1301148488,kv=769410566,logm=3652785373}) 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940913816}) 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778617634}) 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940913816}) 2024-09-15T01:59:27.329 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3925850941}) 2024-09-15T01:59:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 msgr2=0x7f0a2003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 0x7f0a2003ac20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c11fd00 tx=0x7f0a2c007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 10 monclient: shutdown 2024-09-15T01:59:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:27.330 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 msgr2=0x7f0a3c11e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800ea20 tx=0x7f0a380090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640670469}) 2024-09-15T01:59:27.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161906573}) 2024-09-15T01:59:27.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640670469}) 2024-09-15T01:59:27.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714189459}) 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 shutdown_connections 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a20038770 0x7f0a2003ac20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3c137130 0x7f0a3c11eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a3c0708a0 0x7f0a3c11eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 --2- 172.21.15.17:0/2517399368 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a3c06a510 0x7f0a3c11e570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 >> 172.21.15.17:0/2517399368 conn(0x7f0a3c067ae0 msgr2=0x7f0a3c09ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 shutdown_connections 2024-09-15T01:59:27.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f0a4335d640 1 -- 172.21.15.17:0/2517399368 wait complete. 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986260871}) 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981489100}) 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986260871}) 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844234671}) 2024-09-15T01:59:27.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353411601}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995787233}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353411601}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.329+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474668761}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=786173725}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259931630}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=786173725}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.330+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017303416}) 2024-09-15T01:59:27.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573084142}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087471338}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573084142}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.331+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507280388}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825982672}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614429315}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825982672}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.332+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183677650}) 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225534596}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1209651429}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225534596}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879167102}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.333+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779860204}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991813851}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779860204}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508768010}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869196393}) 2024-09-15T01:59:27.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2393588399}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869196393}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.334+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475691369}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153725803}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617566002}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153725803}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402242808}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95696857}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068615230}) 2024-09-15T01:59:27.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95696857}) 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1092987274}) 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991556488}) 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453107915}) 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991556488}) 2024-09-15T01:59:27.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.337+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469156952}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168222296}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682760403}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168222296}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266467701}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783077221}) 2024-09-15T01:59:27.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344412805}) 2024-09-15T01:59:27.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783077221}) 2024-09-15T01:59:27.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735059185}) 2024-09-15T01:59:27.348 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T01:59:27.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745866314}) 2024-09-15T01:59:27.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824518654}) 2024-09-15T01:59:27.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745866314}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.339+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557943094}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738291622}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158045166}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738291622}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.340+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441992515}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526854929}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372155451}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526854929}) 2024-09-15T01:59:27.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=682419788}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.341+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=548124472}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223379653}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=548124472}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247749493}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535914624}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101812529}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535914624}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.342+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318336571}) 2024-09-15T01:59:27.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239054105}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162009480}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239054105}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097331617}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335494680}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075475081}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335494680}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.343+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402895114}) 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127677684}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422607130}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127677684}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792846763}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309243898}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=269702863}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309243898}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.345+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899456726}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056913418}) 2024-09-15T01:59:27.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480955921}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056913418}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708756366}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83295956}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691214218}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83295956}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.346+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442875425}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976088256}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173126717}) 2024-09-15T01:59:27.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976088256}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.347+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861562510}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163988055}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055180781}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163988055}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751382807}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996727431}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931474403}) 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996727431}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.348+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255942298}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718156870}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587442708}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718156870}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.349+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815659853}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725284335}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060589983}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725284335}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.350+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1498719344}) 2024-09-15T01:59:27.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906275422}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234491711}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906275422}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.351+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781028795}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270757851}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3185035916}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270757851}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886719676}) 2024-09-15T01:59:27.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600193258}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970247520}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600193258}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.352+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594259391}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673769759}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264762075}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673769759}) 2024-09-15T01:59:27.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.353+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460681171}) 2024-09-15T01:59:27.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964056405}) 2024-09-15T01:59:27.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=137958706}) 2024-09-15T01:59:27.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964056405}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.354+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1959129995}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=3} crc {logm=1440958731,mdsmap=1531151339,mgr=1206855712,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3521467529,mon_config_key=2516852880}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=2,mgr=7,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=3} crc {logm=3648901345,mdsmap=3140514009,mgr=1655507991,mgr_metadata=2029235354,mgrstat=3074098653,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2956365539}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=3} crc {logm=1440958731,mdsmap=1531151339,mgr=1206855712,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3521467529,mon_config_key=2516852880}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.355+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=5,monmap=1} crc {logm=3918889262,mdsmap=1531151339,mgr=211645576,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3521467529,mon_config_key=1583035879,monmap=662845760}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=4269123339,monmap=3178194974,osd_metadata=2099894339,osd_pg_creating=537121367,osd_snap=1063953654,osdmap=3777580240}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=86} crc {mon_config_key=4269123339,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=537121367,osd_snap=1063953654,osdmap=4179818174}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=4269123339,monmap=3178194974,osd_metadata=2099894339,osd_pg_creating=537121367,osd_snap=1063953654,osdmap=3777580240}) 2024-09-15T01:59:27.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.356+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=86} crc {mon_config_key=4269123339,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=537121367,osd_snap=1063953654,osdmap=3175235622}) 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=519134898}) 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=792925887}) 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=519134898}) 2024-09-15T01:59:27.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T01:59:27.361+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=3444266094}) 2024-09-15T01:59:27.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.545+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v210) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:27.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.545+0000 7f2c8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.545+0000 7f2c8b7fe640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-15T01:59:27.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.645+0000 7f2ca9d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.645+0000 7f2ca9d1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.645+0000 7f2ca9d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.645+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:27.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.646+0000 7f2c8b7fe640 1 -- 172.21.15.67:0/857367062 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v210) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2ca4133ff0 con 0x7f2ca4138130 2024-09-15T01:59:27.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.646+0000 7f2c8b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.646+0000 7f2c8b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-15T01:59:27.648 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h25' already exists 2024-09-15T01:59:27.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 msgr2=0x7f2c9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 0x7f2c9003abf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007fd0 tx=0x7f2c94009040 comp rx=0 tx=0).stop 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 10 monclient: shutdown 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 msgr2=0x7f2ca409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009e90 tx=0x7f2c9c037000 comp rx=0 tx=0).stop 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 shutdown_connections 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2c90038740 0x7f2c9003abf0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca4138130 0x7f2ca409a320 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ca4131f90 0x7f2ca4099de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 --2- 172.21.15.67:0/857367062 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ca4098260 0x7f2ca40998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 >> 172.21.15.67:0/857367062 conn(0x7f2ca4071690 msgr2=0x7f2ca41300c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:27.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 shutdown_connections 2024-09-15T01:59:27.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.649+0000 7f2ca9d1d640 1 -- 172.21.15.67:0/857367062 wait complete. 2024-09-15T01:59:27.659 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:27.659 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 10 monclient: get_monmap_and_config 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 10 monclient: build_initial_monmap 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 10 monclient: monmap: 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:27.736189+0000 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:27.736189+0000 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:27.738 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 1 Processor -- start 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 1 -- start start 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.736+0000 7fc8cba20640 10 monclient: init 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8c4131aa0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8c409f490 0x7fc8c4131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c409f490 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c4138de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c40a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c40a1820 con 0x7fc8c4131aa0 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c4132930 con 0x7fc8c409f490 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a1b00 con 0x7fc8c40a0a60 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8cba20640 10 monclient(hunting): authenticate will time out at 1200.205312s 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8c409f490 0x7fc8c4131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c4138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc8c409f490 0x7fc8c4131e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:59216/0 (socket says 172.21.15.67:59216) 2024-09-15T01:59:27.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c97bc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc8c4131aa0 0x7fc8c409ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:40268/0 (socket says 172.21.15.67:40268) 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): get_auth_request con 0x7fc8c40a0a60 auth_method 0 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 -- 172.21.15.67:0/3419623890 learned_addr learned my addr 172.21.15.67:0/3419623890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): get_auth_request con 0x7fc8c409f490 auth_method 0 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_done global_id 104373 payload 274 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 -- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 msgr2=0x7fc8c4138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c4138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 -- 172.21.15.67:0/3419623890 >> v1:172.21.15.17:6789/0 conn(0x7fc8c4131aa0 legacy=0x7fc8c409ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient: _finish_hunting 0 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 monclient: found mon.noname-a 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c8fbb640 1 -- 172.21.15.67:0/3419623890 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40a03d0 con 0x7fc8c409f490 2024-09-15T01:59:27.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.737+0000 7fc8c9fbd640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c4138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.740+0000 7fc8c8fbb640 1 -- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 msgr2=0x7fc8c4131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.740+0000 7fc8c8fbb640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4131e70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009910 tx=0x7fc8b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbb3215cec12e3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 1 -- 172.21.15.67:0/3419623890 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b802aea0 con 0x7fc8c409f490 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: dump: 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.750 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 1 -- 172.21.15.67:0/3419623890 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b8032440 con 0x7fc8c409f490 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 5 monclient: authenticate success, global_id 104373 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 4 set_mon_vals no callback set 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 1 -- 172.21.15.67:0/3419623890 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80327e0 con 0x7fc8c409f490 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8c3fff640 10 monclient: dump: 2024-09-15T01:59:27.751 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 10 monclient: shutdown 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 -- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 msgr2=0x7fc8c4131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4131e70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009910 tx=0x7fc8b80091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 -- 172.21.15.67:0/3419623890 shutdown_connections 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c4138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 --2- 172.21.15.67:0/3419623890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4131e70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 -- 172.21.15.67:0/3419623890 >> 172.21.15.67:0/3419623890 conn(0x7fc8c4071390 msgr2=0x7fc8c40a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 -- 172.21.15.67:0/3419623890 shutdown_connections 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.741+0000 7fc8cba20640 1 -- 172.21.15.67:0/3419623890 wait complete. 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.742+0000 7fc8cba20640 10 monclient: build_initial_monmap 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.742+0000 7fc8cba20640 10 monclient: monmap: 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:27.753 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.742+0000 7fc8cba20640 1 Processor -- start 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.742+0000 7fc8cba20640 1 -- start start 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.742+0000 7fc8cba20640 10 monclient: init 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:27.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4130190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c409f490 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c41306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c40a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c4131aa0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4136c60 con 0x7fc8c409f490 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a0f40 con 0x7fc8c4131aa0 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a4320 con 0x7fc8c40a0a60 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8cba20640 10 monclient(hunting): authenticate will time out at 1200.211257s 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59194/0 (socket says 172.21.15.67:59194) 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 -- 172.21.15.67:0/4021049505 learned_addr learned my addr 172.21.15.67:0/4021049505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): get_auth_request con 0x7fc8c4131aa0 auth_method 0 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c41306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): get_auth_request con 0x7fc8c40a0a60 auth_method 0 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c97bc640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4130190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c8fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c97bc640 10 monclient(hunting): get_auth_request con 0x7fc8c409f490 auth_method 0 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c97bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c97bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c97bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient(hunting): handle_auth_done global_id 114118 payload 824 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 -- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 msgr2=0x7fc8c41306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c41306d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 -- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 msgr2=0x7fc8c4130190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4130190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 monclient: found mon.noname-b 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.743+0000 7fc8c9fbd640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4009390 con 0x7fc8c4131aa0 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.745+0000 7fc8c9fbd640 1 -- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 msgr2=0x7fc8c41346e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.745+0000 7fc8c9fbd640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400ea30 tx=0x7fc8b40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400cc20 con 0x7fc8c4131aa0 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: dump: 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.760 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b400cd80 con 0x7fc8c4131aa0 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 5 monclient: authenticate success, global_id 114118 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:27.761 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b40136a0 con 0x7fc8c4131aa0 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 10 monclient: _renew_subs 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8009450 con 0x7fc8c4131aa0 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 10 monclient: dump: 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:27.762 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c37fe640 4 set_mon_vals no callback set 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 10 monclient: _renew_subs 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c4136220 con 0x7fc8c4131aa0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.746+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc8b4013a90 con 0x7fc8c4131aa0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8cba20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8cba20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8cba20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c40639e0 con 0x7fc8c4131aa0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8c17fa640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 0x7fc8b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(210..210 src has 148..210) v4 ==== 13475+0+0 (secure 0 0 0) 0x7fc8b404def0 con 0x7fc8c4131aa0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8c97bc640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 0x7fc8b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:27.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.747+0000 7fc8c97bc640 10 monclient: get_auth_request con 0x7fc8b0038740 auth_method 0 2024-09-15T01:59:27.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.749+0000 7fc8c97bc640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 0x7fc8b003abf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc007810 tx=0x7fc8bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:27.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.751+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc8b401a7d0 con 0x7fc8c4131aa0 2024-09-15T01:59:27.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.751+0000 7fc8c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:27.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.751+0000 7fc8c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:27.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.916+0000 7fc8cba20640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.916+0000 7fc8cba20640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:27.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.916+0000 7fc8cba20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:27.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:27.916+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc8c40a1540 con 0x7fc8c4131aa0 2024-09-15T01:59:28.348 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T01:59:28.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:28.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.455+0000 7f33d52c0640 10 monclient: get_monmap_and_config 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.455+0000 7f33d52c0640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.455+0000 7f33d52c0640 10 monclient: monmap: 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:28.456724+0000 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:28.456724+0000 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:28.458 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:28.459 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.456+0000 7f33d52c0640 1 Processor -- start 2024-09-15T01:59:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.456+0000 7f33d52c0640 1 -- start start 2024-09-15T01:59:28.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.456+0000 7f33d52c0640 10 monclient: init 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-a con 0x7f33d009c2c0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33d009c690 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-b con 0x7f33d009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d006a5c0 con 0x7f33d009c690 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d009d190 con 0x7f33d009c2c0 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0099f50 con 0x7f33d009e970 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33d52c0640 10 monclient(hunting): authenticate will time out at 1228.453195s 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36908/0 (socket says 172.21.15.17:36908) 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51262/0 (socket says 172.21.15.17:51262) 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 -- 172.21.15.17:0/2403218061 learned_addr learned my addr 172.21.15.17:0/2403218061 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): get_auth_request con 0x7f33d009e970 auth_method 0 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): get_auth_request con 0x7f33d009c2c0 auth_method 0 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_done global_id 94538 payload 274 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 -- 172.21.15.17:0/2403218061 >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 msgr2=0x7f33d009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 --2- 172.21.15.17:0/2403218061 >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 -- 172.21.15.17:0/2403218061 >> v1:172.21.15.67:6789/0 conn(0x7f33d009c690 legacy=0x7f33d006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 monclient: found mon.noname-b 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33cf7fe640 1 -- 172.21.15.17:0/2403218061 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d0068ac0 con 0x7f33d009e970 2024-09-15T01:59:28.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.457+0000 7f33ceffd640 1 --2- 172.21.15.17:0/2403218061 >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:28.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cf7fe640 1 -- 172.21.15.17:0/2403218061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 msgr2=0x7f33d006f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cf7fe640 1 --2- 172.21.15.17:0/2403218061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f20 tx=0x7f33c8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b88ea4be578be511 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 1 -- 172.21.15.17:0/2403218061 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8007610 con 0x7f33d009e970 2024-09-15T01:59:28.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: dump: 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 1 -- 172.21.15.17:0/2403218061 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c802d050 con 0x7f33d009e970 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33d52c0640 5 monclient: authenticate success, global_id 94538 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33d52c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 1 -- 172.21.15.17:0/2403218061 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c80079a0 con 0x7f33d009e970 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33cdffb640 10 monclient: dump: 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.472 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33d52c0640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.460+0000 7f33d52c0640 4 set_mon_vals no callback set 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 10 monclient: shutdown 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 -- 172.21.15.17:0/2403218061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 msgr2=0x7f33d006f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 --2- 172.21.15.17:0/2403218061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f20 tx=0x7f33c8009840 comp rx=0 tx=0).stop 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 -- 172.21.15.17:0/2403218061 shutdown_connections 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 --2- 172.21.15.17:0/2403218061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009e970 0x7f33d006f240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 --2- 172.21.15.17:0/2403218061 >> v2:172.21.15.17:3300/0 conn(0x7f33d009c2c0 0x7f33d009a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 -- 172.21.15.17:0/2403218061 >> 172.21.15.17:0/2403218061 conn(0x7f33d00a2f70 msgr2=0x7f33d00a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 -- 172.21.15.17:0/2403218061 shutdown_connections 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.461+0000 7f33d52c0640 1 -- 172.21.15.17:0/2403218061 wait complete. 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.462+0000 7f33d52c0640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.462+0000 7f33d52c0640 10 monclient: monmap: 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:28.474 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.462+0000 7f33d52c0640 1 Processor -- start 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.462+0000 7f33d52c0640 1 -- start start 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.462+0000 7f33d52c0640 10 monclient: init 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33d009c2c0 0x7f33d0140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-b con 0x7f33d009c2c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:28.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-c con 0x7f33d009c690 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): picked mon.noname-a con 0x7f33d009e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d009ce60 con 0x7f33d009e970 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d01222d0 con 0x7f33d009c2c0 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00a2b30 con 0x7f33d009c690 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33d52c0640 10 monclient(hunting): authenticate will time out at 1228.459463s 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36922/0 (socket says 172.21.15.17:36922) 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51274/0 (socket says 172.21.15.17:51274) 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 -- 172.21.15.17:0/3764959926 learned_addr learned my addr 172.21.15.17:0/3764959926 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): get_auth_request con 0x7f33d009c690 auth_method 0 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): get_auth_request con 0x7f33d009e970 auth_method 0 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33d009c2c0 0x7f33d0140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): get_auth_request con 0x7f33d009c2c0 auth_method 0 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient(hunting): handle_auth_done global_id 94541 payload 824 2024-09-15T01:59:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 -- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33d009c2c0 msgr2=0x7f33d0140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33d009c2c0 0x7f33d0140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 -- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 msgr2=0x7f33d011eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 monclient: found mon.noname-c 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33ce7fc640 1 -- 172.21.15.17:0/3764959926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009390 con 0x7f33d009c690 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.463+0000 7f33cf7fe640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33ce7fc640 1 -- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 msgr2=0x7f33d013df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33ce7fc640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c640 tx=0x7f33c4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4012840 con 0x7f33d009c690 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: dump: 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c40129a0 con 0x7f33d009c690 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 5 monclient: authenticate success, global_id 94541 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 10 monclient: _renew_subs 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c8009ae0 con 0x7f33d009c690 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4012d60 con 0x7f33d009c690 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33b6ffd640 10 monclient: dump: 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 10 monclient: _renew_subs 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33cd7fa640 4 set_mon_vals no callback set 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.466+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d01212e0 con 0x7f33d009c690 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f33c4011590 con 0x7f33d009c690 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33b6ffd640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 0x7f33bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(210..210 src has 148..210) v4 ==== 13475+0+0 (secure 0 0 0) 0x7f33c404de60 con 0x7f33d009c690 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33d52c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33d52c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33d52c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d0122100 con 0x7f33d009c690 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33ceffd640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 0x7f33bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.467+0000 7f33ceffd640 10 monclient: get_auth_request con 0x7f33bc038740 auth_method 0 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.470+0000 7f33ceffd640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 0x7f33bc03abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009eb0 tx=0x7f33c00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.472+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f33c4016030 con 0x7f33d009c690 2024-09-15T01:59:28.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.472+0000 7f33b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.472+0000 7f33b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33d52c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33d52c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33d52c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f33d00724a0 con 0x7f33d009c690 2024-09-15T01:59:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33b6ffd640 1 -- 172.21.15.17:0/3764959926 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f33c4016020 con 0x7f33d009c690 2024-09-15T01:59:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.702+0000 7f33b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:28.705 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 msgr2=0x7f33bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 0x7f33bc03abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009eb0 tx=0x7f33c00090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 10 monclient: shutdown 2024-09-15T01:59:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 msgr2=0x7f33d013df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c640 tx=0x7f33c4007ed0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 shutdown_connections 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f33bc038740 0x7f33bc03abf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f33d009e970 0x7f33d011eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f33d009c690 0x7f33d013df40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 --2- 172.21.15.17:0/3764959926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33d009c2c0 0x7f33d0140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 >> 172.21.15.17:0/3764959926 conn(0x7f33d00a2f70 msgr2=0x7f33d0098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 shutdown_connections 2024-09-15T01:59:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.705+0000 7f33d52c0640 1 -- 172.21.15.17:0/3764959926 wait complete. 2024-09-15T01:59:28.717 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T01:59:28.717 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T01:59:28.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T01:59:28.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:59:28.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.804+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v211) v1 ==== 168+0+0 (secure 0 0 0) 0x7fc8b401a7d0 con 0x7fc8c4131aa0 2024-09-15T01:59:28.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.805+0000 7fc8c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:28.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.805+0000 7fc8c17fa640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.814+0000 7fe67b909640 10 monclient: get_monmap_and_config 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.814+0000 7fe67b909640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.814+0000 7fe67b909640 10 monclient: monmap: 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:28.815303+0000 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:28.815303+0000 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:28.817 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 Processor -- start 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 -- start start 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient: init 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T01:59:28.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe67409c1b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe67409e9e0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67406eb30 0x7fe67406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-c con 0x7fe67406eb30 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67409f790 con 0x7fe67409c1b0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674068b60 con 0x7fe67409e9e0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe674069d40 con 0x7fe67406eb30 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe67b909640 10 monclient(hunting): authenticate will time out at 1228.811784s 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe6796a5640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe67409c1b0 0x7fe67409a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42636/0 (socket says 172.21.15.17:42636) 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe6796a5640 1 -- 172.21.15.17:0/41646520 learned_addr learned my addr 172.21.15.17:0/41646520 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.815+0000 7fe679ea6640 1 --2- 172.21.15.17:0/41646520 >> v2:172.21.15.67:3300/0 conn(0x7fe67406eb30 0x7fe67406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): get_auth_request con 0x7fe67406eb30 auth_method 0 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient(hunting): handle_auth_done global_id 114124 payload 274 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 1 -- 172.21.15.17:0/41646520 >> v1:172.21.15.67:6789/0 conn(0x7fe67409e9e0 legacy=0x7fe67409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 1 -- 172.21.15.17:0/41646520 >> v1:172.21.15.17:6789/0 conn(0x7fe67409c1b0 legacy=0x7fe67409a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 1 monclient: found mon.noname-c 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T01:59:28.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.816+0000 7fe679ea6640 1 -- 172.21.15.17:0/41646520 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674069aa0 con 0x7fe67406eb30 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe679ea6640 1 -- 172.21.15.17:0/41646520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67406eb30 msgr2=0x7fe67406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe679ea6640 1 --2- 172.21.15.17:0/41646520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67406eb30 0x7fe67406ef90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe664009e60 tx=0x7fe664009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e870bad5e035446c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 1 -- 172.21.15.17:0/41646520 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664007630 con 0x7fe67406eb30 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: dump: 2024-09-15T01:59:28.825 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 1 -- 172.21.15.17:0/41646520 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66402d050 con 0x7fe67406eb30 2024-09-15T01:59:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 5 monclient: authenticate success, global_id 114124 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 1 -- 172.21.15.17:0/41646520 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6640079c0 con 0x7fe67406eb30 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 4 set_mon_vals no callback set 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe673fff640 10 monclient: dump: 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.827 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 10 monclient: shutdown 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 1 -- 172.21.15.17:0/41646520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67406eb30 msgr2=0x7fe67406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 1 --2- 172.21.15.17:0/41646520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67406eb30 0x7fe67406ef90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe664009e60 tx=0x7fe664009780 comp rx=0 tx=0).stop 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 1 -- 172.21.15.17:0/41646520 shutdown_connections 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.819+0000 7fe67b909640 1 --2- 172.21.15.17:0/41646520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67406eb30 0x7fe67406ef90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 1 -- 172.21.15.17:0/41646520 >> 172.21.15.17:0/41646520 conn(0x7fe674071300 msgr2=0x7fe6740a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 1 -- 172.21.15.17:0/41646520 shutdown_connections 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 1 -- 172.21.15.17:0/41646520 wait complete. 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 10 monclient: monmap: 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:28.829 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 1 Processor -- start 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 1 -- start start 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.820+0000 7fe67b909640 10 monclient: init 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 0x7fe674133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-c con 0x7fe67406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67409c1b0 0x7fe67411da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-b con 0x7fe67409c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe67409e9e0 0x7fe67411df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): picked mon.noname-a con 0x7fe67409e9e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67411ffa0 con 0x7fe67409e9e0 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674141ff0 con 0x7fe67409c1b0 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe674121070 con 0x7fe67406eb30 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe67b909640 10 monclient(hunting): authenticate will time out at 1228.817485s 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 0x7fe674133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 0x7fe674133c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36928/0 (socket says 172.21.15.17:36928) 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 -- 172.21.15.17:0/730214682 learned_addr learned my addr 172.21.15.17:0/730214682 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): get_auth_request con 0x7fe67406eb30 auth_method 0 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67409c1b0 0x7fe67411da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): get_auth_request con 0x7fe67409c1b0 auth_method 0 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient(hunting): handle_auth_done global_id 94544 payload 824 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67409c1b0 msgr2=0x7fe67411da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67409c1b0 0x7fe67411da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe67409e9e0 msgr2=0x7fe67411df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe67409e9e0 0x7fe67411df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 monclient: found mon.noname-c 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.821+0000 7fe6796a5640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664009a20 con 0x7fe67406eb30 2024-09-15T01:59:28.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6796a5640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 msgr2=0x7fe674133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6796a5640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 0x7fe674133c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c640 tx=0x7fe66c004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c005cb0 con 0x7fe67406eb30 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: dump: 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.839 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 5 monclient: authenticate success, global_id 94544 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 10 monclient: _renew_subs 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c005e30 con 0x7fe67406eb30 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674120d90 con 0x7fe67406eb30 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c007340 con 0x7fe67406eb30 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6717fa640 10 monclient: dump: 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.841 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe6737fe640 4 set_mon_vals no callback set 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 10 monclient: _renew_subs 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.824+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe674134a10 con 0x7fe67406eb30 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe67b909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe67b909640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 0x7fe67411e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674063bb0 con 0x7fe674121230 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 0x7fe67411e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe66c01b400 con 0x7fe67406eb30 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient: get_auth_request con 0x7fe674121230 auth_method 0 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 10 monclient(hunting): handle_auth_done global_id 104385 payload 824 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe6717fa640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe660038740 0x7fe66003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe678ea4640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe674134c90 con 0x7fe674121230 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7fe66c04d8e0 con 0x7fe67406eb30 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe679ea6640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe660038740 0x7fe66003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.825+0000 7fe679ea6640 10 monclient: get_auth_request con 0x7fe660038740 auth_method 0 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.827+0000 7fe678ea4640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 msgr2=0x7fe67411e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.827+0000 7fe678ea4640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 0x7fe67411e4d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66800d8b0 con 0x7fe674121230 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 10 monclient: dump: 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe66800da10 con 0x7fe674121230 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 msgr2=0x7fe67411e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 0x7fe67411e4d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 msgr2=0x7fe67411e4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe6717fa640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe674121230 0x7fe67411e4d0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.828+0000 7fe679ea6640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe660038740 0x7fe66003abf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe674068b30 tx=0x7fe664037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe67b909640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe67b909640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 0x7fe67411ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe67b909640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674136750 con 0x7fe6741341b0 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 0x7fe67411ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient: get_auth_request con 0x7fe6741341b0 auth_method 0 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 10 monclient(hunting): handle_auth_done global_id 104388 payload 824 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.834+0000 7fe678ea4640 1 -- 172.21.15.17:0/730214682 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe6740724a0 con 0x7fe6741341b0 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe678ea4640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 msgr2=0x7fe67411ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe678ea4640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 0x7fe67411ea10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66801d610 con 0x7fe6741341b0 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 10 monclient: dump: 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.847 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4397 (secure 0 0 0) 0x7fe66801d770 con 0x7fe6741341b0 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 msgr2=0x7fe67411ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 0x7fe67411ea10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668019000 comp rx=0 tx=0).stop 2024-09-15T01:59:28.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 msgr2=0x7fe67411ea10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:28.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.836+0000 7fe6717fa640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe6741341b0 0x7fe67411ea10 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe66800f780 tx=0x7fe668019000 comp rx=0 tx=0).stop 2024-09-15T01:59:28.849 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.850 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:28.851 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:28.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:28.853 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:28.854 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:28.855 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:28.856 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:28.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:28.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:28.860 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.862 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.863 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:28.864 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe660038740 msgr2=0x7fe66003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe660038740 0x7fe66003abf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe674068b30 tx=0x7fe664037000 comp rx=0 tx=0).stop 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 10 monclient: shutdown 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 msgr2=0x7fe674133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 --2- 172.21.15.17:0/730214682 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe67406eb30 0x7fe674133c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c640 tx=0x7fe66c004960 comp rx=0 tx=0).stop 2024-09-15T01:59:28.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe679ea6640 1 -- 172.21.15.17:0/730214682 reap_dead start 2024-09-15T01:59:28.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe679ea6640 1 -- 172.21.15.17:0/730214682 reap_dead start 2024-09-15T01:59:28.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 shutdown_connections 2024-09-15T01:59:28.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 >> 172.21.15.17:0/730214682 conn(0x7fe674071300 msgr2=0x7fe674134ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 shutdown_connections 2024-09-15T01:59:28.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.837+0000 7fe67b909640 1 -- 172.21.15.17:0/730214682 wait complete. 2024-09-15T01:59:28.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:28.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T01:59:28.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:59:28.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.907+0000 7fc8cba20640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:28.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.907+0000 7fc8cba20640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:28.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.907+0000 7fc8cba20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T01:59:28.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.907+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc8c4137e30 con 0x7fc8c4131aa0 2024-09-15T01:59:28.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.908+0000 7fc8c17fa640 1 -- 172.21.15.67:0/4021049505 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v211) v1 ==== 147+0+0 (secure 0 0 0) 0x7fc8b401ad90 con 0x7fc8c4131aa0 2024-09-15T01:59:28.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.908+0000 7fc8c17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:28.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.908+0000 7fc8c17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-15T01:59:28.910 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h26' already exists 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 msgr2=0x7fc8b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 0x7fc8b003abf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc007810 tx=0x7fc8bc009040 comp rx=0 tx=0).stop 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 10 monclient: shutdown 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 msgr2=0x7fc8c41346e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400ea30 tx=0x7fc8b40090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 shutdown_connections 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc8b0038740 0x7fc8b003abf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4131aa0 0x7fc8c41346e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc8c40a0a60 0x7fc8c41306d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 --2- 172.21.15.67:0/4021049505 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc8c409f490 0x7fc8c4130190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 >> 172.21.15.67:0/4021049505 conn(0x7fc8c4071390 msgr2=0x7fc8c40a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 shutdown_connections 2024-09-15T01:59:28.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.910+0000 7fc8cba20640 1 -- 172.21.15.67:0/4021049505 wait complete. 2024-09-15T01:59:28.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:28.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.961+0000 7fe46279d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.961+0000 7fe46279d640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.961+0000 7fe46279d640 10 monclient: monmap: 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:28.962053+0000 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:28.962053+0000 2024-09-15T01:59:28.963 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:28.964 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 Processor -- start 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 -- start start 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient: init 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T01:59:28.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe45c0a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe45c132040 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe45c0987b0 0x7fe45c13a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-a con 0x7fe45c0987b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c0a1580 con 0x7fe45c132040 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c0a5c80 con 0x7fe45c0a58b0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c132440 con 0x7fe45c0987b0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe46279d640 10 monclient(hunting): authenticate will time out at 1228.958646s 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe45c0987b0 0x7fe45c13a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe453fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe45c132040 0x7fe45c136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42646/0 (socket says 172.21.15.17:42646) 2024-09-15T01:59:28.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe453fff640 1 -- 172.21.15.17:0/1926710856 learned_addr learned my addr 172.21.15.17:0/1926710856 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 1 --2- 172.21.15.17:0/1926710856 >> v2:172.21.15.17:3300/0 conn(0x7fe45c0987b0 0x7fe45c13a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51318/0 (socket says 172.21.15.17:51318) 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): get_auth_request con 0x7fe45c0987b0 auth_method 0 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.962+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 10 monclient(hunting): handle_auth_done global_id 104391 payload 274 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 1 -- 172.21.15.17:0/1926710856 >> v1:172.21.15.67:6789/0 conn(0x7fe45c0a58b0 legacy=0x7fe45c0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 1 -- 172.21.15.17:0/1926710856 >> v1:172.21.15.17:6789/0 conn(0x7fe45c132040 legacy=0x7fe45c136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 1 monclient: found mon.noname-a 2024-09-15T01:59:28.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:28.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.963+0000 7fe460d3a640 1 -- 172.21.15.17:0/1926710856 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c098bb0 con 0x7fe45c0987b0 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.965+0000 7fe460d3a640 1 -- 172.21.15.17:0/1926710856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0987b0 msgr2=0x7fe45c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.965+0000 7fe460d3a640 1 --2- 172.21.15.17:0/1926710856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0987b0 0x7fe45c13a500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0a60e0 tx=0x7fe454009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5c417ed231aaba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 1 -- 172.21.15.17:0/1926710856 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454007840 con 0x7fe45c0987b0 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:28.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: dump: 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 1 -- 172.21.15.17:0/1926710856 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45402d050 con 0x7fe45c0987b0 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 5 monclient: authenticate success, global_id 104391 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 1 -- 172.21.15.17:0/1926710856 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454007bd0 con 0x7fe45c0987b0 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 4 set_mon_vals no callback set 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe45b7fe640 10 monclient: dump: 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.977 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 10 monclient: shutdown 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 -- 172.21.15.17:0/1926710856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0987b0 msgr2=0x7fe45c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 --2- 172.21.15.17:0/1926710856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0987b0 0x7fe45c13a500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0a60e0 tx=0x7fe454009780 comp rx=0 tx=0).stop 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 -- 172.21.15.17:0/1926710856 shutdown_connections 2024-09-15T01:59:28.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 --2- 172.21.15.17:0/1926710856 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0987b0 0x7fe45c13a500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 -- 172.21.15.17:0/1926710856 >> 172.21.15.17:0/1926710856 conn(0x7fe45c070ed0 msgr2=0x7fe45c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 -- 172.21.15.17:0/1926710856 shutdown_connections 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.966+0000 7fe46279d640 1 -- 172.21.15.17:0/1926710856 wait complete. 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.967+0000 7fe46279d640 10 monclient: build_initial_monmap 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.967+0000 7fe46279d640 10 monclient: monmap: 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:28.979 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.967+0000 7fe46279d640 1 Processor -- start 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.967+0000 7fe46279d640 1 -- start start 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.967+0000 7fe46279d640 10 monclient: init 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c0987b0 0x7fe45c134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-c con 0x7fe45c0987b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-a con 0x7fe45c0a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c132040 0x7fe45c1411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): picked mon.noname-b con 0x7fe45c132040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c1357c0 con 0x7fe45c0a58b0 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c098dc0 con 0x7fe45c132040 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c09e910 con 0x7fe45c0987b0 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe46279d640 10 monclient(hunting): authenticate will time out at 1228.964144s 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51328/0 (socket says 172.21.15.17:51328) 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 -- 172.21.15.17:0/3402504145 learned_addr learned my addr 172.21.15.17:0/3402504145 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): get_auth_request con 0x7fe45c0a58b0 auth_method 0 2024-09-15T01:59:28.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c0987b0 0x7fe45c134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): get_auth_request con 0x7fe45c0987b0 auth_method 0 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c132040 0x7fe45c1411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): get_auth_request con 0x7fe45c132040 auth_method 0 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe460d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient(hunting): handle_auth_done global_id 104394 payload 824 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c0987b0 msgr2=0x7fe45c134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c0987b0 0x7fe45c134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c132040 msgr2=0x7fe45c1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c132040 0x7fe45c1411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:28.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 monclient: found mon.noname-a 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.968+0000 7fe453fff640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe454009a20 con 0x7fe45c0a58b0 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe453fff640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 msgr2=0x7fe45c130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe453fff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c5e0 tx=0x7fe444004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444005cb0 con 0x7fe45c0a58b0 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 10 monclient: dump: 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444005e30 con 0x7fe45c0a58b0 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.970+0000 7fe46279d640 5 monclient: authenticate success, global_id 104394 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:28.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444007340 con 0x7fe45c0a58b0 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 monclient: _renew_subs 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c11dd70 con 0x7fe45c0a58b0 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 10 monclient: dump: 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe45affd640 4 set_mon_vals no callback set 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 monclient: _renew_subs 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c11bca0 con 0x7fe45c0a58b0 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 0x7fe45c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c11e1f0 con 0x7fe45c131190 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.971+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe44401b400 con 0x7fe45c0a58b0 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 0x7fe45c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient: get_auth_request con 0x7fe45c131190 auth_method 0 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe458ff9640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe43c038740 0x7fe43c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7fe44404d920 con 0x7fe45c0a58b0 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 10 monclient(hunting): handle_auth_done global_id 94550 payload 824 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe45bfff640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe45c11bf20 con 0x7fe45c131190 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe460d3a640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe43c038740 0x7fe43c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.972+0000 7fe460d3a640 10 monclient: get_auth_request con 0x7fe43c038740 auth_method 0 2024-09-15T01:59:28.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe45bfff640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 msgr2=0x7fe45c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe45bfff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 0x7fe45c0298c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe460d3a640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe43c038740 0x7fe43c03abf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe45c09f780 tx=0x7fe454037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00d8b0 con 0x7fe45c131190 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe458ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe458ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.974+0000 7fe458ff9640 10 monclient: dump: 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.992 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe44c00da10 con 0x7fe45c131190 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 msgr2=0x7fe45c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 0x7fe45c0298c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 msgr2=0x7fe45c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.975+0000 7fe458ff9640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c131190 0x7fe45c0298c0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c007e10 comp rx=0 tx=0).stop 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.980+0000 7fe46279d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:28.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.980+0000 7fe46279d640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 0x7fe45c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.980+0000 7fe46279d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.980+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c063b20 con 0x7fe45c029e00 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 0x7fe45c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient: get_auth_request con 0x7fe45c029e00 auth_method 0 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 10 monclient(hunting): handle_auth_done global_id 94553 payload 824 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.981+0000 7fe45bfff640 1 -- 172.21.15.17:0/3402504145 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe45c0724a0 con 0x7fe45c029e00 2024-09-15T01:59:28.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe45bfff640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 msgr2=0x7fe45c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe45bfff640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 0x7fe45c02a200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c01d610 con 0x7fe45c029e00 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 10 monclient: dump: 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:28.995 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7fe44c01d770 con 0x7fe45c029e00 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 msgr2=0x7fe45c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 0x7fe45c02a200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c019000 comp rx=0 tx=0).stop 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 msgr2=0x7fe45c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.983+0000 7fe458ff9640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe45c029e00 0x7fe45c02a200 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00f780 tx=0x7fe44c019000 comp rx=0 tx=0).stop 2024-09-15T01:59:28.996 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:28.997 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:28.998 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:28.999 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:29.000 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:29.001 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:29.002 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.003 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.004 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.006 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:29.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.009 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe43c038740 msgr2=0x7fe43c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe43c038740 0x7fe43c03abf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe45c09f780 tx=0x7fe454037000 comp rx=0 tx=0).stop 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 10 monclient: shutdown 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 msgr2=0x7fe45c130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c5e0 tx=0x7fe444004960 comp rx=0 tx=0).stop 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe460d3a640 1 -- 172.21.15.17:0/3402504145 reap_dead start 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 shutdown_connections 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 --2- 172.21.15.17:0/3402504145 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe45c0a58b0 0x7fe45c130c50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 >> 172.21.15.17:0/3402504145 conn(0x7fe45c070ed0 msgr2=0x7fe45c09ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 shutdown_connections 2024-09-15T01:59:29.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:28.984+0000 7fe46279d640 1 -- 172.21.15.17:0/3402504145 wait complete. 2024-09-15T01:59:29.014 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:29.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T01:59:29.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.998+0000 7f4890f9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.998+0000 7f4890f9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.998+0000 7f4890f9d640 10 monclient: monmap: 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:28.999057+0000 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:28.999057+0000 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.999+0000 7f4890f9d640 1 Processor -- start 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.999+0000 7f4890f9d640 1 -- start start 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:28.999+0000 7f4890f9d640 10 monclient: init 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 0x7f488c131e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f488c131aa0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 0x7f488c09ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f488c0a0a60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f488c09f2c0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c09f6e0 con 0x7f488c09f2c0 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c0a1820 con 0x7f488c131aa0 2024-09-15T01:59:29.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c132930 con 0x7f488c0a0a60 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4890f9d640 10 monclient(hunting): authenticate will time out at 1201.468226s 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f488c09f2c0 0x7f488c068e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60664/0 (socket says 172.21.15.67:60664) 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488ad9d640 1 -- 172.21.15.67:0/2351376218 learned_addr learned my addr 172.21.15.67:0/2351376218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 0x7f488c09ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 1 --2- 172.21.15.67:0/2351376218 >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 0x7f488c131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): get_auth_request con 0x7f488c131aa0 auth_method 0 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): get_auth_request con 0x7f488c0a0a60 auth_method 0 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient(hunting): handle_auth_done global_id 94556 payload 274 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 -- 172.21.15.67:0/2351376218 >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 msgr2=0x7f488c131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2351376218 >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 0x7f488c131e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 -- 172.21.15.67:0/2351376218 >> v1:172.21.15.67:6789/0 conn(0x7f488c09f2c0 legacy=0x7f488c068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:29.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 monclient: found mon.noname-b 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f4889d9b640 1 -- 172.21.15.67:0/2351376218 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0a1b00 con 0x7f488c0a0a60 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.000+0000 7f488a59c640 1 --2- 172.21.15.67:0/2351376218 >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 0x7f488c131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.003+0000 7f4889d9b640 1 -- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 msgr2=0x7f488c09ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.003+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 0x7f488c09ed80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4880009850 tx=0x7f48800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b68b16cff0e6e46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 1 -- 172.21.15.67:0/2351376218 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488002adf0 con 0x7f488c0a0a60 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: dump: 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:29.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 5 monclient: authenticate success, global_id 94556 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 1 -- 172.21.15.67:0/2351376218 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880032440 con 0x7f488c0a0a60 2024-09-15T01:59:29.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 4 set_mon_vals no callback set 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 1 -- 172.21.15.67:0/2351376218 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800327e0 con 0x7f488c0a0a60 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f488959a640 10 monclient: dump: 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.029 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 10 monclient: shutdown 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 -- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 msgr2=0x7f488c09ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 0x7f488c09ed80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4880009850 tx=0x7f48800090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 -- 172.21.15.67:0/2351376218 shutdown_connections 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2351376218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c0a0a60 0x7f488c09ed80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2351376218 >> v2:172.21.15.17:3300/0 conn(0x7f488c131aa0 0x7f488c131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 -- 172.21.15.67:0/2351376218 >> 172.21.15.67:0/2351376218 conn(0x7f488c071390 msgr2=0x7f488c0a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 -- 172.21.15.67:0/2351376218 shutdown_connections 2024-09-15T01:59:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.004+0000 7f4890f9d640 1 -- 172.21.15.67:0/2351376218 wait complete. 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 10 monclient: monmap: 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 1 Processor -- start 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 1 -- start start 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 10 monclient: init 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.005+0000 7f4890f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f488c09f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f488c0a0a60 0x7f488c130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f488c0a0a60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f488c131aa0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c11e2c0 con 0x7f488c0a0a60 2024-09-15T01:59:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c09b8a0 con 0x7f488c131aa0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c1421b0 con 0x7f488c09f2c0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4890f9d640 10 monclient(hunting): authenticate will time out at 1201.474048s 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59204/0 (socket says 172.21.15.67:59204) 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f488c0a0a60 0x7f488c130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 1 -- 172.21.15.67:0/2679147918 learned_addr learned my addr 172.21.15.67:0/2679147918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): get_auth_request con 0x7f488c09f2c0 auth_method 0 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): get_auth_request con 0x7f488c131aa0 auth_method 0 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): get_auth_request con 0x7f488c0a0a60 auth_method 0 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f4889d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient(hunting): handle_auth_done global_id 94559 payload 824 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 -- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 msgr2=0x7f488c11b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 -- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f488c0a0a60 msgr2=0x7f488c130ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f488c0a0a60 0x7f488c130ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 monclient: found mon.noname-c 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488a59c640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c009390 con 0x7f488c09f2c0 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.006+0000 7f488ad9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.008+0000 7f488a59c640 1 -- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 msgr2=0x7f488c09aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.008+0000 7f488a59c640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f487c00ea50 tx=0x7f487c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c009670 con 0x7f488c09f2c0 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: dump: 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.036 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 5 monclient: authenticate success, global_id 94559 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _renew_subs 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4880009390 con 0x7f488c09f2c0 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c00cc20 con 0x7f488c09f2c0 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c013470 con 0x7f488c09f2c0 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f487a7fc640 10 monclient: dump: 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:29.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4888d99640 4 set_mon_vals no callback set 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _renew_subs 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c11e090 con 0x7f488c09f2c0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.009+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c0639e0 con 0x7f488c09f2c0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.010+0000 7f488a59c640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f487c00ea50 tx=0x7f487c0090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f487c013820 mgrmap(e 5) v1 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.010+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f487c013820 con 0x7f488c09f2c0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.010+0000 7f487a7fc640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 0x7f486c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.010+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 0x7f486c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.010+0000 7f4889d9b640 10 monclient: get_auth_request con 0x7f486c038740 auth_method 0 2024-09-15T01:59:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.012+0000 7f4889d9b640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 0x7f486c03abf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4880031000 tx=0x7f4880032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.014+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7f487c04e560 con 0x7f488c09f2c0 2024-09-15T01:59:29.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.112+0000 7f8115f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:29.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.112+0000 7f8115f5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.112+0000 7f8115f5d640 10 monclient: monmap: 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:29.113136+0000 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:29.113136+0000 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:29.116 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:29.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 Processor -- start 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 -- start start 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient: init 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f811009c4f0 0x7f811009a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f811009c4f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f811009aed0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811009c8c0 0x7f811009e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f811009c8c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811006a390 con 0x7f811009aed0 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8110099f50 con 0x7f811009c4f0 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811009f790 con 0x7f811009c8c0 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f8115f5d640 10 monclient(hunting): authenticate will time out at 1229.109578s 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f811009c4f0 0x7f811009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811009c8c0 0x7f811009e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810effd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f811009aed0 0x7f811006eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42658/0 (socket says 172.21.15.17:42658) 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810effd640 1 -- 172.21.15.17:0/359693000 learned_addr learned my addr 172.21.15.17:0/359693000 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): get_auth_request con 0x7f811009c4f0 auth_method 0 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): get_auth_request con 0x7f811009c8c0 auth_method 0 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient(hunting): handle_auth_done global_id 104403 payload 274 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 1 -- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811009c8c0 msgr2=0x7f811009e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 1 --2- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811009c8c0 0x7f811009e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 1 -- 172.21.15.17:0/359693000 >> v1:172.21.15.17:6789/0 conn(0x7f811009aed0 legacy=0x7f811006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.113+0000 7f810f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.114+0000 7f810f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.114+0000 7f810f7fe640 1 -- 172.21.15.17:0/359693000 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110069d40 con 0x7f811009c4f0 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.116+0000 7f810f7fe640 1 -- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 msgr2=0x7f811009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.116+0000 7f810f7fe640 1 --2- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f811009a990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8100009850 tx=0x7f81000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8e03a74aef3ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 1 -- 172.21.15.17:0/359693000 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810002adf0 con 0x7f811009c4f0 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: dump: 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.131 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 5 monclient: authenticate success, global_id 104403 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 1 -- 172.21.15.17:0/359693000 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100032440 con 0x7f811009c4f0 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:29.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 4 set_mon_vals no callback set 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 1 -- 172.21.15.17:0/359693000 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000327e0 con 0x7f811009c4f0 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f810e7fc640 10 monclient: dump: 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.133 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 10 monclient: shutdown 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 -- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 msgr2=0x7f811009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 --2- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f811009a990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8100009850 tx=0x7f81000090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 -- 172.21.15.17:0/359693000 shutdown_connections 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 --2- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811009c8c0 0x7f811009e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 --2- 172.21.15.17:0/359693000 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f811009a990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 -- 172.21.15.17:0/359693000 >> 172.21.15.17:0/359693000 conn(0x7f8110071300 msgr2=0x7f81100a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 -- 172.21.15.17:0/359693000 shutdown_connections 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.117+0000 7f8115f5d640 1 -- 172.21.15.17:0/359693000 wait complete. 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.118+0000 7f8115f5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.118+0000 7f8115f5d640 10 monclient: monmap: 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:29.135 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.118+0000 7f8115f5d640 1 Processor -- start 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.118+0000 7f8115f5d640 1 -- start start 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.118+0000 7f8115f5d640 10 monclient: init 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f811009c4f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009c8c0 0x7f8110133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f811009c8c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81101343d0 0x7f811011da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f81101343d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110134840 con 0x7f811009c4f0 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811009cdb0 con 0x7f811009c8c0 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100a2b60 con 0x7f81101343d0 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f8115f5d640 10 monclient(hunting): authenticate will time out at 1229.115280s 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51350/0 (socket says 172.21.15.17:51350) 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81101343d0 0x7f811011da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 -- 172.21.15.17:0/1385314743 learned_addr learned my addr 172.21.15.17:0/1385314743 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): get_auth_request con 0x7f811009c4f0 auth_method 0 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): get_auth_request con 0x7f81101343d0 auth_method 0 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810effd640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009c8c0 0x7f8110133e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810effd640 10 monclient(hunting): get_auth_request con 0x7f811009c8c0 auth_method 0 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient(hunting): handle_auth_done global_id 104406 payload 824 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81101343d0 msgr2=0x7f811011da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81101343d0 0x7f811011da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009c8c0 msgr2=0x7f8110133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009c8c0 0x7f8110133e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810f7fe640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100009390 con 0x7f811009c4f0 2024-09-15T01:59:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.119+0000 7f810ffff640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81101343d0 0x7f811011da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.121+0000 7f810f7fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 msgr2=0x7f81101410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.121+0000 7f810f7fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8100009850 tx=0x7f8100032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100031a10 con 0x7f811009c4f0 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: dump: 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.141 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100031b70 con 0x7f811009c4f0 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 5 monclient: authenticate success, global_id 104406 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 10 monclient: _renew_subs 2024-09-15T01:59:29.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8110120050 con 0x7f811009c4f0 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810003d440 con 0x7f811009c4f0 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f80f77fe640 10 monclient: dump: 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.143 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 10 monclient: _renew_subs 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f810dffb640 4 set_mon_vals no callback set 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.122+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8110120fb0 con 0x7f811009c4f0 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f8115f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f8115f5d640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 0x7f811011df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110063bb0 con 0x7f811009aed0 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8100039040 con 0x7f811009c4f0 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 0x7f811011df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f80f77fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80fc038790 0x7f80fc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient: get_auth_request con 0x7f811009aed0 auth_method 0 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7f810006dc00 con 0x7f811009c4f0 2024-09-15T01:59:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810ffff640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80fc038790 0x7f80fc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810ffff640 10 monclient: get_auth_request con 0x7f80fc038790 auth_method 0 2024-09-15T01:59:29.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 10 monclient(hunting): handle_auth_done global_id 114139 payload 824 2024-09-15T01:59:29.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.123+0000 7f810effd640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81100724a0 con 0x7f811009aed0 2024-09-15T01:59:29.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.125+0000 7f810effd640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 msgr2=0x7f811011df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.125+0000 7f810effd640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 0x7f811011df90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104004e60 con 0x7f811009aed0 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 10 monclient: dump: 2024-09-15T01:59:29.151 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f810ffff640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80fc038790 0x7f80fc03ac40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f810800e950 tx=0x7f810800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8104007170 con 0x7f811009aed0 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 msgr2=0x7f811011df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 0x7f811011df90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104009040 comp rx=0 tx=0).stop 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 msgr2=0x7f811011df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.126+0000 7f80f77fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811009aed0 0x7f811011df90 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104009040 comp rx=0 tx=0).stop 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f8115f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f8115f5d640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 0x7f8110096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f8115f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811011e8d0 con 0x7f811011e4d0 2024-09-15T01:59:29.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 0x7f8110096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient: get_auth_request con 0x7f811011e4d0 auth_method 0 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 10 monclient(hunting): handle_auth_done global_id 114142 payload 824 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.132+0000 7f810effd640 1 -- 172.21.15.17:0/1385314743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8110067220 con 0x7f811011e4d0 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f810effd640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 msgr2=0x7f8110096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f810effd640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 0x7f8110096a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040208a0 con 0x7f811011e4d0 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 10 monclient: dump: 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.155 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f810401c400 con 0x7f811011e4d0 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 msgr2=0x7f8110096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 0x7f8110096a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104018000 comp rx=0 tx=0).stop 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 msgr2=0x7f8110096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f80f77fe640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811011e4d0 0x7f8110096a10 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f810400e740 tx=0x7f8104018000 comp rx=0 tx=0).stop 2024-09-15T01:59:29.157 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:29.157 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:29.157 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:29.157 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 96, 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:29.158 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.159 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:29.161 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:29.162 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.164 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.166 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.169 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:29.170 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80fc038790 msgr2=0x7f80fc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80fc038790 0x7f80fc03ac40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f810800e950 tx=0x7f810800b040 comp rx=0 tx=0).stop 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 10 monclient: shutdown 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 msgr2=0x7f81101410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f8115f5d640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8100009850 tx=0x7f8100032000 comp rx=0 tx=0).stop 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.135+0000 7f810ffff640 1 -- 172.21.15.17:0/1385314743 reap_dead start 2024-09-15T01:59:29.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.136+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 shutdown_connections 2024-09-15T01:59:29.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.136+0000 7f8115f5d640 1 --2- 172.21.15.17:0/1385314743 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811009c4f0 0x7f81101410d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.136+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 >> 172.21.15.17:0/1385314743 conn(0x7f8110071300 msgr2=0x7f811009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.136+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 shutdown_connections 2024-09-15T01:59:29.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.136+0000 7f8115f5d640 1 -- 172.21.15.17:0/1385314743 wait complete. 2024-09-15T01:59:29.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T01:59:29.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T01:59:29.174 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T01:59:29.174 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T01:59:29.174 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T01:59:29.174 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T01:59:29.175 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T01:59:29.186 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T01:59:29.186 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T01:59:29.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.279+0000 7f2f4e59d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.279+0000 7f2f4e59d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.280+0000 7f2f4e59d640 10 monclient: monmap: 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:29.280831+0000 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:29.280831+0000 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:29.283 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.280+0000 7f2f4e59d640 1 Processor -- start 2024-09-15T01:59:29.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.280+0000 7f2f4e59d640 1 -- start start 2024-09-15T01:59:29.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.280+0000 7f2f4e59d640 10 monclient: init 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 0x7f2f481321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f48068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2f480692f0 0x7f2f480696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f480692f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 0x7f2f4812ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4812e900 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f480a1950 con 0x7f2f480692f0 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f480a1670 con 0x7f2f4812e900 2024-09-15T01:59:29.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480a0020 con 0x7f2f48068f20 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4e59d640 10 monclient(hunting): authenticate will time out at 1229.277420s 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2f480692f0 0x7f2f480696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 0x7f2f481321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4cb3a640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 msgr2=0x7f2f4812ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2f480692f0 0x7f2f480696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51364/0 (socket says 172.21.15.17:51364) 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f4cb3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 0x7f2f4812ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 -- 172.21.15.17:0/857575993 learned_addr learned my addr 172.21.15.17:0/857575993 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): get_auth_request con 0x7f2f48068f20 auth_method 0 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): get_auth_request con 0x7f2f480692f0 auth_method 0 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient(hunting): handle_auth_done global_id 104409 payload 274 2024-09-15T01:59:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 -- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 msgr2=0x7f2f481321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 0x7f2f481321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 -- 172.21.15.17:0/857575993 >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 msgr2=0x7f2f4812ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 --2- 172.21.15.17:0/857575993 >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 0x7f2f4812ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f477fe640 1 -- 172.21.15.17:0/857575993 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f480a1390 con 0x7f2f480692f0 2024-09-15T01:59:29.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.281+0000 7f2f47fff640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 0x7f2f481321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f477fe640 1 -- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f480692f0 msgr2=0x7f2f480696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f477fe640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f480692f0 0x7f2f480696c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f480a2f70 tx=0x7f2f3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e4b2ba3392325d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 1 -- 172.21.15.17:0/857575993 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c032440 con 0x7f2f480692f0 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: dump: 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.296 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f4e59d640 5 monclient: authenticate success, global_id 104409 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f4e59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 1 -- 172.21.15.17:0/857575993 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0325a0 con 0x7f2f480692f0 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 1 -- 172.21.15.17:0/857575993 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c032940 con 0x7f2f480692f0 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f4e59d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f4e59d640 4 set_mon_vals no callback set 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.284+0000 7f2f46ffd640 10 monclient: dump: 2024-09-15T01:59:29.298 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 10 monclient: shutdown 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 -- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f480692f0 msgr2=0x7f2f480696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f480692f0 0x7f2f480696c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f480a2f70 tx=0x7f2f3c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 -- 172.21.15.17:0/857575993 shutdown_connections 2024-09-15T01:59:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/857575993 >> v2:172.21.15.67:3300/0 conn(0x7f2f4812e900 0x7f2f4812ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f480692f0 0x7f2f480696c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/857575993 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f48068f20 0x7f2f481321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 -- 172.21.15.17:0/857575993 >> 172.21.15.17:0/857575993 conn(0x7f2f48070fb0 msgr2=0x7f2f480a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 -- 172.21.15.17:0/857575993 shutdown_connections 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 1 -- 172.21.15.17:0/857575993 wait complete. 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 10 monclient: build_initial_monmap 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.285+0000 7f2f4e59d640 10 monclient: monmap: 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:29.301 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 Processor -- start 2024-09-15T01:59:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 -- start start 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient: init 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f48068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f480692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 0x7f2f4812e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4812e900 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:29.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4809a2c0 con 0x7f2f48068f20 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480a0750 con 0x7f2f4812e900 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f48130e00 con 0x7f2f480692f0 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.286+0000 7f2f4e59d640 10 monclient(hunting): authenticate will time out at 1229.282963s 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51380/0 (socket says 172.21.15.17:51380) 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37000/0 (socket says 172.21.15.17:37000) 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 1 -- 172.21.15.17:0/2498405513 learned_addr learned my addr 172.21.15.17:0/2498405513 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): get_auth_request con 0x7f2f48068f20 auth_method 0 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f4cb3a640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 msgr2=0x7f2f4812e110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): get_auth_request con 0x7f2f480692f0 auth_method 0 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f4cb3a640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 0x7f2f4812e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:29.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient(hunting): handle_auth_done global_id 94571 payload 824 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 msgr2=0x7f2f4812e110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 0x7f2f4812e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 msgr2=0x7f2f48141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 monclient: found mon.noname-c 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f477fe640 1 -- 172.21.15.17:0/2498405513 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38009390 con 0x7f2f480692f0 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.287+0000 7f2f47fff640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:29.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f477fe640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 msgr2=0x7f2f4812dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f477fe640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c031000 tx=0x7f2f3c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0319c0 con 0x7f2f480692f0 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: dump: 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.308 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: _finish_auth 0 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 5 monclient: authenticate success, global_id 94571 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c031b60 con 0x7f2f480692f0 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:29.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 10 monclient: _renew_subs 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c03d440 con 0x7f2f480692f0 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c009450 con 0x7f2f480692f0 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f2ffff640 10 monclient: dump: 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:29.310 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:29.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f467fc640 4 set_mon_vals no callback set 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 10 monclient: _renew_subs 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.289+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4809b550 con 0x7f2f480692f0 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2f3c039040 con 0x7f2f480692f0 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f4e59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f4e59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f4e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4809b6e0 con 0x7f2f480692f0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f2ffff640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 0x7f2f3403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.290+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7f2f3c06d660 con 0x7f2f480692f0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.291+0000 7f2f47fff640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 0x7f2f3403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.291+0000 7f2f47fff640 10 monclient: get_auth_request con 0x7f2f34038720 auth_method 0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.293+0000 7f2f47fff640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 0x7f2f3403abd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2f38011640 tx=0x7f2f380090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:29.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.295+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2f3c02f8a0 con 0x7f2f480692f0 2024-09-15T01:59:29.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.295+0000 7f2f2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.295+0000 7f2f2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.521+0000 7f2f4e59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.521+0000 7f2f4e59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.521+0000 7f2f4e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.521+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2f480724a0 con 0x7f2f480692f0 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.522+0000 7f2f2ffff640 1 -- 172.21.15.17:0/2498405513 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1527 (secure 0 0 0) 0x7f2f3c037020 con 0x7f2f480692f0 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.522+0000 7f2f2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.522+0000 7f2f2ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:29.525 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:29.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 msgr2=0x7f2f3403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 0x7f2f3403abd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2f38011640 tx=0x7f2f380090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 10 monclient: shutdown 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 msgr2=0x7f2f4812dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c031000 tx=0x7f2f3c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 shutdown_connections 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2f34038720 0x7f2f3403abd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4812e900 0x7f2f4812e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2f480692f0 0x7f2f4812dbd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 --2- 172.21.15.17:0/2498405513 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2f48068f20 0x7f2f48141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 >> 172.21.15.17:0/2498405513 conn(0x7f2f48070fb0 msgr2=0x7f2f4812f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 shutdown_connections 2024-09-15T01:59:29.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:29.524+0000 7f2f4e59d640 1 -- 172.21.15.17:0/2498405513 wait complete. 2024-09-15T01:59:29.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.838+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f487c01bb90 con 0x7f488c09f2c0 2024-09-15T01:59:29.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.838+0000 7f487a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:29.840 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:29.838+0000 7f487a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:30.010+0000 7f4890f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:30.010+0000 7f4890f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:30.010+0000 7f4890f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:30.010+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f488c11d080 con 0x7f488c09f2c0 2024-09-15T01:59:32.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:32.005+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:32.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:32.005+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:32.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:32.005+0000 7f48797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:32.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:32.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.643+0000 7ff35d15d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:32.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.643+0000 7ff35d15d640 10 monclient: build_initial_monmap 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.643+0000 7ff35d15d640 10 monclient: monmap: 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:32.644649+0000 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:32.644649+0000 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:32.646 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 1 Processor -- start 2024-09-15T01:59:32.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 1 -- start start 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 10 monclient: init 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.644+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff358068f20 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 0x7ff3580696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3580692f0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff358136fe0 0x7ff3581393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-a con 0x7ff358136fe0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580a1670 con 0x7ff358068f20 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3581328d0 con 0x7ff358136fe0 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580a0020 con 0x7ff3580692f0 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35d15d640 10 monclient(hunting): authenticate will time out at 1232.641057s 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff356d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff358068f20 0x7ff3581321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42664/0 (socket says 172.21.15.17:42664) 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff358136fe0 0x7ff3581393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff356d9d640 1 -- 172.21.15.17:0/3249957589 learned_addr learned my addr 172.21.15.17:0/3249957589 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35659c640 1 -- 172.21.15.17:0/3249957589 >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 msgr2=0x7ff3580696f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): get_auth_request con 0x7ff358136fe0 auth_method 0 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35659c640 1 --2- 172.21.15.17:0/3249957589 >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 0x7ff3580696f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:32.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient(hunting): handle_auth_done global_id 104415 payload 274 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 -- 172.21.15.17:0/3249957589 >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 msgr2=0x7ff3580696f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 --2- 172.21.15.17:0/3249957589 >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 0x7ff3580696f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 -- 172.21.15.17:0/3249957589 >> v1:172.21.15.17:6789/0 conn(0x7ff358068f20 legacy=0x7ff3581321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient: _finish_hunting 0 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 monclient: found mon.noname-a 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:32.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.645+0000 7ff35759e640 1 -- 172.21.15.17:0/3249957589 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580a1390 con 0x7ff358136fe0 2024-09-15T01:59:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff35759e640 1 -- 172.21.15.17:0/3249957589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 msgr2=0x7ff3581393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff35759e640 1 --2- 172.21.15.17:0/3249957589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff3581393c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff350009e60 tx=0x7ff350009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86335951918edb42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 1 -- 172.21.15.17:0/3249957589 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35002aea0 con 0x7ff358136fe0 2024-09-15T01:59:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: dump: 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:32.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: _finish_auth 0 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 1 -- 172.21.15.17:0/3249957589 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35002d050 con 0x7ff358136fe0 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 1 -- 172.21.15.17:0/3249957589 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350007a70 con 0x7ff358136fe0 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff35d15d640 5 monclient: authenticate success, global_id 104415 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff35d15d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff35d15d640 4 set_mon_vals no callback set 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.648+0000 7ff355d9b640 10 monclient: dump: 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 10 monclient: shutdown 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 -- 172.21.15.17:0/3249957589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 msgr2=0x7ff3581393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 --2- 172.21.15.17:0/3249957589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff3581393c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff350009e60 tx=0x7ff350009780 comp rx=0 tx=0).stop 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 -- 172.21.15.17:0/3249957589 shutdown_connections 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 --2- 172.21.15.17:0/3249957589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff3581393c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 --2- 172.21.15.17:0/3249957589 >> v2:172.21.15.67:3300/0 conn(0x7ff3580692f0 0x7ff3580696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 -- 172.21.15.17:0/3249957589 >> 172.21.15.17:0/3249957589 conn(0x7ff358070fb0 msgr2=0x7ff3580a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 -- 172.21.15.17:0/3249957589 shutdown_connections 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.649+0000 7ff35d15d640 1 -- 172.21.15.17:0/3249957589 wait complete. 2024-09-15T01:59:32.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.650+0000 7ff35d15d640 10 monclient: build_initial_monmap 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.650+0000 7ff35d15d640 10 monclient: monmap: 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.650+0000 7ff35d15d640 1 Processor -- start 2024-09-15T01:59:32.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.650+0000 7ff35d15d640 1 -- start start 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.650+0000 7ff35d15d640 10 monclient: init 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-c con 0x7ff358068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 0x7ff35809a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3580692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): picked mon.noname-a con 0x7ff358136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35811c600 con 0x7ff358136fe0 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:32.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580a0780 con 0x7ff3580692f0 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35811cb30 con 0x7ff358068f20 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35d15d640 10 monclient(hunting): authenticate will time out at 1232.647181s 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35659c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 msgr2=0x7ff35809a1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:32.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 0x7ff35809a1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51402/0 (socket says 172.21.15.17:51402) 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 1 -- 172.21.15.17:0/288014263 learned_addr learned my addr 172.21.15.17:0/288014263 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): get_auth_request con 0x7ff358136fe0 auth_method 0 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37004/0 (socket says 172.21.15.17:37004) 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): get_auth_request con 0x7ff358068f20 auth_method 0 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:32.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient(hunting): handle_auth_done global_id 94574 payload 824 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 -- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 msgr2=0x7ff35809a1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 0x7ff35809a1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 -- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 msgr2=0x7ff35809a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 monclient: found mon.noname-c 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:32.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff356d9d640 1 -- 172.21.15.17:0/288014263 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350009a20 con 0x7ff358068f20 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.651+0000 7ff35759e640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.653+0000 7ff356d9d640 1 -- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 msgr2=0x7ff358099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.653+0000 7ff356d9d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c490 tx=0x7ff348007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348005cb0 con 0x7ff358068f20 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: dump: 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.667 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: _finish_auth 0 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 5 monclient: authenticate success, global_id 94574 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 10 monclient: _renew_subs 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35809b5b0 con 0x7ff358068f20 2024-09-15T01:59:32.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348005e30 con 0x7ff358068f20 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348007320 con 0x7ff358068f20 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff33effd640 10 monclient: dump: 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 10 monclient: _renew_subs 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35811b8f0 con 0x7ff358068f20 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.654+0000 7ff354d99640 4 set_mon_vals no callback set 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff34801b3b0 con 0x7ff358068f20 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35d15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35d15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35d15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3580724a0 con 0x7ff358068f20 2024-09-15T01:59:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff33effd640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 0x7ff34403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7ff34804d1e0 con 0x7ff358068f20 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35659c640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 0x7ff34403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.655+0000 7ff35659c640 10 monclient: get_auth_request con 0x7ff3440386f0 auth_method 0 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.658+0000 7ff35659c640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 0x7ff34403aba0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0038f0 tx=0x7ff34c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.659+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff348018680 con 0x7ff358068f20 2024-09-15T01:59:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.659+0000 7ff33effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:32.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.659+0000 7ff33effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:32.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff35d15d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff35d15d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff35d15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff35811cea0 con 0x7ff358068f20 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff33effd640 1 -- 172.21.15.17:0/288014263 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7ff348017070 con 0x7ff358068f20 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff33effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.884+0000 7ff33effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:32.887 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 msgr2=0x7ff34403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 0x7ff34403aba0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0038f0 tx=0x7ff34c006d80 comp rx=0 tx=0).stop 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 10 monclient: shutdown 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 msgr2=0x7ff358099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c490 tx=0x7ff348007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 shutdown_connections 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff3440386f0 0x7ff34403aba0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff358136fe0 0x7ff35809a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3580692f0 0x7ff35809a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 --2- 172.21.15.17:0/288014263 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff358068f20 0x7ff358099cb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:32.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 >> 172.21.15.17:0/288014263 conn(0x7ff358070fb0 msgr2=0x7ff3580a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:32.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 shutdown_connections 2024-09-15T01:59:32.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:32.886+0000 7ff35d15d640 1 -- 172.21.15.17:0/288014263 wait complete. 2024-09-15T01:59:33.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:33.005+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:33.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:33.005+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:34.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:34.005+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:34.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:34.005+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:35.005+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:35.005+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:35.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:36.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:36.005+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:36.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:36.005+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.007+0000 7feea775d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.007+0000 7feea775d640 10 monclient: build_initial_monmap 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.007+0000 7feea775d640 10 monclient: monmap: 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:36.008272+0000 2024-09-15T01:59:36.009 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:36.008272+0000 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:36.010 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 Processor -- start 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 -- start start 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient: init 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 0x7feea01321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): picked mon.noname-c con 0x7feea0068f20 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7feea00692f0 0x7feea00696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): picked mon.noname-a con 0x7feea00692f0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:36.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feea012e900 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea00a1670 con 0x7feea012e900 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea00a0020 con 0x7feea00692f0 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea00a1390 con 0x7feea0068f20 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea775d640 10 monclient(hunting): authenticate will time out at 1236.004853s 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.008+0000 7feea4cf8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7feea00692f0 0x7feea00696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:36.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea54f9640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 msgr2=0x7feea01321c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea54f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 0x7feea01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea5cfa640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7feea012e900 legacy=0x7feea012ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7feea00692f0 0x7feea00696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42526/0 (socket says 172.21.15.17:42526) 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 -- 172.21.15.17:0/2585390494 learned_addr learned my addr 172.21.15.17:0/2585390494 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): get_auth_request con 0x7feea00692f0 auth_method 0 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7fee93fff640 1 -- 172.21.15.17:0/2585390494 DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:36.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient(hunting): handle_auth_done global_id 104421 payload 274 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 -- 172.21.15.17:0/2585390494 >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 msgr2=0x7feea01321c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 --2- 172.21.15.17:0/2585390494 >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 0x7feea01321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 -- 172.21.15.17:0/2585390494 >> v1:172.21.15.67:6789/0 conn(0x7feea012e900 legacy=0x7feea012ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 monclient: found mon.noname-a 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.009+0000 7feea4cf8640 1 -- 172.21.15.17:0/2585390494 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0069c30 con 0x7feea00692f0 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea4cf8640 1 -- 172.21.15.17:0/2585390494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 msgr2=0x7feea00696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea4cf8640 1 --2- 172.21.15.17:0/2585390494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea00696f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feea00687a0 tx=0x7fee940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47881b4f5352a1ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 1 -- 172.21.15.17:0/2585390494 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9402ae70 con 0x7feea00692f0 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: dump: 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 5 monclient: authenticate success, global_id 104421 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 1 -- 172.21.15.17:0/2585390494 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9402f4b0 con 0x7feea00692f0 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 1 -- 172.21.15.17:0/2585390494 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9402f850 con 0x7feea00692f0 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:36.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 4 set_mon_vals no callback set 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7fee93fff640 10 monclient: dump: 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 10 monclient: shutdown 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 -- 172.21.15.17:0/2585390494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 msgr2=0x7feea00696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 --2- 172.21.15.17:0/2585390494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea00696f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feea00687a0 tx=0x7fee940091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 -- 172.21.15.17:0/2585390494 shutdown_connections 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 --2- 172.21.15.17:0/2585390494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea00696f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 --2- 172.21.15.17:0/2585390494 >> v2:172.21.15.67:3300/0 conn(0x7feea0068f20 0x7feea01321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.012+0000 7feea775d640 1 -- 172.21.15.17:0/2585390494 >> 172.21.15.17:0/2585390494 conn(0x7feea0070fb0 msgr2=0x7feea00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 1 -- 172.21.15.17:0/2585390494 shutdown_connections 2024-09-15T01:59:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 1 -- 172.21.15.17:0/2585390494 wait complete. 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 10 monclient: build_initial_monmap 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 10 monclient: monmap: 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 1 Processor -- start 2024-09-15T01:59:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 1 -- start start 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.013+0000 7feea775d640 10 monclient: init 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feea0068f20 0x7feea01413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): picked mon.noname-c con 0x7feea0068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): picked mon.noname-a con 0x7feea00692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 0x7feea012e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): picked mon.noname-b con 0x7feea012e900 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea01422d0 con 0x7feea00692f0 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:36.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0130d90 con 0x7feea012e900 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea009a820 con 0x7feea0068f20 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea775d640 10 monclient(hunting): authenticate will time out at 1236.010620s 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42532/0 (socket says 172.21.15.17:42532) 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feea0068f20 0x7feea01413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 1 -- 172.21.15.17:0/3570820558 learned_addr learned my addr 172.21.15.17:0/3570820558 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): get_auth_request con 0x7feea00692f0 auth_method 0 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:36.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea5cfa640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 msgr2=0x7feea012e350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea5cfa640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 0x7feea012e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): get_auth_request con 0x7feea0068f20 auth_method 0 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea4cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.014+0000 7feea54f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 10 monclient(hunting): handle_auth_done global_id 104424 payload 824 2024-09-15T01:59:36.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feea0068f20 msgr2=0x7feea01413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feea0068f20 0x7feea01413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 msgr2=0x7feea012e350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 0x7feea012e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 monclient: found mon.noname-a 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.015+0000 7feea4cf8640 1 -- 172.21.15.17:0/3570820558 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94009450 con 0x7feea00692f0 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea4cf8640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 msgr2=0x7feea012de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea4cf8640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee94035040 tx=0x7fee94032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee940329b0 con 0x7feea00692f0 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:36.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: dump: 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:36.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 5 monclient: authenticate success, global_id 104424 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 10 monclient: _renew_subs 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94032b30 con 0x7feea00692f0 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea009b780 con 0x7feea00692f0 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9403b400 con 0x7feea00692f0 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee917fa640 10 monclient: dump: 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:36.032 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7fee937fe640 4 set_mon_vals no callback set 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 10 monclient: _renew_subs 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.017+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea0099c50 con 0x7feea00692f0 2024-09-15T01:59:36.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fee94043030 con 0x7feea00692f0 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7feea775d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7feea775d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7feea775d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea00a07c0 con 0x7feea00692f0 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7fee917fa640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 0x7fee8403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.018+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7fee94041070 con 0x7feea00692f0 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.019+0000 7feea54f9640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 0x7fee8403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:36.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.019+0000 7feea54f9640 10 monclient: get_auth_request con 0x7fee840386f0 auth_method 0 2024-09-15T01:59:36.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.021+0000 7feea54f9640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 0x7fee8403aba0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0038f0 tx=0x7fee9c0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:36.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.023+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feea00a07c0 con 0x7feea00692f0 2024-09-15T01:59:36.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.023+0000 7fee917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:36.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.023+0000 7fee917fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7feea775d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7feea775d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7feea775d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feea00a07c0 con 0x7feea00692f0 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7fee917fa640 1 -- 172.21.15.17:0/3570820558 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7feea00a07c0 con 0x7feea00692f0 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7fee917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:36.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.253+0000 7fee917fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:36.259 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:36.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 msgr2=0x7fee8403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 0x7fee8403aba0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0038f0 tx=0x7fee9c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 10 monclient: shutdown 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 msgr2=0x7feea012de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee94035040 tx=0x7fee94032000 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 shutdown_connections 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fee840386f0 0x7fee8403aba0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7feea00692f0 0x7feea012de10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea012e900 0x7feea012e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 --2- 172.21.15.17:0/3570820558 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7feea0068f20 0x7feea01413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 >> 172.21.15.17:0/3570820558 conn(0x7feea0070fb0 msgr2=0x7feea01311e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:36.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 shutdown_connections 2024-09-15T01:59:36.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:36.255+0000 7feea775d640 1 -- 172.21.15.17:0/3570820558 wait complete. 2024-09-15T01:59:37.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:37.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:37.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:37.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:38.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:38.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:39.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:39.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:39.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.375+0000 7f7fedd5d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.375+0000 7f7fedd5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.375+0000 7f7fedd5d640 10 monclient: monmap: 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:39.376124+0000 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:39.376124+0000 2024-09-15T01:59:39.377 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:39.378 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 Processor -- start 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 -- start start 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient: init 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe80a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7fe80980a0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:39.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe8136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe80a01e0 con 0x7f7fe80980a0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80a1580 con 0x7f7fe8136dd0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80a5c80 con 0x7f7fe80a58b0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fedd5d640 10 monclient(hunting): authenticate will time out at 1239.372810s 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fe77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 msgr2=0x7f7fe8132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fdeffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f7fe80980a0 legacy=0x7f7fe80984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fe7fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fe77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fe7fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42534/0 (socket says 172.21.15.17:42534) 2024-09-15T01:59:39.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.376+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 learned_addr learned my addr 172.21.15.17:0/3650397137 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request con 0x7f7fe8136dd0 auth_method 0 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:39.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.377+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:39.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.577+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 msgr2=0x7f7fe8132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:39.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.577+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.67:6789/0 conn(0x7f7fe80980a0 legacy=0x7f7fe80984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:39.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.577+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:59:39.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:39.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request con 0x7f7fe8136dd0 auth_method 0 2024-09-15T01:59:39.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:39.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.578+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:59:39.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.978+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 msgr2=0x7f7fe8132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:39.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.978+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.67:6789/0 conn(0x7f7fe80980a0 legacy=0x7f7fe80984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:39.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.978+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request con 0x7f7fe8136dd0 auth_method 0 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:39.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:39.979+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:59:40.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:40.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:40.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:40.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:40.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.779+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 msgr2=0x7f7fe8132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:40.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.779+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.67:6789/0 conn(0x7f7fe80980a0 legacy=0x7f7fe80984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:40.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.779+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request con 0x7f7fe8136dd0 auth_method 0 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:40.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:40.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:40.780+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:59:41.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:41.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:41.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:41.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:42.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:42.006+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:42.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:42.006+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:42.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:42.006+0000 7f48797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:42.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 10 monclient(hunting): tick 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 monclient(hunting): continuing hunt 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 msgr2=0x7f7fe8132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.67:3300/0 conn(0x7f7fe80a58b0 0x7f7fe8132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 -- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 msgr2=0x7f7fe81391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 --2- 172.21.15.17:0/3650397137 >> v2:172.21.15.17:3300/0 conn(0x7f7fe8136dd0 0x7f7fe81391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.376+0000 7f7fe5ffb640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.67:6789/0 conn(0x7f7fe80980a0 legacy=0x7f7fe80984a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 10 monclient: _add_conns ranks=[2,0] 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc8003650 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fc80060b0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:42.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 1 -- 172.21.15.17:0/3650397137 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe8098bb0 con 0x7f7fc80060b0 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 1 -- 172.21.15.17:0/3650397137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80a1580 con 0x7f7fc8003650 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe5ffb640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request con 0x7f7fc8003650 auth_method 0 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.377+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:42.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.379+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.17:6789/0 conn(0x7f7fc80060b0 legacy=0x7f7fc8008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request con 0x7f7fc8003650 auth_method 0 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.578+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T01:59:42.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request con 0x7f7fc8003650 auth_method 0 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:42.979+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T01:59:43.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:43.007+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:43.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:43.007+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request con 0x7f7fc8003650 auth_method 0 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T01:59:43.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.780+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T01:59:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.813+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667194603 0 0) 0x7f7fe8098bb0 con 0x7f7fc80060b0 2024-09-15T01:59:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.813+0000 7f7fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.813+0000 7f7fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.813+0000 7f7fe6ffd640 10 monclient(hunting): my global_id is 104427 2024-09-15T01:59:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.813+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd40034e0 con 0x7f7fc80060b0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525856305 0 0) 0x7f7fd40034e0 con 0x7f7fc80060b0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 msgr2=0x7f7fc8005a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fc8003650 0x7f7fc8005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe80a01e0 con 0x7f7fc80060b0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f7fd0004050 con 0x7f7fc80060b0 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fedd5d640 5 monclient: authenticate success, global_id 104427 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fedd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: dump: 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:43.820 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:59:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 >> v1:172.21.15.17:6789/0 conn(0x7f7fc80060b0 legacy=0x7f7fc8008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 reap_dead start 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): picked mon.a con 0x7f7fd40039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 0x7f7fd4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): picked mon.c con 0x7f7fd4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 0x7f7fd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): picked mon.b con 0x7f7fd4005180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80a1580 con 0x7f7fd40039d0 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd400c120 con 0x7f7fd4005180 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd400c2e0 con 0x7f7fd4004780 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fedd5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 0x7f7fd4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.815+0000 7f7fe7fff640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 msgr2=0x7f7fd400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): get_auth_request con 0x7f7fd4004780 auth_method 0 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 0x7f7fd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient(hunting): handle_auth_done global_id 104427 payload 293 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 msgr2=0x7f7fd400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 0x7f7fd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 msgr2=0x7f7fd4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient: _finish_hunting 0 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 monclient: found mon.c 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40034e0 con 0x7f7fd4004780 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.816+0000 7f7fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.819+0000 7f7fdeffd640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 msgr2=0x7f7fd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.819+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 0x7f7fd4004bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00038f0 tx=0x7f7fd0007960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40ce682d5afa6d14 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0007150 con 0x7f7fd4004780 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: dump: 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd00072d0 con 0x7f7fd4004780 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 1 -- 172.21.15.17:0/3650397137 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0007440 con 0x7f7fd4004780 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fedd5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fedd5d640 4 set_mon_vals no callback set 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.820+0000 7f7fe6ffd640 10 monclient: dump: 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 10 monclient: shutdown 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 -- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 msgr2=0x7f7fd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 0x7f7fd4004bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00038f0 tx=0x7f7fd0007960 comp rx=0 tx=0).stop 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 -- 172.21.15.17:0/3650397137 shutdown_connections 2024-09-15T01:59:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4005180 0x7f7fd400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4004780 0x7f7fd4004bf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/3650397137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fd4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 -- 172.21.15.17:0/3650397137 >> 172.21.15.17:0/3650397137 conn(0x7f7fe8070ed0 msgr2=0x7f7fe80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 -- 172.21.15.17:0/3650397137 shutdown_connections 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.821+0000 7f7fedd5d640 1 -- 172.21.15.17:0/3650397137 wait complete. 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.822+0000 7f7fedd5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.822+0000 7f7fedd5d640 10 monclient: monmap: 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.822+0000 7f7fedd5d640 1 Processor -- start 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 -- start start 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient: init 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fd40039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 0x7f7fe8141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fd4004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fd4005180 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80a49f0 con 0x7f7fd40039d0 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe811de50 con 0x7f7fd4004780 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe811ca00 con 0x7f7fd4005180 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.823+0000 7f7fedd5d640 10 monclient(hunting): authenticate will time out at 1243.819978s 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fdeffd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 msgr2=0x7f7fe8141780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 0x7f7fe8141780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:43.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33396/0 (socket says 172.21.15.17:33396) 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34538/0 (socket says 172.21.15.17:34538) 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 -- 172.21.15.17:0/2859377063 learned_addr learned my addr 172.21.15.17:0/2859377063 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request con 0x7f7fd40039d0 auth_method 0 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request con 0x7f7fd4005180 auth_method 0 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:43.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient(hunting): handle_auth_done global_id 104430 payload 824 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 -- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 msgr2=0x7f7fe8141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 -- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 msgr2=0x7f7fe8141780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 0x7f7fe8141780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe77fe640 1 -- 172.21.15.17:0/2859377063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe811cce0 con 0x7f7fd40039d0 2024-09-15T01:59:43.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.824+0000 7f7fe7fff640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fe77fe640 1 -- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 msgr2=0x7f7fe8141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fe77fe640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8003c20 tx=0x7f7fd800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800e030 con 0x7f7fd40039d0 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: dump: 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.839 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: _finish_auth 0 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 5 monclient: authenticate success, global_id 104430 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 10 monclient: _renew_subs 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8007d90 con 0x7f7fd40039d0 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe811cfc0 con 0x7f7fd40039d0 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80176c0 con 0x7f7fd40039d0 2024-09-15T01:59:43.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fdffff640 10 monclient: dump: 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 10 monclient: _renew_subs 2024-09-15T01:59:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fe67fc640 4 set_mon_vals no callback set 2024-09-15T01:59:43.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:43.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.827+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe811baa0 con 0x7f7fd40039d0 2024-09-15T01:59:43.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.828+0000 7f7fedd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:43.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.828+0000 7f7fedd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:43.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.828+0000 7f7fedd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:43.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.828+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe8098dc0 con 0x7f7fd40039d0 2024-09-15T01:59:43.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.828+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7fd8017b30 con 0x7f7fd40039d0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.829+0000 7f7fdffff640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 0x7f7fc803b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.829+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7f7fd804e800 con 0x7f7fd40039d0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.829+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 0x7f7fc803b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.829+0000 7f7fdeffd640 10 monclient: get_auth_request con 0x7f7fc8038e00 auth_method 0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.832+0000 7f7fdeffd640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 0x7f7fc803b2b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7fd002f000 tx=0x7f7fd0034000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.833+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7fd80132c0 con 0x7f7fd40039d0 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.833+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:43.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:43.833+0000 7f7fdffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:44.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:44.007+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:44.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:44.007+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:44.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fedd5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T01:59:44.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fedd5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:44.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fedd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:44.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7fe80724a0 con 0x7f7fd40039d0 2024-09-15T01:59:44.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fdffff640 1 -- 172.21.15.17:0/2859377063 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f7fd80142c0 con 0x7f7fd40039d0 2024-09-15T01:59:44.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T01:59:44.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.061+0000 7f7fdffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:44.064 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T01:59:44.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 msgr2=0x7f7fc803b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 0x7f7fc803b2b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7fd002f000 tx=0x7f7fd0034000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 10 monclient: shutdown 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 msgr2=0x7f7fe8141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8003c20 tx=0x7f7fd800b040 comp rx=0 tx=0).stop 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 shutdown_connections 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7fc8038e00 0x7f7fc803b2b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7fd4005180 0x7f7fe8141cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fd4004780 0x7f7fe8141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 --2- 172.21.15.17:0/2859377063 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7fd40039d0 0x7f7fe8141240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 >> 172.21.15.17:0/2859377063 conn(0x7f7fe8070ed0 msgr2=0x7f7fe8097820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 shutdown_connections 2024-09-15T01:59:44.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.063+0000 7f7fedd5d640 1 -- 172.21.15.17:0/2859377063 wait complete. 2024-09-15T01:59:44.076 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T01:59:44.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T01:59:44.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 10 monclient: build_initial_monmap 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 10 monclient: monmap: 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:44.172969+0000 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:44.172969+0000 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:44.174 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:44.175 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 1 Processor -- start 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 1 -- start start 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.172+0000 7ff52b31d640 10 monclient: init 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 0x7ff524132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5240a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5240980a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff524136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:44.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5240a01e0 con 0x7ff5240980a0 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5240a1580 con 0x7ff524136dd0 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240a5c80 con 0x7ff5240a58b0 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff52b31d640 10 monclient(hunting): authenticate will time out at 1244.169428s 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 0x7ff524132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5288b8640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff5240980a0 0x7ff524098470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39216/0 (socket says 172.21.15.17:39216) 2024-09-15T01:59:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5298ba640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7ff524136dd0 legacy=0x7ff5241391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5288b8640 1 -- 172.21.15.17:0/2777171221 learned_addr learned my addr 172.21.15.17:0/2777171221 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): get_auth_request con 0x7ff5240a58b0 auth_method 0 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient(hunting): handle_auth_done global_id 94583 payload 274 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 1 -- 172.21.15.17:0/2777171221 >> v1:172.21.15.67:6789/0 conn(0x7ff524136dd0 legacy=0x7ff5241391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 1 -- 172.21.15.17:0/2777171221 >> v1:172.21.15.17:6789/0 conn(0x7ff5240980a0 legacy=0x7ff524098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient: _finish_hunting 0 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 1 monclient: found mon.noname-b 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.173+0000 7ff5290b9640 1 -- 172.21.15.17:0/2777171221 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5241326c0 con 0x7ff5240a58b0 2024-09-15T01:59:44.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.176+0000 7ff5290b9640 1 -- 172.21.15.17:0/2777171221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 msgr2=0x7ff524132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.176+0000 7ff5290b9640 1 --2- 172.21.15.17:0/2777171221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 0x7ff524132040 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff514009e90 tx=0x7ff514009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310b492ef3b6ea31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 1 -- 172.21.15.17:0/2777171221 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514007840 con 0x7ff5240a58b0 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: dump: 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.187 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 5 monclient: authenticate success, global_id 94583 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 1 -- 172.21.15.17:0/2777171221 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51402d050 con 0x7ff5240a58b0 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:44.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 1 -- 172.21.15.17:0/2777171221 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514007bd0 con 0x7ff5240a58b0 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 4 set_mon_vals no callback set 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff523fff640 10 monclient: dump: 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.189 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 10 monclient: shutdown 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 -- 172.21.15.17:0/2777171221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 msgr2=0x7ff524132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 --2- 172.21.15.17:0/2777171221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 0x7ff524132040 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff514009e90 tx=0x7ff514009780 comp rx=0 tx=0).stop 2024-09-15T01:59:44.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 -- 172.21.15.17:0/2777171221 shutdown_connections 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 --2- 172.21.15.17:0/2777171221 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240a58b0 0x7ff524132040 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 -- 172.21.15.17:0/2777171221 >> 172.21.15.17:0/2777171221 conn(0x7ff524070ed0 msgr2=0x7ff5240a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 -- 172.21.15.17:0/2777171221 shutdown_connections 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.177+0000 7ff52b31d640 1 -- 172.21.15.17:0/2777171221 wait complete. 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.178+0000 7ff52b31d640 10 monclient: build_initial_monmap 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.178+0000 7ff52b31d640 10 monclient: monmap: 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.178+0000 7ff52b31d640 1 Processor -- start 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.178+0000 7ff52b31d640 1 -- start start 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.178+0000 7ff52b31d640 10 monclient: init 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5240980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5240a58b0 0x7ff52409a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5240a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:44.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff524136dd0 0x7ff52409af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): picked mon.noname-a con 0x7ff524136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:44.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240a3430 con 0x7ff524136dd0 2024-09-15T01:59:44.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5241350f0 con 0x7ff5240a58b0 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff524134e10 con 0x7ff5240980a0 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff52b31d640 10 monclient(hunting): authenticate will time out at 1244.175167s 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34560/0 (socket says 172.21.15.17:34560) 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 -- 172.21.15.17:0/1505057251 learned_addr learned my addr 172.21.15.17:0/1505057251 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:44.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): get_auth_request con 0x7ff5240980a0 auth_method 0 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff524136dd0 0x7ff52409af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5288b8640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5240a58b0 msgr2=0x7ff52409a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): get_auth_request con 0x7ff524136dd0 auth_method 0 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5288b8640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5240a58b0 0x7ff52409a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient(hunting): handle_auth_done global_id 94586 payload 824 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5240a58b0 msgr2=0x7ff52409a9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5240a58b0 0x7ff52409a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff524136dd0 msgr2=0x7ff52409af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff524136dd0 0x7ff52409af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient: _finish_hunting 0 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 monclient: found mon.noname-c 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5290b9640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff514009a20 con 0x7ff5240980a0 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.179+0000 7ff5298ba640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff524136dd0 0x7ff52409af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5290b9640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 msgr2=0x7ff5241357f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5290b9640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff514009ef0 tx=0x7ff514037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514031a20 con 0x7ff5240980a0 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 10 monclient: dump: 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.197 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514040070 con 0x7ff5240980a0 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff52b31d640 5 monclient: authenticate success, global_id 94586 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff52b31d640 10 monclient: _renew_subs 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff52b31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.181+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff524134b30 con 0x7ff5240980a0 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514031d90 con 0x7ff5240980a0 2024-09-15T01:59:44.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 10 monclient: dump: 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5237fe640 4 set_mon_vals no callback set 2024-09-15T01:59:44.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 10 monclient: _renew_subs 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff524135d30 con 0x7ff5240980a0 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff514042030 con 0x7ff5240980a0 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 0x7ff52409b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.182+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff524063b20 con 0x7ff52402aa40 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 0x7ff52409b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient: get_auth_request con 0x7ff52402aa40 auth_method 0 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5217fa640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5100386f0 0x7ff51003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7ff51402ae40 con 0x7ff5240980a0 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 10 monclient(hunting): handle_auth_done global_id 104436 payload 824 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5288b8640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff52402ae10 con 0x7ff52402aa40 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5298ba640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5100386f0 0x7ff51003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.183+0000 7ff5298ba640 10 monclient: get_auth_request con 0x7ff5100386f0 auth_method 0 2024-09-15T01:59:44.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5288b8640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 msgr2=0x7ff52409b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5288b8640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 0x7ff52409b460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5180047d0 tx=0x7ff518007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518005d70 con 0x7ff52402aa40 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 10 monclient: dump: 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.205 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff5180070d0 con 0x7ff52402aa40 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 msgr2=0x7ff52409b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 0x7ff52409b460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5180047d0 tx=0x7ff518007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:44.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 msgr2=0x7ff52409b460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.185+0000 7ff5217fa640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52402aa40 0x7ff52409b460 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5180047d0 tx=0x7ff518007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.186+0000 7ff5298ba640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5100386f0 0x7ff51003aba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff52406a6d0 tx=0x7ff51c006f70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff52b31d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff52b31d640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 0x7ff52405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff52b31d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240724a0 con 0x7ff52405e310 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 0x7ff52405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient: get_auth_request con 0x7ff52405e310 auth_method 0 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.191+0000 7ff5288b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.192+0000 7ff5288b8640 10 monclient(hunting): handle_auth_done global_id 104439 payload 824 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.192+0000 7ff5288b8640 1 -- 172.21.15.17:0/1505057251 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff52409fc90 con 0x7ff52405e310 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5288b8640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 msgr2=0x7ff52405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5288b8640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 0x7ff52405e710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff518014000 tx=0x7ff518012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51801f610 con 0x7ff52405e310 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:44.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 10 monclient: dump: 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4387 (secure 0 0 0) 0x7ff51801e360 con 0x7ff52405e310 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:44.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 msgr2=0x7ff52405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 0x7ff52405e710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff518014000 tx=0x7ff518012000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 msgr2=0x7ff52405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.194+0000 7ff5217fa640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff52405e310 0x7ff52405e710 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff518014000 tx=0x7ff518012000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T01:59:44.210 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 100, 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:59:44.211 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:44.213 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.214 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.214 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.214 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.214 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.214 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:44.216 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:44.217 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.218 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:44.223 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:44.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.226 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:44.227 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5100386f0 msgr2=0x7ff51003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff5100386f0 0x7ff51003aba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff52406a6d0 tx=0x7ff51c006f70 comp rx=0 tx=0).stop 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 10 monclient: shutdown 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 msgr2=0x7ff5241357f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff514009ef0 tx=0x7ff514037000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff5298ba640 1 -- 172.21.15.17:0/1505057251 reap_dead start 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 shutdown_connections 2024-09-15T01:59:44.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 --2- 172.21.15.17:0/1505057251 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5240980a0 0x7ff5241357f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 >> 172.21.15.17:0/1505057251 conn(0x7ff524070ed0 msgr2=0x7ff5241391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:44.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 shutdown_connections 2024-09-15T01:59:44.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.195+0000 7ff52b31d640 1 -- 172.21.15.17:0/1505057251 wait complete. 2024-09-15T01:59:44.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:59:44.230 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T01:59:44.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.326+0000 7f5386d9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.326+0000 7f5386d9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.326+0000 7f5386d9d640 10 monclient: monmap: 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:59:44.327150+0000 2024-09-15T01:59:44.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:59:44.327150+0000 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:44.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 Processor -- start 2024-09-15T01:59:44.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 -- start start 2024-09-15T01:59:44.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient: init 2024-09-15T01:59:44.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 0x7f5380132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5380067090 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f538009ce00 0x7f5380067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f538009ce00 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 0x7f538012e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f53800679a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538009e4c0 con 0x7f538009ce00 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538009cbf0 con 0x7f5380067090 2024-09-15T01:59:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538009e1e0 con 0x7f53800679a0 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5386d9d640 10 monclient(hunting): authenticate will time out at 1244.323674s 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5384b39640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 msgr2=0x7f5380132140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f5384b39640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 0x7f5380132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f538009ce00 0x7f5380067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f538009ce00 0x7f5380067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33418/0 (socket says 172.21.15.17:33418) 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 1 -- 172.21.15.17:0/4196772863 learned_addr learned my addr 172.21.15.17:0/4196772863 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): get_auth_request con 0x7f538009ce00 auth_method 0 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 0x7f538012e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): get_auth_request con 0x7f53800679a0 auth_method 0 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.327+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 10 monclient(hunting): handle_auth_done global_id 104442 payload 274 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 -- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 msgr2=0x7f538012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 0x7f538012e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 -- 172.21.15.17:0/4196772863 >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 msgr2=0x7f5380132140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 --2- 172.21.15.17:0/4196772863 >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 0x7f5380132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 monclient: found mon.noname-a 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f537ffff640 1 -- 172.21.15.17:0/4196772863 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380067e00 con 0x7f538009ce00 2024-09-15T01:59:44.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.328+0000 7f538533a640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 0x7f538012e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.330+0000 7f537ffff640 1 -- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 msgr2=0x7f5380067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.330+0000 7f537ffff640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f5380067460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f538004b6a0 tx=0x7f536c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99aecb0f5e720a2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 1 -- 172.21.15.17:0/4196772863 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c032440 con 0x7f538009ce00 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: dump: 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.342 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 5 monclient: authenticate success, global_id 104442 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 1 -- 172.21.15.17:0/4196772863 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c0325a0 con 0x7f538009ce00 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 4 set_mon_vals no callback set 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 1 -- 172.21.15.17:0/4196772863 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c032940 con 0x7f538009ce00 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f537f7fe640 10 monclient: dump: 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.344 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 10 monclient: shutdown 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 -- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 msgr2=0x7f5380067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f5380067460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f538004b6a0 tx=0x7f536c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 -- 172.21.15.17:0/4196772863 shutdown_connections 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53800679a0 0x7f538012e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 --2- 172.21.15.17:0/4196772863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f5380067460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 --2- 172.21.15.17:0/4196772863 >> v2:172.21.15.67:3300/0 conn(0x7f5380067090 0x7f5380132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 -- 172.21.15.17:0/4196772863 >> 172.21.15.17:0/4196772863 conn(0x7f53800a1ee0 msgr2=0x7f53800a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 -- 172.21.15.17:0/4196772863 shutdown_connections 2024-09-15T01:59:44.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.331+0000 7f5386d9d640 1 -- 172.21.15.17:0/4196772863 wait complete. 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.332+0000 7f5386d9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.332+0000 7f5386d9d640 10 monclient: monmap: 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.332+0000 7f5386d9d640 1 Processor -- start 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.332+0000 7f5386d9d640 1 -- start start 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.332+0000 7f5386d9d640 10 monclient: init 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 0x7f53801309e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5380067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800679a0 0x7f538011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f53800679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f538011aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f538009ce00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538011e150 con 0x7f538009ce00 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538011dde0 con 0x7f53800679a0 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538011bfd0 con 0x7f5380067090 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5386d9d640 10 monclient(hunting): authenticate will time out at 1244.329273s 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 0x7f53801309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 0x7f53801309e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34568/0 (socket says 172.21.15.17:34568) 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f538011aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 -- 172.21.15.17:0/3301754982 learned_addr learned my addr 172.21.15.17:0/3301754982 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T01:59:44.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): get_auth_request con 0x7f5380067090 auth_method 0 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f537ffff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800679a0 msgr2=0x7f538011a570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): get_auth_request con 0x7f538009ce00 auth_method 0 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f537ffff640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800679a0 0x7f538011a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient(hunting): handle_auth_done global_id 94592 payload 824 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800679a0 msgr2=0x7f538011a570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800679a0 0x7f538011a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 msgr2=0x7f538011aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f538011aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient: _finish_hunting 0 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 monclient: found mon.noname-c 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f5384b39640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378009a20 con 0x7f5380067090 2024-09-15T01:59:44.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.333+0000 7f538533a640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f538009ce00 0x7f538011aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.335+0000 7f5384b39640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 msgr2=0x7f53801309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.335+0000 7f5384b39640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 0x7f53801309e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f537400b600 tx=0x7f53740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374003d90 con 0x7f5380067090 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: dump: 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.351 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 5 monclient: authenticate success, global_id 94592 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 10 monclient: _renew_subs 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c009450 con 0x7f5380067090 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 10 monclient: _renew_subs 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538011cba0 con 0x7f5380067090 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374006150 con 0x7f5380067090 2024-09-15T01:59:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374006510 con 0x7f5380067090 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537cff9640 10 monclient: dump: 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.353 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.336+0000 7f537effd640 4 set_mon_vals no callback set 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f5386d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f5386d9d640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 0x7f53800298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5380063870 con 0x7f538011b7b0 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f537401b3b0 con 0x7f5380067090 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 0x7f53800298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient: get_auth_request con 0x7f538011b7b0 auth_method 0 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537cff9640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f53640386f0 0x7f536403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(211..211 src has 148..211) v4 ==== 13688+0+0 (secure 0 0 0) 0x7f537404d1e0 con 0x7f5380067090 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 10 monclient(hunting): handle_auth_done global_id 94595 payload 824 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f537ffff640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f538011dc10 con 0x7f538011b7b0 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f538533a640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f53640386f0 0x7f536403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.337+0000 7f538533a640 10 monclient: get_auth_request con 0x7f53640386f0 auth_method 0 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537ffff640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 msgr2=0x7f53800298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537ffff640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 0x7f53800298c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f536c034000 tx=0x7f536c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c032440 con 0x7f538011b7b0 2024-09-15T01:59:44.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.339+0000 7f537cff9640 10 monclient: dump: 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f536c032920 con 0x7f538011b7b0 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 msgr2=0x7f53800298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 0x7f53800298c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f536c034000 tx=0x7f536c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 msgr2=0x7f53800298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f537cff9640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f538011b7b0 0x7f53800298c0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f536c034000 tx=0x7f536c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.340+0000 7f538533a640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f53640386f0 0x7f536403aba0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f537800eb00 tx=0x7f537800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.345+0000 7f5386d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.345+0000 7f5386d9d640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 0x7f5380029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.345+0000 7f5386d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.345+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53800724a0 con 0x7f53801364e0 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 0x7f5380029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient: get_auth_request con 0x7f53801364e0 auth_method 0 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:44.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 10 monclient(hunting): handle_auth_done global_id 94598 payload 824 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.346+0000 7f537ffff640 1 -- 172.21.15.17:0/3301754982 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f538009a990 con 0x7f53801364e0 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537ffff640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 msgr2=0x7f5380029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537ffff640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 0x7f5380029e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f536c009210 tx=0x7f536c03c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c006e00 con 0x7f53801364e0 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 10 monclient: dump: 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T01:59:44.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4725 (secure 0 0 0) 0x7f536c031eb0 con 0x7f53801364e0 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T01:59:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 msgr2=0x7f5380029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 0x7f5380029e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f536c009210 tx=0x7f536c03c000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 msgr2=0x7f5380029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.348+0000 7f537cff9640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53801364e0 0x7f5380029e00 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f536c009210 tx=0x7f536c03c000 comp rx=0 tx=0).stop 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 100, 2024-09-15T01:59:44.362 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.363 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:44.365 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T01:59:44.366 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T01:59:44.367 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.369 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T01:59:44.370 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T01:59:44.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T01:59:44.372 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T01:59:44.375 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T01:59:44.376 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T01:59:44.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f53640386f0 msgr2=0x7f536403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f53640386f0 0x7f536403aba0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f537800eb00 tx=0x7f537800b040 comp rx=0 tx=0).stop 2024-09-15T01:59:44.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 10 monclient: shutdown 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 msgr2=0x7f53801309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 --2- 172.21.15.17:0/3301754982 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5380067090 0x7f53801309e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f537400b600 tx=0x7f53740091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f538533a640 1 -- 172.21.15.17:0/3301754982 reap_dead start 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f538533a640 1 -- 172.21.15.17:0/3301754982 reap_dead start 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 shutdown_connections 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 >> 172.21.15.17:0/3301754982 conn(0x7f53800a1ee0 msgr2=0x7f5380098760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 shutdown_connections 2024-09-15T01:59:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T01:59:44.349+0000 7f5386d9d640 1 -- 172.21.15.17:0/3301754982 wait complete. 2024-09-15T01:59:44.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T01:59:44.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T01:59:45.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.007+0000 7f48797fa640 10 monclient: tick 2024-09-15T01:59:45.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.007+0000 7f48797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:45.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.404+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v212) v1 ==== 168+0+0 (secure 0 0 0) 0x7f487c021070 con 0x7f488c09f2c0 2024-09-15T01:59:45.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.404+0000 7f487a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:45.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.404+0000 7f487a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-15T01:59:45.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.508+0000 7f4890f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:45.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.508+0000 7f4890f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:45.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.508+0000 7f4890f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:45.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.508+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f488c06f0b0 con 0x7f488c09f2c0 2024-09-15T01:59:45.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.730+0000 7f487a7fc640 1 -- 172.21.15.67:0/2679147918 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v212) v1 ==== 147+0+0 (secure 0 0 0) 0x7f487c01b3b0 con 0x7f488c09f2c0 2024-09-15T01:59:45.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.730+0000 7f487a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:45.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.730+0000 7f487a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-15T01:59:45.732 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h27' already exists 2024-09-15T01:59:45.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 msgr2=0x7f486c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 0x7f486c03abf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4880031000 tx=0x7f4880032000 comp rx=0 tx=0).stop 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 10 monclient: shutdown 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 msgr2=0x7f488c09aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f487c00ea50 tx=0x7f487c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 shutdown_connections 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f486c038740 0x7f486c03abf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f488c0a0a60 0x7f488c130ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c131aa0 0x7f488c11b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 --2- 172.21.15.67:0/2679147918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f488c09f2c0 0x7f488c09aea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 >> 172.21.15.67:0/2679147918 conn(0x7f488c071390 msgr2=0x7f488c134db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:45.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 shutdown_connections 2024-09-15T01:59:45.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.732+0000 7f4890f9d640 1 -- 172.21.15.67:0/2679147918 wait complete. 2024-09-15T01:59:45.745 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:45.745 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-15T01:59:45.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.824+0000 7f2ef6b1d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.824+0000 7f2ef6b1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.824+0000 7f2ef6b1d640 10 monclient: monmap: 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:45.826074+0000 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:45.826074+0000 2024-09-15T01:59:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.825+0000 7f2ef6b1d640 1 Processor -- start 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.825+0000 7f2ef6b1d640 1 -- start start 2024-09-15T01:59:45.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.825+0000 7f2ef6b1d640 10 monclient: init 2024-09-15T01:59:45.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:45.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T01:59:45.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 0x7f2ef0132240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef0131e70 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:45.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef0098420 0x7f2ef0098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef0098420 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ef0068e70 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef00a13e0 con 0x7f2ef0068e70 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef00a0440 con 0x7f2ef0098420 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef00a01e0 con 0x7f2ef0131e70 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef6b1d640 10 monclient(hunting): authenticate will time out at 1218.295322s 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef48b9640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 msgr2=0x7f2ef0132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:45.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef48b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 0x7f2ef0132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef0098420 0x7f2ef0098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef0098420 0x7f2ef0098850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:37932/0 (socket says 172.21.15.67:37932) 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 -- 172.21.15.67:0/1309572040 learned_addr learned my addr 172.21.15.67:0/1309572040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2ef50ba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2ef0068e70 0x7f2ef00692f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:50610/0 (socket says 172.21.15.67:50610) 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): get_auth_request con 0x7f2ef0098420 auth_method 0 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:45.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient(hunting): handle_auth_done global_id 104448 payload 274 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 -- 172.21.15.67:0/1309572040 >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 msgr2=0x7f2ef0132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 --2- 172.21.15.67:0/1309572040 >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 0x7f2ef0132240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 -- 172.21.15.67:0/1309572040 >> v1:172.21.15.17:6789/0 conn(0x7f2ef0068e70 legacy=0x7f2ef00692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 monclient: found mon.noname-a 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.826+0000 7f2eeffff640 1 -- 172.21.15.67:0/1309572040 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef00a19b0 con 0x7f2ef0098420 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.829+0000 7f2eeffff640 1 -- 172.21.15.67:0/1309572040 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0098420 msgr2=0x7f2ef0098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.829+0000 7f2eeffff640 1 --2- 172.21.15.67:0/1309572040 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0098420 0x7f2ef0098850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00a6250 tx=0x7f2ee0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ed11bc849bf7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:45.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1309572040 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee002aea0 con 0x7f2ef0098420 2024-09-15T01:59:45.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:45.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: dump: 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.842 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1309572040 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee002d050 con 0x7f2ef0098420 2024-09-15T01:59:45.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1309572040 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0007840 con 0x7f2ef0098420 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2eef7fe640 10 monclient: dump: 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.844 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2ef6b1d640 5 monclient: authenticate success, global_id 104448 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2ef6b1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.830+0000 7f2ef6b1d640 4 set_mon_vals no callback set 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 10 monclient: shutdown 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/1309572040 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0098420 msgr2=0x7f2ef0098850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/1309572040 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0098420 0x7f2ef0098850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00a6250 tx=0x7f2ee0009840 comp rx=0 tx=0).stop 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/1309572040 shutdown_connections 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/1309572040 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0098420 0x7f2ef0098850 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/1309572040 >> v2:172.21.15.67:3300/0 conn(0x7f2ef0131e70 0x7f2ef0132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/1309572040 >> 172.21.15.67:0/1309572040 conn(0x7f2ef00a5c10 msgr2=0x7f2ef00a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/1309572040 shutdown_connections 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/1309572040 wait complete. 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 10 monclient: monmap: 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.846 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.831+0000 7f2ef6b1d640 1 Processor -- start 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 -- start start 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient: init 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef0068e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 0x7f2ef009c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef0098420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef0131e70 0x7f2ef009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef0131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0134f10 con 0x7f2ef0068e70 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00a0d80 con 0x7f2ef0098420 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef011c1d0 con 0x7f2ef0131e70 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef6b1d640 10 monclient(hunting): authenticate will time out at 1218.301687s 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2eeffff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 msgr2=0x7f2ef009c620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2eeffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 0x7f2ef009c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef50ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef0131e70 0x7f2ef009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:45.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:37944/0 (socket says 172.21.15.67:37944) 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 1 -- 172.21.15.67:0/3529029523 learned_addr learned my addr 172.21.15.67:0/3529029523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): get_auth_request con 0x7f2ef0068e70 auth_method 0 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef50ba640 10 monclient(hunting): get_auth_request con 0x7f2ef0131e70 auth_method 0 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef50ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef50ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef50ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.832+0000 7f2ef48b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:45.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef50ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef50ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef50ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 10 monclient(hunting): handle_auth_done global_id 104451 payload 824 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 -- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef0131e70 msgr2=0x7f2ef009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef0131e70 0x7f2ef009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 -- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 msgr2=0x7f2ef009c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 0x7f2ef009c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 10 monclient: _finish_hunting 0 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 monclient: found mon.noname-a 2024-09-15T01:59:45.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.833+0000 7f2ef48b9640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0009ae0 con 0x7f2ef0068e70 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef48b9640 1 -- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 msgr2=0x7f2ef01419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef48b9640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800b5c0 tx=0x7f2ed8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8003d90 con 0x7f2ef0068e70 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 10 monclient: dump: 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.851 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 5 monclient: authenticate success, global_id 104451 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 10 monclient: _renew_subs 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef009df10 con 0x7f2ef0068e70 2024-09-15T01:59:45.852 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 10 monclient: _renew_subs 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8006150 con 0x7f2ef0068e70 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef011b8a0 con 0x7f2ef0068e70 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.835+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8006550 con 0x7f2ef0068e70 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2eecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2eecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2eecff9640 10 monclient: dump: 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:45.853 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2eeeffd640 4 set_mon_vals no callback set 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2ef6b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2ed80103c0 con 0x7f2ef0068e70 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2ef6b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2ef6b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.836+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef00639e0 con 0x7f2ef0068e70 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.837+0000 7f2eecff9640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 0x7f2ed003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.837+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(212..212 src has 148..212) v4 ==== 13901+0+0 (secure 0 0 0) 0x7f2ed804d620 con 0x7f2ef0068e70 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.837+0000 7f2eeffff640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 0x7f2ed003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.837+0000 7f2eeffff640 10 monclient: get_auth_request con 0x7f2ed00386f0 auth_method 0 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.839+0000 7f2eeffff640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 0x7f2ed003aba0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0030000 tx=0x7f2ee0036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.841+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2ed800ed30 con 0x7f2ef0068e70 2024-09-15T01:59:45.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.841+0000 7f2eecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:45.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:45.841+0000 7f2eecff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.008+0000 7f2ef6b1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.008+0000 7f2ef6b1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.008+0000 7f2ef6b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.008+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ef0067a50 con 0x7f2ef0068e70 2024-09-15T01:59:46.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.830+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v213) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2ed8017070 con 0x7f2ef0068e70 2024-09-15T01:59:46.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.830+0000 7f2eecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.830+0000 7f2eecff9640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-15T01:59:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.934+0000 7f2ef6b1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.934+0000 7f2ef6b1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.934+0000 7f2ef6b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.934+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ef006f120 con 0x7f2ef0068e70 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.935+0000 7f2eecff9640 1 -- 172.21.15.67:0/3529029523 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v213) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2ed800e730 con 0x7f2ef0068e70 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.935+0000 7f2eecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.935+0000 7f2eecff9640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-15T01:59:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h28' already exists 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 msgr2=0x7f2ed003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 0x7f2ed003aba0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0030000 tx=0x7f2ee0036000 comp rx=0 tx=0).stop 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 10 monclient: shutdown 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 msgr2=0x7f2ef01419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:46.940 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800b5c0 tx=0x7f2ed8009120 comp rx=0 tx=0).stop 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 shutdown_connections 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ed00386f0 0x7f2ed003aba0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef0131e70 0x7f2ef009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef0098420 0x7f2ef009c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 --2- 172.21.15.67:0/3529029523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef0068e70 0x7f2ef01419e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 >> 172.21.15.67:0/3529029523 conn(0x7f2ef00a5c10 msgr2=0x7f2ef00971e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 shutdown_connections 2024-09-15T01:59:46.941 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:46.937+0000 7f2ef6b1d640 1 -- 172.21.15.67:0/3529029523 wait complete. 2024-09-15T01:59:46.950 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:46.950 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-15T01:59:47.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.026+0000 7f01cb79d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:47.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.026+0000 7f01cb79d640 10 monclient: build_initial_monmap 2024-09-15T01:59:47.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.026+0000 7f01cb79d640 10 monclient: monmap: 2024-09-15T01:59:47.029 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:47.027658+0000 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:47.027658+0000 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:47.031 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:47.031 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:47.031 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:47.031 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 Processor -- start 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 -- start start 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient: init 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T01:59:47.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c4132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-b con 0x7f01c4131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f01c4098260 0x7f01c4098630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-a con 0x7f01c4098260 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01c4098b70 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c4096a70 con 0x7f01c4098b70 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c4098fd0 con 0x7f01c4098260 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c406a6b0 con 0x7f01c4131f90 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.027+0000 7f01cb79d640 10 monclient(hunting): authenticate will time out at 1219.496812s 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c4132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f01c4098260 0x7f01c4098630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9d3a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f01c4098b70 0x7f01c4068e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:50616/0 (socket says 172.21.15.67:50616) 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9d3a640 1 -- 172.21.15.67:0/83189335 learned_addr learned my addr 172.21.15.67:0/83189335 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): get_auth_request con 0x7f01c4131f90 auth_method 0 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): get_auth_request con 0x7f01c4098260 auth_method 0 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:47.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:47.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:47.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:47.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:47.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:47.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient(hunting): handle_auth_done global_id 104454 payload 274 2024-09-15T01:59:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 -- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 msgr2=0x7f01c4132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c4132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 -- 172.21.15.67:0/83189335 >> v1:172.21.15.17:6789/0 conn(0x7f01c4098b70 legacy=0x7f01c4068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient: _finish_hunting 0 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 monclient: found mon.noname-a 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c8d38640 1 -- 172.21.15.67:0/83189335 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c40a16e0 con 0x7f01c4098260 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.028+0000 7f01c9539640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c4132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01c8d38640 1 -- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 msgr2=0x7f01c4098630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01c8d38640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c4098630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01c4064e70 tx=0x7f01b40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd0ce57edf57c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01bbfff640 1 -- 172.21.15.67:0/83189335 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b402adb0 con 0x7f01c4098260 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.031+0000 7f01bbfff640 10 monclient: dump: 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:47.050 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 10 monclient: _finish_auth 0 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 1 -- 172.21.15.67:0/83189335 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b4032440 con 0x7f01c4098260 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 5 monclient: authenticate success, global_id 104454 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:47.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 4 set_mon_vals no callback set 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 1 -- 172.21.15.67:0/83189335 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40327e0 con 0x7f01c4098260 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01bbfff640 10 monclient: dump: 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 10 monclient: shutdown 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:47.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 -- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 msgr2=0x7f01c4098630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c4098630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01c4064e70 tx=0x7f01b40090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 -- 172.21.15.67:0/83189335 shutdown_connections 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c4098630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 --2- 172.21.15.67:0/83189335 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c4132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 -- 172.21.15.67:0/83189335 >> 172.21.15.67:0/83189335 conn(0x7f01c4071690 msgr2=0x7f01c40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 -- 172.21.15.67:0/83189335 shutdown_connections 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.032+0000 7f01cb79d640 1 -- 172.21.15.67:0/83189335 wait complete. 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient: build_initial_monmap 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient: monmap: 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:47.054 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 1 Processor -- start 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 1 -- start start 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient: init 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c413de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.033+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-a con 0x7f01c4098260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:47.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 0x7f01c412f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-b con 0x7f01c4098b70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): picked mon.noname-c con 0x7f01c4131f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c411dc70 con 0x7f01c4098260 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c40a0d80 con 0x7f01c4098b70 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c4130e10 con 0x7f01c4131f90 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01cb79d640 10 monclient(hunting): authenticate will time out at 1219.503039s 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c8d38640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 msgr2=0x7f01c412f550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:47.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c8d38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 0x7f01c412f550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c413de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:42366/0 (socket says 172.21.15.67:42366) 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 -- 172.21.15.67:0/1411623134 learned_addr learned my addr 172.21.15.67:0/1411623134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): get_auth_request con 0x7f01c4131f90 auth_method 0 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): get_auth_request con 0x7f01c4098260 auth_method 0 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:47.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient(hunting): handle_auth_done global_id 94607 payload 824 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 -- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 msgr2=0x7f01c412f550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 0x7f01c412f550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 -- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 msgr2=0x7f01c413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c413de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 monclient: found mon.noname-c 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:47.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9d3a640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0009390 con 0x7f01c4131f90 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.034+0000 7f01c9539640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.036+0000 7f01c9d3a640 1 -- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 msgr2=0x7f01c41413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.036+0000 7f01c9d3a640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00ccf0 tx=0x7f01bc005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc00aa60 con 0x7f01c4131f90 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: dump: 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:47.059 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:47.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 5 monclient: authenticate success, global_id 94607 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 10 monclient: _renew_subs 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b4009390 con 0x7f01c4131f90 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 10 monclient: _renew_subs 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c411d2d0 con 0x7f01c4131f90 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc00abe0 con 0x7f01c4131f90 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0070f0 con 0x7f01c4131f90 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:47.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 10 monclient: dump: 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:47.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01bb7fe640 4 set_mon_vals no callback set 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.037+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01bc007460 con 0x7f01c4131f90 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01cb79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:47.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01cb79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01cb79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01b97fa640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 0x7f01a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c411d570 con 0x7f01c4131f90 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(213..213 src has 148..213) v4 ==== 14114+0+0 (secure 0 0 0) 0x7f01bc04e070 con 0x7f01c4131f90 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01c9539640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 0x7f01a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.038+0000 7f01c9539640 10 monclient: get_auth_request con 0x7f01a80386f0 auth_method 0 2024-09-15T01:59:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.040+0000 7f01c9539640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 0x7f01a803aba0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011580 tx=0x7f01c00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:47.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.043+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01bc01a590 con 0x7f01c4131f90 2024-09-15T01:59:47.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.043+0000 7f01b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:47.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.043+0000 7f01b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:47.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.210+0000 7f01cb79d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.210+0000 7f01cb79d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.210+0000 7f01cb79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.210+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f01c40639e0 con 0x7f01c4131f90 2024-09-15T01:59:47.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.799+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v214) v1 ==== 168+0+0 (secure 0 0 0) 0x7f01bc010af0 con 0x7f01c4131f90 2024-09-15T01:59:47.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.799+0000 7f01b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.799+0000 7f01b97fa640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-15T01:59:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.901+0000 7f01cb79d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.902+0000 7f01cb79d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.902+0000 7f01cb79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.902+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f01c4132950 con 0x7f01c4131f90 2024-09-15T01:59:47.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.903+0000 7f01b97fa640 1 -- 172.21.15.67:0/1411623134 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v214) v1 ==== 147+0+0 (secure 0 0 0) 0x7f01bc01a360 con 0x7f01c4131f90 2024-09-15T01:59:47.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.903+0000 7f01b97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:47.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.903+0000 7f01b97fa640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-15T01:59:47.906 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h29' already exists 2024-09-15T01:59:47.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 msgr2=0x7f01a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:47.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 0x7f01a803aba0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011580 tx=0x7f01c00090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 10 monclient: shutdown 2024-09-15T01:59:47.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 msgr2=0x7f01c41413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00ccf0 tx=0x7f01bc005370 comp rx=0 tx=0).stop 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 shutdown_connections 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01a80386f0 0x7f01a803aba0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01c4131f90 0x7f01c41413a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01c4098b70 0x7f01c412f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.905+0000 7f01cb79d640 1 --2- 172.21.15.67:0/1411623134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01c4098260 0x7f01c413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.906+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 >> 172.21.15.67:0/1411623134 conn(0x7f01c4071690 msgr2=0x7f01c4130230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.906+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 shutdown_connections 2024-09-15T01:59:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.906+0000 7f01cb79d640 1 -- 172.21.15.67:0/1411623134 wait complete. 2024-09-15T01:59:47.917 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:47.917 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.992+0000 7f1ec2d1d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.992+0000 7f1ec2d1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.992+0000 7f1ec2d1d640 10 monclient: monmap: 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:47.993896+0000 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:47.993896+0000 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:47.996 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:47.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 1 Processor -- start 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 1 -- start start 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 10 monclient: init 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc09cb20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1ebc09cef0 0x7f1ebc09afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.993+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc09cef0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 0x7f1ebc096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc09d520 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc09b520 con 0x7f1ebc09cef0 2024-09-15T01:59:47.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc099290 con 0x7f1ebc09d520 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc098fb0 con 0x7f1ebc09cb20 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec2d1d640 10 monclient(hunting): authenticate will time out at 1220.463036s 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec12ba640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 msgr2=0x7f1ebc096a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec12ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 0x7f1ebc096a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1ebc09cef0 0x7f1ebc09afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:42380/0 (socket says 172.21.15.67:42380) 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 1 -- 172.21.15.67:0/3041290296 learned_addr learned my addr 172.21.15.67:0/3041290296 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): get_auth_request con 0x7f1ebc09cb20 auth_method 0 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): get_auth_request con 0x7f1ebc09cef0 auth_method 0 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:48.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient(hunting): handle_auth_done global_id 104460 payload 274 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 -- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 msgr2=0x7f1ebc09aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 -- 172.21.15.67:0/3041290296 >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 msgr2=0x7f1ebc096a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 --2- 172.21.15.67:0/3041290296 >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 0x7f1ebc096a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 monclient: found mon.noname-a 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ebbfff640 1 -- 172.21.15.67:0/3041290296 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc12f810 con 0x7f1ebc09cef0 2024-09-15T01:59:48.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.994+0000 7f1ec0ab9640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.997+0000 7f1ebbfff640 1 -- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc09cef0 msgr2=0x7f1ebc09afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.997+0000 7f1ebbfff640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc09cef0 0x7f1ebc09afe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc097e60 tx=0x7f1ea80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bd3ee1ffe4280e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 1 -- 172.21.15.67:0/3041290296 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea802ae00 con 0x7f1ebc09cef0 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:48.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: dump: 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:48.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 1 -- 172.21.15.67:0/3041290296 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea8032440 con 0x7f1ebc09cef0 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 1 -- 172.21.15.67:0/3041290296 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80327e0 con 0x7f1ebc09cef0 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ebb7fe640 10 monclient: dump: 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 5 monclient: authenticate success, global_id 104460 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 4 set_mon_vals no callback set 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 10 monclient: shutdown 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc09cef0 msgr2=0x7f1ebc09afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.998+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc09cef0 0x7f1ebc09afe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc097e60 tx=0x7f1ea80090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/3041290296 shutdown_connections 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/3041290296 >> v2:172.21.15.67:3300/0 conn(0x7f1ebc09d520 0x7f1ebc096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc09cef0 0x7f1ebc09afe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/3041290296 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc09aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/3041290296 >> 172.21.15.67:0/3041290296 conn(0x7f1ebc0a1020 msgr2=0x7f1ebc0a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/3041290296 shutdown_connections 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/3041290296 wait complete. 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 10 monclient: build_initial_monmap 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 10 monclient: monmap: 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:48.012 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:47.999+0000 7f1ec2d1d640 1 Processor -- start 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 -- start start 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient: init 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc09cb20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 0x7f1ebc129cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc09d520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 0x7f1ebc12a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc141680 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc06e060 con 0x7f1ebc141680 2024-09-15T01:59:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc142040 con 0x7f1ebc09d520 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc06d200 con 0x7f1ebc09cb20 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec2d1d640 10 monclient(hunting): authenticate will time out at 1220.469524s 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ebbfff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 msgr2=0x7f1ebc129cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ebbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 0x7f1ebc129cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 0x7f1ebc12a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc141140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:42382/0 (socket says 172.21.15.67:42382) 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): get_auth_request con 0x7f1ebc141680 auth_method 0 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 1 -- 172.21.15.67:0/1425130585 learned_addr learned my addr 172.21.15.67:0/1425130585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:48.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): get_auth_request con 0x7f1ebc09cb20 auth_method 0 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec12ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.000+0000 7f1ec0ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 10 monclient(hunting): handle_auth_done global_id 104463 payload 824 2024-09-15T01:59:48.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 -- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 msgr2=0x7f1ebc141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc141140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 -- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 msgr2=0x7f1ebc129cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 0x7f1ebc129cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 10 monclient: _finish_hunting 0 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 monclient: found mon.noname-a 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.001+0000 7f1ec12ba640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0009450 con 0x7f1ebc141680 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec12ba640 1 -- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 msgr2=0x7f1ebc12a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec12ba640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 0x7f1ebc12a230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400bdf0 tx=0x7f1eb40096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4006570 con 0x7f1ebc141680 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 10 monclient: dump: 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1eb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:48.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 5 monclient: authenticate success, global_id 104463 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 10 monclient: _renew_subs 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea8009390 con 0x7f1ebc141680 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 10 monclient: _renew_subs 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.003+0000 7f1ec2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc06c9f0 con 0x7f1ebc141680 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb400f070 con 0x7f1ebc141680 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40066d0 con 0x7f1ebc141680 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ebaffd640 4 set_mon_vals no callback set 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 10 monclient: dump: 2024-09-15T01:59:48.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ec2d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:48.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ec2d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:48.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ec2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c005010 con 0x7f1ebc141680 2024-09-15T01:59:48.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.004+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1eb401e3f0 con 0x7f1ebc141680 2024-09-15T01:59:48.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.005+0000 7f1eb8ff9640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 0x7f1ea003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:48.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.005+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(214..214 src has 148..214) v4 ==== 14327+0+0 (secure 0 0 0) 0x7f1eb404ea00 con 0x7f1ebc141680 2024-09-15T01:59:48.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.005+0000 7f1ec0ab9640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 0x7f1ea003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:48.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.005+0000 7f1ec0ab9640 10 monclient: get_auth_request con 0x7f1ea00386f0 auth_method 0 2024-09-15T01:59:48.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.007+0000 7f1ec0ab9640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 0x7f1ea003aba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009f70 tx=0x7f1eb00091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:48.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.009+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1eb401d950 con 0x7f1ebc141680 2024-09-15T01:59:48.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.009+0000 7f1eb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:48.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.009+0000 7f1eb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:48.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.174+0000 7f1ec2d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.174+0000 7f1ec2d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.174+0000 7f1ec2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.174+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1e9c0052c0 con 0x7f1ebc141680 2024-09-15T01:59:48.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.840+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v215) v1 ==== 168+0+0 (secure 0 0 0) 0x7f1eb401d350 con 0x7f1ebc141680 2024-09-15T01:59:48.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.840+0000 7f1eb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.840+0000 7f1eb8ff9640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-15T01:59:48.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.943+0000 7f1ec2d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.943+0000 7f1ec2d1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.943+0000 7f1ec2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:48.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.943+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1e9c005a00 con 0x7f1ebc141680 2024-09-15T01:59:48.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.944+0000 7f1eb8ff9640 1 -- 172.21.15.67:0/1425130585 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v215) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1eb401d530 con 0x7f1ebc141680 2024-09-15T01:59:48.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.944+0000 7f1eb8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:48.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.944+0000 7f1eb8ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-15T01:59:48.947 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h30' already exists 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 msgr2=0x7f1ea003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 0x7f1ea003aba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009f70 tx=0x7f1eb00091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 10 monclient: shutdown 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 msgr2=0x7f1ebc12a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 0x7f1ebc12a230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400bdf0 tx=0x7f1eb40096a0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 shutdown_connections 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ea00386f0 0x7f1ea003aba0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ebc141680 0x7f1ebc12a230 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ebc09d520 0x7f1ebc129cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 --2- 172.21.15.67:0/1425130585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1ebc09cb20 0x7f1ebc141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 >> 172.21.15.67:0/1425130585 conn(0x7f1ebc0a1020 msgr2=0x7f1ebc09f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:48.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 shutdown_connections 2024-09-15T01:59:48.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:48.946+0000 7f1ec2d1d640 1 -- 172.21.15.67:0/1425130585 wait complete. 2024-09-15T01:59:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-15T01:59:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.033+0000 7f26a459d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.033+0000 7f26a459d640 10 monclient: build_initial_monmap 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.033+0000 7f26a459d640 10 monclient: monmap: 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:49.034900+0000 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:49.034900+0000 2024-09-15T01:59:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 Processor -- start 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 -- start start 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient: init 2024-09-15T01:59:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f269c098190 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-b con 0x7f269c098560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 0x7f269c1391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-c con 0x7f269c136e10 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c0a16c0 con 0x7f269c098190 2024-09-15T01:59:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269c1328c0 con 0x7f269c136e10 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0a0700 con 0x7f269c098560 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.034+0000 7f26a459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a459d640 10 monclient(hunting): authenticate will time out at 1221.504008s 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a2339640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f269c098190 legacy=0x7f269c1321b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a2b3a640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 msgr2=0x7f269c1391f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a2b3a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 0x7f269c1391f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:42398/0 (socket says 172.21.15.67:42398) 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 -- 172.21.15.67:0/1772751681 learned_addr learned my addr 172.21.15.67:0/1772751681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): get_auth_request con 0x7f269c098560 auth_method 0 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient(hunting): handle_auth_done global_id 94616 payload 274 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 -- 172.21.15.67:0/1772751681 >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 msgr2=0x7f269c1391f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 --2- 172.21.15.67:0/1772751681 >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 0x7f269c1391f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 -- 172.21.15.67:0/1772751681 >> v1:172.21.15.67:6789/0 conn(0x7f269c098190 legacy=0x7f269c1321b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient: _finish_hunting 0 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 monclient: found mon.noname-b 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.035+0000 7f26a1b38640 1 -- 172.21.15.67:0/1772751681 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c0a13e0 con 0x7f269c098560 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.038+0000 7f26a1b38640 1 -- 172.21.15.67:0/1772751681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 msgr2=0x7f269c098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.038+0000 7f26a1b38640 1 --2- 172.21.15.67:0/1772751681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f269c096c20 tx=0x7f268c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7eb8052acc0de3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.038+0000 7f26a1337640 1 -- 172.21.15.67:0/1772751681 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c032440 con 0x7f269c098560 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.038+0000 7f26a1337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:49.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: dump: 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: _finish_auth 0 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:49.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 5 monclient: authenticate success, global_id 94616 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 1 -- 172.21.15.67:0/1772751681 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c0325a0 con 0x7f269c098560 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 4 set_mon_vals no callback set 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 1 -- 172.21.15.67:0/1772751681 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c032940 con 0x7f269c098560 2024-09-15T01:59:49.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:49.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:49.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a1337640 10 monclient: dump: 2024-09-15T01:59:49.060 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 10 monclient: shutdown 2024-09-15T01:59:49.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 -- 172.21.15.67:0/1772751681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 msgr2=0x7f269c098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 --2- 172.21.15.67:0/1772751681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f269c096c20 tx=0x7f268c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 -- 172.21.15.67:0/1772751681 shutdown_connections 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 --2- 172.21.15.67:0/1772751681 >> v2:172.21.15.67:3300/0 conn(0x7f269c136e10 0x7f269c1391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 --2- 172.21.15.67:0/1772751681 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c098960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 -- 172.21.15.67:0/1772751681 >> 172.21.15.67:0/1772751681 conn(0x7f269c0711e0 msgr2=0x7f269c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 -- 172.21.15.67:0/1772751681 shutdown_connections 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.039+0000 7f26a459d640 1 -- 172.21.15.67:0/1772751681 wait complete. 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient: build_initial_monmap 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient: monmap: 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:49.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 Processor -- start 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 -- start start 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient: init 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-a con 0x7f269c098190 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-c con 0x7f269c098560 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 0x7f269c134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): picked mon.noname-b con 0x7f269c136e10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c098eb0 con 0x7f269c098190 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0a3510 con 0x7f269c136e10 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c1363c0 con 0x7f269c098560 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a459d640 10 monclient(hunting): authenticate will time out at 1221.509908s 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a2b3a640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 msgr2=0x7f269c134e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.040+0000 7f26a2b3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 0x7f269c134e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:37996/0 (socket says 172.21.15.67:37996) 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 -- 172.21.15.67:0/808477573 learned_addr learned my addr 172.21.15.67:0/808477573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): get_auth_request con 0x7f269c098190 auth_method 0 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): get_auth_request con 0x7f269c098560 auth_method 0 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:49.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient(hunting): handle_auth_done global_id 104466 payload 824 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 -- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 msgr2=0x7f269c134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 -- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 msgr2=0x7f269c134e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:49.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 0x7f269c134e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient: _finish_hunting 0 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 monclient: found mon.noname-a 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.041+0000 7f26a2339640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c009450 con 0x7f269c098190 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a2339640 1 -- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 msgr2=0x7f269c130bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a2339640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f269800b600 tx=0x7f26980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698015070 con 0x7f269c098190 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: dump: 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:49.067 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 5 monclient: authenticate success, global_id 104466 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 10 monclient: _renew_subs 2024-09-15T01:59:49.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2698003d90 con 0x7f269c098190 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c142070 con 0x7f269c098190 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698006380 con 0x7f269c098190 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26927fc640 10 monclient: dump: 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:49.069 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a0b36640 4 set_mon_vals no callback set 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 10 monclient: _renew_subs 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.044+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c1376b0 con 0x7f269c098190 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c0639e0 con 0x7f269c098190 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2698010440 con 0x7f269c098190 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26927fc640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 0x7f268003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(215..215 src has 148..215) v4 ==== 14540+0+0 (secure 0 0 0) 0x7f269804dba0 con 0x7f269c098190 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a1b38640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 0x7f268003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.045+0000 7f26a1b38640 10 monclient: get_auth_request con 0x7f26800386f0 auth_method 0 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.048+0000 7f26a1b38640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 0x7f268003aba0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f269c09c360 tx=0x7f268c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.050+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2698006730 con 0x7f269c098190 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.050+0000 7f26927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.050+0000 7f26927fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.216+0000 7f26a459d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.216+0000 7f26a459d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.216+0000 7f26a459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.216+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f269c11d0f0 con 0x7f269c098190 2024-09-15T01:59:49.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.844+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v216) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2698018780 con 0x7f269c098190 2024-09-15T01:59:49.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.844+0000 7f26927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.844+0000 7f26927fc640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-15T01:59:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.949+0000 7f26a459d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.949+0000 7f26a459d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.949+0000 7f26a459d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:49.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.949+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f269c06f0b0 con 0x7f269c098190 2024-09-15T01:59:49.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.949+0000 7f26927fc640 1 -- 172.21.15.67:0/808477573 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v216) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2698018d60 con 0x7f269c098190 2024-09-15T01:59:49.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.950+0000 7f26927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:49.952 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.950+0000 7f26927fc640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-15T01:59:49.952 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h31' already exists 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 msgr2=0x7f268003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 0x7f268003aba0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f269c09c360 tx=0x7f268c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 10 monclient: shutdown 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 msgr2=0x7f269c130bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:49.954 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f269800b600 tx=0x7f26980091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 shutdown_connections 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f26800386f0 0x7f268003aba0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f269c136e10 0x7f269c134e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f269c098560 0x7f269c134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 --2- 172.21.15.67:0/808477573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269c098190 0x7f269c130bb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.951+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 >> 172.21.15.67:0/808477573 conn(0x7f269c0711e0 msgr2=0x7f269c09e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.952+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 shutdown_connections 2024-09-15T01:59:49.955 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:49.952+0000 7f26a459d640 1 -- 172.21.15.67:0/808477573 wait complete. 2024-09-15T01:59:49.963 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:49.963 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-15T01:59:50.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 10 monclient: get_monmap_and_config 2024-09-15T01:59:50.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 10 monclient: build_initial_monmap 2024-09-15T01:59:50.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 10 monclient: monmap: 2024-09-15T01:59:50.041 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:50.040249+0000 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:50.040249+0000 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:50.042 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:50.043 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 1 Processor -- start 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 1 -- start start 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.039+0000 7fc75682b640 10 monclient: init 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc750131e70 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc750098420 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 0x7fc75013a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-b con 0x7fc7501381b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500a13e0 con 0x7fc750098420 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500a0440 con 0x7fc750131e70 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:50.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7500a01e0 con 0x7fc7501381b0 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc75682b640 10 monclient(hunting): authenticate will time out at 1222.509379s 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc74ffff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fc750131e70 legacy=0x7fc750132240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 0x7fc75013a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc74f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc750098420 0x7fc750098850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:50628/0 (socket says 172.21.15.67:50628) 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc74f7fe640 1 -- 172.21.15.67:0/3625973459 learned_addr learned my addr 172.21.15.67:0/3625973459 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): get_auth_request con 0x7fc7501381b0 auth_method 0 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:50.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.040+0000 7fc754dc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 10 monclient(hunting): handle_auth_done global_id 94622 payload 274 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 1 -- 172.21.15.67:0/3625973459 >> v1:172.21.15.67:6789/0 conn(0x7fc750131e70 legacy=0x7fc750132240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 1 -- 172.21.15.67:0/3625973459 >> v1:172.21.15.17:6789/0 conn(0x7fc750098420 legacy=0x7fc750098850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 10 monclient: _finish_hunting 0 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 1 monclient: found mon.noname-b 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:50.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.041+0000 7fc754dc8640 1 -- 172.21.15.67:0/3625973459 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7500a19b0 con 0x7fc7501381b0 2024-09-15T01:59:50.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.043+0000 7fc754dc8640 1 -- 172.21.15.67:0/3625973459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 msgr2=0x7fc75013a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:50.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.043+0000 7fc754dc8640 1 --2- 172.21.15.67:0/3625973459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 0x7fc75013a590 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc7500a6250 tx=0x7fc748009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8e7e1249b409380 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:50.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 1 -- 172.21.15.67:0/3625973459 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748007840 con 0x7fc7501381b0 2024-09-15T01:59:50.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:50.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:50.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: dump: 2024-09-15T01:59:50.053 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:50.053 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:50.053 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:50.054 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: _finish_auth 0 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 1 -- 172.21.15.67:0/3625973459 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74802d050 con 0x7fc7501381b0 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 1 -- 172.21.15.67:0/3625973459 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748007bd0 con 0x7fc7501381b0 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc74effd640 10 monclient: dump: 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:50.055 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 5 monclient: authenticate success, global_id 94622 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 4 set_mon_vals no callback set 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 10 monclient: shutdown 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 -- 172.21.15.67:0/3625973459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 msgr2=0x7fc75013a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 --2- 172.21.15.67:0/3625973459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 0x7fc75013a590 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc7500a6250 tx=0x7fc748009780 comp rx=0 tx=0).stop 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 -- 172.21.15.67:0/3625973459 shutdown_connections 2024-09-15T01:59:50.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 --2- 172.21.15.67:0/3625973459 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7501381b0 0x7fc75013a590 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 -- 172.21.15.67:0/3625973459 >> 172.21.15.67:0/3625973459 conn(0x7fc7500a5c10 msgr2=0x7fc7500a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 -- 172.21.15.67:0/3625973459 shutdown_connections 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.044+0000 7fc75682b640 1 -- 172.21.15.67:0/3625973459 wait complete. 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.045+0000 7fc75682b640 10 monclient: build_initial_monmap 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.045+0000 7fc75682b640 10 monclient: monmap: 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.057 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.045+0000 7fc75682b640 1 Processor -- start 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.045+0000 7fc75682b640 1 -- start start 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.045+0000 7fc75682b640 10 monclient: init 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:50.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-a con 0x7fc750098420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 0x7fc7500994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-c con 0x7fc750131e70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 0x7fc750099a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): picked mon.noname-b con 0x7fc7501381b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc750133a50 con 0x7fc750098420 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc750069df0 con 0x7fc7501381b0 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75009b910 con 0x7fc750131e70 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc75682b640 10 monclient(hunting): authenticate will time out at 1222.515399s 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc754dc8640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 msgr2=0x7fc750099a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:50.059 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc754dc8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 0x7fc750099a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 0x7fc7500994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:38000/0 (socket says 172.21.15.67:38000) 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 -- 172.21.15.67:0/1055527170 learned_addr learned my addr 172.21.15.67:0/1055527170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): get_auth_request con 0x7fc750098420 auth_method 0 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): get_auth_request con 0x7fc750131e70 auth_method 0 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:50.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient(hunting): handle_auth_done global_id 104469 payload 824 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 -- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 msgr2=0x7fc7500994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 0x7fc7500994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 -- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 msgr2=0x7fc750099a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 0x7fc750099a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient: _finish_hunting 0 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 monclient: found mon.noname-a 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:50.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74ffff640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748009a20 con 0x7fc750098420 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.046+0000 7fc74f7fe640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 0x7fc7500994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc74ffff640 1 -- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 msgr2=0x7fc7501379b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc74ffff640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b630 tx=0x7fc7400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740015070 con 0x7fc750098420 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: dump: 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:50.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 5 monclient: authenticate success, global_id 104469 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 10 monclient: _renew_subs 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc750136fc0 con 0x7fc750098420 2024-09-15T01:59:50.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 10 monclient: _renew_subs 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740003d90 con 0x7fc750098420 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75009b150 con 0x7fc750098420 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740006380 con 0x7fc750098420 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc747fff640 10 monclient: dump: 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:50.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:50.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.049+0000 7fc74e7fc640 4 set_mon_vals no callback set 2024-09-15T01:59:50.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc75682b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:50.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc75682b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc75682b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7501347c0 con 0x7fc750098420 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc740010440 con 0x7fc750098420 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc747fff640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 0x7fc73003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.050+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(216..216 src has 148..216) v4 ==== 14753+0+0 (secure 0 0 0) 0x7fc74004d9f0 con 0x7fc750098420 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.051+0000 7fc74f7fe640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 0x7fc73003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.051+0000 7fc74f7fe640 10 monclient: get_auth_request con 0x7fc730038720 auth_method 0 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.053+0000 7fc74f7fe640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 0x7fc73003abd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc750134500 tx=0x7fc738006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.055+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc74001a770 con 0x7fc750098420 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.055+0000 7fc747fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.055+0000 7fc747fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:50.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.223+0000 7fc75682b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.223+0000 7fc75682b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.223+0000 7fc75682b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.223+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc7500639e0 con 0x7fc750098420 2024-09-15T01:59:50.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.847+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v217) v1 ==== 168+0+0 (secure 0 0 0) 0x7fc740018780 con 0x7fc750098420 2024-09-15T01:59:50.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.847+0000 7fc747fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.847+0000 7fc747fff640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-15T01:59:50.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.953+0000 7fc75682b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.953+0000 7fc75682b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.953+0000 7fc75682b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:50.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.953+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc750067a50 con 0x7fc750098420 2024-09-15T01:59:50.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.954+0000 7fc747fff640 1 -- 172.21.15.67:0/1055527170 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v217) v1 ==== 147+0+0 (secure 0 0 0) 0x7fc740018d60 con 0x7fc750098420 2024-09-15T01:59:50.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.954+0000 7fc747fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T01:59:50.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.954+0000 7fc747fff640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-15T01:59:50.957 INFO:tasks.workunit.client.0.smithi067.stderr:bucket 'h32' already exists 2024-09-15T01:59:50.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 msgr2=0x7fc73003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:50.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 0x7fc73003abd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc750134500 tx=0x7fc738006f50 comp rx=0 tx=0).stop 2024-09-15T01:59:50.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 10 monclient: shutdown 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 msgr2=0x7fc7501379b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.956+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b630 tx=0x7fc7400091a0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 shutdown_connections 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc730038720 0x7fc73003abd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7501381b0 0x7fc750099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc750131e70 0x7fc7500994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 --2- 172.21.15.67:0/1055527170 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc750098420 0x7fc7501379b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 >> 172.21.15.67:0/1055527170 conn(0x7fc7500a5c10 msgr2=0x7fc7500a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 shutdown_connections 2024-09-15T01:59:50.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:50.957+0000 7fc75682b640 1 -- 172.21.15.67:0/1055527170 wait complete. 2024-09-15T01:59:50.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ seq 1 32 2024-09-15T01:59:50.970 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:50.970 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h1 2024-09-15T01:59:51.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.044+0000 7f6e28f5d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:51.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.044+0000 7f6e28f5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.044+0000 7f6e28f5d640 10 monclient: monmap: 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:51.045487+0000 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:51.045487+0000 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:51.048 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:51.048 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 Processor -- start 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 -- start start 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient: init 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e240a0b50 0x7f6e24069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e240a0b50 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e2409eea0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 0x7f6e24132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2409f2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240a61a0 con 0x7f6e2409eea0 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e240a1610 con 0x7f6e240a0b50 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240a04c0 con 0x7f6e2409f2c0 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e28f5d640 10 monclient(hunting): authenticate will time out at 1223.514692s 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 0x7f6e24132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.045+0000 7f6e2259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e240a0b50 0x7f6e24069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e1bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6e2409eea0 0x7f6e24131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56262/0 (socket says 172.21.15.67:56262) 2024-09-15T01:59:51.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6e240a0b50 0x7f6e24069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43568/0 (socket says 172.21.15.67:43568) 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 0x7f6e24132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35702/0 (socket says 172.21.15.67:35702) 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e1bfff640 1 -- 172.21.15.67:0/4118749800 learned_addr learned my addr 172.21.15.67:0/4118749800 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): get_auth_request con 0x7f6e240a0b50 auth_method 0 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): get_auth_request con 0x7f6e2409f2c0 auth_method 0 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:51.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient(hunting): handle_auth_done global_id 104472 payload 274 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 -- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 msgr2=0x7f6e24132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 --2- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 0x7f6e24132220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 -- 172.21.15.67:0/4118749800 >> v1:172.21.15.17:6789/0 conn(0x7f6e2409eea0 legacy=0x7f6e24131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient: _finish_hunting 0 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 monclient: found mon.noname-a 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.046+0000 7f6e2259c640 1 -- 172.21.15.67:0/4118749800 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240a0f20 con 0x7f6e240a0b50 2024-09-15T01:59:51.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e2259c640 1 -- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 msgr2=0x7f6e24069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:51.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e2259c640 1 --2- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e24069380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009850 tx=0x7f6e140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3931112ca6f1d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:51.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 1 -- 172.21.15.67:0/4118749800 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1402adf0 con 0x7f6e240a0b50 2024-09-15T01:59:51.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: dump: 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:51.061 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: _finish_auth 0 2024-09-15T01:59:51.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:51.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 1 -- 172.21.15.67:0/4118749800 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14032440 con 0x7f6e240a0b50 2024-09-15T01:59:51.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 1 -- 172.21.15.67:0/4118749800 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140327e0 con 0x7f6e240a0b50 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 5 monclient: authenticate success, global_id 104472 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 4 set_mon_vals no callback set 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e21d9b640 10 monclient: dump: 2024-09-15T01:59:51.063 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 10 monclient: shutdown 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 1 -- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 msgr2=0x7f6e24069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.049+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e24069380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009850 tx=0x7f6e140090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 -- 172.21.15.67:0/4118749800 shutdown_connections 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409f2c0 0x7f6e24132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/4118749800 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e24069380 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 -- 172.21.15.67:0/4118749800 >> 172.21.15.67:0/4118749800 conn(0x7f6e24071330 msgr2=0x7f6e240a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:51.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 -- 172.21.15.67:0/4118749800 shutdown_connections 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 -- 172.21.15.67:0/4118749800 wait complete. 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 10 monclient: build_initial_monmap 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 10 monclient: monmap: 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:51.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 Processor -- start 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 1 -- start start 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.050+0000 7f6e28f5d640 10 monclient: init 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 0x7f6e24131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 0x7f6e2411b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2409f2c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e2411be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e240a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240a1260 con 0x7f6e240a0b50 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24134c70 con 0x7f6e2409f2c0 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2411ef90 con 0x7f6e2409eea0 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:51.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e28f5d640 10 monclient(hunting): authenticate will time out at 1223.520305s 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e1bfff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 msgr2=0x7f6e2411b940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e1bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 0x7f6e2411b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 0x7f6e24131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e2411be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 0x7f6e24131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35710/0 (socket says 172.21.15.67:35710) 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 1 -- 172.21.15.67:0/2868284042 learned_addr learned my addr 172.21.15.67:0/2868284042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): get_auth_request con 0x7f6e240a0b50 auth_method 0 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): get_auth_request con 0x7f6e2409eea0 auth_method 0 2024-09-15T01:59:51.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient(hunting): handle_auth_done global_id 104475 payload 824 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 -- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 msgr2=0x7f6e24131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 0x7f6e24131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 -- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 msgr2=0x7f6e2411b940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 0x7f6e2411b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 10 monclient: _finish_hunting 0 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.051+0000 7f6e22d9d640 1 monclient: found mon.noname-a 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.052+0000 7f6e22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.052+0000 7f6e22d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.052+0000 7f6e22d9d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14009390 con 0x7f6e240a0b50 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e22d9d640 1 -- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 msgr2=0x7f6e2411be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e22d9d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e2411be80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009e50 tx=0x7f6e1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c014480 con 0x7f6e240a0b50 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: dump: 2024-09-15T01:59:51.071 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 5 monclient: authenticate success, global_id 104475 2024-09-15T01:59:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c00f040 con 0x7f6e240a0b50 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 10 monclient: _renew_subs 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2411ecb0 con 0x7f6e240a0b50 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c014850 con 0x7f6e240a0b50 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e1a7fc640 10 monclient: dump: 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:51.073 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e2159a640 4 set_mon_vals no callback set 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 10 monclient: _renew_subs 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.054+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e24136570 con 0x7f6e240a0b50 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e28f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e28f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e28f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2404bf60 con 0x7f6e240a0b50 2024-09-15T01:59:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6e1c014bc0 con 0x7f6e240a0b50 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.055+0000 7f6e1a7fc640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 0x7f6e0403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.056+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(217..217 src has 148..217) v4 ==== 14966+0+0 (secure 0 0 0) 0x7f6e1c04f100 con 0x7f6e240a0b50 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.056+0000 7f6e2259c640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 0x7f6e0403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.056+0000 7f6e2259c640 10 monclient: get_auth_request con 0x7f6e040386f0 auth_method 0 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.058+0000 7f6e2259c640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 0x7f6e0403aba0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009130 tx=0x7f6e14032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.060+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6e1c024900 con 0x7f6e240a0b50 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.060+0000 7f6e1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:51.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.060+0000 7f6e1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:51.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.227+0000 7f6e28f5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.227+0000 7f6e28f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.227+0000 7f6e28f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.227+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f6e240a0220 con 0x7f6e240a0b50 2024-09-15T01:59:51.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.855+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v218) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6e1c018020 con 0x7f6e240a0b50 2024-09-15T01:59:51.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.855+0000 7f6e1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.855+0000 7f6e1a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-15T01:59:51.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.959+0000 7f6e28f5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.959+0000 7f6e28f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.959+0000 7f6e28f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.959+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f6e24142020 con 0x7f6e240a0b50 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.960+0000 7f6e1a7fc640 1 -- 172.21.15.67:0/2868284042 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v218) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6e1c012340 con 0x7f6e240a0b50 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.960+0000 7f6e1a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.960+0000 7f6e1a7fc640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-15T01:59:51.962 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h1' does not appear in the crush map 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 msgr2=0x7f6e0403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 0x7f6e0403aba0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009130 tx=0x7f6e14032000 comp rx=0 tx=0).stop 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 10 monclient: shutdown 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 msgr2=0x7f6e2411be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e2411be80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009e50 tx=0x7f6e1c00b040 comp rx=0 tx=0).stop 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 shutdown_connections 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6e040386f0 0x7f6e0403aba0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6e240a0b50 0x7f6e2411be80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2409f2c0 0x7f6e2411b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 --2- 172.21.15.67:0/2868284042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6e2409eea0 0x7f6e24131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 >> 172.21.15.67:0/2868284042 conn(0x7f6e24071330 msgr2=0x7f6e240a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 shutdown_connections 2024-09-15T01:59:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:51.962+0000 7f6e28f5d640 1 -- 172.21.15.67:0/2868284042 wait complete. 2024-09-15T01:59:51.974 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:51.974 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h2 2024-09-15T01:59:52.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.048+0000 7fbe3339d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.048+0000 7fbe3339d640 10 monclient: build_initial_monmap 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.048+0000 7fbe3339d640 10 monclient: monmap: 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:52.049739+0000 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:52.049739+0000 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:52.051 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:52.052 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:52.052 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:52.052 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 Processor -- start 2024-09-15T01:59:52.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 -- start start 2024-09-15T01:59:52.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient: init 2024-09-15T01:59:52.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:52.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c0a3830 0x7fbe2c06a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c0a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe2c06bef0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe2c136d00 0x7fbe2c1390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c136d00 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c0a3c00 con 0x7fbe2c06bef0 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2c0696a0 con 0x7fbe2c136d00 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c069940 con 0x7fbe2c0a3830 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe3339d640 10 monclient(hunting): authenticate will time out at 1224.518865s 2024-09-15T01:59:52.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.049+0000 7fbe30938640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fbe2c06bef0 legacy=0x7fbe2c099aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe2c136d00 0x7fbe2c1390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c0a3830 0x7fbe2c06a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe2c136d00 0x7fbe2c1390e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43586/0 (socket says 172.21.15.67:43586) 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 -- 172.21.15.67:0/2992558694 learned_addr learned my addr 172.21.15.67:0/2992558694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): get_auth_request con 0x7fbe2c136d00 auth_method 0 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): get_auth_request con 0x7fbe2c0a3830 auth_method 0 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:52.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe31139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient(hunting): handle_auth_done global_id 104478 payload 274 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 -- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c0a3830 msgr2=0x7fbe2c06a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 --2- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c0a3830 0x7fbe2c06a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 -- 172.21.15.67:0/2992558694 >> v1:172.21.15.67:6789/0 conn(0x7fbe2c06bef0 legacy=0x7fbe2c099aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 monclient: found mon.noname-a 2024-09-15T01:59:52.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.050+0000 7fbe3193a640 1 -- 172.21.15.67:0/2992558694 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c069c20 con 0x7fbe2c136d00 2024-09-15T01:59:52.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe3193a640 1 -- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c136d00 msgr2=0x7fbe2c1390e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:52.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe3193a640 1 --2- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c136d00 0x7fbe2c1390e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f50 tx=0x7fbe24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282078f3b7483d58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:52.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 1 -- 172.21.15.67:0/2992558694 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007610 con 0x7fbe2c136d00 2024-09-15T01:59:52.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:52.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: dump: 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:52.064 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: _finish_auth 0 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 1 -- 172.21.15.67:0/2992558694 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2402d050 con 0x7fbe2c136d00 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 1 -- 172.21.15.67:0/2992558694 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240079a0 con 0x7fbe2c136d00 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe23fff640 10 monclient: dump: 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:52.065 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe3339d640 5 monclient: authenticate success, global_id 104478 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe3339d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.053+0000 7fbe3339d640 4 set_mon_vals no callback set 2024-09-15T01:59:52.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 10 monclient: shutdown 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 -- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c136d00 msgr2=0x7fbe2c1390e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 --2- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c136d00 0x7fbe2c1390e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f50 tx=0x7fbe24009840 comp rx=0 tx=0).stop 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 -- 172.21.15.67:0/2992558694 shutdown_connections 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 --2- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c136d00 0x7fbe2c1390e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 --2- 172.21.15.67:0/2992558694 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c0a3830 0x7fbe2c06a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 -- 172.21.15.67:0/2992558694 >> 172.21.15.67:0/2992558694 conn(0x7fbe2c070fb0 msgr2=0x7fbe2c0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 -- 172.21.15.67:0/2992558694 shutdown_connections 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 1 -- 172.21.15.67:0/2992558694 wait complete. 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 10 monclient: build_initial_monmap 2024-09-15T01:59:52.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.054+0000 7fbe3339d640 10 monclient: monmap: 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:52.068 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 Processor -- start 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 -- start start 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient: init 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 0x7fbe2c09c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c06bef0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c0a3830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c136d00 0x7fbe2c098140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2c136d00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe31139640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 msgr2=0x7fbe2c09c900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c06ea50 con 0x7fbe2c0a3830 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0a3480 con 0x7fbe2c06bef0 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c09bf50 con 0x7fbe2c136d00 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe3339d640 10 monclient(hunting): authenticate will time out at 1224.524865s 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.055+0000 7fbe31139640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 0x7fbe2c09c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:52.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c136d00 0x7fbe2c098140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43600/0 (socket says 172.21.15.67:43600) 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 -- 172.21.15.67:0/1837729124 learned_addr learned my addr 172.21.15.67:0/1837729124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): get_auth_request con 0x7fbe2c0a3830 auth_method 0 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): get_auth_request con 0x7fbe2c136d00 auth_method 0 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:52.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe3193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient(hunting): handle_auth_done global_id 104481 payload 824 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 -- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c136d00 msgr2=0x7fbe2c098140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c136d00 0x7fbe2c098140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 -- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 msgr2=0x7fbe2c09c900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:52.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 0x7fbe2c09c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient: _finish_hunting 0 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 monclient: found mon.noname-a 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.056+0000 7fbe30938640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe24009ae0 con 0x7fbe2c0a3830 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.058+0000 7fbe30938640 1 -- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 msgr2=0x7fbe2c097c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.058+0000 7fbe30938640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b5c0 tx=0x7fbe1c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c003d90 con 0x7fbe2c0a3830 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:52.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: dump: 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c006150 con 0x7fbe2c0a3830 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 5 monclient: authenticate success, global_id 104481 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 10 monclient: _renew_subs 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2c09ace0 con 0x7fbe2c0a3830 2024-09-15T01:59:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c006510 con 0x7fbe2c0a3830 2024-09-15T01:59:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe217fa640 10 monclient: dump: 2024-09-15T01:59:52.077 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:52.077 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:52.078 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.078 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 10 monclient: _renew_subs 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe237fe640 4 set_mon_vals no callback set 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.059+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2c09bcf0 con 0x7fbe2c0a3830 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe1c01b3b0 con 0x7fbe2c0a3830 2024-09-15T01:59:52.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe3339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe3339d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe3339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2c099100 con 0x7fbe2c0a3830 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe217fa640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 0x7fbe1003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(218..218 src has 148..218) v4 ==== 14756+0+0 (secure 0 0 0) 0x7fbe1c04d730 con 0x7fbe2c0a3830 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe31139640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 0x7fbe1003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:52.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.060+0000 7fbe31139640 10 monclient: get_auth_request con 0x7fbe10038720 auth_method 0 2024-09-15T01:59:52.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.062+0000 7fbe31139640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 0x7fbe1003abd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009480 tx=0x7fbe280090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:52.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.065+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbe1c00e830 con 0x7fbe2c0a3830 2024-09-15T01:59:52.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.065+0000 7fbe217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:52.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.065+0000 7fbe217fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:52.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.233+0000 7fbe3339d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.233+0000 7fbe3339d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.233+0000 7fbe3339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.233+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7fbe2c09c160 con 0x7fbe2c0a3830 2024-09-15T01:59:52.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.856+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v219) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe1c017030 con 0x7fbe2c0a3830 2024-09-15T01:59:52.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.856+0000 7fbe217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.856+0000 7fbe217fa640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-15T01:59:52.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.961+0000 7fbe3339d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.961+0000 7fbe3339d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.961+0000 7fbe3339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:52.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.961+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7fbe2c04bf60 con 0x7fbe2c0a3830 2024-09-15T01:59:52.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.962+0000 7fbe217fa640 1 -- 172.21.15.67:0/1837729124 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v219) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe1c017020 con 0x7fbe2c0a3830 2024-09-15T01:59:52.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.962+0000 7fbe217fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T01:59:52.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.962+0000 7fbe217fa640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-15T01:59:52.966 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h2' does not appear in the crush map 2024-09-15T01:59:52.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 msgr2=0x7fbe1003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:52.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 0x7fbe1003abd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009480 tx=0x7fbe280090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 10 monclient: shutdown 2024-09-15T01:59:52.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 msgr2=0x7fbe2c097c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b5c0 tx=0x7fbe1c009120 comp rx=0 tx=0).stop 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 shutdown_connections 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe10038720 0x7fbe1003abd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2c0a3830 0x7fbe2c097c00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe2c136d00 0x7fbe2c098140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 --2- 172.21.15.67:0/1837729124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06bef0 0x7fbe2c09c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 >> 172.21.15.67:0/1837729124 conn(0x7fbe2c070fb0 msgr2=0x7fbe2c06c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 shutdown_connections 2024-09-15T01:59:52.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:52.964+0000 7fbe3339d640 1 -- 172.21.15.67:0/1837729124 wait complete. 2024-09-15T01:59:52.976 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:52.976 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h3 2024-09-15T01:59:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.050+0000 7f168c094640 10 monclient: get_monmap_and_config 2024-09-15T01:59:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.050+0000 7f168c094640 10 monclient: build_initial_monmap 2024-09-15T01:59:53.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.051+0000 7f168c094640 10 monclient: monmap: 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:53.052164+0000 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:53.052164+0000 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:53.054 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:53.055 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:53.055 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:53.055 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.051+0000 7f168c094640 1 Processor -- start 2024-09-15T01:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.051+0000 7f168c094640 1 -- start start 2024-09-15T01:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.051+0000 7f168c094640 10 monclient: init 2024-09-15T01:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 0x7f1684069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): picked mon.noname-c con 0x7f16840a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f168409eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f168409f2a0 0x7f1684132220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): picked mon.noname-a con 0x7f168409f2a0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168406a270 con 0x7f168409eea0 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168406f290 con 0x7f168409f2a0 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16840a0f20 con 0x7f16840a0b50 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168c094640 10 monclient(hunting): authenticate will time out at 1225.521296s 2024-09-15T01:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168962f640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f168409eea0 legacy=0x7f1684131b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f1689e30640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 msgr2=0x7f1684069340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f1689e30640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 0x7f1684069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f168409f2a0 0x7f1684132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f168409f2a0 0x7f1684132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43616/0 (socket says 172.21.15.67:43616) 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 1 -- 172.21.15.67:0/3968153403 learned_addr learned my addr 172.21.15.67:0/3968153403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): get_auth_request con 0x7f168409f2a0 auth_method 0 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:53.067 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 10 monclient(hunting): handle_auth_done global_id 104484 payload 274 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 1 -- 172.21.15.67:0/3968153403 >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 msgr2=0x7f1684069340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.052+0000 7f168a631640 1 --2- 172.21.15.67:0/3968153403 >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 0x7f1684069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 1 -- 172.21.15.67:0/3968153403 >> v1:172.21.15.67:6789/0 conn(0x7f168409eea0 legacy=0x7f1684131b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 10 monclient: _finish_hunting 0 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 1 monclient: found mon.noname-a 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.053+0000 7f168a631640 1 -- 172.21.15.67:0/3968153403 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840a6160 con 0x7f168409f2a0 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.055+0000 7f168a631640 1 -- 172.21.15.67:0/3968153403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f168409f2a0 msgr2=0x7f1684132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:53.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.055+0000 7f168a631640 1 --2- 172.21.15.67:0/3968153403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f168409f2a0 0x7f1684132220 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f168406a450 tx=0x7f167c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c835b2d05a097dfa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 1 -- 172.21.15.67:0/3968153403 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c032440 con 0x7f168409f2a0 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: dump: 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:53.069 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: _finish_auth 0 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 1 -- 172.21.15.67:0/3968153403 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c0325a0 con 0x7f168409f2a0 2024-09-15T01:59:53.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 1 -- 172.21.15.67:0/3968153403 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c032940 con 0x7f168409f2a0 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 5 monclient: authenticate success, global_id 104484 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 4 set_mon_vals no callback set 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:53.071 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f1688e2e640 10 monclient: dump: 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:53.072 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 10 monclient: shutdown 2024-09-15T01:59:53.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 -- 172.21.15.67:0/3968153403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f168409f2a0 msgr2=0x7f1684132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 --2- 172.21.15.67:0/3968153403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f168409f2a0 0x7f1684132220 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f168406a450 tx=0x7f167c0091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 -- 172.21.15.67:0/3968153403 shutdown_connections 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 --2- 172.21.15.67:0/3968153403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f168409f2a0 0x7f1684132220 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 --2- 172.21.15.67:0/3968153403 >> v2:172.21.15.67:3300/0 conn(0x7f16840a0b50 0x7f1684069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 -- 172.21.15.67:0/3968153403 >> 172.21.15.67:0/3968153403 conn(0x7f1684071290 msgr2=0x7f16840a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 -- 172.21.15.67:0/3968153403 shutdown_connections 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.056+0000 7f168c094640 1 -- 172.21.15.67:0/3968153403 wait complete. 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.057+0000 7f168c094640 10 monclient: build_initial_monmap 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.057+0000 7f168c094640 10 monclient: monmap: 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:53.074 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.057+0000 7f168c094640 1 Processor -- start 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.057+0000 7f168c094640 1 -- start start 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.057+0000 7f168c094640 10 monclient: init 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:53.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 0x7f1684141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): picked mon.noname-c con 0x7f168409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 0x7f1684134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): picked mon.noname-b con 0x7f168409f2a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): picked mon.noname-a con 0x7f16840a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684122090 con 0x7f16840a0b50 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841225c0 con 0x7f168409f2a0 2024-09-15T01:59:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840a1260 con 0x7f168409eea0 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168c094640 10 monclient(hunting): authenticate will time out at 1225.527221s 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168962f640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 msgr2=0x7f1684134e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168962f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 0x7f1684134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 0x7f1684141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43620/0 (socket says 172.21.15.67:43620) 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 -- 172.21.15.67:0/700759494 learned_addr learned my addr 172.21.15.67:0/700759494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): get_auth_request con 0x7f16840a0b50 auth_method 0 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 0x7f1684141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35726/0 (socket says 172.21.15.67:35726) 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): get_auth_request con 0x7f168409eea0 auth_method 0 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f1689e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient(hunting): handle_auth_done global_id 104487 payload 824 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 -- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 msgr2=0x7f1684141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:53.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 0x7f1684141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 -- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 msgr2=0x7f1684134e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 0x7f1684134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient: _finish_hunting 0 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 monclient: found mon.noname-a 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.058+0000 7f168a631640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c009450 con 0x7f16840a0b50 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168a631640 1 -- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 msgr2=0x7f168411e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168a631640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f167c02a720 tx=0x7f167c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c031c70 con 0x7f16840a0b50 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:53.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:53.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: dump: 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:53.081 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 5 monclient: authenticate success, global_id 104487 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 10 monclient: _renew_subs 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1684141ee0 con 0x7f16840a0b50 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c031dd0 con 0x7f16840a0b50 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c045620 con 0x7f16840a0b50 2024-09-15T01:59:53.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1681ffb640 10 monclient: dump: 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:53.083 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 10 monclient: _renew_subs 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16841210a0 con 0x7f16840a0b50 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.061+0000 7f1683fff640 4 set_mon_vals no callback set 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.062+0000 7f168c094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.062+0000 7f168c094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.062+0000 7f168c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.062+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168404bf60 con 0x7f16840a0b50 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.062+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f167c039070 con 0x7f16840a0b50 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.063+0000 7f1681ffb640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 0x7f167003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:53.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.063+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(219..219 src has 148..219) v4 ==== 14546+0+0 (secure 0 0 0) 0x7f167c06eb60 con 0x7f16840a0b50 2024-09-15T01:59:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.063+0000 7f1689e30640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 0x7f167003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.063+0000 7f1689e30640 10 monclient: get_auth_request con 0x7f16700386f0 auth_method 0 2024-09-15T01:59:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.065+0000 7f1689e30640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 0x7f167003aba0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1674003bd0 tx=0x7f167400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:53.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.067+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f167c03a300 con 0x7f16840a0b50 2024-09-15T01:59:53.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.067+0000 7f1681ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:53.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.067+0000 7f1681ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.234+0000 7f168c094640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.234+0000 7f168c094640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.234+0000 7f168c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.234+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f168404bf60 con 0x7f16840a0b50 2024-09-15T01:59:53.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.864+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v220) v1 ==== 118+0+0 (secure 0 0 0) 0x7f167c037020 con 0x7f16840a0b50 2024-09-15T01:59:53.867 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.864+0000 7f1681ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.867 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.864+0000 7f1681ffb640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-15T01:59:53.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.971+0000 7f168c094640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.971+0000 7f168c094640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.971+0000 7f168c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:53.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.971+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f168404bf60 con 0x7f16840a0b50 2024-09-15T01:59:53.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.972+0000 7f1681ffb640 1 -- 172.21.15.67:0/700759494 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v220) v1 ==== 118+0+0 (secure 0 0 0) 0x7f167c04c300 con 0x7f16840a0b50 2024-09-15T01:59:53.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.972+0000 7f1681ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T01:59:53.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.972+0000 7f1681ffb640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-15T01:59:53.974 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h3' does not appear in the crush map 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 msgr2=0x7f167003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 0x7f167003aba0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1674003bd0 tx=0x7f167400d040 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 10 monclient: shutdown 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 msgr2=0x7f168411e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f167c02a720 tx=0x7f167c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 shutdown_connections 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16700386f0 0x7f167003aba0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16840a0b50 0x7f168411e940 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f168409f2a0 0x7f1684134e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 --2- 172.21.15.67:0/700759494 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f168409eea0 0x7f1684141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:53.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 >> 172.21.15.67:0/700759494 conn(0x7f1684071290 msgr2=0x7f168409ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:53.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 shutdown_connections 2024-09-15T01:59:53.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:53.974+0000 7f168c094640 1 -- 172.21.15.67:0/700759494 wait complete. 2024-09-15T01:59:53.985 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:53.985 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h4 2024-09-15T01:59:54.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.059+0000 7fbf4f109640 10 monclient: get_monmap_and_config 2024-09-15T01:59:54.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.059+0000 7fbf4f109640 10 monclient: build_initial_monmap 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.059+0000 7fbf4f109640 10 monclient: monmap: 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:54.060448+0000 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:54.060448+0000 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:54.062 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:54.063 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:54.063 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.059+0000 7fbf4f109640 1 Processor -- start 2024-09-15T01:59:54.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 1 -- start start 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient: init 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf480a0b50 0x7fbf48069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-c con 0x7fbf480a0b50 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf4809eea0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf4809f2a0 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4806a270 con 0x7fbf4809f2a0 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480a0f20 con 0x7fbf4809eea0 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4806f290 con 0x7fbf480a0b50 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4f109640 10 monclient(hunting): authenticate will time out at 1226.529572s 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf47fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fbf4809eea0 legacy=0x7fbf48131b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4cea5640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fbf480a0b50 msgr2=0x7fbf48069340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4cea5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf480a0b50 0x7fbf48069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4d6a6640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbf4809f2a0 0x7fbf48132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56276/0 (socket says 172.21.15.67:56276) 2024-09-15T01:59:54.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.060+0000 7fbf4d6a6640 1 -- 172.21.15.67:0/461563232 learned_addr learned my addr 172.21.15.67:0/461563232 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf4d6a6640 1 -- 172.21.15.67:0/461563232 >> v1:172.21.15.17:6789/0 conn(0x7fbf4809f2a0 legacy=0x7fbf48132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298663505 0 0) 0x7fbf4806a270 con 0x7fbf4809f2a0 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): my global_id is 104490 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf380034e0 con 0x7fbf4809f2a0 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787588653 0 0) 0x7fbf380034e0 con 0x7fbf4809f2a0 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 >> v2:172.21.15.67:3300/0 conn(0x7fbf480a0b50 msgr2=0x7fbf48069340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 --2- 172.21.15.67:0/461563232 >> v2:172.21.15.67:3300/0 conn(0x7fbf480a0b50 0x7fbf48069340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 >> v1:172.21.15.67:6789/0 conn(0x7fbf4809eea0 legacy=0x7fbf48131b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:54.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf48132930 con 0x7fbf4809f2a0 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fbf40001aa0 con 0x7fbf4809f2a0 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: dump: 2024-09-15T01:59:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:54.079 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 >> v1:172.21.15.17:6789/0 conn(0x7fbf4809f2a0 legacy=0x7fbf48132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 0x7fbf38003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): picked mon.a con 0x7fbf38003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38004780 0x7fbf38004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): picked mon.c con 0x7fbf38004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380099c0 0x7fbf3800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): picked mon.b con 0x7fbf380099c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:54.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3800c330 con 0x7fbf38003a50 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3800c4a0 con 0x7fbf380099c0 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3800c660 con 0x7fbf38004780 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf477fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf4d6a6640 1 -- 172.21.15.67:0/461563232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380099c0 msgr2=0x7fbf3800bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.063+0000 7fbf4d6a6640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380099c0 0x7fbf3800bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 0x7fbf38003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:54.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): get_auth_request con 0x7fbf38003a50 auth_method 0 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38004780 0x7fbf38004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): get_auth_request con 0x7fbf38004780 auth_method 0 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:59:54.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_done global_id 104490 payload 293 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 -- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38004780 msgr2=0x7fbf38004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38004780 0x7fbf38004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 -- 172.21.15.67:0/461563232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380099c0 msgr2=0x7fbf3800bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380099c0 0x7fbf3800bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient: _finish_hunting 0 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 monclient: found mon.a 2024-09-15T01:59:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4d6a6640 1 -- 172.21.15.67:0/461563232 reap_dead start 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 1 -- 172.21.15.67:0/461563232 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380034e0 con 0x7fbf38003a50 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 10 monclient: _finish_auth 0 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4cea5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4f109640 5 monclient: authenticate success, global_id 104490 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.064+0000 7fbf4f109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf4cea5640 1 -- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 msgr2=0x7fbf38003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 0x7fbf38003e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098f0 tx=0x7fbf3c006ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10b764d8adda44a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c003b80 con 0x7fbf38003a50 2024-09-15T01:59:54.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: dump: 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c003ce0 con 0x7fbf38003a50 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf4f109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf4f109640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf4f109640 4 set_mon_vals no callback set 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 1 -- 172.21.15.67:0/461563232 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c02aac0 con 0x7fbf38003a50 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.067+0000 7fbf477fe640 10 monclient: dump: 2024-09-15T01:59:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 10 monclient: shutdown 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 -- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 msgr2=0x7fbf38003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 0x7fbf38003e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098f0 tx=0x7fbf3c006ed0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 -- 172.21.15.67:0/461563232 shutdown_connections 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 --2- 172.21.15.67:0/461563232 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf38003a50 0x7fbf38003e40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 -- 172.21.15.67:0/461563232 >> 172.21.15.67:0/461563232 conn(0x7fbf48071290 msgr2=0x7fbf480a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 -- 172.21.15.67:0/461563232 shutdown_connections 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 1 -- 172.21.15.67:0/461563232 wait complete. 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 10 monclient: build_initial_monmap 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.068+0000 7fbf4f109640 10 monclient: monmap: 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 Processor -- start 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 -- start start 2024-09-15T01:59:54.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient: init 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-c con 0x7fbf38003a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf48141320 0x7fbf48130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-a con 0x7fbf48141320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 0x7fbf4811d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): picked mon.noname-b con 0x7fbf48131330 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4811fc40 con 0x7fbf48141320 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4811f4a0 con 0x7fbf48131330 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480a1260 con 0x7fbf38003a50 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4f109640 10 monclient(hunting): authenticate will time out at 1226.538774s 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4d6a6640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 msgr2=0x7fbf4811d960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4d6a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 0x7fbf4811d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf44ff9640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:54.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35740/0 (socket says 172.21.15.67:35740) 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 1 -- 172.21.15.67:0/2255429847 learned_addr learned my addr 172.21.15.67:0/2255429847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 10 monclient(hunting): get_auth_request con 0x7fbf38003a50 auth_method 0 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf4cea5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf47fff640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf48141320 0x7fbf48130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf47fff640 10 monclient(hunting): get_auth_request con 0x7fbf48141320 auth_method 0 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.069+0000 7fbf47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:54.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient(hunting): handle_auth_done global_id 94643 payload 824 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 -- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 msgr2=0x7fbf4811d960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 0x7fbf4811d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 -- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf48141320 msgr2=0x7fbf48130df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:54.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf48141320 0x7fbf48130df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient: _finish_hunting 0 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 monclient: found mon.noname-c 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.070+0000 7fbf4cea5640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4809e9f0 con 0x7fbf38003a50 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4cea5640 1 -- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 msgr2=0x7fbf48140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4cea5640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009920 tx=0x7fbf3c030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c03b070 con 0x7fbf38003a50 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:54.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: dump: 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: _finish_auth 0 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 5 monclient: authenticate success, global_id 94643 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 10 monclient: _renew_subs 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf48120f40 con 0x7fbf38003a50 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c005560 con 0x7fbf38003a50 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0336c0 con 0x7fbf38003a50 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:54.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf44ff9640 10 monclient: dump: 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:54.097 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf46ffd640 4 set_mon_vals no callback set 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 10 monclient: _renew_subs 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.072+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4811ffe0 con 0x7fbf38003a50 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf4f109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf4f109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf4f109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbf3c03e070 con 0x7fbf38003a50 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4804bf60 con 0x7fbf38003a50 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.073+0000 7fbf44ff9640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 0x7fbf2c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.074+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(220..220 src has 148..220) v4 ==== 14336+0+0 (secure 0 0 0) 0x7fbf3c06d520 con 0x7fbf38003a50 2024-09-15T01:59:54.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.074+0000 7fbf47fff640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 0x7fbf2c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:54.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.074+0000 7fbf47fff640 10 monclient: get_auth_request con 0x7fbf2c0386f0 auth_method 0 2024-09-15T01:59:54.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.076+0000 7fbf47fff640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 0x7fbf2c03aba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbf48069ff0 tx=0x7fbf34009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:54.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.078+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbf3c030920 con 0x7fbf38003a50 2024-09-15T01:59:54.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.078+0000 7fbf44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:54.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.078+0000 7fbf44ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:54.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.244+0000 7fbf4f109640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.244+0000 7fbf4f109640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.244+0000 7fbf4f109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.244+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fbf4806f000 con 0x7fbf38003a50 2024-09-15T01:59:54.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.869+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v221) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbf3c045030 con 0x7fbf38003a50 2024-09-15T01:59:54.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.869+0000 7fbf44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.869+0000 7fbf44ff9640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-15T01:59:54.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.976+0000 7fbf4f109640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.976+0000 7fbf4f109640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.976+0000 7fbf4f109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:54.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.976+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fbf48141b90 con 0x7fbf38003a50 2024-09-15T01:59:54.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.977+0000 7fbf44ff9640 1 -- 172.21.15.67:0/2255429847 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v221) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbf3c045020 con 0x7fbf38003a50 2024-09-15T01:59:54.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.977+0000 7fbf44ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T01:59:54.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.977+0000 7fbf44ff9640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-15T01:59:54.980 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h4' does not appear in the crush map 2024-09-15T01:59:54.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 msgr2=0x7fbf2c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:54.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 0x7fbf2c03aba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbf48069ff0 tx=0x7fbf34009060 comp rx=0 tx=0).stop 2024-09-15T01:59:54.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 10 monclient: shutdown 2024-09-15T01:59:54.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 msgr2=0x7fbf48140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.979+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009920 tx=0x7fbf3c030000 comp rx=0 tx=0).stop 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 shutdown_connections 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf2c0386f0 0x7fbf2c03aba0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf48141320 0x7fbf48130df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf48131330 0x7fbf4811d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 --2- 172.21.15.67:0/2255429847 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf38003a50 0x7fbf48140de0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 >> 172.21.15.67:0/2255429847 conn(0x7fbf48071290 msgr2=0x7fbf48131cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 shutdown_connections 2024-09-15T01:59:54.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:54.980+0000 7fbf4f109640 1 -- 172.21.15.67:0/2255429847 wait complete. 2024-09-15T01:59:54.992 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:54.992 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h5 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.066+0000 7f9565b9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.066+0000 7f9565b9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.066+0000 7f9565b9d640 10 monclient: monmap: 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:55.068017+0000 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:55.068017+0000 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:55.070 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.067+0000 7f9565b9d640 1 Processor -- start 2024-09-15T01:59:55.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.067+0000 7f9565b9d640 1 -- start start 2024-09-15T01:59:55.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.067+0000 7f9565b9d640 10 monclient: init 2024-09-15T01:59:55.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f95600a1a00 0x7f956009aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f95600a1a00 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 0x7f956009b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f956009d5a0 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 0x7f9560096ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9560096a70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956009cf80 con 0x7f95600a1a00 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956009cc40 con 0x7f956009d5a0 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600a1dd0 con 0x7f9560096a70 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f9565b9d640 10 monclient(hunting): authenticate will time out at 1227.537133s 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955effd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 msgr2=0x7f956009b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 0x7f956009b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f95600a1a00 0x7f956009aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 0x7f9560096ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f95600a1a00 0x7f956009aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43648/0 (socket says 172.21.15.67:43648) 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 -- 172.21.15.67:0/2680717891 learned_addr learned my addr 172.21.15.67:0/2680717891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 0x7f9560096ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35742/0 (socket says 172.21.15.67:35742) 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): get_auth_request con 0x7f95600a1a00 auth_method 0 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f9560096a70 auth_method 0 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient(hunting): handle_auth_done global_id 104496 payload 274 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 -- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 msgr2=0x7f9560096ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 --2- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 0x7f9560096ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 -- 172.21.15.67:0/2680717891 >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 msgr2=0x7f956009b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 --2- 172.21.15.67:0/2680717891 >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 0x7f956009b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.068+0000 7f955f7fe640 1 -- 172.21.15.67:0/2680717891 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956012efa0 con 0x7f95600a1a00 2024-09-15T01:59:55.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.071+0000 7f955f7fe640 1 -- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f95600a1a00 msgr2=0x7f956009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:55.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.071+0000 7f955f7fe640 1 --2- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f95600a1a00 0x7f956009aae0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f95600a2080 tx=0x7f95500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9041d87364ffcb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:55.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 1 -- 172.21.15.67:0/2680717891 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550032440 con 0x7f95600a1a00 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: dump: 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:55.083 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: _finish_auth 0 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 1 -- 172.21.15.67:0/2680717891 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95500325a0 con 0x7f95600a1a00 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 5 monclient: authenticate success, global_id 104496 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 1 -- 172.21.15.67:0/2680717891 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550032940 con 0x7f95600a1a00 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 4 set_mon_vals no callback set 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f955e7fc640 10 monclient: dump: 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:55.084 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 10 monclient: shutdown 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 -- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f95600a1a00 msgr2=0x7f956009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 --2- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f95600a1a00 0x7f956009aae0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f95600a2080 tx=0x7f95500091d0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 -- 172.21.15.67:0/2680717891 shutdown_connections 2024-09-15T01:59:55.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 --2- 172.21.15.67:0/2680717891 >> v2:172.21.15.67:3300/0 conn(0x7f956009d5a0 0x7f956009b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 --2- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9560096a70 0x7f9560096ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 --2- 172.21.15.67:0/2680717891 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f95600a1a00 0x7f956009aae0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 -- 172.21.15.67:0/2680717891 >> 172.21.15.67:0/2680717891 conn(0x7f9560071340 msgr2=0x7f95600a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 -- 172.21.15.67:0/2680717891 shutdown_connections 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.072+0000 7f9565b9d640 1 -- 172.21.15.67:0/2680717891 wait complete. 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.073+0000 7f9565b9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.073+0000 7f9565b9d640 10 monclient: monmap: 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.086 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.073+0000 7f9565b9d640 1 Processor -- start 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.073+0000 7f9565b9d640 1 -- start start 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.073+0000 7f9565b9d640 10 monclient: init 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9560096a70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 0x7f9560135010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f956009d5a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95600a1a00 0x7f95601410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f95600a1a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9560130cf0 con 0x7f9560096a70 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600985b0 con 0x7f956009d5a0 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600a09c0 con 0x7f95600a1a00 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f9565b9d640 10 monclient(hunting): authenticate will time out at 1227.543087s 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955effd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 msgr2=0x7f9560135010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 0x7f9560135010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95600a1a00 0x7f95601410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:55.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43652/0 (socket says 172.21.15.67:43652) 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 -- 172.21.15.67:0/3025729585 learned_addr learned my addr 172.21.15.67:0/3025729585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): get_auth_request con 0x7f9560096a70 auth_method 0 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f95600a1a00 auth_method 0 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:55.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient(hunting): handle_auth_done global_id 104499 payload 824 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 -- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95600a1a00 msgr2=0x7f95601410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95600a1a00 0x7f95601410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 -- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 msgr2=0x7f9560135010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 0x7f9560135010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient: _finish_hunting 0 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 monclient: found mon.noname-a 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.074+0000 7f955f7fe640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550009450 con 0x7f9560096a70 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f955f7fe640 1 -- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 msgr2=0x7f9560130440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f955f7fe640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9550009210 tx=0x7f9550032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500319c0 con 0x7f9560096a70 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: dump: 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: _finish_auth 0 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550031b40 con 0x7f9560096a70 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 5 monclient: authenticate success, global_id 104499 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 10 monclient: _renew_subs 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955003d440 con 0x7f9560096a70 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9560132510 con 0x7f9560096a70 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:55.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f955dffb640 4 set_mon_vals no callback set 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f95477fe640 10 monclient: dump: 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 10 monclient: _renew_subs 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.077+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956006e230 con 0x7f9560096a70 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9550039040 con 0x7f9560096a70 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f9565b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f9565b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f9565b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956004bf60 con 0x7f9560096a70 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f95477fe640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 0x7f954c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(221..221 src has 148..221) v4 ==== 14126+0+0 (secure 0 0 0) 0x7f955006d9e0 con 0x7f9560096a70 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f955effd640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 0x7f954c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:55.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.078+0000 7f955effd640 10 monclient: get_auth_request con 0x7f954c038720 auth_method 0 2024-09-15T01:59:55.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.081+0000 7f955effd640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 0x7f954c03abd0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f956006eb90 tx=0x7f9554006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:55.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.083+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f955002f2a0 con 0x7f9560096a70 2024-09-15T01:59:55.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.083+0000 7f95477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:55.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.083+0000 7f95477fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:55.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.249+0000 7f9565b9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.249+0000 7f9565b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.249+0000 7f9565b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.249+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f9560067180 con 0x7f9560096a70 2024-09-15T01:59:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.873+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9550042030 con 0x7f9560096a70 2024-09-15T01:59:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.873+0000 7f95477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.873+0000 7f95477fe640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-15T01:59:55.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.980+0000 7f9565b9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.980+0000 7f9565b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.980+0000 7f9565b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.980+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f956012fd80 con 0x7f9560096a70 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.981+0000 7f95477fe640 1 -- 172.21.15.67:0/3025729585 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9550042030 con 0x7f9560096a70 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.981+0000 7f95477fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.981+0000 7f95477fe640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-15T01:59:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h5' does not appear in the crush map 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 msgr2=0x7f954c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 0x7f954c03abd0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f956006eb90 tx=0x7f9554006d80 comp rx=0 tx=0).stop 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 10 monclient: shutdown 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 msgr2=0x7f9560130440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9550009210 tx=0x7f9550032000 comp rx=0 tx=0).stop 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 shutdown_connections 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f954c038720 0x7f954c03abd0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95600a1a00 0x7f95601410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956009d5a0 0x7f9560135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 --2- 172.21.15.67:0/3025729585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9560096a70 0x7f9560130440 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:55.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 >> 172.21.15.67:0/3025729585 conn(0x7f9560071340 msgr2=0x7f95600a0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:55.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 shutdown_connections 2024-09-15T01:59:55.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:55.983+0000 7f9565b9d640 1 -- 172.21.15.67:0/3025729585 wait complete. 2024-09-15T01:59:55.995 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:55.995 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h6 2024-09-15T01:59:56.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.070+0000 7f4cbbb9d640 10 monclient: get_monmap_and_config 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.070+0000 7f4cbbb9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.070+0000 7f4cbbb9d640 10 monclient: monmap: 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:56.071691+0000 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:56.071691+0000 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:56.074 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:56.074 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:56.074 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:56.074 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:56.074 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 Processor -- start 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 -- start start 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient: init 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cb40a3830 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cb406b9b0 0x7f4cb4136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb406b9b0 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 0x7f4cb4139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb4137240 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb4069c60 con 0x7f4cb40a3830 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40a3c00 con 0x7f4cb406b9b0 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4069650 con 0x7f4cb4137240 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cbbb9d640 10 monclient(hunting): authenticate will time out at 1228.540801s 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.071+0000 7f4cb9939640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f4cb40a3830 legacy=0x7f4cb406e2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 0x7f4cb4139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cb406b9b0 0x7f4cb4136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 0x7f4cb4139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35762/0 (socket says 172.21.15.67:35762) 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 1 -- 172.21.15.67:0/801445022 learned_addr learned my addr 172.21.15.67:0/801445022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4cb406b9b0 0x7f4cb4136d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43664/0 (socket says 172.21.15.67:43664) 2024-09-15T01:59:56.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): get_auth_request con 0x7f4cb4137240 auth_method 0 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): get_auth_request con 0x7f4cb406b9b0 auth_method 0 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:56.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient(hunting): handle_auth_done global_id 104502 payload 274 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 -- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 msgr2=0x7f4cb4139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 --2- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 0x7f4cb4139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 -- 172.21.15.67:0/801445022 >> v1:172.21.15.67:6789/0 conn(0x7f4cb40a3830 legacy=0x7f4cb406e2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient: _finish_hunting 0 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 monclient: found mon.noname-a 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.072+0000 7f4cb9138640 1 -- 172.21.15.67:0/801445022 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb409f500 con 0x7f4cb406b9b0 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.075+0000 7f4cb9138640 1 -- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb406b9b0 msgr2=0x7f4cb4136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.075+0000 7f4cb9138640 1 --2- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4136d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009850 tx=0x7f4ca40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53a34bfabd54d2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 1 -- 172.21.15.67:0/801445022 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca402ae80 con 0x7f4cb406b9b0 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: dump: 2024-09-15T01:59:56.087 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: _finish_auth 0 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 1 -- 172.21.15.67:0/801445022 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4032440 con 0x7f4cb406b9b0 2024-09-15T01:59:56.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 1 -- 172.21.15.67:0/801445022 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40327c0 con 0x7f4cb406b9b0 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 5 monclient: authenticate success, global_id 104502 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 4 set_mon_vals no callback set 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cb8937640 10 monclient: dump: 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:56.089 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 10 monclient: shutdown 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb406b9b0 msgr2=0x7f4cb4136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4136d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009850 tx=0x7f4ca40090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/801445022 shutdown_connections 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb4137240 0x7f4cb4139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/801445022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4136d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/801445022 >> 172.21.15.67:0/801445022 conn(0x7f4cb4070ff0 msgr2=0x7f4cb40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/801445022 shutdown_connections 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.076+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/801445022 wait complete. 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.077+0000 7f4cbbb9d640 10 monclient: build_initial_monmap 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.077+0000 7f4cbbb9d640 10 monclient: monmap: 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.077+0000 7f4cbbb9d640 1 Processor -- start 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.077+0000 7f4cbbb9d640 1 -- start start 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.077+0000 7f4cbbb9d640 10 monclient: init 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb406b9b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 0x7f4cb4135210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb40a3830 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb4137240 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4142580 con 0x7f4cb4137240 2024-09-15T01:59:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb411f830 con 0x7f4cb406b9b0 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40a2900 con 0x7f4cb40a3830 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9939640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 msgr2=0x7f4cb4134cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cbbb9d640 10 monclient(hunting): authenticate will time out at 1228.547170s 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9939640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4134cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 0x7f4cb4135210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 0x7f4cb4135210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35766/0 (socket says 172.21.15.67:35766) 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43676/0 (socket says 172.21.15.67:43676) 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 1 -- 172.21.15.67:0/2678011557 learned_addr learned my addr 172.21.15.67:0/2678011557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): get_auth_request con 0x7f4cb40a3830 auth_method 0 2024-09-15T01:59:56.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): get_auth_request con 0x7f4cb4137240 auth_method 0 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cb9138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient(hunting): handle_auth_done global_id 104505 payload 824 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 -- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 msgr2=0x7f4cb4135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 0x7f4cb4135210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 -- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 msgr2=0x7f4cb4134cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4134cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient: _finish_hunting 0 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 monclient: found mon.noname-a 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.078+0000 7f4cba13a640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4009390 con 0x7f4cb4137240 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cba13a640 1 -- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 msgr2=0x7f4cb411db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cba13a640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009e50 tx=0x7f4cac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac009c30 con 0x7f4cb4137240 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:56.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: dump: 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:56.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac00f040 con 0x7f4cb4137240 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 5 monclient: authenticate success, global_id 104505 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 10 monclient: _renew_subs 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4099e80 con 0x7f4cb4137240 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac014440 con 0x7f4cb4137240 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4ca9ffb640 10 monclient: dump: 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cabfff640 4 set_mon_vals no callback set 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 10 monclient: _renew_subs 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.081+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb411f2b0 con 0x7f4cb4137240 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4cac0147b0 con 0x7f4cb4137240 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cbbb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cbbb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cbbb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb404bf60 con 0x7f4cb4137240 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4ca9ffb640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 0x7f4c9803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(222..222 src has 148..222) v4 ==== 13916+0+0 (secure 0 0 0) 0x7f4cac04e3d0 con 0x7f4cb4137240 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cb9939640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 0x7f4c9803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.082+0000 7f4cb9939640 10 monclient: get_auth_request con 0x7f4c980386f0 auth_method 0 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.085+0000 7f4cb9939640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 0x7f4c9803aba0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.087+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4cac01c360 con 0x7f4cb4137240 2024-09-15T01:59:56.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.087+0000 7f4ca9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:56.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.087+0000 7f4ca9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:56.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.253+0000 7f4cbbb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.253+0000 7f4cbbb9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.253+0000 7f4cbbb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.253+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f4cb406c070 con 0x7f4cb4137240 2024-09-15T01:59:56.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.878+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4cac012e60 con 0x7f4cb4137240 2024-09-15T01:59:56.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.878+0000 7f4ca9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.878+0000 7f4ca9ffb640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-15T01:59:56.985 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.983+0000 7f4cbbb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.983+0000 7f4cbbb9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.983+0000 7f4cbbb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:56.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.983+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f4cb409f980 con 0x7f4cb4137240 2024-09-15T01:59:56.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.984+0000 7f4ca9ffb640 1 -- 172.21.15.67:0/2678011557 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4cac01be90 con 0x7f4cb4137240 2024-09-15T01:59:56.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.984+0000 7f4ca9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T01:59:56.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.984+0000 7f4ca9ffb640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-15T01:59:56.987 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h6' does not appear in the crush map 2024-09-15T01:59:56.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 msgr2=0x7f4c9803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 0x7f4c9803aba0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006d30 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 10 monclient: shutdown 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 msgr2=0x7f4cb411db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009e50 tx=0x7f4cac00b040 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 shutdown_connections 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4c980386f0 0x7f4c9803aba0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4cb4137240 0x7f4cb411db90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4cb40a3830 0x7f4cb4135210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 --2- 172.21.15.67:0/2678011557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb406b9b0 0x7f4cb4134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 >> 172.21.15.67:0/2678011557 conn(0x7f4cb4070ff0 msgr2=0x7f4cb406dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:56.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 shutdown_connections 2024-09-15T01:59:56.990 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:56.986+0000 7f4cbbb9d640 1 -- 172.21.15.67:0/2678011557 wait complete. 2024-09-15T01:59:56.998 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:56.998 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h7 2024-09-15T01:59:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.072+0000 7f4bcf620640 10 monclient: get_monmap_and_config 2024-09-15T01:59:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.072+0000 7f4bcf620640 10 monclient: build_initial_monmap 2024-09-15T01:59:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.072+0000 7f4bcf620640 10 monclient: monmap: 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:57.073500+0000 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:57.073500+0000 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:57.075 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.072+0000 7f4bcf620640 1 Processor -- start 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 -- start start 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient: init 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bc80a0b50 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:57.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 0x7f4bc8139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc8136e40 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80a61a0 con 0x7f4bc80a0b50 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80a1610 con 0x7f4bc8136e40 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc809f7d0 con 0x7f4bc809eea0 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcf620640 10 monclient(hunting): authenticate will time out at 1229.542630s 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcdbbd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 msgr2=0x7f4bc8139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.073+0000 7f4bcdbbd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 0x7f4bc8139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bcd3bc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4bc80a0b50 0x7f4bc8069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56280/0 (socket says 172.21.15.67:56280) 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35770/0 (socket says 172.21.15.67:35770) 2024-09-15T01:59:57.079 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bcd3bc640 1 -- 172.21.15.67:0/1098537309 learned_addr learned my addr 172.21.15.67:0/1098537309 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): get_auth_request con 0x7f4bc809eea0 auth_method 0 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_done global_id 94658 payload 274 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 -- 172.21.15.67:0/1098537309 >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 msgr2=0x7f4bc8139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 --2- 172.21.15.67:0/1098537309 >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 0x7f4bc8139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 -- 172.21.15.67:0/1098537309 >> v1:172.21.15.17:6789/0 conn(0x7f4bc80a0b50 legacy=0x7f4bc8069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient: _finish_hunting 0 2024-09-15T01:59:57.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 monclient: found mon.noname-b 2024-09-15T01:59:57.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:57.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.074+0000 7f4bccbbb640 1 -- 172.21.15.67:0/1098537309 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80a04c0 con 0x7f4bc809eea0 2024-09-15T01:59:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bccbbb640 1 -- 172.21.15.67:0/1098537309 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 msgr2=0x7f4bc809f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bccbbb640 1 --2- 172.21.15.67:0/1098537309 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009850 tx=0x7f4bb80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e25329d8fa77de04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 1 -- 172.21.15.67:0/1098537309 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8032440 con 0x7f4bc809eea0 2024-09-15T01:59:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:57.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: dump: 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: _finish_auth 0 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 1 -- 172.21.15.67:0/1098537309 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb80325a0 con 0x7f4bc809eea0 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 5 monclient: authenticate success, global_id 94658 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 1 -- 172.21.15.67:0/1098537309 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8032940 con 0x7f4bc809eea0 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 4 set_mon_vals no callback set 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bbffff640 10 monclient: dump: 2024-09-15T01:59:57.087 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 10 monclient: shutdown 2024-09-15T01:59:57.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 1 -- 172.21.15.67:0/1098537309 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 msgr2=0x7f4bc809f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.077+0000 7f4bcf620640 1 --2- 172.21.15.67:0/1098537309 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009850 tx=0x7f4bb80090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 -- 172.21.15.67:0/1098537309 shutdown_connections 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 --2- 172.21.15.67:0/1098537309 >> v2:172.21.15.67:3300/0 conn(0x7f4bc8136e40 0x7f4bc8139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 --2- 172.21.15.67:0/1098537309 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809f290 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 -- 172.21.15.67:0/1098537309 >> 172.21.15.67:0/1098537309 conn(0x7f4bc8071330 msgr2=0x7f4bc80a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 -- 172.21.15.67:0/1098537309 shutdown_connections 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 -- 172.21.15.67:0/1098537309 wait complete. 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 10 monclient: build_initial_monmap 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 10 monclient: monmap: 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:57.089 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 Processor -- start 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 1 -- start start 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.078+0000 7f4bcf620640 10 monclient: init 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:57.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc809eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 0x7f4bc8131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc80a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 0x7f4bc811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc8136e40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc809b9b0 con 0x7f4bc80a0b50 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80a1260 con 0x7f4bc8136e40 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:57.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80a3a70 con 0x7f4bc809eea0 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcf620640 10 monclient(hunting): authenticate will time out at 1229.548558s 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcdbbd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 msgr2=0x7f4bc811b940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcdbbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 0x7f4bc811b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcd3bc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 0x7f4bc8131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 0x7f4bc8131bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:43690/0 (socket says 172.21.15.67:43690) 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 1 -- 172.21.15.67:0/2192521803 learned_addr learned my addr 172.21.15.67:0/2192521803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 10 monclient(hunting): get_auth_request con 0x7f4bc80a0b50 auth_method 0 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:57.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bccbbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcd3bc640 10 monclient(hunting): get_auth_request con 0x7f4bc809eea0 auth_method 0 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcd3bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcd3bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.079+0000 7f4bcd3bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bccbbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient(hunting): handle_auth_done global_id 94661 payload 824 2024-09-15T01:59:57.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 -- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 msgr2=0x7f4bc811b940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 0x7f4bc811b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 -- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 msgr2=0x7f4bc8131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 0x7f4bc8131bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient: _finish_hunting 0 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 monclient: found mon.noname-c 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.080+0000 7f4bcd3bc640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009390 con 0x7f4bc809eea0 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcd3bc640 1 -- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 msgr2=0x7f4bc809a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcd3bc640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809a8b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c490 tx=0x7f4bc4007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4005cb0 con 0x7f4bc809eea0 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: dump: 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: _finish_auth 0 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:57.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc4005e30 con 0x7f4bc809eea0 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 5 monclient: authenticate success, global_id 94661 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 10 monclient: _renew_subs 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc811ce10 con 0x7f4bc809eea0 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4007300 con 0x7f4bc809eea0 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbd7fa640 10 monclient: dump: 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.096 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 10 monclient: _renew_subs 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc809b430 con 0x7f4bc809eea0 2024-09-15T01:59:57.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.082+0000 7f4bbf7fe640 4 set_mon_vals no callback set 2024-09-15T01:59:57.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bcf620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:57.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bcf620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bcf620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc811e030 con 0x7f4bc809eea0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4bc401b3b0 con 0x7f4bc809eea0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.083+0000 7f4bbd7fa640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 0x7f4bac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.084+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(223..223 src has 148..223) v4 ==== 13706+0+0 (secure 0 0 0) 0x7f4bc404d280 con 0x7f4bc809eea0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.084+0000 7f4bccbbb640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 0x7f4bac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.084+0000 7f4bccbbb640 10 monclient: get_auth_request con 0x7f4bac038720 auth_method 0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.086+0000 7f4bccbbb640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 0x7f4bac03abd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009130 tx=0x7f4bb8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.088+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4bc4018650 con 0x7f4bc809eea0 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.088+0000 7f4bbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:57.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.088+0000 7f4bbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:57.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.255+0000 7f4bcf620640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.256+0000 7f4bcf620640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.256+0000 7f4bcf620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.256+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f4bc804bf60 con 0x7f4bc809eea0 2024-09-15T01:59:57.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.801+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4bc4017070 con 0x7f4bc809eea0 2024-09-15T01:59:57.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.801+0000 7f4bbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.803 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.801+0000 7f4bbd7fa640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-15T01:59:57.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.904+0000 7f4bcf620640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.904+0000 7f4bcf620640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.904+0000 7f4bcf620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:57.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.904+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f4bc80008d0 con 0x7f4bc809eea0 2024-09-15T01:59:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.906+0000 7f4bbd7fa640 1 -- 172.21.15.67:0/2192521803 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4bc400e290 con 0x7f4bc809eea0 2024-09-15T01:59:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.906+0000 7f4bbd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T01:59:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.906+0000 7f4bbd7fa640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-15T01:59:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h7' does not appear in the crush map 2024-09-15T01:59:57.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 msgr2=0x7f4bac03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 0x7f4bac03abd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009130 tx=0x7f4bb8032000 comp rx=0 tx=0).stop 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 10 monclient: shutdown 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 msgr2=0x7f4bc809a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809a8b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c490 tx=0x7f4bc4007f00 comp rx=0 tx=0).stop 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 shutdown_connections 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4bac038720 0x7f4bac03abd0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4bc80a0b50 0x7f4bc8131bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc8136e40 0x7f4bc811b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 --2- 172.21.15.67:0/2192521803 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4bc809eea0 0x7f4bc809a8b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:57.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 >> 172.21.15.67:0/2192521803 conn(0x7f4bc8071330 msgr2=0x7f4bc809dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:57.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 shutdown_connections 2024-09-15T01:59:57.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.908+0000 7f4bcf620640 1 -- 172.21.15.67:0/2192521803 wait complete. 2024-09-15T01:59:57.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:57.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h8 2024-09-15T01:59:57.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.993+0000 7f7c23f54640 10 monclient: get_monmap_and_config 2024-09-15T01:59:57.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.993+0000 7f7c23f54640 10 monclient: build_initial_monmap 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.993+0000 7f7c23f54640 10 monclient: monmap: 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:57.994595+0000 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:57.994595+0000 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:57.997 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:57.997 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 Processor -- start 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 -- start start 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient: init 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c1c068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7c1c069190 0x7f7c1c069590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1c069190 addr v2:172.21.15.17:3300/0 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c1c137150 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:57.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c096c10 con 0x7f7c1c137150 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c132960 con 0x7f7c1c068dc0 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c0a14d0 con 0x7f7c1c069190 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c23f54640 10 monclient(hunting): authenticate will time out at 1230.463737s 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.994+0000 7f7c21cf0640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f7c1c068dc0 legacy=0x7f7c1c132250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c224f1640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7c1c137150 0x7f7c1c139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56286/0 (socket says 172.21.15.67:56286) 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c224f1640 1 -- 172.21.15.67:0/1464830666 learned_addr learned my addr 172.21.15.67:0/1464830666 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 1 --2- 172.21.15.67:0/1464830666 >> v2:172.21.15.17:3300/0 conn(0x7f7c1c069190 0x7f7c1c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): get_auth_request con 0x7f7c1c069190 auth_method 0 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient(hunting): handle_auth_done global_id 104511 payload 274 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 1 -- 172.21.15.67:0/1464830666 >> v1:172.21.15.67:6789/0 conn(0x7f7c1c068dc0 legacy=0x7f7c1c132250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 1 -- 172.21.15.67:0/1464830666 >> v1:172.21.15.17:6789/0 conn(0x7f7c1c137150 legacy=0x7f7c1c139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient: _finish_hunting 0 2024-09-15T01:59:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 1 monclient: found mon.noname-a 2024-09-15T01:59:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.995+0000 7f7c214ef640 1 -- 172.21.15.67:0/1464830666 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c09ff20 con 0x7f7c1c069190 2024-09-15T01:59:58.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c214ef640 1 -- 172.21.15.67:0/1464830666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c069190 msgr2=0x7f7c1c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c214ef640 1 --2- 172.21.15.67:0/1464830666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c069190 0x7f7c1c069590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009850 tx=0x7f7c0c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3974dc6f68995dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 1 -- 172.21.15.67:0/1464830666 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c032440 con 0x7f7c1c069190 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: dump: 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.008 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: _finish_auth 0 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c23f54640 5 monclient: authenticate success, global_id 104511 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c23f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 1 -- 172.21.15.67:0/1464830666 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c0325a0 con 0x7f7c1c069190 2024-09-15T01:59:58.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c23f54640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c23f54640 4 set_mon_vals no callback set 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 1 -- 172.21.15.67:0/1464830666 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c032940 con 0x7f7c1c069190 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.998+0000 7f7c20cee640 10 monclient: dump: 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:58.010 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 10 monclient: shutdown 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 -- 172.21.15.67:0/1464830666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c069190 msgr2=0x7f7c1c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 --2- 172.21.15.67:0/1464830666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c069190 0x7f7c1c069590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009850 tx=0x7f7c0c0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 -- 172.21.15.67:0/1464830666 shutdown_connections 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 --2- 172.21.15.67:0/1464830666 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c069190 0x7f7c1c069590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 -- 172.21.15.67:0/1464830666 >> 172.21.15.67:0/1464830666 conn(0x7f7c1c0711b0 msgr2=0x7f7c1c0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 -- 172.21.15.67:0/1464830666 shutdown_connections 2024-09-15T01:59:58.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 1 -- 172.21.15.67:0/1464830666 wait complete. 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 10 monclient: build_initial_monmap 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:57.999+0000 7f7c23f54640 10 monclient: monmap: 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:58.012 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 Processor -- start 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 -- start start 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient: init 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1c068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 0x7f7c1c12f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-b con 0x7f7c1c069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c137150 0x7f7c1c097fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1c137150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:58.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c09b720 con 0x7f7c1c137150 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c09bc50 con 0x7f7c1c069190 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c0a0890 con 0x7f7c1c068dc0 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c23f54640 10 monclient(hunting): authenticate will time out at 1230.469788s 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c214ef640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 msgr2=0x7f7c1c12f050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c214ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 0x7f7c1c12f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c21cf0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c21cf0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35790/0 (socket says 172.21.15.67:35790) 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c21cf0640 1 -- 172.21.15.67:0/3887940056 learned_addr learned my addr 172.21.15.67:0/3887940056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.000+0000 7f7c224f1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c137150 0x7f7c1c097fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): get_auth_request con 0x7f7c1c068dc0 auth_method 0 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): get_auth_request con 0x7f7c1c137150 auth_method 0 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c224f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:58.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient(hunting): handle_auth_done global_id 94664 payload 824 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 -- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 msgr2=0x7f7c1c12f050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 0x7f7c1c12f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 -- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c137150 msgr2=0x7f7c1c097fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c137150 0x7f7c1c097fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient: _finish_hunting 0 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 monclient: found mon.noname-c 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.001+0000 7f7c21cf0640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c009390 con 0x7f7c1c068dc0 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c21cf0640 1 -- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 msgr2=0x7f7c1c12eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c21cf0640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800b620 tx=0x7f7c18009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:58.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18015070 con 0x7f7c1c068dc0 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 10 monclient: dump: 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:58.017 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 10 monclient: _finish_auth 0 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c23f54640 5 monclient: authenticate success, global_id 94664 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c23f54640 10 monclient: _renew_subs 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c23f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c09a000 con 0x7f7c1c068dc0 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.003+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c18006150 con 0x7f7c1c068dc0 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c180062c0 con 0x7f7c1c068dc0 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 10 monclient: dump: 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:58.018 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 10 monclient: _renew_subs 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c13fff640 4 set_mon_vals no callback set 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c142370 con 0x7f7c1c068dc0 2024-09-15T01:59:58.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7c180066b0 con 0x7f7c1c068dc0 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.004+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c131a10 con 0x7f7c1c068dc0 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.005+0000 7f7c11ffb640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 0x7f7c0003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.005+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(224..224 src has 148..224) v4 ==== 13496+0+0 (secure 0 0 0) 0x7f7c1804d5f0 con 0x7f7c1c068dc0 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.005+0000 7f7c214ef640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 0x7f7c0003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:58.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.005+0000 7f7c214ef640 10 monclient: get_auth_request con 0x7f7c000386f0 auth_method 0 2024-09-15T01:59:58.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.007+0000 7f7c214ef640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 0x7f7c0003aba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c034000 tx=0x7f7c0c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:58.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.009+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7c18018af0 con 0x7f7c1c068dc0 2024-09-15T01:59:58.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.009+0000 7f7c11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:58.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.009+0000 7f7c11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.180+0000 7f7c23f54640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.180+0000 7f7c23f54640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.180+0000 7f7c23f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.180+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f7c1c09a780 con 0x7f7c1c068dc0 2024-09-15T01:59:58.888 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.886+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7c18018310 con 0x7f7c1c068dc0 2024-09-15T01:59:58.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.886+0000 7f7c11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.886+0000 7f7c11ffb640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-15T01:59:58.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.992+0000 7f7c23f54640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.992+0000 7f7c23f54640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.992+0000 7f7c23f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T01:59:58.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.992+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f7c1c04bf60 con 0x7f7c1c068dc0 2024-09-15T01:59:58.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.994+0000 7f7c11ffb640 1 -- 172.21.15.67:0/3887940056 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7c180184f0 con 0x7f7c1c068dc0 2024-09-15T01:59:58.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.994+0000 7f7c11ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T01:59:58.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.994+0000 7f7c11ffb640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-15T01:59:58.996 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h8' does not appear in the crush map 2024-09-15T01:59:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 msgr2=0x7f7c0003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 0x7f7c0003aba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c034000 tx=0x7f7c0c032000 comp rx=0 tx=0).stop 2024-09-15T01:59:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 10 monclient: shutdown 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 msgr2=0x7f7c1c12eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800b620 tx=0x7f7c18009120 comp rx=0 tx=0).stop 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 shutdown_connections 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7c000386f0 0x7f7c0003aba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7c1c137150 0x7f7c1c097fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1c069190 0x7f7c1c12f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 --2- 172.21.15.67:0/3887940056 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7c1c068dc0 0x7f7c1c12eb10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 >> 172.21.15.67:0/3887940056 conn(0x7f7c1c0711b0 msgr2=0x7f7c1c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:58.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 shutdown_connections 2024-09-15T01:59:59.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:58.996+0000 7f7c23f54640 1 -- 172.21.15.67:0/3887940056 wait complete. 2024-09-15T01:59:59.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T01:59:59.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h9 2024-09-15T01:59:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 10 monclient: get_monmap_and_config 2024-09-15T01:59:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 10 monclient: build_initial_monmap 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 10 monclient: monmap: 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:59:59.083202+0000 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:59:59.083202+0000 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T01:59:59.085 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 1 Processor -- start 2024-09-15T01:59:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 1 -- start start 2024-09-15T01:59:59.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.082+0000 7ff909080640 10 monclient: init 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 0x7ff9041321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): picked mon.noname-c con 0x7ff904098140 addr v2:172.21.15.67:3300/0 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9040a0370 addr v1:172.21.15.17:6789/0 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): picked mon.noname-b con 0x7ff904098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:59.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040a0b80 con 0x7ff9040a0370 2024-09-15T01:59:59.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:59.090 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9040a1730 con 0x7ff904098140 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040a00b0 con 0x7ff904098c20 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff909080640 10 monclient(hunting): authenticate will time out at 1231.552186s 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff902d9d640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 msgr2=0x7ff9041321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff902d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 0x7ff9041321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90259c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff9040a0370 0x7ff904098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56296/0 (socket says 172.21.15.67:56296) 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35806/0 (socket says 172.21.15.67:35806) 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90259c640 1 -- 172.21.15.67:0/1607052663 learned_addr learned my addr 172.21.15.67:0/1607052663 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): get_auth_request con 0x7ff904098c20 auth_method 0 2024-09-15T01:59:59.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient(hunting): handle_auth_done global_id 94667 payload 274 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 -- 172.21.15.67:0/1607052663 >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 msgr2=0x7ff9041321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 --2- 172.21.15.67:0/1607052663 >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 0x7ff9041321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 -- 172.21.15.67:0/1607052663 >> v1:172.21.15.17:6789/0 conn(0x7ff9040a0370 legacy=0x7ff904098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient: _finish_hunting 0 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 monclient: found mon.noname-b 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T01:59:59.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.083+0000 7ff90359e640 1 -- 172.21.15.67:0/1607052663 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff904071390 con 0x7ff904098c20 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff90359e640 1 -- 172.21.15.67:0/1607052663 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 msgr2=0x7ff9041390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff90359e640 1 --2- 172.21.15.67:0/1607052663 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009850 tx=0x7ff8fc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145add41617ccff7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 1 -- 172.21.15.67:0/1607052663 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc032440 con 0x7ff904098c20 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 10 monclient: dump: 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:59.098 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 10 monclient: _finish_auth 0 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff901d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.086+0000 7ff909080640 5 monclient: authenticate success, global_id 94667 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 1 -- 172.21.15.67:0/1607052663 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc0325a0 con 0x7ff904098c20 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 10 monclient: get_monmap_and_config success 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 4 set_mon_vals no callback set 2024-09-15T01:59:59.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 1 -- 172.21.15.67:0/1607052663 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc032940 con 0x7ff904098c20 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff901d9b640 10 monclient: dump: 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:59.100 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 10 monclient: shutdown 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 -- 172.21.15.67:0/1607052663 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 msgr2=0x7ff9041390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 --2- 172.21.15.67:0/1607052663 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009850 tx=0x7ff8fc0090f0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 -- 172.21.15.67:0/1607052663 shutdown_connections 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 --2- 172.21.15.67:0/1607052663 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098c20 0x7ff9041390c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 --2- 172.21.15.67:0/1607052663 >> v2:172.21.15.67:3300/0 conn(0x7ff904098140 0x7ff9041321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 -- 172.21.15.67:0/1607052663 >> 172.21.15.67:0/1607052663 conn(0x7ff9040a5780 msgr2=0x7ff9040a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 -- 172.21.15.67:0/1607052663 shutdown_connections 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.087+0000 7ff909080640 1 -- 172.21.15.67:0/1607052663 wait complete. 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient: build_initial_monmap 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient: monmap: 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T01:59:59.101 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 Processor -- start 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 -- start start 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient: init 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient: _reopen_session rank -1 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T01:59:59.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): picked mon.noname-c con 0x7ff904098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 0x7ff904130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): picked mon.noname-b con 0x7ff904098c20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 0x7ff904131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): picked mon.noname-a con 0x7ff9040a0370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff904134f70 con 0x7ff9040a0370 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040a3cf0 con 0x7ff904098c20 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): start opening mon connection 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff904137e90 con 0x7ff904098140 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): _renew_subs 2024-09-15T01:59:59.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff909080640 10 monclient(hunting): authenticate will time out at 1231.557965s 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff90259c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 msgr2=0x7ff904130d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.088+0000 7ff90259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 0x7ff904130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 0x7ff904131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35816/0 (socket says 172.21.15.67:35816) 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 1 -- 172.21.15.67:0/1357266882 learned_addr learned my addr 172.21.15.67:0/1357266882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): get_auth_request con 0x7ff904098140 auth_method 0 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): get_auth_request con 0x7ff9040a0370 auth_method 0 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T01:59:59.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient(hunting): handle_auth_done global_id 104517 payload 824 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 -- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 msgr2=0x7ff904138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 -- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 msgr2=0x7ff904130d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 0x7ff904130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient: _finish_hunting 0 2024-09-15T01:59:59.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 monclient: found mon.noname-a 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff90359e640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc009390 con 0x7ff9040a0370 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.089+0000 7ff902d9d640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff90359e640 1 -- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 msgr2=0x7ff904131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff90359e640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 0x7ff904131250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc031000 tx=0x7ff8fc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc031c70 con 0x7ff9040a0370 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: dump: 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: _finish_auth 0 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 5 monclient: authenticate success, global_id 104517 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc031dd0 con 0x7ff9040a0370 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 10 monclient: _renew_subs 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9041349b0 con 0x7ff9040a0370 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc03b630 con 0x7ff9040a0370 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff8faffd640 10 monclient: dump: 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T01:59:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 10 monclient: _renew_subs 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff904099090 con 0x7ff9040a0370 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.092+0000 7ff90159a640 4 set_mon_vals no callback set 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff909080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff909080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff909080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff904132e30 con 0x7ff9040a0370 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff8fc039040 con 0x7ff9040a0370 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff8faffd640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 0x7ff8e403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(225..225 src has 148..225) v4 ==== 13286+0+0 (secure 0 0 0) 0x7ff8fc06daf0 con 0x7ff9040a0370 2024-09-15T01:59:59.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff902d9d640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 0x7ff8e403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T01:59:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.093+0000 7ff902d9d640 10 monclient: get_auth_request con 0x7ff8e40386f0 auth_method 0 2024-09-15T01:59:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.096+0000 7ff902d9d640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 0x7ff8e403aba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40066f0 tx=0x7ff8f40097f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T01:59:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.098+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff8fc042b40 con 0x7ff9040a0370 2024-09-15T01:59:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.098+0000 7ff8faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T01:59:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.098+0000 7ff8faffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T01:59:59.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.264+0000 7ff909080640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.264+0000 7ff909080640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.264+0000 7ff909080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.264+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7ff90404bf60 con 0x7ff9040a0370 2024-09-15T01:59:59.893 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.891+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff8fc03f030 con 0x7ff9040a0370 2024-09-15T01:59:59.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.891+0000 7ff8faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.894 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.891+0000 7ff8faffd640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-15T01:59:59.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff909080640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff909080640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff909080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T01:59:59.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7ff90404bf60 con 0x7ff9040a0370 2024-09-15T01:59:59.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff8faffd640 1 -- 172.21.15.67:0/1357266882 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff8fc03f020 con 0x7ff9040a0370 2024-09-15T01:59:59.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.993+0000 7ff8faffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T01:59:59.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.994+0000 7ff8faffd640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-15T01:59:59.996 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h9' does not appear in the crush map 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 msgr2=0x7ff8e403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 0x7ff8e403aba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40066f0 tx=0x7ff8f40097f0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 10 monclient: shutdown 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 msgr2=0x7ff904131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 0x7ff904131250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc031000 tx=0x7ff8fc032000 comp rx=0 tx=0).stop 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 shutdown_connections 2024-09-15T01:59:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff8e40386f0 0x7ff8e403aba0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9040a0370 0x7ff904131250 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff904098c20 0x7ff904130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 --2- 172.21.15.67:0/1357266882 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff904098140 0x7ff904138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 >> 172.21.15.67:0/1357266882 conn(0x7ff9040a5780 msgr2=0x7ff90409b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 shutdown_connections 2024-09-15T01:59:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T01:59:59.995+0000 7ff909080640 1 -- 172.21.15.67:0/1357266882 wait complete. 2024-09-15T02:00:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h10 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.080+0000 7efedbf9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.080+0000 7efedbf9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.080+0000 7efedbf9d640 10 monclient: monmap: 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:00.081977+0000 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:00.081977+0000 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.083 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:00.084 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.081+0000 7efedbf9d640 1 Processor -- start 2024-09-15T02:00:00.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.081+0000 7efedbf9d640 1 -- start start 2024-09-15T02:00:00.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.081+0000 7efedbf9d640 10 monclient: init 2024-09-15T02:00:00.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:00.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efed40a3830 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 0x7efed4136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-c con 0x7efed406b9b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efed4137240 0x7efed4139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-a con 0x7efed4137240 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4069c60 con 0x7efed40a3830 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40a3c20 con 0x7efed4137240 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed4069650 con 0x7efed406b9b0 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efedbf9d640 10 monclient(hunting): authenticate will time out at 1232.551170s 2024-09-15T02:00:00.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efed9538640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 msgr2=0x7efed4136d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efed9538640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 0x7efed4136d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efed9d39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7efed40a3830 0x7efed406e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56310/0 (socket says 172.21.15.67:56310) 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efed4137240 0x7efed4139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efed9d39640 1 -- 172.21.15.67:0/1709899359 learned_addr learned my addr 172.21.15.67:0/1709899359 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): get_auth_request con 0x7efed4137240 auth_method 0 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient(hunting): handle_auth_done global_id 104520 payload 274 2024-09-15T02:00:00.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 -- 172.21.15.67:0/1709899359 >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 msgr2=0x7efed4136d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 --2- 172.21.15.67:0/1709899359 >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 0x7efed4136d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 -- 172.21.15.67:0/1709899359 >> v1:172.21.15.17:6789/0 conn(0x7efed40a3830 legacy=0x7efed406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 monclient: found mon.noname-a 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:00.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.082+0000 7efeda53a640 1 -- 172.21.15.67:0/1709899359 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed409f500 con 0x7efed4137240 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.085+0000 7efeda53a640 1 -- 172.21.15.67:0/1709899359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed4137240 msgr2=0x7efed4139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.085+0000 7efeda53a640 1 --2- 172.21.15.67:0/1709899359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed4137240 0x7efed4139620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efed409f140 tx=0x7efecc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285c0d81458596fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 1 -- 172.21.15.67:0/1709899359 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc007840 con 0x7efed4137240 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: dump: 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:00.094 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: _finish_auth 0 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 1 -- 172.21.15.67:0/1709899359 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc02d050 con 0x7efed4137240 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:00.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 1 -- 172.21.15.67:0/1709899359 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc007bd0 con 0x7efed4137240 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 5 monclient: authenticate success, global_id 104520 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 4 set_mon_vals no callback set 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efed8d37640 10 monclient: dump: 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.096 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 10 monclient: shutdown 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 -- 172.21.15.67:0/1709899359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed4137240 msgr2=0x7efed4139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 --2- 172.21.15.67:0/1709899359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed4137240 0x7efed4139620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efed409f140 tx=0x7efecc009780 comp rx=0 tx=0).stop 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 -- 172.21.15.67:0/1709899359 shutdown_connections 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 --2- 172.21.15.67:0/1709899359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed4137240 0x7efed4139620 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:00.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 --2- 172.21.15.67:0/1709899359 >> v2:172.21.15.67:3300/0 conn(0x7efed406b9b0 0x7efed4136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:00.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 -- 172.21.15.67:0/1709899359 >> 172.21.15.67:0/1709899359 conn(0x7efed4070ff0 msgr2=0x7efed40a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:00.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 -- 172.21.15.67:0/1709899359 shutdown_connections 2024-09-15T02:00:00.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.086+0000 7efedbf9d640 1 -- 172.21.15.67:0/1709899359 wait complete. 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.087+0000 7efedbf9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.087+0000 7efedbf9d640 10 monclient: monmap: 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:00.099 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:00.100 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:00.100 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.100 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:00.100 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:00.100 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.087+0000 7efedbf9d640 1 Processor -- start 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.087+0000 7efedbf9d640 1 -- start start 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.087+0000 7efedbf9d640 10 monclient: init 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 0x7efed41361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-a con 0x7efed406b9b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 0x7efed411ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-b con 0x7efed40a3830 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): picked mon.noname-c con 0x7efed4137240 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:00.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed4099e80 con 0x7efed406b9b0 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed406e640 con 0x7efed40a3830 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed4120cc0 con 0x7efed4137240 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efedbf9d640 10 monclient(hunting): authenticate will time out at 1232.557130s 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9538640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 msgr2=0x7efed411ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:00.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9538640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 0x7efed411ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 0x7efed41361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:35820/0 (socket says 172.21.15.67:35820) 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 -- 172.21.15.67:0/3268948078 learned_addr learned my addr 172.21.15.67:0/3268948078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): get_auth_request con 0x7efed4137240 auth_method 0 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:00.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): get_auth_request con 0x7efed406b9b0 auth_method 0 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient(hunting): handle_auth_done global_id 94673 payload 824 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 -- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 msgr2=0x7efed411ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 0x7efed411ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 -- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 msgr2=0x7efed41361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 0x7efed41361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 monclient: found mon.noname-c 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efeda53a640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc009a20 con 0x7efed4137240 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.088+0000 7efed9d39640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 0x7efed41361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efeda53a640 1 -- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 msgr2=0x7efed411f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efeda53a640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efecc0099f0 tx=0x7efecc037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc031cd0 con 0x7efed4137240 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: dump: 2024-09-15T02:00:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 5 monclient: authenticate success, global_id 94673 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 10 monclient: _renew_subs 2024-09-15T02:00:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed4120810 con 0x7efed4137240 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc040070 con 0x7efed4137240 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc045640 con 0x7efed4137240 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efec9ffb640 10 monclient: dump: 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 10 monclient: _renew_subs 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efecbfff640 4 set_mon_vals no callback set 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.091+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed4137af0 con 0x7efed4137240 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efedbf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efedbf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efedbf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed404bf60 con 0x7efed4137240 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efecc003020 con 0x7efed4137240 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efec9ffb640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 0x7efeb803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(226..226 src has 148..226) v4 ==== 13076+0+0 (secure 0 0 0) 0x7efecc06f6f0 con 0x7efed4137240 2024-09-15T02:00:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efed9d39640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 0x7efeb803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.092+0000 7efed9d39640 10 monclient: get_auth_request con 0x7efeb80386f0 auth_method 0 2024-09-15T02:00:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.095+0000 7efed9d39640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 0x7efeb803aba0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7efed0007fd0 tx=0x7efed0007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.097+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efecc02ae60 con 0x7efed4137240 2024-09-15T02:00:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.097+0000 7efec9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.097+0000 7efec9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:00.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.265+0000 7efedbf9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:00.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.265+0000 7efedbf9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:00.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.265+0000 7efedbf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:00.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.265+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7efed409f980 con 0x7efed4137240 2024-09-15T02:00:00.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.893+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v227) v1 ==== 120+0+0 (secure 0 0 0) 0x7efecc039bb0 con 0x7efed4137240 2024-09-15T02:00:00.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.893+0000 7efec9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:00.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.893+0000 7efec9ffb640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-15T02:00:01.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.998+0000 7efedbf9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:01.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.998+0000 7efedbf9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:01.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.998+0000 7efedbf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:00.998+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7efed41222e0 con 0x7efed4137240 2024-09-15T02:00:01.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.001+0000 7efec9ffb640 1 -- 172.21.15.67:0/3268948078 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v227) v1 ==== 120+0+0 (secure 0 0 0) 0x7efed41222e0 con 0x7efed4137240 2024-09-15T02:00:01.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.001+0000 7efec9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T02:00:01.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.001+0000 7efec9ffb640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-15T02:00:01.003 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h10' does not appear in the crush map 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 msgr2=0x7efeb803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 0x7efeb803aba0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7efed0007fd0 tx=0x7efed0007f00 comp rx=0 tx=0).stop 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 10 monclient: shutdown 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 msgr2=0x7efed411f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efecc0099f0 tx=0x7efecc037000 comp rx=0 tx=0).stop 2024-09-15T02:00:01.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 shutdown_connections 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efeb80386f0 0x7efeb803aba0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efed4137240 0x7efed411f380 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efed40a3830 0x7efed411ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 --2- 172.21.15.67:0/3268948078 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efed406b9b0 0x7efed41361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 >> 172.21.15.67:0/3268948078 conn(0x7efed4070ff0 msgr2=0x7efed406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 shutdown_connections 2024-09-15T02:00:01.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.002+0000 7efedbf9d640 1 -- 172.21.15.67:0/3268948078 wait complete. 2024-09-15T02:00:01.017 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:01.017 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h11 2024-09-15T02:00:01.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.089+0000 7fc23f59d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:01.091 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.089+0000 7fc23f59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:01.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.089+0000 7fc23f59d640 10 monclient: monmap: 2024-09-15T02:00:01.092 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:01.092 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:01.092 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:01.090387+0000 2024-09-15T02:00:01.092 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:01.090387+0000 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:01.093 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.089+0000 7fc23f59d640 1 Processor -- start 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 1 -- start start 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient: init 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-b con 0x7fc238068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc238069190 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc238137150 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc238096c10 con 0x7fc238069190 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23809ff20 con 0x7fc238137150 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2380a14d0 con 0x7fc238068dc0 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23f59d640 10 monclient(hunting): authenticate will time out at 1233.559561s 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23db3a640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fc238137150 legacy=0x7fc238139530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:33812/0 (socket says 172.21.15.67:33812) 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23cb38640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc238069190 0x7fc238069560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:59120/0 (socket says 172.21.15.67:59120) 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 1 -- 172.21.15.67:0/2993119771 learned_addr learned my addr 172.21.15.67:0/2993119771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 10 monclient(hunting): get_auth_request con 0x7fc238068dc0 auth_method 0 2024-09-15T02:00:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.090+0000 7fc23d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient(hunting): handle_auth_done global_id 94676 payload 274 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 1 -- 172.21.15.67:0/2993119771 >> v1:172.21.15.67:6789/0 conn(0x7fc238137150 legacy=0x7fc238139530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 1 -- 172.21.15.67:0/2993119771 >> v1:172.21.15.17:6789/0 conn(0x7fc238069190 legacy=0x7fc238069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:01.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient: _finish_hunting 0 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 1 monclient: found mon.noname-b 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.091+0000 7fc23d339640 1 -- 172.21.15.67:0/2993119771 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2380a0180 con 0x7fc238068dc0 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23d339640 1 -- 172.21.15.67:0/2993119771 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 msgr2=0x7fc238132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23d339640 1 --2- 172.21.15.67:0/2993119771 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc234009940 tx=0x7fc2340091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b2afd269a2438e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 1 -- 172.21.15.67:0/2993119771 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234032440 con 0x7fc238068dc0 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: dump: 2024-09-15T02:00:01.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: _finish_auth 0 2024-09-15T02:00:01.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 5 monclient: authenticate success, global_id 94676 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 1 -- 172.21.15.67:0/2993119771 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2340325a0 con 0x7fc238068dc0 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 1 -- 172.21.15.67:0/2993119771 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234032940 con 0x7fc238068dc0 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 4 set_mon_vals no callback set 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc22ffff640 10 monclient: dump: 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:01.110 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 10 monclient: shutdown 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:01.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 -- 172.21.15.67:0/2993119771 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 msgr2=0x7fc238132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 --2- 172.21.15.67:0/2993119771 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc234009940 tx=0x7fc2340091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 -- 172.21.15.67:0/2993119771 shutdown_connections 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 --2- 172.21.15.67:0/2993119771 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc238132250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 -- 172.21.15.67:0/2993119771 >> 172.21.15.67:0/2993119771 conn(0x7fc2380711b0 msgr2=0x7fc2380a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 -- 172.21.15.67:0/2993119771 shutdown_connections 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.094+0000 7fc23f59d640 1 -- 172.21.15.67:0/2993119771 wait complete. 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.095+0000 7fc23f59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.095+0000 7fc23f59d640 10 monclient: monmap: 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:01.112 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.095+0000 7fc23f59d640 1 Processor -- start 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.095+0000 7fc23f59d640 1 -- start start 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.095+0000 7fc23f59d640 10 monclient: init 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:01.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-c con 0x7fc238068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 0x7fc238141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-b con 0x7fc238069190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc238137150 0x7fc238142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): picked mon.noname-a con 0x7fc238137150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238099e90 con 0x7fc238137150 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23809b720 con 0x7fc238069190 2024-09-15T02:00:01.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23812fac0 con 0x7fc238068dc0 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23f59d640 10 monclient(hunting): authenticate will time out at 1233.565366s 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23cb38640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 msgr2=0x7fc238141b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23cb38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 0x7fc238141b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:33824/0 (socket says 172.21.15.67:33824) 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 -- 172.21.15.67:0/4010458376 learned_addr learned my addr 172.21.15.67:0/4010458376 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): get_auth_request con 0x7fc238068dc0 auth_method 0 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:01.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc238137150 0x7fc238142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): get_auth_request con 0x7fc238137150 auth_method 0 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:01.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23db3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient(hunting): handle_auth_done global_id 94679 payload 824 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 -- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 msgr2=0x7fc238141b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 0x7fc238141b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 -- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc238137150 msgr2=0x7fc238142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc238137150 0x7fc238142040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient: _finish_hunting 0 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 monclient: found mon.noname-c 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.096+0000 7fc23d339640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234009450 con 0x7fc238068dc0 2024-09-15T02:00:01.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23d339640 1 -- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 msgr2=0x7fc2381415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23d339640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2340099a0 tx=0x7fc234032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23403c960 con 0x7fc238068dc0 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: dump: 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23403cae0 con 0x7fc238068dc0 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 5 monclient: authenticate success, global_id 94679 2024-09-15T02:00:01.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 10 monclient: _renew_subs 2024-09-15T02:00:01.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2380a0890 con 0x7fc238068dc0 2024-09-15T02:00:01.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234031420 con 0x7fc238068dc0 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22d7fa640 10 monclient: dump: 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:01.121 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 10 monclient: _renew_subs 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc22f7fe640 4 set_mon_vals no callback set 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.099+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23811bd90 con 0x7fc238068dc0 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23f59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23f59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc234039040 con 0x7fc238068dc0 2024-09-15T02:00:01.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23804bf60 con 0x7fc238068dc0 2024-09-15T02:00:01.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc22d7fa640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 0x7fc21c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:01.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(227..227 src has 148..227) v4 ==== 12863+0+0 (secure 0 0 0) 0x7fc23406e2d0 con 0x7fc238068dc0 2024-09-15T02:00:01.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23cb38640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 0x7fc21c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:01.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.100+0000 7fc23cb38640 10 monclient: get_auth_request con 0x7fc21c0386f0 auth_method 0 2024-09-15T02:00:01.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.103+0000 7fc23cb38640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 0x7fc21c03aba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc23809f110 tx=0x7fc228007ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:01.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.105+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc234043370 con 0x7fc238068dc0 2024-09-15T02:00:01.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.105+0000 7fc22d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:01.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.105+0000 7fc22d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:01.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.272+0000 7fc23f59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:01.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.272+0000 7fc23f59d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:01.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.272+0000 7fc23f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:01.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.272+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fc23804bf60 con 0x7fc238068dc0 2024-09-15T02:00:01.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.901+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v228) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc23404a090 con 0x7fc238068dc0 2024-09-15T02:00:01.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.902+0000 7fc22d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:01.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:01.902+0000 7fc22d7fa640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-15T02:00:02.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.010+0000 7fc23f59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:02.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.010+0000 7fc23f59d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:02.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.010+0000 7fc23f59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.010+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fc23804bf60 con 0x7fc238068dc0 2024-09-15T02:00:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.011+0000 7fc22d7fa640 1 -- 172.21.15.67:0/4010458376 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v228) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc234037020 con 0x7fc238068dc0 2024-09-15T02:00:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.011+0000 7fc22d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T02:00:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.011+0000 7fc22d7fa640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-15T02:00:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h11' does not appear in the crush map 2024-09-15T02:00:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 msgr2=0x7fc21c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 0x7fc21c03aba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc23809f110 tx=0x7fc228007ac0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 10 monclient: shutdown 2024-09-15T02:00:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 msgr2=0x7fc2381415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2340099a0 tx=0x7fc234032000 comp rx=0 tx=0).stop 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 shutdown_connections 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc21c0386f0 0x7fc21c03aba0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc238137150 0x7fc238142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238069190 0x7fc238141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 --2- 172.21.15.67:0/4010458376 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc238068dc0 0x7fc2381415c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 >> 172.21.15.67:0/4010458376 conn(0x7fc2380711b0 msgr2=0x7fc23809e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 shutdown_connections 2024-09-15T02:00:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.013+0000 7fc23f59d640 1 -- 172.21.15.67:0/4010458376 wait complete. 2024-09-15T02:00:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h12 2024-09-15T02:00:02.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.100+0000 7f2491cdd640 10 monclient: get_monmap_and_config 2024-09-15T02:00:02.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.100+0000 7f2491cdd640 10 monclient: build_initial_monmap 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.100+0000 7f2491cdd640 10 monclient: monmap: 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:02.101545+0000 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:02.101545+0000 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:02.103 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:02.104 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:02.104 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:02.104 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:02.104 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 Processor -- start 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 -- start start 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient: init 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T02:00:02.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 0x7f248c069380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f248c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f248c09eea0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f248c09f290 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c0a61a0 con 0x7f248c09eea0 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c0a04c0 con 0x7f248c09f290 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0a1610 con 0x7f248c0a0b50 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f2491cdd640 10 monclient(hunting): authenticate will time out at 1234.570680s 2024-09-15T02:00:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248bfff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f248c09f290 legacy=0x7f248c132220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 0x7f248c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248affd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f248c09eea0 0x7f248c131b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:59126/0 (socket says 172.21.15.67:59126) 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248affd640 1 -- 172.21.15.67:0/3357073951 learned_addr learned my addr 172.21.15.67:0/3357073951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248b7fe640 10 monclient(hunting): get_auth_request con 0x7f248c0a0b50 auth_method 0 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.101+0000 7f248b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient(hunting): handle_auth_done global_id 94682 payload 274 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 1 -- 172.21.15.67:0/3357073951 >> v1:172.21.15.67:6789/0 conn(0x7f248c09f290 legacy=0x7f248c132220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 1 -- 172.21.15.67:0/3357073951 >> v1:172.21.15.17:6789/0 conn(0x7f248c09eea0 legacy=0x7f248c131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 1 monclient: found mon.noname-b 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.102+0000 7f248b7fe640 1 -- 172.21.15.67:0/3357073951 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0a0f20 con 0x7f248c0a0b50 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248b7fe640 1 -- 172.21.15.67:0/3357073951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 msgr2=0x7f248c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248b7fe640 1 --2- 172.21.15.67:0/3357073951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 0x7f248c069380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f247c009880 tx=0x7f247c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c94770a9bd50ac72 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 1 -- 172.21.15.67:0/3357073951 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c032440 con 0x7f248c0a0b50 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: dump: 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 1 -- 172.21.15.67:0/3357073951 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c0325a0 con 0x7f248c0a0b50 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 1 -- 172.21.15.67:0/3357073951 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c032940 con 0x7f248c0a0b50 2024-09-15T02:00:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f248a7fc640 10 monclient: dump: 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f2491cdd640 5 monclient: authenticate success, global_id 94682 2024-09-15T02:00:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f2491cdd640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.105+0000 7f2491cdd640 4 set_mon_vals no callback set 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 10 monclient: shutdown 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 -- 172.21.15.67:0/3357073951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 msgr2=0x7f248c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 --2- 172.21.15.67:0/3357073951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 0x7f248c069380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f247c009880 tx=0x7f247c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 -- 172.21.15.67:0/3357073951 shutdown_connections 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 --2- 172.21.15.67:0/3357073951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c0a0b50 0x7f248c069380 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 -- 172.21.15.67:0/3357073951 >> 172.21.15.67:0/3357073951 conn(0x7f248c071330 msgr2=0x7f248c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 -- 172.21.15.67:0/3357073951 shutdown_connections 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 1 -- 172.21.15.67:0/3357073951 wait complete. 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 10 monclient: build_initial_monmap 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.106+0000 7f2491cdd640 10 monclient: monmap: 2024-09-15T02:00:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 Processor -- start 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 -- start start 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient: init 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 0x7f248c09a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f248c09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f248c09f290 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 0x7f248c1412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f248c0a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c131ac0 con 0x7f248c0a0b50 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c09b480 con 0x7f248c09eea0 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c11de50 con 0x7f248c09f290 2024-09-15T02:00:02.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f2491cdd640 10 monclient(hunting): authenticate will time out at 1234.576593s 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248b7fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 msgr2=0x7f248c09a950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 0x7f248c09a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 0x7f248c1412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:33836/0 (socket says 172.21.15.67:33836) 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 1 -- 172.21.15.67:0/2585322543 learned_addr learned my addr 172.21.15.67:0/2585322543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 10 monclient(hunting): get_auth_request con 0x7f248c09f290 auth_method 0 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248bfff640 10 monclient(hunting): get_auth_request con 0x7f248c0a0b50 auth_method 0 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.107+0000 7f248bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient(hunting): handle_auth_done global_id 94685 payload 824 2024-09-15T02:00:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 -- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 msgr2=0x7f248c09a950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 0x7f248c09a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 -- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 msgr2=0x7f248c1412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 0x7f248c1412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 monclient: found mon.noname-c 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248affd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c009390 con 0x7f248c09f290 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.108+0000 7f248bfff640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 0x7f248c1412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f248affd640 1 -- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 msgr2=0x7f248c130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f248affd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f248000c5e0 tx=0x7f2480004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480005cb0 con 0x7f248c09f290 2024-09-15T02:00:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: dump: 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2480005e30 con 0x7f248c09f290 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480007340 con 0x7f248c09f290 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 5 monclient: authenticate success, global_id 94685 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 10 monclient: _renew_subs 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c11e130 con 0x7f248c09f290 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2489ffb640 4 set_mon_vals no callback set 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f24737fe640 10 monclient: dump: 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 10 monclient: _renew_subs 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.110+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c11c880 con 0x7f248c09f290 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f2491cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f2491cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f2491cdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c11cb00 con 0x7f248c09f290 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f248001b400 con 0x7f248c09f290 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f24737fe640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 0x7f247803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(228..228 src has 148..228) v4 ==== 12650+0+0 (secure 0 0 0) 0x7f248004d550 con 0x7f248c09f290 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f248b7fe640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 0x7f247803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.112+0000 7f248b7fe640 10 monclient: get_auth_request con 0x7f24780386f0 auth_method 0 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.114+0000 7f248b7fe640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 0x7f247803aba0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f248c071940 tx=0x7f247c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:02.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.116+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f248001a540 con 0x7f248c09f290 2024-09-15T02:00:02.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.116+0000 7f24737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:02.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.116+0000 7f24737fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.282+0000 7f2491cdd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.282+0000 7f2491cdd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.282+0000 7f2491cdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.282+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f248c04bf60 con 0x7f248c09f290 2024-09-15T02:00:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.903+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v229) v1 ==== 120+0+0 (secure 0 0 0) 0x7f248001a360 con 0x7f248c09f290 2024-09-15T02:00:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.903+0000 7f24737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:02.903+0000 7f24737fe640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-15T02:00:03.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.007+0000 7f2491cdd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:03.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.007+0000 7f2491cdd640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:03.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.007+0000 7f2491cdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.007+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f248c0008d0 con 0x7f248c09f290 2024-09-15T02:00:03.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.008+0000 7f24737fe640 1 -- 172.21.15.67:0/2585322543 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v229) v1 ==== 120+0+0 (secure 0 0 0) 0x7f248001a540 con 0x7f248c09f290 2024-09-15T02:00:03.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.008+0000 7f24737fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T02:00:03.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.008+0000 7f24737fe640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-15T02:00:03.011 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h12' does not appear in the crush map 2024-09-15T02:00:03.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 msgr2=0x7f247803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:03.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 0x7f247803aba0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f248c071940 tx=0x7f247c032000 comp rx=0 tx=0).stop 2024-09-15T02:00:03.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 10 monclient: shutdown 2024-09-15T02:00:03.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:03.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 msgr2=0x7f248c130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:03.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f248000c5e0 tx=0x7f2480004960 comp rx=0 tx=0).stop 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 shutdown_connections 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24780386f0 0x7f247803aba0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f248c0a0b50 0x7f248c1412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f248c09f290 0x7f248c130d30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 --2- 172.21.15.67:0/2585322543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c09eea0 0x7f248c09a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 >> 172.21.15.67:0/2585322543 conn(0x7f248c071330 msgr2=0x7f248c131db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 shutdown_connections 2024-09-15T02:00:03.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.010+0000 7f2491cdd640 1 -- 172.21.15.67:0/2585322543 wait complete. 2024-09-15T02:00:03.022 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:03.023 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h13 2024-09-15T02:00:03.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 10 monclient: get_monmap_and_config 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 10 monclient: build_initial_monmap 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 10 monclient: monmap: 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:03.098209+0000 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:03.098209+0000 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:03.100 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 1 Processor -- start 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 1 -- start start 2024-09-15T02:00:03.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.097+0000 7fd14cc17640 10 monclient: init 2024-09-15T02:00:03.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:03.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T02:00:03.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd148098140 0x7fd1481321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-c con 0x7fd148098140 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1480a0370 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd148098c20 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1480a0b80 con 0x7fd148098c20 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1480a00b0 con 0x7fd1480a0370 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd148071390 con 0x7fd148098140 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14cc17640 10 monclient(hunting): authenticate will time out at 1235.567327s 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14659c640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fd148098140 msgr2=0x7fd1481321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd145d9b640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fd1480a0370 legacy=0x7fd148098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd14659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd148098140 0x7fd1481321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd146d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd148098c20 0x7fd1481390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:59130/0 (socket says 172.21.15.67:59130) 2024-09-15T02:00:03.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.098+0000 7fd146d9d640 1 -- 172.21.15.67:0/4242742511 learned_addr learned my addr 172.21.15.67:0/4242742511 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.100+0000 7fd146d9d640 1 -- 172.21.15.67:0/4242742511 >> v1:172.21.15.17:6789/0 conn(0x7fd148098c20 legacy=0x7fd1481390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232107117 0 0) 0x7fd1480a0b80 con 0x7fd148098c20 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): my global_id is 104532 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1300034e0 con 0x7fd148098c20 2024-09-15T02:00:03.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328560490 0 0) 0x7fd1300034e0 con 0x7fd148098c20 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 >> v2:172.21.15.67:3300/0 conn(0x7fd148098140 msgr2=0x7fd1481321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 --2- 172.21.15.67:0/4242742511 >> v2:172.21.15.67:3300/0 conn(0x7fd148098140 0x7fd1481321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 >> v1:172.21.15.67:6789/0 conn(0x7fd1480a0370 legacy=0x7fd148098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480714d0 con 0x7fd148098c20 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: _finish_auth 0 2024-09-15T02:00:03.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fd140001720 con 0x7fd148098c20 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: dump: 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.116 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 >> v1:172.21.15.17:6789/0 conn(0x7fd148098c20 legacy=0x7fd1481390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 0x7fd130003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): picked mon.a con 0x7fd1300039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130004780 0x7fd130004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): picked mon.b con 0x7fd130004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd130005330 0x7fd13000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): picked mon.c con 0x7fd130005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd148071390 con 0x7fd1300039d0 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13000c290 con 0x7fd130004780 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13000c450 con 0x7fd130005330 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:03.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14559a640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd145d9b640 1 -- 172.21.15.67:0/4242742511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130004780 msgr2=0x7fd130004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd145d9b640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130004780 0x7fd130004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 0x7fd130003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd130005330 0x7fd13000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): get_auth_request con 0x7fd1300039d0 auth_method 0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): get_auth_request con 0x7fd130005330 auth_method 0 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:03.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd14659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.101+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 10 monclient(hunting): handle_auth_done global_id 104532 payload 293 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 -- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd130005330 msgr2=0x7fd13000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd130005330 0x7fd13000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 -- 172.21.15.67:0/4242742511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130004780 msgr2=0x7fd130004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130004780 0x7fd130004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 monclient: found mon.a 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 1 -- 172.21.15.67:0/4242742511 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1300034e0 con 0x7fd1300039d0 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 10 monclient: _finish_auth 0 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14cc17640 5 monclient: authenticate success, global_id 104532 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd14cc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd146d9d640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd130005330 0x7fd13000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.102+0000 7fd146d9d640 1 -- 172.21.15.67:0/4242742511 reap_dead start 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14659c640 1 -- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 msgr2=0x7fd130003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14659c640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 0x7fd130003dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd138009f90 tx=0x7fd1380095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e843e62be26ccd1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138032430 con 0x7fd1300039d0 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: dump: 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.122 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13802d070 con 0x7fd1300039d0 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 1 -- 172.21.15.67:0/4242742511 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138032820 con 0x7fd1300039d0 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 4 set_mon_vals no callback set 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14559a640 10 monclient: dump: 2024-09-15T02:00:03.123 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 10 monclient: shutdown 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 1 -- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 msgr2=0x7fd130003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:03.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 0x7fd130003dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd138009f90 tx=0x7fd1380095d0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 1 -- 172.21.15.67:0/4242742511 shutdown_connections 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 1 --2- 172.21.15.67:0/4242742511 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd1300039d0 0x7fd130003dc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.105+0000 7fd14cc17640 1 -- 172.21.15.67:0/4242742511 >> 172.21.15.67:0/4242742511 conn(0x7fd1480a5780 msgr2=0x7fd1480a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 1 -- 172.21.15.67:0/4242742511 shutdown_connections 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 1 -- 172.21.15.67:0/4242742511 wait complete. 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 10 monclient: build_initial_monmap 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 10 monclient: monmap: 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:03.125 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 1 Processor -- start 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 1 -- start start 2024-09-15T02:00:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.106+0000 7fd14cc17640 10 monclient: init 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 0x7fd148130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-b con 0x7fd1300039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-c con 0x7fd1481311f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd14811dd80 0x7fd14811e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): picked mon.noname-a con 0x7fd14811dd80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481315c0 con 0x7fd14811dd80 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480a38a0 con 0x7fd1300039d0 2024-09-15T02:00:03.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14659c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 msgr2=0x7fd148130cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481210e0 con 0x7fd1481311f0 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14cc17640 10 monclient(hunting): authenticate will time out at 1235.576328s 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd1367fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd14659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 0x7fd148130cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd14811dd80 0x7fd14811e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:33844/0 (socket says 172.21.15.67:33844) 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 1 -- 172.21.15.67:0/3753593841 learned_addr learned my addr 172.21.15.67:0/3753593841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): get_auth_request con 0x7fd1481311f0 auth_method 0 2024-09-15T02:00:03.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): get_auth_request con 0x7fd14811dd80 auth_method 0 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd145d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.107+0000 7fd146d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:03.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 10 monclient(hunting): handle_auth_done global_id 94688 payload 824 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 -- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 msgr2=0x7fd148130cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 0x7fd148130cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 -- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd14811dd80 msgr2=0x7fd14811e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd14811dd80 0x7fd14811e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 monclient: found mon.noname-c 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.108+0000 7fd145d9b640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148099090 con 0x7fd1481311f0 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd145d9b640 1 -- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 msgr2=0x7fd14811d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd145d9b640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00b810 tx=0x7fd13c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003dd0 con 0x7fd1481311f0 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: dump: 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 5 monclient: authenticate success, global_id 94688 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 10 monclient: _renew_subs 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd148120e00 con 0x7fd1481311f0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 10 monclient: _renew_subs 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14811ff10 con 0x7fd1481311f0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c006150 con 0x7fd1481311f0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c006510 con 0x7fd1481311f0 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:03.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd1367fc640 10 monclient: dump: 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.110+0000 7fd144d99640 4 set_mon_vals no callback set 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd14cc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd14cc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd14cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14811f300 con 0x7fd1481311f0 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd13c0103c0 con 0x7fd1481311f0 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd1367fc640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 0x7fd12803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(229..229 src has 148..229) v4 ==== 12437+0+0 (secure 0 0 0) 0x7fd13c04cee0 con 0x7fd1481311f0 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.111+0000 7fd14659c640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 0x7fd12803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.112+0000 7fd14659c640 10 monclient: get_auth_request con 0x7fd1280386f0 auth_method 0 2024-09-15T02:00:03.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.114+0000 7fd14659c640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 0x7fd12803aba0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd138003cc0 tx=0x7fd138035000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:03.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.117+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd13c00eaa0 con 0x7fd1481311f0 2024-09-15T02:00:03.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.117+0000 7fd1367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:03.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.117+0000 7fd1367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:03.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.282+0000 7fd14cc17640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:03.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.282+0000 7fd14cc17640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:03.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.282+0000 7fd14cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:03.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.282+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7fd148098510 con 0x7fd1481311f0 2024-09-15T02:00:03.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.911+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v230) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd13c04e070 con 0x7fd1481311f0 2024-09-15T02:00:03.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.911+0000 7fd1367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:03.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:03.911+0000 7fd1367fc640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-15T02:00:04.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.017+0000 7fd14cc17640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:04.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.017+0000 7fd14cc17640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:04.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.017+0000 7fd14cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.017+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7fd14804bf60 con 0x7fd1481311f0 2024-09-15T02:00:04.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.019+0000 7fd1367fc640 1 -- 172.21.15.67:0/3753593841 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v230) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd13c017030 con 0x7fd1481311f0 2024-09-15T02:00:04.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.019+0000 7fd1367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T02:00:04.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.019+0000 7fd1367fc640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-15T02:00:04.021 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h13' does not appear in the crush map 2024-09-15T02:00:04.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 msgr2=0x7fd12803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 0x7fd12803aba0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd138003cc0 tx=0x7fd138035000 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 10 monclient: shutdown 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 msgr2=0x7fd14811d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00b810 tx=0x7fd13c0091a0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 shutdown_connections 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd1280386f0 0x7fd12803aba0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd14811dd80 0x7fd14811e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd1481311f0 0x7fd14811d840 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 --2- 172.21.15.67:0/3753593841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300039d0 0x7fd148130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 >> 172.21.15.67:0/3753593841 conn(0x7fd1480a5780 msgr2=0x7fd14809b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 shutdown_connections 2024-09-15T02:00:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.020+0000 7fd14cc17640 1 -- 172.21.15.67:0/3753593841 wait complete. 2024-09-15T02:00:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h14 2024-09-15T02:00:04.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 10 monclient: get_monmap_and_config 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 10 monclient: build_initial_monmap 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 10 monclient: monmap: 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:04.106323+0000 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:04.106323+0000 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 1 Processor -- start 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 1 -- start start 2024-09-15T02:00:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.105+0000 7fd76b494640 10 monclient: init 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd764098140 0x7fd7641321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-c con 0x7fd764098140 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7640a0370 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd764098c20 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640a0b80 con 0x7fd7640a0370 2024-09-15T02:00:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd769230640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fd764098140 msgr2=0x7fd7641321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640a00b0 con 0x7fd764098c20 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd764071390 con 0x7fd764098140 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd769230640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd764098140 0x7fd7641321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd76b494640 10 monclient(hunting): authenticate will time out at 1236.575517s 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd769a31640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fd764098c20 legacy=0x7fd7641390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd768a2f640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd7640a0370 0x7fd764098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:59138/0 (socket says 172.21.15.67:59138) 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.106+0000 7fd768a2f640 1 -- 172.21.15.67:0/582201607 learned_addr learned my addr 172.21.15.67:0/582201607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd768a2f640 1 -- 172.21.15.67:0/582201607 >> v1:172.21.15.17:6789/0 conn(0x7fd7640a0370 legacy=0x7fd764098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600016544 0 0) 0x7fd7640a0b80 con 0x7fd7640a0370 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 10 monclient(hunting): my global_id is 104538 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7500034e0 con 0x7fd7640a0370 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.109+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3121177733 0 0) 0x7fd7500034e0 con 0x7fd7640a0370 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 >> v2:172.21.15.67:3300/0 conn(0x7fd764098140 msgr2=0x7fd7641321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 --2- 172.21.15.67:0/582201607 >> v2:172.21.15.67:3300/0 conn(0x7fd764098140 0x7fd7641321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 >> v1:172.21.15.67:6789/0 conn(0x7fd764098c20 legacy=0x7fd7641390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:04.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640714d0 con 0x7fd7640a0370 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: _finish_auth 0 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fd754001720 con 0x7fd7640a0370 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd76b494640 5 monclient: authenticate success, global_id 104538 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd76b494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: dump: 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.123 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 >> v1:172.21.15.17:6789/0 conn(0x7fd7640a0370 legacy=0x7fd764098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:04.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 0x7fd750003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): picked mon.c con 0x7fd7500039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd750004780 0x7fd750004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): picked mon.a con 0x7fd750004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd750005330 0x7fd75000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): picked mon.b con 0x7fd750005330 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd764071390 con 0x7fd750004780 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000c290 con 0x7fd750005330 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000c450 con 0x7fd7500039d0 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd75bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:04.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd76b494640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769a31640 1 -- 172.21.15.67:0/582201607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd750005330 msgr2=0x7fd75000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769a31640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd750005330 0x7fd75000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 0x7fd750003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): get_auth_request con 0x7fd7500039d0 auth_method 0 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd750004780 0x7fd750004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): get_auth_request con 0x7fd750004780 auth_method 0 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:04.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient(hunting): handle_auth_done global_id 104538 payload 293 2024-09-15T02:00:04.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 -- 172.21.15.67:0/582201607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd750005330 msgr2=0x7fd75000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd750005330 0x7fd75000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 -- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd750004780 msgr2=0x7fd750004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd750004780 0x7fd750004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient: _finish_hunting 0 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 monclient: found mon.c 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769a31640 1 -- 172.21.15.67:0/582201607 reap_dead start 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 1 -- 172.21.15.67:0/582201607 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500034e0 con 0x7fd7500039d0 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 10 monclient: _finish_auth 0 2024-09-15T02:00:04.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.110+0000 7fd769230640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd769230640 1 -- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 msgr2=0x7fd750003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd769230640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 0x7fd750003dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd7600099c0 tx=0x7fd760009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5875a8e3c7ae6ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760003b80 con 0x7fd7500039d0 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd75bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd75bfff640 10 monclient: dump: 2024-09-15T02:00:04.129 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.130 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.113+0000 7fd76b494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760003d20 con 0x7fd7500039d0 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 1 -- 172.21.15.67:0/582201607 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76002ab80 con 0x7fd7500039d0 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 4 set_mon_vals no callback set 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd75bfff640 10 monclient: dump: 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.132 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 10 monclient: shutdown 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 -- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 msgr2=0x7fd750003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 0x7fd750003dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd7600099c0 tx=0x7fd760009060 comp rx=0 tx=0).stop 2024-09-15T02:00:04.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 -- 172.21.15.67:0/582201607 shutdown_connections 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 --2- 172.21.15.67:0/582201607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd7500039d0 0x7fd750003dc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 -- 172.21.15.67:0/582201607 >> 172.21.15.67:0/582201607 conn(0x7fd7640a5780 msgr2=0x7fd7640a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 -- 172.21.15.67:0/582201607 shutdown_connections 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.114+0000 7fd76b494640 1 -- 172.21.15.67:0/582201607 wait complete. 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient: build_initial_monmap 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient: monmap: 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:04.134 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 Processor -- start 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 -- start start 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient: init 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 0x7fd76409e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-b con 0x7fd7500039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:04.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-a con 0x7fd764130bf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd76411ba40 0x7fd76411beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): picked mon.noname-c con 0x7fd76411ba40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76411e360 con 0x7fd764130bf0 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd769230640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 msgr2=0x7fd76409e600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd764131990 con 0x7fd7500039d0 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd764099090 con 0x7fd76411ba40 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.115+0000 7fd76b494640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd76b494640 10 monclient(hunting): authenticate will time out at 1236.585009s 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769230640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 0x7fd76409e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd76411ba40 0x7fd76411beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:49534/0 (socket says 172.21.15.67:49534) 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 -- 172.21.15.67:0/1530685484 learned_addr learned my addr 172.21.15.67:0/1530685484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:04.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): get_auth_request con 0x7fd764130bf0 auth_method 0 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): get_auth_request con 0x7fd76411ba40 auth_method 0 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd769a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient(hunting): handle_auth_done global_id 104541 payload 824 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 -- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd76411ba40 msgr2=0x7fd76411beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd76411ba40 0x7fd76411beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 -- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 msgr2=0x7fd76409e600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 0x7fd76409e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient: _finish_hunting 0 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 monclient: found mon.noname-a 2024-09-15T02:00:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.116+0000 7fd768a2f640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764131bc0 con 0x7fd764130bf0 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.118+0000 7fd768a2f640 1 -- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 msgr2=0x7fd764130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.118+0000 7fd768a2f640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7540018d0 tx=0x7fd754007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7540116d0 con 0x7fd764130bf0 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: dump: 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.141 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd754011830 con 0x7fd764130bf0 2024-09-15T02:00:04.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 5 monclient: authenticate success, global_id 104541 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _renew_subs 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd764141f10 con 0x7fd764130bf0 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754011bd0 con 0x7fd764130bf0 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd7597fa640 10 monclient: dump: 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.143 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd75b7fe640 4 set_mon_vals no callback set 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _renew_subs 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd764136550 con 0x7fd764130bf0 2024-09-15T02:00:04.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.119+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76404bf60 con 0x7fd764130bf0 2024-09-15T02:00:04.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.120+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd75401d3b0 con 0x7fd764130bf0 2024-09-15T02:00:04.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.120+0000 7fd7597fa640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 0x7fd74803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.120+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(230..230 src has 148..230) v4 ==== 12224+0+0 (secure 0 0 0) 0x7fd75404d060 con 0x7fd764130bf0 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.120+0000 7fd769230640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 0x7fd74803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.120+0000 7fd769230640 10 monclient: get_auth_request con 0x7fd7480386f0 auth_method 0 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.122+0000 7fd769230640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 0x7fd74803aba0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd760003940 tx=0x7fd760030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.125+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd75401b6e0 con 0x7fd764130bf0 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.125+0000 7fd7597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.125+0000 7fd7597fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:04.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.292+0000 7fd76b494640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:04.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.292+0000 7fd76b494640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:04.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.292+0000 7fd76b494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.292+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7fd764098510 con 0x7fd764130bf0 2024-09-15T02:00:04.380 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:00:04.380 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:00:04.380 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:00:04.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:00:04.383 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:00:04.383 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:00:04.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:04.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.491+0000 7fe77195d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.491+0000 7fe77195d640 10 monclient: build_initial_monmap 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.491+0000 7fe77195d640 10 monclient: monmap: 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:04.492185+0000 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:04.492185+0000 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:04.494 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:04.495 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:04.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.491+0000 7fe77195d640 1 Processor -- start 2024-09-15T02:00:04.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 -- start start 2024-09-15T02:00:04.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient: init 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe76c067090 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe76c067460 0x7fe76c067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-a con 0x7fe76c067460 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-c con 0x7fe76c136f90 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c09cbc0 con 0x7fe76c067090 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c09e550 con 0x7fe76c067460 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c1328b0 con 0x7fe76c136f90 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe77195d640 10 monclient(hunting): authenticate will time out at 1264.488574s 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76affd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe76c067090 0x7fe76c1321a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42936/0 (socket says 172.21.15.17:42936) 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76affd640 1 -- 172.21.15.17:0/2574298588 learned_addr learned my addr 172.21.15.17:0/2574298588 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.17:3300/0 conn(0x7fe76c067460 0x7fe76c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): get_auth_request con 0x7fe76c067460 auth_method 0 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 10 monclient(hunting): get_auth_request con 0x7fe76c136f90 auth_method 0 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.492+0000 7fe76b7fe640 1 -- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 msgr2=0x7fe76c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76b7fe640 1 -- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 msgr2=0x7fe76c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 10 monclient(hunting): handle_auth_done global_id 104544 payload 274 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 1 -- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 msgr2=0x7fe76c139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 1 -- 172.21.15.17:0/2574298588 >> v1:172.21.15.17:6789/0 conn(0x7fe76c067090 legacy=0x7fe76c1321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:04.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.493+0000 7fe76a7fc640 1 -- 172.21.15.17:0/2574298588 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c09e1e0 con 0x7fe76c067460 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.495+0000 7fe76a7fc640 1 -- 172.21.15.17:0/2574298588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067460 msgr2=0x7fe76c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.495+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/2574298588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067460 0x7fe76c067830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe76c131ea0 tx=0x7fe7600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f34b895bfb3e816 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 1 -- 172.21.15.17:0/2574298588 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76002ae00 con 0x7fe76c067460 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: dump: 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.501 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 5 monclient: authenticate success, global_id 104544 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 1 -- 172.21.15.17:0/2574298588 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760032440 con 0x7fe76c067460 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:04.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 1 -- 172.21.15.17:0/2574298588 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600327e0 con 0x7fe76c067460 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe769ffb640 10 monclient: dump: 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.503 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:04.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 4 set_mon_vals no callback set 2024-09-15T02:00:04.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 10 monclient: shutdown 2024-09-15T02:00:04.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:04.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 -- 172.21.15.17:0/2574298588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067460 msgr2=0x7fe76c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 --2- 172.21.15.17:0/2574298588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067460 0x7fe76c067830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe76c131ea0 tx=0x7fe7600090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 -- 172.21.15.17:0/2574298588 shutdown_connections 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 --2- 172.21.15.17:0/2574298588 >> v2:172.21.15.67:3300/0 conn(0x7fe76c136f90 0x7fe76c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 --2- 172.21.15.17:0/2574298588 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067460 0x7fe76c067830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 -- 172.21.15.17:0/2574298588 >> 172.21.15.17:0/2574298588 conn(0x7fe76c0a1ee0 msgr2=0x7fe76c0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 -- 172.21.15.17:0/2574298588 shutdown_connections 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.496+0000 7fe77195d640 1 -- 172.21.15.17:0/2574298588 wait complete. 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.497+0000 7fe77195d640 10 monclient: build_initial_monmap 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.497+0000 7fe77195d640 10 monclient: monmap: 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:04.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.497+0000 7fe77195d640 1 Processor -- start 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.497+0000 7fe77195d640 1 -- start start 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.497+0000 7fe77195d640 10 monclient: init 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-a con 0x7fe76c067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:04.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-b con 0x7fe76c067460 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 0x7fe76c118900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): picked mon.noname-c con 0x7fe76c136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c09d530 con 0x7fe76c067090 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0a0140 con 0x7fe76c067460 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c11a830 con 0x7fe76c136f90 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe77195d640 10 monclient(hunting): authenticate will time out at 1264.494425s 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46850/0 (socket says 172.21.15.17:46850) 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 0x7fe76c118900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 1 -- 172.21.15.17:0/3078384723 learned_addr learned my addr 172.21.15.17:0/3078384723 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): get_auth_request con 0x7fe76c067090 auth_method 0 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): get_auth_request con 0x7fe76c136f90 auth_method 0 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 10 monclient(hunting): get_auth_request con 0x7fe76c067460 auth_method 0 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:04.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 msgr2=0x7fe76c141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 msgr2=0x7fe76c141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76a7fc640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient(hunting): handle_auth_done global_id 94694 payload 824 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 msgr2=0x7fe76c141b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:04.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 msgr2=0x7fe76c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 monclient: found mon.noname-c 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76b7fe640 1 -- 172.21.15.17:0/3078384723 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760009390 con 0x7fe76c136f90 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.498+0000 7fe76affd640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe76b7fe640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 msgr2=0x7fe76c118900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe76b7fe640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 0x7fe76c118900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe764007e70 tx=0x7fe7640097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764014420 con 0x7fe76c136f90 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:04.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: dump: 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.516 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 5 monclient: authenticate success, global_id 94694 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7640145c0 con 0x7fe76c136f90 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 10 monclient: _renew_subs 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c11aef0 con 0x7fe76c136f90 2024-09-15T02:00:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764014980 con 0x7fe76c136f90 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe752ffd640 10 monclient: dump: 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:04.518 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 10 monclient: _renew_subs 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c11d000 con 0x7fe76c136f90 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.501+0000 7fe7697fa640 4 set_mon_vals no callback set 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe76401e460 con 0x7fe76c136f90 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe752ffd640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 0x7fe75803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(230..230 src has 148..230) v4 ==== 12224+0+0 (secure 0 0 0) 0x7fe76404de70 con 0x7fe76c136f90 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe77195d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe77195d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe77195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c11a380 con 0x7fe76c136f90 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe76affd640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 0x7fe75803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.502+0000 7fe76affd640 10 monclient: get_auth_request con 0x7fe758038740 auth_method 0 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.505+0000 7fe76affd640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 0x7fe75803abf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009f00 tx=0x7fe75c0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.507+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe764013380 con 0x7fe76c136f90 2024-09-15T02:00:04.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.507+0000 7fe752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:04.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.507+0000 7fe752ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe77195d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe77195d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe77195d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe76c0724a0 con 0x7fe76c136f90 2024-09-15T02:00:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe752ffd640 1 -- 172.21.15.17:0/3078384723 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7fe76404ae60 con 0x7fe76c136f90 2024-09-15T02:00:04.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:04.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.733+0000 7fe752ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:04.736 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 msgr2=0x7fe75803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 0x7fe75803abf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009f00 tx=0x7fe75c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 10 monclient: shutdown 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 msgr2=0x7fe76c118900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 0x7fe76c118900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe764007e70 tx=0x7fe7640097e0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 shutdown_connections 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe758038740 0x7fe75803abf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe76c136f90 0x7fe76c118900 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c067460 0x7fe76c141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 --2- 172.21.15.17:0/3078384723 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe76c067090 0x7fe76c141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 >> 172.21.15.17:0/3078384723 conn(0x7fe76c0a1ee0 msgr2=0x7fe76c097ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 shutdown_connections 2024-09-15T02:00:04.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:04.735+0000 7fe77195d640 1 -- 172.21.15.17:0/3078384723 wait complete. 2024-09-15T02:00:04.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.919+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v231) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd75401a260 con 0x7fd764130bf0 2024-09-15T02:00:04.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.919+0000 7fd7597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:04.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:04.919+0000 7fd7597fa640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.026+0000 7fd76b494640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.026+0000 7fd76b494640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.026+0000 7fd76b494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.026+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7fd7640008d0 con 0x7fd764130bf0 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.027+0000 7fd7597fa640 1 -- 172.21.15.67:0/1530685484 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v231) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd75401ae20 con 0x7fd764130bf0 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.027+0000 7fd7597fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T02:00:05.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.027+0000 7fd7597fa640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-15T02:00:05.032 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h14' does not appear in the crush map 2024-09-15T02:00:05.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 msgr2=0x7fd74803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 0x7fd74803aba0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd760003940 tx=0x7fd760030000 comp rx=0 tx=0).stop 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 10 monclient: shutdown 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 msgr2=0x7fd764130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7540018d0 tx=0x7fd754007a90 comp rx=0 tx=0).stop 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 shutdown_connections 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7480386f0 0x7fd74803aba0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd764130bf0 0x7fd764130fc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd76411ba40 0x7fd76411beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 --2- 172.21.15.67:0/1530685484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7500039d0 0x7fd76409e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 >> 172.21.15.67:0/1530685484 conn(0x7fd7640a5780 msgr2=0x7fd7640a4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 shutdown_connections 2024-09-15T02:00:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.029+0000 7fd76b494640 1 -- 172.21.15.67:0/1530685484 wait complete. 2024-09-15T02:00:05.041 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:05.041 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h15 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.114+0000 7f328bc0f640 10 monclient: get_monmap_and_config 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.114+0000 7f328bc0f640 10 monclient: build_initial_monmap 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.114+0000 7f328bc0f640 10 monclient: monmap: 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:05.115964+0000 2024-09-15T02:00:05.117 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:05.115964+0000 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:05.118 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.115+0000 7f328bc0f640 1 Processor -- start 2024-09-15T02:00:05.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.115+0000 7f328bc0f640 1 -- start start 2024-09-15T02:00:05.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.115+0000 7f328bc0f640 10 monclient: init 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 0x7f328409aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f32840a1a00 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 0x7f328409b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f328409d5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3284096a70 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328409cf80 con 0x7f32840a1a00 2024-09-15T02:00:05.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328409cc40 con 0x7f3284096a70 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840a1dd0 con 0x7f328409d5a0 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328bc0f640 10 monclient(hunting): authenticate will time out at 1237.585120s 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45592/0 (socket says 172.21.15.67:45592) 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 -- 172.21.15.67:0/30363789 learned_addr learned my addr 172.21.15.67:0/30363789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 10 monclient(hunting): get_auth_request con 0x7f3284096a70 auth_method 0 2024-09-15T02:00:05.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 -- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 msgr2=0x7f3284096ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 -- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 msgr2=0x7f3284096ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 --2- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 0x7f328409b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 0x7f328409aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f328a1ac640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): get_auth_request con 0x7f328409d5a0 auth_method 0 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.124 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): get_auth_request con 0x7f32840a1a00 auth_method 0 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient(hunting): handle_auth_done global_id 94697 payload 274 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 -- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 msgr2=0x7f3284096ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 -- 172.21.15.67:0/30363789 >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 msgr2=0x7f328409aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 0x7f328409aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient: _finish_hunting 0 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 monclient: found mon.noname-b 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32891aa640 1 -- 172.21.15.67:0/30363789 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328412efa0 con 0x7f328409d5a0 2024-09-15T02:00:05.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.116+0000 7f32899ab640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 0x7f328409aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:05.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.119+0000 7f32891aa640 1 -- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 msgr2=0x7f328409b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:05.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.119+0000 7f32891aa640 1 --2- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 0x7f328409b020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3284099a80 tx=0x7f32780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f37e6b9013bbaed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:05.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 1 -- 172.21.15.67:0/30363789 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327802adf0 con 0x7f328409d5a0 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: dump: 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:05.132 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: _finish_auth 0 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 5 monclient: authenticate success, global_id 94697 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 1 -- 172.21.15.67:0/30363789 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278032440 con 0x7f328409d5a0 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 4 set_mon_vals no callback set 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 1 -- 172.21.15.67:0/30363789 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780327e0 con 0x7f328409d5a0 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f32889a9640 10 monclient: dump: 2024-09-15T02:00:05.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 10 monclient: shutdown 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 -- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 msgr2=0x7f328409b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 --2- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 0x7f328409b020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3284099a80 tx=0x7f32780090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 -- 172.21.15.67:0/30363789 shutdown_connections 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.67:3300/0 conn(0x7f3284096a70 0x7f3284096ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 --2- 172.21.15.67:0/30363789 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f328409d5a0 0x7f328409b020 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 --2- 172.21.15.67:0/30363789 >> v2:172.21.15.17:3300/0 conn(0x7f32840a1a00 0x7f328409aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 -- 172.21.15.67:0/30363789 >> 172.21.15.67:0/30363789 conn(0x7f3284071340 msgr2=0x7f32840a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 -- 172.21.15.67:0/30363789 shutdown_connections 2024-09-15T02:00:05.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.120+0000 7f328bc0f640 1 -- 172.21.15.67:0/30363789 wait complete. 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient: build_initial_monmap 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient: monmap: 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.136 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 1 Processor -- start 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 1 -- start start 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient: init 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 0x7f328406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3284096a70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 0x7f3284141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.121+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f328409d5a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:05.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f32840a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284141f30 con 0x7f328409d5a0 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328406e7f0 con 0x7f32840a1a00 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3284132650 con 0x7f3284096a70 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328bc0f640 10 monclient(hunting): authenticate will time out at 1237.591029s 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45598/0 (socket says 172.21.15.67:45598) 2024-09-15T02:00:05.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 -- 172.21.15.67:0/4277642466 learned_addr learned my addr 172.21.15.67:0/4277642466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 10 monclient(hunting): get_auth_request con 0x7f32840a1a00 auth_method 0 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 0x7f328406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): get_auth_request con 0x7f3284096a70 auth_method 0 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 0x7f3284141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): get_auth_request con 0x7f328409d5a0 auth_method 0 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:05.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 msgr2=0x7f3284134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 msgr2=0x7f3284134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f328a1ac640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient(hunting): handle_auth_done global_id 104553 payload 824 2024-09-15T02:00:05.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 msgr2=0x7f328406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 0x7f328406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 msgr2=0x7f3284134e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient: _finish_hunting 0 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 monclient: found mon.noname-a 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32891aa640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274009a20 con 0x7f328409d5a0 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.122+0000 7f32899ab640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 0x7f328406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.124+0000 7f32891aa640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 msgr2=0x7f3284141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.124+0000 7f32891aa640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 0x7f3284141520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32840a1f10 tx=0x7f3278032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278031a10 con 0x7f328409d5a0 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:05.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: dump: 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 5 monclient: authenticate success, global_id 104553 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278031b70 con 0x7f328409d5a0 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 10 monclient: _renew_subs 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3278009390 con 0x7f328409d5a0 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327803d440 con 0x7f328409d5a0 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3281ffb640 10 monclient: dump: 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:05.143 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 10 monclient: _renew_subs 2024-09-15T02:00:05.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3284132090 con 0x7f328409d5a0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.125+0000 7f3283fff640 4 set_mon_vals no callback set 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3278039040 con 0x7f328409d5a0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f328bc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f328bc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f328bc0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328404bf60 con 0x7f328409d5a0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f3281ffb640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(231..231 src has 148..231) v4 ==== 12011+0+0 (secure 0 0 0) 0x7f327806d550 con 0x7f328409d5a0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f32899ab640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.126+0000 7f32899ab640 10 monclient: get_auth_request con 0x7f3270038740 auth_method 0 2024-09-15T02:00:05.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.129+0000 7f32899ab640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 0x7f327003abf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f327400eae0 tx=0x7f327400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:05.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.131+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f327802f2a0 con 0x7f328409d5a0 2024-09-15T02:00:05.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.131+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:05.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.131+0000 7f3281ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:05.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.301+0000 7f328bc0f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:05.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.301+0000 7f328bc0f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:05.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.301+0000 7f328bc0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:05.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.301+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f3284067180 con 0x7f328409d5a0 2024-09-15T02:00:05.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.932+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3278042090 con 0x7f328409d5a0 2024-09-15T02:00:05.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.932+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:05.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:05.932+0000 7f3281ffb640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-15T02:00:06.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.040+0000 7f328bc0f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.040+0000 7f328bc0f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.040+0000 7f328bc0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.040+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f3284132280 con 0x7f328409d5a0 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.041+0000 7f3281ffb640 1 -- 172.21.15.67:0/4277642466 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3278037070 con 0x7f328409d5a0 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.041+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T02:00:06.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.041+0000 7f3281ffb640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-15T02:00:06.044 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h15' does not appear in the crush map 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 msgr2=0x7f327003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 0x7f327003abf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f327400eae0 tx=0x7f327400b040 comp rx=0 tx=0).stop 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 10 monclient: shutdown 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 msgr2=0x7f3284141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 0x7f3284141520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32840a1f10 tx=0x7f3278032000 comp rx=0 tx=0).stop 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 shutdown_connections 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f328409d5a0 0x7f3284141520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840a1a00 0x7f3284134e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 --2- 172.21.15.67:0/4277642466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3284096a70 0x7f328406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 >> 172.21.15.67:0/4277642466 conn(0x7f3284071340 msgr2=0x7f32840a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:06.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 shutdown_connections 2024-09-15T02:00:06.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.043+0000 7f328bc0f640 1 -- 172.21.15.67:0/4277642466 wait complete. 2024-09-15T02:00:06.055 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:06.055 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h16 2024-09-15T02:00:06.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.129+0000 7f16c879d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:06.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.129+0000 7f16c879d640 10 monclient: build_initial_monmap 2024-09-15T02:00:06.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.129+0000 7f16c879d640 10 monclient: monmap: 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:06.131066+0000 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:06.131066+0000 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:06.133 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.130+0000 7f16c879d640 1 Processor -- start 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.130+0000 7f16c879d640 1 -- start start 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.130+0000 7f16c879d640 10 monclient: init 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 0x7f16c01321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-b con 0x7f16c0098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-c con 0x7f16c00a0370 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:06.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16c0098a50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c0098ef0 con 0x7f16c0098a50 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c00a1730 con 0x7f16c00a0370 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00a0b80 con 0x7f16c0098140 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c879d640 10 monclient(hunting): authenticate will time out at 1238.600319s 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45606/0 (socket says 172.21.15.67:45606) 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 -- 172.21.15.67:0/1219407041 learned_addr learned my addr 172.21.15.67:0/1219407041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 10 monclient(hunting): get_auth_request con 0x7f16c00a0370 auth_method 0 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:06.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6d3a640 1 --1- 172.21.15.67:0/1219407041 >> v1:172.21.15.67:6789/0 conn(0x7f16c0098a50 0x7f16c00695c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42154/0 (socket says 172.21.15.67:42154) 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 -- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 msgr2=0x7f16c0098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 -- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 msgr2=0x7f16c0098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 --2- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c5d38640 1 --2- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 --2- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 0x7f16c01321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): get_auth_request con 0x7f16c0098140 auth_method 0 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:06.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient(hunting): handle_auth_done global_id 94703 payload 274 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 -- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 msgr2=0x7f16c0098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 --2- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 -- 172.21.15.67:0/1219407041 >> v1:172.21.15.67:6789/0 conn(0x7f16c0098a50 legacy=0x7f16c00695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient: _finish_hunting 0 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 monclient: found mon.noname-b 2024-09-15T02:00:06.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:06.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:06.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.131+0000 7f16c6539640 1 -- 172.21.15.67:0/1219407041 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c00a00b0 con 0x7f16c0098140 2024-09-15T02:00:06.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.134+0000 7f16c6539640 1 -- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 msgr2=0x7f16c01321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:06.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.134+0000 7f16c6539640 1 --2- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 0x7f16c01321e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f16c00a52f0 tx=0x7f16b40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4766beb83781eff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:06.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 1 -- 172.21.15.67:0/1219407041 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b402adb0 con 0x7f16c0098140 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: dump: 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: _finish_auth 0 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 1 -- 172.21.15.67:0/1219407041 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4032440 con 0x7f16c0098140 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 1 -- 172.21.15.67:0/1219407041 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b40327e0 con 0x7f16c0098140 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 5 monclient: authenticate success, global_id 94703 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 4 set_mon_vals no callback set 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c5537640 10 monclient: dump: 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:06.147 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 10 monclient: shutdown 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 -- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 msgr2=0x7f16c01321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 --2- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 0x7f16c01321e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f16c00a52f0 tx=0x7f16b40090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 -- 172.21.15.67:0/1219407041 shutdown_connections 2024-09-15T02:00:06.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 --2- 172.21.15.67:0/1219407041 >> v2:172.21.15.67:3300/0 conn(0x7f16c00a0370 0x7f16c0098510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 --2- 172.21.15.67:0/1219407041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098140 0x7f16c01321e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 -- 172.21.15.67:0/1219407041 >> 172.21.15.67:0/1219407041 conn(0x7f16c00a5780 msgr2=0x7f16c00a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 -- 172.21.15.67:0/1219407041 shutdown_connections 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.135+0000 7f16c879d640 1 -- 172.21.15.67:0/1219407041 wait complete. 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.136+0000 7f16c879d640 10 monclient: build_initial_monmap 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.136+0000 7f16c879d640 10 monclient: monmap: 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:06.149 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.136+0000 7f16c879d640 1 Processor -- start 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.136+0000 7f16c879d640 1 -- start start 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.136+0000 7f16c879d640 10 monclient: init 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 0x7f16c009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-a con 0x7f16c0098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 0x7f16c0130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-c con 0x7f16c0098a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:06.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): picked mon.noname-b con 0x7f16c00a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c011cb00 con 0x7f16c0098140 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0131ba0 con 0x7f16c00a0370 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0099090 con 0x7f16c0098a50 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c879d640 10 monclient(hunting): authenticate will time out at 1238.606169s 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45614/0 (socket says 172.21.15.67:45614) 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 -- 172.21.15.67:0/695077068 learned_addr learned my addr 172.21.15.67:0/695077068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 10 monclient(hunting): get_auth_request con 0x7f16c00a0370 auth_method 0 2024-09-15T02:00:06.151 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 msgr2=0x7f16c0131460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 msgr2=0x7f16c0131460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6d3a640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 0x7f16c0130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 0x7f16c009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): get_auth_request con 0x7f16c0098140 auth_method 0 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): get_auth_request con 0x7f16c0098a50 auth_method 0 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient(hunting): handle_auth_done global_id 104556 payload 824 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 msgr2=0x7f16c0130f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 0x7f16c0130f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 msgr2=0x7f16c0131460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient: _finish_hunting 0 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 monclient: found mon.noname-a 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c6539640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4009390 con 0x7f16c0098140 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.137+0000 7f16c5d38640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 0x7f16c0130f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c6539640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 msgr2=0x7f16c009b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c6539640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 0x7f16c009b4a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f16b4031000 tx=0x7f16b4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4031990 con 0x7f16c0098140 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: dump: 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 5 monclient: authenticate success, global_id 104556 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 10 monclient: _renew_subs 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c011e040 con 0x7f16c0098140 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 10 monclient: _renew_subs 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c0142190 con 0x7f16c0098140 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4031d00 con 0x7f16c0098140 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4031e90 con 0x7f16c0098140 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16b27fc640 10 monclient: dump: 2024-09-15T02:00:06.156 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.140+0000 7f16c4d36640 4 set_mon_vals no callback set 2024-09-15T02:00:06.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16c879d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16c879d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c004bf60 con 0x7f16c0098140 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f16b4039040 con 0x7f16c0098140 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16b27fc640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 0x7f16a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.141+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(232..232 src has 148..232) v4 ==== 11798+0+0 (secure 0 0 0) 0x7f16b406d790 con 0x7f16c0098140 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.142+0000 7f16c5d38640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 0x7f16a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:06.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.142+0000 7f16c5d38640 10 monclient: get_auth_request con 0x7f16a4038740 auth_method 0 2024-09-15T02:00:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.144+0000 7f16c5d38640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 0x7f16a403abf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16c011dc60 tx=0x7f16bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.146+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f16b4030340 con 0x7f16c0098140 2024-09-15T02:00:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.146+0000 7f16b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:06.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.146+0000 7f16b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:06.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.318+0000 7f16c879d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:06.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.318+0000 7f16c879d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:06.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.318+0000 7f16c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:06.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.318+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f16c0098ef0 con 0x7f16c0098140 2024-09-15T02:00:06.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.934+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f16c0098ef0 con 0x7f16c0098140 2024-09-15T02:00:06.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.934+0000 7f16b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:06.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:06.934+0000 7f16b27fc640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-15T02:00:07.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.050+0000 7f16c879d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.050+0000 7f16c879d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.050+0000 7f16c879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.050+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f16c0098ef0 con 0x7f16c0098140 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.051+0000 7f16b27fc640 1 -- 172.21.15.67:0/695077068 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f16c0098ef0 con 0x7f16c0098140 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.051+0000 7f16b27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.051+0000 7f16b27fc640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-15T02:00:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h16' does not appear in the crush map 2024-09-15T02:00:07.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 msgr2=0x7f16a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 0x7f16a403abf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16c011dc60 tx=0x7f16bc008040 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 10 monclient: shutdown 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 msgr2=0x7f16c009b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 0x7f16c009b4a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f16b4031000 tx=0x7f16b4032000 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 shutdown_connections 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a4038740 0x7f16a403abf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c00a0370 0x7f16c0131460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c0098a50 0x7f16c0130f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 --2- 172.21.15.67:0/695077068 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c0098140 0x7f16c009b4a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 >> 172.21.15.67:0/695077068 conn(0x7f16c00a5780 msgr2=0x7f16c0134e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 shutdown_connections 2024-09-15T02:00:07.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.053+0000 7f16c879d640 1 -- 172.21.15.67:0/695077068 wait complete. 2024-09-15T02:00:07.064 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:07.064 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h17 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.137+0000 7fc420e99640 10 monclient: get_monmap_and_config 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.137+0000 7fc420e99640 10 monclient: build_initial_monmap 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.137+0000 7fc420e99640 10 monclient: monmap: 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.140 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:07.139119+0000 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:07.139119+0000 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:07.141 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.138+0000 7fc420e99640 1 Processor -- start 2024-09-15T02:00:07.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.138+0000 7fc420e99640 1 -- start start 2024-09-15T02:00:07.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.138+0000 7fc420e99640 10 monclient: init 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c0a1a00 0x7fc41c09aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c0a1a00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc41c09d5a0 0x7fc41c09b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c09d5a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc41c096a70 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c09cc40 con 0x7fc41c096a70 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c09cf80 con 0x7fc41c09d5a0 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0a1dd0 con 0x7fc41c0a1a00 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc420e99640 10 monclient(hunting): authenticate will time out at 1239.608262s 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc41c09d5a0 0x7fc41c09b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c0a1a00 0x7fc41c09aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41ad9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc41c096a70 0x7fc41c096ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:59140/0 (socket says 172.21.15.67:59140) 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41ad9d640 1 -- 172.21.15.67:0/2998817028 learned_addr learned my addr 172.21.15.67:0/2998817028 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): get_auth_request con 0x7fc41c09d5a0 auth_method 0 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): get_auth_request con 0x7fc41c0a1a00 auth_method 0 2024-09-15T02:00:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient(hunting): handle_auth_done global_id 104559 payload 274 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 -- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c0a1a00 msgr2=0x7fc41c09aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 --2- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c0a1a00 0x7fc41c09aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 -- 172.21.15.67:0/2998817028 >> v1:172.21.15.17:6789/0 conn(0x7fc41c096a70 legacy=0x7fc41c096ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:00:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 monclient: found mon.noname-a 2024-09-15T02:00:07.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:07.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:07.147 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.139+0000 7fc419d9b640 1 -- 172.21.15.67:0/2998817028 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c12efa0 con 0x7fc41c09d5a0 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.142+0000 7fc419d9b640 1 -- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 msgr2=0x7fc41c09b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.142+0000 7fc419d9b640 1 --2- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c09b020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc410009850 tx=0x7fc4100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ab1ed754429c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 1 -- 172.21.15.67:0/2998817028 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41002adf0 con 0x7fc41c09d5a0 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: dump: 2024-09-15T02:00:07.152 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: _finish_auth 0 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 1 -- 172.21.15.67:0/2998817028 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc410032440 con 0x7fc41c09d5a0 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:07.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 1 -- 172.21.15.67:0/2998817028 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4100327e0 con 0x7fc41c09d5a0 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 5 monclient: authenticate success, global_id 104559 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 4 set_mon_vals no callback set 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc41959a640 10 monclient: dump: 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.154 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 10 monclient: shutdown 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 -- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 msgr2=0x7fc41c09b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 --2- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c09b020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc410009850 tx=0x7fc4100090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 -- 172.21.15.67:0/2998817028 shutdown_connections 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 --2- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c09b020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 --2- 172.21.15.67:0/2998817028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c0a1a00 0x7fc41c09aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 -- 172.21.15.67:0/2998817028 >> 172.21.15.67:0/2998817028 conn(0x7fc41c071340 msgr2=0x7fc41c0a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 -- 172.21.15.67:0/2998817028 shutdown_connections 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.143+0000 7fc420e99640 1 -- 172.21.15.67:0/2998817028 wait complete. 2024-09-15T02:00:07.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient: build_initial_monmap 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient: monmap: 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 Processor -- start 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 -- start start 2024-09-15T02:00:07.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient: init 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 0x7fc41c136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c096a70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c132310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c09d5a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c0a1a00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0985b0 con 0x7fc41c09d5a0 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c134280 con 0x7fc41c0a1a00 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0a09d0 con 0x7fc41c096a70 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.144+0000 7fc420e99640 10 monclient(hunting): authenticate will time out at 1239.613887s 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45618/0 (socket says 172.21.15.67:45618) 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 0x7fc41c136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 -- 172.21.15.67:0/1680966572 learned_addr learned my addr 172.21.15.67:0/1680966572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 10 monclient(hunting): get_auth_request con 0x7fc41c0a1a00 auth_method 0 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): get_auth_request con 0x7fc41c096a70 auth_method 0 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c132310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): get_auth_request con 0x7fc41c09d5a0 auth_method 0 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 msgr2=0x7fc41c132850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 msgr2=0x7fc41c132850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41ad9d640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:07.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient(hunting): handle_auth_done global_id 94712 payload 824 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 msgr2=0x7fc41c132850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 msgr2=0x7fc41c132310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c132310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 monclient: found mon.noname-c 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc41a59c640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009390 con 0x7fc41c096a70 2024-09-15T02:00:07.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.145+0000 7fc419d9b640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c132310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc41a59c640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 msgr2=0x7fc41c136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc41a59c640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 0x7fc41c136b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e960 tx=0x7fc40c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c009540 con 0x7fc41c096a70 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 10 monclient: dump: 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.161 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c00cb50 con 0x7fc41c096a70 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c013470 con 0x7fc41c096a70 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc420e99640 5 monclient: authenticate success, global_id 94712 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc40a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc40a7fc640 10 monclient: dump: 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:07.162 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.147+0000 7fc418d99640 4 set_mon_vals no callback set 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _renew_subs 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c135900 con 0x7fc41c096a70 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _renew_subs 2024-09-15T02:00:07.163 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c1352d0 con 0x7fc41c096a70 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc40c013860 con 0x7fc41c096a70 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.148+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c135d10 con 0x7fc41c096a70 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.149+0000 7fc40a7fc640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 0x7fc3fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.149+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(233..233 src has 148..233) v4 ==== 11585+0+0 (secure 0 0 0) 0x7fc40c04d870 con 0x7fc41c096a70 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.149+0000 7fc419d9b640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 0x7fc3fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.164 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.149+0000 7fc419d9b640 10 monclient: get_auth_request con 0x7fc3fc038740 auth_method 0 2024-09-15T02:00:07.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.151+0000 7fc419d9b640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 0x7fc3fc03abf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc41c099a80 tx=0x7fc410032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.153+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc40c023a90 con 0x7fc41c096a70 2024-09-15T02:00:07.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.153+0000 7fc40a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.165 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.153+0000 7fc40a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:07.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.319+0000 7fc420e99640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.319+0000 7fc420e99640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.319+0000 7fc420e99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.322 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.319+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7fc41c133dd0 con 0x7fc41c096a70 2024-09-15T02:00:07.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:07.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.803+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc40c021030 con 0x7fc41c096a70 2024-09-15T02:00:07.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.803+0000 7fc40a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.803+0000 7fc40a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-15T02:00:07.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.861+0000 7ff192d5d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:07.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.861+0000 7ff192d5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:07.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.861+0000 7ff192d5d640 10 monclient: monmap: 2024-09-15T02:00:07.863 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:07.862373+0000 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:07.862373+0000 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:07.864 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 Processor -- start 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 -- start start 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient: init 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff18c0692f0 0x7ff18c0696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c0692f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff18c12e900 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0a1950 con 0x7ff18c0692f0 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0a1670 con 0x7ff18c12e900 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0a0020 con 0x7ff18c068f20 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.862+0000 7ff192d5d640 10 monclient(hunting): authenticate will time out at 1267.858886s 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff18c0692f0 0x7ff18c0696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff18c0692f0 0x7ff18c0696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46864/0 (socket says 172.21.15.17:46864) 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 -- 172.21.15.17:0/881203834 learned_addr learned my addr 172.21.15.17:0/881203834 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): get_auth_request con 0x7ff18c0692f0 auth_method 0 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): get_auth_request con 0x7ff18c068f20 auth_method 0 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 1 --2- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 10 monclient(hunting): get_auth_request con 0x7ff18c12e900 auth_method 0 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 1 -- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 msgr2=0x7ff18c12ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 1 -- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 msgr2=0x7ff18c12ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 1 --2- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff1912fa640 1 --2- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient(hunting): handle_auth_done global_id 104565 payload 274 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 -- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 msgr2=0x7ff18c1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 --2- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c1321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 -- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 msgr2=0x7ff18c12ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:07.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 --2- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:07.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 monclient: found mon.noname-a 2024-09-15T02:00:07.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:07.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:07.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.863+0000 7ff18bfff640 1 -- 172.21.15.17:0/881203834 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0a1390 con 0x7ff18c0692f0 2024-09-15T02:00:07.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18bfff640 1 -- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c0692f0 msgr2=0x7ff18c0696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:07.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18bfff640 1 --2- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c0692f0 0x7ff18c0696c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0a2f70 tx=0x7ff1780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d29a6e01e836e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 1 -- 172.21.15.17:0/881203834 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178032440 con 0x7ff18c0692f0 2024-09-15T02:00:07.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: dump: 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:07.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 5 monclient: authenticate success, global_id 104565 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 1 -- 172.21.15.17:0/881203834 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1780325a0 con 0x7ff18c0692f0 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 4 set_mon_vals no callback set 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 1 -- 172.21.15.17:0/881203834 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178032940 con 0x7ff18c0692f0 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff18b7fe640 10 monclient: dump: 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 10 monclient: shutdown 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:07.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 1 -- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c0692f0 msgr2=0x7ff18c0696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.866+0000 7ff192d5d640 1 --2- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c0692f0 0x7ff18c0696c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0a2f70 tx=0x7ff1780091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 -- 172.21.15.17:0/881203834 shutdown_connections 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 --2- 172.21.15.17:0/881203834 >> v2:172.21.15.67:3300/0 conn(0x7ff18c12e900 0x7ff18c12ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 --2- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c0692f0 0x7ff18c0696c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 --2- 172.21.15.17:0/881203834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c1321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 -- 172.21.15.17:0/881203834 >> 172.21.15.17:0/881203834 conn(0x7ff18c070fb0 msgr2=0x7ff18c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 -- 172.21.15.17:0/881203834 shutdown_connections 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 -- 172.21.15.17:0/881203834 wait complete. 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 10 monclient: monmap: 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:07.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.867+0000 7ff192d5d640 1 Processor -- start 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 -- start start 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient: init 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff18c068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c0692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c12e900 0x7ff18c11b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c12e900 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c11fb50 con 0x7ff18c12e900 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c130cb0 con 0x7ff18c0692f0 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c11ce20 con 0x7ff18c068f20 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:07.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff192d5d640 10 monclient(hunting): authenticate will time out at 1267.864728s 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48538/0 (socket says 172.21.15.17:48538) 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 1 -- 172.21.15.17:0/2332595429 learned_addr learned my addr 172.21.15.17:0/2332595429 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): get_auth_request con 0x7ff18c068f20 auth_method 0 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c12e900 0x7ff18c11b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): get_auth_request con 0x7ff18c12e900 auth_method 0 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff190af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 10 monclient(hunting): get_auth_request con 0x7ff18c0692f0 auth_method 0 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff1912fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 msgr2=0x7ff18c141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.868+0000 7ff18bfff640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 msgr2=0x7ff18c141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff18bfff640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff18bfff640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 10 monclient(hunting): handle_auth_done global_id 94718 payload 824 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 msgr2=0x7ff18c141ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c12e900 msgr2=0x7ff18c11b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c12e900 0x7ff18c11b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 10 monclient: _finish_hunting 0 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 monclient: found mon.noname-c 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.869+0000 7ff190af9640 1 -- 172.21.15.17:0/2332595429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178009450 con 0x7ff18c068f20 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff190af9640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 msgr2=0x7ff18c141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff190af9640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff18000ea20 tx=0x7ff1800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180009600 con 0x7ff18c068f20 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: dump: 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: _finish_auth 0 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000cc20 con 0x7ff18c068f20 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 5 monclient: authenticate success, global_id 94718 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 10 monclient: _renew_subs 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c11e950 con 0x7ff18c068f20 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180013470 con 0x7ff18c068f20 2024-09-15T02:00:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff188ff9640 10 monclient: dump: 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:07.888 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 10 monclient: _renew_subs 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff18affd640 4 set_mon_vals no callback set 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.871+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c11e660 con 0x7ff18c068f20 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff180013860 con 0x7ff18c068f20 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff188ff9640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 0x7ff17003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff192d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff192d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff192d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c0724a0 con 0x7ff18c068f20 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(234..234 src has 148..234) v4 ==== 11372+0+0 (secure 0 0 0) 0x7ff18004d8a0 con 0x7ff18c068f20 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.872+0000 7ff18bfff640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 0x7ff17003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.873+0000 7ff18bfff640 10 monclient: get_auth_request con 0x7ff170038740 auth_method 0 2024-09-15T02:00:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.875+0000 7ff18bfff640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 0x7ff17003abf0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff18c071760 tx=0x7ff178032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.877+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff18003f070 con 0x7ff18c068f20 2024-09-15T02:00:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.877+0000 7ff188ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:07.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:07.877+0000 7ff188ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:07.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.907+0000 7fc420e99640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.907+0000 7fc420e99640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.907+0000 7fc420e99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:07.910 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.907+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7fc41c04bf60 con 0x7fc41c096a70 2024-09-15T02:00:07.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.908+0000 7fc40a7fc640 1 -- 172.21.15.67:0/1680966572 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc40c021020 con 0x7fc41c096a70 2024-09-15T02:00:07.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.908+0000 7fc40a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T02:00:07.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.908+0000 7fc40a7fc640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-15T02:00:07.911 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h17' does not appear in the crush map 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 msgr2=0x7fc3fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 0x7fc3fc03abf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc41c099a80 tx=0x7fc410032000 comp rx=0 tx=0).stop 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 10 monclient: shutdown 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 msgr2=0x7fc41c136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 0x7fc41c136b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e960 tx=0x7fc40c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 shutdown_connections 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc3fc038740 0x7fc3fc03abf0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc41c09d5a0 0x7fc41c132310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc41c0a1a00 0x7fc41c132850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 --2- 172.21.15.67:0/1680966572 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc41c096a70 0x7fc41c136b10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 >> 172.21.15.67:0/1680966572 conn(0x7fc41c071340 msgr2=0x7fc41c0a0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 shutdown_connections 2024-09-15T02:00:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.910+0000 7fc420e99640 1 -- 172.21.15.67:0/1680966572 wait complete. 2024-09-15T02:00:07.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:07.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h18 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.995+0000 7f607159d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.995+0000 7f607159d640 10 monclient: build_initial_monmap 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.995+0000 7f607159d640 10 monclient: monmap: 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:07.997043+0000 2024-09-15T02:00:07.998 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:07.997043+0000 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:07.999 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.996+0000 7f607159d640 1 Processor -- start 2024-09-15T02:00:08.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.996+0000 7f607159d640 1 -- start start 2024-09-15T02:00:08.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.996+0000 7f607159d640 10 monclient: init 2024-09-15T02:00:08.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f606c0a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 0x7f606c09f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): picked mon.noname-b con 0x7f606c09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): picked mon.noname-c con 0x7f606c136e40 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0a61a0 con 0x7f606c0a0b50 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0a1610 con 0x7f606c136e40 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:08.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c09f7d0 con 0x7f606c09eea0 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f606c0a0b50 0x7f606c069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42166/0 (socket says 172.21.15.67:42166) 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606affd640 1 -- 172.21.15.67:0/1459207075 learned_addr learned my addr 172.21.15.67:0/1459207075 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f607159d640 10 monclient(hunting): authenticate will time out at 1240.466262s 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 10 monclient(hunting): get_auth_request con 0x7f606c136e40 auth_method 0 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:08.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 --2- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 0x7f606c09f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): get_auth_request con 0x7f606c09eea0 auth_method 0 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 1 -- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 msgr2=0x7f606c139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 1 -- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 msgr2=0x7f606c139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 1 --2- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606b7fe640 1 --2- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:08.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient(hunting): handle_auth_done global_id 94721 payload 274 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 -- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 msgr2=0x7f606c139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 --2- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 -- 172.21.15.67:0/1459207075 >> v1:172.21.15.67:6789/0 conn(0x7f606c0a0b50 legacy=0x7f606c069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 monclient: found mon.noname-b 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:08.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:07.997+0000 7f606a7fc640 1 -- 172.21.15.67:0/1459207075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0a04c0 con 0x7f606c09eea0 2024-09-15T02:00:08.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f606a7fc640 1 -- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 msgr2=0x7f606c09f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:08.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f606a7fc640 1 --2- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 0x7f606c09f290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f605c009850 tx=0x7f605c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01522e1b6f94215 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 1 -- 172.21.15.67:0/1459207075 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c02adf0 con 0x7f606c09eea0 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: dump: 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.013 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 1 -- 172.21.15.67:0/1459207075 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c032440 con 0x7f606c09eea0 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 1 -- 172.21.15.67:0/1459207075 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c0327e0 con 0x7f606c09eea0 2024-09-15T02:00:08.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f6069ffb640 10 monclient: dump: 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:08.015 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f607159d640 5 monclient: authenticate success, global_id 94721 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f607159d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.000+0000 7f607159d640 4 set_mon_vals no callback set 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 10 monclient: shutdown 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 -- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 msgr2=0x7f606c09f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 --2- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 0x7f606c09f290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f605c009850 tx=0x7f605c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 -- 172.21.15.67:0/1459207075 shutdown_connections 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 --2- 172.21.15.67:0/1459207075 >> v2:172.21.15.67:3300/0 conn(0x7f606c136e40 0x7f606c139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 --2- 172.21.15.67:0/1459207075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c09eea0 0x7f606c09f290 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 -- 172.21.15.67:0/1459207075 >> 172.21.15.67:0/1459207075 conn(0x7f606c071330 msgr2=0x7f606c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 -- 172.21.15.67:0/1459207075 shutdown_connections 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.001+0000 7f607159d640 1 -- 172.21.15.67:0/1459207075 wait complete. 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient: build_initial_monmap 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient: monmap: 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:08.017 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 Processor -- start 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 -- start start 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient: init 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): picked mon.noname-b con 0x7f606c09eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 0x7f606c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): picked mon.noname-c con 0x7f606c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 0x7f606c11b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): picked mon.noname-a con 0x7f606c136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c11dda0 con 0x7f606c136e40 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c11e2d0 con 0x7f606c09eea0 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0a1260 con 0x7f606c0a0b50 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.002+0000 7f607159d640 10 monclient(hunting): authenticate will time out at 1240.471971s 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45636/0 (socket says 172.21.15.67:45636) 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 -- 172.21.15.67:0/3743039250 learned_addr learned my addr 172.21.15.67:0/3743039250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 10 monclient(hunting): get_auth_request con 0x7f606c09eea0 auth_method 0 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 0x7f606c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 msgr2=0x7f606c09a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 msgr2=0x7f606c09a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): get_auth_request con 0x7f606c0a0b50 auth_method 0 2024-09-15T02:00:08.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 0x7f606c11b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): get_auth_request con 0x7f606c136e40 auth_method 0 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606affd640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:08.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient(hunting): handle_auth_done global_id 104571 payload 824 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 msgr2=0x7f606c131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 0x7f606c131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 msgr2=0x7f606c09a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 monclient: found mon.noname-a 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606b7fe640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c009390 con 0x7f606c136e40 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.003+0000 7f606a7fc640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 0x7f606c131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f606b7fe640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 msgr2=0x7f606c11b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:08.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f606b7fe640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 0x7f606c11b940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6064007e70 tx=0x7f60640097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064018070 con 0x7f606c136e40 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 10 monclient: dump: 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:08.023 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f607159d640 5 monclient: authenticate success, global_id 104571 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064014420 con 0x7f606c136e40 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60640147c0 con 0x7f606c136e40 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f607159d640 10 monclient: _renew_subs 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c1420d0 con 0x7f606c136e40 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.005+0000 7f60697fa640 4 set_mon_vals no callback set 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f6062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f6062ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f6062ffd640 10 monclient: dump: 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 monclient: _renew_subs 2024-09-15T02:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c11cae0 con 0x7f606c136e40 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6064003020 con 0x7f606c136e40 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.006+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c04bf60 con 0x7f606c136e40 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.007+0000 7f6062ffd640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 0x7f604c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.007+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(234..234 src has 148..234) v4 ==== 11372+0+0 (secure 0 0 0) 0x7f606404d6a0 con 0x7f606c136e40 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.007+0000 7f606affd640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 0x7f604c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.007+0000 7f606affd640 10 monclient: get_auth_request con 0x7f604c038770 auth_method 0 2024-09-15T02:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.009+0000 7f606affd640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 0x7f604c03ac20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6054007dd0 tx=0x7f6054009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.012+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6064009920 con 0x7f606c136e40 2024-09-15T02:00:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.012+0000 7f6062ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.012+0000 7f6062ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:08.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.105+0000 7ff192d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:08.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.105+0000 7ff192d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:08.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.105+0000 7ff192d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:08.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.105+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff18c12feb0 con 0x7ff18c068f20 2024-09-15T02:00:08.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.106+0000 7ff188ff9640 1 -- 172.21.15.17:0/2332595429 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7ff180023510 con 0x7ff18c068f20 2024-09-15T02:00:08.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.106+0000 7ff188ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:08.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.106+0000 7ff188ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:08.108 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 msgr2=0x7ff17003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 0x7ff17003abf0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff18c071760 tx=0x7ff178032000 comp rx=0 tx=0).stop 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 10 monclient: shutdown 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 msgr2=0x7ff18c141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff18000ea20 tx=0x7ff1800090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 shutdown_connections 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff170038740 0x7ff17003abf0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff18c12e900 0x7ff18c11b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff18c0692f0 0x7ff18c141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 --2- 172.21.15.17:0/2332595429 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff18c068f20 0x7ff18c141760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:08.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 >> 172.21.15.17:0/2332595429 conn(0x7ff18c070fb0 msgr2=0x7ff18c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:08.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 shutdown_connections 2024-09-15T02:00:08.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:08.108+0000 7ff192d5d640 1 -- 172.21.15.17:0/2332595429 wait complete. 2024-09-15T02:00:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.179+0000 7f607159d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.179+0000 7f607159d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.179+0000 7f607159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:08.179+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f606c06f000 con 0x7f606c136e40 2024-09-15T02:00:09.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.180+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6064005cf0 con 0x7f606c136e40 2024-09-15T02:00:09.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.180+0000 7f6062ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:09.181 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.180+0000 7f6062ffd640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-15T02:00:09.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.286+0000 7f607159d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:09.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.286+0000 7f607159d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:09.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.286+0000 7f607159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.286+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f606c11cdb0 con 0x7f606c136e40 2024-09-15T02:00:09.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.287+0000 7f6062ffd640 1 -- 172.21.15.67:0/3743039250 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f60640122e0 con 0x7f606c136e40 2024-09-15T02:00:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.287+0000 7f6062ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T02:00:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.287+0000 7f6062ffd640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-15T02:00:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h18' does not appear in the crush map 2024-09-15T02:00:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.289+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 msgr2=0x7f604c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 0x7f604c03ac20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6054007dd0 tx=0x7f6054009040 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 10 monclient: shutdown 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 msgr2=0x7f606c11b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 0x7f606c11b940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6064007e70 tx=0x7f60640097e0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 shutdown_connections 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f604c038770 0x7f604c03ac20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f606c136e40 0x7f606c11b940 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f606c0a0b50 0x7f606c131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 --2- 172.21.15.67:0/3743039250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f606c09eea0 0x7f606c09a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 >> 172.21.15.67:0/3743039250 conn(0x7f606c071330 msgr2=0x7f606c09dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 shutdown_connections 2024-09-15T02:00:09.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.290+0000 7f607159d640 1 -- 172.21.15.67:0/3743039250 wait complete. 2024-09-15T02:00:09.300 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:09.300 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h19 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.376+0000 7f827a6b8640 10 monclient: get_monmap_and_config 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.376+0000 7f827a6b8640 10 monclient: build_initial_monmap 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.376+0000 7f827a6b8640 10 monclient: monmap: 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:09.376326+0000 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:09.376326+0000 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:09.378 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.376+0000 7f827a6b8640 1 Processor -- start 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.376+0000 7f827a6b8640 1 -- start start 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient: init 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8274098140 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 0x7f8274098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f82740a0370 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8274098a50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8274098ef0 con 0x7f8274098140 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740a0b80 con 0x7f8274098a50 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82741328f0 con 0x7f82740a0370 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f827a6b8640 10 monclient(hunting): authenticate will time out at 1241.845585s 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 0x7f8274098510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f8278c55640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8274098a50 0x7f82741390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42174/0 (socket says 172.21.15.67:42174) 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 learned_addr learned my addr 172.21.15.67:0/4039795350 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f82740a0370 auth_method 0 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:09.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 1 -- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 msgr2=0x7f8274098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:09.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 1 -- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 msgr2=0x7f8274098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:09.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 1 --2- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 0x7f8274098510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:09.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.377+0000 7f82737fe640 1 --2- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 0x7f8274098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.379+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 >> v1:172.21.15.67:6789/0 conn(0x7f8274098a50 legacy=0x7f82741390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.379+0000 7f8273fff640 1 -- 172.21.15.67:0/4039795350 >> v1:172.21.15.17:6789/0 conn(0x7f8274098140 legacy=0x7f82741321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080949948 0 0) 0x7f82640014a0 con 0x7f8274098140 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:09.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): my global_id is 104574 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c0034e0 con 0x7f8274098140 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795130321 0 0) 0x7f825c0034e0 con 0x7f8274098140 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 msgr2=0x7f8274098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 --2- 172.21.15.67:0/4039795350 >> v2:172.21.15.67:3300/0 conn(0x7f82740a0370 0x7f8274098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 >> v1:172.21.15.67:6789/0 conn(0x7f8274098a50 legacy=0x7f82741390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740a00b0 con 0x7f8274098140 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:09.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f82640019c0 con 0x7f8274098140 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f827a6b8640 5 monclient: authenticate success, global_id 104574 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f827a6b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: dump: 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.388 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 >> v1:172.21.15.17:6789/0 conn(0x7f8274098140 legacy=0x7f82741321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f825c003a50 0x7f825c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): picked mon.a con 0x7f825c003a50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f825c004780 0x7f825c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): picked mon.c con 0x7f825c004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f825c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): picked mon.b con 0x7f825c0052e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82741328f0 con 0x7f825c003a50 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c00c2b0 con 0x7f825c0052e0 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c00c470 con 0x7f825c004780 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8272ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f827a6b8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f825c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): get_auth_request con 0x7f825c0052e0 auth_method 0 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8273fff640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f825c003a50 0x7f825c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f82737fe640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f825c004780 0x7f825c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8273fff640 10 monclient(hunting): get_auth_request con 0x7f825c003a50 auth_method 0 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8273fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient(hunting): handle_auth_done global_id 104574 payload 293 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f825c004780 msgr2=0x7f825c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f825c004780 0x7f825c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f825c003a50 msgr2=0x7f825c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f825c003a50 0x7f825c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient: _finish_hunting 0 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 monclient: found mon.b 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0034e0 con 0x7f825c0052e0 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 10 monclient: _finish_auth 0 2024-09-15T02:00:09.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f8278c55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f82737fe640 10 monclient: get_auth_request con 0x7f825c004780 auth_method 0 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.380+0000 7f82737fe640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f825c004780 0x7f825c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.381+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 reap_dead start 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.383+0000 7f8278c55640 1 -- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 msgr2=0x7f825c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.383+0000 7f8278c55640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f825c00b9d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f826c00c500 tx=0x7f826c001ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f9ba0ba46715cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c0070f0 con 0x7f825c0052e0 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: dump: 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.393 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.394 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.396 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:09.396 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.396 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:09.396 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c02d070 con 0x7f825c0052e0 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 1 -- 172.21.15.67:0/4039795350 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c007270 con 0x7f825c0052e0 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 4 set_mon_vals no callback set 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f8272ffd640 10 monclient: dump: 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 10 monclient: shutdown 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 -- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 msgr2=0x7f825c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f825c00b9d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f826c00c500 tx=0x7f826c001ce0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 -- 172.21.15.67:0/4039795350 shutdown_connections 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 --2- 172.21.15.67:0/4039795350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f825c00b9d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 -- 172.21.15.67:0/4039795350 >> 172.21.15.67:0/4039795350 conn(0x7f82740a5780 msgr2=0x7f82740a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 -- 172.21.15.67:0/4039795350 shutdown_connections 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.384+0000 7f827a6b8640 1 -- 172.21.15.67:0/4039795350 wait complete. 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.385+0000 7f827a6b8640 10 monclient: build_initial_monmap 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.385+0000 7f827a6b8640 10 monclient: monmap: 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:09.400 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:09.400 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:09.400 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.385+0000 7f827a6b8640 1 Processor -- start 2024-09-15T02:00:09.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.385+0000 7f827a6b8640 1 -- start start 2024-09-15T02:00:09.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.385+0000 7f827a6b8640 10 monclient: init 2024-09-15T02:00:09.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:09.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:09.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-b con 0x7f825c0052e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82741332c0 0x7f827411e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f82741332c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 0x7f827411ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): picked mon.noname-a con 0x7f8274131fa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274120080 con 0x7f8274131fa0 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274121e90 con 0x7f825c0052e0 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8274141d40 con 0x7f82741332c0 2024-09-15T02:00:09.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f827a6b8640 10 monclient(hunting): authenticate will time out at 1241.854156s 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45668/0 (socket says 172.21.15.67:45668) 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 -- 172.21.15.67:0/3660744932 learned_addr learned my addr 172.21.15.67:0/3660744932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 10 monclient(hunting): get_auth_request con 0x7f825c0052e0 auth_method 0 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82741332c0 0x7f827411e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 0x7f827411ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 msgr2=0x7f8274131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 msgr2=0x7f8274131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f82741332c0 auth_method 0 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): get_auth_request con 0x7f8274131fa0 auth_method 0 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8273fff640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient(hunting): handle_auth_done global_id 104577 payload 824 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82741332c0 msgr2=0x7f827411e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82741332c0 0x7f827411e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 msgr2=0x7f8274131a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient: _finish_hunting 0 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 monclient: found mon.noname-a 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.386+0000 7f8278c55640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274121bb0 con 0x7f8274131fa0 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f8278c55640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 msgr2=0x7f827411ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f8278c55640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 0x7f827411ead0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8274096df0 tx=0x7f826c033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c032920 con 0x7f8274131fa0 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: dump: 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:09.409 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: _finish_auth 0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 5 monclient: authenticate success, global_id 104577 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 10 monclient: _renew_subs 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8274122170 con 0x7f8274131fa0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 10 monclient: _renew_subs 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8274120e70 con 0x7f8274131fa0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c032c90 con 0x7f8274131fa0 2024-09-15T02:00:09.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c032e00 con 0x7f8274131fa0 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f825bfff640 10 monclient: dump: 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:09.411 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.389+0000 7f82727fc640 4 set_mon_vals no callback set 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f827a6b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f827a6b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f827a6b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254005010 con 0x7f8274131fa0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f826c02f030 con 0x7f8274131fa0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f825bfff640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 0x7f826003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(235..235 src has 148..235) v4 ==== 11159+0+0 (secure 0 0 0) 0x7f826c06df20 con 0x7f8274131fa0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f8273fff640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 0x7f826003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.390+0000 7f8273fff640 10 monclient: get_auth_request con 0x7f8260038740 auth_method 0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.393+0000 7f8273fff640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 0x7f826003abf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c00 tx=0x7f826400a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:09.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.395+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f826c03d5b0 con 0x7f8274131fa0 2024-09-15T02:00:09.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.395+0000 7f825bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:09.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.395+0000 7f825bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.563+0000 7f827a6b8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.563+0000 7f827a6b8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.563+0000 7f827a6b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:09.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:09.563+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f82540052c0 con 0x7f8274131fa0 2024-09-15T02:00:10.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.184+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f826c045360 con 0x7f8274131fa0 2024-09-15T02:00:10.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.184+0000 7f825bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:10.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.184+0000 7f825bfff640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-15T02:00:10.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.288+0000 7f827a6b8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:10.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.288+0000 7f827a6b8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:10.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.288+0000 7f827a6b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:10.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.288+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f82540052c0 con 0x7f8274131fa0 2024-09-15T02:00:10.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.289+0000 7f825bfff640 1 -- 172.21.15.67:0/3660744932 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f826c045540 con 0x7f8274131fa0 2024-09-15T02:00:10.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.289+0000 7f825bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T02:00:10.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.289+0000 7f825bfff640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-15T02:00:10.291 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h19' does not appear in the crush map 2024-09-15T02:00:10.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 msgr2=0x7f826003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:10.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 0x7f826003abf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c00 tx=0x7f826400a040 comp rx=0 tx=0).stop 2024-09-15T02:00:10.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 10 monclient: shutdown 2024-09-15T02:00:10.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:10.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 msgr2=0x7f827411ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 0x7f827411ead0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8274096df0 tx=0x7f826c033000 comp rx=0 tx=0).stop 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 shutdown_connections 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8260038740 0x7f826003abf0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8274131fa0 0x7f827411ead0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f82741332c0 0x7f827411e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.291+0000 7f827a6b8640 1 --2- 172.21.15.67:0/3660744932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f825c0052e0 0x7f8274131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.292+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 >> 172.21.15.67:0/3660744932 conn(0x7f82740a5780 msgr2=0x7f82740a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.292+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 shutdown_connections 2024-09-15T02:00:10.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.292+0000 7f827a6b8640 1 -- 172.21.15.67:0/3660744932 wait complete. 2024-09-15T02:00:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:10.302 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h20 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 10 monclient: monmap: 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:10.377244+0000 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:10.377244+0000 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:10.379 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 1 Processor -- start 2024-09-15T02:00:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 1 -- start start 2024-09-15T02:00:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.377+0000 7f8c1ad9d640 10 monclient: init 2024-09-15T02:00:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c14098140 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c140a0370 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 0x7f8c1413a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c14098c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c140a0b80 con 0x7f8c140a0370 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c141328f0 con 0x7f8c14098140 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140a1730 con 0x7f8c14098c20 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1ad9d640 10 monclient(hunting): authenticate will time out at 1242.846554s 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c18b39640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c14098140 0x7f8c141321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55244/0 (socket says 172.21.15.67:55244) 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c18b39640 1 -- 172.21.15.67:0/224525218 learned_addr learned my addr 172.21.15.67:0/224525218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:10.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 1 --2- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 0x7f8c1413a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): get_auth_request con 0x7f8c14098c20 auth_method 0 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.378+0000 7f8c1933a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 10 monclient(hunting): handle_auth_done global_id 94730 payload 274 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 1 -- 172.21.15.67:0/224525218 >> v1:172.21.15.67:6789/0 conn(0x7f8c14098140 legacy=0x7f8c141321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 1 -- 172.21.15.67:0/224525218 >> v1:172.21.15.17:6789/0 conn(0x7f8c140a0370 legacy=0x7f8c14098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 1 monclient: found mon.noname-b 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:10.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.379+0000 7f8c1933a640 1 -- 172.21.15.67:0/224525218 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140a00b0 con 0x7f8c14098c20 2024-09-15T02:00:10.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.381+0000 7f8c1933a640 1 -- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 msgr2=0x7f8c1413a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1933a640 1 --2- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 0x7f8c1413a0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009880 tx=0x7f8c0c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dd51db24dcea601 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 1 -- 172.21.15.67:0/224525218 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c032440 con 0x7f8c14098c20 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: dump: 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:10.388 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 1 -- 172.21.15.67:0/224525218 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0325a0 con 0x7f8c14098c20 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 5 monclient: authenticate success, global_id 94730 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 1 -- 172.21.15.67:0/224525218 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c032940 con 0x7f8c14098c20 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:10.389 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c137fe640 10 monclient: dump: 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 4 set_mon_vals no callback set 2024-09-15T02:00:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 10 monclient: shutdown 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 msgr2=0x7f8c1413a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.382+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 0x7f8c1413a0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009880 tx=0x7f8c0c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/224525218 shutdown_connections 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/224525218 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098c20 0x7f8c1413a0d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/224525218 >> 172.21.15.67:0/224525218 conn(0x7f8c140a5780 msgr2=0x7f8c140a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/224525218 shutdown_connections 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/224525218 wait complete. 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 10 monclient: monmap: 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:10.391 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.383+0000 7f8c1ad9d640 1 Processor -- start 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 -- start start 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient: init 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 0x7f8c14139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c14098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 0x7f8c14140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c14098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c140a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141339d0 con 0x7f8c14098c20 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14131710 con 0x7f8c140a0370 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1409f1f0 con 0x7f8c14098140 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1ad9d640 10 monclient(hunting): authenticate will time out at 1242.852615s 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40880/0 (socket says 172.21.15.67:40880) 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 -- 172.21.15.67:0/807757790 learned_addr learned my addr 172.21.15.67:0/807757790 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 10 monclient(hunting): get_auth_request con 0x7f8c140a0370 auth_method 0 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 0x7f8c14139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 msgr2=0x7f8c1412fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 msgr2=0x7f8c1412fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): get_auth_request con 0x7f8c14098140 auth_method 0 2024-09-15T02:00:10.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c1933a640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 0x7f8c14140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): get_auth_request con 0x7f8c14098c20 auth_method 0 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c18b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.384+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 10 monclient(hunting): handle_auth_done global_id 94733 payload 824 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 msgr2=0x7f8c1412fc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 msgr2=0x7f8c14140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 0x7f8c14140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 10 monclient: _finish_hunting 0 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 monclient: found mon.noname-c 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c18b39640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c009390 con 0x7f8c14098140 2024-09-15T02:00:10.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.385+0000 7f8c13fff640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 0x7f8c14140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c18b39640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 msgr2=0x7f8c14139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c18b39640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 0x7f8c14139500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c490 tx=0x7f8c04007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04005cb0 con 0x7f8c14098140 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:10.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: dump: 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: _finish_auth 0 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04005e10 con 0x7f8c14098140 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04007320 con 0x7f8c14098140 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c12ffd640 4 set_mon_vals no callback set 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c10ff9640 10 monclient: dump: 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 5 monclient: authenticate success, global_id 94733 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 10 monclient: _renew_subs 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c141336f0 con 0x7f8c14098140 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 10 monclient: _renew_subs 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.387+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c14139a40 con 0x7f8c14098140 2024-09-15T02:00:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c1ad9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c1ad9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8c0401b3b0 con 0x7f8c14098140 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c1ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1404bf60 con 0x7f8c14098140 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.388+0000 7f8c10ff9640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 0x7f8bf403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.389+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(236..236 src has 148..236) v4 ==== 10946+0+0 (secure 0 0 0) 0x7f8c0404c740 con 0x7f8c14098140 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.389+0000 7f8c13fff640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 0x7f8bf403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.389+0000 7f8c13fff640 10 monclient: get_auth_request con 0x7f8bf4038740 auth_method 0 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.391+0000 7f8c13fff640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 0x7f8bf403abf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc003260 tx=0x7f8bfc002630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.393+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8c0400f860 con 0x7f8c14098140 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.393+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:10.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.393+0000 7f8c10ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:10.563 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.562+0000 7f8c1ad9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.562+0000 7f8c1ad9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.562+0000 7f8c1ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:10.564 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:10.562+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f8c14133c10 con 0x7f8c14098140 2024-09-15T02:00:11.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:11.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.197+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8c04017020 con 0x7f8c14098140 2024-09-15T02:00:11.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.197+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:11.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.197+0000 7f8c10ff9640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-15T02:00:11.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.232+0000 7ff5b595d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:11.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.232+0000 7ff5b595d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.232+0000 7ff5b595d640 10 monclient: monmap: 2024-09-15T02:00:11.234 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:11.233261+0000 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:11.233261+0000 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:11.235 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 Processor -- start 2024-09-15T02:00:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 -- start start 2024-09-15T02:00:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient: init 2024-09-15T02:00:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff5b006ec20 0x7ff5b009c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b006ec20 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b006eff0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5b006f3c0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b0069bb0 con 0x7ff5b006f3c0 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b0099ea0 con 0x7ff5b006ec20 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b009cc80 con 0x7ff5b006eff0 2024-09-15T02:00:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5b595d640 10 monclient(hunting): authenticate will time out at 1271.229813s 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5aeffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff5b006ec20 0x7ff5b009c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5af7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff5b006f3c0 0x7ff5b009ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42942/0 (socket says 172.21.15.17:42942) 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5aeffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff5b006ec20 0x7ff5b009c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46874/0 (socket says 172.21.15.17:46874) 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.233+0000 7ff5af7fe640 1 -- 172.21.15.17:0/389857101 learned_addr learned my addr 172.21.15.17:0/389857101 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): get_auth_request con 0x7ff5b006ec20 auth_method 0 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 10 monclient(hunting): get_auth_request con 0x7ff5b006eff0 auth_method 0 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 1 -- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 msgr2=0x7ff5b009d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 1 -- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 msgr2=0x7ff5b009d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_done global_id 104583 payload 274 2024-09-15T02:00:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 1 -- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 msgr2=0x7ff5b009d870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 1 --2- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 1 -- 172.21.15.17:0/389857101 >> v1:172.21.15.17:6789/0 conn(0x7ff5b006f3c0 legacy=0x7ff5b009ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 1 monclient: found mon.noname-a 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.234+0000 7ff5aeffd640 1 -- 172.21.15.17:0/389857101 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b006a1a0 con 0x7ff5b006ec20 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5aeffd640 1 -- 172.21.15.17:0/389857101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006ec20 msgr2=0x7ff5b009c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5aeffd640 1 --2- 172.21.15.17:0/389857101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006ec20 0x7ff5b009c740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff5b009f510 tx=0x7ff5a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da8ee0564a3faf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 1 -- 172.21.15.17:0/389857101 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a002ae90 con 0x7ff5b006ec20 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: dump: 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.249 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 5 monclient: authenticate success, global_id 104583 2024-09-15T02:00:11.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 1 -- 172.21.15.17:0/389857101 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a0032440 con 0x7ff5b006ec20 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 1 -- 172.21.15.17:0/389857101 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0032820 con 0x7ff5b006ec20 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 4 set_mon_vals no callback set 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5ae7fc640 10 monclient: dump: 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.251 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 10 monclient: shutdown 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 1 -- 172.21.15.17:0/389857101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006ec20 msgr2=0x7ff5b009c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 1 --2- 172.21.15.17:0/389857101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006ec20 0x7ff5b009c740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff5b009f510 tx=0x7ff5a00090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 1 -- 172.21.15.17:0/389857101 shutdown_connections 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 1 --2- 172.21.15.17:0/389857101 >> v2:172.21.15.67:3300/0 conn(0x7ff5b006eff0 0x7ff5b009d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.237+0000 7ff5b595d640 1 --2- 172.21.15.17:0/389857101 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006ec20 0x7ff5b009c740 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 1 -- 172.21.15.17:0/389857101 >> 172.21.15.17:0/389857101 conn(0x7ff5b00a2e80 msgr2=0x7ff5b00a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 1 -- 172.21.15.17:0/389857101 shutdown_connections 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 1 -- 172.21.15.17:0/389857101 wait complete. 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 10 monclient: monmap: 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 1 Processor -- start 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 1 -- start start 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.238+0000 7ff5b595d640 10 monclient: init 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b006ec20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b006eff0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 0x7ff5b011e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b006f3c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0120470 con 0x7ff5b006f3c0 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0122180 con 0x7ff5b006eff0 2024-09-15T02:00:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0122460 con 0x7ff5b006ec20 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5b595d640 10 monclient(hunting): authenticate will time out at 1271.235466s 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48546/0 (socket says 172.21.15.17:48546) 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 -- 172.21.15.17:0/3983739777 learned_addr learned my addr 172.21.15.17:0/3983739777 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): get_auth_request con 0x7ff5b006ec20 auth_method 0 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 0x7ff5b011e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): get_auth_request con 0x7ff5b006f3c0 auth_method 0 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 10 monclient(hunting): get_auth_request con 0x7ff5b006eff0 auth_method 0 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 msgr2=0x7ff5b0134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 msgr2=0x7ff5b0134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient(hunting): handle_auth_done global_id 94736 payload 824 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 msgr2=0x7ff5b0134d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:11.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 msgr2=0x7ff5b011e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 0x7ff5b011e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 monclient: found mon.noname-c 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5aeffd640 1 -- 172.21.15.17:0/3983739777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0009390 con 0x7ff5b006ec20 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.239+0000 7ff5af7fe640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 0x7ff5b011e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5aeffd640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 msgr2=0x7ff5b0140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5aeffd640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff5a002a6c0 tx=0x7ff5a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0031940 con 0x7ff5b006ec20 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: dump: 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.260 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 5 monclient: authenticate success, global_id 94736 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 10 monclient: _renew_subs 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b0120f40 con 0x7ff5b006ec20 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a002ae90 con 0x7ff5b006ec20 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0031cb0 con 0x7ff5b006ec20 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5a6ffd640 10 monclient: dump: 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.262 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5adffb640 4 set_mon_vals no callback set 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 10 monclient: _renew_subs 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.242+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b009c2f0 con 0x7ff5b006ec20 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff5a0039070 con 0x7ff5b006ec20 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5b595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5b595d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5b595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b00724a0 con 0x7ff5b006ec20 2024-09-15T02:00:11.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5a6ffd640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 0x7ff59003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(237..237 src has 148..237) v4 ==== 10733+0+0 (secure 0 0 0) 0x7ff5a006db90 con 0x7ff5b006ec20 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 0x7ff59003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.243+0000 7ff5a7fff640 10 monclient: get_auth_request con 0x7ff590038740 auth_method 0 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.246+0000 7ff5a7fff640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 0x7ff59003abf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00a1230 tx=0x7ff598008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.247+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff5a003c620 con 0x7ff5b006ec20 2024-09-15T02:00:11.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.247+0000 7ff5a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.247+0000 7ff5a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.300+0000 7f8c1ad9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.300+0000 7f8c1ad9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.300+0000 7f8c1ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.300+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f8c14139c60 con 0x7f8c14098140 2024-09-15T02:00:11.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.301+0000 7f8c10ff9640 1 -- 172.21.15.67:0/807757790 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8c04017070 con 0x7f8c14098140 2024-09-15T02:00:11.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.301+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T02:00:11.303 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.301+0000 7f8c10ff9640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-15T02:00:11.303 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h20' does not appear in the crush map 2024-09-15T02:00:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 msgr2=0x7f8bf403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 0x7f8bf403abf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc003260 tx=0x7f8bfc002630 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 10 monclient: shutdown 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 msgr2=0x7f8c14139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 0x7f8c14139500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c490 tx=0x7f8c04007f00 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 shutdown_connections 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bf4038740 0x7f8bf403abf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8c14098c20 0x7f8c14140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c140a0370 0x7f8c1412fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 --2- 172.21.15.67:0/807757790 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8c14098140 0x7f8c14139500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 >> 172.21.15.67:0/807757790 conn(0x7f8c140a5780 msgr2=0x7f8c140a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 shutdown_connections 2024-09-15T02:00:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.304+0000 7f8c1ad9d640 1 -- 172.21.15.67:0/807757790 wait complete. 2024-09-15T02:00:11.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:11.314 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h21 2024-09-15T02:00:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 10 monclient: get_monmap_and_config 2024-09-15T02:00:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 10 monclient: monmap: 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:11.390355+0000 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:11.390355+0000 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:11.392 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 1 Processor -- start 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 1 -- start start 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.390+0000 7f922e646640 10 monclient: init 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T02:00:11.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9228098140 0x7f92281321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): picked mon.noname-a con 0x7f9228098140 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92280a0370 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9228098c20 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280a0b80 con 0x7f9228098c20 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280a00b0 con 0x7f92280a0370 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9228071390 con 0x7f9228098140 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f922e646640 10 monclient(hunting): authenticate will time out at 1243.859483s 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f92277fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92280a0370 0x7f9228098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55250/0 (socket says 172.21.15.67:55250) 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f92277fe640 1 -- 172.21.15.67:0/2257991202 learned_addr learned my addr 172.21.15.67:0/2257991202 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 1 --2- 172.21.15.67:0/2257991202 >> v2:172.21.15.17:3300/0 conn(0x7f9228098140 0x7f92281321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): get_auth_request con 0x7f9228098140 auth_method 0 2024-09-15T02:00:11.395 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.391+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 10 monclient(hunting): handle_auth_done global_id 104589 payload 274 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 1 -- 172.21.15.67:0/2257991202 >> v1:172.21.15.67:6789/0 conn(0x7f92280a0370 legacy=0x7f9228098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 1 -- 172.21.15.67:0/2257991202 >> v1:172.21.15.17:6789/0 conn(0x7f9228098c20 legacy=0x7f92281390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 1 monclient: found mon.noname-a 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.392+0000 7f9227fff640 1 -- 172.21.15.67:0/2257991202 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280714d0 con 0x7f9228098140 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.394+0000 7f9227fff640 1 -- 172.21.15.67:0/2257991202 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 msgr2=0x7f92281321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9227fff640 1 --2- 172.21.15.67:0/2257991202 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f92281321e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9218009850 tx=0x7f92180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82708665494e97f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 1 -- 172.21.15.67:0/2257991202 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218032440 con 0x7f9228098140 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: dump: 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 1 -- 172.21.15.67:0/2257991202 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92180325a0 con 0x7f9228098140 2024-09-15T02:00:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 1 -- 172.21.15.67:0/2257991202 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218032940 con 0x7f9228098140 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f9226ffd640 10 monclient: dump: 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.402 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 5 monclient: authenticate success, global_id 104589 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 4 set_mon_vals no callback set 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 10 monclient: shutdown 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 1 -- 172.21.15.67:0/2257991202 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 msgr2=0x7f92281321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.395+0000 7f922e646640 1 --2- 172.21.15.67:0/2257991202 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f92281321e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9218009850 tx=0x7f92180090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 -- 172.21.15.67:0/2257991202 shutdown_connections 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 --2- 172.21.15.67:0/2257991202 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f92281321e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 -- 172.21.15.67:0/2257991202 >> 172.21.15.67:0/2257991202 conn(0x7f92280a5780 msgr2=0x7f92280a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 -- 172.21.15.67:0/2257991202 shutdown_connections 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 -- 172.21.15.67:0/2257991202 wait complete. 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 10 monclient: monmap: 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.396+0000 7f922e646640 1 Processor -- start 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 -- start start 2024-09-15T02:00:11.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient: init 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f9228139520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): picked mon.noname-a con 0x7f9228098140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): picked mon.noname-b con 0x7f9228098c20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 0x7f92281356e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): picked mon.noname-c con 0x7f92280a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281387b0 con 0x7f9228098140 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228099090 con 0x7f9228098c20 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92280a38e0 con 0x7f92280a0370 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922e646640 10 monclient(hunting): authenticate will time out at 1243.865497s 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40886/0 (socket says 172.21.15.67:40886) 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 -- 172.21.15.67:0/2934924075 learned_addr learned my addr 172.21.15.67:0/2934924075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 10 monclient(hunting): get_auth_request con 0x7f9228098c20 auth_method 0 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f9228139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 0x7f92281356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): get_auth_request con 0x7f9228098140 auth_method 0 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): get_auth_request con 0x7f92280a0370 auth_method 0 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 msgr2=0x7f92281351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 msgr2=0x7f92281351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f92277fe640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 10 monclient(hunting): handle_auth_done global_id 94739 payload 824 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 msgr2=0x7f92281351a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.397+0000 7f922cbe3640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 msgr2=0x7f9228139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f9228139520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 1 monclient: found mon.noname-c 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f922cbe3640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9218009390 con 0x7f92280a0370 2024-09-15T02:00:11.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.398+0000 7f9227fff640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f9228139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922cbe3640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 msgr2=0x7f92281356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922cbe3640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 0x7f92281356e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9228099060 tx=0x7f92200025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922000a820 con 0x7f92280a0370 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: dump: 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.411 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922000a980 con 0x7f92280a0370 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922000ad80 con 0x7f92280a0370 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f920ffff640 10 monclient: dump: 2024-09-15T02:00:11.412 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 5 monclient: authenticate success, global_id 94739 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 10 monclient: _renew_subs 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.413 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92281373d0 con 0x7f92280a0370 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f92267fc640 4 set_mon_vals no callback set 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 10 monclient: _renew_subs 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.400+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9228136c80 con 0x7f92280a0370 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9220012630 con 0x7f92280a0370 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f922e646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f922e646640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f922e646640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9228138af0 con 0x7f92280a0370 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f920ffff640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 0x7f921403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(237..237 src has 148..237) v4 ==== 10733+0+0 (secure 0 0 0) 0x7f922004d3e0 con 0x7f92280a0370 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f9227fff640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 0x7f921403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.401+0000 7f9227fff640 10 monclient: get_auth_request con 0x7f9214038770 auth_method 0 2024-09-15T02:00:11.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.404+0000 7f9227fff640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 0x7f921403ac20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f92280a1b60 tx=0x7f9218032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.406+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9228138af0 con 0x7f92280a0370 2024-09-15T02:00:11.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.406+0000 7f920ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.406+0000 7f920ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.476+0000 7ff5b595d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.476+0000 7ff5b595d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.476+0000 7ff5b595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.476+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff5b00008d0 con 0x7ff5b006ec20 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.477+0000 7ff5a6ffd640 1 -- 172.21.15.17:0/3983739777 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7ff5a004a070 con 0x7ff5b006ec20 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.477+0000 7ff5a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:11.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.477+0000 7ff5a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:11.480 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 msgr2=0x7ff59003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 0x7ff59003abf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00a1230 tx=0x7ff598008040 comp rx=0 tx=0).stop 2024-09-15T02:00:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 10 monclient: shutdown 2024-09-15T02:00:11.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 msgr2=0x7ff5b0140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff5a002a6c0 tx=0x7ff5a0032000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 shutdown_connections 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff590038740 0x7ff59003abf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff5b006f3c0 0x7ff5b011e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b006eff0 0x7ff5b0134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 --2- 172.21.15.17:0/3983739777 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff5b006ec20 0x7ff5b0140f90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 >> 172.21.15.17:0/3983739777 conn(0x7ff5b00a2e80 msgr2=0x7ff5b00a1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 shutdown_connections 2024-09-15T02:00:11.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.479+0000 7ff5b595d640 1 -- 172.21.15.17:0/3983739777 wait complete. 2024-09-15T02:00:11.491 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:00:11.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:00:11.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:00:11.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.574+0000 7f922e646640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:11.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.574+0000 7f922e646640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:11.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.574+0000 7f922e646640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:11.574+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f922804bf60 con 0x7f92280a0370 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.587+0000 7f72be59d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.587+0000 7f72be59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.587+0000 7f72be59d640 10 monclient: monmap: 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:11.587949+0000 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:11.587949+0000 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:11.589 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:11.590 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:11.590 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:11.590 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:11.590 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.587+0000 7f72be59d640 1 Processor -- start 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 -- start start 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient: init 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72b8131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-b con 0x7f72b80a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-c con 0x7f72b8132430 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:11.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b80a0040 con 0x7f72b8131b20 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b809bff0 con 0x7f72b8132430 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80a14a0 con 0x7f72b80a0280 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72be59d640 10 monclient(hunting): authenticate will time out at 1271.584543s 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48548/0 (socket says 172.21.15.17:48548) 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 -- 172.21.15.17:0/2708787589 learned_addr learned my addr 172.21.15.17:0/2708787589 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): get_auth_request con 0x7f72b80a0280 auth_method 0 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 10 monclient(hunting): get_auth_request con 0x7f72b8132430 auth_method 0 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 1 -- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 msgr2=0x7f72b8138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 1 -- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 msgr2=0x7f72b8138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 10 monclient(hunting): handle_auth_done global_id 94742 payload 274 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 -- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 msgr2=0x7f72b8138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 --2- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.588+0000 7f72affff640 1 -- 172.21.15.17:0/2708787589 >> v1:172.21.15.67:6789/0 conn(0x7f72b8131b20 legacy=0x7f72b809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.589+0000 7f72affff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.589+0000 7f72affff640 1 monclient: found mon.noname-b 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.589+0000 7f72affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.589+0000 7f72affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.589+0000 7f72affff640 1 -- 172.21.15.17:0/2708787589 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b809f3b0 con 0x7f72b80a0280 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72affff640 1 -- 172.21.15.17:0/2708787589 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 msgr2=0x7f72b8131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72affff640 1 --2- 172.21.15.17:0/2708787589 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009910 tx=0x7f72a00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4fa9b348de5677 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72b77fe640 1 -- 172.21.15.17:0/2708787589 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a002aea0 con 0x7f72b80a0280 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72b77fe640 10 monclient: dump: 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.601 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.591+0000 7f72b77fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 1 -- 172.21.15.17:0/2708787589 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a0032440 con 0x7f72b80a0280 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 5 monclient: authenticate success, global_id 94742 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 1 -- 172.21.15.17:0/2708787589 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00327e0 con 0x7f72b80a0280 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 4 set_mon_vals no callback set 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72b77fe640 10 monclient: dump: 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.603 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 10 monclient: shutdown 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 -- 172.21.15.17:0/2708787589 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 msgr2=0x7f72b8131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 --2- 172.21.15.17:0/2708787589 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009910 tx=0x7f72a00091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 -- 172.21.15.17:0/2708787589 shutdown_connections 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 --2- 172.21.15.17:0/2708787589 >> v2:172.21.15.67:3300/0 conn(0x7f72b8132430 0x7f72b8138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 --2- 172.21.15.17:0/2708787589 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b80a0280 0x7f72b8131ef0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 -- 172.21.15.17:0/2708787589 >> 172.21.15.17:0/2708787589 conn(0x7f72b8071270 msgr2=0x7f72b80a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 -- 172.21.15.17:0/2708787589 shutdown_connections 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.592+0000 7f72be59d640 1 -- 172.21.15.17:0/2708787589 wait complete. 2024-09-15T02:00:11.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient: monmap: 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 Processor -- start 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 -- start start 2024-09-15T02:00:11.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient: init 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b80a0280 0x7f72b8137630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-a con 0x7f72b80a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8131b20 0x7f72b81333b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-b con 0x7f72b8131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): picked mon.noname-c con 0x7f72b8132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8137b70 con 0x7f72b80a0280 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80a09b0 con 0x7f72b8131b20 2024-09-15T02:00:11.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8138f60 con 0x7f72b8132430 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72be59d640 10 monclient(hunting): authenticate will time out at 1271.589858s 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72bcb3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.593+0000 7f72b7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b80a0280 0x7f72b8137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48554/0 (socket says 172.21.15.17:48554) 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 learned_addr learned my addr 172.21.15.17:0/1310335279 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): get_auth_request con 0x7f72b8132430 auth_method 0 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): get_auth_request con 0x7f72b80a0280 auth_method 0 2024-09-15T02:00:11.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72affff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8131b20 0x7f72b81333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72affff640 10 monclient(hunting): get_auth_request con 0x7f72b8131b20 auth_method 0 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient(hunting): handle_auth_done global_id 94745 payload 824 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8131b20 msgr2=0x7f72b81333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8131b20 0x7f72b81333b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b80a0280 msgr2=0x7f72b8137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b80a0280 0x7f72b8137630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 monclient: found mon.noname-c 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0009450 con 0x7f72b8132430 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.594+0000 7f72b7fff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b80a0280 0x7f72b8137630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 msgr2=0x7f72b81338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72bcb3a640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009fd0 tx=0x7f72b0009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b000ee30 con 0x7f72b8132430 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: dump: 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.610 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b0013590 con 0x7f72b8132430 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 5 monclient: authenticate success, global_id 94745 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 10 monclient: _renew_subs 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0013950 con 0x7f72b8132430 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b8135a80 con 0x7f72b8132430 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b6ffd640 4 set_mon_vals no callback set 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72b4ff9640 10 monclient: dump: 2024-09-15T02:00:11.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 10 monclient: _renew_subs 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.596+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b8134eb0 con 0x7f72b8132430 2024-09-15T02:00:11.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72be59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72be59d640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 0x7f72b8133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8136b10 con 0x7f72b802aa40 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f72b001e460 con 0x7f72b8132430 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 0x7f72b8133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient: get_auth_request con 0x7f72b802aa40 auth_method 0 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 10 monclient(hunting): handle_auth_done global_id 104598 payload 824 2024-09-15T02:00:11.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b7fff640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72b8135450 con 0x7f72b802aa40 2024-09-15T02:00:11.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b4ff9640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7298038770 0x7f729803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.597+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(237..237 src has 148..237) v4 ==== 10733+0+0 (secure 0 0 0) 0x7f72b004d930 con 0x7f72b8132430 2024-09-15T02:00:11.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.598+0000 7f72affff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7298038770 0x7f729803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.598+0000 7f72affff640 10 monclient: get_auth_request con 0x7f7298038770 auth_method 0 2024-09-15T02:00:11.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b7fff640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 msgr2=0x7f72b8133e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b7fff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 0x7f72b8133e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f72a800fa50 tx=0x7f72a8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800dda0 con 0x7f72b802aa40 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 10 monclient: dump: 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f72a8014470 con 0x7f72b802aa40 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 msgr2=0x7f72b8133e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 0x7f72b8133e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f72a800fa50 tx=0x7f72a8007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 msgr2=0x7f72b8133e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72b4ff9640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b802aa40 0x7f72b8133e30 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f72a800fa50 tx=0x7f72a8007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.600+0000 7f72affff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7298038770 0x7f729803ac20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f72b809f710 tx=0x7f72a0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72be59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72be59d640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 0x7f72b805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72be59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8063a40 con 0x7f72b805e310 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 0x7f72b805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient: get_auth_request con 0x7f72b805e310 auth_method 0 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 10 monclient(hunting): handle_auth_done global_id 104601 payload 824 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.606+0000 7f72b7fff640 1 -- 172.21.15.17:0/1310335279 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f72b802ae10 con 0x7f72b805e310 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b7fff640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 msgr2=0x7f72b805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b7fff640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 0x7f72b805e710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f72a8019000 tx=0x7f72a801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a801ed30 con 0x7f72b805e310 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.608+0000 7f72b4ff9640 10 monclient: dump: 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7f72a802b440 con 0x7f72b805e310 2024-09-15T02:00:11.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 msgr2=0x7f72b805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 0x7f72b805e710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f72a8019000 tx=0x7f72a801a000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 msgr2=0x7f72b805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72b4ff9640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72b805e310 0x7f72b805e710 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f72a8019000 tx=0x7f72a801a000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:00:11.621 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.622 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.623 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:11.624 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.625 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.626 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.627 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:11.629 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.631 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:11.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:11.633 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7298038770 msgr2=0x7f729803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7298038770 0x7f729803ac20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f72b809f710 tx=0x7f72a0032000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 10 monclient: shutdown 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 msgr2=0x7f72b81338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009fd0 tx=0x7f72b0009870 comp rx=0 tx=0).stop 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72bcb3a640 1 -- 172.21.15.17:0/1310335279 reap_dead start 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 shutdown_connections 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 --2- 172.21.15.17:0/1310335279 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72b8132430 0x7f72b81338f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 >> 172.21.15.17:0/1310335279 conn(0x7f72b8071270 msgr2=0x7f72b8096a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 shutdown_connections 2024-09-15T02:00:11.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.609+0000 7f72be59d640 1 -- 172.21.15.17:0/1310335279 wait complete. 2024-09-15T02:00:11.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:11.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:00:11.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.731+0000 7fc74d99d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.731+0000 7fc74d99d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.731+0000 7fc74d99d640 10 monclient: monmap: 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:11.732575+0000 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:11.732575+0000 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 Processor -- start 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 -- start start 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient: init 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc748130e50 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc748132350 0x7fc748136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-a con 0x7fc748132350 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc748137230 0x7fc748139610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-c con 0x7fc748137230 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc748131f60 con 0x7fc748130e50 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74812f8c0 con 0x7fc748132350 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74809e130 con 0x7fc748137230 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc74d99d640 10 monclient(hunting): authenticate will time out at 1271.728865s 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc7467fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc748132350 0x7fc748136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.732+0000 7fc746ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc748130e50 0x7fc748131220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42944/0 (socket says 172.21.15.17:42944) 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc746ffd640 1 -- 172.21.15.17:0/213116781 learned_addr learned my addr 172.21.15.17:0/213116781 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): get_auth_request con 0x7fc748132350 auth_method 0 2024-09-15T02:00:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient(hunting): handle_auth_done global_id 104604 payload 274 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7477fe640 1 --2- 172.21.15.17:0/213116781 >> v2:172.21.15.67:3300/0 conn(0x7fc748137230 0x7fc748139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 1 -- 172.21.15.17:0/213116781 >> v2:172.21.15.67:3300/0 conn(0x7fc748137230 msgr2=0x7fc748139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 1 --2- 172.21.15.17:0/213116781 >> v2:172.21.15.67:3300/0 conn(0x7fc748137230 0x7fc748139610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 1 -- 172.21.15.17:0/213116781 >> v1:172.21.15.17:6789/0 conn(0x7fc748130e50 legacy=0x7fc748131220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.733+0000 7fc7467fc640 1 -- 172.21.15.17:0/213116781 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74809e410 con 0x7fc748132350 2024-09-15T02:00:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc7467fc640 1 -- 172.21.15.17:0/213116781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 msgr2=0x7fc748136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc7467fc640 1 --2- 172.21.15.17:0/213116781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc748136cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc74806f8e0 tx=0x7fc73c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a641edf34bd4d8f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 1 -- 172.21.15.17:0/213116781 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c032440 con 0x7fc748132350 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: dump: 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 5 monclient: authenticate success, global_id 104604 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 1 -- 172.21.15.17:0/213116781 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c0325a0 con 0x7fc748132350 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 4 set_mon_vals no callback set 2024-09-15T02:00:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 1 -- 172.21.15.17:0/213116781 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c032940 con 0x7fc748132350 2024-09-15T02:00:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc745ffb640 10 monclient: dump: 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 10 monclient: shutdown 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 -- 172.21.15.17:0/213116781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 msgr2=0x7fc748136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 --2- 172.21.15.17:0/213116781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc748136cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc74806f8e0 tx=0x7fc73c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 -- 172.21.15.17:0/213116781 shutdown_connections 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 --2- 172.21.15.17:0/213116781 >> v2:172.21.15.67:3300/0 conn(0x7fc748137230 0x7fc748139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 --2- 172.21.15.17:0/213116781 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc748136cf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 -- 172.21.15.17:0/213116781 >> 172.21.15.17:0/213116781 conn(0x7fc7480a1ee0 msgr2=0x7fc7480a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 -- 172.21.15.17:0/213116781 shutdown_connections 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.736+0000 7fc74d99d640 1 -- 172.21.15.17:0/213116781 wait complete. 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.737+0000 7fc74d99d640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.737+0000 7fc74d99d640 10 monclient: monmap: 2024-09-15T02:00:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.737+0000 7fc74d99d640 1 Processor -- start 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.737+0000 7fc74d99d640 1 -- start start 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.737+0000 7fc74d99d640 10 monclient: init 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 0x7fc7481354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-c con 0x7fc748130e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc7481359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-a con 0x7fc748132350 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc748137230 0x7fc748135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): picked mon.noname-b con 0x7fc748137230 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74809d480 con 0x7fc748132350 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748096fd0 con 0x7fc748137230 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74811a720 con 0x7fc748130e50 2024-09-15T02:00:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc74d99d640 10 monclient(hunting): authenticate will time out at 1271.734352s 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc7481359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 0x7fc7481354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 0x7fc7481354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48562/0 (socket says 172.21.15.17:48562) 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc7481359e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46918/0 (socket says 172.21.15.17:46918) 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 1 -- 172.21.15.17:0/466150022 learned_addr learned my addr 172.21.15.17:0/466150022 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): get_auth_request con 0x7fc748130e50 auth_method 0 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): get_auth_request con 0x7fc748132350 auth_method 0 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7477fe640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc748137230 0x7fc748135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7477fe640 10 monclient(hunting): get_auth_request con 0x7fc748137230 auth_method 0 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient(hunting): handle_auth_done global_id 104607 payload 824 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 msgr2=0x7fc7481354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 0x7fc7481354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc748137230 msgr2=0x7fc748135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc748137230 0x7fc748135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc7467fc640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c009390 con 0x7fc748132350 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.738+0000 7fc746ffd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748130e50 0x7fc7481354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc7467fc640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 msgr2=0x7fc7481359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc7467fc640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc7481359e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc73c031000 tx=0x7fc73c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c031c70 con 0x7fc748132350 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: dump: 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.759 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c031dd0 con 0x7fc748132350 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 5 monclient: authenticate success, global_id 104607 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 10 monclient: _renew_subs 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74811a440 con 0x7fc748132350 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c03d690 con 0x7fc748132350 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc72effd640 10 monclient: dump: 2024-09-15T02:00:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 10 monclient: _renew_subs 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc7457fa640 4 set_mon_vals no callback set 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.741+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748119550 con 0x7fc748132350 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc74d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc74d99d640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 0x7fc74809fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc748063870 con 0x7fc748136460 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc73c039040 con 0x7fc748132350 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 0x7fc74809fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient: get_auth_request con 0x7fc748136460 auth_method 0 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc72effd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc734038740 0x7fc73403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 10 monclient(hunting): handle_auth_done global_id 94751 payload 824 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc746ffd640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc74811a8b0 con 0x7fc748136460 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(237..237 src has 148..237) v4 ==== 10733+0+0 (secure 0 0 0) 0x7fc73c06d090 con 0x7fc748132350 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc7477fe640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc734038740 0x7fc73403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.742+0000 7fc7477fe640 10 monclient: get_auth_request con 0x7fc734038740 auth_method 0 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc746ffd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 msgr2=0x7fc74809fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc746ffd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 0x7fc74809fce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc738007320 tx=0x7fc738004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73800fd50 con 0x7fc748136460 2024-09-15T02:00:11.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc72effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.744+0000 7fc72effd640 10 monclient: dump: 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc7477fe640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc734038740 0x7fc73403abf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc74000e4c0 tx=0x7fc7400076d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc73800feb0 con 0x7fc748136460 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 msgr2=0x7fc74809fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 0x7fc74809fce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc738007320 tx=0x7fc738004960 comp rx=0 tx=0).stop 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 msgr2=0x7fc74809fce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.745+0000 7fc72effd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc748136460 0x7fc74809fce0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.750+0000 7fc74d99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.750+0000 7fc74d99d640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 0x7fc7480a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.750+0000 7fc74d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.750+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480724a0 con 0x7fc7480a0220 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 0x7fc7480a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient: get_auth_request con 0x7fc7480a0220 auth_method 0 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 10 monclient(hunting): handle_auth_done global_id 94754 payload 824 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.751+0000 7fc746ffd640 1 -- 172.21.15.17:0/466150022 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc7480689a0 con 0x7fc7480a0220 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc746ffd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 msgr2=0x7fc7480a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc746ffd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 0x7fc7480a0620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc738017000 tx=0x7fc738018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73801caa0 con 0x7fc7480a0220 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 10 monclient: dump: 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fc73800fb40 con 0x7fc7480a0220 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 msgr2=0x7fc7480a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.753+0000 7fc72effd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 0x7fc7480a0620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc738017000 tx=0x7fc738018000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc72effd640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 msgr2=0x7fc7480a0620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:11.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc72effd640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc7480a0220 0x7fc7480a0620 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc738017000 tx=0x7fc738018000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.771 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:00:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:11.792 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc734038740 msgr2=0x7fc73403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc734038740 0x7fc73403abf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc74000e4c0 tx=0x7fc7400076d0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 10 monclient: shutdown 2024-09-15T02:00:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 msgr2=0x7fc7481359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 --2- 172.21.15.17:0/466150022 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc748132350 0x7fc7481359e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc73c031000 tx=0x7fc73c032000 comp rx=0 tx=0).stop 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc7477fe640 1 -- 172.21.15.17:0/466150022 reap_dead start 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc7477fe640 1 -- 172.21.15.17:0/466150022 reap_dead start 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 shutdown_connections 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 >> 172.21.15.17:0/466150022 conn(0x7fc7480a1ee0 msgr2=0x7fc7480a1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 shutdown_connections 2024-09-15T02:00:11.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.754+0000 7fc74d99d640 1 -- 172.21.15.17:0/466150022 wait complete. 2024-09-15T02:00:11.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:11.796 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:00:11.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.892+0000 7f7f7a280640 10 monclient: get_monmap_and_config 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.892+0000 7f7f7a280640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.892+0000 7f7f7a280640 10 monclient: monmap: 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.895 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:11.893292+0000 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:11.893292+0000 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:11.896 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 Processor -- start 2024-09-15T02:00:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 -- start start 2024-09-15T02:00:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient: init 2024-09-15T02:00:11.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T02:00:11.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f74131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:11.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-a con 0x7f7f740a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-b con 0x7f7f74132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f740a0040 con 0x7f7f74131b20 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7409bff0 con 0x7f7f740a0280 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f740a14a0 con 0x7f7f74132430 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7a280640 10 monclient(hunting): authenticate will time out at 1271.889762s 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46926/0 (socket says 172.21.15.17:46926) 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48604/0 (socket says 172.21.15.17:48604) 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 1 -- 172.21.15.17:0/3708133434 learned_addr learned my addr 172.21.15.17:0/3708133434 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 10 monclient(hunting): get_auth_request con 0x7f7f740a0280 auth_method 0 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 10 monclient(hunting): get_auth_request con 0x7f7f74132430 auth_method 0 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.893+0000 7f7f7881d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient(hunting): handle_auth_done global_id 94757 payload 274 2024-09-15T02:00:11.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 1 -- 172.21.15.17:0/3708133434 >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 msgr2=0x7f7f74131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 1 --2- 172.21.15.17:0/3708133434 >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 1 -- 172.21.15.17:0/3708133434 >> v1:172.21.15.67:6789/0 conn(0x7f7f74131b20 legacy=0x7f7f7409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 1 monclient: found mon.noname-b 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f7881d640 1 -- 172.21.15.17:0/3708133434 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7409f3b0 con 0x7f7f74132430 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.894+0000 7f7f737fe640 1 --2- 172.21.15.17:0/3708133434 >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7881d640 1 -- 172.21.15.17:0/3708133434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 msgr2=0x7f7f74138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7881d640 1 --2- 172.21.15.17:0/3708133434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009e60 tx=0x7f7f6c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a9153850bc096f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 1 -- 172.21.15.17:0/3708133434 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c007760 con 0x7f7f74132430 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: dump: 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.908 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 1 -- 172.21.15.17:0/3708133434 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6c02d050 con 0x7f7f74132430 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 1 -- 172.21.15.17:0/3708133434 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c007af0 con 0x7f7f74132430 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 5 monclient: authenticate success, global_id 94757 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 4 set_mon_vals no callback set 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f72ffd640 10 monclient: dump: 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 10 monclient: shutdown 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:11.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 -- 172.21.15.17:0/3708133434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 msgr2=0x7f7f74138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 --2- 172.21.15.17:0/3708133434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009e60 tx=0x7f7f6c009780 comp rx=0 tx=0).stop 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 -- 172.21.15.17:0/3708133434 shutdown_connections 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 --2- 172.21.15.17:0/3708133434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f74132430 0x7f7f74138e60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 --2- 172.21.15.17:0/3708133434 >> v2:172.21.15.17:3300/0 conn(0x7f7f740a0280 0x7f7f74131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 -- 172.21.15.17:0/3708133434 >> 172.21.15.17:0/3708133434 conn(0x7f7f74071270 msgr2=0x7f7f740a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 -- 172.21.15.17:0/3708133434 shutdown_connections 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.897+0000 7f7f7a280640 1 -- 172.21.15.17:0/3708133434 wait complete. 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.898+0000 7f7f7a280640 10 monclient: build_initial_monmap 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.898+0000 7f7f7a280640 10 monclient: monmap: 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:11.911 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.898+0000 7f7f7a280640 1 Processor -- start 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.898+0000 7f7f7a280640 1 -- start start 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.898+0000 7f7f7a280640 10 monclient: init 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 0x7f7f74137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-c con 0x7f7f740a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 0x7f7f74132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-b con 0x7f7f74131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7f74132430 0x7f7f74132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): picked mon.noname-a con 0x7f7f74132430 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7409ee40 con 0x7f7f74132430 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74134360 con 0x7f7f74131b20 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74134850 con 0x7f7f740a0280 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7a280640 10 monclient(hunting): authenticate will time out at 1271.895472s 2024-09-15T02:00:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 0x7f7f74137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 0x7f7f74137f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48620/0 (socket says 172.21.15.17:48620) 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 -- 172.21.15.17:0/1843635466 learned_addr learned my addr 172.21.15.17:0/1843635466 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): get_auth_request con 0x7f7f740a0280 auth_method 0 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7f74132430 0x7f7f74132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): get_auth_request con 0x7f7f74132430 auth_method 0 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 0x7f7f74132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 10 monclient(hunting): get_auth_request con 0x7f7f74131b20 auth_method 0 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 msgr2=0x7f7f74132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:11.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 msgr2=0x7f7f74132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:11.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 0x7f7f74132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 0x7f7f74132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient(hunting): handle_auth_done global_id 94760 payload 824 2024-09-15T02:00:11.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 msgr2=0x7f7f74132970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:11.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74131b20 0x7f7f74132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7f74132430 msgr2=0x7f7f74132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7f74132430 0x7f7f74132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 monclient: found mon.noname-c 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f73fff640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f68009450 con 0x7f7f740a0280 2024-09-15T02:00:11.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.899+0000 7f7f7881d640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7f74132430 0x7f7f74132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f73fff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 msgr2=0x7f7f74137f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f73fff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 0x7f7f74137f10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c5e0 tx=0x7f7f64007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64012840 con 0x7f7f740a0280 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: dump: 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.920 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: _finish_auth 0 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f640129a0 con 0x7f7f740a0280 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 5 monclient: authenticate success, global_id 94760 2024-09-15T02:00:11.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 10 monclient: _renew_subs 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c009a20 con 0x7f7f740a0280 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64012da0 con 0x7f7f740a0280 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f5bfff640 10 monclient: dump: 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:11.922 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 10 monclient: _renew_subs 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f71ffb640 4 set_mon_vals no callback set 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.902+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f74134c90 con 0x7f7f740a0280 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7f64011590 con 0x7f7f740a0280 2024-09-15T02:00:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7a280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7a280640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74063a40 con 0x7f7f741333f0 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f5bfff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7f60038770 0x7f7f6003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(237..237 src has 148..237) v4 ==== 10733+0+0 (secure 0 0 0) 0x7f7f6404d340 con 0x7f7f740a0280 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7881d640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7f60038770 0x7f7f6003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f741333f0 auth_method 0 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f7881d640 10 monclient: get_auth_request con 0x7f7f60038770 auth_method 0 2024-09-15T02:00:11.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:11.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:11.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:11.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.903+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:11.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:11.906+0000 7f7f7881d640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7f60038770 0x7f7f6003ac20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7f7409f710 tx=0x7f7f6c037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f741333f0 auth_method 0 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.104+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:00:12.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f741333f0 auth_method 0 2024-09-15T02:00:12.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:12.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:12.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:12.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:12.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:12.505+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.514+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9220016020 con 0x7f92280a0370 2024-09-15T02:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.514+0000 7f920ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.514+0000 7f920ffff640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-15T02:00:12.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.621+0000 7f922e646640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:12.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.621+0000 7f922e646640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:12.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.621+0000 7f922e646640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.621+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f922809f7f0 con 0x7f92280a0370 2024-09-15T02:00:12.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.623+0000 7f920ffff640 1 -- 172.21.15.67:0/2934924075 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9220016070 con 0x7f92280a0370 2024-09-15T02:00:12.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.623+0000 7f920ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T02:00:12.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.623+0000 7f920ffff640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-15T02:00:12.624 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h21' does not appear in the crush map 2024-09-15T02:00:12.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 msgr2=0x7f921403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:12.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 0x7f921403ac20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f92280a1b60 tx=0x7f9218032000 comp rx=0 tx=0).stop 2024-09-15T02:00:12.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 10 monclient: shutdown 2024-09-15T02:00:12.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:12.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 msgr2=0x7f92281356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.624+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 0x7f92281356e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9228099060 tx=0x7f92200025b0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 shutdown_connections 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9214038770 0x7f921403ac20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f92280a0370 0x7f92281356e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9228098c20 0x7f92281351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 --2- 172.21.15.67:0/2934924075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9228098140 0x7f9228139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 >> 172.21.15.67:0/2934924075 conn(0x7f92280a5780 msgr2=0x7f922809b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 shutdown_connections 2024-09-15T02:00:12.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.625+0000 7f922e646640 1 -- 172.21.15.67:0/2934924075 wait complete. 2024-09-15T02:00:12.635 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:12.635 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h22 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.710+0000 7f4a0b878640 10 monclient: get_monmap_and_config 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.710+0000 7f4a0b878640 10 monclient: build_initial_monmap 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.710+0000 7f4a0b878640 10 monclient: monmap: 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:12.710988+0000 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:12.710988+0000 2024-09-15T02:00:12.712 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:12.713 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.711+0000 7f4a0b878640 1 Processor -- start 2024-09-15T02:00:12.715 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.711+0000 7f4a0b878640 1 -- start start 2024-09-15T02:00:12.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.711+0000 7f4a0b878640 10 monclient: init 2024-09-15T02:00:12.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:12.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T02:00:12.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a0406e210 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:12.716 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 0x7f4a04136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0406e5e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a04137240 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0406aa60 con 0x7f4a0406e210 2024-09-15T02:00:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a04069bd0 con 0x7f4a04137240 2024-09-15T02:00:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0406a190 con 0x7f4a0406e5e0 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a0b878640 10 monclient(hunting): authenticate will time out at 1245.180115s 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a09e15640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4a04137240 0x7f4a04139620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55262/0 (socket says 172.21.15.67:55262) 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a09e15640 1 -- 172.21.15.67:0/2055525061 learned_addr learned my addr 172.21.15.67:0/2055525061 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 1 --2- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 0x7f4a04136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): get_auth_request con 0x7f4a0406e5e0 auth_method 0 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.718 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient(hunting): handle_auth_done global_id 94763 payload 274 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 1 -- 172.21.15.67:0/2055525061 >> v1:172.21.15.67:6789/0 conn(0x7f4a04137240 legacy=0x7f4a04139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 1 -- 172.21.15.67:0/2055525061 >> v1:172.21.15.17:6789/0 conn(0x7f4a0406e210 legacy=0x7f4a0406b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient: _finish_hunting 0 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 1 monclient: found mon.noname-b 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.719 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.712+0000 7f4a08e13640 1 -- 172.21.15.67:0/2055525061 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040698f0 con 0x7f4a0406e5e0 2024-09-15T02:00:12.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.715+0000 7f4a08e13640 1 -- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 msgr2=0x7f4a04136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:12.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.715+0000 7f4a08e13640 1 --2- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 0x7f4a04136d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f49f80098b0 tx=0x7f49f80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e47128d5f0a8895 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 1 -- 172.21.15.67:0/2055525061 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8032440 con 0x7f4a0406e5e0 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: dump: 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:12.728 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 1 -- 172.21.15.67:0/2055525061 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f80325a0 con 0x7f4a0406e5e0 2024-09-15T02:00:12.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 5 monclient: authenticate success, global_id 94763 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 1 -- 172.21.15.67:0/2055525061 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8032940 con 0x7f4a0406e5e0 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 4 set_mon_vals no callback set 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a03fff640 10 monclient: dump: 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:12.730 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 10 monclient: shutdown 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 1 -- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 msgr2=0x7f4a04136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 1 --2- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 0x7f4a04136d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f49f80098b0 tx=0x7f49f80090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 1 -- 172.21.15.67:0/2055525061 shutdown_connections 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 1 --2- 172.21.15.67:0/2055525061 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e5e0 0x7f4a04136d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.716+0000 7f4a0b878640 1 -- 172.21.15.67:0/2055525061 >> 172.21.15.67:0/2055525061 conn(0x7f4a04070ff0 msgr2=0x7f4a040a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 1 -- 172.21.15.67:0/2055525061 shutdown_connections 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 1 -- 172.21.15.67:0/2055525061 wait complete. 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 10 monclient: build_initial_monmap 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 10 monclient: monmap: 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:12.732 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 1 Processor -- start 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 1 -- start start 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.717+0000 7f4a0b878640 10 monclient: init 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:12.733 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 0x7f4a04141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0406e210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4a0406e5e0 0x7f4a04141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0406e5e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): picked mon.noname-b con 0x7f4a04137240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a04141cd0 con 0x7f4a0406e5e0 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0409ca90 con 0x7f4a04137240 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:12.734 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a04099ee0 con 0x7f4a0406e210 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a0b878640 10 monclient(hunting): authenticate will time out at 1245.186520s 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40900/0 (socket says 172.21.15.67:40900) 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 -- 172.21.15.67:0/3094131180 learned_addr learned my addr 172.21.15.67:0/3094131180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 10 monclient(hunting): get_auth_request con 0x7f4a04137240 auth_method 0 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 0x7f4a04141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.735 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 msgr2=0x7f4a0409a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 msgr2=0x7f4a0409a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): get_auth_request con 0x7f4a0406e210 auth_method 0 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4a0406e5e0 0x7f4a04141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09e15640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): get_auth_request con 0x7f4a0406e5e0 auth_method 0 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:12.736 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a09614640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.718+0000 7f4a08e13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 10 monclient(hunting): handle_auth_done global_id 94766 payload 824 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 msgr2=0x7f4a0409a5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4a0406e5e0 msgr2=0x7f4a04141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:12.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4a0406e5e0 0x7f4a04141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 10 monclient: _finish_hunting 0 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 monclient: found mon.noname-c 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.719+0000 7f4a09614640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f8009390 con 0x7f4a0406e210 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a09614640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 msgr2=0x7f4a04141790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a09614640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 0x7f4a04141790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49f400c890 tx=0x7f49f4002630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f400ce20 con 0x7f4a0406e210 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: dump: 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:12.738 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:12.739 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 5 monclient: authenticate success, global_id 94766 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 10 monclient: _renew_subs 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0409c5a0 con 0x7f4a0406e210 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 10 monclient: _renew_subs 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0411e5a0 con 0x7f4a0406e210 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f400aa60 con 0x7f4a0406e210 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f400ae20 con 0x7f4a0406e210 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:12.740 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a017fa640 10 monclient: dump: 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.721+0000 7f4a037fe640 4 set_mon_vals no callback set 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.722+0000 7f4a0b878640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:12.741 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.722+0000 7f4a0b878640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.722+0000 7f4a0b878640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.722+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f49f4012630 con 0x7f4a0406e210 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.722+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0409cd20 con 0x7f4a0406e210 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.723+0000 7f4a017fa640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 0x7f49f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.723+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(238..238 src has 148..238) v4 ==== 10520+0+0 (secure 0 0 0) 0x7f49f404d4b0 con 0x7f4a0406e210 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.723+0000 7f4a08e13640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 0x7f49f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.723+0000 7f4a08e13640 10 monclient: get_auth_request con 0x7f49f0038740 auth_method 0 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.725+0000 7f4a08e13640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 0x7f49f003abf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4a0409ebe0 tx=0x7f49f8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:12.742 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.727+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f49f4022d70 con 0x7f4a0406e210 2024-09-15T02:00:12.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.727+0000 7f4a017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:12.743 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.727+0000 7f4a017fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.894+0000 7f4a0b878640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.894+0000 7f4a0b878640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:12.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.894+0000 7f4a0b878640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:12.896 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:12.894+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f4a0404bf60 con 0x7f4a0406e210 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f741333f0 auth_method 0 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:00:13.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:13.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:13.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:13.306+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:00:13.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.809+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f49f4016020 con 0x7f4a0406e210 2024-09-15T02:00:13.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.809+0000 7f4a017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:13.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.809+0000 7f4a017fa640 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-15T02:00:13.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.913+0000 7f4a0b878640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:13.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.913+0000 7f4a0b878640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:13.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.913+0000 7f4a0b878640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:13.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.913+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f4a04069eb0 con 0x7f4a0406e210 2024-09-15T02:00:13.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.915+0000 7f4a017fa640 1 -- 172.21.15.67:0/3094131180 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f49f4022770 con 0x7f4a0406e210 2024-09-15T02:00:13.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.915+0000 7f4a017fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T02:00:13.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.915+0000 7f4a017fa640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-15T02:00:13.917 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h22' does not appear in the crush map 2024-09-15T02:00:13.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 msgr2=0x7f49f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:13.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 0x7f49f003abf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4a0409ebe0 tx=0x7f49f8032000 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 10 monclient: shutdown 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 msgr2=0x7f4a04141790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 0x7f4a04141790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49f400c890 tx=0x7f49f4002630 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 shutdown_connections 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f49f0038740 0x7f49f003abf0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4a0406e5e0 0x7f4a04141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a04137240 0x7f4a0409a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 --2- 172.21.15.67:0/3094131180 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4a0406e210 0x7f4a04141790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 >> 172.21.15.67:0/3094131180 conn(0x7f4a04070ff0 msgr2=0x7f4a040a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 shutdown_connections 2024-09-15T02:00:13.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:13.917+0000 7f4a0b878640 1 -- 172.21.15.67:0/3094131180 wait complete. 2024-09-15T02:00:13.927 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:13.927 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h23 2024-09-15T02:00:14.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.003+0000 7ff6bf509640 10 monclient: get_monmap_and_config 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.003+0000 7ff6bf509640 10 monclient: build_initial_monmap 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.003+0000 7ff6bf509640 10 monclient: monmap: 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:14.003492+0000 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:14.003492+0000 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:14.006 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:14.006 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:14.006 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:14.006 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:14.006 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.003+0000 7ff6bf509640 1 Processor -- start 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 1 -- start start 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient: init 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6b80a3830 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:14.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b8136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-b con 0x7ff6b806b9b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6b8137240 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b8069c60 con 0x7ff6b80a3830 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b80a3c00 con 0x7ff6b8137240 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b8069650 con 0x7ff6b806b9b0 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bf509640 10 monclient(hunting): authenticate will time out at 1246.472754s 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bdaa6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6b8137240 0x7ff6b8139620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55264/0 (socket says 172.21.15.67:55264) 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.004+0000 7ff6bdaa6640 1 -- 172.21.15.67:0/3062203152 learned_addr learned my addr 172.21.15.67:0/3062203152 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b8136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): get_auth_request con 0x7ff6b806b9b0 auth_method 0 2024-09-15T02:00:14.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_done global_id 94769 payload 274 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/3062203152 >> v1:172.21.15.67:6789/0 conn(0x7ff6b8137240 legacy=0x7ff6b8139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/3062203152 >> v1:172.21.15.17:6789/0 conn(0x7ff6b80a3830 legacy=0x7ff6b806e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient: _finish_hunting 0 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 1 monclient: found mon.noname-b 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:14.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.005+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/3062203152 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b809f500 con 0x7ff6b806b9b0 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 msgr2=0x7ff6b8136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b8136d00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009880 tx=0x7ff6a80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9dd2af1a6ebafeb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 1 -- 172.21.15.67:0/3062203152 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8032440 con 0x7ff6b806b9b0 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: dump: 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: _finish_auth 0 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 1 -- 172.21.15.67:0/3062203152 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a80325a0 con 0x7ff6b806b9b0 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 1 -- 172.21.15.67:0/3062203152 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8032940 con 0x7ff6b806b9b0 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6bf509640 5 monclient: authenticate success, global_id 94769 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6bf509640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6bf509640 4 set_mon_vals no callback set 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.008+0000 7ff6affff640 10 monclient: dump: 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.020 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 10 monclient: shutdown 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 -- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 msgr2=0x7ff6b8136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 --2- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b8136d00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009880 tx=0x7ff6a80090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 -- 172.21.15.67:0/3062203152 shutdown_connections 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 --2- 172.21.15.67:0/3062203152 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b8136d00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 -- 172.21.15.67:0/3062203152 >> 172.21.15.67:0/3062203152 conn(0x7ff6b8070ff0 msgr2=0x7ff6b80a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 -- 172.21.15.67:0/3062203152 shutdown_connections 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 1 -- 172.21.15.67:0/3062203152 wait complete. 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 10 monclient: build_initial_monmap 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.009+0000 7ff6bf509640 10 monclient: monmap: 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 Processor -- start 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 -- start start 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient: init 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b80976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-c con 0x7ff6b806b9b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 0x7ff6b8097be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-a con 0x7ff6b80a3830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): picked mon.noname-b con 0x7ff6b8137240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b811f050 con 0x7ff6b80a3830 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b811de10 con 0x7ff6b8137240 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b8099e80 con 0x7ff6b806b9b0 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:14.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bf509640 10 monclient(hunting): authenticate will time out at 1246.478745s 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40904/0 (socket says 172.21.15.67:40904) 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 -- 172.21.15.67:0/2156866120 learned_addr learned my addr 172.21.15.67:0/2156866120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 10 monclient(hunting): get_auth_request con 0x7ff6b8137240 auth_method 0 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 msgr2=0x7ff6b8098120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 msgr2=0x7ff6b8098120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bdaa6640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.010+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 0x7ff6b8097be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b80976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): get_auth_request con 0x7ff6b80a3830 auth_method 0 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): get_auth_request con 0x7ff6b806b9b0 auth_method 0 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:14.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bd2a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient(hunting): handle_auth_done global_id 104619 payload 824 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 msgr2=0x7ff6b80976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b80976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 msgr2=0x7ff6b8098120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient: _finish_hunting 0 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 monclient: found mon.noname-a 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.011+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8009390 con 0x7ff6b80a3830 2024-09-15T02:00:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.013+0000 7ff6bcaa4640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 msgr2=0x7ff6b8097be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.013+0000 7ff6bcaa4640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 0x7ff6b8097be0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8031000 tx=0x7ff6a8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8031990 con 0x7ff6b80a3830 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: dump: 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a8031af0 con 0x7ff6b80a3830 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a803b400 con 0x7ff6b80a3830 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6ad7fa640 10 monclient: dump: 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 5 monclient: authenticate success, global_id 104619 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 10 monclient: _renew_subs 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6af7fe640 4 set_mon_vals no callback set 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b813a640 con 0x7ff6b80a3830 2024-09-15T02:00:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 10 monclient: _renew_subs 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.014+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b80991e0 con 0x7ff6b80a3830 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bf509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bf509640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bf509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b804bf60 con 0x7ff6b80a3830 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff6a8039040 con 0x7ff6b80a3830 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6ad7fa640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 0x7ff69c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(239..239 src has 148..239) v4 ==== 10307+0+0 (secure 0 0 0) 0x7ff6a806d2a0 con 0x7ff6b80a3830 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bd2a5640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 0x7ff69c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.015+0000 7ff6bd2a5640 10 monclient: get_auth_request con 0x7ff69c038790 auth_method 0 2024-09-15T02:00:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.018+0000 7ff6bd2a5640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 0x7ff69c03ac40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4007fd0 tx=0x7ff6b4007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.020+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff6a803b830 con 0x7ff6b80a3830 2024-09-15T02:00:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.020+0000 7ff6ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.020+0000 7ff6ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:14.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.188+0000 7ff6bf509640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.188+0000 7ff6bf509640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.188+0000 7ff6bf509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.188+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7ff6b804bf60 con 0x7ff6b80a3830 2024-09-15T02:00:14.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.810+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff6a803f020 con 0x7ff6b80a3830 2024-09-15T02:00:14.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.810+0000 7ff6ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.812 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.810+0000 7ff6ad7fa640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-15T02:00:14.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.899+0000 7f7f5affd640 10 monclient: tick 2024-09-15T02:00:14.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.899+0000 7f7f5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:14.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.899+0000 7f7f5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f741333f0 auth_method 0 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:14.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 10 monclient(hunting): handle_auth_done global_id 124099 payload 824 2024-09-15T02:00:14.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.909+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f741337c0 con 0x7f7f741333f0 2024-09-15T02:00:14.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.911+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:14.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.911+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f680233f0 con 0x7f7f741333f0 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 10 monclient: dump: 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.914 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7f68023550 con 0x7f7f741333f0 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68008040 comp rx=0 tx=0).stop 2024-09-15T02:00:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 msgr2=0x7f7f74097a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:14.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.912+0000 7f7f5bfff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f741333f0 0x7f7f74097a20 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68008040 comp rx=0 tx=0).stop 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.915+0000 7ff6bf509640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.915+0000 7ff6bf509640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.915+0000 7ff6bf509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.915+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7ff6b804bf60 con 0x7ff6b80a3830 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.916+0000 7ff6ad7fa640 1 -- 172.21.15.67:0/2156866120 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff6a8037070 con 0x7ff6b80a3830 2024-09-15T02:00:14.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.916+0000 7ff6ad7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T02:00:14.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.916+0000 7ff6ad7fa640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-15T02:00:14.919 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h23' does not appear in the crush map 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 msgr2=0x7ff69c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 0x7ff69c03ac40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4007fd0 tx=0x7ff6b4007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 10 monclient: shutdown 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 msgr2=0x7ff6b8097be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 0x7ff6b8097be0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8031000 tx=0x7ff6a8032000 comp rx=0 tx=0).stop 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 shutdown_connections 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff69c038790 0x7ff69c03ac40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff6b80a3830 0x7ff6b8097be0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b8137240 0x7ff6b8098120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 --2- 172.21.15.67:0/2156866120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff6b806b9b0 0x7ff6b80976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 >> 172.21.15.67:0/2156866120 conn(0x7ff6b8070ff0 msgr2=0x7ff6b80a2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:14.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 shutdown_connections 2024-09-15T02:00:14.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:14.918+0000 7ff6bf509640 1 -- 172.21.15.67:0/2156866120 wait complete. 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f7a280640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f7a280640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 0x7f7f74098360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f7a280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f740a0a60 con 0x7f7f74097f60 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 0x7f7f74098360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f74097f60 auth_method 0 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:14.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 10 monclient(hunting): handle_auth_done global_id 124102 payload 824 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.919+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f740724a0 con 0x7f7f74097f60 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f737fe640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 msgr2=0x7f7f74098360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f737fe640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 0x7f7f74098360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68022eb0 con 0x7f7f74097f60 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 10 monclient: dump: 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.927 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f7f680153c0 con 0x7f7f74097f60 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 msgr2=0x7f7f74098360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 0x7f7f74098360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68022000 comp rx=0 tx=0).stop 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 msgr2=0x7f7f74098360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:14.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.922+0000 7f7f5bfff640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74097f60 0x7f7f74098360 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f74131af0 tx=0x7f7f68022000 comp rx=0 tx=0).stop 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:14.929 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:14.930 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:14.931 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:14.932 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:14.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.934 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:14.935 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:14.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:14.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:14.939 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:14.940 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:14.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:14.941 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:14.941 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:14.942 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:14.942 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h24 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7f60038770 msgr2=0x7f7f6003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7f60038770 0x7f7f6003ac20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7f7409f710 tx=0x7f7f6c037000 comp rx=0 tx=0).stop 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 10 monclient: shutdown 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 msgr2=0x7f7f74137f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 --2- 172.21.15.17:0/1843635466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7f740a0280 0x7f7f74137f10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c5e0 tx=0x7f7f64007ed0 comp rx=0 tx=0).stop 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7881d640 1 -- 172.21.15.17:0/1843635466 reap_dead start 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7881d640 1 -- 172.21.15.17:0/1843635466 reap_dead start 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 shutdown_connections 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 >> 172.21.15.17:0/1843635466 conn(0x7f7f74071270 msgr2=0x7f7f740a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 shutdown_connections 2024-09-15T02:00:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:14.923+0000 7f7f7a280640 1 -- 172.21.15.17:0/1843635466 wait complete. 2024-09-15T02:00:14.944 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:14.944 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:00:14.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.004+0000 7fe85c59d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.004+0000 7fe85c59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.004+0000 7fe85c59d640 10 monclient: monmap: 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:15.006056+0000 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:15.006056+0000 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:15.007 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:15.008 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.005+0000 7fe85c59d640 1 Processor -- start 2024-09-15T02:00:15.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.005+0000 7fe85c59d640 1 -- start start 2024-09-15T02:00:15.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.005+0000 7fe85c59d640 10 monclient: init 2024-09-15T02:00:15.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:15.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8540a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe85409eea0 0x7fe85409f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-c con 0x7fe85409eea0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe854136e40 0x7fe854139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-b con 0x7fe854136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85406a270 con 0x7fe8540a0b50 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85406f290 con 0x7fe85409eea0 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85409f7e0 con 0x7fe854136e40 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85c59d640 10 monclient(hunting): authenticate will time out at 1247.475198s 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe85409eea0 0x7fe85409f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85a339640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8540a0b50 0x7fe854069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55272/0 (socket says 172.21.15.67:55272) 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85a339640 1 -- 172.21.15.67:0/2662662446 learned_addr learned my addr 172.21.15.67:0/2662662446 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): get_auth_request con 0x7fe85409eea0 auth_method 0 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85ab3a640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe854136e40 0x7fe854139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85ab3a640 10 monclient(hunting): get_auth_request con 0x7fe854136e40 auth_method 0 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe85ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient(hunting): handle_auth_done global_id 124105 payload 274 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 -- 172.21.15.67:0/2662662446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe854136e40 msgr2=0x7fe854139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe854136e40 0x7fe854139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 -- 172.21.15.67:0/2662662446 >> v1:172.21.15.67:6789/0 conn(0x7fe8540a0b50 legacy=0x7fe854069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient: _finish_hunting 0 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 monclient: found mon.noname-c 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.006+0000 7fe859b38640 1 -- 172.21.15.67:0/2662662446 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540a0f20 con 0x7fe85409eea0 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859b38640 1 -- 172.21.15.67:0/2662662446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe85409eea0 msgr2=0x7fe85409f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859b38640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe85409eea0 0x7fe85409f2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe844009880 tx=0x7fe8440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65f841bf03696f0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 1 -- 172.21.15.67:0/2662662446 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84402ae20 con 0x7fe85409eea0 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: dump: 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: _finish_auth 0 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 1 -- 172.21.15.67:0/2662662446 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe844032440 con 0x7fe85409eea0 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 1 -- 172.21.15.67:0/2662662446 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8440327e0 con 0x7fe85409eea0 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe85c59d640 5 monclient: authenticate success, global_id 124105 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe85c59d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:15.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe85c59d640 4 set_mon_vals no callback set 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.009+0000 7fe859337640 10 monclient: dump: 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.021 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 10 monclient: shutdown 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 -- 172.21.15.67:0/2662662446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe85409eea0 msgr2=0x7fe85409f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe85409eea0 0x7fe85409f2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe844009880 tx=0x7fe8440090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 -- 172.21.15.67:0/2662662446 shutdown_connections 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe854136e40 0x7fe854139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2662662446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe85409eea0 0x7fe85409f2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 -- 172.21.15.67:0/2662662446 >> 172.21.15.67:0/2662662446 conn(0x7fe854071290 msgr2=0x7fe8540a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 -- 172.21.15.67:0/2662662446 shutdown_connections 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 1 -- 172.21.15.67:0/2662662446 wait complete. 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 10 monclient: build_initial_monmap 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.010+0000 7fe85c59d640 10 monclient: monmap: 2024-09-15T02:00:15.022 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 Processor -- start 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 -- start start 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient: init 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:15.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe85409eea0 0x7fe854134ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-c con 0x7fe85409eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8540a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe854136e40 0x7fe85411db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): picked mon.noname-a con 0x7fe854136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540a1260 con 0x7fe854136e40 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540a32e0 con 0x7fe8540a0b50 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85411f620 con 0x7fe85409eea0 2024-09-15T02:00:15.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85c59d640 10 monclient(hunting): authenticate will time out at 1247.480666s 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40918/0 (socket says 172.21.15.67:40918) 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 1 -- 172.21.15.67:0/2447335285 learned_addr learned my addr 172.21.15.67:0/2447335285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): get_auth_request con 0x7fe8540a0b50 auth_method 0 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe85409eea0 0x7fe854134ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe859b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe854136e40 0x7fe85411db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): get_auth_request con 0x7fe85409eea0 auth_method 0 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): get_auth_request con 0x7fe854136e40 auth_method 0 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85a339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.011+0000 7fe85ab3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe85ab3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 10 monclient(hunting): handle_auth_done global_id 124108 payload 824 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 -- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe85409eea0 msgr2=0x7fe854134ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe85409eea0 0x7fe854134ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 -- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe854136e40 msgr2=0x7fe85411db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe854136e40 0x7fe85411db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 10 monclient: _finish_hunting 0 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 monclient: found mon.noname-b 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.012+0000 7fe859b38640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe844009390 con 0x7fe8540a0b50 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe859b38640 1 -- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 msgr2=0x7fe85411d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe859b38640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe844009360 tx=0x7fe844032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844031d40 con 0x7fe8540a0b50 2024-09-15T02:00:15.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: dump: 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 5 monclient: authenticate success, global_id 124108 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84403d440 con 0x7fe8540a0b50 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 10 monclient: _renew_subs 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85411f170 con 0x7fe8540a0b50 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844031ea0 con 0x7fe8540a0b50 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe84a7fc640 10 monclient: dump: 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.029 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe858b36640 4 set_mon_vals no callback set 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 10 monclient: _renew_subs 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.014+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8541376d0 con 0x7fe8540a0b50 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe844039040 con 0x7fe8540a0b50 2024-09-15T02:00:15.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85c59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85c59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85404bf60 con 0x7fe8540a0b50 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe84a7fc640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 0x7fe83803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(240..240 src has 148..240) v4 ==== 10094+0+0 (secure 0 0 0) 0x7fe84406cf90 con 0x7fe8540a0b50 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85a339640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 0x7fe83803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.015+0000 7fe85a339640 10 monclient: get_auth_request con 0x7fe838038740 auth_method 0 2024-09-15T02:00:15.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.018+0000 7fe85a339640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 0x7fe83803abf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe850007fd0 tx=0x7fe850007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.020+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe844030340 con 0x7fe8540a0b50 2024-09-15T02:00:15.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.020+0000 7fe84a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.020+0000 7fe84a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:15.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.047+0000 7f0dd3d5d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.048+0000 7f0dd3d5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.048+0000 7f0dd3d5d640 10 monclient: monmap: 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:15.048844+0000 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:15.048844+0000 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:15.051 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:15.051 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.048+0000 7f0dd3d5d640 1 Processor -- start 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.048+0000 7f0dd3d5d640 1 -- start start 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.048+0000 7f0dd3d5d640 10 monclient: init 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0dcc067090 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dcc067460 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0dcc137fa0 0x7f0dcc13a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0dcc137fa0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc09cbc0 con 0x7f0dcc067090 2024-09-15T02:00:15.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc09e550 con 0x7f0dcc067460 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc067f40 con 0x7f0dcc137fa0 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd3d5d640 10 monclient(hunting): authenticate will time out at 1275.045359s 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0dcc137fa0 0x7f0dcc13a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd1af9640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f0dcc067090 0x7f0dcc1321a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59152/0 (socket says 172.21.15.17:59152) 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd1af9640 1 -- 172.21.15.17:0/1860574094 learned_addr learned my addr 172.21.15.17:0/1860574094 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): get_auth_request con 0x7f0dcc137fa0 auth_method 0 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_done global_id 104625 payload 274 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 1 -- 172.21.15.17:0/1860574094 >> v1:172.21.15.67:6789/0 conn(0x7f0dcc067460 legacy=0x7f0dcc067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 1 -- 172.21.15.17:0/1860574094 >> v1:172.21.15.17:6789/0 conn(0x7f0dcc067090 legacy=0x7f0dcc1321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient: _finish_hunting 0 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 1 monclient: found mon.noname-a 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.049+0000 7f0dd22fa640 1 -- 172.21.15.17:0/1860574094 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc1328b0 con 0x7f0dcc137fa0 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd22fa640 1 -- 172.21.15.17:0/1860574094 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc137fa0 msgr2=0x7f0dcc13a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd22fa640 1 --2- 172.21.15.17:0/1860574094 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc137fa0 0x7f0dcc13a380 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009e60 tx=0x7f0dc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54e052d1536f14b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 1 -- 172.21.15.17:0/1860574094 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4007630 con 0x7f0dcc137fa0 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: dump: 2024-09-15T02:00:15.059 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.060 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: _finish_auth 0 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 1 -- 172.21.15.17:0/1860574094 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc402d050 con 0x7f0dcc137fa0 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd3d5d640 5 monclient: authenticate success, global_id 104625 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd3d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd3d5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 1 -- 172.21.15.17:0/1860574094 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40079c0 con 0x7f0dcc137fa0 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd3d5d640 4 set_mon_vals no callback set 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.052+0000 7f0dd0af7640 10 monclient: dump: 2024-09-15T02:00:15.061 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 10 monclient: shutdown 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/1860574094 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc137fa0 msgr2=0x7f0dcc13a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/1860574094 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc137fa0 0x7f0dcc13a380 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009e60 tx=0x7f0dc4009780 comp rx=0 tx=0).stop 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/1860574094 shutdown_connections 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/1860574094 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc137fa0 0x7f0dcc13a380 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/1860574094 >> 172.21.15.17:0/1860574094 conn(0x7f0dcc0a1ee0 msgr2=0x7f0dcc0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/1860574094 shutdown_connections 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/1860574094 wait complete. 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.053+0000 7f0dd3d5d640 10 monclient: monmap: 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:15.063 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 Processor -- start 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 -- start start 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient: init 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0dcc067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 0x7f0dcc118a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0dcc067460 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0dcc137fa0 0x7f0dcc118fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f0dcc137fa0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc09d530 con 0x7f0dcc067090 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc09fce0 con 0x7f0dcc067460 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc130a80 con 0x7f0dcc137fa0 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd3d5d640 10 monclient(hunting): authenticate will time out at 1275.050837s 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd22fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0dcc137fa0 0x7f0dcc118fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37988/0 (socket says 172.21.15.17:37988) 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 1 -- 172.21.15.17:0/496004529 learned_addr learned my addr 172.21.15.17:0/496004529 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 10 monclient(hunting): get_auth_request con 0x7f0dcc067090 auth_method 0 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd1af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.054+0000 7f0dd12f8640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 0x7f0dcc118a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): get_auth_request con 0x7f0dcc137fa0 auth_method 0 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): get_auth_request con 0x7f0dcc067460 auth_method 0 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:15.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd22fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient(hunting): handle_auth_done global_id 104628 payload 824 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 -- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0dcc137fa0 msgr2=0x7f0dcc118fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0dcc137fa0 0x7f0dcc118fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 -- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 msgr2=0x7f0dcc118a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 0x7f0dcc118a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient: _finish_hunting 0 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 monclient: found mon.noname-a 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd1af9640 1 -- 172.21.15.17:0/496004529 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4009a20 con 0x7f0dcc067090 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.055+0000 7f0dd12f8640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 0x7f0dcc118a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.057+0000 7f0dd1af9640 1 -- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 msgr2=0x7f0dcc118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.057+0000 7f0dd1af9640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800c5e0 tx=0x7f0dc8004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.057+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8005cb0 con 0x7f0dcc067090 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: dump: 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.070 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8005e30 con 0x7f0dcc067090 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 5 monclient: authenticate success, global_id 104628 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 10 monclient: _renew_subs 2024-09-15T02:00:15.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc11a9d0 con 0x7f0dcc067090 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8007340 con 0x7f0dcc067090 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc1ffb640 10 monclient: dump: 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dc3fff640 4 set_mon_vals no callback set 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 10 monclient: _renew_subs 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.058+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc136d20 con 0x7f0dcc067090 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0dc801b400 con 0x7f0dcc067090 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd3d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd3d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd3d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc0724a0 con 0x7f0dcc067090 2024-09-15T02:00:15.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dc1ffb640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 0x7f0db003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(240..240 src has 148..240) v4 ==== 10094+0+0 (secure 0 0 0) 0x7f0dc804cb50 con 0x7f0dcc067090 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd12f8640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 0x7f0db003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.059+0000 7f0dd12f8640 10 monclient: get_auth_request con 0x7f0db0038740 auth_method 0 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.062+0000 7f0dd12f8640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 0x7f0db003abf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc007fd0 tx=0x7f0dbc007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.063+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0dc801b6b0 con 0x7f0dcc067090 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.063+0000 7f0dc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:15.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.063+0000 7f0dc1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:15.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.187+0000 7fe85c59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:15.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.187+0000 7fe85c59d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:15.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.187+0000 7fe85c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:15.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.187+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7fe85406f000 con 0x7fe8540a0b50 2024-09-15T02:00:15.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.290+0000 7f0dd3d5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:00:15.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.290+0000 7f0dd3d5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:00:15.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.290+0000 7f0dd3d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:15.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.290+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0dcc11afc0 con 0x7f0dcc067090 2024-09-15T02:00:15.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.291+0000 7f0dc1ffb640 1 -- 172.21.15.17:0/496004529 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0dc801a360 con 0x7f0dcc067090 2024-09-15T02:00:15.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.291+0000 7f0dc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:00:15.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.291+0000 7f0dc1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:15.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=3462109831,config=3140514009,health=2033643596,kv=769410566,logm=4152409845}) 2024-09-15T02:00:15.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=2303401494,config=3140514009,health=2033643596,kv=1529054990,logm=829881043}) 2024-09-15T02:00:15.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=3462109831,config=3140514009,health=2033643596,kv=769410566,logm=4152409845}) 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=11,kv=8,logm=61} crc {auth=102691780,config=3140514009,health=555194467,kv=1529054990,logm=1961140063}) 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123264180}) 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729771887}) 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123264180}) 2024-09-15T02:00:15.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791869427}) 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 msgr2=0x7f0db003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 0x7f0db003abf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc007fd0 tx=0x7f0dbc007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 10 monclient: shutdown 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 msgr2=0x7f0dcc118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800c5e0 tx=0x7f0dc8004960 comp rx=0 tx=0).stop 2024-09-15T02:00:15.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 shutdown_connections 2024-09-15T02:00:15.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0db0038740 0x7f0db003abf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0dcc137fa0 0x7f0dcc118fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc067460 0x7f0dcc118a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 --2- 172.21.15.17:0/496004529 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0dcc067090 0x7f0dcc118540 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:15.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 >> 172.21.15.17:0/496004529 conn(0x7f0dcc0a1ee0 msgr2=0x7f0dcc131270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:15.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914952943}) 2024-09-15T02:00:15.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978257492}) 2024-09-15T02:00:15.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914952943}) 2024-09-15T02:00:15.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2215325232}) 2024-09-15T02:00:15.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 shutdown_connections 2024-09-15T02:00:15.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:15.294+0000 7f0dd3d5d640 1 -- 172.21.15.17:0/496004529 wait complete. 2024-09-15T02:00:15.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676491823}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970803335}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676491823}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527708539}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414227536}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=7621516}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414227536}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964007198}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392538431}) 2024-09-15T02:00:15.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3113285577}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392538431}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984724360}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209003540}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768758096}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209003540}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2469371621}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871098035}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949565653}) 2024-09-15T02:00:15.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871098035}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=303717069}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820584388}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279012434}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820584388}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797152747}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267364415}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230308125}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267364415}) 2024-09-15T02:00:15.305 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276956509}) 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164019504}) 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2800603696}) 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164019504}) 2024-09-15T02:00:15.307 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421398408}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093547747}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1227606769}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093547747}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380332506}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348442727}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421938318}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348442727}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645201560}) 2024-09-15T02:00:15.308 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2196663388}) 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3647141252}) 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2196663388}) 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241487312}) 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.309 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54213494}) 2024-09-15T02:00:15.310 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523524649}) 2024-09-15T02:00:15.310 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.310 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54213494}) 2024-09-15T02:00:15.310 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992478727}) 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826614002}) 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39527995}) 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826614002}) 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691302584}) 2024-09-15T02:00:15.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957733566}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819781142}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957733566}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123311126}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851210328}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256333910}) 2024-09-15T02:00:15.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851210328}) 2024-09-15T02:00:15.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4203116129}) 2024-09-15T02:00:15.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070675603}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481791263}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070675603}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654474112}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481420945}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3378058410}) 2024-09-15T02:00:15.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481420945}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288727782}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217716962}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3045714179}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217716962}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4026661641}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123340276}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3892314984}) 2024-09-15T02:00:15.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123340276}) 2024-09-15T02:00:15.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239360510}) 2024-09-15T02:00:15.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415164232}) 2024-09-15T02:00:15.343 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239777971}) 2024-09-15T02:00:15.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415164232}) 2024-09-15T02:00:15.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3901462514}) 2024-09-15T02:00:15.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859288920}) 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=764492043}) 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859288920}) 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635953913}) 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852455536}) 2024-09-15T02:00:15.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610332854}) 2024-09-15T02:00:15.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852455536}) 2024-09-15T02:00:15.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542162333}) 2024-09-15T02:00:15.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647758027}) 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3457358002}) 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647758027}) 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=384419857}) 2024-09-15T02:00:15.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669940168}) 2024-09-15T02:00:15.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165045518}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669940168}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4281338069}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399453579}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528968679}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399453579}) 2024-09-15T02:00:15.349 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134703892}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883446219}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2800330526}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883446219}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335014074}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664894384}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2298059784}) 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.317+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664894384}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895826192}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670112696}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669622372}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670112696}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.318+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721353167}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598327340}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893809426}) 2024-09-15T02:00:15.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598327340}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595584968}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409406001}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851471411}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409406001}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.320+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=426751826}) 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.352 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469493527}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232255091}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469493527}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607974724}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2708773473}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259819401}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2708773473}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230890359}) 2024-09-15T02:00:15.353 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129488793}) 2024-09-15T02:00:15.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264437096}) 2024-09-15T02:00:15.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129488793}) 2024-09-15T02:00:15.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.322+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140970803}) 2024-09-15T02:00:15.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407053314}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746842404}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407053314}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.323+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480519593}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914267627}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626300667}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914267627}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224539039}) 2024-09-15T02:00:15.356 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.324+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630751269}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338611893}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630751269}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858854026}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587381020}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010052455}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587381020}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.325+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490660361}) 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.357 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334990339}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=826639074}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334990339}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.326+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777524449}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736942412}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=331690502}) 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736942412}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.327+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3113387456}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683389411}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233246179}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683389411}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.328+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1768135966}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859645105}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636002518}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859645105}) 2024-09-15T02:00:15.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.335+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2331517329}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2} crc {logm=3543192533,mdsmap=1531151339,mgr=1727179341,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2263380069,mon_config_key=1583035879,monmap=3178194974,osd_metadata=611002287,osd_pg_creating=1202600274,osd_snap=1063953654}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=5} crc {logm=2552077219,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2510903899,mon_config_key=199500175,monmap=3178194974,osd_metadata=1418025409}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2} crc {logm=3543192533,mdsmap=1531151339,mgr=1727179341,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2263380069,mon_config_key=1583035879,monmap=3178194974,osd_metadata=611002287,osd_pg_creating=1202600274,osd_snap=1063953654}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.336+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=3556738082,mdsmap=3140514009,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2602235713,mon_config_key=1583035879,monmap=3178194974,osd_metadata=2170749948,osd_pg_creating=1202600274,osd_snap=1063953654,osdmap=3015745646}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2684782967}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2153093395}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2684782967}) 2024-09-15T02:00:15.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.338+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4230570998}) 2024-09-15T02:00:15.361 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2468284698}) 2024-09-15T02:00:15.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=10812319}) 2024-09-15T02:00:15.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:00:15.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2468284698}) 2024-09-15T02:00:15.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:15.344+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=50} crc {osdmap=2309006742}) 2024-09-15T02:00:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.976+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe844042070 con 0x7fe8540a0b50 2024-09-15T02:00:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.976+0000 7fe84a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:15.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:15.976+0000 7fe84a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-15T02:00:16.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.080+0000 7fe85c59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:16.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.080+0000 7fe85c59d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:16.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.080+0000 7fe85c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.080+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7fe85406f000 con 0x7fe8540a0b50 2024-09-15T02:00:16.315 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:00:16.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:16.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.410+0000 7eff00034640 10 monclient: get_monmap_and_config 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.410+0000 7eff00034640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.410+0000 7eff00034640 10 monclient: monmap: 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:16.411353+0000 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:16.411353+0000 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:16.413 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:16.414 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 Processor -- start 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 -- start start 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient: init 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efef809a9f0 0x7efef809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): picked mon.noname-a con 0x7efef809a9f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef812e420 0x7efef809b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): picked mon.noname-c con 0x7efef812e420 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:16.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efef812e810 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef8098fb0 con 0x7efef812e810 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef812e130 con 0x7efef809a9f0 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef809b300 con 0x7efef812e420 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7eff00034640 10 monclient(hunting): authenticate will time out at 1276.407879s 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7efefddd0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efef809a9f0 0x7efef809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7efefddd0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efef809a9f0 0x7efef809adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38004/0 (socket says 172.21.15.17:38004) 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7efefddd0640 1 -- 172.21.15.17:0/928433166 learned_addr learned my addr 172.21.15.17:0/928433166 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.411+0000 7efefddd0640 10 monclient(hunting): get_auth_request con 0x7efef809a9f0 auth_method 0 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 1 --2- 172.21.15.17:0/928433166 >> v2:172.21.15.67:3300/0 conn(0x7efef812e420 0x7efef809b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): get_auth_request con 0x7efef812e420 auth_method 0 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient(hunting): handle_auth_done global_id 104631 payload 274 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 1 -- 172.21.15.17:0/928433166 >> v2:172.21.15.67:3300/0 conn(0x7efef812e420 msgr2=0x7efef809b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 1 --2- 172.21.15.17:0/928433166 >> v2:172.21.15.67:3300/0 conn(0x7efef812e420 0x7efef809b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 1 -- 172.21.15.17:0/928433166 >> v1:172.21.15.67:6789/0 conn(0x7efef812e810 legacy=0x7efef809bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 1 monclient: found mon.noname-a 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.412+0000 7efefddd0640 1 -- 172.21.15.17:0/928433166 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef8098cd0 con 0x7efef809a9f0 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefddd0640 1 -- 172.21.15.17:0/928433166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 msgr2=0x7efef809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefddd0640 1 --2- 172.21.15.17:0/928433166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef809adc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efef80769e0 tx=0x7efef400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60385cef27dad507 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 1 -- 172.21.15.17:0/928433166 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4032530 con 0x7efef809a9f0 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:16.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: dump: 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 1 -- 172.21.15.17:0/928433166 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4032690 con 0x7efef809a9f0 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 5 monclient: authenticate success, global_id 104631 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 4 set_mon_vals no callback set 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 1 -- 172.21.15.17:0/928433166 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4032a30 con 0x7efef809a9f0 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7efefcdce640 10 monclient: dump: 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.426 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 10 monclient: shutdown 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 -- 172.21.15.17:0/928433166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 msgr2=0x7efef809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 --2- 172.21.15.17:0/928433166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef809adc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efef80769e0 tx=0x7efef400b460 comp rx=0 tx=0).stop 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 -- 172.21.15.17:0/928433166 shutdown_connections 2024-09-15T02:00:16.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 --2- 172.21.15.17:0/928433166 >> v2:172.21.15.67:3300/0 conn(0x7efef812e420 0x7efef809b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 --2- 172.21.15.17:0/928433166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef809adc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 -- 172.21.15.17:0/928433166 >> 172.21.15.17:0/928433166 conn(0x7efef80a1000 msgr2=0x7efef80a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 -- 172.21.15.17:0/928433166 shutdown_connections 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.415+0000 7eff00034640 1 -- 172.21.15.17:0/928433166 wait complete. 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.416+0000 7eff00034640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.416+0000 7eff00034640 10 monclient: monmap: 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.428 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.416+0000 7eff00034640 1 Processor -- start 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 -- start start 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient: init 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): picked mon.noname-a con 0x7efef809a9f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): picked mon.noname-c con 0x7efef812e420 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:16.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef812e810 0x7efef806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): picked mon.noname-b con 0x7efef812e810 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8117bb0 con 0x7efef809a9f0 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef809f230 con 0x7efef812e810 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef80983b0 con 0x7efef812e420 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7eff00034640 10 monclient(hunting): authenticate will time out at 1276.413517s 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34800/0 (socket says 172.21.15.17:34800) 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38012/0 (socket says 172.21.15.17:38012) 2024-09-15T02:00:16.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 -- 172.21.15.17:0/2192095746 learned_addr learned my addr 172.21.15.17:0/2192095746 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): get_auth_request con 0x7efef812e420 auth_method 0 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): get_auth_request con 0x7efef809a9f0 auth_method 0 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef812e810 0x7efef806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): get_auth_request con 0x7efef812e810 auth_method 0 2024-09-15T02:00:16.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefe5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient(hunting): handle_auth_done global_id 94784 payload 824 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 -- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef812e810 msgr2=0x7efef806ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef812e810 0x7efef806ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 -- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 msgr2=0x7efef813d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 monclient: found mon.noname-c 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefd5cf640 1 -- 172.21.15.17:0/2192095746 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef400b700 con 0x7efef812e420 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.417+0000 7efefddd0640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efefd5cf640 1 -- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 msgr2=0x7efef8127a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efefd5cf640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efee800ea20 tx=0x7efee80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8009600 con 0x7efef812e420 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: dump: 2024-09-15T02:00:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee800cc20 con 0x7efef812e420 2024-09-15T02:00:16.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 5 monclient: authenticate success, global_id 94784 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 10 monclient: _renew_subs 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef806f130 con 0x7efef812e420 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8013470 con 0x7efef812e420 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeedffb640 10 monclient: dump: 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.435 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 10 monclient: _renew_subs 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7efeeffff640 4 set_mon_vals no callback set 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.420+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef813dee0 con 0x7efef812e420 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.421+0000 7eff00034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.421+0000 7eff00034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.421+0000 7eff00034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.421+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef80724a0 con 0x7efef812e420 2024-09-15T02:00:16.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.421+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efee8013860 con 0x7efef812e420 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.422+0000 7efeedffb640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 0x7efedc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.422+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7efee804d250 con 0x7efef812e420 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.422+0000 7efefddd0640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 0x7efedc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.422+0000 7efefddd0640 10 monclient: get_auth_request con 0x7efedc038740 auth_method 0 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.425+0000 7efefddd0640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 0x7efedc03abf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efef8098720 tx=0x7efef403a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.426+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efee80236b0 con 0x7efef812e420 2024-09-15T02:00:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.426+0000 7efeedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.426+0000 7efeedffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:16.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.470+0000 7fe84a7fc640 1 -- 172.21.15.67:0/2447335285 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe844037030 con 0x7fe8540a0b50 2024-09-15T02:00:16.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.470+0000 7fe84a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T02:00:16.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.470+0000 7fe84a7fc640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-15T02:00:16.474 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h24' does not appear in the crush map 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 msgr2=0x7fe83803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 0x7fe83803abf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe850007fd0 tx=0x7fe850007f00 comp rx=0 tx=0).stop 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 10 monclient: shutdown 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 msgr2=0x7fe85411d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe844009360 tx=0x7fe844032000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 shutdown_connections 2024-09-15T02:00:16.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe838038740 0x7fe83803abf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe854136e40 0x7fe85411db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8540a0b50 0x7fe85411d5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 --2- 172.21.15.67:0/2447335285 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe85409eea0 0x7fe854134ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 >> 172.21.15.67:0/2447335285 conn(0x7fe854071290 msgr2=0x7fe85409ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 shutdown_connections 2024-09-15T02:00:16.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.472+0000 7fe85c59d640 1 -- 172.21.15.67:0/2447335285 wait complete. 2024-09-15T02:00:16.485 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:16.485 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h25 2024-09-15T02:00:16.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.567+0000 7fc3a5f9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:16.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.567+0000 7fc3a5f9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.567+0000 7fc3a5f9d640 10 monclient: monmap: 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:16.568329+0000 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:16.568329+0000 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:16.571 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:16.571 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.567+0000 7fc3a5f9d640 1 Processor -- start 2024-09-15T02:00:16.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 1 -- start start 2024-09-15T02:00:16.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient: init 2024-09-15T02:00:16.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3a0098140 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3a00a0370 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a0098c20 0x7fc3a013a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a0098c20 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00a0b80 con 0x7fc3a0098140 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a01328f0 con 0x7fc3a00a0370 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a00a00b0 con 0x7fc3a0098c20 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc3a5f9d640 10 monclient(hunting): authenticate will time out at 1249.037549s 2024-09-15T02:00:16.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a0098c20 0x7fc3a013a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a0098c20 0x7fc3a013a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40932/0 (socket says 172.21.15.67:40932) 2024-09-15T02:00:16.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 -- 172.21.15.67:0/3281947582 learned_addr learned my addr 172.21.15.67:0/3281947582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:16.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3a00a0370 0x7fc3a0098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55288/0 (socket says 172.21.15.67:55288) 2024-09-15T02:00:16.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): get_auth_request con 0x7fc3a0098c20 auth_method 0 2024-09-15T02:00:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient(hunting): handle_auth_done global_id 124120 payload 274 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 -- 172.21.15.67:0/3281947582 >> v1:172.21.15.67:6789/0 conn(0x7fc3a00a0370 legacy=0x7fc3a0098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 -- 172.21.15.67:0/3281947582 >> v1:172.21.15.17:6789/0 conn(0x7fc3a0098140 legacy=0x7fc3a01321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 monclient: found mon.noname-c 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.568+0000 7fc39ffff640 1 -- 172.21.15.67:0/3281947582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0071390 con 0x7fc3a0098c20 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.571+0000 7fc39ffff640 1 -- 172.21.15.67:0/3281947582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a0098c20 msgr2=0x7fc3a013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.571+0000 7fc39ffff640 1 --2- 172.21.15.67:0/3281947582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a0098c20 0x7fc3a013a0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc398009f20 tx=0x7fc398009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7048850d910e19e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 1 -- 172.21.15.67:0/3281947582 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39802aea0 con 0x7fc3a0098c20 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: dump: 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 1 -- 172.21.15.67:0/3281947582 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39802d050 con 0x7fc3a0098c20 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 5 monclient: authenticate success, global_id 124120 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:16.588 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 1 -- 172.21.15.67:0/3281947582 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398007840 con 0x7fc3a0098c20 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 4 set_mon_vals no callback set 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc39e7fc640 10 monclient: dump: 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 10 monclient: shutdown 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/3281947582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a0098c20 msgr2=0x7fc3a013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/3281947582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a0098c20 0x7fc3a013a0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc398009f20 tx=0x7fc398009840 comp rx=0 tx=0).stop 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/3281947582 shutdown_connections 2024-09-15T02:00:16.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/3281947582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a0098c20 0x7fc3a013a0d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/3281947582 >> 172.21.15.67:0/3281947582 conn(0x7fc3a00a5780 msgr2=0x7fc3a00a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/3281947582 shutdown_connections 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.572+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/3281947582 wait complete. 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.573+0000 7fc3a5f9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.573+0000 7fc3a5f9d640 10 monclient: monmap: 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:16.591 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.573+0000 7fc3a5f9d640 1 Processor -- start 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.573+0000 7fc3a5f9d640 1 -- start start 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.573+0000 7fc3a5f9d640 10 monclient: init 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:16.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a0098140 0x7fc3a0139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a0098140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a0098c20 0x7fc3a012fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a0098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a00a0370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0099090 con 0x7fc3a0098c20 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00a3c70 con 0x7fc3a00a0370 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0131cd0 con 0x7fc3a0098140 2024-09-15T02:00:16.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc3a5f9d640 10 monclient(hunting): authenticate will time out at 1249.043299s 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40934/0 (socket says 172.21.15.67:40934) 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 -- 172.21.15.67:0/2318160625 learned_addr learned my addr 172.21.15.67:0/2318160625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): get_auth_request con 0x7fc3a00a0370 auth_method 0 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a0098c20 0x7fc3a012fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a0098140 0x7fc3a0139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): get_auth_request con 0x7fc3a0098c20 auth_method 0 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): get_auth_request con 0x7fc3a0098140 auth_method 0 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient(hunting): handle_auth_done global_id 124123 payload 824 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 -- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a0098140 msgr2=0x7fc3a0139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a0098140 0x7fc3a0139780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 -- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a0098c20 msgr2=0x7fc3a012fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a0098c20 0x7fc3a012fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 monclient: found mon.noname-b 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.574+0000 7fc39ffff640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398009ae0 con 0x7fc3a00a0370 2024-09-15T02:00:16.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc39ffff640 1 -- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 msgr2=0x7fc3a0130210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc39ffff640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc398009ab0 tx=0x7fc398036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398031cb0 con 0x7fc3a00a0370 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: dump: 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.597 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398031e10 con 0x7fc3a00a0370 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 5 monclient: authenticate success, global_id 124123 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 10 monclient: _renew_subs 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a0133dc0 con 0x7fc3a00a0370 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39803c670 con 0x7fc3a00a0370 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:16.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3877fe640 10 monclient: dump: 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.600 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 10 monclient: _renew_subs 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a0138c90 con 0x7fc3a00a0370 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.577+0000 7fc39dffb640 4 set_mon_vals no callback set 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc39803caa0 con 0x7fc3a00a0370 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3a5f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3a5f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3a5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a004bf60 con 0x7fc3a00a0370 2024-09-15T02:00:16.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3877fe640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 0x7fc38c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7fc39806d690 con 0x7fc3a00a0370 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc39f7fe640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 0x7fc38c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.578+0000 7fc39f7fe640 10 monclient: get_auth_request con 0x7fc38c038740 auth_method 0 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.581+0000 7fc39f7fe640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 0x7fc38c03abf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc390007fd0 tx=0x7fc390007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.583+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc39803ad80 con 0x7fc3a00a0370 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.583+0000 7fc3877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.583+0000 7fc3877fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:16.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7eff00034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7eff00034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7eff00034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efef809d6c0 con 0x7efef812e420 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7efeedffb640 1 -- 172.21.15.17:0/2192095746 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7efee80232f0 con 0x7efef812e420 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7efeedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:16.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.652+0000 7efeedffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:16.656 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:16.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 msgr2=0x7efedc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 0x7efedc03abf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efef8098720 tx=0x7efef403a000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 10 monclient: shutdown 2024-09-15T02:00:16.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 msgr2=0x7efef8127a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efee800ea20 tx=0x7efee80090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 shutdown_connections 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efedc038740 0x7efedc03abf0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef812e810 0x7efef806ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efef812e420 0x7efef8127a60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 --2- 172.21.15.17:0/2192095746 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efef809a9f0 0x7efef813d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.654+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 >> 172.21.15.17:0/2192095746 conn(0x7efef80a1000 msgr2=0x7efef80997b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.655+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 shutdown_connections 2024-09-15T02:00:16.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.655+0000 7eff00034640 1 -- 172.21.15.17:0/2192095746 wait complete. 2024-09-15T02:00:16.666 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:00:16.666 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:00:16.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:00:16.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:00:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.750+0000 7fc3a5f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.750+0000 7fc3a5f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.750+0000 7fc3a5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:16.750+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7fc3a006f000 con 0x7fc3a00a0370 2024-09-15T02:00:16.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.762+0000 7f779519d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:16.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.762+0000 7f779519d640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.762+0000 7f779519d640 10 monclient: monmap: 2024-09-15T02:00:16.769 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.769 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.769 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:16.763514+0000 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:16.763514+0000 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:16.770 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 Processor -- start 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 -- start start 2024-09-15T02:00:16.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient: init 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7790067090 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): picked mon.noname-b con 0x7f7790067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:16.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): picked mon.noname-a con 0x7f7790136f90 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779009cbc0 con 0x7f7790067090 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779009e550 con 0x7f7790136f90 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77901328b0 con 0x7f7790067460 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f779519d640 10 monclient(hunting): authenticate will time out at 1276.759807s 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38014/0 (socket says 172.21.15.17:38014) 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 1 -- 172.21.15.17:0/3642166196 learned_addr learned my addr 172.21.15.17:0/3642166196 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:16.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778e59c640 10 monclient(hunting): get_auth_request con 0x7f7790067460 auth_method 0 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 10 monclient(hunting): get_auth_request con 0x7f7790136f90 auth_method 0 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.763+0000 7f778f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient(hunting): handle_auth_done global_id 94790 payload 274 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 1 -- 172.21.15.17:0/3642166196 >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 msgr2=0x7f7790139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 1 --2- 172.21.15.17:0/3642166196 >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 1 -- 172.21.15.17:0/3642166196 >> v1:172.21.15.67:6789/0 conn(0x7f7790067090 legacy=0x7f77901321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 1 monclient: found mon.noname-b 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778e59c640 1 -- 172.21.15.17:0/3642166196 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779009e1e0 con 0x7f7790067460 2024-09-15T02:00:16.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.764+0000 7f778f59e640 1 --2- 172.21.15.17:0/3642166196 >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:16.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778e59c640 1 -- 172.21.15.17:0/3642166196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 msgr2=0x7f7790067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778e59c640 1 --2- 172.21.15.17:0/3642166196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790067830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7784009850 tx=0x7f77840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a3b0054663a048b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 1 -- 172.21.15.17:0/3642166196 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778402adf0 con 0x7f7790067460 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: dump: 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 1 -- 172.21.15.17:0/3642166196 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784032440 con 0x7f7790067460 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 5 monclient: authenticate success, global_id 94790 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 1 -- 172.21.15.17:0/3642166196 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784032820 con 0x7f7790067460 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 4 set_mon_vals no callback set 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f778dd9b640 10 monclient: dump: 2024-09-15T02:00:16.787 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 10 monclient: shutdown 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 1 -- 172.21.15.17:0/3642166196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 msgr2=0x7f7790067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.767+0000 7f779519d640 1 --2- 172.21.15.17:0/3642166196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790067830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7784009850 tx=0x7f77840090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 -- 172.21.15.17:0/3642166196 shutdown_connections 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 --2- 172.21.15.17:0/3642166196 >> v2:172.21.15.17:3300/0 conn(0x7f7790136f90 0x7f7790139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 --2- 172.21.15.17:0/3642166196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790067830 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 -- 172.21.15.17:0/3642166196 >> 172.21.15.17:0/3642166196 conn(0x7f77900a1ee0 msgr2=0x7f77900a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 -- 172.21.15.17:0/3642166196 shutdown_connections 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 -- 172.21.15.17:0/3642166196 wait complete. 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 10 monclient: monmap: 2024-09-15T02:00:16.789 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.768+0000 7f779519d640 1 Processor -- start 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 -- start start 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient: init 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7790067090 0x7f7790141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): picked mon.noname-b con 0x7f7790067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): picked mon.noname-c con 0x7f7790067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 0x7f77901189c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): picked mon.noname-a con 0x7f7790136f90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779011c590 con 0x7f7790136f90 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779009d560 con 0x7f7790067090 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779011afb0 con 0x7f7790067460 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f779519d640 10 monclient(hunting): authenticate will time out at 1276.765464s 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 0x7f77901189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34824/0 (socket says 172.21.15.17:34824) 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 1 -- 172.21.15.17:0/3660987168 learned_addr learned my addr 172.21.15.17:0/3660987168 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): get_auth_request con 0x7f7790067460 auth_method 0 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): get_auth_request con 0x7f7790136f90 auth_method 0 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7790067090 0x7f7790141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): get_auth_request con 0x7f7790067090 auth_method 0 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient(hunting): handle_auth_done global_id 104643 payload 824 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 msgr2=0x7f7790141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7790067090 msgr2=0x7f7790141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7790067090 0x7f7790141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 monclient: found mon.noname-a 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:16.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788009ae0 con 0x7f7790136f90 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.769+0000 7f778e59c640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7790067460 0x7f7790141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 msgr2=0x7f77901189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f778f59e640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 0x7f77901189c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7788009ee0 tx=0x7f778800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788018070 con 0x7f7790136f90 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: dump: 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 5 monclient: authenticate success, global_id 104643 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778800f040 con 0x7f7790136f90 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 10 monclient: _renew_subs 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7784009390 con 0x7f7790136f90 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880092e0 con 0x7f7790136f90 2024-09-15T02:00:16.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f7776ffd640 10 monclient: dump: 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 10 monclient: _renew_subs 2024-09-15T02:00:16.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779011a3e0 con 0x7f7790136f90 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.772+0000 7f778d59a640 4 set_mon_vals no callback set 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f779519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f779519d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 0x7f77901192d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779011c0b0 con 0x7f7790118f00 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 0x7f77901192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient: get_auth_request con 0x7f7790118f00 auth_method 0 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7788003020 con 0x7f7790136f90 2024-09-15T02:00:16.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 10 monclient(hunting): handle_auth_done global_id 104646 payload 824 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778ed9d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f779011c280 con 0x7f7790118f00 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f7776ffd640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f777c038740 0x7f777c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f778804d090 con 0x7f7790136f90 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778e59c640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f777c038740 0x7f777c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.773+0000 7f778e59c640 10 monclient: get_auth_request con 0x7f777c038740 auth_method 0 2024-09-15T02:00:16.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f778ed9d640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 msgr2=0x7f77901192d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f778ed9d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 0x7f77901192d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778000d8b0 con 0x7f7790118f00 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f7776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f7776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.775+0000 7f7776ffd640 10 monclient: dump: 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.800 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f778000da10 con 0x7f7790118f00 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 msgr2=0x7f77901192d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 0x7f77901192d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 msgr2=0x7f77901192d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:16.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f7776ffd640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790118f00 0x7f77901192d0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.776+0000 7f778e59c640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f777c038740 0x7f777c03abf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7784009130 tx=0x7f7784032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f779519d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f779519d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 0x7f7790132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f779519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790063870 con 0x7f7790119810 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f778ed9d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 0x7f7790132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f778ed9d640 10 monclient: get_auth_request con 0x7f7790119810 auth_method 0 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f778ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f778ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.781+0000 7f778ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.782+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.782+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.782+0000 7f778ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.782+0000 7f778ed9d640 10 monclient(hunting): handle_auth_done global_id 104649 payload 824 2024-09-15T02:00:16.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.782+0000 7f778ed9d640 1 -- 172.21.15.17:0/3660987168 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f779011a8f0 con 0x7f7790119810 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f778ed9d640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 msgr2=0x7f7790132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f778ed9d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 0x7f7790132ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778001d610 con 0x7f7790119810 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 10 monclient: dump: 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7f778001d770 con 0x7f7790119810 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 msgr2=0x7f7790132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 0x7f7790132ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780019000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 msgr2=0x7f7790132ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:16.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.784+0000 7f7776ffd640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790119810 0x7f7790132ad0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f778000f780 tx=0x7f7780019000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:00:16.806 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.807 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.808 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:16.809 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.810 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:16.813 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.814 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.815 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:16.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:16.817 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:16.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f777c038740 msgr2=0x7f777c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f777c038740 0x7f777c03abf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7784009130 tx=0x7f7784032000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 10 monclient: shutdown 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 msgr2=0x7f77901189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 --2- 172.21.15.17:0/3660987168 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7790136f90 0x7f77901189c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7788009ee0 tx=0x7f778800b040 comp rx=0 tx=0).stop 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 reap_dead start 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f778f59e640 1 -- 172.21.15.17:0/3660987168 reap_dead start 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 shutdown_connections 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 >> 172.21.15.17:0/3660987168 conn(0x7f77900a1ee0 msgr2=0x7f7790097ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 shutdown_connections 2024-09-15T02:00:16.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.785+0000 7f779519d640 1 -- 172.21.15.17:0/3660987168 wait complete. 2024-09-15T02:00:16.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:16.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:00:16.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:00:16.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.916+0000 7f478f4c9640 10 monclient: get_monmap_and_config 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.916+0000 7f478f4c9640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.916+0000 7f478f4c9640 10 monclient: monmap: 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:16.917451+0000 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:16.917451+0000 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:16.920 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:16.920 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:16.920 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:16.920 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 Processor -- start 2024-09-15T02:00:16.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 -- start start 2024-09-15T02:00:16.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient: init 2024-09-15T02:00:16.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f478809c0d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f478809e970 0x7f478809c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f478809e970 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f478806eb30 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880699c0 con 0x7f478806eb30 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478809ed70 con 0x7f478809c0d0 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478806a5e0 con 0x7f478809e970 2024-09-15T02:00:16.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.917+0000 7f478f4c9640 10 monclient(hunting): authenticate will time out at 1276.913910s 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478da66640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f478806eb30 0x7f478806ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59168/0 (socket says 172.21.15.17:59168) 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478da66640 1 -- 172.21.15.17:0/2802622930 learned_addr learned my addr 172.21.15.17:0/2802622930 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 1 --2- 172.21.15.17:0/2802622930 >> v2:172.21.15.67:3300/0 conn(0x7f478809e970 0x7f478809c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): get_auth_request con 0x7f478809e970 auth_method 0 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient(hunting): handle_auth_done global_id 124129 payload 274 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 1 -- 172.21.15.17:0/2802622930 >> v1:172.21.15.67:6789/0 conn(0x7f478809c0d0 legacy=0x7f478809a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 1 -- 172.21.15.17:0/2802622930 >> v1:172.21.15.17:6789/0 conn(0x7f478806eb30 legacy=0x7f478806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 1 monclient: found mon.noname-c 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:16.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.918+0000 7f478ca64640 1 -- 172.21.15.17:0/2802622930 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478809f6e0 con 0x7f478809e970 2024-09-15T02:00:16.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f478ca64640 1 -- 172.21.15.17:0/2802622930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478809e970 msgr2=0x7f478809c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f478ca64640 1 --2- 172.21.15.17:0/2802622930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478809e970 0x7f478809c4a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4778009850 tx=0x7f47780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f71f4d4e73c33aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 1 -- 172.21.15.17:0/2802622930 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778032440 con 0x7f478809e970 2024-09-15T02:00:16.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: dump: 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 1 -- 172.21.15.17:0/2802622930 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47780325a0 con 0x7f478809e970 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 1 -- 172.21.15.17:0/2802622930 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778032940 con 0x7f478809e970 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f478f4c9640 5 monclient: authenticate success, global_id 124129 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f478f4c9640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f478f4c9640 4 set_mon_vals no callback set 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.921+0000 7f477ffff640 10 monclient: dump: 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.934 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 10 monclient: shutdown 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 -- 172.21.15.17:0/2802622930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478809e970 msgr2=0x7f478809c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 --2- 172.21.15.17:0/2802622930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478809e970 0x7f478809c4a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4778009850 tx=0x7f47780090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 -- 172.21.15.17:0/2802622930 shutdown_connections 2024-09-15T02:00:16.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 --2- 172.21.15.17:0/2802622930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478809e970 0x7f478809c4a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 -- 172.21.15.17:0/2802622930 >> 172.21.15.17:0/2802622930 conn(0x7f47880712f0 msgr2=0x7f47880a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 -- 172.21.15.17:0/2802622930 shutdown_connections 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 1 -- 172.21.15.17:0/2802622930 wait complete. 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 10 monclient: build_initial_monmap 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.922+0000 7f478f4c9640 10 monclient: monmap: 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:16.936 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 Processor -- start 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 -- start start 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient: init 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478806eb30 0x7f4788141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f478806eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:16.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 0x7f4788133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f478809c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 0x7f478811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f478809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4788121190 con 0x7f478809c0d0 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881416d0 con 0x7f478806eb30 2024-09-15T02:00:16.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788120eb0 con 0x7f478809e970 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.923+0000 7f478f4c9640 10 monclient(hunting): authenticate will time out at 1276.919944s 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 0x7f4788133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 0x7f478811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 0x7f4788133d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38040/0 (socket says 172.21.15.17:38040) 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 1 -- 172.21.15.17:0/822603834 learned_addr learned my addr 172.21.15.17:0/822603834 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 0x7f478811d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34838/0 (socket says 172.21.15.17:34838) 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): get_auth_request con 0x7f478809c0d0 auth_method 0 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): get_auth_request con 0x7f478809e970 auth_method 0 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478806eb30 0x7f4788141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): get_auth_request con 0x7f478806eb30 auth_method 0 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient(hunting): handle_auth_done global_id 94796 payload 824 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478806eb30 msgr2=0x7f4788141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478806eb30 0x7f4788141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 msgr2=0x7f4788133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 0x7f4788133d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient: _finish_hunting 0 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 monclient: found mon.noname-c 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4778009390 con 0x7f478809e970 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478806eb30 0x7f4788141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.924+0000 7f478ca64640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f478809c0d0 0x7f4788133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 msgr2=0x7f478811d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478da66640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 0x7f478811d9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f478000cb60 tx=0x7f47800052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478000a850 con 0x7f478809e970 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: dump: 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.942 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 5 monclient: authenticate success, global_id 94796 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478000a9d0 con 0x7f478809e970 2024-09-15T02:00:16.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478000ad90 con 0x7f478809e970 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 10 monclient: _renew_subs 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4788121470 con 0x7f478809e970 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477d7fa640 10 monclient: dump: 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.944 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f477f7fe640 4 set_mon_vals no callback set 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 10 monclient: _renew_subs 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.926+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4788134b30 con 0x7f478809e970 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478f4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478f4c9640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 0x7f478811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788063b90 con 0x7f478811ff50 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4780007300 con 0x7f478809e970 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 0x7f478811df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient: get_auth_request con 0x7f478811ff50 auth_method 0 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.927+0000 7f477d7fa640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f476c038740 0x7f476c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.928+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f478004cf80 con 0x7f478809e970 2024-09-15T02:00:16.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.928+0000 7f478d265640 10 monclient(hunting): handle_auth_done global_id 94799 payload 824 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.928+0000 7f478d265640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f478811f410 con 0x7f478811ff50 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.928+0000 7f478ca64640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f476c038740 0x7f476c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.928+0000 7f478ca64640 10 monclient: get_auth_request con 0x7f476c038740 auth_method 0 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f478d265640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 msgr2=0x7f478811df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 0x7f478811df20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f4784007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478400dda0 con 0x7f478811ff50 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 10 monclient: dump: 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.947 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4784014470 con 0x7f478811ff50 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f478ca64640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f476c038740 0x7f476c03abf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4778009130 tx=0x7f4778032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 msgr2=0x7f478811df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 0x7f478811df20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f4784007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:16.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 msgr2=0x7f478811df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:16.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.930+0000 7f477d7fa640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478811ff50 0x7f478811df20 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f4784007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478f4c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478f4c9640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 0x7f478811e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478f4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788121300 con 0x7f47881342d0 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 0x7f478811e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient: get_auth_request con 0x7f47881342d0 auth_method 0 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:16.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 10 monclient(hunting): handle_auth_done global_id 94802 payload 824 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.936+0000 7f478d265640 1 -- 172.21.15.17:0/822603834 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47880724a0 con 0x7f47881342d0 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478d265640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 msgr2=0x7f478811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478d265640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 0x7f478811e460 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f478401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478401ec30 con 0x7f47881342d0 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 10 monclient: dump: 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:16.952 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f478401ed90 con 0x7f47881342d0 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 msgr2=0x7f478811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 0x7f478811e460 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f478401a000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 msgr2=0x7f478811e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f477d7fa640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f47881342d0 0x7f478811e460 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f478400fa50 tx=0x7f478401a000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:16.954 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.955 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.956 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:16.957 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:16.958 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:16.959 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:16.959 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:16.959 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.959 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.962 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:16.963 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:16.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:16.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.966 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:00:16.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:16.968 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f476c038740 msgr2=0x7f476c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f476c038740 0x7f476c03abf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4778009130 tx=0x7f4778032000 comp rx=0 tx=0).stop 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 10 monclient: shutdown 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 msgr2=0x7f478811d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 --2- 172.21.15.17:0/822603834 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f478809e970 0x7f478811d9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f478000cb60 tx=0x7f47800052b0 comp rx=0 tx=0).stop 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 reap_dead start 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478da66640 1 -- 172.21.15.17:0/822603834 reap_dead start 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 shutdown_connections 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.939+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 >> 172.21.15.17:0/822603834 conn(0x7f47880712f0 msgr2=0x7f4788134e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.940+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 shutdown_connections 2024-09-15T02:00:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:16.940+0000 7f478f4c9640 1 -- 172.21.15.17:0/822603834 wait complete. 2024-09-15T02:00:16.971 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:16.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:00:16.971 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 10 monclient: get_monmap_and_config 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 10 monclient: build_initial_monmap 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 10 monclient: monmap: 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:17.066862+0000 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:17.066862+0000 2024-09-15T02:00:17.068 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:17.069 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 1 Processor -- start 2024-09-15T02:00:17.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 1 -- start start 2024-09-15T02:00:17.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.066+0000 7f342fe34640 10 monclient: init 2024-09-15T02:00:17.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:17.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T02:00:17.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f34280a58b0 0x7f3428132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-a con 0x7f34280a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:17.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34280980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:17.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-b con 0x7f3428136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34280a01e0 con 0x7f34280980a0 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34280a1580 con 0x7f34280a58b0 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34280a5c80 con 0x7f3428136dd0 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342fe34640 10 monclient(hunting): authenticate will time out at 1277.063383s 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f34280a58b0 0x7f3428132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f34280a58b0 0x7f3428132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38048/0 (socket says 172.21.15.17:38048) 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 -- 172.21.15.17:0/3964023623 learned_addr learned my addr 172.21.15.17:0/3964023623 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:17.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): get_auth_request con 0x7f34280a58b0 auth_method 0 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:34866/0 (socket says 172.21.15.17:34866) 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): get_auth_request con 0x7f3428136dd0 auth_method 0 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient(hunting): handle_auth_done global_id 104655 payload 274 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 -- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 msgr2=0x7f34281391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 -- 172.21.15.17:0/3964023623 >> v1:172.21.15.17:6789/0 conn(0x7f34280980a0 legacy=0x7f34280984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient: _finish_hunting 0 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 monclient: found mon.noname-a 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342dbd0640 1 -- 172.21.15.17:0/3964023623 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428098bb0 con 0x7f34280a58b0 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.067+0000 7f342e3d1640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342dbd0640 1 -- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280a58b0 msgr2=0x7f3428132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342dbd0640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280a58b0 0x7f3428132040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3424009910 tx=0x7f34240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db56f6dd677a2601 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 1 -- 172.21.15.17:0/3964023623 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424032440 con 0x7f34280a58b0 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: dump: 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.079 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: _finish_auth 0 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 1 -- 172.21.15.17:0/3964023623 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34240325a0 con 0x7f34280a58b0 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 1 -- 172.21.15.17:0/3964023623 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424032940 con 0x7f34280a58b0 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342fe34640 5 monclient: authenticate success, global_id 104655 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342fe34640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342fe34640 4 set_mon_vals no callback set 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.070+0000 7f342cbce640 10 monclient: dump: 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.080 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 10 monclient: shutdown 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 -- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280a58b0 msgr2=0x7f3428132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280a58b0 0x7f3428132040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3424009910 tx=0x7f34240091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 -- 172.21.15.17:0/3964023623 shutdown_connections 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 --2- 172.21.15.17:0/3964023623 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280a58b0 0x7f3428132040 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 -- 172.21.15.17:0/3964023623 >> 172.21.15.17:0/3964023623 conn(0x7f3428070ed0 msgr2=0x7f34280a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 -- 172.21.15.17:0/3964023623 shutdown_connections 2024-09-15T02:00:17.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.071+0000 7f342fe34640 1 -- 172.21.15.17:0/3964023623 wait complete. 2024-09-15T02:00:17.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient: build_initial_monmap 2024-09-15T02:00:17.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient: monmap: 2024-09-15T02:00:17.088 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:17.088 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 1 Processor -- start 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 1 -- start start 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient: init 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:17.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 0x7f3428132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-a con 0x7f34280980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34280a58b0 0x7f3428133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.072+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-b con 0x7f34280a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): picked mon.noname-c con 0x7f3428136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34280a32a0 con 0x7f34280980a0 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428134520 con 0x7f34280a58b0 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3428135ad0 con 0x7f3428136dd0 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:17.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342fe34640 10 monclient(hunting): authenticate will time out at 1277.069051s 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 0x7f3428132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 0x7f3428132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38060/0 (socket says 172.21.15.17:38060) 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 -- 172.21.15.17:0/1243010706 learned_addr learned my addr 172.21.15.17:0/1243010706 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): get_auth_request con 0x7f34280980a0 auth_method 0 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): get_auth_request con 0x7f3428136dd0 auth_method 0 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34280a58b0 0x7f3428133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): get_auth_request con 0x7f34280a58b0 auth_method 0 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342e3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient(hunting): handle_auth_done global_id 104658 payload 824 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 msgr2=0x7f34281335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3428136dd0 0x7f34281335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34280a58b0 msgr2=0x7f3428133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34280a58b0 0x7f3428133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient: _finish_hunting 0 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 monclient: found mon.noname-a 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:17.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.073+0000 7f342dbd0640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420009a20 con 0x7f34280980a0 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342dbd0640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 msgr2=0x7f3428132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342dbd0640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 0x7f3428132b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3424035040 tx=0x7f3424032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240319c0 con 0x7f34280980a0 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: dump: 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.094 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424031b60 con 0x7f34280980a0 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342fe34640 5 monclient: authenticate success, global_id 104658 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342fe34640 10 monclient: _renew_subs 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342fe34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424009450 con 0x7f34280980a0 2024-09-15T02:00:17.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342403d420 con 0x7f34280980a0 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341dffb640 10 monclient: dump: 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.096 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 10 monclient: _renew_subs 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3428136790 con 0x7f34280980a0 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.075+0000 7f341ffff640 4 set_mon_vals no callback set 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 0x7f34281371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428063b20 con 0x7f3428133ae0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.076+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3424039040 con 0x7f34280980a0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 0x7f34281371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient: get_auth_request con 0x7f3428133ae0 auth_method 0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f341dffb640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f340c038740 0x7f340c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f342406c7f0 con 0x7f34280980a0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342e3d1640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f340c038740 0x7f340c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342e3d1640 10 monclient: get_auth_request con 0x7f340c038740 auth_method 0 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 10 monclient(hunting): handle_auth_done global_id 124138 payload 824 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.077+0000 7f342d3cf640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3428135e30 con 0x7f3428133ae0 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f342d3cf640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 msgr2=0x7f34281371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f342d3cf640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 0x7f34281371a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f341800e780 tx=0x7f3418006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f342e3d1640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f340c038740 0x7f340c03abf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f342000f700 tx=0x7f342000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341800cac0 con 0x7f3428133ae0 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f341dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f341dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.079+0000 7f341dffb640 10 monclient: dump: 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.099 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f341800cc20 con 0x7f3428133ae0 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 msgr2=0x7f34281371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 0x7f34281371a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f341800e780 tx=0x7f3418006f50 comp rx=0 tx=0).stop 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 msgr2=0x7f34281371a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:17.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.080+0000 7f341dffb640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3428133ae0 0x7f34281371a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342fe34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342fe34640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 0x7f3428137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342fe34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428098e80 con 0x7f34281376e0 2024-09-15T02:00:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 0x7f3428137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient: get_auth_request con 0x7f34281376e0 auth_method 0 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 10 monclient(hunting): handle_auth_done global_id 124141 payload 824 2024-09-15T02:00:17.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.086+0000 7f342d3cf640 1 -- 172.21.15.17:0/1243010706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f34280724a0 con 0x7f34281376e0 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342d3cf640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 msgr2=0x7f3428137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342d3cf640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 0x7f3428137ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3418009fd0 tx=0x7f3418015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341801cc30 con 0x7f34281376e0 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 10 monclient: dump: 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.104 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f341800c8b0 con 0x7f34281376e0 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 msgr2=0x7f3428137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 0x7f3428137ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3418009fd0 tx=0x7f3418015000 comp rx=0 tx=0).stop 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 msgr2=0x7f3428137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:17.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f341dffb640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34281376e0 0x7f3428137ae0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3418009fd0 tx=0x7f3418015000 comp rx=0 tx=0).stop 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 102, 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:17.106 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:17.107 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:17.108 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:17.109 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:17.110 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:17.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:17.112 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:17.114 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:17.115 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:17.116 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:17.117 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:17.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:17.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:17.119 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:17.119 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:17.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f340c038740 msgr2=0x7f340c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f340c038740 0x7f340c03abf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f342000f700 tx=0x7f342000b040 comp rx=0 tx=0).stop 2024-09-15T02:00:17.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 10 monclient: shutdown 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 msgr2=0x7f3428132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 --2- 172.21.15.17:0/1243010706 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f34280980a0 0x7f3428132b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3424035040 tx=0x7f3424032000 comp rx=0 tx=0).stop 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342e3d1640 1 -- 172.21.15.17:0/1243010706 reap_dead start 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342e3d1640 1 -- 172.21.15.17:0/1243010706 reap_dead start 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 shutdown_connections 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 >> 172.21.15.17:0/1243010706 conn(0x7f3428070ed0 msgr2=0x7f3428068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.089+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 shutdown_connections 2024-09-15T02:00:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.090+0000 7f342fe34640 1 -- 172.21.15.17:0/1243010706 wait complete. 2024-09-15T02:00:17.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:00:17.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T02:00:17.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:00:17.122 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T02:00:17.122 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:00:17.123 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:00:17.123 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:00:17.138 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:00:17.138 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:00:17.139 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.271+0000 7fbe6739d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.271+0000 7fbe6739d640 10 monclient: build_initial_monmap 2024-09-15T02:00:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.271+0000 7fbe6739d640 10 monclient: monmap: 2024-09-15T02:00:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:17.272628+0000 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:17.272628+0000 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.272+0000 7fbe6739d640 1 Processor -- start 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.272+0000 7fbe6739d640 1 -- start start 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.272+0000 7fbe6739d640 10 monclient: init 2024-09-15T02:00:17.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.272+0000 7fbe6739d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe6006eb30 0x7fbe6006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6006eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 0x7fbe6009bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6006f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe6009c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6009f6a0 con 0x7fbe6009c4d0 2024-09-15T02:00:17.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe60069f40 con 0x7fbe6006eb30 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60069950 con 0x7fbe6006f440 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe6739d640 10 monclient(hunting): authenticate will time out at 1277.269094s 2024-09-15T02:00:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe64938640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 msgr2=0x7fbe6009bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe64938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 0x7fbe6009bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe6006eb30 0x7fbe6006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe6006eb30 0x7fbe6006ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38114/0 (socket says 172.21.15.17:38114) 2024-09-15T02:00:17.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 -- 172.21.15.17:0/2893005978 learned_addr learned my addr 172.21.15.17:0/2893005978 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:17.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): get_auth_request con 0x7fbe6006eb30 auth_method 0 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient(hunting): handle_auth_done global_id 104661 payload 274 2024-09-15T02:00:17.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 -- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 msgr2=0x7fbe6009bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 --2- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 0x7fbe6009bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 -- 172.21.15.17:0/2893005978 >> v1:172.21.15.67:6789/0 conn(0x7fbe6009c4d0 legacy=0x7fbe6009d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient: _finish_hunting 0 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 monclient: found mon.noname-a 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.273+0000 7fbe65139640 1 -- 172.21.15.17:0/2893005978 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6009ad80 con 0x7fbe6006eb30 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe65139640 1 -- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 msgr2=0x7fbe6006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe65139640 1 --2- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe6006ef00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009910 tx=0x7fbe5c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1b0a5e80233093 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 1 -- 172.21.15.17:0/2893005978 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c032440 con 0x7fbe6006eb30 2024-09-15T02:00:17.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: dump: 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:17.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 5 monclient: authenticate success, global_id 104661 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 1 -- 172.21.15.17:0/2893005978 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c0325a0 con 0x7fbe6006eb30 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 1 -- 172.21.15.17:0/2893005978 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c032940 con 0x7fbe6006eb30 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 4 set_mon_vals no callback set 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe57fff640 10 monclient: dump: 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.300 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 10 monclient: shutdown 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 1 -- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 msgr2=0x7fbe6006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.276+0000 7fbe6739d640 1 --2- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe6006ef00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009910 tx=0x7fbe5c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 -- 172.21.15.17:0/2893005978 shutdown_connections 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 --2- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6006f440 0x7fbe6009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 --2- 172.21.15.17:0/2893005978 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe6006ef00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 -- 172.21.15.17:0/2893005978 >> 172.21.15.17:0/2893005978 conn(0x7fbe600712f0 msgr2=0x7fbe600a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 -- 172.21.15.17:0/2893005978 shutdown_connections 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 -- 172.21.15.17:0/2893005978 wait complete. 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 10 monclient: build_initial_monmap 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 10 monclient: monmap: 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:17.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.277+0000 7fbe6739d640 1 Processor -- start 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 -- start start 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient: init 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:17.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6006eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe6006f440 0x7fbe60133d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6006f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 0x7fbe60134260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe6009c4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6009cd10 con 0x7fbe6006eb30 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60121750 con 0x7fbe6006f440 2024-09-15T02:00:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60134ba0 con 0x7fbe6009c4d0 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6739d640 10 monclient(hunting): authenticate will time out at 1277.274681s 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6593a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 msgr2=0x7fbe60134260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe6593a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 0x7fbe60134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38122/0 (socket says 172.21.15.17:38122) 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 -- 172.21.15.17:0/1685275120 learned_addr learned my addr 172.21.15.17:0/1685275120 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): get_auth_request con 0x7fbe6006eb30 auth_method 0 2024-09-15T02:00:17.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe64938640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe6006f440 0x7fbe60133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe64938640 10 monclient(hunting): get_auth_request con 0x7fbe6006f440 auth_method 0 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe64938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe64938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe64938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient(hunting): handle_auth_done global_id 104664 payload 824 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 -- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 msgr2=0x7fbe60134260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 0x7fbe60134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 -- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe6006f440 msgr2=0x7fbe60133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe6006f440 0x7fbe60133d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient: _finish_hunting 0 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 1 monclient: found mon.noname-a 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.278+0000 7fbe65139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.279+0000 7fbe65139640 1 -- 172.21.15.17:0/1685275120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c009450 con 0x7fbe6006eb30 2024-09-15T02:00:17.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe65139640 1 -- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 msgr2=0x7fbe601410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe65139640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c035040 tx=0x7fbe5c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c03c960 con 0x7fbe6006eb30 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: dump: 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.308 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 5 monclient: authenticate success, global_id 104664 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c03cae0 con 0x7fbe6006eb30 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:17.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 10 monclient: _renew_subs 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c03cea0 con 0x7fbe6006eb30 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6011f310 con 0x7fbe6006eb30 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe557fa640 10 monclient: dump: 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:17.310 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe577fe640 4 set_mon_vals no callback set 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 10 monclient: _renew_subs 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.281+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe60121120 con 0x7fbe6006eb30 2024-09-15T02:00:17.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe5c039040 con 0x7fbe6006eb30 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe6739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe6739d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe6739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe600724a0 con 0x7fbe6006eb30 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe557fa640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 0x7fbe4403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7fbe5c06ced0 con 0x7fbe6006eb30 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe64938640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 0x7fbe4403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.282+0000 7fbe64938640 10 monclient: get_auth_request con 0x7fbe440386f0 auth_method 0 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.285+0000 7fbe64938640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 0x7fbe4403aba0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbe60099ab0 tx=0x7fbe50006ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:17.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:17.287+0000 7fbe65139640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c035040 tx=0x7fbe5c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fbe5c042c90 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T02:00:18.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.115+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbe5c042c90 con 0x7fbe6006eb30 2024-09-15T02:00:18.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.115+0000 7fbe557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:18.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.115+0000 7fbe557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe6739d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe6739d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe6739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe60121520 con 0x7fbe6006eb30 2024-09-15T02:00:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe557fa640 1 -- 172.21.15.17:0/1685275120 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fbe5c040080 con 0x7fbe6006eb30 2024-09-15T02:00:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.346+0000 7fbe557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:18.349 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 msgr2=0x7fbe4403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 0x7fbe4403aba0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbe60099ab0 tx=0x7fbe50006ef0 comp rx=0 tx=0).stop 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 10 monclient: shutdown 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 msgr2=0x7fbe601410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c035040 tx=0x7fbe5c032000 comp rx=0 tx=0).stop 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 shutdown_connections 2024-09-15T02:00:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe440386f0 0x7fbe4403aba0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe6009c4d0 0x7fbe60134260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe6006f440 0x7fbe60133d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 --2- 172.21.15.17:0/1685275120 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe6006eb30 0x7fbe601410c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 >> 172.21.15.17:0/1685275120 conn(0x7fbe600712f0 msgr2=0x7fbe600a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 shutdown_connections 2024-09-15T02:00:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:18.348+0000 7fbe6739d640 1 -- 172.21.15.17:0/1685275120 wait complete. 2024-09-15T02:00:19.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:19.574+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:19.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:19.574+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:19.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:19.574+0000 7fc3867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:20.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:20.574+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:20.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:20.574+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:21.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.468+0000 7f16cd460640 10 monclient: get_monmap_and_config 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.468+0000 7f16cd460640 10 monclient: build_initial_monmap 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.468+0000 7f16cd460640 10 monclient: monmap: 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:21.469125+0000 2024-09-15T02:00:21.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:21.469125+0000 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.468+0000 7f16cd460640 1 Processor -- start 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 -- start start 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient: init 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T02:00:21.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-b con 0x7f16c80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16c809ccf0 0x7f16c8069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-c con 0x7f16c809ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16c8069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c80a5b50 con 0x7f16c8069ac0 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c80a18b0 con 0x7f16c809ccf0 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c809c5f0 con 0x7f16c80a5780 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16cd460640 10 monclient(hunting): authenticate will time out at 1281.465575s 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c6ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 msgr2=0x7f16c809bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:21.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c77fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f16c8069ac0 0x7f16c8138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59194/0 (socket says 172.21.15.17:59194) 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c77fe640 1 -- 172.21.15.17:0/3570580633 learned_addr learned my addr 172.21.15.17:0/3570580633 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c67fc640 1 --2- 172.21.15.17:0/3570580633 >> v2:172.21.15.67:3300/0 conn(0x7f16c809ccf0 0x7f16c8069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c67fc640 10 monclient(hunting): get_auth_request con 0x7f16c809ccf0 auth_method 0 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.469+0000 7f16c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient(hunting): handle_auth_done global_id 124147 payload 274 2024-09-15T02:00:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 1 -- 172.21.15.17:0/3570580633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 msgr2=0x7f16c809bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 1 --2- 172.21.15.17:0/3570580633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c809bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 1 -- 172.21.15.17:0/3570580633 >> v1:172.21.15.17:6789/0 conn(0x7f16c8069ac0 legacy=0x7f16c8138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 1 monclient: found mon.noname-c 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.470+0000 7f16c67fc640 1 -- 172.21.15.17:0/3570580633 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c809d210 con 0x7f16c809ccf0 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c67fc640 1 -- 172.21.15.17:0/3570580633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 msgr2=0x7f16c8069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c67fc640 1 --2- 172.21.15.17:0/3570580633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c8069580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009850 tx=0x7f16b00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bbcc4515d083b3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 1 -- 172.21.15.17:0/3570580633 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0032440 con 0x7f16c809ccf0 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:21.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: dump: 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:21.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 5 monclient: authenticate success, global_id 124147 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 1 -- 172.21.15.17:0/3570580633 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00325a0 con 0x7f16c809ccf0 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 1 -- 172.21.15.17:0/3570580633 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0032940 con 0x7f16c809ccf0 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 4 set_mon_vals no callback set 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16c5ffb640 10 monclient: dump: 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.487 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 10 monclient: shutdown 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 1 -- 172.21.15.17:0/3570580633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 msgr2=0x7f16c8069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.473+0000 7f16cd460640 1 --2- 172.21.15.17:0/3570580633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c8069580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009850 tx=0x7f16b00090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 -- 172.21.15.17:0/3570580633 shutdown_connections 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 --2- 172.21.15.17:0/3570580633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c8069580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 --2- 172.21.15.17:0/3570580633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c809bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 -- 172.21.15.17:0/3570580633 >> 172.21.15.17:0/3570580633 conn(0x7f16c8070e10 msgr2=0x7f16c80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 -- 172.21.15.17:0/3570580633 shutdown_connections 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 1 -- 172.21.15.17:0/3570580633 wait complete. 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 10 monclient: build_initial_monmap 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.474+0000 7f16cd460640 10 monclient: monmap: 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:21.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 Processor -- start 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 -- start start 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient: init 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-a con 0x7f16c8069ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c813d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-b con 0x7f16c809ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c8141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): picked mon.noname-c con 0x7f16c80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:21.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8120df0 con 0x7f16c8069ac0 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8122ae0 con 0x7f16c809ccf0 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c813eba0 con 0x7f16c80a5780 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16cd460640 10 monclient(hunting): authenticate will time out at 1281.471685s 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c77fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 msgr2=0x7f16c8141620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c8141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38126/0 (socket says 172.21.15.17:38126) 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 1 -- 172.21.15.17:0/91990004 learned_addr learned my addr 172.21.15.17:0/91990004 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:21.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): get_auth_request con 0x7f16c8069ac0 auth_method 0 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c67fc640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c67fc640 10 monclient(hunting): get_auth_request con 0x7f16c809ccf0 auth_method 0 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.475+0000 7f16c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:21.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 10 monclient(hunting): handle_auth_done global_id 104667 payload 824 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 -- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 msgr2=0x7f16c8141620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c8141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 -- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 msgr2=0x7f16c813d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c813d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 monclient: found mon.noname-a 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.476+0000 7f16c6ffd640 1 -- 172.21.15.17:0/91990004 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0009390 con 0x7f16c8069ac0 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16c6ffd640 1 -- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 msgr2=0x7f16c813e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:21.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16c6ffd640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f16b800b600 tx=0x7f16b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8015070 con 0x7f16c8069ac0 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: dump: 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:21.494 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 5 monclient: authenticate success, global_id 104667 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8003d90 con 0x7f16c8069ac0 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 10 monclient: _renew_subs 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8006380 con 0x7f16c8069ac0 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c8122800 con 0x7f16c8069ac0 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16beffd640 10 monclient: dump: 2024-09-15T02:00:21.495 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16c4ff9640 4 set_mon_vals no callback set 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 10 monclient: _renew_subs 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:21.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.478+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c813de40 con 0x7f16c8069ac0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f16b8010440 con 0x7f16c8069ac0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16cd460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16cd460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16cd460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c81230f0 con 0x7f16c8069ac0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16beffd640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 0x7f16a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f16b804c8f0 con 0x7f16c8069ac0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.479+0000 7f16c67fc640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 0x7f16a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:21.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.480+0000 7f16c67fc640 10 monclient: get_auth_request con 0x7f16a80386f0 auth_method 0 2024-09-15T02:00:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.482+0000 7f16c67fc640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 0x7f16a803aba0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f16c80a0c50 tx=0x7f16b0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.484+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f16b8018a70 con 0x7f16c8069ac0 2024-09-15T02:00:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.484+0000 7f16beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:21.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.484+0000 7f16beffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:21.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:21.574+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:21.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:21.574+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16cd460640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16cd460640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16cd460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f16c8122ca0 con 0x7f16c8069ac0 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16beffd640 1 -- 172.21.15.17:0/91990004 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f16b8018a70 con 0x7f16c8069ac0 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.712+0000 7f16beffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:21.715 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 msgr2=0x7f16a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 0x7f16a803aba0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f16c80a0c50 tx=0x7f16b0032000 comp rx=0 tx=0).stop 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 10 monclient: shutdown 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 msgr2=0x7f16c813e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f16b800b600 tx=0x7f16b80091a0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 shutdown_connections 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16a80386f0 0x7f16a803aba0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16c80a5780 0x7f16c8141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c809ccf0 0x7f16c813d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 --2- 172.21.15.17:0/91990004 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16c8069ac0 0x7f16c813e0b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 >> 172.21.15.17:0/91990004 conn(0x7f16c8070e10 msgr2=0x7f16c809b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 shutdown_connections 2024-09-15T02:00:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:21.714+0000 7f16cd460640 1 -- 172.21.15.17:0/91990004 wait complete. 2024-09-15T02:00:22.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:22.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:22.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:22.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:23.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:23.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:23.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:23.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:24.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:24.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:24.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:24.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:24.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 10 monclient: get_monmap_and_config 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 10 monclient: build_initial_monmap 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 10 monclient: monmap: 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:24.834947+0000 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:24.834947+0000 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:24.837 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 1 Processor -- start 2024-09-15T02:00:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 1 -- start start 2024-09-15T02:00:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.834+0000 7f9ad88d4640 10 monclient: init 2024-09-15T02:00:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:24.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 0x7f9ad409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad4131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ad40a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ad409f1e0 0x7f9ad4138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad409f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40a0040 con 0x7f9ad40a0280 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad409bff0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40a14a0 con 0x7f9ad4131b20 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad88d4640 10 monclient(hunting): authenticate will time out at 1284.831502s 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad259c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 msgr2=0x7f9ad409eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 0x7f9ad409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad1d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9ad40a0280 0x7f9ad4131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60918/0 (socket says 172.21.15.17:60918) 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad1d9b640 1 -- 172.21.15.17:0/3226048437 learned_addr learned my addr 172.21.15.17:0/3226048437 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad2d9d640 1 --2- 172.21.15.17:0/3226048437 >> v2:172.21.15.67:3300/0 conn(0x7f9ad409f1e0 0x7f9ad4138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad2d9d640 10 monclient(hunting): get_auth_request con 0x7f9ad409f1e0 auth_method 0 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.835+0000 7f9ad2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:24.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_done global_id 124153 payload 274 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 1 -- 172.21.15.17:0/3226048437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 msgr2=0x7f9ad409eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 1 --2- 172.21.15.17:0/3226048437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 0x7f9ad409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 1 -- 172.21.15.17:0/3226048437 >> v1:172.21.15.17:6789/0 conn(0x7f9ad40a0280 legacy=0x7f9ad4131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 1 monclient: found mon.noname-c 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:24.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.836+0000 7f9ad2d9d640 1 -- 172.21.15.17:0/3226048437 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4132600 con 0x7f9ad409f1e0 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad2d9d640 1 -- 172.21.15.17:0/3226048437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 msgr2=0x7f9ad4138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad2d9d640 1 --2- 172.21.15.17:0/3226048437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad4138e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009e60 tx=0x7f9acc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6211292a210810fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 1 -- 172.21.15.17:0/3226048437 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc007840 con 0x7f9ad409f1e0 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: dump: 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:24.846 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: _finish_auth 0 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 1 -- 172.21.15.17:0/3226048437 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc02d050 con 0x7f9ad409f1e0 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 1 -- 172.21.15.17:0/3226048437 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc007bd0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad159a640 10 monclient: dump: 2024-09-15T02:00:24.847 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:24.848 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad88d4640 5 monclient: authenticate success, global_id 124153 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad88d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.839+0000 7f9ad88d4640 4 set_mon_vals no callback set 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 10 monclient: shutdown 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 -- 172.21.15.17:0/3226048437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 msgr2=0x7f9ad4138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/3226048437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad4138e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009e60 tx=0x7f9acc009780 comp rx=0 tx=0).stop 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 -- 172.21.15.17:0/3226048437 shutdown_connections 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/3226048437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad4138e60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/3226048437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad4131b20 0x7f9ad409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 -- 172.21.15.17:0/3226048437 >> 172.21.15.17:0/3226048437 conn(0x7f9ad4071270 msgr2=0x7f9ad40a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 -- 172.21.15.17:0/3226048437 shutdown_connections 2024-09-15T02:00:24.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 1 -- 172.21.15.17:0/3226048437 wait complete. 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 10 monclient: build_initial_monmap 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.840+0000 7f9ad88d4640 10 monclient: monmap: 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 Processor -- start 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 -- start start 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient: init 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad409b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad409f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 0x7f9ad4132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad40a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad4131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41363f0 con 0x7f9ad4131b20 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:24.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad4137bc0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40a09b0 con 0x7f9ad40a0280 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad88d4640 10 monclient(hunting): authenticate will time out at 1284.837776s 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad1d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 msgr2=0x7f9ad4132970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad1d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 0x7f9ad4132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad409b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52118/0 (socket says 172.21.15.17:52118) 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 1 -- 172.21.15.17:0/1264662766 learned_addr learned my addr 172.21.15.17:0/1264662766 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 10 monclient(hunting): get_auth_request con 0x7f9ad4131b20 auth_method 0 2024-09-15T02:00:24.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad259c640 10 monclient(hunting): get_auth_request con 0x7f9ad409f1e0 auth_method 0 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.841+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient(hunting): handle_auth_done global_id 124156 payload 824 2024-09-15T02:00:24.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 -- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 msgr2=0x7f9ad4132970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 0x7f9ad4132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 -- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 msgr2=0x7f9ad4132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 monclient: found mon.noname-b 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad259c640 1 -- 172.21.15.17:0/1264662766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc009a20 con 0x7f9ad409f1e0 2024-09-15T02:00:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.842+0000 7f9ad2d9d640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad259c640 1 -- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 msgr2=0x7f9ad409b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad259c640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad409b230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400b5e0 tx=0x7f9ac40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4003d90 con 0x7f9ad409f1e0 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: dump: 2024-09-15T02:00:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac4006150 con 0x7f9ad409f1e0 2024-09-15T02:00:24.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 5 monclient: authenticate success, global_id 124156 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 10 monclient: _renew_subs 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad4134080 con 0x7f9ad409f1e0 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4006510 con 0x7f9ad409f1e0 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad0d99640 4 set_mon_vals no callback set 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ac27fc640 10 monclient: dump: 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 10 monclient: _renew_subs 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.844+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad4133b70 con 0x7f9ad409f1e0 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.845+0000 7f9ad88d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.845+0000 7f9ad88d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.845+0000 7f9ad88d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.845+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad40724a0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.846+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ac40103e0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.846+0000 7f9ac27fc640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 0x7f9ab403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.846+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(241..241 src has 148..241) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f9ac404c510 con 0x7f9ad409f1e0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.846+0000 7f9ad1d9b640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 0x7f9ab403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.846+0000 7f9ad1d9b640 10 monclient: get_auth_request con 0x7f9ab40386f0 auth_method 0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.849+0000 7f9ad1d9b640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 0x7f9ab403aba0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9ad409ba00 tx=0x7f9ac8007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.851+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ac400eac0 con 0x7f9ad409f1e0 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.851+0000 7f9ac27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:24.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:24.851+0000 7f9ac27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.076+0000 7f9ad88d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:25.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.076+0000 7f9ad88d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.076+0000 7f9ad88d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.076+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9ad406efc0 con 0x7f9ad409f1e0 2024-09-15T02:00:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.077+0000 7f9ac27fc640 1 -- 172.21.15.17:0/1264662766 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f9ac4017070 con 0x7f9ad409f1e0 2024-09-15T02:00:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.077+0000 7f9ac27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:25.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.077+0000 7f9ac27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:25.080 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 msgr2=0x7f9ab403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 0x7f9ab403aba0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9ad409ba00 tx=0x7f9ac8007b80 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 10 monclient: shutdown 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 msgr2=0x7f9ad409b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad409b230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400b5e0 tx=0x7f9ac40090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 shutdown_connections 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9ab40386f0 0x7f9ab403aba0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ad4131b20 0x7f9ad4132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9ad40a0280 0x7f9ad4132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 --2- 172.21.15.17:0/1264662766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ad409f1e0 0x7f9ad409b230 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:25.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 >> 172.21.15.17:0/1264662766 conn(0x7f9ad4071270 msgr2=0x7f9ad40a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:25.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 shutdown_connections 2024-09-15T02:00:25.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:25.079+0000 7f9ad88d4640 1 -- 172.21.15.17:0/1264662766 wait complete. 2024-09-15T02:00:25.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:25.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:25.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:25.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:26.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:26.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:26.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:26.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:27.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:27.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:27.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:27.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:28.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:28.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:28.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 10 monclient: monmap: 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:28.200889+0000 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:28.200889+0000 2024-09-15T02:00:28.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:28.208 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 1 Processor -- start 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 1 -- start start 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.200+0000 7ff245f9d640 10 monclient: init 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:28.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2400a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff240132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff240136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400a01e0 con 0x7ff2400a58b0 2024-09-15T02:00:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2400a1580 con 0x7ff240132040 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400a5c80 con 0x7ff240136dd0 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff245f9d640 10 monclient(hunting): authenticate will time out at 1288.197425s 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 msgr2=0x7ff2401391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:28.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52130/0 (socket says 172.21.15.17:52130) 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 learned_addr learned my addr 172.21.15.17:0/2912263050 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff240132040 auth_method 0 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.201+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:28.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.203+0000 7ff23f7fe640 1 -- 172.21.15.17:0/2912263050 >> v1:172.21.15.67:6789/0 conn(0x7ff2400a58b0 legacy=0x7ff2400980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:28.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23ffff640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 msgr2=0x7ff2401391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23ffff640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff240132040 auth_method 0 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:00:28.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:28.576+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:28.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:28.576+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23ffff640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 msgr2=0x7ff2401391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:28.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23ffff640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff240132040 auth_method 0 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:28.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:28.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:28.802+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:00:29.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:29.575+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:29.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:29.575+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:29.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:29.575+0000 7fc3867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23ffff640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 msgr2=0x7ff2401391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23ffff640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:00:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff240132040 auth_method 0 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:29.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:29.603+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:00:30.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:30.576+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:30.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:30.576+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): tick 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 monclient(hunting): continuing hunt 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 msgr2=0x7ff2401391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff240136dd0 0x7ff2401391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 msgr2=0x7ff2400987b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:31.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.17:3300/0 conn(0x7ff240132040 0x7ff2400987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 -- 172.21.15.17:0/2912263050 >> v1:172.21.15.67:6789/0 conn(0x7ff2400a58b0 legacy=0x7ff2400980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff22c003720 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff22c007440 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 -- 172.21.15.17:0/2912263050 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff240132440 con 0x7ff22c003720 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 1 -- 172.21.15.17:0/2912263050 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2400a1580 con 0x7ff22c007440 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23d7fa640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff22c007440 auth_method 0 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:31.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.201+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:31.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.203+0000 7ff23f7fe640 1 -- 172.21.15.17:0/2912263050 >> v1:172.21.15.17:6789/0 conn(0x7ff22c003720 legacy=0x7ff22c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:31.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff22c007440 auth_method 0 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:31.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.402+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:00:31.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:31.576+0000 7fc3867fc640 10 monclient: tick 2024-09-15T02:00:31.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:31.576+0000 7fc3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:31.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:31.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff22c007440 auth_method 0 2024-09-15T02:00:31.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.803+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:00:31.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.992+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226897135 0 0) 0x7ff240132440 con 0x7ff22c003720 2024-09-15T02:00:31.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.992+0000 7ff23e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:31.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.992+0000 7ff23e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:31.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.992+0000 7ff23e7fc640 10 monclient(hunting): my global_id is 104673 2024-09-15T02:00:31.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.992+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2280034e0 con 0x7ff22c003720 2024-09-15T02:00:31.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:31.993+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc398040090 con 0x7fc3a00a0370 2024-09-15T02:00:31.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:31.993+0000 7fc3877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:31.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:31.993+0000 7fc3877fe640 10 monclient: _finish_command 2 = system:0 device 'h25' does not appear in the crush map 2024-09-15T02:00:32.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1630091731 0 0) 0x7ff2280034e0 con 0x7ff22c003720 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 msgr2=0x7ff22c0098f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 --2- 172.21.15.17:0/2912263050 >> v2:172.21.15.67:3300/0 conn(0x7ff22c007440 0x7ff22c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c00b0c0 con 0x7ff22c003720 2024-09-15T02:00:32.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7ff230001730 con 0x7ff22c003720 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: dump: 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:00:32.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 >> v1:172.21.15.17:6789/0 conn(0x7ff22c003720 legacy=0x7ff22c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23ffff640 1 -- 172.21.15.17:0/2912263050 reap_dead start 2024-09-15T02:00:32.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff228003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient(hunting): picked mon.b con 0x7ff2280039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 0x7ff228004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient(hunting): picked mon.a con 0x7ff228004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 0x7ff22800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient(hunting): picked mon.c con 0x7ff228005180 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.995+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400a1580 con 0x7ff228004780 2024-09-15T02:00:32.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22800c120 con 0x7ff2280039d0 2024-09-15T02:00:32.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22800c2e0 con 0x7ff228005180 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23ffff640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 msgr2=0x7ff22800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23ffff640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 0x7ff22800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 0x7ff228004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff228003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): get_auth_request con 0x7ff228004780 auth_method 0 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): get_auth_request con 0x7ff2280039d0 auth_method 0 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient(hunting): handle_auth_done global_id 104673 payload 293 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 msgr2=0x7ff22800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 0x7ff22800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 msgr2=0x7ff228003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff228003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 monclient: found mon.a 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280034e0 con 0x7ff228004780 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff23effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff245f9d640 5 monclient: authenticate success, global_id 104673 2024-09-15T02:00:32.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.996+0000 7ff245f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23effd640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 msgr2=0x7ff228004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23effd640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 0x7ff228004bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff234006f60 tx=0x7ff234008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f17afc61be2d8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23402c680 con 0x7ff228004780 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: dump: 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234030070 con 0x7ff228004780 2024-09-15T02:00:32.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 1 -- 172.21.15.17:0/2912263050 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23402ca70 con 0x7ff228004780 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff23e7fc640 10 monclient: dump: 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.021 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff245f9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:31.999+0000 7ff245f9d640 4 set_mon_vals no callback set 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 10 monclient: shutdown 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 -- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 msgr2=0x7ff228004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 0x7ff228004bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff234006f60 tx=0x7ff234008040 comp rx=0 tx=0).stop 2024-09-15T02:00:32.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 -- 172.21.15.17:0/2912263050 shutdown_connections 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228004780 0x7ff228004bf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228005180 0x7ff22800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 --2- 172.21.15.17:0/2912263050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff228003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 -- 172.21.15.17:0/2912263050 >> 172.21.15.17:0/2912263050 conn(0x7ff240070ed0 msgr2=0x7ff2400a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 -- 172.21.15.17:0/2912263050 shutdown_connections 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.000+0000 7ff245f9d640 1 -- 172.21.15.17:0/2912263050 wait complete. 2024-09-15T02:00:32.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.001+0000 7ff245f9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.001+0000 7ff245f9d640 10 monclient: monmap: 2024-09-15T02:00:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:32.025 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.025 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.025 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:32.025 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:32.025 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 Processor -- start 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 -- start start 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient: init 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:32.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff240130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff2280039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:32.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 0x7ff2401348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff228004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff228005180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:32.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24011bcd0 con 0x7ff228005180 2024-09-15T02:00:32.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24011ce10 con 0x7ff2280039d0 2024-09-15T02:00:32.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24011d0f0 con 0x7ff228004780 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff245f9d640 10 monclient(hunting): authenticate will time out at 1291.999715s 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23effd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 msgr2=0x7ff2401348e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 0x7ff2401348e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff240130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52166/0 (socket says 172.21.15.17:52166) 2024-09-15T02:00:32.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 1 -- 172.21.15.17:0/1360357131 learned_addr learned my addr 172.21.15.17:0/1360357131 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 10 monclient(hunting): get_auth_request con 0x7ff228005180 auth_method 0 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23f7fe640 10 monclient(hunting): get_auth_request con 0x7ff2280039d0 auth_method 0 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.002+0000 7ff23f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient(hunting): handle_auth_done global_id 104676 payload 824 2024-09-15T02:00:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 -- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 msgr2=0x7ff2401348e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 0x7ff2401348e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 -- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 msgr2=0x7ff240130a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff240130a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 monclient: found mon.noname-a 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.003+0000 7ff23ffff640 1 -- 172.21.15.17:0/1360357131 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24011d3d0 con 0x7ff228005180 2024-09-15T02:00:32.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff23ffff640 1 -- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 msgr2=0x7ff240134e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff23ffff640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0073b0 tx=0x7ff238009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23800bea0 con 0x7ff228005180 2024-09-15T02:00:32.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:32.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: dump: 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.037 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23800c040 con 0x7ff228005180 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 5 monclient: authenticate success, global_id 104676 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 10 monclient: _renew_subs 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24011c7d0 con 0x7ff228005180 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23800e700 con 0x7ff228005180 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff2277fe640 10 monclient: dump: 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 10 monclient: _renew_subs 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff23dffb640 4 set_mon_vals no callback set 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.005+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24011b970 con 0x7ff228005180 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff23800eab0 con 0x7ff228005180 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff245f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff245f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff245f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff240136410 con 0x7ff228005180 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff2277fe640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 0x7ff22c03b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(242..242 src has 148..242) v4 ==== 9668+0+0 (secure 0 0 0) 0x7ff23804c030 con 0x7ff228005180 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff23f7fe640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 0x7ff22c03b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.007+0000 7ff23f7fe640 10 monclient: get_auth_request con 0x7ff22c038ec0 auth_method 0 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.010+0000 7ff23f7fe640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 0x7ff22c03b370 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff230000e50 tx=0x7ff230009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.012+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff23807de40 con 0x7ff228005180 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.012+0000 7ff2277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.012+0000 7ff2277fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.099+0000 7fc3a5f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.099+0000 7fc3a5f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.099+0000 7fc3a5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.099+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7fc3a006f000 con 0x7fc3a00a0370 2024-09-15T02:00:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.100+0000 7fc3877fe640 1 -- 172.21.15.67:0/2318160625 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc39803a780 con 0x7fc3a00a0370 2024-09-15T02:00:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.100+0000 7fc3877fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T02:00:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.100+0000 7fc3877fe640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-15T02:00:32.103 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h25' does not appear in the crush map 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 msgr2=0x7fc38c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 0x7fc38c03abf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc390007fd0 tx=0x7fc390007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 10 monclient: shutdown 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 msgr2=0x7fc3a0130210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc398009ab0 tx=0x7fc398036000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 shutdown_connections 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc38c038740 0x7fc38c03abf0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc3a0098c20 0x7fc3a012fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a00a0370 0x7fc3a0130210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 --2- 172.21.15.67:0/2318160625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc3a0098140 0x7fc3a0139780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 >> 172.21.15.67:0/2318160625 conn(0x7fc3a00a5780 msgr2=0x7fc3a009ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 shutdown_connections 2024-09-15T02:00:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.102+0000 7fc3a5f9d640 1 -- 172.21.15.67:0/2318160625 wait complete. 2024-09-15T02:00:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h26 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.188+0000 7fb371a34640 10 monclient: get_monmap_and_config 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.188+0000 7fb371a34640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.188+0000 7fb371a34640 10 monclient: monmap: 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:32.190123+0000 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:32.190123+0000 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.191 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:32.192 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.189+0000 7fb371a34640 1 Processor -- start 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.189+0000 7fb371a34640 1 -- start start 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.189+0000 7fb371a34640 10 monclient: init 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb36c0a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb36c09eea0 0x7fb36c09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-c con 0x7fb36c09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 0x7fb36c139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-b con 0x7fb36c136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c06a270 con 0x7fb36c0a0b50 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c06f290 con 0x7fb36c09eea0 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c09f7e0 con 0x7fb36c136e40 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb371a34640 10 monclient(hunting): authenticate will time out at 1264.659115s 2024-09-15T02:00:32.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36b7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 msgr2=0x7fb36c139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 0x7fb36c139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb36c09eea0 0x7fb36c09f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb36c0a0b50 0x7fb36c069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47716/0 (socket says 172.21.15.67:47716) 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36affd640 1 -- 172.21.15.67:0/3554922311 learned_addr learned my addr 172.21.15.67:0/3554922311 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): get_auth_request con 0x7fb36c09eea0 auth_method 0 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_done global_id 124162 payload 274 2024-09-15T02:00:32.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 -- 172.21.15.67:0/3554922311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 msgr2=0x7fb36c139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:32.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 --2- 172.21.15.67:0/3554922311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 0x7fb36c139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 -- 172.21.15.67:0/3554922311 >> v1:172.21.15.67:6789/0 conn(0x7fb36c0a0b50 legacy=0x7fb36c069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 monclient: found mon.noname-c 2024-09-15T02:00:32.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.190+0000 7fb36a7fc640 1 -- 172.21.15.67:0/3554922311 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0a0f20 con 0x7fb36c09eea0 2024-09-15T02:00:32.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb36a7fc640 1 -- 172.21.15.67:0/3554922311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c09eea0 msgr2=0x7fb36c09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb36a7fc640 1 --2- 172.21.15.67:0/3554922311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c09eea0 0x7fb36c09f2a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb360009880 tx=0x7fb3600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1822610911980202 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 1 -- 172.21.15.67:0/3554922311 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360032440 con 0x7fb36c09eea0 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: dump: 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.205 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 1 -- 172.21.15.67:0/3554922311 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3600325a0 con 0x7fb36c09eea0 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 5 monclient: authenticate success, global_id 124162 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 1 -- 172.21.15.67:0/3554922311 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360032940 con 0x7fb36c09eea0 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 4 set_mon_vals no callback set 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb369ffb640 10 monclient: dump: 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 10 monclient: shutdown 2024-09-15T02:00:32.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 1 -- 172.21.15.67:0/3554922311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c09eea0 msgr2=0x7fb36c09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.193+0000 7fb371a34640 1 --2- 172.21.15.67:0/3554922311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c09eea0 0x7fb36c09f2a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb360009880 tx=0x7fb3600090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 -- 172.21.15.67:0/3554922311 shutdown_connections 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 --2- 172.21.15.67:0/3554922311 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c136e40 0x7fb36c139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 --2- 172.21.15.67:0/3554922311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c09eea0 0x7fb36c09f2a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 -- 172.21.15.67:0/3554922311 >> 172.21.15.67:0/3554922311 conn(0x7fb36c071290 msgr2=0x7fb36c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 -- 172.21.15.67:0/3554922311 shutdown_connections 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 -- 172.21.15.67:0/3554922311 wait complete. 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 10 monclient: monmap: 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:32.208 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.194+0000 7fb371a34640 1 Processor -- start 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 -- start start 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient: init 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:32.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 0x7fb36c134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-c con 0x7fb36c09eea0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-b con 0x7fb36c0a0b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb36c136e40 0x7fb36c11dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): picked mon.noname-a con 0x7fb36c136e40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c137720 con 0x7fb36c136e40 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0a12d0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0a3450 con 0x7fb36c09eea0 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb371a34640 10 monclient(hunting): authenticate will time out at 1264.664500s 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36affd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 msgr2=0x7fb36c134be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 0x7fb36c134be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37626/0 (socket says 172.21.15.67:37626) 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 -- 172.21.15.67:0/2489031968 learned_addr learned my addr 172.21.15.67:0/2489031968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): get_auth_request con 0x7fb36c0a0b50 auth_method 0 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb36c136e40 0x7fb36c11dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): get_auth_request con 0x7fb36c136e40 auth_method 0 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient(hunting): handle_auth_done global_id 124165 payload 824 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 -- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 msgr2=0x7fb36c134be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 0x7fb36c134be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 -- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb36c136e40 msgr2=0x7fb36c11dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb36c136e40 0x7fb36c11dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 monclient: found mon.noname-b 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.195+0000 7fb36a7fc640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb360009390 con 0x7fb36c0a0b50 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb36a7fc640 1 -- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 msgr2=0x7fb36c11d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb36a7fc640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb360009360 tx=0x7fb360032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3600318c0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: dump: 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:32.213 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb360031a20 con 0x7fb36c0a0b50 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 5 monclient: authenticate success, global_id 124165 2024-09-15T02:00:32.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 10 monclient: _renew_subs 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c120fd0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360031de0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb352ffd640 10 monclient: dump: 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 10 monclient: _renew_subs 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c142070 con 0x7fb36c0a0b50 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.198+0000 7fb3697fa640 4 set_mon_vals no callback set 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb360039050 con 0x7fb36c0a0b50 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb352ffd640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 0x7fb35803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 148..242) v4 ==== 9668+0+0 (secure 0 0 0) 0x7fb36006cdc0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb371a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb371a34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb36affd640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 0x7fb35803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb371a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c11feb0 con 0x7fb36c0a0b50 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.199+0000 7fb36affd640 10 monclient: get_auth_request con 0x7fb3580386f0 auth_method 0 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.201+0000 7fb36affd640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 0x7fb35803aba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0047d0 tx=0x7fb35c007af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.204+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb360040020 con 0x7fb36c0a0b50 2024-09-15T02:00:32.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.204+0000 7fb352ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.204+0000 7fb352ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.240+0000 7ff245f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.240+0000 7ff245f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.240+0000 7ff245f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.240+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff2400724a0 con 0x7ff228005180 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.241+0000 7ff2277fe640 1 -- 172.21.15.17:0/1360357131 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7ff238017020 con 0x7ff228005180 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.241+0000 7ff2277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:32.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.241+0000 7ff2277fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:32.245 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":104,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 msgr2=0x7ff22c03b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 0x7ff22c03b370 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff230000e50 tx=0x7ff230009040 comp rx=0 tx=0).stop 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 10 monclient: shutdown 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 msgr2=0x7ff240134e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0073b0 tx=0x7ff238009390 comp rx=0 tx=0).stop 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 shutdown_connections 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff22c038ec0 0x7ff22c03b370 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff228005180 0x7ff240134e20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff228004780 0x7ff2401348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 --2- 172.21.15.17:0/1360357131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280039d0 0x7ff240130a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 >> 172.21.15.17:0/1360357131 conn(0x7ff240070ed0 msgr2=0x7ff2400a4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 shutdown_connections 2024-09-15T02:00:32.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.243+0000 7ff245f9d640 1 -- 172.21.15.17:0/1360357131 wait complete. 2024-09-15T02:00:32.257 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:00:32.257 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:00:32.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:00:32.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.351+0000 7f63b288b640 10 monclient: get_monmap_and_config 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.351+0000 7f63b288b640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.351+0000 7f63b288b640 10 monclient: monmap: 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:32.353649+0000 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:32.353649+0000 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:32.356 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:32.356 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:32.356 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.352+0000 7f63b288b640 1 Processor -- start 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.352+0000 7f63b288b640 1 -- start start 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.352+0000 7f63b288b640 10 monclient: init 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.352+0000 7f63b288b640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.352+0000 7f63b288b640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63ac070a40 0x7f63ac09ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-c con 0x7f63ac070a40 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63ac06a550 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63ac137130 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac06eb80 con 0x7f63ac137130 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac09dd00 con 0x7f63ac06a550 2024-09-15T02:00:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac070f30 con 0x7f63ac070a40 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b288b640 10 monclient(hunting): authenticate will time out at 1292.350086s 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b0e28640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f63ac137130 0x7f63ac139510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60938/0 (socket says 172.21.15.17:60938) 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63b0e28640 1 -- 172.21.15.17:0/2322951887 learned_addr learned my addr 172.21.15.17:0/2322951887 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 1 --2- 172.21.15.17:0/2322951887 >> v2:172.21.15.67:3300/0 conn(0x7f63ac070a40 0x7f63ac09ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): get_auth_request con 0x7f63ac070a40 auth_method 0 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient(hunting): handle_auth_done global_id 124168 payload 274 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 1 -- 172.21.15.17:0/2322951887 >> v1:172.21.15.67:6789/0 conn(0x7f63ac06a550 legacy=0x7f63ac09b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 1 -- 172.21.15.17:0/2322951887 >> v1:172.21.15.17:6789/0 conn(0x7f63ac137130 legacy=0x7f63ac139510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 1 monclient: found mon.noname-c 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.353+0000 7f63abfff640 1 -- 172.21.15.17:0/2322951887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac09c470 con 0x7f63ac070a40 2024-09-15T02:00:32.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63abfff640 1 -- 172.21.15.17:0/2322951887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 msgr2=0x7f63ac09ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63abfff640 1 --2- 172.21.15.17:0/2322951887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac09ae20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f639c009850 tx=0x7f639c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2629a51c5787b889 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 1 -- 172.21.15.17:0/2322951887 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c032440 con 0x7f63ac070a40 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: dump: 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.369 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 1 -- 172.21.15.17:0/2322951887 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c0325a0 con 0x7f63ac070a40 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63b288b640 5 monclient: authenticate success, global_id 124168 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63b288b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63b288b640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 1 -- 172.21.15.17:0/2322951887 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c032940 con 0x7f63ac070a40 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63b288b640 4 set_mon_vals no callback set 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.356+0000 7f63aaffd640 10 monclient: dump: 2024-09-15T02:00:32.370 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 10 monclient: shutdown 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 -- 172.21.15.17:0/2322951887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 msgr2=0x7f63ac09ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 --2- 172.21.15.17:0/2322951887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac09ae20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f639c009850 tx=0x7f639c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 -- 172.21.15.17:0/2322951887 shutdown_connections 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 --2- 172.21.15.17:0/2322951887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac09ae20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 -- 172.21.15.17:0/2322951887 >> 172.21.15.17:0/2322951887 conn(0x7f63ac067ae0 msgr2=0x7f63ac0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 -- 172.21.15.17:0/2322951887 shutdown_connections 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 1 -- 172.21.15.17:0/2322951887 wait complete. 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.357+0000 7f63b288b640 10 monclient: monmap: 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.372 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 Processor -- start 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 -- start start 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient: init 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-a con 0x7f63ac06a550 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac11f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-b con 0x7f63ac070a40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63ac137130 0x7f63ac11f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): picked mon.noname-c con 0x7f63ac137130 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac06deb0 con 0x7f63ac06a550 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0a17d0 con 0x7f63ac070a40 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.358+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63b288b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac122000 con 0x7f63ac137130 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63b288b640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63b288b640 10 monclient(hunting): authenticate will time out at 1292.356011s 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63b0e28640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63ac137130 msgr2=0x7f63ac11f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63b0e28640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63ac137130 0x7f63ac11f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52170/0 (socket says 172.21.15.17:52170) 2024-09-15T02:00:32.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 -- 172.21.15.17:0/1093036482 learned_addr learned my addr 172.21.15.17:0/1093036482 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): get_auth_request con 0x7f63ac06a550 auth_method 0 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac11f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): get_auth_request con 0x7f63ac070a40 auth_method 0 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient(hunting): handle_auth_done global_id 104682 payload 824 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63ac137130 msgr2=0x7f63ac11f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f63ac137130 0x7f63ac11f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 msgr2=0x7f63ac11f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63ac070a40 0x7f63ac11f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 monclient: found mon.noname-a 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.359+0000 7f63abfff640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009390 con 0x7f63ac06a550 2024-09-15T02:00:32.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63abfff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 msgr2=0x7f63ac11eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63abfff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098b0 tx=0x7f639c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031940 con 0x7f63ac06a550 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: dump: 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.377 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c031aa0 con 0x7f63ac06a550 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 5 monclient: authenticate success, global_id 104682 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 10 monclient: _renew_subs 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac122530 con 0x7f63ac06a550 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031e40 con 0x7f63ac06a550 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f6393fff640 10 monclient: dump: 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.379 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 10 monclient: _renew_subs 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63aa7fc640 4 set_mon_vals no callback set 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.361+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac121010 con 0x7f63ac06a550 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f639c039070 con 0x7f63ac06a550 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63b288b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63b288b640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 0x7f63ac02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac063b00 con 0x7f63ac11faf0 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 0x7f63ac02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient: get_auth_request con 0x7f63ac11faf0 auth_method 0 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f6393fff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63980386f0 0x7f639803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.362+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.363+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(242..242 src has 148..242) v4 ==== 9668+0+0 (secure 0 0 0) 0x7f639c06d4b0 con 0x7f63ac06a550 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.363+0000 7f63b0e28640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63980386f0 0x7f639803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.363+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_done global_id 104685 payload 824 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.363+0000 7f63ab7fe640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac120500 con 0x7f63ac11faf0 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.363+0000 7f63b0e28640 10 monclient: get_auth_request con 0x7f63980386f0 auth_method 0 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f63ab7fe640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 msgr2=0x7f63ac02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f63ab7fe640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 0x7f63ac02aa40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a000d8b0 con 0x7f63ac11faf0 2024-09-15T02:00:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 10 monclient: dump: 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f63b0e28640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63980386f0 0x7f639803aba0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f63ac122d00 tx=0x7f63a4005080 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f63a000da10 con 0x7f63ac11faf0 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 msgr2=0x7f63ac02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 0x7f63ac02aa40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 msgr2=0x7f63ac02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.365+0000 7f6393fff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac11faf0 0x7f63ac02aa40 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:32.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.373+0000 7fb371a34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:32.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.373+0000 7fb371a34640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:32.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.373+0000 7fb371a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:32.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.373+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fb36c04bf60 con 0x7fb36c0a0b50 2024-09-15T02:00:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63b288b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63b288b640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 0x7f63ac05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63b288b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0724a0 con 0x7f63ac05e310 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 0x7f63ac05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient: get_auth_request con 0x7f63ac05e310 auth_method 0 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 10 monclient(hunting): handle_auth_done global_id 104688 payload 824 2024-09-15T02:00:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.371+0000 7f63ab7fe640 1 -- 172.21.15.17:0/1093036482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f63ac11b430 con 0x7f63ac05e310 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f63ab7fe640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 msgr2=0x7f63ac05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f63ab7fe640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 0x7f63ac05e710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a001d610 con 0x7f63ac05e310 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 10 monclient: dump: 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f63a001d770 con 0x7f63ac05e310 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 msgr2=0x7f63ac05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 0x7f63ac05e710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0019000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 msgr2=0x7f63ac05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f6393fff640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac05e310 0x7f63ac05e710 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f63a000f780 tx=0x7f63a0019000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 104, 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:32.389 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:32.390 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:32.391 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:32.392 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:32.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:32.395 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.396 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.397 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.398 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.399 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.400 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:32.401 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63980386f0 msgr2=0x7f639803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.374+0000 7f63b288b640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f63980386f0 0x7f639803aba0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f63ac122d00 tx=0x7f63a4005080 comp rx=0 tx=0).stop 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 10 monclient: shutdown 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 msgr2=0x7f63ac11eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b0e28640 1 -- 172.21.15.17:0/1093036482 reap_dead start 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098b0 tx=0x7f639c032000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 shutdown_connections 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 --2- 172.21.15.17:0/1093036482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f63ac06a550 0x7f63ac11eb30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 >> 172.21.15.17:0/1093036482 conn(0x7f63ac067ae0 msgr2=0x7f63ac09fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 shutdown_connections 2024-09-15T02:00:32.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.375+0000 7f63b288b640 1 -- 172.21.15.17:0/1093036482 wait complete. 2024-09-15T02:00:32.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:00:32.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:00:32.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:00:32.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.498+0000 7f7b7371d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:32.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.498+0000 7f7b7371d640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.498+0000 7f7b7371d640 10 monclient: monmap: 2024-09-15T02:00:32.502 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:32.500585+0000 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:32.500585+0000 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:32.503 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:32.504 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 1 Processor -- start 2024-09-15T02:00:32.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 1 -- start start 2024-09-15T02:00:32.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 10 monclient: init 2024-09-15T02:00:32.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T02:00:32.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7b6c06eb30 0x7f7b6c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c06eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:32.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.499+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b6c06f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b6c09e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c069c60 con 0x7f7b6c09e970 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c09f690 con 0x7f7b6c06f440 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6c06a920 con 0x7f7b6c06eb30 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b7371d640 10 monclient(hunting): authenticate will time out at 1292.497052s 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7b6c06eb30 0x7f7b6c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b71cba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7b6c09e970 0x7f7b6c09c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60940/0 (socket says 172.21.15.17:60940) 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b71cba640 1 -- 172.21.15.17:0/42953527 learned_addr learned my addr 172.21.15.17:0/42953527 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): get_auth_request con 0x7f7b6c06eb30 auth_method 0 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient(hunting): handle_auth_done global_id 104691 payload 274 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 1 -- 172.21.15.17:0/42953527 >> v1:172.21.15.67:6789/0 conn(0x7f7b6c06f440 legacy=0x7f7b6c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 1 -- 172.21.15.17:0/42953527 >> v1:172.21.15.17:6789/0 conn(0x7f7b6c09e970 legacy=0x7f7b6c09c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 1 monclient: found mon.noname-a 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.500+0000 7f7b714b9640 1 -- 172.21.15.17:0/42953527 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c09ad40 con 0x7f7b6c06eb30 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b714b9640 1 -- 172.21.15.17:0/42953527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06eb30 msgr2=0x7f7b6c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b714b9640 1 --2- 172.21.15.17:0/42953527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06eb30 0x7f7b6c06ef00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009850 tx=0x7f7b680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e451b0f7d0f965 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 1 -- 172.21.15.17:0/42953527 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68032440 con 0x7f7b6c06eb30 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: dump: 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 1 -- 172.21.15.17:0/42953527 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b680325a0 con 0x7f7b6c06eb30 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 1 -- 172.21.15.17:0/42953527 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68032940 con 0x7f7b6c06eb30 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b63fff640 10 monclient: dump: 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.511 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 5 monclient: authenticate success, global_id 104691 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:32.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 4 set_mon_vals no callback set 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 10 monclient: shutdown 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 1 -- 172.21.15.17:0/42953527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06eb30 msgr2=0x7f7b6c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.503+0000 7f7b7371d640 1 --2- 172.21.15.17:0/42953527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06eb30 0x7f7b6c06ef00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009850 tx=0x7f7b680090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 -- 172.21.15.17:0/42953527 shutdown_connections 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 --2- 172.21.15.17:0/42953527 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06eb30 0x7f7b6c06ef00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 -- 172.21.15.17:0/42953527 >> 172.21.15.17:0/42953527 conn(0x7f7b6c0712f0 msgr2=0x7f7b6c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 -- 172.21.15.17:0/42953527 shutdown_connections 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 -- 172.21.15.17:0/42953527 wait complete. 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 10 monclient: build_initial_monmap 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 10 monclient: monmap: 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:32.522 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:32.523 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:32.523 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 Processor -- start 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 1 -- start start 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.504+0000 7f7b7371d640 10 monclient: init 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b6c06eb30 0x7f7b6c141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c06f440 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c09e970 0x7f7b6c11d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c09e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:32.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b714b9640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b6c06eb30 msgr2=0x7f7b6c141130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c11f470 con 0x7f7b6c06f440 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c134b70 con 0x7f7b6c09e970 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c1211a0 con 0x7f7b6c06eb30 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b714b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b6c06eb30 0x7f7b6c141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:32.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b7371d640 10 monclient(hunting): authenticate will time out at 1292.502576s 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:52204/0 (socket says 172.21.15.17:52204) 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 -- 172.21.15.17:0/3412838250 learned_addr learned my addr 172.21.15.17:0/3412838250 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): get_auth_request con 0x7f7b6c06f440 auth_method 0 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c09e970 0x7f7b6c11d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 10 monclient(hunting): get_auth_request con 0x7f7b6c09e970 auth_method 0 2024-09-15T02:00:32.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient(hunting): handle_auth_done global_id 104694 payload 824 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b6c06eb30 msgr2=0x7f7b6c141130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7b6c06eb30 0x7f7b6c141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c09e970 msgr2=0x7f7b6c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c09e970 0x7f7b6c11d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient: _finish_hunting 0 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 monclient: found mon.noname-a 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b70cb8640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68009390 con 0x7f7b6c06f440 2024-09-15T02:00:32.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.505+0000 7f7b71cba640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c09e970 0x7f7b6c11d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b70cb8640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 msgr2=0x7f7b6c133da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b70cb8640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00c5e0 tx=0x7f7b5c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c012840 con 0x7f7b6c06f440 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: dump: 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 5 monclient: authenticate success, global_id 104694 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 10 monclient: _renew_subs 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c0129a0 con 0x7f7b6c06f440 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c120ec0 con 0x7f7b6c06f440 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c012d60 con 0x7f7b6c06f440 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b617fa640 10 monclient: dump: 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b637fe640 4 set_mon_vals no callback set 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 10 monclient: _renew_subs 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.508+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c141810 con 0x7f7b6c06f440 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b7371d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b7371d640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 0x7f7b6c11df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c063b90 con 0x7f7b6c1342e0 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7b5c011590 con 0x7f7b6c06f440 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b617fa640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b500386f0 0x7f7b5003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 0x7f7b6c11df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(242..242 src has 148..242) v4 ==== 9668+0+0 (secure 0 0 0) 0x7f7b5c04cf40 con 0x7f7b6c06f440 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient: get_auth_request con 0x7f7b6c1342e0 auth_method 0 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b71cba640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b500386f0 0x7f7b5003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b71cba640 10 monclient: get_auth_request con 0x7f7b500386f0 auth_method 0 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 10 monclient(hunting): handle_auth_done global_id 124177 payload 824 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.509+0000 7f7b714b9640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b6c1215f0 con 0x7f7b6c1342e0 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b714b9640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 msgr2=0x7f7b6c11df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b714b9640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 0x7f7b6c11df20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b68034000 tx=0x7f7b68032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b71cba640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b500386f0 0x7f7b5003aba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400e1b0 tx=0x7f7b64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6803d9d0 con 0x7f7b6c1342e0 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 10 monclient: dump: 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7b6803db30 con 0x7f7b6c1342e0 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 msgr2=0x7f7b6c11df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 0x7f7b6c11df20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b68034000 tx=0x7f7b68032000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 msgr2=0x7f7b6c11df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.512+0000 7f7b617fa640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c1342e0 0x7f7b6c11df20 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b68034000 tx=0x7f7b68032000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b7371d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b7371d640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 0x7f7b6c0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:32.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b7371d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:32.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c1346b0 con 0x7f7b6c11e460 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 0x7f7b6c0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient: get_auth_request con 0x7f7b6c11e460 auth_method 0 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:32.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 10 monclient(hunting): handle_auth_done global_id 124180 payload 824 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.518+0000 7f7b714b9640 1 -- 172.21.15.17:0/3412838250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b6c11e860 con 0x7f7b6c11e460 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b714b9640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 msgr2=0x7f7b6c0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b714b9640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 0x7f7b6c0a1480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009130 tx=0x7f7b6803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680499e0 con 0x7f7b6c11e460 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:32.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 10 monclient: dump: 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:32.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f7b68049b40 con 0x7f7b6c11e460 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 msgr2=0x7f7b6c0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 0x7f7b6c0a1480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009130 tx=0x7f7b6803d000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 msgr2=0x7f7b6c0a1480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b617fa640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6c11e460 0x7f7b6c0a1480 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009130 tx=0x7f7b6803d000 comp rx=0 tx=0).stop 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 104, 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:00:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:00:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:00:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:00:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:00:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:00:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:00:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:00:32.554 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:00:32.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b500386f0 msgr2=0x7f7b5003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7b500386f0 0x7f7b5003aba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400e1b0 tx=0x7f7b64009040 comp rx=0 tx=0).stop 2024-09-15T02:00:32.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 10 monclient: shutdown 2024-09-15T02:00:32.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:32.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 msgr2=0x7f7b6c133da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b7371d640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00c5e0 tx=0x7f7b5c007ed0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.521+0000 7f7b71cba640 1 -- 172.21.15.17:0/3412838250 reap_dead start 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.522+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 shutdown_connections 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.522+0000 7f7b7371d640 1 --2- 172.21.15.17:0/3412838250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7b6c06f440 0x7f7b6c133da0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.522+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 >> 172.21.15.17:0/3412838250 conn(0x7f7b6c0712f0 msgr2=0x7f7b6c134d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.522+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 shutdown_connections 2024-09-15T02:00:32.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:32.522+0000 7f7b7371d640 1 -- 172.21.15.17:0/3412838250 wait complete. 2024-09-15T02:00:32.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:00:32.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:00:33.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.997+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb360037070 con 0x7fb36c0a0b50 2024-09-15T02:00:33.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.997+0000 7fb352ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:33.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:32.997+0000 7fb352ffd640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-15T02:00:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.102+0000 7fb371a34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.102+0000 7fb371a34640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.102+0000 7fb371a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.102+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fb36c04bf60 con 0x7fb36c0a0b50 2024-09-15T02:00:33.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.103+0000 7fb352ffd640 1 -- 172.21.15.67:0/2489031968 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb3600443b0 con 0x7fb36c0a0b50 2024-09-15T02:00:33.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.103+0000 7fb352ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T02:00:33.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.103+0000 7fb352ffd640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-15T02:00:33.105 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h26' does not appear in the crush map 2024-09-15T02:00:33.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 msgr2=0x7fb35803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 0x7fb35803aba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0047d0 tx=0x7fb35c007af0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 10 monclient: shutdown 2024-09-15T02:00:33.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 msgr2=0x7fb36c11d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb360009360 tx=0x7fb360032000 comp rx=0 tx=0).stop 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 shutdown_connections 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3580386f0 0x7fb35803aba0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb36c136e40 0x7fb36c11dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb36c0a0b50 0x7fb36c11d6a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 --2- 172.21.15.67:0/2489031968 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb36c09eea0 0x7fb36c134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 >> 172.21.15.67:0/2489031968 conn(0x7fb36c071290 msgr2=0x7fb36c135e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 shutdown_connections 2024-09-15T02:00:33.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.105+0000 7fb371a34640 1 -- 172.21.15.67:0/2489031968 wait complete. 2024-09-15T02:00:33.116 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:33.116 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h27 2024-09-15T02:00:33.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:33.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 10 monclient: monmap: 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:33.191305+0000 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:33.191305+0000 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:33.193 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:33.194 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 1 Processor -- start 2024-09-15T02:00:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 1 -- start start 2024-09-15T02:00:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.190+0000 7f4740d9d640 10 monclient: init 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f473c098140 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f473c098510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f473c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f473c098a50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c098ef0 con 0x7f473c098140 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0a0b80 con 0x7f473c098a50 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0a1730 con 0x7f473c0a0370 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4740d9d640 10 monclient(hunting): authenticate will time out at 1265.660469s 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4739d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 msgr2=0x7f473c098510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f4739d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f473c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f473ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f473c098a50 0x7f473c1390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47730/0 (socket says 172.21.15.67:47730) 2024-09-15T02:00:33.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.191+0000 7f473ad9d640 1 -- 172.21.15.67:0/746602165 learned_addr learned my addr 172.21.15.67:0/746602165 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473ad9d640 1 -- 172.21.15.67:0/746602165 >> v1:172.21.15.67:6789/0 conn(0x7f473c098a50 legacy=0x7f473c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162525996 0 0) 0x7f473c0a0b80 con 0x7f473c098a50 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473959a640 10 monclient(hunting): my global_id is 124183 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473a59c640 1 -- 172.21.15.67:0/746602165 >> v1:172.21.15.17:6789/0 conn(0x7f473c098140 legacy=0x7f473c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.193+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47240034e0 con 0x7f473c098a50 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009747173 0 0) 0x7f472c0014a0 con 0x7f473c098140 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): my global_id is 104697 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473c0a0b80 con 0x7f473c098140 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773343063 0 0) 0x7f47240034e0 con 0x7f473c098a50 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 msgr2=0x7f473c098510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f473c098510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 >> v1:172.21.15.17:6789/0 conn(0x7f473c098140 legacy=0x7f473c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c1328f0 con 0x7f473c098a50 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: _finish_auth 0 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4734001aa0 con 0x7f473c098a50 2024-09-15T02:00:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: dump: 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 >> v1:172.21.15.67:6789/0 conn(0x7f473c098a50 legacy=0x7f473c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724003b60 0x7f4724003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): picked mon.a con 0x7f4724003b60 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 0x7f4724004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): picked mon.b con 0x7f4724004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f472400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): picked mon.c con 0x7f473c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0a1730 con 0x7f4724003b60 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240055c0 con 0x7f4724004780 2024-09-15T02:00:33.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472400bf10 con 0x7f473c0a0370 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473ad9d640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 msgr2=0x7f472400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473ad9d640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f472400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 0x7f4724004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724003b60 0x7f4724003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): get_auth_request con 0x7f4724004780 auth_method 0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): get_auth_request con 0x7f4724003b60 auth_method 0 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:33.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient(hunting): handle_auth_done global_id 124183 payload 293 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 msgr2=0x7f472400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c0a0370 0x7f472400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724003b60 msgr2=0x7f4724003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724003b60 0x7f4724003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:00:33.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 monclient: found mon.b 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 1 -- 172.21.15.67:0/746602165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240034e0 con 0x7f4724004780 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 10 monclient: _finish_auth 0 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4739d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4740d9d640 5 monclient: authenticate success, global_id 124183 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f473a59c640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724003b60 0x7f4724003f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.194+0000 7f4740d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f4739d9b640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 msgr2=0x7f4724004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f4739d9b640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 0x7f4724004bf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47300099e0 tx=0x7f4730006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d3bc0db0719d52 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730003b70 con 0x7f4724004780 2024-09-15T02:00:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: dump: 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f4740d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:33.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4730003d10 con 0x7f4724004780 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f4740d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 1 -- 172.21.15.67:0/746602165 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473002ab60 con 0x7f4724004780 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f4740d9d640 4 set_mon_vals no callback set 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.197+0000 7f473959a640 10 monclient: dump: 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 10 monclient: shutdown 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 -- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 msgr2=0x7f4724004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 --2- 172.21.15.67:0/746602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4724004780 0x7f4724004bf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47300099e0 tx=0x7f4730006f90 comp rx=0 tx=0).stop 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f473ad9d640 1 -- 172.21.15.67:0/746602165 reap_dead start 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 -- 172.21.15.67:0/746602165 shutdown_connections 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 -- 172.21.15.67:0/746602165 >> 172.21.15.67:0/746602165 conn(0x7f473c0a5780 msgr2=0x7f473c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 -- 172.21.15.67:0/746602165 shutdown_connections 2024-09-15T02:00:33.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 1 -- 172.21.15.67:0/746602165 wait complete. 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.198+0000 7f4740d9d640 10 monclient: monmap: 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:33.217 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 Processor -- start 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 -- start start 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient: init 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724004780 0x7f473c1412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4724004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 0x7f473c131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f473c141810 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f473c132370 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c120550 con 0x7f4724004780 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c099090 con 0x7f473c132370 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0a49a0 con 0x7f473c141810 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4740d9d640 10 monclient(hunting): authenticate will time out at 1265.668612s 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4739d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 msgr2=0x7f473c131e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37648/0 (socket says 172.21.15.67:37648) 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 -- 172.21.15.67:0/1890389359 learned_addr learned my addr 172.21.15.67:0/1890389359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): get_auth_request con 0x7f473c132370 auth_method 0 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:33.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f4739d9b640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 0x7f473c131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473a59c640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724004780 0x7f473c1412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473a59c640 10 monclient(hunting): get_auth_request con 0x7f4724004780 auth_method 0 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient(hunting): handle_auth_done global_id 124186 payload 824 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 -- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 msgr2=0x7f473c131e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 0x7f473c131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 -- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724004780 msgr2=0x7f473c1412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724004780 0x7f473c1412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 monclient: found mon.noname-b 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.199+0000 7f473ad9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c121f80 con 0x7f473c132370 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f473ad9d640 1 -- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 msgr2=0x7f473c11e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f473ad9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f473c09fa10 tx=0x7f473c068d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734004d30 con 0x7f473c132370 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: dump: 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:33.221 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734004e90 con 0x7f473c132370 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 5 monclient: authenticate success, global_id 124186 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 10 monclient: _renew_subs 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c142140 con 0x7f473c132370 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47340073c0 con 0x7f473c132370 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:33.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f472a7fc640 10 monclient: dump: 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4738d99640 4 set_mon_vals no callback set 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 10 monclient: _renew_subs 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.202+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c11eeb0 con 0x7f473c132370 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4734007770 con 0x7f473c132370 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f4740d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f4740d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f4740d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c122380 con 0x7f473c132370 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f472a7fc640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 0x7f471c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(243..243 src has 217..243) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f473404c280 con 0x7f473c132370 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f473a59c640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 0x7f471c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:33.225 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.203+0000 7f473a59c640 10 monclient: get_auth_request con 0x7f471c038720 auth_method 0 2024-09-15T02:00:33.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.206+0000 7f473a59c640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 0x7f471c03abd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f472c000f60 tx=0x7f472c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:33.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.207+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4734010ae0 con 0x7f473c132370 2024-09-15T02:00:33.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.207+0000 7f472a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:33.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.207+0000 7f472a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:33.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.374+0000 7f4740d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:33.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.374+0000 7f4740d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:33.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.374+0000 7f4740d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:33.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:33.374+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f473c122170 con 0x7f473c132370 2024-09-15T02:00:34.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.002+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4734010300 con 0x7f473c132370 2024-09-15T02:00:34.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.002+0000 7f472a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:34.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.002+0000 7f472a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-15T02:00:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.104+0000 7f4740d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.104+0000 7f4740d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.104+0000 7f4740d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.104+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f473c071390 con 0x7f473c132370 2024-09-15T02:00:34.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.106+0000 7f472a7fc640 1 -- 172.21.15.67:0/1890389359 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7f47340104e0 con 0x7f473c132370 2024-09-15T02:00:34.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.106+0000 7f472a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T02:00:34.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.106+0000 7f472a7fc640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-15T02:00:34.108 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h27' does not appear in the crush map 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 msgr2=0x7f471c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 0x7f471c03abd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f472c000f60 tx=0x7f472c009040 comp rx=0 tx=0).stop 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 10 monclient: shutdown 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 msgr2=0x7f473c11e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f473c09fa10 tx=0x7f473c068d50 comp rx=0 tx=0).stop 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 shutdown_connections 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f471c038720 0x7f471c03abd0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f473c132370 0x7f473c11e970 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f473c141810 0x7f473c131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 --2- 172.21.15.67:0/1890389359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4724004780 0x7f473c1412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 >> 172.21.15.67:0/1890389359 conn(0x7f473c0a5780 msgr2=0x7f473c0992a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:34.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.107+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 shutdown_connections 2024-09-15T02:00:34.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.108+0000 7f4740d9d640 1 -- 172.21.15.67:0/1890389359 wait complete. 2024-09-15T02:00:34.119 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:34.119 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h28 2024-09-15T02:00:34.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.192+0000 7f0ea2c8b640 10 monclient: get_monmap_and_config 2024-09-15T02:00:34.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.192+0000 7f0ea2c8b640 10 monclient: build_initial_monmap 2024-09-15T02:00:34.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.192+0000 7f0ea2c8b640 10 monclient: monmap: 2024-09-15T02:00:34.195 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:34.195 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:34.194040+0000 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:34.194040+0000 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:34.196 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:34.197 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:34.197 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.193+0000 7f0ea2c8b640 1 Processor -- start 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.193+0000 7f0ea2c8b640 1 -- start start 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.193+0000 7f0ea2c8b640 10 monclient: init 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0e9c0a3830 0x7f0e9c06b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c0a3830 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e9c06e2b0 0x7f0e9c09db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c06e2b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e9c09e050 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c069c60 con 0x7f0e9c09e050 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9c0a3c00 con 0x7f0e9c0a3830 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9c069650 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:34.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea2c8b640 10 monclient(hunting): authenticate will time out at 1266.663212s 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e9c06e2b0 0x7f0e9c09db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea1228640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e9c09e050 0x7f0e9c138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47736/0 (socket says 172.21.15.67:47736) 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea1228640 1 -- 172.21.15.67:0/2463174491 learned_addr learned my addr 172.21.15.67:0/2463174491 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request con 0x7f0e9c06e2b0 auth_method 0 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:34.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_done global_id 124189 payload 274 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 -- 172.21.15.67:0/2463174491 >> v2:172.21.15.17:3300/0 conn(0x7f0e9c0a3830 msgr2=0x7f0e9c06b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0ea0a27640 1 --2- 172.21.15.67:0/2463174491 >> v2:172.21.15.17:3300/0 conn(0x7f0e9c0a3830 0x7f0e9c06b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 --2- 172.21.15.67:0/2463174491 >> v2:172.21.15.17:3300/0 conn(0x7f0e9c0a3830 0x7f0e9c06b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 -- 172.21.15.67:0/2463174491 >> v1:172.21.15.67:6789/0 conn(0x7f0e9c09e050 legacy=0x7f0e9c138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 monclient: found mon.noname-c 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:34.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.194+0000 7f0e9bfff640 1 -- 172.21.15.67:0/2463174491 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c09f500 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9bfff640 1 -- 172.21.15.67:0/2463174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c06e2b0 msgr2=0x7f0e9c09db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9bfff640 1 --2- 172.21.15.67:0/2463174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c06e2b0 0x7f0e9c09db10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c09f140 tx=0x7f0e880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da1cfb7b6114546b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 1 -- 172.21.15.67:0/2463174491 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88032440 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: dump: 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.207 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 1 -- 172.21.15.67:0/2463174491 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e880325a0 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0ea2c8b640 5 monclient: authenticate success, global_id 124189 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0ea2c8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0ea2c8b640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 1 -- 172.21.15.67:0/2463174491 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88032940 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0ea2c8b640 4 set_mon_vals no callback set 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.197+0000 7f0e9b7fe640 10 monclient: dump: 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.209 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 10 monclient: shutdown 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/2463174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c06e2b0 msgr2=0x7f0e9c09db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/2463174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c06e2b0 0x7f0e9c09db10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c09f140 tx=0x7f0e880090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/2463174491 shutdown_connections 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/2463174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c06e2b0 0x7f0e9c09db10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/2463174491 >> v2:172.21.15.17:3300/0 conn(0x7f0e9c0a3830 0x7f0e9c06b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/2463174491 >> 172.21.15.67:0/2463174491 conn(0x7f0e9c070ff0 msgr2=0x7f0e9c0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/2463174491 shutdown_connections 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/2463174491 wait complete. 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 10 monclient: build_initial_monmap 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 10 monclient: monmap: 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:34.211 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.198+0000 7f0ea2c8b640 1 Processor -- start 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 -- start start 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient: init 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 0x7f0e9c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c06e2b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c09e050 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e9c0a3830 0x7f0e9c09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c0a3830 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c0a2cd0 con 0x7f0e9c0a3830 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c142310 con 0x7f0e9c09e050 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c09d070 con 0x7f0e9c06e2b0 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea2c8b640 10 monclient(hunting): authenticate will time out at 1266.668609s 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37666/0 (socket says 172.21.15.67:37666) 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 -- 172.21.15.67:0/678964463 learned_addr learned my addr 172.21.15.67:0/678964463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request con 0x7f0e9c09e050 auth_method 0 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:34.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea0a27640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 msgr2=0x7f0e9c1414b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea0a27640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 0x7f0e9c1414b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea1228640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e9c0a3830 0x7f0e9c09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea1228640 10 monclient(hunting): get_auth_request con 0x7f0e9c0a3830 auth_method 0 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea1228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea1228640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0ea1228640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_done global_id 124192 payload 824 2024-09-15T02:00:34.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 msgr2=0x7f0e9c1414b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 0x7f0e9c1414b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e9c0a3830 msgr2=0x7f0e9c09b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e9c0a3830 0x7f0e9c09b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 monclient: found mon.noname-b 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.199+0000 7f0e9bfff640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e88009390 con 0x7f0e9c09e050 2024-09-15T02:00:34.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e9bfff640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 msgr2=0x7f0e9c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:34.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e9bfff640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e88035040 tx=0x7f0e88032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:34.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88000cf0 con 0x7f0e9c09e050 2024-09-15T02:00:34.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: dump: 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: _finish_auth 0 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 5 monclient: authenticate success, global_id 124192 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e88031c70 con 0x7f0e9c09e050 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8803d440 con 0x7f0e9c09e050 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e98ff9640 10 monclient: dump: 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:34.219 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 10 monclient: _renew_subs 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c11f0e0 con 0x7f0e9c09e050 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0e9affd640 4 set_mon_vals no callback set 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 10 monclient: _renew_subs 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.202+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c11e180 con 0x7f0e9c09e050 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0e88039040 con 0x7f0e9c09e050 2024-09-15T02:00:34.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0e98ff9640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 0x7f0e8003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(244..244 src has 217..244) v4 ==== 9242+0+0 (secure 0 0 0) 0x7f0e8806c870 con 0x7f0e9c09e050 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea2c8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea2c8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea2c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c11f560 con 0x7f0e9c09e050 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea0a27640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 0x7f0e8003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.203+0000 7f0ea0a27640 10 monclient: get_auth_request con 0x7f0e800386f0 auth_method 0 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.206+0000 7f0ea0a27640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 0x7f0e8003aba0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006fd0 tx=0x7f0e90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.207+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0e88042020 con 0x7f0e9c09e050 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.207+0000 7f0e98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:34.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.207+0000 7f0e98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:34.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.375+0000 7f0ea2c8b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:34.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.375+0000 7f0ea2c8b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:34.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.375+0000 7f0ea2c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:34.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:34.375+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f0e9c04bf60 con 0x7f0e9c09e050 2024-09-15T02:00:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.006+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v245) v1 ==== 121+0+0 (secure 0 0 0) 0x7f0e880432b0 con 0x7f0e9c09e050 2024-09-15T02:00:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.006+0000 7f0e98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.006+0000 7f0e98ff9640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-15T02:00:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.108+0000 7f0ea2c8b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.108+0000 7f0ea2c8b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.108+0000 7f0ea2c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.108+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f0e9c04bf60 con 0x7f0e9c09e050 2024-09-15T02:00:35.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.109+0000 7f0e98ff9640 1 -- 172.21.15.67:0/678964463 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0e88030e60 con 0x7f0e9c09e050 2024-09-15T02:00:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.109+0000 7f0e98ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T02:00:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.109+0000 7f0e98ff9640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-15T02:00:35.112 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h28' does not appear in the crush map 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 msgr2=0x7f0e8003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 0x7f0e8003aba0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006fd0 tx=0x7f0e90008040 comp rx=0 tx=0).stop 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 10 monclient: shutdown 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 msgr2=0x7f0e9c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e88035040 tx=0x7f0e88032000 comp rx=0 tx=0).stop 2024-09-15T02:00:35.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 shutdown_connections 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0e800386f0 0x7f0e8003aba0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e9c0a3830 0x7f0e9c09b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c09e050 0x7f0e9c131fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 --2- 172.21.15.67:0/678964463 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0e9c06e2b0 0x7f0e9c1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 >> 172.21.15.67:0/678964463 conn(0x7f0e9c070ff0 msgr2=0x7f0e9c0a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 shutdown_connections 2024-09-15T02:00:35.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.111+0000 7f0ea2c8b640 1 -- 172.21.15.67:0/678964463 wait complete. 2024-09-15T02:00:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h29 2024-09-15T02:00:35.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.195+0000 7f25826af640 10 monclient: get_monmap_and_config 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.195+0000 7f25826af640 10 monclient: build_initial_monmap 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.195+0000 7f25826af640 10 monclient: monmap: 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:35.196705+0000 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:35.196705+0000 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:35.199 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 Processor -- start 2024-09-15T02:00:35.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 -- start start 2024-09-15T02:00:35.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient: init 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c0a3850 0x7f257c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): picked mon.noname-c con 0x7f257c0a3850 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c09db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): picked mon.noname-b con 0x7f257c06a6e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f257c09e050 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c09f9a0 con 0x7f257c09e050 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:35.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c069c20 con 0x7f257c0a3850 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0696c0 con 0x7f257c06a6e0 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.196+0000 7f25826af640 10 monclient(hunting): authenticate will time out at 1267.665998s 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257b7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 msgr2=0x7f257c09db10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c09db10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f2580c4c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f257c09e050 0x7f257c138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47748/0 (socket says 172.21.15.67:47748) 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c0a3850 0x7f257c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f2580c4c640 1 -- 172.21.15.67:0/4171231120 learned_addr learned my addr 172.21.15.67:0/4171231120 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): get_auth_request con 0x7f257c0a3850 auth_method 0 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:35.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient(hunting): handle_auth_done global_id 124195 payload 274 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 -- 172.21.15.67:0/4171231120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 msgr2=0x7f257c09db10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 --2- 172.21.15.67:0/4171231120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c09db10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 -- 172.21.15.67:0/4171231120 >> v1:172.21.15.67:6789/0 conn(0x7f257c09e050 legacy=0x7f257c138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 monclient: found mon.noname-c 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.197+0000 7f257bfff640 1 -- 172.21.15.67:0/4171231120 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0a3c20 con 0x7f257c0a3850 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257bfff640 1 -- 172.21.15.67:0/4171231120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0a3850 msgr2=0x7f257c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257bfff640 1 --2- 172.21.15.67:0/4171231120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0a3850 0x7f257c06e1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2570009850 tx=0x7f25700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae5abf38d631531 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 1 -- 172.21.15.67:0/4171231120 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570032440 con 0x7f257c0a3850 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:35.210 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: dump: 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: _finish_auth 0 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:35.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 1 -- 172.21.15.67:0/4171231120 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25700325a0 con 0x7f257c0a3850 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 5 monclient: authenticate success, global_id 124195 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 1 -- 172.21.15.67:0/4171231120 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570032940 con 0x7f257c0a3850 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 4 set_mon_vals no callback set 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f257affd640 10 monclient: dump: 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.212 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 10 monclient: shutdown 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 -- 172.21.15.67:0/4171231120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0a3850 msgr2=0x7f257c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 --2- 172.21.15.67:0/4171231120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0a3850 0x7f257c06e1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2570009850 tx=0x7f25700090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 -- 172.21.15.67:0/4171231120 shutdown_connections 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 --2- 172.21.15.67:0/4171231120 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c09db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 --2- 172.21.15.67:0/4171231120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0a3850 0x7f257c06e1a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 -- 172.21.15.67:0/4171231120 >> 172.21.15.67:0/4171231120 conn(0x7f257c070ff0 msgr2=0x7f257c0a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 -- 172.21.15.67:0/4171231120 shutdown_connections 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.200+0000 7f25826af640 1 -- 172.21.15.67:0/4171231120 wait complete. 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.201+0000 7f25826af640 10 monclient: build_initial_monmap 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.201+0000 7f25826af640 10 monclient: monmap: 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:35.214 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.201+0000 7f25826af640 1 Processor -- start 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.201+0000 7f25826af640 1 -- start start 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.201+0000 7f25826af640 10 monclient: init 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): picked mon.noname-c con 0x7f257c06a6e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): picked mon.noname-b con 0x7f257c09e050 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f257c0a3850 0x7f257c11e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): picked mon.noname-a con 0x7f257c0a3850 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:35.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c06e9e0 con 0x7f257c0a3850 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c134d00 con 0x7f257c09e050 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c11f340 con 0x7f257c06a6e0 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f25826af640 10 monclient(hunting): authenticate will time out at 1267.671355s 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257bfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 msgr2=0x7f257c133d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37694/0 (socket says 172.21.15.67:37694) 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 -- 172.21.15.67:0/3890739606 learned_addr learned my addr 172.21.15.67:0/3890739606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): get_auth_request con 0x7f257c09e050 auth_method 0 2024-09-15T02:00:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257bfff640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c133d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient(hunting): handle_auth_done global_id 124198 payload 824 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 -- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 msgr2=0x7f257c133d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c133d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 -- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f257c0a3850 msgr2=0x7f257c11e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f257c0a3850 0x7f257c11e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 monclient: found mon.noname-b 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.202+0000 7f257b7fe640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570009390 con 0x7f257c09e050 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.204+0000 7f257b7fe640 1 -- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 msgr2=0x7f257c11db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f257b7fe640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f256c00b600 tx=0x7f256c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f257b7fe640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f256c00b600 tx=0x7f256c0091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f256c006380 mon_map magic: 0 v1 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c015070 con 0x7f257c09e050 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 10 monclient: dump: 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:35.218 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c003d90 con 0x7f257c09e050 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 5 monclient: authenticate success, global_id 124198 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 10 monclient: _renew_subs 2024-09-15T02:00:35.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c11fef0 con 0x7f257c09e050 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f2563fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f257a7fc640 4 set_mon_vals no callback set 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 10 monclient: _renew_subs 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.205+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c120e50 con 0x7f257c09e050 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c006380 con 0x7f257c09e050 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 10 monclient: dump: 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:35.220 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f256c010440 con 0x7f257c09e050 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f25826af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f25826af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f25826af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c121090 con 0x7f257c09e050 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f2563fff640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 0x7f256803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:35.221 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f257bfff640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 0x7f256803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:35.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.206+0000 7f257bfff640 10 monclient: get_auth_request con 0x7f2568038740 auth_method 0 2024-09-15T02:00:35.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.209+0000 7f257bfff640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 0x7f256803abf0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f257c09d730 tx=0x7f2570032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:35.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.211+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(245..245 src has 217..245) v4 ==== 9029+0+0 (secure 0 0 0) 0x7f256c04c730 con 0x7f257c09e050 2024-09-15T02:00:35.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.828+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f256c018d60 con 0x7f257c09e050 2024-09-15T02:00:35.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.828+0000 7f2563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:35.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.828+0000 7f2563fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.995+0000 7f25826af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.995+0000 7f25826af640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.995+0000 7f25826af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:35.995+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f257c04bf60 con 0x7f257c09e050 2024-09-15T02:00:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.711+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v246) v1 ==== 121+0+0 (secure 0 0 0) 0x7f256c018780 con 0x7f257c09e050 2024-09-15T02:00:36.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.711+0000 7f2563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:36.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.711+0000 7f2563fff640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.815+0000 7f25826af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.815+0000 7f25826af640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.815+0000 7f25826af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.815+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f257c067090 con 0x7f257c09e050 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.817+0000 7f2563fff640 1 -- 172.21.15.67:0/3890739606 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7f256c04a080 con 0x7f257c09e050 2024-09-15T02:00:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.817+0000 7f2563fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.817+0000 7f2563fff640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h29' does not appear in the crush map 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 msgr2=0x7f256803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 0x7f256803abf0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f257c09d730 tx=0x7f2570032000 comp rx=0 tx=0).stop 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 10 monclient: shutdown 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 msgr2=0x7f257c11db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f256c00b600 tx=0x7f256c0091a0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 shutdown_connections 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2568038740 0x7f256803abf0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f257c0a3850 0x7f257c11e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c09e050 0x7f257c11db90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 --2- 172.21.15.67:0/3890739606 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f257c06a6e0 0x7f257c133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 >> 172.21.15.67:0/3890739606 conn(0x7f257c070ff0 msgr2=0x7f257c06d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 shutdown_connections 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.819+0000 7f25826af640 1 -- 172.21.15.67:0/3890739606 wait complete. 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h30 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.904+0000 7faa9cf5d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.904+0000 7faa9cf5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.905+0000 7faa9cf5d640 10 monclient: monmap: 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:36.906157+0000 2024-09-15T02:00:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:36.906157+0000 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.905+0000 7faa9cf5d640 1 Processor -- start 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.905+0000 7faa9cf5d640 1 -- start start 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.905+0000 7faa9cf5d640 10 monclient: init 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa98068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa98069190 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faa98138160 0x7faa9813a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-a con 0x7faa98138160 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa98096c10 con 0x7faa98068dc0 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa98132960 con 0x7faa98069190 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9806a2a0 con 0x7faa98138160 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa9cf5d640 10 monclient(hunting): authenticate will time out at 1269.375424s 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa95d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faa98069190 0x7faa98069590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47764/0 (socket says 172.21.15.67:47764) 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa95d9b640 1 -- 172.21.15.67:0/1319184478 learned_addr learned my addr 172.21.15.67:0/1319184478 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 1 --2- 172.21.15.67:0/1319184478 >> v2:172.21.15.17:3300/0 conn(0x7faa98138160 0x7faa9813a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): get_auth_request con 0x7faa98138160 auth_method 0 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.906+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 10 monclient(hunting): handle_auth_done global_id 104706 payload 274 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 1 -- 172.21.15.67:0/1319184478 >> v1:172.21.15.67:6789/0 conn(0x7faa98069190 legacy=0x7faa98069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 1 -- 172.21.15.67:0/1319184478 >> v1:172.21.15.17:6789/0 conn(0x7faa98068dc0 legacy=0x7faa98132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 1 monclient: found mon.noname-a 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.907+0000 7faa96d9d640 1 -- 172.21.15.67:0/1319184478 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9809ff20 con 0x7faa98138160 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.909+0000 7faa96d9d640 1 -- 172.21.15.67:0/1319184478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98138160 msgr2=0x7faa9813a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.909+0000 7faa96d9d640 1 --2- 172.21.15.67:0/1319184478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98138160 0x7faa9813a540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa90009e60 tx=0x7faa90009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adda5abafd8eaa93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 1 -- 172.21.15.67:0/1319184478 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90007630 con 0x7faa98138160 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: dump: 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.965 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: _finish_auth 0 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 5 monclient: authenticate success, global_id 104706 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 1 -- 172.21.15.67:0/1319184478 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9002d050 con 0x7faa98138160 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:36.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 1 -- 172.21.15.67:0/1319184478 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa900079c0 con 0x7faa98138160 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 4 set_mon_vals no callback set 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9559a640 10 monclient: dump: 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 10 monclient: shutdown 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 -- 172.21.15.67:0/1319184478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98138160 msgr2=0x7faa9813a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/1319184478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98138160 0x7faa9813a540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa90009e60 tx=0x7faa90009780 comp rx=0 tx=0).stop 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 -- 172.21.15.67:0/1319184478 shutdown_connections 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/1319184478 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98138160 0x7faa9813a540 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 -- 172.21.15.67:0/1319184478 >> 172.21.15.67:0/1319184478 conn(0x7faa980711b0 msgr2=0x7faa980a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 -- 172.21.15.67:0/1319184478 shutdown_connections 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.910+0000 7faa9cf5d640 1 -- 172.21.15.67:0/1319184478 wait complete. 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.911+0000 7faa9cf5d640 10 monclient: build_initial_monmap 2024-09-15T02:00:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.911+0000 7faa9cf5d640 10 monclient: monmap: 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.911+0000 7faa9cf5d640 1 Processor -- start 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.911+0000 7faa9cf5d640 1 -- start start 2024-09-15T02:00:36.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.911+0000 7faa9cf5d640 10 monclient: init 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98068dc0 0x7faa9809b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-a con 0x7faa98068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 0x7faa980970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-c con 0x7faa98069190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): picked mon.noname-b con 0x7faa98138160 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9809aaa0 con 0x7faa98068dc0 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa98131940 con 0x7faa98138160 2024-09-15T02:00:36.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa980a0890 con 0x7faa98069190 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9cf5d640 10 monclient(hunting): authenticate will time out at 1269.381204s 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa95d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 msgr2=0x7faa980970c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37702/0 (socket says 172.21.15.67:37702) 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 -- 172.21.15.67:0/2624693565 learned_addr learned my addr 172.21.15.67:0/2624693565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): get_auth_request con 0x7faa98138160 auth_method 0 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa95d9b640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 0x7faa980970c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:36.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9659c640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98068dc0 0x7faa9809b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9659c640 10 monclient(hunting): get_auth_request con 0x7faa98068dc0 auth_method 0 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient(hunting): handle_auth_done global_id 124201 payload 824 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 -- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 msgr2=0x7faa980970c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 0x7faa980970c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 -- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98068dc0 msgr2=0x7faa9809b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98068dc0 0x7faa9809b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:36.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 monclient: found mon.noname-b 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.912+0000 7faa96d9d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa90009a20 con 0x7faa98138160 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa96d9d640 1 -- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 msgr2=0x7faa98097600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa96d9d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faa90009e60 tx=0x7faa90037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90007ec0 con 0x7faa98138160 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.914+0000 7faa867fc640 10 monclient: dump: 2024-09-15T02:00:36.973 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa90040070 con 0x7faa98138160 2024-09-15T02:00:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90031d10 con 0x7faa98138160 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 5 monclient: authenticate success, global_id 124201 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _renew_subs 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa98098e30 con 0x7faa98138160 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa94d99640 4 set_mon_vals no callback set 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 10 monclient: dump: 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _renew_subs 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa980994f0 con 0x7faa98138160 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faa90042030 con 0x7faa98138160 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.915+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9804bf60 con 0x7faa98138160 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.916+0000 7faa867fc640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 0x7faa7803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.916+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 217..246) v4 ==== 8816+0+0 (secure 0 0 0) 0x7faa9006e2c0 con 0x7faa98138160 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.916+0000 7faa9659c640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 0x7faa7803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.916+0000 7faa9659c640 10 monclient: get_auth_request con 0x7faa780386f0 auth_method 0 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.918+0000 7faa9659c640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 0x7faa7803aba0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faa880037d0 tx=0x7faa88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.920+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faa9003ab80 con 0x7faa98138160 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.920+0000 7faa867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:36.920+0000 7faa867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:37.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:37.090+0000 7faa9cf5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:37.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:37.090+0000 7faa9cf5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:37.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:37.090+0000 7faa9cf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:37.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:37.090+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7faa9804bf60 con 0x7faa98138160 2024-09-15T02:00:38.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.024+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v247) v1 ==== 121+0+0 (secure 0 0 0) 0x7faa9003a3a0 con 0x7faa98138160 2024-09-15T02:00:38.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.024+0000 7faa867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:38.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.024+0000 7faa867fc640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-15T02:00:38.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.131+0000 7faa9cf5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:38.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.131+0000 7faa9cf5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:38.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.131+0000 7faa9cf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.131+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7faa9804bf60 con 0x7faa98138160 2024-09-15T02:00:38.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.132+0000 7faa867fc640 1 -- 172.21.15.67:0/2624693565 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7faa9003a580 con 0x7faa98138160 2024-09-15T02:00:38.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.132+0000 7faa867fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T02:00:38.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.132+0000 7faa867fc640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-15T02:00:38.135 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h30' does not appear in the crush map 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 msgr2=0x7faa7803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 0x7faa7803aba0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7faa880037d0 tx=0x7faa88009040 comp rx=0 tx=0).stop 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 10 monclient: shutdown 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 msgr2=0x7faa98097600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faa90009e60 tx=0x7faa90037000 comp rx=0 tx=0).stop 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 shutdown_connections 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa780386f0 0x7faa7803aba0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa98138160 0x7faa98097600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa98069190 0x7faa980970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 --2- 172.21.15.67:0/2624693565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa98068dc0 0x7faa9809b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 >> 172.21.15.67:0/2624693565 conn(0x7faa980711b0 msgr2=0x7faa9809ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:38.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 shutdown_connections 2024-09-15T02:00:38.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.134+0000 7faa9cf5d640 1 -- 172.21.15.67:0/2624693565 wait complete. 2024-09-15T02:00:38.146 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:38.146 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h31 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.220+0000 7eff0924f640 10 monclient: get_monmap_and_config 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.220+0000 7eff0924f640 10 monclient: build_initial_monmap 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.220+0000 7eff0924f640 10 monclient: monmap: 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:38.221941+0000 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:38.221941+0000 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:38.224 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 Processor -- start 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 -- start start 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient: init 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff040a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff0409eea0 0x7eff0409f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-c con 0x7eff0409eea0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff04139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-b con 0x7eff04136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:38.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0406a270 con 0x7eff040a0b50 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0406f290 con 0x7eff0409eea0 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0409f7e0 con 0x7eff04136e40 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.221+0000 7eff0924f640 10 monclient(hunting): authenticate will time out at 1270.690940s 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff02d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff040a0b50 0x7eff04069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47766/0 (socket says 172.21.15.67:47766) 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0359e640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 msgr2=0x7eff04139220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff02d9d640 1 -- 172.21.15.67:0/3578825240 learned_addr learned my addr 172.21.15.67:0/3578825240 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff0409eea0 0x7eff0409f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0359e640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff04139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): get_auth_request con 0x7eff0409eea0 auth_method 0 2024-09-15T02:00:38.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient(hunting): handle_auth_done global_id 124204 payload 274 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 -- 172.21.15.67:0/3578825240 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 msgr2=0x7eff04139220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff04139220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 -- 172.21.15.67:0/3578825240 >> v1:172.21.15.67:6789/0 conn(0x7eff040a0b50 legacy=0x7eff04069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 monclient: found mon.noname-c 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.222+0000 7eff0259c640 1 -- 172.21.15.67:0/3578825240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040a0f20 con 0x7eff0409eea0 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0259c640 1 -- 172.21.15.67:0/3578825240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 msgr2=0x7eff0409f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0259c640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff0409f2a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efeec009880 tx=0x7efeec0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8ef6f003b06c6a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 1 -- 172.21.15.67:0/3578825240 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec032440 con 0x7eff0409eea0 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: dump: 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.234 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: _finish_auth 0 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 1 -- 172.21.15.67:0/3578825240 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec0325a0 con 0x7eff0409eea0 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 5 monclient: authenticate success, global_id 124204 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:38.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 1 -- 172.21.15.67:0/3578825240 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec032940 con 0x7eff0409eea0 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 4 set_mon_vals no callback set 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff01d9b640 10 monclient: dump: 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.236 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 10 monclient: shutdown 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 -- 172.21.15.67:0/3578825240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 msgr2=0x7eff0409f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff0409f2a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efeec009880 tx=0x7efeec0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 -- 172.21.15.67:0/3578825240 shutdown_connections 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff04139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 --2- 172.21.15.67:0/3578825240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff0409f2a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 -- 172.21.15.67:0/3578825240 >> 172.21.15.67:0/3578825240 conn(0x7eff04071290 msgr2=0x7eff040a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 -- 172.21.15.67:0/3578825240 shutdown_connections 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.225+0000 7eff0924f640 1 -- 172.21.15.67:0/3578825240 wait complete. 2024-09-15T02:00:38.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.226+0000 7eff0924f640 10 monclient: build_initial_monmap 2024-09-15T02:00:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.226+0000 7eff0924f640 10 monclient: monmap: 2024-09-15T02:00:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.226+0000 7eff0924f640 1 Processor -- start 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.226+0000 7eff0924f640 1 -- start start 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.226+0000 7eff0924f640 10 monclient: init 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-b con 0x7eff0409eea0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 0x7eff0411d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-a con 0x7eff040a0b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff0411dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): picked mon.noname-c con 0x7eff04136e40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04137720 con 0x7eff040a0b50 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040a12d0 con 0x7eff0409eea0 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff040a3450 con 0x7eff04136e40 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0924f640 10 monclient(hunting): authenticate will time out at 1270.696269s 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37722/0 (socket says 172.21.15.67:37722) 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 -- 172.21.15.67:0/2610990693 learned_addr learned my addr 172.21.15.67:0/2610990693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): get_auth_request con 0x7eff0409eea0 auth_method 0 2024-09-15T02:00:38.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0359e640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 msgr2=0x7eff0411dbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0359e640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff0411dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0259c640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 0x7eff0411d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient(hunting): handle_auth_done global_id 124207 payload 824 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 msgr2=0x7eff0411dbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff0411dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 msgr2=0x7eff0411d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:38.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 0x7eff0411d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 monclient: found mon.noname-b 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff02d9d640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec009390 con 0x7eff0409eea0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0259c640 10 monclient: get_auth_request con 0x7eff040a0b50 auth_method 0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.227+0000 7eff0259c640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 0x7eff0411d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7eff02d9d640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 msgr2=0x7eff04134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7eff02d9d640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efef400cca0 tx=0x7efef40053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7eff02d9d640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efef400cca0 tx=0x7efef40053f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7efef400abe0 config(0 keys) v1 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef400aa60 con 0x7eff0409eea0 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 10 monclient: dump: 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef400abe0 con 0x7eff0409eea0 2024-09-15T02:00:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.229+0000 7eff0924f640 5 monclient: authenticate success, global_id 124207 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _renew_subs 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff04120fd0 con 0x7eff0409eea0 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0159a640 4 set_mon_vals no callback set 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _renew_subs 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff04120cf0 con 0x7eff0409eea0 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef40070f0 con 0x7eff0409eea0 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 10 monclient: dump: 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:38.245 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efef40074e0 con 0x7eff0409eea0 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.230+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff04142130 con 0x7eff0409eea0 2024-09-15T02:00:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.231+0000 7efefaffd640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 0x7efee403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.231+0000 7eff0259c640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 0x7efee403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.231+0000 7eff0259c640 10 monclient: get_auth_request con 0x7efee40386f0 auth_method 0 2024-09-15T02:00:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.233+0000 7eff0259c640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 0x7efee403aba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efeec031000 tx=0x7efeec032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.235+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(247..247 src has 217..247) v4 ==== 8603+0+0 (secure 0 0 0) 0x7efef404cf10 con 0x7eff0409eea0 2024-09-15T02:00:38.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.852+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efef401a360 con 0x7eff0409eea0 2024-09-15T02:00:38.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.852+0000 7efefaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:38.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:38.852+0000 7efefaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:39.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:39.020+0000 7eff0924f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:39.020+0000 7eff0924f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:39.020+0000 7eff0924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:39.020+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7eff0404bf60 con 0x7eff0409eea0 2024-09-15T02:00:40.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.025+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v248) v1 ==== 121+0+0 (secure 0 0 0) 0x7efef4010af0 con 0x7eff0409eea0 2024-09-15T02:00:40.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.025+0000 7efefaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:40.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.025+0000 7efefaffd640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-15T02:00:40.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.129+0000 7eff0924f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:40.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.129+0000 7eff0924f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:40.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.129+0000 7eff0924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.129+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7eff0411f230 con 0x7eff0409eea0 2024-09-15T02:00:40.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.131+0000 7efefaffd640 1 -- 172.21.15.67:0/2610990693 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7efef401a360 con 0x7eff0409eea0 2024-09-15T02:00:40.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.131+0000 7efefaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T02:00:40.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.131+0000 7efefaffd640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-15T02:00:40.133 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h31' does not appear in the crush map 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 msgr2=0x7efee403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 0x7efee403aba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efeec031000 tx=0x7efeec032000 comp rx=0 tx=0).stop 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 10 monclient: shutdown 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 msgr2=0x7eff04134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:40.135 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.132+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efef400cca0 tx=0x7efef40053f0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 shutdown_connections 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efee40386f0 0x7efee403aba0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7eff040a0b50 0x7eff0411d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7eff04136e40 0x7eff0411dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 --2- 172.21.15.67:0/2610990693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0409eea0 0x7eff04134be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 >> 172.21.15.67:0/2610990693 conn(0x7eff04071290 msgr2=0x7eff04135e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 shutdown_connections 2024-09-15T02:00:40.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.133+0000 7eff0924f640 1 -- 172.21.15.67:0/2610990693 wait complete. 2024-09-15T02:00:40.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in `seq 1 32` 2024-09-15T02:00:40.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm h32 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.219+0000 7f3f828c0640 10 monclient: get_monmap_and_config 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.219+0000 7f3f828c0640 10 monclient: build_initial_monmap 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.219+0000 7f3f828c0640 10 monclient: monmap: 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:40.221082+0000 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:40.221082+0000 2024-09-15T02:00:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:40.223 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.220+0000 7f3f828c0640 1 Processor -- start 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.220+0000 7f3f828c0640 1 -- start start 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.220+0000 7f3f828c0640 10 monclient: init 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f7c0a0b50 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f7c09eea0 0x7f3f7c09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c09eea0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f7c136e40 0x7f3f7c139220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c136e40 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c06a270 con 0x7f3f7c0a0b50 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c06f290 con 0x7f3f7c136e40 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c09f7e0 con 0x7f3f7c09eea0 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f828c0640 10 monclient(hunting): authenticate will time out at 1272.690354s 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f7c09eea0 0x7f3f7c09f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3f7c0a0b50 0x7f3f7c069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47778/0 (socket says 172.21.15.67:47778) 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7bfff640 1 -- 172.21.15.67:0/3950786749 learned_addr learned my addr 172.21.15.67:0/3950786749 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f7c09eea0 auth_method 0 2024-09-15T02:00:40.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f80e5d640 1 --2- 172.21.15.67:0/3950786749 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c136e40 0x7f3f7c139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f80e5d640 10 monclient(hunting): get_auth_request con 0x7f3f7c136e40 auth_method 0 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f80e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f80e5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f80e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_done global_id 124210 payload 274 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 -- 172.21.15.67:0/3950786749 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c136e40 msgr2=0x7f3f7c139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 --2- 172.21.15.67:0/3950786749 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c136e40 0x7f3f7c139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 -- 172.21.15.67:0/3950786749 >> v1:172.21.15.67:6789/0 conn(0x7f3f7c0a0b50 legacy=0x7f3f7c069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 monclient: found mon.noname-c 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.221+0000 7f3f7b7fe640 1 -- 172.21.15.67:0/3950786749 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0a0f20 con 0x7f3f7c09eea0 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7b7fe640 1 -- 172.21.15.67:0/3950786749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09eea0 msgr2=0x7f3f7c09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7b7fe640 1 --2- 172.21.15.67:0/3950786749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c09f2a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0a08b0 tx=0x7f3f700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f64875b04545b66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7affd640 1 -- 172.21.15.67:0/3950786749 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7002ae00 con 0x7f3f7c09eea0 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.224+0000 7f3f7affd640 10 monclient: dump: 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.235 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 10 monclient: _finish_auth 0 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 1 -- 172.21.15.67:0/3950786749 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70032440 con 0x7f3f7c09eea0 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 5 monclient: authenticate success, global_id 124210 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:40.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 4 set_mon_vals no callback set 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 1 -- 172.21.15.67:0/3950786749 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700327e0 con 0x7f3f7c09eea0 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f7affd640 10 monclient: dump: 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 10 monclient: shutdown 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 -- 172.21.15.67:0/3950786749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09eea0 msgr2=0x7f3f7c09f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 --2- 172.21.15.67:0/3950786749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c09f2a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0a08b0 tx=0x7f3f700090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 -- 172.21.15.67:0/3950786749 shutdown_connections 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 --2- 172.21.15.67:0/3950786749 >> v2:172.21.15.17:3300/0 conn(0x7f3f7c136e40 0x7f3f7c139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 --2- 172.21.15.67:0/3950786749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c09f2a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 -- 172.21.15.67:0/3950786749 >> 172.21.15.67:0/3950786749 conn(0x7f3f7c071290 msgr2=0x7f3f7c0a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 -- 172.21.15.67:0/3950786749 shutdown_connections 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.225+0000 7f3f828c0640 1 -- 172.21.15.67:0/3950786749 wait complete. 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.226+0000 7f3f828c0640 10 monclient: build_initial_monmap 2024-09-15T02:00:40.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.226+0000 7f3f828c0640 10 monclient: monmap: 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.226+0000 7f3f828c0640 1 Processor -- start 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.226+0000 7f3f828c0640 1 -- start start 2024-09-15T02:00:40.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.226+0000 7f3f828c0640 10 monclient: init 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c09eea0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 0x7f3f7c11e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a0b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c136e40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a1260 con 0x7f3f7c09eea0 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c09e280 con 0x7f3f7c136e40 2024-09-15T02:00:40.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c121b40 con 0x7f3f7c0a0b50 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f828c0640 10 monclient(hunting): authenticate will time out at 1272.696203s 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7b7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 msgr2=0x7f3f7c11e810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37734/0 (socket says 172.21.15.67:37734) 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 0x7f3f7c11e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 -- 172.21.15.67:0/48683492 learned_addr learned my addr 172.21.15.67:0/48683492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): get_auth_request con 0x7f3f7c136e40 auth_method 0 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:40.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7bfff640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7bfff640 10 monclient(hunting): get_auth_request con 0x7f3f7c09eea0 auth_method 0 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient(hunting): handle_auth_done global_id 124213 payload 824 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 -- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 msgr2=0x7f3f7c11e810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 0x7f3f7c11e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 -- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09eea0 msgr2=0x7f3f7c135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:40.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c135d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 monclient: found mon.noname-b 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.227+0000 7f3f80e5d640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70009390 con 0x7f3f7c136e40 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.229+0000 7f3f80e5d640 1 -- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 msgr2=0x7f3f7c11ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.229+0000 7f3f80e5d640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f74007f30 tx=0x7f3f740098e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.229+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7400ee30 con 0x7f3f7c136e40 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: dump: 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:40.243 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 5 monclient: authenticate success, global_id 124213 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _renew_subs 2024-09-15T02:00:40.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c120f30 con 0x7f3f7c136e40 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f74013590 con 0x7f3f7c136e40 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74013950 con 0x7f3f7c136e40 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 10 monclient: dump: 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:40.245 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f7a7fc640 4 set_mon_vals no callback set 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _renew_subs 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c120300 con 0x7f3f7c136e40 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f74013d40 con 0x7f3f7c136e40 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.230+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c1222b0 con 0x7f3f7c136e40 2024-09-15T02:00:40.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.231+0000 7f3f63fff640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 0x7f3f6803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.231+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(248..248 src has 217..248) v4 ==== 8390+0+0 (secure 0 0 0) 0x7f3f7404d140 con 0x7f3f7c136e40 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.231+0000 7f3f7bfff640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 0x7f3f6803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.231+0000 7f3f7bfff640 10 monclient: get_auth_request con 0x7f3f680386f0 auth_method 0 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.233+0000 7f3f7bfff640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 0x7f3f6803aba0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c007fd0 tx=0x7f3f6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.235+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f7401ecb0 con 0x7f3f7c136e40 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.235+0000 7f3f63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:40.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.235+0000 7f3f63fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:40.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.403+0000 7f3f828c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.403+0000 7f3f828c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.403+0000 7f3f828c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:40.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:40.403+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f3f7c04bf60 con 0x7f3f7c136e40 2024-09-15T02:00:41.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.030+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v249) v1 ==== 121+0+0 (secure 0 0 0) 0x7f3f7401b310 con 0x7f3f7c136e40 2024-09-15T02:00:41.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.030+0000 7f3f63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:41.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.030+0000 7f3f63fff640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-15T02:00:41.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.138+0000 7f3f828c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:41.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.138+0000 7f3f828c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:41.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.138+0000 7f3f828c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.138+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f3f7c0008d0 con 0x7f3f7c136e40 2024-09-15T02:00:41.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.139+0000 7f3f63fff640 1 -- 172.21.15.67:0/48683492 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3f740122e0 con 0x7f3f7c136e40 2024-09-15T02:00:41.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.139+0000 7f3f63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T02:00:41.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.139+0000 7f3f63fff640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-15T02:00:41.142 INFO:tasks.workunit.client.0.smithi067.stderr:device 'h32' does not appear in the crush map 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 msgr2=0x7f3f6803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 0x7f3f6803aba0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c007fd0 tx=0x7f3f6c009040 comp rx=0 tx=0).stop 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 10 monclient: shutdown 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 msgr2=0x7f3f7c11ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f74007f30 tx=0x7f3f740098e0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 shutdown_connections 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f680386f0 0x7f3f6803aba0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f7c136e40 0x7f3f7c11ed50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f7c0a0b50 0x7f3f7c11e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 --2- 172.21.15.67:0/48683492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f7c09eea0 0x7f3f7c135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 >> 172.21.15.67:0/48683492 conn(0x7f3f7c071290 msgr2=0x7f3f7c09ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:41.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 shutdown_connections 2024-09-15T02:00:41.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.141+0000 7f3f828c0640 1 -- 172.21.15.67:0/48683492 wait complete. 2024-09-15T02:00:41.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush rm r1 2024-09-15T02:00:41.230 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.228+0000 7fd4f5d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.228+0000 7fd4f5d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.228+0000 7fd4f5d9d640 10 monclient: monmap: 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:41.229672+0000 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:41.229672+0000 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:41.232 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:41.232 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:41.232 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 Processor -- start 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 -- start start 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient: init 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 0x7fd4f006e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4f00a3850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4f006a640 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4f009e220 0x7fd4f0139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4f009e220 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f0069c60 con 0x7fd4f006a640 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:41.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f0069670 con 0x7fd4f009e220 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00a3c20 con 0x7fd4f00a3850 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4f5d9d640 10 monclient(hunting): authenticate will time out at 1273.698787s 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4ef7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 msgr2=0x7fd4f006e210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4ef7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 0x7fd4f006e210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4f009e220 0x7fd4f0139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4f009e220 0x7fd4f0139d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45528/0 (socket says 172.21.15.67:45528) 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 1 -- 172.21.15.67:0/4136051252 learned_addr learned my addr 172.21.15.67:0/4136051252 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 10 monclient(hunting): get_auth_request con 0x7fd4f009e220 auth_method 0 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:41.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.229+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:41.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4eeffd640 1 -- 172.21.15.67:0/4136051252 >> v1:172.21.15.67:6789/0 conn(0x7fd4f006a640 legacy=0x7fd4f009db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:41.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:41.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:41.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:41.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient(hunting): handle_auth_done global_id 124216 payload 274 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 1 -- 172.21.15.67:0/4136051252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 msgr2=0x7fd4f006e210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 1 --2- 172.21.15.67:0/4136051252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 0x7fd4f006e210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 1 -- 172.21.15.67:0/4136051252 >> v1:172.21.15.67:6789/0 conn(0x7fd4f006a640 legacy=0x7fd4f009db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 1 monclient: found mon.noname-c 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.232+0000 7fd4effff640 1 -- 172.21.15.67:0/4136051252 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f0069980 con 0x7fd4f009e220 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4effff640 1 -- 172.21.15.67:0/4136051252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f009e220 msgr2=0x7fd4f0139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4effff640 1 --2- 172.21.15.67:0/4136051252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f009e220 0x7fd4f0139d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009fa0 tx=0x7fd4e8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e42ef345fd96df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 1 -- 172.21.15.67:0/4136051252 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8007610 con 0x7fd4f009e220 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:41.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: dump: 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:41.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:41.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 1 -- 172.21.15.67:0/4136051252 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e802d050 con 0x7fd4f009e220 2024-09-15T02:00:41.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 5 monclient: authenticate success, global_id 124216 2024-09-15T02:00:41.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 1 -- 172.21.15.67:0/4136051252 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80079a0 con 0x7fd4f009e220 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 4 set_mon_vals no callback set 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4ee7fc640 10 monclient: dump: 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.242 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 10 monclient: shutdown 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/4136051252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f009e220 msgr2=0x7fd4f0139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/4136051252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f009e220 0x7fd4f0139d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009fa0 tx=0x7fd4e8009840 comp rx=0 tx=0).stop 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/4136051252 shutdown_connections 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/4136051252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f009e220 0x7fd4f0139d20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/4136051252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f00a3850 0x7fd4f006e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/4136051252 >> 172.21.15.67:0/4136051252 conn(0x7fd4f0070fb0 msgr2=0x7fd4f00a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:41.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/4136051252 shutdown_connections 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.235+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/4136051252 wait complete. 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.236+0000 7fd4f5d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.236+0000 7fd4f5d9d640 10 monclient: monmap: 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:41.244 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.236+0000 7fd4f5d9d640 1 Processor -- start 2024-09-15T02:00:41.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.236+0000 7fd4f5d9d640 1 -- start start 2024-09-15T02:00:41.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.236+0000 7fd4f5d9d640 10 monclient: init 2024-09-15T02:00:41.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4f006a640 0x7fd4f0134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4f006a640 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 0x7fd4f011e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4f009e220 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4f00a3850 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0135b00 con 0x7fd4f006a640 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f006ea50 con 0x7fd4f00a3850 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00a08c0 con 0x7fd4f009e220 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4f5d9d640 10 monclient(hunting): authenticate will time out at 1273.706275s 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4eeffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 msgr2=0x7fd4f011e8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45536/0 (socket says 172.21.15.67:45536) 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4eeffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 0x7fd4f011e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 -- 172.21.15.67:0/1702837320 learned_addr learned my addr 172.21.15.67:0/1702837320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): get_auth_request con 0x7fd4f00a3850 auth_method 0 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:41.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient(hunting): handle_auth_done global_id 124222 payload 824 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 -- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 msgr2=0x7fd4f011e8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 0x7fd4f011e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 -- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4f006a640 msgr2=0x7fd4f0134d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4f006a640 0x7fd4f0134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 monclient: found mon.noname-b 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.237+0000 7fd4effff640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009ae0 con 0x7fd4f00a3850 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.239+0000 7fd4effff640 1 -- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 msgr2=0x7fd4f011ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.239+0000 7fd4effff640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009ab0 tx=0x7fd4e8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.239+0000 7fd4effff640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009ab0 tx=0x7fd4e8036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fd4e8031cb0 mon_map magic: 0 v1 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8031cb0 con 0x7fd4f00a3850 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: dump: 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e8031e10 con 0x7fd4f00a3850 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 5 monclient: authenticate success, global_id 124222 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:41.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 10 monclient: _renew_subs 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f0120f90 con 0x7fd4f00a3850 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4edffb640 4 set_mon_vals no callback set 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 10 monclient: _renew_subs 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f0141e50 con 0x7fd4f00a3850 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e803c670 con 0x7fd4f00a3850 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:41.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 10 monclient: dump: 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.240+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd4e803ca20 con 0x7fd4f00a3850 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4f5d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4f5d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:41.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4f5d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f004bf60 con 0x7fd4f00a3850 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4d77fe640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 0x7fd4dc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 0x7fd4dc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.241+0000 7fd4ef7fe640 10 monclient: get_auth_request con 0x7fd4dc038740 auth_method 0 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.243+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 0x7fd4dc03abf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0006cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.245+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(249..249 src has 217..249) v4 ==== 8177+0+0 (secure 0 0 0) 0x7fd4e806d7f0 con 0x7fd4f00a3850 2024-09-15T02:00:41.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.862+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd4e803b380 con 0x7fd4f00a3850 2024-09-15T02:00:41.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.862+0000 7fd4d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:41.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:41.862+0000 7fd4d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:42.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:42.037+0000 7fd4f5d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:42.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:42.037+0000 7fd4f5d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:42.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:42.037+0000 7fd4f5d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:42.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:42.037+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7fd4f0069f40 con 0x7fd4f00a3850 2024-09-15T02:00:43.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.039+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v250) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd4e8040030 con 0x7fd4f00a3850 2024-09-15T02:00:43.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.039+0000 7fd4d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:43.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.039+0000 7fd4d77fe640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-15T02:00:43.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.146+0000 7fd4f5d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:43.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.146+0000 7fd4f5d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:43.148 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.146+0000 7fd4f5d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.146+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7fd4f0067140 con 0x7fd4f00a3850 2024-09-15T02:00:43.149 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.147+0000 7fd4d77fe640 1 -- 172.21.15.67:0/1702837320 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v250) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd4e8040020 con 0x7fd4f00a3850 2024-09-15T02:00:43.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.147+0000 7fd4d77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T02:00:43.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.147+0000 7fd4d77fe640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-15T02:00:43.150 INFO:tasks.workunit.client.0.smithi067.stderr:device 'r1' does not appear in the crush map 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 msgr2=0x7fd4dc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 0x7fd4dc03abf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0006cf0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 10 monclient: shutdown 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 msgr2=0x7fd4f011ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009ab0 tx=0x7fd4e8036000 comp rx=0 tx=0).stop 2024-09-15T02:00:43.152 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 shutdown_connections 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd4dc038740 0x7fd4dc03abf0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f00a3850 0x7fd4f011ee00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd4f009e220 0x7fd4f011e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 --2- 172.21.15.67:0/1702837320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd4f006a640 0x7fd4f0134d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.149+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 >> 172.21.15.67:0/1702837320 conn(0x7fd4f0070fb0 msgr2=0x7fd4f006d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.150+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 shutdown_connections 2024-09-15T02:00:43.157 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.150+0000 7fd4f5d9d640 1 -- 172.21.15.67:0/1702837320 wait complete. 2024-09-15T02:00:43.161 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T02:00:43.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.236+0000 7fdc83878640 10 monclient: get_monmap_and_config 2024-09-15T02:00:43.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.236+0000 7fdc83878640 10 monclient: build_initial_monmap 2024-09-15T02:00:43.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.236+0000 7fdc83878640 10 monclient: monmap: 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:43.237446+0000 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:43.237446+0000 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:43.239 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.236+0000 7fdc83878640 1 Processor -- start 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 -- start start 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient: init 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdc7c098140 0x7fdc7c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c098140 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc7c0a0370 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 0x7fdc7c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c098c20 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0a0b80 con 0x7fdc7c0a0370 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0a00b0 con 0x7fdc7c098140 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c071390 con 0x7fdc7c098c20 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc83878640 10 monclient(hunting): authenticate will time out at 1275.706602s 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 0x7fdc7c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 0x7fdc7c1390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45552/0 (socket says 172.21.15.67:45552) 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 1 -- 172.21.15.67:0/3016248335 learned_addr learned my addr 172.21.15.67:0/3016248335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): get_auth_request con 0x7fdc7c098c20 auth_method 0 2024-09-15T02:00:43.243 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc80e13640 1 --1- 172.21.15.67:0/3016248335 >> v1:172.21.15.67:6789/0 conn(0x7fdc7c0a0370 0x7fdc7c098510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49114/0 (socket says 172.21.15.67:49114) 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81614640 1 --2- 172.21.15.67:0/3016248335 >> v2:172.21.15.17:3300/0 conn(0x7fdc7c098140 0x7fdc7c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81e15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81614640 10 monclient(hunting): get_auth_request con 0x7fdc7c098140 auth_method 0 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81614640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81614640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.237+0000 7fdc81614640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:43.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient(hunting): handle_auth_done global_id 104718 payload 274 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 1 -- 172.21.15.67:0/3016248335 >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 msgr2=0x7fdc7c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 1 --2- 172.21.15.67:0/3016248335 >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 0x7fdc7c1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 1 -- 172.21.15.67:0/3016248335 >> v1:172.21.15.67:6789/0 conn(0x7fdc7c0a0370 legacy=0x7fdc7c098510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient: _finish_hunting 0 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 1 monclient: found mon.noname-a 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:43.245 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.238+0000 7fdc81614640 1 -- 172.21.15.67:0/3016248335 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c071500 con 0x7fdc7c098140 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc81614640 1 -- 172.21.15.67:0/3016248335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098140 msgr2=0x7fdc7c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc81614640 1 --2- 172.21.15.67:0/3016248335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098140 0x7fdc7c1321e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a1b60 tx=0x7fdc780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879ea3ade19fb0f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 1 -- 172.21.15.67:0/3016248335 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7802ae00 con 0x7fdc7c098140 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: dump: 2024-09-15T02:00:43.250 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 5 monclient: authenticate success, global_id 104718 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:43.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 1 -- 172.21.15.67:0/3016248335 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc78032440 con 0x7fdc7c098140 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 4 set_mon_vals no callback set 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 1 -- 172.21.15.67:0/3016248335 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78032820 con 0x7fdc7c098140 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc73fff640 10 monclient: dump: 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:43.252 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 10 monclient: shutdown 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 -- 172.21.15.67:0/3016248335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098140 msgr2=0x7fdc7c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 --2- 172.21.15.67:0/3016248335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098140 0x7fdc7c1321e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a1b60 tx=0x7fdc780090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 -- 172.21.15.67:0/3016248335 shutdown_connections 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 --2- 172.21.15.67:0/3016248335 >> v2:172.21.15.67:3300/0 conn(0x7fdc7c098c20 0x7fdc7c1390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 --2- 172.21.15.67:0/3016248335 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098140 0x7fdc7c1321e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.241+0000 7fdc83878640 1 -- 172.21.15.67:0/3016248335 >> 172.21.15.67:0/3016248335 conn(0x7fdc7c0a5780 msgr2=0x7fdc7c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 1 -- 172.21.15.67:0/3016248335 shutdown_connections 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 1 -- 172.21.15.67:0/3016248335 wait complete. 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 10 monclient: build_initial_monmap 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 10 monclient: monmap: 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:43.254 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 1 Processor -- start 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 1 -- start start 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.242+0000 7fdc83878640 10 monclient: init 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c098140 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 0x7fdc7c1310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c098c20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 0x7fdc7c1315e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c0a0370 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c099090 con 0x7fdc7c098c20 2024-09-15T02:00:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0a3940 con 0x7fdc7c098140 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c134d90 con 0x7fdc7c0a0370 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc83878640 10 monclient(hunting): authenticate will time out at 1275.712605s 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45564/0 (socket says 172.21.15.67:45564) 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 -- 172.21.15.67:0/2025216550 learned_addr learned my addr 172.21.15.67:0/2025216550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): get_auth_request con 0x7fdc7c098140 auth_method 0 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:43.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81e15640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 msgr2=0x7fdc7c1315e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81e15640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 0x7fdc7c1315e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc80e13640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 0x7fdc7c1310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient(hunting): handle_auth_done global_id 124228 payload 824 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 msgr2=0x7fdc7c1315e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 0x7fdc7c1315e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 msgr2=0x7fdc7c1310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 0x7fdc7c1310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient: _finish_hunting 0 2024-09-15T02:00:43.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 monclient: found mon.noname-b 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc81614640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78009390 con 0x7fdc7c098140 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc80e13640 10 monclient: get_auth_request con 0x7fdc7c098c20 auth_method 0 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.243+0000 7fdc80e13640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 0x7fdc7c1310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc81614640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 msgr2=0x7fdc7c130b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc81614640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc78009880 tx=0x7fdc78032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc780319f0 con 0x7fdc7c098140 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: dump: 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:43.258 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc78031b50 con 0x7fdc7c098140 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7803c410 con 0x7fdc7c098140 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 5 monclient: authenticate success, global_id 124228 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc717fa640 10 monclient: dump: 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:43.260 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 10 monclient: _renew_subs 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc737fe640 4 set_mon_vals no callback set 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c138040 con 0x7fdc7c098140 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 10 monclient: _renew_subs 2024-09-15T02:00:43.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.246+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c132f80 con 0x7fdc7c098140 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc83878640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc83878640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc83878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdc78039070 con 0x7fdc7c098140 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c04bf60 con 0x7fdc7c098140 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc717fa640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 0x7fdc6003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 224..250) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fdc7806ccc0 con 0x7fdc7c098140 2024-09-15T02:00:43.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc80e13640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 0x7fdc6003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:43.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.247+0000 7fdc80e13640 10 monclient: get_auth_request con 0x7fdc600386f0 auth_method 0 2024-09-15T02:00:43.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.250+0000 7fdc80e13640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 0x7fdc6003aba0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a0760 tx=0x7fdc6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:43.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.251+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdc7803bb50 con 0x7fdc7c098140 2024-09-15T02:00:43.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.251+0000 7fdc717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:43.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.251+0000 7fdc717fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:43.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.422+0000 7fdc83878640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:43.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.422+0000 7fdc83878640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:43.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.422+0000 7fdc83878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:43.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:43.422+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdc7c132720 con 0x7fdc7c098140 2024-09-15T02:00:44.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.048+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v251) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdc78040070 con 0x7fdc7c098140 2024-09-15T02:00:44.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.048+0000 7fdc717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:44.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.048+0000 7fdc717fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:44.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.153+0000 7fdc83878640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:44.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.153+0000 7fdc83878640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:44.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.153+0000 7fdc83878640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:44.156 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:44.153+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdc7c132720 con 0x7fdc7c098140 2024-09-15T02:00:45.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.050+0000 7fdc717fa640 1 -- 172.21.15.67:0/2025216550 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v252) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdc78037030 con 0x7fdc7c098140 2024-09-15T02:00:45.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.050+0000 7fdc717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T02:00:45.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.050+0000 7fdc717fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 msgr2=0x7fdc6003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 0x7fdc6003aba0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0a0760 tx=0x7fdc6c009040 comp rx=0 tx=0).stop 2024-09-15T02:00:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 10 monclient: shutdown 2024-09-15T02:00:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 msgr2=0x7fdc7c130b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.052+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdc78009880 tx=0x7fdc78032000 comp rx=0 tx=0).stop 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 shutdown_connections 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc600386f0 0x7fdc6003aba0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc7c098c20 0x7fdc7c1310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc7c0a0370 0x7fdc7c1315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 --2- 172.21.15.67:0/2025216550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c098140 0x7fdc7c130b60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 >> 172.21.15.67:0/2025216550 conn(0x7fdc7c0a5780 msgr2=0x7fdc7c09b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 shutdown_connections 2024-09-15T02:00:45.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.053+0000 7fdc83878640 1 -- 172.21.15.67:0/2025216550 wait complete. 2024-09-15T02:00:45.067 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo OK 2024-09-15T02:00:45.068 INFO:tasks.workunit.client.0.smithi067.stdout:OK 2024-09-15T02:00:45.068 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:00:45.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:00:45.099 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-15T02:00:45.100 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-15T02:00:45.101 DEBUG:teuthology.orchestra.run.smithi067:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-15T02:00:45.161 INFO:tasks.workunit.client.0.smithi067.stderr:+ set -e 2024-09-15T02:00:45.161 INFO:tasks.workunit.client.0.smithi067.stderr:++ uuidgen 2024-09-15T02:00:45.172 INFO:tasks.workunit.client.0.smithi067.stderr:+ ua=7b19bad9-2bed-4846-b592-7009ac3d12e0 2024-09-15T02:00:45.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ uuidgen 2024-09-15T02:00:45.172 INFO:tasks.workunit.client.0.smithi067.stderr:+ ub=230aff7d-6869-4287-a1fa-cd4abe8412b2 2024-09-15T02:00:45.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 7b19bad9-2bed-4846-b592-7009ac3d12e0 2024-09-15T02:00:45.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 10 monclient: get_monmap_and_config 2024-09-15T02:00:45.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 10 monclient: build_initial_monmap 2024-09-15T02:00:45.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 10 monclient: monmap: 2024-09-15T02:00:45.251 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:45.250339+0000 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:45.250339+0000 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:45.252 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 1 Processor -- start 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 1 -- start start 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.249+0000 7f83f44af640 10 monclient: init 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 0x7f83ec06a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.255 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-c con 0x7f83ec0a36e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83ec06bdc0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f83ec136bb0 0x7f83ec138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-a con 0x7f83ec136bb0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ec09f640 con 0x7f83ec06bdc0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ec099fa0 con 0x7f83ec136bb0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ec069f00 con 0x7f83ec0a36e0 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:45.256 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f44af640 10 monclient(hunting): authenticate will time out at 1277.719472s 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 0x7f83ec06a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 0x7f83ec06a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45572/0 (socket says 172.21.15.67:45572) 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 1 -- 172.21.15.67:0/360558272 learned_addr learned my addr 172.21.15.67:0/360558272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 10 monclient(hunting): get_auth_request con 0x7f83ec0a36e0 auth_method 0 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f224b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/360558272 >> v2:172.21.15.17:3300/0 conn(0x7f83ec136bb0 0x7f83ec138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): get_auth_request con 0x7f83ec136bb0 auth_method 0 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.250+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_done global_id 104721 payload 274 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 1 -- 172.21.15.67:0/360558272 >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 msgr2=0x7f83ec06a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/360558272 >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 0x7f83ec06a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 1 -- 172.21.15.67:0/360558272 >> v1:172.21.15.67:6789/0 conn(0x7f83ec06bdc0 legacy=0x7f83ec06e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 1 monclient: found mon.noname-a 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:45.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:45.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.251+0000 7f83f2a4c640 1 -- 172.21.15.67:0/360558272 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec069c20 con 0x7f83ec136bb0 2024-09-15T02:00:45.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f2a4c640 1 -- 172.21.15.67:0/360558272 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec136bb0 msgr2=0x7f83ec138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:45.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/360558272 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec136bb0 0x7f83ec138f90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83ec09eae0 tx=0x7f83e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44746c735d277f0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 1 -- 172.21.15.67:0/360558272 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e4007760 con 0x7f83ec136bb0 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: dump: 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: _finish_auth 0 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 1 -- 172.21.15.67:0/360558272 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e402d050 con 0x7f83ec136bb0 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f44af640 5 monclient: authenticate success, global_id 104721 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f44af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f44af640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 1 -- 172.21.15.67:0/360558272 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e4007af0 con 0x7f83ec136bb0 2024-09-15T02:00:45.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f44af640 4 set_mon_vals no callback set 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.254+0000 7f83f1249640 10 monclient: dump: 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:45.267 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 10 monclient: shutdown 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 -- 172.21.15.67:0/360558272 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec136bb0 msgr2=0x7f83ec138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 --2- 172.21.15.67:0/360558272 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec136bb0 0x7f83ec138f90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83ec09eae0 tx=0x7f83e4009780 comp rx=0 tx=0).stop 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 -- 172.21.15.67:0/360558272 shutdown_connections 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 --2- 172.21.15.67:0/360558272 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec136bb0 0x7f83ec138f90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 --2- 172.21.15.67:0/360558272 >> v2:172.21.15.67:3300/0 conn(0x7f83ec0a36e0 0x7f83ec06a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 -- 172.21.15.67:0/360558272 >> 172.21.15.67:0/360558272 conn(0x7f83ec071020 msgr2=0x7f83ec0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 -- 172.21.15.67:0/360558272 shutdown_connections 2024-09-15T02:00:45.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 1 -- 172.21.15.67:0/360558272 wait complete. 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 10 monclient: build_initial_monmap 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.255+0000 7f83f44af640 10 monclient: monmap: 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:45.269 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 Processor -- start 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 -- start start 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient: init 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec06bdc0 0x7f83ec097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-a con 0x7f83ec06bdc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 0x7f83ec09b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-c con 0x7f83ec0a36e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): picked mon.noname-b con 0x7f83ec136bb0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec06ea30 con 0x7f83ec06bdc0 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0a3450 con 0x7f83ec136bb0 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec098830 con 0x7f83ec0a36e0 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f44af640 10 monclient(hunting): authenticate will time out at 1277.725614s 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f1a4a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 msgr2=0x7f83ec09b1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45588/0 (socket says 172.21.15.67:45588) 2024-09-15T02:00:45.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 1 -- 172.21.15.67:0/1564637701 learned_addr learned my addr 172.21.15.67:0/1564637701 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): get_auth_request con 0x7f83ec136bb0 auth_method 0 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f1a4a640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 0x7f83ec09b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f224b640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec06bdc0 0x7f83ec097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f224b640 10 monclient(hunting): get_auth_request con 0x7f83ec06bdc0 auth_method 0 2024-09-15T02:00:45.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f224b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f224b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.256+0000 7f83f224b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f224b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f224b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f224b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 10 monclient(hunting): handle_auth_done global_id 124234 payload 824 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 -- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 msgr2=0x7f83ec09b1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 0x7f83ec09b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 -- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec06bdc0 msgr2=0x7f83ec097790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec06bdc0 0x7f83ec097790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:45.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 monclient: found mon.noname-b 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.257+0000 7f83f2a4c640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e4009a20 con 0x7f83ec136bb0 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f2a4c640 1 -- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 msgr2=0x7f83ec09b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f2a4c640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f83e40097c0 tx=0x7f83e4037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e4031ce0 con 0x7f83ec136bb0 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: dump: 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:45.274 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:45.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 5 monclient: authenticate success, global_id 124234 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 10 monclient: _renew_subs 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec09cd10 con 0x7f83ec136bb0 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e4040070 con 0x7f83ec136bb0 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e4045690 con 0x7f83ec136bb0 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83e27fc640 10 monclient: dump: 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:45.276 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f0a48640 4 set_mon_vals no callback set 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 10 monclient: _renew_subs 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.259+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec096fa0 con 0x7f83ec136bb0 2024-09-15T02:00:45.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f83e4042030 con 0x7f83ec136bb0 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83f44af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83f44af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83f44af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ec04bf60 con 0x7f83ec136bb0 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83e27fc640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 0x7f83d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.260+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(252..252 src has 224..252) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f83e406ddf0 con 0x7f83ec136bb0 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.261+0000 7f83f224b640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 0x7f83d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:45.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.261+0000 7f83f224b640 10 monclient: get_auth_request con 0x7f83d00386f0 auth_method 0 2024-09-15T02:00:45.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.263+0000 7f83f224b640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 0x7f83d003aba0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f83e800bca0 tx=0x7f83e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:45.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.265+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f83e4071030 con 0x7f83ec136bb0 2024-09-15T02:00:45.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.265+0000 7f83e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:45.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.265+0000 7f83e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:45.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.438+0000 7f83f44af640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.438+0000 7f83f44af640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.438+0000 7f83f44af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:45.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:45.438+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f83ec069940 con 0x7f83ec136bb0 2024-09-15T02:00:46.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.057+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v253) v1 ==== 106+0+3 (secure 0 0 0) 0x7f83e404a020 con 0x7f83ec136bb0 2024-09-15T02:00:46.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.057+0000 7f83e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.057+0000 7f83e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.168+0000 7f83f44af640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.168+0000 7f83f44af640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.168+0000 7f83f44af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.168+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f83ec0008d0 con 0x7f83ec136bb0 2024-09-15T02:00:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.169+0000 7f83e27fc640 1 -- 172.21.15.67:0/1564637701 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v253) v1 ==== 106+0+3 (secure 0 0 0) 0x7f83e40305d0 con 0x7f83ec136bb0 2024-09-15T02:00:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.169+0000 7f83e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.169+0000 7f83e27fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 msgr2=0x7f83d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 0x7f83d003aba0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f83e800bca0 tx=0x7f83e8008040 comp rx=0 tx=0).stop 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 10 monclient: shutdown 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 msgr2=0x7f83ec09b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f83e40097c0 tx=0x7f83e4037000 comp rx=0 tx=0).stop 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 shutdown_connections 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f83d00386f0 0x7f83d003aba0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ec136bb0 0x7f83ec09b720 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f83ec0a36e0 0x7f83ec09b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 --2- 172.21.15.67:0/1564637701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f83ec06bdc0 0x7f83ec097790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 >> 172.21.15.67:0/1564637701 conn(0x7f83ec071020 msgr2=0x7f83ec131eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 shutdown_connections 2024-09-15T02:00:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.171+0000 7f83f44af640 1 -- 172.21.15.67:0/1564637701 wait complete. 2024-09-15T02:00:46.183 INFO:tasks.workunit.client.0.smithi067.stderr:+ na=12 2024-09-15T02:00:46.183 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 7b19bad9-2bed-4846-b592-7009ac3d12e0 2024-09-15T02:00:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 10 monclient: build_initial_monmap 2024-09-15T02:00:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 10 monclient: monmap: 2024-09-15T02:00:46.259 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:46.258313+0000 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:46.258313+0000 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:46.260 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 1 Processor -- start 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 1 -- start start 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.257+0000 7f0bf439d640 10 monclient: init 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0bec098370 0x7f0bec132350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bec098370 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bec098740 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bec098b40 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec0a17b0 con 0x7f0bec098740 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec069c90 con 0x7f0bec098b40 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec096fc0 con 0x7f0bec098370 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf439d640 10 monclient(hunting): authenticate will time out at 1278.727441s 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0bec098370 0x7f0bec132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0bec098370 0x7f0bec132350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45602/0 (socket says 172.21.15.67:45602) 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 1 -- 172.21.15.67:0/2857663211 learned_addr learned my addr 172.21.15.67:0/2857663211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 10 monclient(hunting): get_auth_request con 0x7f0bec098370 auth_method 0 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.258+0000 7f0bf2139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.260+0000 7f0bf293a640 1 -- 172.21.15.67:0/2857663211 >> v1:172.21.15.67:6789/0 conn(0x7f0bec098b40 legacy=0x7f0bec069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.260+0000 7f0bf2139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.260+0000 7f0bf2139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:46.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.260+0000 7f0bf2139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.260+0000 7f0bf1938640 1 -- 172.21.15.67:0/2857663211 >> v1:172.21.15.17:6789/0 conn(0x7f0bec098740 legacy=0x7f0bec068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 10 monclient(hunting): handle_auth_done global_id 124237 payload 274 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 1 -- 172.21.15.67:0/2857663211 >> v1:172.21.15.67:6789/0 conn(0x7f0bec098b40 legacy=0x7f0bec069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 1 -- 172.21.15.67:0/2857663211 >> v1:172.21.15.17:6789/0 conn(0x7f0bec098740 legacy=0x7f0bec068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 10 monclient: _finish_hunting 0 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 1 monclient: found mon.noname-c 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.261+0000 7f0bf2139640 1 -- 172.21.15.67:0/2857663211 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec09fef0 con 0x7f0bec098370 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.263+0000 7f0bf2139640 1 -- 172.21.15.67:0/2857663211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098370 msgr2=0x7f0bec132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.263+0000 7f0bf2139640 1 --2- 172.21.15.67:0/2857663211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098370 0x7f0bec132350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009a10 tx=0x7f0be80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fafbc2f3e18d0a1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 1 -- 172.21.15.67:0/2857663211 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be802f4b0 con 0x7f0bec098370 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: dump: 2024-09-15T02:00:46.266 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: _finish_auth 0 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 1 -- 172.21.15.67:0/2857663211 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be802f610 con 0x7f0bec098370 2024-09-15T02:00:46.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 5 monclient: authenticate success, global_id 124237 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 1 -- 172.21.15.67:0/2857663211 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be802f9f0 con 0x7f0bec098370 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 4 set_mon_vals no callback set 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf1137640 10 monclient: dump: 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.268 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.269 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.269 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 10 monclient: shutdown 2024-09-15T02:00:46.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:46.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 -- 172.21.15.67:0/2857663211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098370 msgr2=0x7f0bec132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 --2- 172.21.15.67:0/2857663211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098370 0x7f0bec132350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009a10 tx=0x7f0be80091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 -- 172.21.15.67:0/2857663211 shutdown_connections 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 --2- 172.21.15.67:0/2857663211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098370 0x7f0bec132350 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 -- 172.21.15.67:0/2857663211 >> 172.21.15.67:0/2857663211 conn(0x7f0bec071150 msgr2=0x7f0bec0a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 -- 172.21.15.67:0/2857663211 shutdown_connections 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.264+0000 7f0bf439d640 1 -- 172.21.15.67:0/2857663211 wait complete. 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient: build_initial_monmap 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient: monmap: 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:46.274 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 1 Processor -- start 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 1 -- start start 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient: init 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bec098370 0x7f0bec09e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.265+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-a con 0x7f0bec098370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:46.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bec098740 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 0x7f0bec1410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bec098b40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0a4290 con 0x7f0bec098370 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec09f080 con 0x7f0bec098740 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec11dd40 con 0x7f0bec098b40 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf439d640 10 monclient(hunting): authenticate will time out at 1278.735038s 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf293a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 msgr2=0x7f0bec1410d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf293a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 0x7f0bec1410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45606/0 (socket says 172.21.15.67:45606) 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 -- 172.21.15.67:0/3637995604 learned_addr learned my addr 172.21.15.67:0/3637995604 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): get_auth_request con 0x7f0bec098740 auth_method 0 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf2139640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bec098370 0x7f0bec09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf2139640 10 monclient(hunting): get_auth_request con 0x7f0bec098370 auth_method 0 2024-09-15T02:00:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf2139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf2139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf2139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient(hunting): handle_auth_done global_id 124243 payload 824 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 -- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 msgr2=0x7f0bec1410d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 0x7f0bec1410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 -- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bec098370 msgr2=0x7f0bec09e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bec098370 0x7f0bec09e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient: _finish_hunting 0 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 monclient: found mon.noname-b 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.266+0000 7f0bf1938640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8009450 con 0x7f0bec098740 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf1938640 1 -- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 msgr2=0x7f0bec130c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf1938640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00c510 tx=0x7f0bdc007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc0074f0 con 0x7f0bec098740 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: dump: 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.279 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 5 monclient: authenticate success, global_id 124243 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc005780 con 0x7f0bec098740 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 10 monclient: _renew_subs 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec11e020 con 0x7f0bec098740 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc005b80 con 0x7f0bec098740 2024-09-15T02:00:46.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0be27fc640 10 monclient: dump: 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf0936640 4 set_mon_vals no callback set 2024-09-15T02:00:46.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 10 monclient: _renew_subs 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.268+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec11c8e0 con 0x7f0bec098740 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.269+0000 7f0bf439d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.269+0000 7f0bf439d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.269+0000 7f0bf439d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.269+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec04bf60 con 0x7f0bec098740 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.269+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0bdc01b3b0 con 0x7f0bec098740 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.270+0000 7f0be27fc640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 0x7f0bd003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.270+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(253..253 src has 224..253) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f0bdc04bb10 con 0x7f0bec098740 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.270+0000 7f0bf2139640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 0x7f0bd003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.270+0000 7f0bf2139640 10 monclient: get_auth_request con 0x7f0bd00386f0 auth_method 0 2024-09-15T02:00:46.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.272+0000 7f0bf2139640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 0x7f0bd003aba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0be8034000 tx=0x7f0be8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.274+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0bdc018860 con 0x7f0bec098740 2024-09-15T02:00:46.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.274+0000 7f0be27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.274+0000 7f0be27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.452+0000 7f0bf439d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.452+0000 7f0bf439d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.452+0000 7f0bf439d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.452+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f0bec06f0b0 con 0x7f0bec098740 2024-09-15T02:00:46.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.454+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v253) v1 ==== 106+0+3 (secure 0 0 0) 0x7f0bdc017070 con 0x7f0bec098740 2024-09-15T02:00:46.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.454+0000 7f0be27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.454+0000 7f0be27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.563+0000 7f0bf439d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.563+0000 7f0bf439d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.563+0000 7f0bf439d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.563+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f0bec141dd0 con 0x7f0bec098740 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.564+0000 7f0be27fc640 1 -- 172.21.15.67:0/3637995604 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v253) v1 ==== 106+0+3 (secure 0 0 0) 0x7f0bdc018260 con 0x7f0bec098740 2024-09-15T02:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.564+0000 7f0be27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:46.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.564+0000 7f0be27fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 msgr2=0x7f0bd003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 0x7f0bd003aba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0be8034000 tx=0x7f0be8032000 comp rx=0 tx=0).stop 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 10 monclient: shutdown 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 msgr2=0x7f0bec130c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00c510 tx=0x7f0bdc007d20 comp rx=0 tx=0).stop 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 shutdown_connections 2024-09-15T02:00:46.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0bd00386f0 0x7f0bd003aba0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0bec098b40 0x7f0bec1410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bec098740 0x7f0bec130c20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 --2- 172.21.15.67:0/3637995604 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0bec098370 0x7f0bec09e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 >> 172.21.15.67:0/3637995604 conn(0x7f0bec071150 msgr2=0x7f0bec09b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 shutdown_connections 2024-09-15T02:00:46.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.566+0000 7f0bf439d640 1 -- 172.21.15.67:0/3637995604 wait complete. 2024-09-15T02:00:46.577 INFO:tasks.workunit.client.0.smithi067.stderr:+ test 12 -eq 12 2024-09-15T02:00:46.577 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 230aff7d-6869-4287-a1fa-cd4abe8412b2 2024-09-15T02:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.650+0000 7f1bca79d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.650+0000 7f1bca79d640 10 monclient: build_initial_monmap 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.650+0000 7f1bca79d640 10 monclient: monmap: 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:46.651496+0000 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:46.651496+0000 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:46.653 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:46.654 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:46.654 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 Processor -- start 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 -- start start 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient: init 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 0x7f1bc4132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc4130e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1bc41311d0 0x7f1bc41315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc41311d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bc4068dc0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc409ff20 con 0x7f1bc4068dc0 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc41320b0 con 0x7f1bc41311d0 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40a17b0 con 0x7f1bc4130e00 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bca79d640 10 monclient(hunting): authenticate will time out at 1279.120626s 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc3fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 msgr2=0x7f1bc4132430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:46.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 0x7f1bc4132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc8d3a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bc4068dc0 0x7f1bc4069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49152/0 (socket says 172.21.15.67:49152) 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc8d3a640 1 -- 172.21.15.67:0/3408288631 learned_addr learned my addr 172.21.15.67:0/3408288631 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3408288631 >> v2:172.21.15.17:3300/0 conn(0x7f1bc41311d0 0x7f1bc41315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): get_auth_request con 0x7f1bc41311d0 auth_method 0 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.651+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_done global_id 104733 payload 274 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 msgr2=0x7f1bc4132430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:46.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 0x7f1bc4132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3408288631 >> v1:172.21.15.67:6789/0 conn(0x7f1bc4068dc0 legacy=0x7f1bc4069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 1 monclient: found mon.noname-a 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:46.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.652+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3408288631 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40a00f0 con 0x7f1bc41311d0 2024-09-15T02:00:46.664 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc41311d0 msgr2=0x7f1bc41315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc41311d0 0x7f1bc41315c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009850 tx=0x7f1bac0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce1de278dfa25d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 1 -- 172.21.15.67:0/3408288631 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac032440 con 0x7f1bc41311d0 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: dump: 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.665 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 1 -- 172.21.15.67:0/3408288631 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0325a0 con 0x7f1bc41311d0 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 5 monclient: authenticate success, global_id 104733 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 4 set_mon_vals no callback set 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 1 -- 172.21.15.67:0/3408288631 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac032980 con 0x7f1bc41311d0 2024-09-15T02:00:46.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bc2ffd640 10 monclient: dump: 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 10 monclient: shutdown 2024-09-15T02:00:46.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 1 -- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc41311d0 msgr2=0x7f1bc41315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.655+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc41311d0 0x7f1bc41315c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009850 tx=0x7f1bac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 -- 172.21.15.67:0/3408288631 shutdown_connections 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc41311d0 0x7f1bc41315c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3408288631 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc4130e00 0x7f1bc4132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 -- 172.21.15.67:0/3408288631 >> 172.21.15.67:0/3408288631 conn(0x7f1bc40710f0 msgr2=0x7f1bc40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 -- 172.21.15.67:0/3408288631 shutdown_connections 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 -- 172.21.15.67:0/3408288631 wait complete. 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 10 monclient: build_initial_monmap 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 10 monclient: monmap: 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:46.668 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.656+0000 7f1bca79d640 1 Processor -- start 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 -- start start 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient: init 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 0x7f1bc412eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc4068dc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc4130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 0x7f1bc4141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc41311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40a0820 con 0x7f1bc4068dc0 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc409eda0 con 0x7f1bc4130e00 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc411c0f0 con 0x7f1bc41311d0 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bca79d640 10 monclient(hunting): authenticate will time out at 1279.126486s 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc8d3a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 msgr2=0x7f1bc4141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45610/0 (socket says 172.21.15.67:45610) 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc8d3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 0x7f1bc4141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3970192971 learned_addr learned my addr 172.21.15.67:0/3970192971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): get_auth_request con 0x7f1bc4130e00 auth_method 0 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 0x7f1bc412eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 10 monclient(hunting): get_auth_request con 0x7f1bc4068dc0 auth_method 0 2024-09-15T02:00:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient(hunting): handle_auth_done global_id 124246 payload 824 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 msgr2=0x7f1bc4141790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 0x7f1bc4141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 msgr2=0x7f1bc412eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 0x7f1bc412eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 monclient: found mon.noname-b 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac009390 con 0x7f1bc4130e00 2024-09-15T02:00:46.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.657+0000 7f1bc3fff640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 0x7f1bc412eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bc37fe640 1 -- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 msgr2=0x7f1bc4141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bc37fe640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009850 tx=0x7f1bac032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac031a90 con 0x7f1bc4130e00 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: dump: 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.673 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: _finish_auth 0 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 5 monclient: authenticate success, global_id 124246 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 10 monclient: _renew_subs 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc411cfe0 con 0x7f1bc4130e00 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac031bf0 con 0x7f1bc4130e00 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac03c410 con 0x7f1bc4130e00 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:46.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bbbfff640 10 monclient: dump: 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bc27fc640 4 set_mon_vals no callback set 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 10 monclient: _renew_subs 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.660+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc411d5a0 con 0x7f1bc4130e00 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1bac039070 con 0x7f1bc4130e00 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bbbfff640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(253..253 src has 224..253) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1bac06cba0 con 0x7f1bc4130e00 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bca79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bca79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bca79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bc3fff640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc404bf60 con 0x7f1bc4130e00 2024-09-15T02:00:46.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.661+0000 7f1bc3fff640 10 monclient: get_auth_request con 0x7f1ba40386f0 auth_method 0 2024-09-15T02:00:46.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.664+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:00:46.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.664+0000 7f1bc3fff640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4004fd0 tx=0x7f1bb4004980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:46.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.666+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1bac040020 con 0x7f1bc4130e00 2024-09-15T02:00:46.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.666+0000 7f1bbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:46.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.666+0000 7f1bbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:46.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.839+0000 7f1bca79d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:46.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.839+0000 7f1bca79d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:46.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.839+0000 7f1bca79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:46.842 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:46.839+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"} v 0) v1 -- 0x7f1bc40a14d0 con 0x7f1bc4130e00 2024-09-15T02:00:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.061+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7f1bac037070 con 0x7f1bc4130e00 2024-09-15T02:00:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.061+0000 7f1bbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.064 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.061+0000 7f1bbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:47.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.171+0000 7f1bca79d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.171+0000 7f1bca79d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.171+0000 7f1bca79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.171+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"} v 0) v1 -- 0x7f1bc40a14d0 con 0x7f1bc4130e00 2024-09-15T02:00:47.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.172+0000 7f1bbbfff640 1 -- 172.21.15.67:0/3970192971 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7f1bac03cda0 con 0x7f1bc4130e00 2024-09-15T02:00:47.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.172+0000 7f1bbbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.172+0000 7f1bbbfff640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:47.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 msgr2=0x7f1ba403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4004fd0 tx=0x7f1bb4004980 comp rx=0 tx=0).stop 2024-09-15T02:00:47.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 10 monclient: shutdown 2024-09-15T02:00:47.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 msgr2=0x7f1bc4141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009850 tx=0x7f1bac032000 comp rx=0 tx=0).stop 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 shutdown_connections 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1ba40386f0 0x7f1ba403aba0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bc41311d0 0x7f1bc4141790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc4130e00 0x7f1bc4141250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 --2- 172.21.15.67:0/3970192971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bc4068dc0 0x7f1bc412eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 >> 172.21.15.67:0/3970192971 conn(0x7f1bc40710f0 msgr2=0x7f1bc4132ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 shutdown_connections 2024-09-15T02:00:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.174+0000 7f1bca79d640 1 -- 172.21.15.67:0/3970192971 wait complete. 2024-09-15T02:00:47.185 INFO:tasks.workunit.client.0.smithi067.stderr:+ nb=14 2024-09-15T02:00:47.185 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 230aff7d-6869-4287-a1fa-cd4abe8412b2 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.257+0000 7fb336d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.257+0000 7fb336d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.257+0000 7fb336d9d640 10 monclient: monmap: 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:47.259088+0000 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:47.259088+0000 2024-09-15T02:00:47.263 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.258+0000 7fb336d9d640 1 Processor -- start 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.258+0000 7fb336d9d640 1 -- start start 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.258+0000 7fb336d9d640 10 monclient: init 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T02:00:47.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3300a36e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:47.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb33009ccc0 0x7fb330099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb33009ccc0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:47.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 0x7fb33009e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb33009c630 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33009f640 con 0x7fb3300a36e0 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb330099fe0 con 0x7fb33009ccc0 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33009b9d0 con 0x7fb33009c630 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb336d9d640 10 monclient(hunting): authenticate will time out at 1279.728337s 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb33533a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 msgr2=0x7fb33009e0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb33009ccc0 0x7fb330099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb33533a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 0x7fb33009e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb33009ccc0 0x7fb330099aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45620/0 (socket says 172.21.15.67:45620) 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 -- 172.21.15.67:0/3401694451 learned_addr learned my addr 172.21.15.67:0/3401694451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:47.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): get_auth_request con 0x7fb33009ccc0 auth_method 0 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient(hunting): handle_auth_done global_id 124249 payload 274 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 -- 172.21.15.67:0/3401694451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 msgr2=0x7fb33009e0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 --2- 172.21.15.67:0/3401694451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 0x7fb33009e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 -- 172.21.15.67:0/3401694451 >> v1:172.21.15.17:6789/0 conn(0x7fb3300a36e0 legacy=0x7fb33009d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 monclient: found mon.noname-c 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.259+0000 7fb32ffff640 1 -- 172.21.15.67:0/3401694451 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33009b6f0 con 0x7fb33009ccc0 2024-09-15T02:00:47.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32ffff640 1 -- 172.21.15.67:0/3401694451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 msgr2=0x7fb330099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32ffff640 1 --2- 172.21.15.67:0/3401694451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb330099aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb320009f20 tx=0x7fb320009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f7efe723703fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 1 -- 172.21.15.67:0/3401694451 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320007610 con 0x7fb33009ccc0 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: dump: 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.274 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb336d9d640 5 monclient: authenticate success, global_id 124249 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 1 -- 172.21.15.67:0/3401694451 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32002d050 con 0x7fb33009ccc0 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb336d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:47.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb336d9d640 4 set_mon_vals no callback set 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 1 -- 172.21.15.67:0/3401694451 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3200079a0 con 0x7fb33009ccc0 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.262+0000 7fb32f7fe640 10 monclient: dump: 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.276 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 10 monclient: shutdown 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 -- 172.21.15.67:0/3401694451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 msgr2=0x7fb330099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3401694451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb330099aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb320009f20 tx=0x7fb320009840 comp rx=0 tx=0).stop 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 -- 172.21.15.67:0/3401694451 shutdown_connections 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3401694451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb33009c630 0x7fb33009e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3401694451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb330099aa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 -- 172.21.15.67:0/3401694451 >> 172.21.15.67:0/3401694451 conn(0x7fb330071020 msgr2=0x7fb3300a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 -- 172.21.15.67:0/3401694451 shutdown_connections 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 1 -- 172.21.15.67:0/3401694451 wait complete. 2024-09-15T02:00:47.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 10 monclient: build_initial_monmap 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.263+0000 7fb336d9d640 10 monclient: monmap: 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 1 Processor -- start 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 1 -- start start 2024-09-15T02:00:47.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 10 monclient: init 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb33009c630 0x7fb33013bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.264+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb33009c630 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb33009ccc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 0x7fb33013f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3300a36e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb330120ad0 con 0x7fb33009c630 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb330120510 con 0x7fb33009ccc0 2024-09-15T02:00:47.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33011dfa0 con 0x7fb3300a36e0 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb336d9d640 10 monclient(hunting): authenticate will time out at 1279.734063s 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb33533a640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 msgr2=0x7fb33013f050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb33533a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 0x7fb33013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45636/0 (socket says 172.21.15.67:45636) 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 -- 172.21.15.67:0/3365593313 learned_addr learned my addr 172.21.15.67:0/3365593313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): get_auth_request con 0x7fb33009ccc0 auth_method 0 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb33009c630 0x7fb33013bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): get_auth_request con 0x7fb33009c630 auth_method 0 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb334b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient(hunting): handle_auth_done global_id 124252 payload 824 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 -- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 msgr2=0x7fb33013f050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:47.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 0x7fb33013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 -- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb33009c630 msgr2=0x7fb33013bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb33009c630 0x7fb33013bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 monclient: found mon.noname-b 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.265+0000 7fb32ffff640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320009ae0 con 0x7fb33009ccc0 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32ffff640 1 -- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 msgr2=0x7fb33013b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32ffff640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb320009f20 tx=0x7fb320036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320031aa0 con 0x7fb33009ccc0 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 10 monclient: dump: 2024-09-15T02:00:47.282 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 10 monclient: _finish_auth 0 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb336d9d640 5 monclient: authenticate success, global_id 124252 2024-09-15T02:00:47.283 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.267+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb320031c00 con 0x7fb33009ccc0 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 10 monclient: _renew_subs 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb330120230 con 0x7fb33009ccc0 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32003c420 con 0x7fb33009ccc0 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 10 monclient: dump: 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.284 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32effd640 4 set_mon_vals no callback set 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 10 monclient: _renew_subs 2024-09-15T02:00:47.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3301208a0 con 0x7fb33009ccc0 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.268+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb32003c7d0 con 0x7fb33009ccc0 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb32cff9640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 0x7fb31003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb336d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb336d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 224..254) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb32006d480 con 0x7fb33009ccc0 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb336d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33004bf60 con 0x7fb33009ccc0 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb334b39640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 0x7fb31003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.269+0000 7fb334b39640 10 monclient: get_auth_request con 0x7fb3100386f0 auth_method 0 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.271+0000 7fb334b39640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 0x7fb31003aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb328007fd0 tx=0x7fb328007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.273+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb320040070 con 0x7fb33009ccc0 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.273+0000 7fb32cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.273+0000 7fb32cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:47.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.450+0000 7fb336d9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.450+0000 7fb336d9d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.450+0000 7fb336d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.453 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.450+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"} v 0) v1 -- 0x7fb330067120 con 0x7fb33009ccc0 2024-09-15T02:00:47.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.451+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb32003ca80 con 0x7fb33009ccc0 2024-09-15T02:00:47.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.451+0000 7fb32cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.451+0000 7fb32cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:47.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.563+0000 7fb336d9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.563+0000 7fb336d9d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.563+0000 7fb336d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.563+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"} v 0) v1 -- 0x7fb33011f360 con 0x7fb33009ccc0 2024-09-15T02:00:47.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.564+0000 7fb32cff9640 1 -- 172.21.15.67:0/3365593313 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb320039310 con 0x7fb33009ccc0 2024-09-15T02:00:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.564+0000 7fb32cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "230aff7d-6869-4287-a1fa-cd4abe8412b2"}] 2024-09-15T02:00:47.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.564+0000 7fb32cff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 msgr2=0x7fb31003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 0x7fb31003aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb328007fd0 tx=0x7fb328007e10 comp rx=0 tx=0).stop 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 10 monclient: shutdown 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 msgr2=0x7fb33013b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb320009f20 tx=0x7fb320036000 comp rx=0 tx=0).stop 2024-09-15T02:00:47.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 shutdown_connections 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3100386f0 0x7fb31003aba0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3300a36e0 0x7fb33013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33009ccc0 0x7fb33013b190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 --2- 172.21.15.67:0/3365593313 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb33009c630 0x7fb33013bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 >> 172.21.15.67:0/3365593313 conn(0x7fb330071020 msgr2=0x7fb33006dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 shutdown_connections 2024-09-15T02:00:47.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.566+0000 7fb336d9d640 1 -- 172.21.15.67:0/3365593313 wait complete. 2024-09-15T02:00:47.578 INFO:tasks.workunit.client.0.smithi067.stderr:+ test 14 -eq 14 2024-09-15T02:00:47.578 INFO:tasks.workunit.client.0.smithi067.stderr:+ test 14 -ne 12 2024-09-15T02:00:47.578 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm 12 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.650+0000 7f0c9d91d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.650+0000 7f0c9d91d640 10 monclient: build_initial_monmap 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.650+0000 7f0c9d91d640 10 monclient: monmap: 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:47.652088+0000 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:47.652088+0000 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:47.653 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.656 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:47.656 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:47.656 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:47.656 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:47.656 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:47.657 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 1 Processor -- start 2024-09-15T02:00:47.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 1 -- start start 2024-09-15T02:00:47.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 10 monclient: init 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 0x7f0c98132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.651+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c98130e00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0c981311d0 0x7f0c981315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c981311d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c98068dc0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9809ff20 con 0x7f0c98068dc0 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c981320b0 con 0x7f0c981311d0 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980a17b0 con 0x7f0c98130e00 2024-09-15T02:00:47.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c9d91d640 10 monclient(hunting): authenticate will time out at 1280.121069s 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c96ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 msgr2=0x7f0c98132430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c96ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 0x7f0c98132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0c981311d0 0x7f0c981315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0c981311d0 0x7f0c981315c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:33710/0 (socket says 172.21.15.67:33710) 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 1 -- 172.21.15.67:0/2444848780 learned_addr learned my addr 172.21.15.67:0/2444848780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): get_auth_request con 0x7f0c981311d0 auth_method 0 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.652+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c977fe640 1 -- 172.21.15.67:0/2444848780 >> v1:172.21.15.17:6789/0 conn(0x7f0c98068dc0 legacy=0x7f0c98069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 10 monclient(hunting): handle_auth_done global_id 104742 payload 274 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 1 -- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 msgr2=0x7f0c98132430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 1 --2- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 0x7f0c98132430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 1 -- 172.21.15.67:0/2444848780 >> v1:172.21.15.17:6789/0 conn(0x7f0c98068dc0 legacy=0x7f0c98069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:47.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.654+0000 7f0c967fc640 1 -- 172.21.15.67:0/2444848780 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c980a00f0 con 0x7f0c981311d0 2024-09-15T02:00:47.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c967fc640 1 -- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 msgr2=0x7f0c981315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:47.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c967fc640 1 --2- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c981315c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009990 tx=0x7f0c8c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bb6f76d23d652 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 1 -- 172.21.15.67:0/2444848780 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c02f4b0 con 0x7f0c981311d0 2024-09-15T02:00:47.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: dump: 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.667 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 1 -- 172.21.15.67:0/2444848780 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c02f610 con 0x7f0c981311d0 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 1 -- 172.21.15.67:0/2444848780 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c02f9b0 con 0x7f0c981311d0 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c95ffb640 10 monclient: dump: 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.668 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c9d91d640 5 monclient: authenticate success, global_id 104742 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c9d91d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.658+0000 7f0c9d91d640 4 set_mon_vals no callback set 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 10 monclient: shutdown 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 -- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 msgr2=0x7f0c981315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c981315c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009990 tx=0x7f0c8c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 -- 172.21.15.67:0/2444848780 shutdown_connections 2024-09-15T02:00:47.669 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c981315c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/2444848780 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98130e00 0x7f0c98132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 -- 172.21.15.67:0/2444848780 >> 172.21.15.67:0/2444848780 conn(0x7f0c980710f0 msgr2=0x7f0c980a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 -- 172.21.15.67:0/2444848780 shutdown_connections 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.659+0000 7f0c9d91d640 1 -- 172.21.15.67:0/2444848780 wait complete. 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.660+0000 7f0c9d91d640 10 monclient: build_initial_monmap 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.660+0000 7f0c9d91d640 10 monclient: monmap: 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.670 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.660+0000 7f0c9d91d640 1 Processor -- start 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.660+0000 7f0c9d91d640 1 -- start start 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.660+0000 7f0c9d91d640 10 monclient: init 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 0x7f0c9809c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c98068dc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c98130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c9809d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c981311d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980a4be0 con 0x7f0c981311d0 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9809eb80 con 0x7f0c98130e00 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980a0820 con 0x7f0c98068dc0 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c9d91d640 10 monclient(hunting): authenticate will time out at 1280.130369s 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c96ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 msgr2=0x7f0c9809c620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c96ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 0x7f0c9809c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45650/0 (socket says 172.21.15.67:45650) 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 -- 172.21.15.67:0/1184464799 learned_addr learned my addr 172.21.15.67:0/1184464799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): get_auth_request con 0x7f0c98130e00 auth_method 0 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c9809d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): get_auth_request con 0x7f0c981311d0 auth_method 0 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient(hunting): handle_auth_done global_id 124255 payload 824 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 -- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 msgr2=0x7f0c9809c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 0x7f0c9809c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 -- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 msgr2=0x7f0c9809d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c9809d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:47.674 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 monclient: found mon.noname-b 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.661+0000 7f0c967fc640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c009450 con 0x7f0c98130e00 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.664+0000 7f0c967fc640 1 -- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 msgr2=0x7f0c9809cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.664+0000 7f0c967fc640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c02a7d0 tx=0x7f0c8c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.675 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.664+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:00:47.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c032960 con 0x7f0c98130e00 2024-09-15T02:00:47.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:47.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: dump: 2024-09-15T02:00:47.677 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: _finish_auth 0 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c032ae0 con 0x7f0c98130e00 2024-09-15T02:00:47.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c032ea0 con 0x7f0c98130e00 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c7effd640 10 monclient: dump: 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:47.679 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 5 monclient: authenticate success, global_id 124255 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 10 monclient: _renew_subs 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9809e690 con 0x7f0c98130e00 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c957fa640 4 set_mon_vals no callback set 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 10 monclient: _renew_subs 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.666+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9809ed50 con 0x7f0c98130e00 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0c8c039070 con 0x7f0c98130e00 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c9d91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c9d91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c9d91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9804bf60 con 0x7f0c98130e00 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c7effd640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 0x7f0c8403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 224..254) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f0c8c06ca30 con 0x7f0c98130e00 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c96ffd640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 0x7f0c8403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.667+0000 7f0c96ffd640 10 monclient: get_auth_request con 0x7f0c84038740 auth_method 0 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.669+0000 7f0c96ffd640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 0x7f0c8403abf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0c88003c70 tx=0x7f0c88009310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.671+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0c8c0313a0 con 0x7f0c98130e00 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.671+0000 7f0c7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:47.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.671+0000 7f0c7effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:47.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.842+0000 7f0c9d91d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:47.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.842+0000 7f0c9d91d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:47.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.842+0000 7f0c9d91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:47.845 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:47.842+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f0c980a14d0 con 0x7f0c98130e00 2024-09-15T02:00:47.845 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:47.843+0000 7f742dbf1640 -1 mon.a@0(leader).osd e254 definitely_dead 0 2024-09-15T02:00:48.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.812+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v255) v1 ==== 83+0+0 (secure 0 0 0) 0x7f0c8c040080 con 0x7f0c98130e00 2024-09-15T02:00:48.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.812+0000 7f0c7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:48.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.812+0000 7f0c7effd640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-15T02:00:48.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.922+0000 7f0c9d91d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:48.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.922+0000 7f0c9d91d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:48.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.922+0000 7f0c9d91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:48.924 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.922+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f0c9809e180 con 0x7f0c98130e00 2024-09-15T02:00:48.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:48.923+0000 7f742dbf1640 -1 mon.a@0(leader).osd e255 definitely_dead 0 2024-09-15T02:00:48.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.923+0000 7f0c7effd640 1 -- 172.21.15.67:0/1184464799 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v255) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0c8c037030 con 0x7f0c98130e00 2024-09-15T02:00:48.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.923+0000 7f0c7effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:48.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.923+0000 7f0c7effd640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T02:00:48.926 INFO:tasks.workunit.client.0.smithi067.stderr:osd.12 does not exist. 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 msgr2=0x7f0c8403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 0x7f0c8403abf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0c88003c70 tx=0x7f0c88009310 comp rx=0 tx=0).stop 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 10 monclient: shutdown 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 msgr2=0x7f0c9809cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c02a7d0 tx=0x7f0c8c032000 comp rx=0 tx=0).stop 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 shutdown_connections 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0c84038740 0x7f0c8403abf0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0c981311d0 0x7f0c9809d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98130e00 0x7f0c9809cb60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 --2- 172.21.15.67:0/1184464799 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0c98068dc0 0x7f0c9809c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 >> 172.21.15.67:0/1184464799 conn(0x7f0c980710f0 msgr2=0x7f0c9812fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 shutdown_connections 2024-09-15T02:00:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:48.925+0000 7f0c9d91d640 1 -- 172.21.15.67:0/1184464799 wait complete. 2024-09-15T02:00:48.937 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm 12 2024-09-15T02:00:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 10 monclient: build_initial_monmap 2024-09-15T02:00:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 10 monclient: monmap: 2024-09-15T02:00:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:49.014222+0000 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:49.014222+0000 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:49.016 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 1 Processor -- start 2024-09-15T02:00:49.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 1 -- start start 2024-09-15T02:00:49.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.013+0000 7fbb7959d640 10 monclient: init 2024-09-15T02:00:49.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbb74098370 0x7fbb74132360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-a con 0x7fbb74098370 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb74098740 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 0x7fbb74069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-b con 0x7fbb74098b10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740a17b0 con 0x7fbb74098740 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7409feb0 con 0x7fbb74098370 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74098f00 con 0x7fbb74098b10 2024-09-15T02:00:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb7959d640 10 monclient(hunting): authenticate will time out at 1281.483300s 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb727fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb74098740 0x7fbb74068e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49168/0 (socket says 172.21.15.67:49168) 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb727fc640 1 -- 172.21.15.67:0/658792230 learned_addr learned my addr 172.21.15.67:0/658792230 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb737fe640 1 -- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 msgr2=0x7fbb74069580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb737fe640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 0x7fbb74069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/658792230 >> v2:172.21.15.17:3300/0 conn(0x7fbb74098370 0x7fbb74132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request con 0x7fbb74098370 auth_method 0 2024-09-15T02:00:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_done global_id 104751 payload 274 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 -- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 msgr2=0x7fbb74069580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 0x7fbb74069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 -- 172.21.15.67:0/658792230 >> v1:172.21.15.67:6789/0 conn(0x7fbb74098740 legacy=0x7fbb74068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 monclient: found mon.noname-a 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:49.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:49.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.014+0000 7fbb72ffd640 1 -- 172.21.15.67:0/658792230 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740a00f0 con 0x7fbb74098370 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb72ffd640 1 -- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 msgr2=0x7fbb74132360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb74132360 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbb740a0780 tx=0x7fbb64009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba8882ebc6ce9a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 1 -- 172.21.15.67:0/658792230 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64007840 con 0x7fbb74098370 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 10 monclient: dump: 2024-09-15T02:00:49.027 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:49.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 1 -- 172.21.15.67:0/658792230 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6402d050 con 0x7fbb74098370 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.017+0000 7fbb71ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb71ffb640 1 -- 172.21.15.67:0/658792230 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64007bd0 con 0x7fbb74098370 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb71ffb640 10 monclient: dump: 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.029 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 5 monclient: authenticate success, global_id 104751 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 4 set_mon_vals no callback set 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 10 monclient: shutdown 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 -- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 msgr2=0x7fbb74132360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb74132360 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbb740a0780 tx=0x7fbb64009780 comp rx=0 tx=0).stop 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 -- 172.21.15.67:0/658792230 shutdown_connections 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098b10 0x7fbb74069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 --2- 172.21.15.67:0/658792230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb74132360 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 -- 172.21.15.67:0/658792230 >> 172.21.15.67:0/658792230 conn(0x7fbb740a59a0 msgr2=0x7fbb740a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 -- 172.21.15.67:0/658792230 shutdown_connections 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.018+0000 7fbb7959d640 1 -- 172.21.15.67:0/658792230 wait complete. 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient: build_initial_monmap 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient: monmap: 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:49.031 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 Processor -- start 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 -- start start 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient: init 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb7409e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-a con 0x7fbb74098370 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 0x7fbb74130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-c con 0x7fbb74098740 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): picked mon.noname-b con 0x7fbb74098b10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740a0820 con 0x7fbb74098370 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7409b2c0 con 0x7fbb74098b10 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7411e100 con 0x7fbb74098740 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.019+0000 7fbb7959d640 10 monclient(hunting): authenticate will time out at 1281.488996s 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb727fc640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 msgr2=0x7fbb74130d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 0x7fbb74130d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45658/0 (socket says 172.21.15.67:45658) 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 -- 172.21.15.67:0/1449771585 learned_addr learned my addr 172.21.15.67:0/1449771585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): get_auth_request con 0x7fbb74098b10 auth_method 0 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb7409e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request con 0x7fbb74098370 auth_method 0 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient(hunting): handle_auth_done global_id 124258 payload 824 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 -- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 msgr2=0x7fbb74130d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 0x7fbb74130d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 -- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 msgr2=0x7fbb7409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb7409e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 monclient: found mon.noname-b 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.020+0000 7fbb737fe640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb64009a20 con 0x7fbb74098b10 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb737fe640 1 -- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 msgr2=0x7fbb7411b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb737fe640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00b4e0 tx=0x7fbb6c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c003d40 con 0x7fbb74098b10 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: dump: 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: _finish_auth 0 2024-09-15T02:00:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c006150 con 0x7fbb74098b10 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 5 monclient: authenticate success, global_id 124258 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 10 monclient: _renew_subs 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb74141b90 con 0x7fbb74098b10 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c006510 con 0x7fbb74098b10 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb6affd640 10 monclient: dump: 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 10 monclient: _renew_subs 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb717fa640 4 set_mon_vals no callback set 2024-09-15T02:00:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.022+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb741416b0 con 0x7fbb74098b10 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbb6c010570 con 0x7fbb74098b10 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb7959d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb7959d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb7959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7411e290 con 0x7fbb74098b10 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb6affd640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 0x7fbb5403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.023+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 229..255) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbb6c04b950 con 0x7fbb74098b10 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.024+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 0x7fbb5403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.024+0000 7fbb72ffd640 10 monclient: get_auth_request con 0x7fbb54038720 auth_method 0 2024-09-15T02:00:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.026+0000 7fbb72ffd640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 0x7fbb5403abd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbb740a5130 tx=0x7fbb64037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.026+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:00:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.028+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbb6c01b3b0 con 0x7fbb74098b10 2024-09-15T02:00:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.028+0000 7fbb6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.028+0000 7fbb6affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:49.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.201+0000 7fbb7959d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.201+0000 7fbb7959d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.201+0000 7fbb7959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.201+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fbb74141b30 con 0x7fbb74098b10 2024-09-15T02:00:49.417 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:49.415+0000 7f742dbf1640 -1 mon.a@0(leader).osd e255 definitely_dead 0 2024-09-15T02:00:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.415+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v255) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbb6c017080 con 0x7fbb74098b10 2024-09-15T02:00:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.415+0000 7fbb6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.415+0000 7fbb6affd640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-15T02:00:49.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.520+0000 7fbb7959d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.520+0000 7fbb7959d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.520+0000 7fbb7959d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.520+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fbb7404bf60 con 0x7fbb74098b10 2024-09-15T02:00:49.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:49.521+0000 7f742dbf1640 -1 mon.a@0(leader).osd e255 definitely_dead 0 2024-09-15T02:00:49.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.522+0000 7fbb6affd640 1 -- 172.21.15.67:0/1449771585 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v255) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbb6c00d290 con 0x7fbb74098b10 2024-09-15T02:00:49.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.522+0000 7fbb6affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T02:00:49.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.522+0000 7fbb6affd640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T02:00:49.524 INFO:tasks.workunit.client.0.smithi067.stderr:osd.12 does not exist. 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 msgr2=0x7fbb5403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 0x7fbb5403abd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbb740a5130 tx=0x7fbb64037000 comp rx=0 tx=0).stop 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 10 monclient: shutdown 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 msgr2=0x7fbb7411b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00b4e0 tx=0x7fbb6c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 shutdown_connections 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb54038720 0x7fbb5403abd0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb74098b10 0x7fbb7411b830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbb74098740 0x7fbb74130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.523+0000 7fbb7959d640 1 --2- 172.21.15.67:0/1449771585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb74098370 0x7fbb7409e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.524+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 >> 172.21.15.67:0/1449771585 conn(0x7fbb740a59a0 msgr2=0x7fbb740a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.524+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 shutdown_connections 2024-09-15T02:00:49.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.524+0000 7fbb7959d640 1 -- 172.21.15.67:0/1449771585 wait complete. 2024-09-15T02:00:49.535 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm 14 2024-09-15T02:00:49.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.607+0000 7fa543460640 10 monclient: get_monmap_and_config 2024-09-15T02:00:49.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.607+0000 7fa543460640 10 monclient: build_initial_monmap 2024-09-15T02:00:49.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.607+0000 7fa543460640 10 monclient: monmap: 2024-09-15T02:00:49.610 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:49.609066+0000 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:49.609066+0000 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.608+0000 7fa543460640 1 Processor -- start 2024-09-15T02:00:49.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.608+0000 7fa543460640 1 -- start start 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.608+0000 7fa543460640 10 monclient: init 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 0x7fa53c06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): picked mon.noname-c con 0x7fa53c06e1d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa53c068d40 0x7fa53c09d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): picked mon.noname-a con 0x7fa53c068d40 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa53c06e5a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c06a0b0 con 0x7fa53c06e5a0 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c099fa0 con 0x7fa53c068d40 2024-09-15T02:00:49.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c06be10 con 0x7fa53c06e1d0 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa543460640 10 monclient(hunting): authenticate will time out at 1282.078249s 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 0x7fa53c06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 0x7fa53c06ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45668/0 (socket says 172.21.15.67:45668) 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 1 -- 172.21.15.67:0/1791626657 learned_addr learned my addr 172.21.15.67:0/1791626657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 10 monclient(hunting): get_auth_request con 0x7fa53c06e1d0 auth_method 0 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5411fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 --2- 172.21.15.67:0/1791626657 >> v2:172.21.15.17:3300/0 conn(0x7fa53c068d40 0x7fa53c09d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): get_auth_request con 0x7fa53c068d40 auth_method 0 2024-09-15T02:00:49.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient(hunting): handle_auth_done global_id 104757 payload 274 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 -- 172.21.15.67:0/1791626657 >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 msgr2=0x7fa53c06ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 --2- 172.21.15.67:0/1791626657 >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 0x7fa53c06ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 -- 172.21.15.67:0/1791626657 >> v1:172.21.15.67:6789/0 conn(0x7fa53c06e5a0 legacy=0x7fa53c09df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient: _finish_hunting 0 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 monclient: found mon.noname-a 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:49.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:49.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.609+0000 7fa5409fb640 1 -- 172.21.15.67:0/1791626657 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c06bf50 con 0x7fa53c068d40 2024-09-15T02:00:49.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.612+0000 7fa5409fb640 1 -- 172.21.15.67:0/1791626657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c068d40 msgr2=0x7fa53c09d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.612+0000 7fa5409fb640 1 --2- 172.21.15.67:0/1791626657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c068d40 0x7fa53c09d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa53c09f270 tx=0x7fa5300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa1899d8e217a06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 1 -- 172.21.15.67:0/1791626657 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53002ae00 con 0x7fa53c068d40 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: dump: 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: _finish_auth 0 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 1 -- 172.21.15.67:0/1791626657 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530032440 con 0x7fa53c068d40 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 1 -- 172.21.15.67:0/1791626657 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300327e0 con 0x7fa53c068d40 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 5 monclient: authenticate success, global_id 104757 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 4 set_mon_vals no callback set 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa53bfff640 10 monclient: dump: 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 10 monclient: shutdown 2024-09-15T02:00:49.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 -- 172.21.15.67:0/1791626657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c068d40 msgr2=0x7fa53c09d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 --2- 172.21.15.67:0/1791626657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c068d40 0x7fa53c09d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa53c09f270 tx=0x7fa5300090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 -- 172.21.15.67:0/1791626657 shutdown_connections 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 --2- 172.21.15.67:0/1791626657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c068d40 0x7fa53c09d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 --2- 172.21.15.67:0/1791626657 >> v2:172.21.15.67:3300/0 conn(0x7fa53c06e1d0 0x7fa53c06ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 -- 172.21.15.67:0/1791626657 >> 172.21.15.67:0/1791626657 conn(0x7fa53c0a2fd0 msgr2=0x7fa53c0a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 -- 172.21.15.67:0/1791626657 shutdown_connections 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.613+0000 7fa543460640 1 -- 172.21.15.67:0/1791626657 wait complete. 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.614+0000 7fa543460640 10 monclient: build_initial_monmap 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.614+0000 7fa543460640 10 monclient: monmap: 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:49.625 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.614+0000 7fa543460640 1 Processor -- start 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.614+0000 7fa543460640 1 -- start start 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.614+0000 7fa543460640 10 monclient: init 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:00:49.626 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): picked mon.noname-b con 0x7fa53c068d40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c06e1d0 0x7fa53c134fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): picked mon.noname-a con 0x7fa53c06e1d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 0x7fa53c11ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): picked mon.noname-c con 0x7fa53c06e5a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1220e0 con 0x7fa53c06e1d0 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c120ba0 con 0x7fa53c068d40 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c142080 con 0x7fa53c06e5a0 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa543460640 10 monclient(hunting): authenticate will time out at 1282.084136s 2024-09-15T02:00:49.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5419fd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 msgr2=0x7fa53c11ea90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45682/0 (socket says 172.21.15.67:45682) 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 -- 172.21.15.67:0/3994435308 learned_addr learned my addr 172.21.15.67:0/3994435308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): get_auth_request con 0x7fa53c068d40 auth_method 0 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5419fd640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 0x7fa53c11ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c06e1d0 0x7fa53c134fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): get_auth_request con 0x7fa53c06e1d0 auth_method 0 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:49.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5409fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient(hunting): handle_auth_done global_id 124264 payload 824 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 -- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 msgr2=0x7fa53c11ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 0x7fa53c11ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 -- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c06e1d0 msgr2=0x7fa53c134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c06e1d0 0x7fa53c134fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:49.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 monclient: found mon.noname-b 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.615+0000 7fa5411fc640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530009390 con 0x7fa53c068d40 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5411fc640 1 -- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 msgr2=0x7fa53c1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5411fc640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa52c007eb0 tx=0x7fa52c009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c015400 con 0x7fa53c068d40 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: dump: 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.630 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c00f040 con 0x7fa53c068d40 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa543460640 5 monclient: authenticate success, global_id 124264 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c0157b0 con 0x7fa53c068d40 2024-09-15T02:00:49.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa5397fa640 10 monclient: dump: 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:49.632 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _renew_subs 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c0a20c0 con 0x7fa53c068d40 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.617+0000 7fa53b7fe640 4 set_mon_vals no callback set 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _renew_subs 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c1211f0 con 0x7fa53c068d40 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa52c015b60 con 0x7fa53c068d40 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.618+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c04bf60 con 0x7fa53c068d40 2024-09-15T02:00:49.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.619+0000 7fa5397fa640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 0x7fa52803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:49.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.619+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 229..255) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa52c04cc10 con 0x7fa53c068d40 2024-09-15T02:00:49.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.619+0000 7fa5409fb640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 0x7fa52803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:49.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.619+0000 7fa5409fb640 10 monclient: get_auth_request con 0x7fa5280386f0 auth_method 0 2024-09-15T02:00:49.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.621+0000 7fa5409fb640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 0x7fa52803aba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa530031000 tx=0x7fa530032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:49.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.623+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa52c0143a0 con 0x7fa53c068d40 2024-09-15T02:00:49.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.623+0000 7fa5397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:49.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.623+0000 7fa5397fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:49.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.798+0000 7fa543460640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:49.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.798+0000 7fa543460640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:49.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.798+0000 7fa543460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:49.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:49.799+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7fa53c120680 con 0x7fa53c068d40 2024-09-15T02:00:49.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:49.800+0000 7f742dbf1640 -1 mon.a@0(leader).osd e255 definitely_dead 0 2024-09-15T02:00:50.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.421+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v256) v1 ==== 83+0+0 (secure 0 0 0) 0x7fa52c013300 con 0x7fa53c068d40 2024-09-15T02:00:50.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.422+0000 7fa5397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:50.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.422+0000 7fa5397fa640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-15T02:00:50.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.530+0000 7fa543460640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:50.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.530+0000 7fa543460640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:50.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.530+0000 7fa543460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.530+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7fa53c09e640 con 0x7fa53c068d40 2024-09-15T02:00:50.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:50.531+0000 7f742dbf1640 -1 mon.a@0(leader).osd e256 definitely_dead 0 2024-09-15T02:00:50.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.531+0000 7fa5397fa640 1 -- 172.21.15.67:0/3994435308 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v256) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa52c01d360 con 0x7fa53c068d40 2024-09-15T02:00:50.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.531+0000 7fa5397fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T02:00:50.534 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.531+0000 7fa5397fa640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-15T02:00:50.534 INFO:tasks.workunit.client.0.smithi067.stderr:osd.14 does not exist. 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 msgr2=0x7fa52803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 0x7fa52803aba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa530031000 tx=0x7fa530032000 comp rx=0 tx=0).stop 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 10 monclient: shutdown 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 msgr2=0x7fa53c1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa52c007eb0 tx=0x7fa52c009330 comp rx=0 tx=0).stop 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 shutdown_connections 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5280386f0 0x7fa52803aba0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa53c06e1d0 0x7fa53c134fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa53c06e5a0 0x7fa53c11ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 --2- 172.21.15.67:0/3994435308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa53c068d40 0x7fa53c1411a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 >> 172.21.15.67:0/3994435308 conn(0x7fa53c0a2fd0 msgr2=0x7fa53c0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:50.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 shutdown_connections 2024-09-15T02:00:50.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.533+0000 7fa543460640 1 -- 172.21.15.67:0/3994435308 wait complete. 2024-09-15T02:00:50.544 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph osd rm 1000 2024-09-15T02:00:50.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.617+0000 7f7c0179d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:50.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.617+0000 7f7c0179d640 10 monclient: build_initial_monmap 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.617+0000 7f7c0179d640 10 monclient: monmap: 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:50.618569+0000 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:50.618569+0000 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:50.620 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.617+0000 7f7c0179d640 1 Processor -- start 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 1 -- start start 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient: init 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bfc0981b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:50.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7bfc098580 0x7f7bfc098980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc098580 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7bfc136ea0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0a07b0 con 0x7f7bfc136ea0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0a16e0 con 0x7f7bfc0981b0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc1329d0 con 0x7f7bfc098580 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7c0179d640 10 monclient(hunting): authenticate will time out at 1283.087559s 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7bfc0981b0 0x7f7bfc1322c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39420/0 (socket says 172.21.15.67:39420) 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfaffd640 1 -- 172.21.15.67:0/3179930466 learned_addr learned my addr 172.21.15.67:0/3179930466 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfa7fc640 1 --2- 172.21.15.67:0/3179930466 >> v2:172.21.15.17:3300/0 conn(0x7f7bfc098580 0x7f7bfc098980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfa7fc640 10 monclient(hunting): get_auth_request con 0x7f7bfc098580 auth_method 0 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.618+0000 7f7bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:50.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.620+0000 7f7bfaffd640 1 -- 172.21.15.67:0/3179930466 >> v1:172.21.15.67:6789/0 conn(0x7f7bfc0981b0 legacy=0x7f7bfc1322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436483348 0 0) 0x7f7bfc0a16e0 con 0x7f7bfc0981b0 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): my global_id is 124267 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be40034e0 con 0x7f7bfc0981b0 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 >> v1:172.21.15.17:6789/0 conn(0x7f7bfc136ea0 legacy=0x7f7bfc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187555374 0 0) 0x7f7be40034e0 con 0x7f7bfc0981b0 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 >> v2:172.21.15.17:3300/0 conn(0x7f7bfc098580 msgr2=0x7f7bfc098980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 --2- 172.21.15.67:0/3179930466 >> v2:172.21.15.17:3300/0 conn(0x7f7bfc098580 0x7f7bfc098980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 >> v1:172.21.15.17:6789/0 conn(0x7f7bfc136ea0 legacy=0x7f7bfc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: _finish_hunting 0 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0a1400 con 0x7f7bfc0981b0 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f7bec001720 con 0x7f7bfc0981b0 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: dump: 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:50.630 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 >> v1:172.21.15.67:6789/0 conn(0x7f7bfc0981b0 legacy=0x7f7bfc1322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7be4003b20 0x7f7be4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): picked mon.a con 0x7f7be4003b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4004780 0x7f7be4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): picked mon.c con 0x7f7be4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 0x7f7be400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): picked mon.b con 0x7f7be4005330 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc1329d0 con 0x7f7be4003b20 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be400c2d0 con 0x7f7be4005330 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be400c490 con 0x7f7be4004780 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bf9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:00:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfa7fc640 1 -- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4004780 msgr2=0x7f7be4004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfa7fc640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4004780 0x7f7be4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 0x7f7be400b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f7be4005330 auth_method 0 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7be4003b20 0x7f7be4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): get_auth_request con 0x7f7be4003b20 auth_method 0 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:50.633 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_done global_id 124267 payload 293 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4004780 msgr2=0x7f7be4004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4004780 0x7f7be4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7be4003b20 msgr2=0x7f7be4003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7be4003b20 0x7f7be4003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.634 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 monclient: found mon.b 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be40034e0 con 0x7f7be4005330 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 10 monclient: _finish_auth 0 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7c0179d640 5 monclient: authenticate success, global_id 124267 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.621+0000 7f7c0179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.622+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 reap_dead start 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 msgr2=0x7f7be400b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 0x7f7be400b9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40021c0 tx=0x7f7bf4001890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7d828b0015140c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4005b00 con 0x7f7be4005330 2024-09-15T02:00:50.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bf9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.624+0000 7f7bf9ffb640 10 monclient: dump: 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:50.636 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf402d070 con 0x7f7be4005330 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 4 set_mon_vals no callback set 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 1 -- 172.21.15.67:0/3179930466 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4005ca0 con 0x7f7be4005330 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7bf9ffb640 10 monclient: dump: 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 10 monclient: shutdown 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 -- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 msgr2=0x7f7be400b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 0x7f7be400b9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40021c0 tx=0x7f7bf4001890 comp rx=0 tx=0).stop 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 -- 172.21.15.67:0/3179930466 shutdown_connections 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 --2- 172.21.15.67:0/3179930466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be4005330 0x7f7be400b9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 -- 172.21.15.67:0/3179930466 >> 172.21.15.67:0/3179930466 conn(0x7f7bfc071150 msgr2=0x7f7bfc0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:50.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 -- 172.21.15.67:0/3179930466 shutdown_connections 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.625+0000 7f7c0179d640 1 -- 172.21.15.67:0/3179930466 wait complete. 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.626+0000 7f7c0179d640 10 monclient: build_initial_monmap 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.626+0000 7f7c0179d640 10 monclient: monmap: 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:50.639 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.626+0000 7f7c0179d640 1 Processor -- start 2024-09-15T02:00:50.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.626+0000 7f7c0179d640 1 -- start start 2024-09-15T02:00:50.640 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.626+0000 7f7c0179d640 10 monclient: init 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 0x7f7bfc141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-c con 0x7f7be4005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7bfc141850 0x7f7bfc141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc141850 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): picked mon.noname-b con 0x7f7bfc134a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc11be00 con 0x7f7bfc141850 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc098f00 con 0x7f7bfc134a50 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfaffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 msgr2=0x7f7bfc141310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc1421e0 con 0x7f7be4005330 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7c0179d640 10 monclient(hunting): authenticate will time out at 1283.096242s 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 0x7f7bfc141310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56182/0 (socket says 172.21.15.67:56182) 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/4264155889 learned_addr learned my addr 172.21.15.67:0/4264155889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f7bfc134a50 auth_method 0 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7bfc141850 0x7f7bfc141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): get_auth_request con 0x7f7bfc141850 auth_method 0 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:50.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient(hunting): handle_auth_done global_id 124270 payload 824 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 msgr2=0x7f7bfc141310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 0x7f7bfc141310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7bfc141850 msgr2=0x7f7bfc141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7bfc141850 0x7f7bfc141c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 monclient: found mon.noname-b 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:50.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.627+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0a40d0 con 0x7f7bfc134a50 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.629+0000 7f7bfb7fe640 1 -- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 msgr2=0x7f7bfc134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.629+0000 7f7bfb7fe640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4001490 tx=0x7f7bf4033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4032640 con 0x7f7bfc134a50 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: dump: 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:50.648 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf40327a0 con 0x7f7bfc134a50 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4032b40 con 0x7f7bfc134a50 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 5 monclient: authenticate success, global_id 124270 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 10 monclient: _renew_subs 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc11b950 con 0x7f7bfc134a50 2024-09-15T02:00:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7bf97fa640 4 set_mon_vals no callback set 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 10 monclient: dump: 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:50.650 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 10 monclient: _renew_subs 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc11d010 con 0x7f7bfc134a50 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.630+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7bf402f030 con 0x7f7bfc134a50 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7c0179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7c0179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7c0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc04bf60 con 0x7f7bfc134a50 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7be2ffd640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 0x7f7be803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 229..256) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f7bf406c730 con 0x7f7bfc134a50 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7bfaffd640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 0x7f7be803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.631+0000 7f7bfaffd640 10 monclient: get_auth_request con 0x7f7be8038740 auth_method 0 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.633+0000 7f7bfaffd640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 0x7f7be803abf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7bec007fd0 tx=0x7f7bec007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.635+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7bf4046410 con 0x7f7bfc134a50 2024-09-15T02:00:50.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.635+0000 7f7be2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:50.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.635+0000 7f7be2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:50.808 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.806+0000 7f7c0179d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.806+0000 7f7c0179d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.806+0000 7f7c0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.806+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7bfc0a19c0 con 0x7f7bfc134a50 2024-09-15T02:00:50.809 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:50.807+0000 7f742dbf1640 -1 mon.a@0(leader).osd e256 definitely_dead 0 2024-09-15T02:00:50.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.807+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v256) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7bf4045370 con 0x7f7bfc134a50 2024-09-15T02:00:50.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.807+0000 7f7be2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.810 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.807+0000 7f7be2ffd640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-15T02:00:50.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.913+0000 7f7c0179d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.913+0000 7f7c0179d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.913+0000 7f7c0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:50.916 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.913+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7bfc096a70 con 0x7f7bfc134a50 2024-09-15T02:00:50.916 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:00:50.914+0000 7f742dbf1640 -1 mon.a@0(leader).osd e256 definitely_dead 0 2024-09-15T02:00:50.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.914+0000 7f7be2ffd640 1 -- 172.21.15.67:0/4264155889 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v256) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7bf4045550 con 0x7f7bfc134a50 2024-09-15T02:00:50.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.914+0000 7f7be2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T02:00:50.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.914+0000 7f7be2ffd640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-15T02:00:50.917 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1000 does not exist. 2024-09-15T02:00:50.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 msgr2=0x7f7be803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 0x7f7be803abf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7bec007fd0 tx=0x7f7bec007a50 comp rx=0 tx=0).stop 2024-09-15T02:00:50.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 10 monclient: shutdown 2024-09-15T02:00:50.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:50.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 msgr2=0x7f7bfc134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4001490 tx=0x7f7bf4033000 comp rx=0 tx=0).stop 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 shutdown_connections 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7be8038740 0x7f7be803abf0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7bfc141850 0x7f7bfc141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bfc134a50 0x7f7bfc134ec0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 --2- 172.21.15.67:0/4264155889 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7be4005330 0x7f7bfc141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.916+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 >> 172.21.15.67:0/4264155889 conn(0x7f7bfc071150 msgr2=0x7f7bfc09ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.917+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 shutdown_connections 2024-09-15T02:00:50.920 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:50.917+0000 7f7c0179d640 1 -- 172.21.15.67:0/4264155889 wait complete. 2024-09-15T02:00:50.928 INFO:tasks.workunit.client.0.smithi067.stderr:++ ceph osd create 7b19bad9-2bed-4846-b592-7009ac3d12e0 2024-09-15T02:00:51.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.000+0000 7f32a891d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:51.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.000+0000 7f32a891d640 10 monclient: build_initial_monmap 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.000+0000 7f32a891d640 10 monclient: monmap: 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:51.001586+0000 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:51.001586+0000 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:51.003 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.000+0000 7f32a891d640 1 Processor -- start 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 -- start start 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient: init 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32a4130e00 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 0x7f32a41315f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-c con 0x7f32a41311d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f32a4136f10 0x7f32a41392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-a con 0x7f32a4136f10 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:51.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a409ff20 con 0x7f32a4130e00 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a41320b0 con 0x7f32a4136f10 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a40a17b0 con 0x7f32a41311d0 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a891d640 10 monclient(hunting): authenticate will time out at 1283.470705s 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 0x7f32a41315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 0x7f32a41315f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56194/0 (socket says 172.21.15.67:56194) 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 1 -- 172.21.15.67:0/1929646541 learned_addr learned my addr 172.21.15.67:0/1929646541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): get_auth_request con 0x7f32a41311d0 auth_method 0 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:51.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a2d9d640 1 --2- 172.21.15.67:0/1929646541 >> v2:172.21.15.17:3300/0 conn(0x7f32a4136f10 0x7f32a41392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request con 0x7f32a4136f10 auth_method 0 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.001+0000 7f32a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:51.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_done global_id 104772 payload 274 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 1 -- 172.21.15.67:0/1929646541 >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 msgr2=0x7f32a41315f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 1 --2- 172.21.15.67:0/1929646541 >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 0x7f32a41315f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 1 -- 172.21.15.67:0/1929646541 >> v1:172.21.15.67:6789/0 conn(0x7f32a4130e00 legacy=0x7f32a4132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 1 monclient: found mon.noname-a 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:51.009 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.002+0000 7f32a2d9d640 1 -- 172.21.15.67:0/1929646541 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a40a00f0 con 0x7f32a4136f10 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a2d9d640 1 -- 172.21.15.67:0/1929646541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 msgr2=0x7f32a41392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a2d9d640 1 --2- 172.21.15.67:0/1929646541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a41392f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32a40990e0 tx=0x7f329c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b076dfc1d6e5dde server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 1 -- 172.21.15.67:0/1929646541 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c007760 con 0x7f32a4136f10 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:51.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: dump: 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: _finish_auth 0 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:51.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 5 monclient: authenticate success, global_id 104772 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 1 -- 172.21.15.67:0/1929646541 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c02d050 con 0x7f32a4136f10 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 1 -- 172.21.15.67:0/1929646541 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c007b30 con 0x7f32a4136f10 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 4 set_mon_vals no callback set 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a159a640 10 monclient: dump: 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.016 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 10 monclient: shutdown 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 1 -- 172.21.15.67:0/1929646541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 msgr2=0x7f32a41392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.005+0000 7f32a891d640 1 --2- 172.21.15.67:0/1929646541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a41392f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32a40990e0 tx=0x7f329c009780 comp rx=0 tx=0).stop 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 -- 172.21.15.67:0/1929646541 shutdown_connections 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 --2- 172.21.15.67:0/1929646541 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a41392f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 --2- 172.21.15.67:0/1929646541 >> v2:172.21.15.67:3300/0 conn(0x7f32a41311d0 0x7f32a41315f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 -- 172.21.15.67:0/1929646541 >> 172.21.15.67:0/1929646541 conn(0x7f32a40710f0 msgr2=0x7f32a40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 -- 172.21.15.67:0/1929646541 shutdown_connections 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 -- 172.21.15.67:0/1929646541 wait complete. 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 10 monclient: build_initial_monmap 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 10 monclient: monmap: 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:51.018 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 Processor -- start 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 1 -- start start 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.006+0000 7f32a891d640 10 monclient: init 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-b con 0x7f32a4130e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 0x7f32a4097f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-c con 0x7f32a41311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a4098460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): picked mon.noname-a con 0x7f32a4136f10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:51.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40a0820 con 0x7f32a4136f10 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40a37c0 con 0x7f32a4130e00 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a4099540 con 0x7f32a41311d0 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a891d640 10 monclient(hunting): authenticate will time out at 1283.476330s 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56200/0 (socket says 172.21.15.67:56200) 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 -- 172.21.15.67:0/1672424134 learned_addr learned my addr 172.21.15.67:0/1672424134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): get_auth_request con 0x7f32a4130e00 auth_method 0 2024-09-15T02:00:51.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a1d9b640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 msgr2=0x7f32a4097f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a1d9b640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 0x7f32a4097f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a2d9d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a4098460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient(hunting): handle_auth_done global_id 124276 payload 824 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 msgr2=0x7f32a4097f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 0x7f32a4097f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 msgr2=0x7f32a4098460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:51.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a4098460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient: _finish_hunting 0 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 monclient: found mon.noname-b 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a259c640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c009a20 con 0x7f32a4130e00 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a2d9d640 10 monclient: get_auth_request con 0x7f32a4136f10 auth_method 0 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.007+0000 7f32a2d9d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a4098460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.009+0000 7f32a259c640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 msgr2=0x7f32a40979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.009+0000 7f32a259c640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f329400c610 tx=0x7f3294007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3294012920 con 0x7f32a4130e00 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:51.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: dump: 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:51.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3294012ac0 con 0x7f32a4130e00 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 5 monclient: authenticate success, global_id 124276 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 10 monclient: _renew_subs 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a409b000 con 0x7f32a4130e00 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3294012e80 con 0x7f32a4130e00 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 10 monclient: dump: 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:51.024 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a0d99640 4 set_mon_vals no callback set 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 10 monclient: _renew_subs 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a409a0a0 con 0x7f32a4130e00 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.010+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3294011590 con 0x7f32a4130e00 2024-09-15T02:00:51.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a891d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a891d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a409b6e0 con 0x7f32a4130e00 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f329a7fc640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 0x7f328403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 229..256) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f329404c2b0 con 0x7f32a4130e00 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a1d9b640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 0x7f328403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.011+0000 7f32a1d9b640 10 monclient: get_auth_request con 0x7f32840386f0 auth_method 0 2024-09-15T02:00:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.013+0000 7f32a1d9b640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 0x7f328403aba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f328c006c10 tx=0x7f328c0091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.015+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f32940198c0 con 0x7f32a4130e00 2024-09-15T02:00:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.015+0000 7f329a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.015+0000 7f329a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.190+0000 7f32a891d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.190+0000 7f32a891d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.190+0000 7f32a891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.190+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f32a40a14d0 con 0x7f32a4130e00 2024-09-15T02:00:51.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.422+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v257) v1 ==== 106+0+3 (secure 0 0 0) 0x7f3294019310 con 0x7f32a4130e00 2024-09-15T02:00:51.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.422+0000 7f329a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.422+0000 7f329a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:51.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.530+0000 7f32a891d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.530+0000 7f32a891d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.530+0000 7f32a891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.530+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"} v 0) v1 -- 0x7f32a404bf60 con 0x7f32a4130e00 2024-09-15T02:00:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.531+0000 7f329a7fc640 1 -- 172.21.15.67:0/1672424134 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}]=0 v257) v1 ==== 106+0+3 (secure 0 0 0) 0x7f32940198c0 con 0x7f32a4130e00 2024-09-15T02:00:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.531+0000 7f329a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "7b19bad9-2bed-4846-b592-7009ac3d12e0"}] 2024-09-15T02:00:51.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.531+0000 7f329a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 msgr2=0x7f328403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 0x7f328403aba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f328c006c10 tx=0x7f328c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 10 monclient: shutdown 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 msgr2=0x7f32a40979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f329400c610 tx=0x7f3294007ed0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 shutdown_connections 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f32840386f0 0x7f328403aba0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32a4136f10 0x7f32a4098460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f32a41311d0 0x7f32a4097f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 --2- 172.21.15.67:0/1672424134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a4130e00 0x7f32a40979e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 >> 172.21.15.67:0/1672424134 conn(0x7f32a40710f0 msgr2=0x7f32a40a39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 shutdown_connections 2024-09-15T02:00:51.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:51.533+0000 7f32a891d640 1 -- 172.21.15.67:0/1672424134 wait complete. 2024-09-15T02:00:51.544 INFO:tasks.workunit.client.0.smithi067.stderr:+ na2=12 2024-09-15T02:00:51.545 INFO:tasks.workunit.client.0.smithi067.stdout:OK 2024-09-15T02:00:51.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo OK 2024-09-15T02:00:51.546 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:00:51.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:00:51.613 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-15T02:00:51.614 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-15T02:00:51.614 DEBUG:teuthology.orchestra.run.smithi067:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-15T02:00:51.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-15T02:00:51.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ exit_on_error=1 2024-09-15T02:00:51.674 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ ! -z '' ]] 2024-09-15T02:00:51.674 INFO:tasks.workunit.client.0.smithi067.stderr:++ uname 2024-09-15T02:00:51.675 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' Linux = FreeBSD ']' 2024-09-15T02:00:51.675 INFO:tasks.workunit.client.0.smithi067.stderr:+ ETIMEDOUT=110 2024-09-15T02:00:51.675 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-15T02:00:51.675 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-15T02:00:51.675 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:00:51.676 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T02:00:51.676 INFO:tasks.workunit.client.0.smithi067.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T02:00:51.676 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T02:00:52.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:00:52.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:00:52.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:52.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-15T02:00:52.167 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T02:00:52.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-15T02:00:52.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:00:52.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T02:00:52.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T02:00:52.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:00:52.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:00:52.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:52.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph auth del client.bazar 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: get_monmap_and_config 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: build_initial_monmap 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: monmap: 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:00:52.319545+0000 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:00:52.319545+0000 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:52.321 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:52.322 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:52.322 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:52.322 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 Processor -- start 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 -- start start 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: init 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa00c130e00 0x7fa00c132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fa00c130e00 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:52.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa00c1311d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 0x7fa00c0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fa00c1315f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00c09ff20 con 0x7fa00c1311d0 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00c1320b0 con 0x7fa00c130e00 2024-09-15T02:00:52.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c131a30 con 0x7fa00c1315f0 2024-09-15T02:00:52.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:52.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa011ec9640 10 monclient(hunting): authenticate will time out at 1284.788684s 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa00c130e00 0x7fa00c132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa00c130e00 0x7fa00c132430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56230/0 (socket says 172.21.15.67:56230) 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 1 -- 172.21.15.67:0/3210621768 learned_addr learned my addr 172.21.15.67:0/3210621768 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): get_auth_request con 0x7fa00c130e00 auth_method 0 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00bfff640 1 -- 172.21.15.67:0/3210621768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 msgr2=0x7fa00c0694d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00bfff640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 0x7fa00c0694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:52.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.319+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_done global_id 124288 payload 274 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 1 -- 172.21.15.67:0/3210621768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 msgr2=0x7fa00c0694d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 0x7fa00c0694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 1 -- 172.21.15.67:0/3210621768 >> v1:172.21.15.17:6789/0 conn(0x7fa00c1311d0 legacy=0x7fa00c068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 1 monclient: found mon.noname-c 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.320+0000 7fa00b7fe640 1 -- 172.21.15.67:0/3210621768 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c0a00f0 con 0x7fa00c130e00 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00b7fe640 1 -- 172.21.15.67:0/3210621768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c130e00 msgr2=0x7fa00c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00b7fe640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c130e00 0x7fa00c132430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009910 tx=0x7f9ffc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a81c3889092c264 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 1 -- 172.21.15.67:0/3210621768 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc032440 con 0x7fa00c130e00 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: dump: 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: _finish_auth 0 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 1 -- 172.21.15.67:0/3210621768 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc0325a0 con 0x7fa00c130e00 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 5 monclient: authenticate success, global_id 124288 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 1 -- 172.21.15.67:0/3210621768 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc032940 con 0x7fa00c130e00 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 4 set_mon_vals no callback set 2024-09-15T02:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa00affd640 10 monclient: dump: 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 10 monclient: shutdown 2024-09-15T02:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 -- 172.21.15.67:0/3210621768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c130e00 msgr2=0x7fa00c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c130e00 0x7fa00c132430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009910 tx=0x7f9ffc0091d0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 -- 172.21.15.67:0/3210621768 shutdown_connections 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1315f0 0x7fa00c0694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 --2- 172.21.15.67:0/3210621768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c130e00 0x7fa00c132430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 -- 172.21.15.67:0/3210621768 >> 172.21.15.67:0/3210621768 conn(0x7fa00c0710f0 msgr2=0x7fa00c0a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 -- 172.21.15.67:0/3210621768 shutdown_connections 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.323+0000 7fa011ec9640 1 -- 172.21.15.67:0/3210621768 wait complete. 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient: build_initial_monmap 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient: monmap: 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 Processor -- start 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 -- start start 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient: init 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa00c130e00 0x7fa00c141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.341 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fa00c130e00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 0x7fa00c12f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fa00c1311d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fa00c1315f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c142590 con 0x7fa00c130e00 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c0a4ba0 con 0x7fa00c1315f0 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c11c770 con 0x7fa00c1311d0 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.324+0000 7fa011ec9640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:52.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa011ec9640 10 monclient(hunting): authenticate will time out at 1284.794007s 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa002ffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 msgr2=0x7fa00c12f000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa002ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 0x7fa00c12f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56236/0 (socket says 172.21.15.67:56236) 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 -- 172.21.15.67:0/834862948 learned_addr learned my addr 172.21.15.67:0/834862948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): get_auth_request con 0x7fa00c1315f0 auth_method 0 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa00c130e00 0x7fa00c141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): get_auth_request con 0x7fa00c130e00 auth_method 0 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.343 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient(hunting): handle_auth_done global_id 124291 payload 824 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 -- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 msgr2=0x7fa00c12f000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 0x7fa00c12f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 -- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa00c130e00 msgr2=0x7fa00c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa00c130e00 0x7fa00c141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:00:52.344 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 monclient: found mon.noname-b 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.325+0000 7fa00bfff640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc009450 con 0x7fa00c1315f0 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa00bfff640 1 -- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 msgr2=0x7fa00c09ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa00bfff640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa00400bb10 tx=0x7fa0040097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400cbc0 con 0x7fa00c1315f0 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: dump: 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.345 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00400cd40 con 0x7fa00c1315f0 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 5 monclient: authenticate success, global_id 124291 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 10 monclient: _renew_subs 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c0a07d0 con 0x7fa00c1315f0 2024-09-15T02:00:52.346 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004006150 con 0x7fa00c1315f0 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa003fff640 10 monclient: dump: 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.347 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa00a7fc640 4 set_mon_vals no callback set 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 10 monclient: _renew_subs 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.327+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c09f0b0 con 0x7fa00c1315f0 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa004012020 con 0x7fa00c1315f0 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa003fff640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 0x7f9fec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa011ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa011ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa011ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa00404c4b0 con 0x7fa00c1315f0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.328+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c04bf60 con 0x7fa00c1315f0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.329+0000 7fa00b7fe640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 0x7f9fec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.329+0000 7fa00b7fe640 10 monclient: get_auth_request con 0x7f9fec038740 auth_method 0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.331+0000 7fa00b7fe640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 0x7f9fec03abf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0990e0 tx=0x7f9ffc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.333+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa00401b030 con 0x7fa00c1315f0 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.333+0000 7fa003fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.349 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.333+0000 7fa003fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:52.557 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:00:52.557 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T02:00:52.557 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:00:52.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:00:52.561 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:00:52.561 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:00:52.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.559+0000 7fa011ec9640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.559+0000 7fa011ec9640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.559+0000 7fa011ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.559+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7fa00c09c140 con 0x7fa00c1315f0 2024-09-15T02:00:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.563+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7fa00401b020 con 0x7fa00c1315f0 2024-09-15T02:00:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.563+0000 7fa003fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.563+0000 7fa003fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:52.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.668+0000 7f77bd75d640 10 monclient: get_monmap_and_config 2024-09-15T02:00:52.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.668+0000 7f77bd75d640 10 monclient: build_initial_monmap 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.668+0000 7f77bd75d640 10 monclient: monmap: 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:52.669588+0000 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:52.669588+0000 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:52.671 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 Processor -- start 2024-09-15T02:00:52.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 -- start start 2024-09-15T02:00:52.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient: init 2024-09-15T02:00:52.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77b80a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f77b8132040 0x7f77b80987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-a con 0x7f77b8132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77b8136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80a01e0 con 0x7f77b80a58b0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80a1580 con 0x7f77b8136dd0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b80a5c80 con 0x7f77b8132040 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.669+0000 7f77bd75d640 10 monclient(hunting): authenticate will time out at 1312.665980s 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f77b8132040 0x7f77b80987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f77b8136dd0 0x7f77b81391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:52572/0 (socket says 172.21.15.17:52572) 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f77b8132040 0x7f77b80987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38028/0 (socket says 172.21.15.17:38028) 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b77fe640 1 -- 172.21.15.17:0/3388045927 learned_addr learned my addr 172.21.15.17:0/3388045927 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): get_auth_request con 0x7f77b8132040 auth_method 0 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.670+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b77fe640 1 -- 172.21.15.17:0/3388045927 >> v1:172.21.15.67:6789/0 conn(0x7f77b8136dd0 legacy=0x7f77b81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b6ffd640 1 -- 172.21.15.17:0/3388045927 >> v1:172.21.15.17:6789/0 conn(0x7f77b80a58b0 legacy=0x7f77b80980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 10 monclient(hunting): handle_auth_done global_id 104790 payload 274 2024-09-15T02:00:52.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 1 -- 172.21.15.17:0/3388045927 >> v1:172.21.15.67:6789/0 conn(0x7f77b8136dd0 legacy=0x7f77b81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 1 -- 172.21.15.17:0/3388045927 >> v1:172.21.15.17:6789/0 conn(0x7f77b80a58b0 legacy=0x7f77b80980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b5ffb640 1 -- 172.21.15.17:0/3388045927 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067464783 0 0) 0x7f77b80a1580 con 0x7f77b8136dd0 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b67fc640 1 -- 172.21.15.17:0/3388045927 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b8132440 con 0x7f77b8132040 2024-09-15T02:00:52.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.672+0000 7f77b5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b67fc640 1 -- 172.21.15.17:0/3388045927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 msgr2=0x7f77b80987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b67fc640 1 --2- 172.21.15.17:0/3388045927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b80987b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f77b80a1710 tx=0x7f77a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6329d8ef141cb00c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 1 -- 172.21.15.17:0/3388045927 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a002f4b0 con 0x7f77b8132040 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: dump: 2024-09-15T02:00:52.681 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 1 -- 172.21.15.17:0/3388045927 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a002f610 con 0x7f77b8132040 2024-09-15T02:00:52.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 5 monclient: authenticate success, global_id 104790 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 4 set_mon_vals no callback set 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 1 -- 172.21.15.17:0/3388045927 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a002f9f0 con 0x7f77b8132040 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77b5ffb640 10 monclient: dump: 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.683 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 10 monclient: shutdown 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 1 -- 172.21.15.17:0/3388045927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 msgr2=0x7f77b80987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.675+0000 7f77bd75d640 1 --2- 172.21.15.17:0/3388045927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b80987b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f77b80a1710 tx=0x7f77a00090f0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 1 -- 172.21.15.17:0/3388045927 shutdown_connections 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 1 --2- 172.21.15.17:0/3388045927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b80987b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 1 -- 172.21.15.17:0/3388045927 >> 172.21.15.17:0/3388045927 conn(0x7f77b8070ed0 msgr2=0x7f77b80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 1 -- 172.21.15.17:0/3388045927 shutdown_connections 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 1 -- 172.21.15.17:0/3388045927 wait complete. 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 10 monclient: build_initial_monmap 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.676+0000 7f77bd75d640 10 monclient: monmap: 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 Processor -- start 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 -- start start 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient: init 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-c con 0x7f77b80a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-a con 0x7f77b8132040 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8136dd0 0x7f77b8133020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): picked mon.noname-b con 0x7f77b8136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b809f3f0 con 0x7f77b8132040 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b809ec50 con 0x7f77b8136dd0 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b8098dc0 con 0x7f77b80a58b0 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77bd75d640 10 monclient(hunting): authenticate will time out at 1312.673659s 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38034/0 (socket says 172.21.15.17:38034) 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 1 -- 172.21.15.17:0/2601964274 learned_addr learned my addr 172.21.15.17:0/2601964274 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): get_auth_request con 0x7f77b8132040 auth_method 0 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 10 monclient(hunting): get_auth_request con 0x7f77b80a58b0 auth_method 0 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8136dd0 0x7f77b8133020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): get_auth_request con 0x7f77b8136dd0 auth_method 0 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 msgr2=0x7f77b809e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 msgr2=0x7f77b809e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:52.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b6ffd640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.677+0000 7f77b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 10 monclient(hunting): handle_auth_done global_id 104793 payload 824 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 msgr2=0x7f77b809e040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8136dd0 msgr2=0x7f77b8133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8136dd0 0x7f77b8133020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 10 monclient: _finish_hunting 0 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 monclient: found mon.noname-a 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.678+0000 7f77b67fc640 1 -- 172.21.15.17:0/2601964274 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0009390 con 0x7f77b8132040 2024-09-15T02:00:52.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77b67fc640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 msgr2=0x7f77b8132ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77b67fc640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f77a002a7f0 tx=0x7f77a0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0032a60 con 0x7f77b8132040 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: dump: 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.693 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: _finish_auth 0 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a0032bc0 con 0x7f77b8132040 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 5 monclient: authenticate success, global_id 104793 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 10 monclient: _renew_subs 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b813a610 con 0x7f77b8132040 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a003c400 con 0x7f77b8132040 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77aeffd640 10 monclient: dump: 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77b57fa640 4 set_mon_vals no callback set 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 10 monclient: _renew_subs 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.680+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b8135da0 con 0x7f77b8132040 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f77a003c7f0 con 0x7f77b8132040 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77bd75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77bd75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77bd75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77aeffd640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 0x7f779803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b8063ae0 con 0x7f77b8132040 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f77a006cc50 con 0x7f77b8132040 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77b6ffd640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 0x7f779803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.681+0000 7f77b6ffd640 10 monclient: get_auth_request con 0x7f7798038740 auth_method 0 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.684+0000 7f77b6ffd640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 0x7f779803abf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f77a8007ae0 tx=0x7f77a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.686+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f77a0037070 con 0x7f77b8132040 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.686+0000 7f77aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:52.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.686+0000 7f77aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:52.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.722+0000 7fa011ec9640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.722+0000 7fa011ec9640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.722+0000 7fa011ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:52.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.722+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7fa00c0008d0 con 0x7fa00c1315f0 2024-09-15T02:00:52.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.723+0000 7fa003fff640 1 -- 172.21.15.67:0/834862948 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7fa00401d900 con 0x7fa00c1315f0 2024-09-15T02:00:52.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.723+0000 7fa003fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T02:00:52.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.723+0000 7fa003fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:00:52.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 msgr2=0x7f9fec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 0x7f9fec03abf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0990e0 tx=0x7f9ffc032000 comp rx=0 tx=0).stop 2024-09-15T02:00:52.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 10 monclient: shutdown 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 msgr2=0x7fa00c09ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa00400bb10 tx=0x7fa0040097d0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 shutdown_connections 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9fec038740 0x7f9fec03abf0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c1315f0 0x7fa00c09ca60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa00c1311d0 0x7fa00c12f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 --2- 172.21.15.67:0/834862948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa00c130e00 0x7fa00c141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 >> 172.21.15.67:0/834862948 conn(0x7fa00c0710f0 msgr2=0x7fa00c12fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 shutdown_connections 2024-09-15T02:00:52.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:00:52.725+0000 7fa011ec9640 1 -- 172.21.15.67:0/834862948 wait complete. 2024-09-15T02:00:52.735 INFO:tasks.workunit.client.0.smithi067.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-15T02:00:52.735 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-15T02:00:52.736 INFO:tasks.workunit.client.0.smithi067.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T02:00:52.736 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-15T02:00:52.736 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:00:52.736 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T02:00:52.736 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77bd75d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77bd75d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77bd75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f77b8134d10 con 0x7f77b8132040 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77aeffd640 1 -- 172.21.15.17:0/2601964274 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f77b8134d10 con 0x7f77b8132040 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:52.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.914+0000 7f77aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:52.917 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":104,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 msgr2=0x7f779803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 0x7f779803abf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f77a8007ae0 tx=0x7f77a8009040 comp rx=0 tx=0).stop 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 10 monclient: shutdown 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 msgr2=0x7f77b8132ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f77a002a7f0 tx=0x7f77a0032000 comp rx=0 tx=0).stop 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 shutdown_connections 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7798038740 0x7f779803abf0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f77b8132040 0x7f77b8132ae0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8136dd0 0x7f77b8133020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 --2- 172.21.15.17:0/2601964274 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f77b80a58b0 0x7f77b809e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 >> 172.21.15.17:0/2601964274 conn(0x7f77b8070ed0 msgr2=0x7f77b809b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 shutdown_connections 2024-09-15T02:00:52.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:52.916+0000 7f77bd75d640 1 -- 172.21.15.17:0/2601964274 wait complete. 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:00:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:00:54.223 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:00:54.223 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:00:54.708 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:00:54.708 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:00:54.708 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:54.708 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-15T02:00:54.708 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-15T02:00:54.709 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:00:54.709 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T02:00:54.709 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T02:00:54.709 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-15T02:00:55.187 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:00:55.188 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T02:00:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T02:00:55.188 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:00:55.509 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T02:00:55.510 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T02:00:55.510 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T02:00:55.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 10 monclient: get_monmap_and_config 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 10 monclient: build_initial_monmap 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 10 monclient: monmap: 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:56.037887+0000 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:56.037887+0000 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:56.040 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:56.040 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 1 Processor -- start 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 1 -- start start 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.037+0000 7f9d7f2af640 10 monclient: init 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T02:00:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9d78131b20 0x7f9d7809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-a con 0x7f9d78131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-b con 0x7f9d780a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d7809f1e0 0x7f9d78132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7809f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d780a0040 con 0x7f9d78131b20 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7809bff0 con 0x7f9d7809f1e0 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a14a0 con 0x7f9d780a0280 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7f2af640 10 monclient(hunting): authenticate will time out at 1316.034480s 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9d78131b20 0x7f9d7809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9d78131b20 0x7f9d7809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48378/0 (socket says 172.21.15.17:48378) 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 -- 172.21.15.17:0/1606082140 learned_addr learned my addr 172.21.15.17:0/1606082140 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): get_auth_request con 0x7f9d78131b20 auth_method 0 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 1 --2- 172.21.15.17:0/1606082140 >> v2:172.21.15.67:3300/0 conn(0x7f9d7809f1e0 0x7f9d78132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 10 monclient(hunting): get_auth_request con 0x7f9d780a0280 auth_method 0 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): get_auth_request con 0x7f9d7809f1e0 auth_method 0 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 1 -- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 msgr2=0x7f9d78131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 1 -- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 msgr2=0x7f9d78131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7c84a640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient(hunting): handle_auth_done global_id 104823 payload 274 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 -- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 msgr2=0x7f9d78131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 -- 172.21.15.17:0/1606082140 >> v2:172.21.15.67:3300/0 conn(0x7f9d7809f1e0 msgr2=0x7f9d78132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/1606082140 >> v2:172.21.15.67:3300/0 conn(0x7f9d7809f1e0 0x7f9d78132430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient: _finish_hunting 0 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 monclient: found mon.noname-a 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:00:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.038+0000 7f9d7d04b640 1 -- 172.21.15.17:0/1606082140 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78068e80 con 0x7f9d78131b20 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.041+0000 7f9d7d04b640 1 -- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 msgr2=0x7f9d7809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.041+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d7809eca0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9d780a2f90 tx=0x7f9d68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65604e8d160a8bfd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.041+0000 7f9d77fff640 1 -- 172.21.15.17:0/1606082140 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68007610 con 0x7f9d78131b20 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: dump: 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.056 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: _finish_auth 0 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 1 -- 172.21.15.17:0/1606082140 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6802d050 con 0x7f9d78131b20 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 5 monclient: authenticate success, global_id 104823 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:56.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 10 monclient: get_monmap_and_config success 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 1 -- 172.21.15.17:0/1606082140 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680079a0 con 0x7f9d78131b20 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 4 set_mon_vals no callback set 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d77fff640 10 monclient: dump: 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 10 monclient: shutdown 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 -- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 msgr2=0x7f9d7809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d7809eca0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9d780a2f90 tx=0x7f9d68009840 comp rx=0 tx=0).stop 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 -- 172.21.15.17:0/1606082140 shutdown_connections 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/1606082140 >> v2:172.21.15.67:3300/0 conn(0x7f9d7809f1e0 0x7f9d78132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d780a0280 0x7f9d78131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/1606082140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d7809eca0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 -- 172.21.15.17:0/1606082140 >> 172.21.15.17:0/1606082140 conn(0x7f9d78071270 msgr2=0x7f9d780a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 -- 172.21.15.17:0/1606082140 shutdown_connections 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.042+0000 7f9d7f2af640 1 -- 172.21.15.17:0/1606082140 wait complete. 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.043+0000 7f9d7f2af640 10 monclient: build_initial_monmap 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.043+0000 7f9d7f2af640 10 monclient: monmap: 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:00:56.060 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.043+0000 7f9d7f2af640 1 Processor -- start 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.043+0000 7f9d7f2af640 1 -- start start 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.043+0000 7f9d7f2af640 10 monclient: init 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7809f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 0x7f9d78141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-b con 0x7f9d780a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d78141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): picked mon.noname-a con 0x7f9d78131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:00:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78120470 con 0x7f9d78131b20 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a09b0 con 0x7f9d780a0280 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7809e780 con 0x7f9d7809f1e0 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7f2af640 10 monclient(hunting): authenticate will time out at 1316.040219s 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d78141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 0x7f9d78141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40816/0 (socket says 172.21.15.17:40816) 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): get_auth_request con 0x7f9d78131b20 auth_method 0 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 -- 172.21.15.17:0/3386906435 learned_addr learned my addr 172.21.15.17:0/3386906435 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): get_auth_request con 0x7f9d780a0280 auth_method 0 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 10 monclient(hunting): get_auth_request con 0x7f9d7809f1e0 auth_method 0 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 msgr2=0x7f9d78140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 msgr2=0x7f9d78140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient(hunting): handle_auth_done global_id 124336 payload 824 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 msgr2=0x7f9d78140fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 msgr2=0x7f9d78141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d78141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient: _finish_hunting 0 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 monclient: found mon.noname-b 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7c84a640 1 -- 172.21.15.17:0/3386906435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68009ae0 con 0x7f9d780a0280 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.044+0000 7f9d7d84c640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d78141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.046+0000 7f9d7c84a640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 msgr2=0x7f9d78141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.046+0000 7f9d7c84a640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 0x7f9d78141520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006ea0 tx=0x7f9d6c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00b8d0 con 0x7f9d780a0280 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: dump: 2024-09-15T02:00:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: _finish_auth 0 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c00ba30 con 0x7f9d780a0280 2024-09-15T02:00:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00bdf0 con 0x7f9d780a0280 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 5 monclient: authenticate success, global_id 124336 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 10 monclient: _renew_subs 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d78121990 con 0x7f9d780a0280 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d757fa640 10 monclient: dump: 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:00:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d777fe640 4 set_mon_vals no callback set 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 10 monclient: _renew_subs 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.047+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7811f8a0 con 0x7f9d780a0280 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d6c014640 con 0x7f9d780a0280 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d757fa640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 0x7f9d6403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9d6c04c9c0 con 0x7f9d780a0280 2024-09-15T02:00:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7f2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7f2af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7f2af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d780724a0 con 0x7f9d780a0280 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 0x7f9d6403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.048+0000 7f9d7d04b640 10 monclient: get_auth_request con 0x7f9d64038740 auth_method 0 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.051+0000 7f9d7d04b640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 0x7f9d6403abf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d680072b0 tx=0x7f9d68036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.053+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d6c01d400 con 0x7f9d780a0280 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.053+0000 7f9d757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:00:56.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.053+0000 7f9d757fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d7f2af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d7f2af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d7f2af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d781215d0 con 0x7f9d780a0280 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d757fa640 1 -- 172.21.15.17:0/3386906435 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f9d6c01b300 con 0x7f9d780a0280 2024-09-15T02:00:56.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:00:56.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.283+0000 7f9d757fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:00:56.286 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":104,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 msgr2=0x7f9d6403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 0x7f9d6403abf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d680072b0 tx=0x7f9d68036000 comp rx=0 tx=0).stop 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 10 monclient: shutdown 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 msgr2=0x7f9d78141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.285+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 0x7f9d78141520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006ea0 tx=0x7f9d6c008040 comp rx=0 tx=0).stop 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 shutdown_connections 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d64038740 0x7f9d6403abf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d78131b20 0x7f9d78141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d780a0280 0x7f9d78141520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 --2- 172.21.15.17:0/3386906435 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d7809f1e0 0x7f9d78140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:00:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 >> 172.21.15.17:0/3386906435 conn(0x7f9d78071270 msgr2=0x7f9d7809cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:00:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 shutdown_connections 2024-09-15T02:00:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:56.286+0000 7f9d7f2af640 1 -- 172.21.15.17:0/3386906435 wait complete. 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:00:56.665 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T02:00:56.666 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T02:00:56.666 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T02:00:56.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:00:56.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:00:56.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:00:56.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T02:00:56.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T02:00:56.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-15T02:00:56.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-15T02:00:56.921 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T02:00:56.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=110 2024-09-15T02:00:56.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T02:00:56.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T02:00:59.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:00:59.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.406+0000 7f3f33add640 10 monclient: get_monmap_and_config 2024-09-15T02:00:59.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.406+0000 7f3f33add640 10 monclient: build_initial_monmap 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.406+0000 7f3f33add640 10 monclient: monmap: 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:00:59.407360+0000 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:00:59.407360+0000 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:00:59.409 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:00:59.410 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:00:59.410 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:00:59.410 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:00:59.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 Processor -- start 2024-09-15T02:00:59.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 -- start start 2024-09-15T02:00:59.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient: init 2024-09-15T02:00:59.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient: _reopen_session rank -1 2024-09-15T02:00:59.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-a con 0x7f3f2c09b850 addr v2:172.21.15.17:3300/0 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f2c09bc20 addr v1:172.21.15.17:6789/0 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-c con 0x7f3f2c13b300 addr v2:172.21.15.67:3300/0 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c09d710 con 0x7f3f2c09bc20 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c097c20 con 0x7f3f2c09b850 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): start opening mon connection 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c09c730 con 0x7f3f2c13b300 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): _renew_subs 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.407+0000 7f3f33add640 10 monclient(hunting): authenticate will time out at 1319.403955s 2024-09-15T02:00:59.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48400/0 (socket says 172.21.15.17:48400) 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 learned_addr learned my addr 172.21.15.17:0/1374632600 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31078640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3f2c09bc20 0x7f3f2c09c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42508/0 (socket says 172.21.15.17:42508) 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f2c09b850 auth_method 0 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f31879640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.408+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.410+0000 7f3f31078640 1 -- 172.21.15.17:0/1374632600 >> v1:172.21.15.17:6789/0 conn(0x7f3f2c09bc20 legacy=0x7f3f2c09c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:00:59.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.410+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:00:59.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.410+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:59.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.410+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:59.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.410+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:00:59.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 10 monclient(hunting): get_auth_request con 0x7f3f2c13b300 auth_method 0 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:59.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.609+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:00:59.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:00:59.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f2c09b850 auth_method 0 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:00:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:00:59.611+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:00.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:00.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 10 monclient(hunting): get_auth_request con 0x7f3f2c13b300 auth_method 0 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:00.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:00.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.010+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:00.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:00.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f2c09b850 auth_method 0 2024-09-15T02:01:00.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:00.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:00.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:00.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:00.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:00.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:00.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.012+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:00.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:00.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 10 monclient(hunting): get_auth_request con 0x7f3f2c13b300 auth_method 0 2024-09-15T02:01:00.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:00.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:00.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:00.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:00.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:00.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:00.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.812+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f2c09b850 auth_method 0 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:00.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:00.813+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): tick 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 monclient(hunting): continuing hunt 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 msgr2=0x7f3f2c13d6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.67:3300/0 conn(0x7f3f2c13b300 0x7f3f2c13d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 -- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 msgr2=0x7f3f2c098ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 --2- 172.21.15.17:0/1374632600 >> v2:172.21.15.17:3300/0 conn(0x7f3f2c09b850 0x7f3f2c098ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 -- 172.21.15.17:0/1374632600 >> v1:172.21.15.17:6789/0 conn(0x7f3f2c09bc20 legacy=0x7f3f2c09c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f3f100036d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f10006240 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 -- 172.21.15.17:0/1374632600 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c09c870 con 0x7f3f10006240 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 1 -- 172.21.15.17:0/1374632600 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c097c20 con 0x7f3f100036d0 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f237fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f100036d0 auth_method 0 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.408+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.410+0000 7f3f31078640 1 -- 172.21.15.17:0/1374632600 >> v1:172.21.15.67:6789/0 conn(0x7f3f10006240 legacy=0x7f3f100086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f100036d0 auth_method 0 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:02.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:02.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:02.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:02.609+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:03.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f100036d0 auth_method 0 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:03.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.010+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:03.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409464303 0 0) 0x7f3f2c09c870 con 0x7f3f10006240 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): my global_id is 124348 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f180034e0 con 0x7f3f10006240 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3160675409 0 0) 0x7f3f180034e0 con 0x7f3f10006240 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 msgr2=0x7f3f10005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f100036d0 0x7f3f10005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: _finish_hunting 0 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:01:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1000b070 con 0x7f3f10006240 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: _finish_auth 0 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f3f1c001720 con 0x7f3f10006240 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f33add640 5 monclient: authenticate success, global_id 124348 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f33add640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: dump: 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 >> v1:172.21.15.67:6789/0 conn(0x7f3f10006240 legacy=0x7f3f100086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 reap_dead start 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 0x7f3f18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): picked mon.c con 0x7f3f180039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 0x7f3f18004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): picked mon.b con 0x7f3f18004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 0x7f3f1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): picked mon.a con 0x7f3f18005250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c097c20 con 0x7f3f18005250 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800c210 con 0x7f3f18004780 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800c3d0 con 0x7f3f180039d0 2024-09-15T02:01:03.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f30877640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.134+0000 7f3f33add640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 0x7f3f18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 0x7f3f18004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 0x7f3f1800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f180039d0 auth_method 0 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): get_auth_request con 0x7f3f18004780 auth_method 0 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): get_auth_request con 0x7f3f18005250 auth_method 0 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:03.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient(hunting): handle_auth_done global_id 124348 payload 293 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 msgr2=0x7f3f18003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 0x7f3f18003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 msgr2=0x7f3f18004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 0x7f3f18004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient: _finish_hunting 0 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 monclient: found mon.a 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180034e0 con 0x7f3f18005250 2024-09-15T02:01:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 10 monclient: _finish_auth 0 2024-09-15T02:01:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f3207a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31078640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 0x7f3f18004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.135+0000 7f3f31879640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 0x7f3f18003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.138+0000 7f3f3207a640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 msgr2=0x7f3f1800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.138+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 0x7f3f1800b9d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c097770 tx=0x7f3f24007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d9b74085b85387 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2402be20 con 0x7f3f18005250 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: dump: 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2402f070 con 0x7f3f18005250 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f33add640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f33add640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 1 -- 172.21.15.17:0/1374632600 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24035650 con 0x7f3f18005250 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f33add640 4 set_mon_vals no callback set 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.139+0000 7f3f30877640 10 monclient: dump: 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 10 monclient: shutdown 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 -- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 msgr2=0x7f3f1800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 0x7f3f1800b9d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c097770 tx=0x7f3f24007670 comp rx=0 tx=0).stop 2024-09-15T02:01:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 -- 172.21.15.17:0/1374632600 shutdown_connections 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18005250 0x7f3f1800b9d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f18004780 0x7f3f18004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 --2- 172.21.15.17:0/1374632600 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f180039d0 0x7f3f18003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 -- 172.21.15.17:0/1374632600 >> 172.21.15.17:0/1374632600 conn(0x7f3f2c0a1000 msgr2=0x7f3f2c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 -- 172.21.15.17:0/1374632600 shutdown_connections 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.140+0000 7f3f33add640 1 -- 172.21.15.17:0/1374632600 wait complete. 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.141+0000 7f3f33add640 10 monclient: build_initial_monmap 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.141+0000 7f3f33add640 10 monclient: monmap: 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 Processor -- start 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 -- start start 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient: init 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 0x7f3f2c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-b con 0x7f3f180039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-a con 0x7f3f18004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): picked mon.noname-c con 0x7f3f18005250 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c120fb0 con 0x7f3f18004780 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c09d300 con 0x7f3f180039d0 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c06efd0 con 0x7f3f18005250 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.142+0000 7f3f33add640 10 monclient(hunting): authenticate will time out at 1323.138922s 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41984/0 (socket says 172.21.15.17:41984) 2024-09-15T02:01:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39108/0 (socket says 172.21.15.17:39108) 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 -- 172.21.15.17:0/1765979603 learned_addr learned my addr 172.21.15.17:0/1765979603 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): get_auth_request con 0x7f3f18004780 auth_method 0 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 10 monclient(hunting): get_auth_request con 0x7f3f18005250 auth_method 0 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 0x7f3f2c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): get_auth_request con 0x7f3f180039d0 auth_method 0 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 msgr2=0x7f3f2c06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 msgr2=0x7f3f2c06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f3207a640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient(hunting): handle_auth_done global_id 104844 payload 824 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 msgr2=0x7f3f2c06cf40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 msgr2=0x7f3f2c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 0x7f3f2c06c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient: _finish_hunting 0 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 monclient: found mon.noname-a 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31078640 1 -- 172.21.15.17:0/1765979603 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c06e510 con 0x7f3f18004780 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.143+0000 7f3f31879640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 0x7f3f2c06c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.145+0000 7f3f31078640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 msgr2=0x7f3f2c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.145+0000 7f3f31078640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0077c0 tx=0x7f3f1c007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c018030 con 0x7f3f18004780 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: dump: 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: _finish_auth 0 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 5 monclient: authenticate success, global_id 104844 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c00fb70 con 0x7f3f18004780 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 10 monclient: _renew_subs 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c135a30 con 0x7f3f18004780 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c0133f0 con 0x7f3f18004780 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f21ffb640 10 monclient: dump: 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:03.165 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 10 monclient: _renew_subs 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c06ebd0 con 0x7f3f18004780 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.146+0000 7f3f23fff640 4 set_mon_vals no callback set 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f33add640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f1c01b070 con 0x7f3f18004780 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f33add640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f33add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c0724a0 con 0x7f3f18004780 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f21ffb640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 0x7f3f1003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.147+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3f1c019070 con 0x7f3f18004780 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.148+0000 7f3f31879640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 0x7f3f1003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.148+0000 7f3f31879640 10 monclient: get_auth_request con 0x7f3f10039100 auth_method 0 2024-09-15T02:01:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.150+0000 7f3f31879640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 0x7f3f1003b5b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3f28018a50 tx=0x7f3f28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.152+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f1c007da0 con 0x7f3f18004780 2024-09-15T02:01:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.152+0000 7f3f21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.152+0000 7f3f21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:03.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.386+0000 7f3f33add640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.386+0000 7f3f33add640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.386+0000 7f3f33add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.386+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f2c120cd0 con 0x7f3f18004780 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.387+0000 7f3f21ffb640 1 -- 172.21.15.17:0/1765979603 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f3f1c01b790 con 0x7f3f18004780 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.387+0000 7f3f21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.387+0000 7f3f21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:03.389 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":108,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 msgr2=0x7f3f1003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 0x7f3f1003b5b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3f28018a50 tx=0x7f3f28008040 comp rx=0 tx=0).stop 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 10 monclient: shutdown 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 msgr2=0x7f3f2c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0077c0 tx=0x7f3f1c007930 comp rx=0 tx=0).stop 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 shutdown_connections 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f10039100 0x7f3f1003b5b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f18004780 0x7f3f2c06ca00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f18005250 0x7f3f2c06cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 --2- 172.21.15.17:0/1765979603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f180039d0 0x7f3f2c06c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:03.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 >> 172.21.15.17:0/1765979603 conn(0x7f3f2c0a1000 msgr2=0x7f3f2c098e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:03.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 shutdown_connections 2024-09-15T02:01:03.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:03.388+0000 7f3f33add640 1 -- 172.21.15.17:0/1765979603 wait complete. 2024-09-15T02:01:06.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:06.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.506+0000 7f7ce351d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:06.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.506+0000 7f7ce351d640 10 monclient: build_initial_monmap 2024-09-15T02:01:06.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.506+0000 7f7ce351d640 10 monclient: monmap: 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:06.507480+0000 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:06.507480+0000 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:06.509 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 1 Processor -- start 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 1 -- start start 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 10 monclient: init 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-b con 0x7f7cdc097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:06.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.507+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7cdc09a590 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7cdc09a960 0x7f7cdc12edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-c con 0x7f7cdc09a960 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc0985d0 con 0x7f7cdc09a590 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cdc09d200 con 0x7f7cdc09a960 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc097c30 con 0x7f7cdc097320 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce351d640 10 monclient(hunting): authenticate will time out at 1326.504035s 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce0ab8640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f7cdc09a590 0x7f7cdc12e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38626/0 (socket says 172.21.15.17:38626) 2024-09-15T02:01:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce0ab8640 1 -- 172.21.15.17:0/1774095707 learned_addr learned my addr 172.21.15.17:0/1774095707 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 10 monclient(hunting): get_auth_request con 0x7f7cdc097320 auth_method 0 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 1 -- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 msgr2=0x7f7cdc0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 1 -- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 msgr2=0x7f7cdc0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce12b9640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 --2- 172.21.15.17:0/1774095707 >> v2:172.21.15.67:3300/0 conn(0x7f7cdc09a960 0x7f7cdc12edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): get_auth_request con 0x7f7cdc09a960 auth_method 0 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient(hunting): handle_auth_done global_id 124354 payload 274 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 -- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 msgr2=0x7f7cdc0976f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 -- 172.21.15.17:0/1774095707 >> v1:172.21.15.17:6789/0 conn(0x7f7cdc09a590 legacy=0x7f7cdc12e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient: _finish_hunting 0 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 monclient: found mon.noname-c 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:01:06.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.508+0000 7f7ce1aba640 1 -- 172.21.15.17:0/1774095707 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0992f0 con 0x7f7cdc09a960 2024-09-15T02:01:06.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.511+0000 7f7ce1aba640 1 -- 172.21.15.17:0/1774095707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a960 msgr2=0x7f7cdc12edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.511+0000 7f7ce1aba640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a960 0x7f7cdc12edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009e60 tx=0x7f7cd4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b62bb69487138c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 1 -- 172.21.15.17:0/1774095707 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4007760 con 0x7f7cdc09a960 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: dump: 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: _finish_auth 0 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 5 monclient: authenticate success, global_id 124354 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 1 -- 172.21.15.17:0/1774095707 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd402d050 con 0x7f7cdc09a960 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 4 set_mon_vals no callback set 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 1 -- 172.21.15.17:0/1774095707 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4007af0 con 0x7f7cdc09a960 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ccffff640 10 monclient: dump: 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 10 monclient: shutdown 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 -- 172.21.15.17:0/1774095707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a960 msgr2=0x7f7cdc12edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a960 0x7f7cdc12edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009e60 tx=0x7f7cd4009780 comp rx=0 tx=0).stop 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 -- 172.21.15.17:0/1774095707 shutdown_connections 2024-09-15T02:01:06.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a960 0x7f7cdc12edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 --2- 172.21.15.17:0/1774095707 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc097320 0x7f7cdc0976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 -- 172.21.15.17:0/1774095707 >> 172.21.15.17:0/1774095707 conn(0x7f7cdc0a1000 msgr2=0x7f7cdc0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.512+0000 7f7ce351d640 1 -- 172.21.15.17:0/1774095707 shutdown_connections 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 1 -- 172.21.15.17:0/1774095707 wait complete. 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 10 monclient: build_initial_monmap 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 10 monclient: monmap: 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.524 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 1 Processor -- start 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 1 -- start start 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.513+0000 7f7ce351d640 10 monclient: init 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-a con 0x7f7cdc097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a590 0x7f7cdc141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-b con 0x7f7cdc09a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): picked mon.noname-c con 0x7f7cdc09a960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc121030 con 0x7f7cdc097320 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc09ad50 con 0x7f7cdc09a590 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc12e760 con 0x7f7cdc09a960 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce351d640 10 monclient(hunting): authenticate will time out at 1326.510500s 2024-09-15T02:01:06.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42014/0 (socket says 172.21.15.17:42014) 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39132/0 (socket says 172.21.15.17:39132) 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 1 -- 172.21.15.17:0/3729120485 learned_addr learned my addr 172.21.15.17:0/3729120485 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): get_auth_request con 0x7f7cdc097320 auth_method 0 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 10 monclient(hunting): get_auth_request con 0x7f7cdc09a960 auth_method 0 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 msgr2=0x7f7cdc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 msgr2=0x7f7cdc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a590 0x7f7cdc141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce1aba640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): get_auth_request con 0x7f7cdc09a590 auth_method 0 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce0ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.514+0000 7f7ce12b9640 10 monclient(hunting): handle_auth_done global_id 104847 payload 824 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 msgr2=0x7f7cdc1422c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a590 msgr2=0x7f7cdc141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a590 0x7f7cdc141d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 10 monclient: _finish_hunting 0 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 monclient: found mon.noname-a 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.515+0000 7f7ce12b9640 1 -- 172.21.15.17:0/3729120485 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4009a20 con 0x7f7cdc097320 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce12b9640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 msgr2=0x7f7cdc141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce12b9640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800d9f0 tx=0x7f7cd8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8007aa0 con 0x7f7cdc097320 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: dump: 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: _finish_auth 0 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 5 monclient: authenticate success, global_id 104847 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8007c00 con 0x7f7cdc097320 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 10 monclient: _renew_subs 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8010470 con 0x7f7cdc097320 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc11f1e0 con 0x7f7cdc097320 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccd7fa640 10 monclient: dump: 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ccf7fe640 4 set_mon_vals no callback set 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 10 monclient: _renew_subs 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.517+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc11ecd0 con 0x7f7cdc097320 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7cd80107e0 con 0x7f7cdc097320 2024-09-15T02:01:06.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ce351d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:06.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ce351d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:06.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ce351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc120b90 con 0x7f7cdc097320 2024-09-15T02:01:06.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ccd7fa640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 0x7f7cc003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.518+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f7cd8014030 con 0x7f7cdc097320 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.519+0000 7f7ce0ab8640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 0x7f7cc003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.519+0000 7f7ce0ab8640 10 monclient: get_auth_request con 0x7f7cc0038790 auth_method 0 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.521+0000 7f7ce0ab8640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 0x7f7cc003ac40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0999d0 tx=0x7f7cd0007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.523+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7cd8052020 con 0x7f7cdc097320 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.523+0000 7f7ccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:06.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.523+0000 7f7ccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.749+0000 7f7ce351d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.749+0000 7f7ce351d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.749+0000 7f7ce351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.749+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7cdc0724a0 con 0x7f7cdc097320 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.750+0000 7f7ccd7fa640 1 -- 172.21.15.17:0/3729120485 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f7cd8010a90 con 0x7f7cdc097320 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.750+0000 7f7ccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:06.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.750+0000 7f7ccd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:06.753 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:06.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 msgr2=0x7f7cc003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 0x7f7cc003ac40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0999d0 tx=0x7f7cd0007f00 comp rx=0 tx=0).stop 2024-09-15T02:01:06.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 10 monclient: shutdown 2024-09-15T02:01:06.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:06.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 msgr2=0x7f7cdc141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800d9f0 tx=0x7f7cd8008040 comp rx=0 tx=0).stop 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 shutdown_connections 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7cc0038790 0x7f7cc003ac40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7cdc09a960 0x7f7cdc1422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cdc09a590 0x7f7cdc141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 --2- 172.21.15.17:0/3729120485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7cdc097320 0x7f7cdc141840 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 >> 172.21.15.17:0/3729120485 conn(0x7f7cdc0a1000 msgr2=0x7f7cdc12dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 shutdown_connections 2024-09-15T02:01:06.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.752+0000 7f7ce351d640 1 -- 172.21.15.17:0/3729120485 wait complete. 2024-09-15T02:01:06.764 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:01:06.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:01:06.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:01:06.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.859+0000 7f9e2b75d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.859+0000 7f9e2b75d640 10 monclient: build_initial_monmap 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.859+0000 7f9e2b75d640 10 monclient: monmap: 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:06.860723+0000 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:06.860723+0000 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:06.862 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:06.863 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:06.863 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:06.863 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:06.863 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.860+0000 7f9e2b75d640 1 Processor -- start 2024-09-15T02:01:06.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.860+0000 7f9e2b75d640 1 -- start start 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.860+0000 7f9e2b75d640 10 monclient: init 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e241319a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9e2409eac0 0x7f9e2409eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e2409eac0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e24136cd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e240a1530 con 0x7f9e241319a0 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2409fca0 con 0x7f9e24136cd0 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240a00e0 con 0x7f9e2409eac0 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e2b75d640 10 monclient(hunting): authenticate will time out at 1326.857237s 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9e2409eac0 0x7f9e2409eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9e2409eac0 0x7f9e2409eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42022/0 (socket says 172.21.15.17:42022) 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3163951589 learned_addr learned my addr 172.21.15.17:0/3163951589 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): get_auth_request con 0x7f9e2409eac0 auth_method 0 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e294f9640 1 --1- 172.21.15.17:0/3163951589 >> v1:172.21.15.17:6789/0 conn(0x7f9e241319a0 0x7f9e24131d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38636/0 (socket says 172.21.15.17:38636) 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.861+0000 7f9e28cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e294f9640 1 -- 172.21.15.17:0/3163951589 >> v1:172.21.15.17:6789/0 conn(0x7f9e241319a0 legacy=0x7f9e24131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 10 monclient(hunting): handle_auth_done global_id 104850 payload 274 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3163951589 >> v1:172.21.15.67:6789/0 conn(0x7f9e24136cd0 legacy=0x7f9e241390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3163951589 >> v1:172.21.15.67:6789/0 conn(0x7f9e24136cd0 legacy=0x7f9e241390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3163951589 >> v1:172.21.15.17:6789/0 conn(0x7f9e241319a0 legacy=0x7f9e24131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 10 monclient: _finish_hunting 0 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 1 monclient: found mon.noname-a 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.863+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3163951589 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240a1250 con 0x7f9e2409eac0 2024-09-15T02:01:06.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3163951589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e2409eac0 msgr2=0x7f9e2409eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e28cf8640 1 --2- 172.21.15.17:0/3163951589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e2409eac0 0x7f9e2409eec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009a10 tx=0x7f9e140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a5def1db0b5d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3163951589 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1402f4b0 con 0x7f9e2409eac0 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: dump: 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.874 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: _finish_auth 0 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e2b75d640 5 monclient: authenticate success, global_id 104850 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3163951589 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1402f610 con 0x7f9e2409eac0 2024-09-15T02:01:06.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e2b75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e2b75d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e2b75d640 4 set_mon_vals no callback set 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3163951589 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1402f9b0 con 0x7f9e2409eac0 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.866+0000 7f9e1bfff640 10 monclient: dump: 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 10 monclient: shutdown 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3163951589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e2409eac0 msgr2=0x7f9e2409eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3163951589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e2409eac0 0x7f9e2409eec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009a10 tx=0x7f9e140091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3163951589 shutdown_connections 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3163951589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e2409eac0 0x7f9e2409eec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3163951589 >> 172.21.15.17:0/3163951589 conn(0x7f9e240a5000 msgr2=0x7f9e240a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3163951589 shutdown_connections 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3163951589 wait complete. 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 10 monclient: build_initial_monmap 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.867+0000 7f9e2b75d640 10 monclient: monmap: 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:06.878 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 Processor -- start 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 -- start start 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient: init 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e2409eac0 0x7f9e24136220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2409eac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e241319a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 0x7f9e24096f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e24136cd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:06.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24097da0 con 0x7f9e24136cd0 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2409bf50 con 0x7f9e2409eac0 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2409db20 con 0x7f9e241319a0 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e2b75d640 10 monclient(hunting): authenticate will time out at 1326.864871s 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e28cf8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.868+0000 7f9e29cfa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 0x7f9e24096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39138/0 (socket says 172.21.15.17:39138) 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3326487349 learned_addr learned my addr 172.21.15.17:0/3326487349 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): get_auth_request con 0x7f9e24136cd0 auth_method 0 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 10 monclient(hunting): get_auth_request con 0x7f9e241319a0 auth_method 0 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 msgr2=0x7f9e24096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:01:06.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 msgr2=0x7f9e24096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e2409eac0 0x7f9e24136220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e28cf8640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 10 monclient(hunting): get_auth_request con 0x7f9e2409eac0 auth_method 0 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient(hunting): handle_auth_done global_id 104856 payload 824 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 msgr2=0x7f9e24096a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e241319a0 0x7f9e24096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e2409eac0 msgr2=0x7f9e24136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e2409eac0 0x7f9e24136220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient: _finish_hunting 0 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 monclient: found mon.noname-a 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14009450 con 0x7f9e24136cd0 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.869+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e2409eac0 0x7f9e24136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 msgr2=0x7f9e24096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e29cfa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 0x7f9e24096f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c7f0 tx=0x7f9e1c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c005d10 con 0x7f9e24136cd0 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:06.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: dump: 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: _finish_auth 0 2024-09-15T02:01:06.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 5 monclient: authenticate success, global_id 104856 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c005e90 con 0x7f9e24136cd0 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 10 monclient: _renew_subs 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2409b760 con 0x7f9e24136cd0 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00a890 con 0x7f9e24136cd0 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e197fa640 10 monclient: dump: 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.885 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e1b7fe640 4 set_mon_vals no callback set 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 10 monclient: _renew_subs 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:06.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.871+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2409aa70 con 0x7f9e24136cd0 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e2b75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 0x7f9e24132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24063b00 con 0x7f9e24097490 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 0x7f9e24132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient: get_auth_request con 0x7f9e24097490 auth_method 0 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9e1c00ac40 con 0x7f9e24136cd0 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 10 monclient(hunting): handle_auth_done global_id 104859 payload 824 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e294f9640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e2409ad20 con 0x7f9e24097490 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.872+0000 7f9e197fa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e08038740 0x7f9e0803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.873+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9e1c04c320 con 0x7f9e24136cd0 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.873+0000 7f9e28cf8640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e08038740 0x7f9e0803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.873+0000 7f9e28cf8640 10 monclient: get_auth_request con 0x7f9e08038740 auth_method 0 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e294f9640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 msgr2=0x7f9e24132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 0x7f9e24132a20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e200127a0 con 0x7f9e24097490 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 10 monclient: dump: 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.892 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9e20012900 con 0x7f9e24097490 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 msgr2=0x7f9e24132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 0x7f9e24132a20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20007f10 comp rx=0 tx=0).stop 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 msgr2=0x7f9e24132a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e197fa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24097490 0x7f9e24132a20 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20007f10 comp rx=0 tx=0).stop 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.875+0000 7f9e28cf8640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e08038740 0x7f9e0803abf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009420 tx=0x7f9e14032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e2b75d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 0x7f9e24133360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e2b75d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240999e0 con 0x7f9e24132f60 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 0x7f9e24133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient: get_auth_request con 0x7f9e24132f60 auth_method 0 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 10 monclient(hunting): handle_auth_done global_id 104862 payload 824 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.881+0000 7f9e294f9640 1 -- 172.21.15.17:0/3326487349 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e24097860 con 0x7f9e24132f60 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.883+0000 7f9e294f9640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 msgr2=0x7f9e24133360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:06.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.883+0000 7f9e294f9640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 0x7f9e24133360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2001c6f0 con 0x7f9e24132f60 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 10 monclient: dump: 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:06.895 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f9e20024070 con 0x7f9e24132f60 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 msgr2=0x7f9e24133360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 0x7f9e24133360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20015000 comp rx=0 tx=0).stop 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 msgr2=0x7f9e24133360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e197fa640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24132f60 0x7f9e24133360 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005f60 tx=0x7f9e20015000 comp rx=0 tx=0).stop 2024-09-15T02:01:06.896 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:06.897 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:06.898 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.899 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:06.900 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:06.901 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:06.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:06.903 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:06.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:06.908 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:06.909 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:06.910 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e08038740 msgr2=0x7f9e0803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e08038740 0x7f9e0803abf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009420 tx=0x7f9e14032000 comp rx=0 tx=0).stop 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 10 monclient: shutdown 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 msgr2=0x7f9e24096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 --2- 172.21.15.17:0/3326487349 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e24136cd0 0x7f9e24096f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c7f0 tx=0x7f9e1c007f70 comp rx=0 tx=0).stop 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 reap_dead start 2024-09-15T02:01:06.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e29cfa640 1 -- 172.21.15.17:0/3326487349 reap_dead start 2024-09-15T02:01:06.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 shutdown_connections 2024-09-15T02:01:06.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 >> 172.21.15.17:0/3326487349 conn(0x7f9e240a5000 msgr2=0x7f9e240a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:06.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 shutdown_connections 2024-09-15T02:01:06.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:06.884+0000 7f9e2b75d640 1 -- 172.21.15.17:0/3326487349 wait complete. 2024-09-15T02:01:06.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:06.914 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:01:06.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:01:07.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.009+0000 7fa240c6f640 10 monclient: get_monmap_and_config 2024-09-15T02:01:07.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.009+0000 7fa240c6f640 10 monclient: build_initial_monmap 2024-09-15T02:01:07.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.009+0000 7fa240c6f640 10 monclient: monmap: 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:07.010497+0000 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:07.010497+0000 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:07.012 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 1 Processor -- start 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 1 -- start start 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 10 monclient: init 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T02:01:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.010+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa23c097320 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa23c09a590 0x7fa23c1339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa23c09a590 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa23c133f10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23c0985d0 con 0x7fa23c097320 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23c09d200 con 0x7fa23c09a590 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0992f0 con 0x7fa23c133f10 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa240c6f640 10 monclient(hunting): authenticate will time out at 1327.007125s 2024-09-15T02:01:07.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23a59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa23c097320 0x7fa23c0976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38646/0 (socket says 172.21.15.17:38646) 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23a59c640 1 -- 172.21.15.17:0/1737010992 learned_addr learned my addr 172.21.15.17:0/1737010992 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 10 monclient(hunting): get_auth_request con 0x7fa23c133f10 auth_method 0 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1737010992 >> v2:172.21.15.67:3300/0 conn(0x7fa23c09a590 0x7fa23c1339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): get_auth_request con 0x7fa23c09a590 auth_method 0 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 1 -- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 msgr2=0x7fa23c1362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:01:07.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 1 -- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 msgr2=0x7fa23c1362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient(hunting): handle_auth_done global_id 124363 payload 274 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 -- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 msgr2=0x7fa23c1362f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 -- 172.21.15.17:0/1737010992 >> v1:172.21.15.17:6789/0 conn(0x7fa23c097320 legacy=0x7fa23c0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 monclient: found mon.noname-c 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:01:07.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.011+0000 7fa239d9b640 1 -- 172.21.15.17:0/1737010992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c09cb90 con 0x7fa23c09a590 2024-09-15T02:01:07.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa239d9b640 1 -- 172.21.15.17:0/1737010992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c09a590 msgr2=0x7fa23c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:07.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c09a590 0x7fa23c1339d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa224009850 tx=0x7fa2240090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b795f68fce87bf21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 1 -- 172.21.15.17:0/1737010992 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22402adf0 con 0x7fa23c09a590 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: dump: 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:07.025 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: _finish_auth 0 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 1 -- 172.21.15.17:0/1737010992 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224032440 con 0x7fa23c09a590 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 1 -- 172.21.15.17:0/1737010992 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240327e0 con 0x7fa23c09a590 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa240c6f640 5 monclient: authenticate success, global_id 124363 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa240c6f640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa240c6f640 4 set_mon_vals no callback set 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.014+0000 7fa23959a640 10 monclient: dump: 2024-09-15T02:01:07.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 10 monclient: shutdown 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:07.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 -- 172.21.15.17:0/1737010992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c09a590 msgr2=0x7fa23c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c09a590 0x7fa23c1339d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa224009850 tx=0x7fa2240090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 -- 172.21.15.17:0/1737010992 shutdown_connections 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c133f10 0x7fa23c1362f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1737010992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c09a590 0x7fa23c1339d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 -- 172.21.15.17:0/1737010992 >> 172.21.15.17:0/1737010992 conn(0x7fa23c0a1000 msgr2=0x7fa23c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 -- 172.21.15.17:0/1737010992 shutdown_connections 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 1 -- 172.21.15.17:0/1737010992 wait complete. 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 10 monclient: build_initial_monmap 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.015+0000 7fa240c6f640 10 monclient: monmap: 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 Processor -- start 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 -- start start 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient: init 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa23c097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 0x7fa23c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa23c09a590 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c133f10 0x7fa23c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa23c133f10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c09ad50 con 0x7fa23c09a590 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0a0790 con 0x7fa23c133f10 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c1184c0 con 0x7fa23c097320 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa240c6f640 10 monclient(hunting): authenticate will time out at 1327.012748s 2024-09-15T02:01:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 0x7fa23c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 0x7fa23c06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42052/0 (socket says 172.21.15.17:42052) 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 1 -- 172.21.15.17:0/1000784041 learned_addr learned my addr 172.21.15.17:0/1000784041 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): get_auth_request con 0x7fa23c09a590 auth_method 0 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39150/0 (socket says 172.21.15.17:39150) 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 10 monclient(hunting): get_auth_request con 0x7fa23c097320 auth_method 0 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa239d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 msgr2=0x7fa23c12cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 msgr2=0x7fa23c12cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.016+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c133f10 0x7fa23c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa23ad9d640 10 monclient(hunting): get_auth_request con 0x7fa23c133f10 auth_method 0 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa23ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 10 monclient(hunting): handle_auth_done global_id 104865 payload 824 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 msgr2=0x7fa23c12cf50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c097320 0x7fa23c12cf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c133f10 msgr2=0x7fa23c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa23c133f10 0x7fa23c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 monclient: found mon.noname-a 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.017+0000 7fa239d9b640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa224009390 con 0x7fa23c09a590 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa239d9b640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 msgr2=0x7fa23c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa239d9b640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 0x7fa23c06cd50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa224009850 tx=0x7fa224032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:07.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224031b10 con 0x7fa23c09a590 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: dump: 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 5 monclient: authenticate success, global_id 104865 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224031c70 con 0x7fa23c09a590 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 10 monclient: _renew_subs 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23c06ed10 con 0x7fa23c09a590 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22403d420 con 0x7fa23c09a590 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa2327fc640 10 monclient: dump: 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 10 monclient: _renew_subs 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa233fff640 4 set_mon_vals no callback set 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:07.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.019+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c06e800 con 0x7fa23c09a590 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa240c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c063c20 con 0x7fa23c06f3d0 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa224039040 con 0x7fa23c09a590 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 10 monclient: get_auth_request con 0x7fa23c06f3d0 auth_method 0 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:07.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa2327fc640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa21c038740 0x7fa21c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.020+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa22406c490 con 0x7fa23c09a590 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.021+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa21c038740 0x7fa21c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.021+0000 7fa23ad9d640 10 monclient: get_auth_request con 0x7fa21c038740 auth_method 0 2024-09-15T02:01:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.023+0000 7fa23ad9d640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa21c038740 0x7fa21c03abf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa23c096e90 tx=0x7fa23400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 10 monclient: get_auth_request con 0x7fa23c06f3d0 auth_method 0 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:07.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:07.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:07.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:07.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.221+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:07.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:07.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 10 monclient: get_auth_request con 0x7fa23c06f3d0 auth_method 0 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:07.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:07.622+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:08.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient: get_auth_request con 0x7fa23c06f3d0 auth_method 0 2024-09-15T02:01:08.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 10 monclient(hunting): handle_auth_done global_id 134099 payload 824 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.423+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa23c0724a0 con 0x7fa23c06f3d0 2024-09-15T02:01:08.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.425+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.425+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa22c011860 tx=0x7fa22c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c00fb50 con 0x7fa23c06f3d0 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 10 monclient: dump: 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.428 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa22c00fcb0 con 0x7fa23c06f3d0 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa22c011860 tx=0x7fa22c009040 comp rx=0 tx=0).stop 2024-09-15T02:01:08.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 msgr2=0x7fa23c06d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:08.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.426+0000 7fa2327fc640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c06f3d0 0x7fa23c06d7d0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa22c011860 tx=0x7fa22c009040 comp rx=0 tx=0).stop 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa240c6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 0x7fa23c069cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa240c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c098f60 con 0x7fa23c0698d0 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 0x7fa23c069cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient: get_auth_request con 0x7fa23c0698d0 auth_method 0 2024-09-15T02:01:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 10 monclient(hunting): handle_auth_done global_id 134102 payload 824 2024-09-15T02:01:08.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.433+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa23c09cfe0 con 0x7fa23c0698d0 2024-09-15T02:01:08.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa23a59c640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 msgr2=0x7fa23c069cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa23a59c640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 0x7fa23c069cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa22c019000 tx=0x7fa22c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c01c8f0 con 0x7fa23c0698d0 2024-09-15T02:01:08.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 10 monclient: dump: 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fa22c01ca50 con 0x7fa23c0698d0 2024-09-15T02:01:08.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 msgr2=0x7fa23c069cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 0x7fa23c069cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa22c019000 tx=0x7fa22c01a000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 msgr2=0x7fa23c069cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.436+0000 7fa2327fc640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa23c0698d0 0x7fa23c069cd0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa22c019000 tx=0x7fa22c01a000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:08.440 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.441 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.442 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:08.443 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:08.444 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.445 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:08.446 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:08.447 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.448 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:08.450 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.451 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:08.452 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:08.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa21c038740 msgr2=0x7fa21c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa21c038740 0x7fa21c03abf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa23c096e90 tx=0x7fa23400d040 comp rx=0 tx=0).stop 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 10 monclient: shutdown 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 msgr2=0x7fa23c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 --2- 172.21.15.17:0/1000784041 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa23c09a590 0x7fa23c06cd50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa224009850 tx=0x7fa224032000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa23ad9d640 1 -- 172.21.15.17:0/1000784041 reap_dead start 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa23ad9d640 1 -- 172.21.15.17:0/1000784041 reap_dead start 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 shutdown_connections 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 >> 172.21.15.17:0/1000784041 conn(0x7fa23c0a1000 msgr2=0x7fa23c132160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 shutdown_connections 2024-09-15T02:01:08.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.437+0000 7fa240c6f640 1 -- 172.21.15.17:0/1000784041 wait complete. 2024-09-15T02:01:08.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:08.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:01:08.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:01:08.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 10 monclient: get_monmap_and_config 2024-09-15T02:01:08.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:08.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 10 monclient: monmap: 2024-09-15T02:01:08.562 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:08.563 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.563 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:08.560956+0000 2024-09-15T02:01:08.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:08.560956+0000 2024-09-15T02:01:08.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:08.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:08.564 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 1 Processor -- start 2024-09-15T02:01:08.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 1 -- start start 2024-09-15T02:01:08.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.560+0000 7fc0760ef640 10 monclient: init 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0700a58b0 0x7fc070132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc0700a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc0700980a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc0700989e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0700a01e0 con 0x7fc0700980a0 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0700a1580 con 0x7fc0700a58b0 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700a5c80 con 0x7fc0700989e0 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc0760ef640 10 monclient(hunting): authenticate will time out at 1328.557488s 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39180/0 (socket says 172.21.15.17:39180) 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42062/0 (socket says 172.21.15.17:42062) 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 -- 172.21.15.17:0/596292812 learned_addr learned my addr 172.21.15.17:0/596292812 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): get_auth_request con 0x7fc0700989e0 auth_method 0 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): get_auth_request con 0x7fc0700980a0 auth_method 0 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.67:3300/0 conn(0x7fc0700a58b0 0x7fc070132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0700a58b0 auth_method 0 2024-09-15T02:01:08.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient(hunting): handle_auth_done global_id 134105 payload 274 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 -- 172.21.15.17:0/596292812 >> v2:172.21.15.67:3300/0 conn(0x7fc0700a58b0 msgr2=0x7fc070132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.67:3300/0 conn(0x7fc0700a58b0 0x7fc070132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 -- 172.21.15.17:0/596292812 >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 msgr2=0x7fc0700984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:08.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 monclient: found mon.noname-b 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06ffff640 1 -- 172.21.15.17:0/596292812 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0701326c0 con 0x7fc0700989e0 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.561+0000 7fc06effd640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.564+0000 7fc06ffff640 1 -- 172.21.15.17:0/596292812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 msgr2=0x7fc070068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.564+0000 7fc06ffff640 1 --2- 172.21.15.17:0/596292812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0701328b0 tx=0x7fc068009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ad6882e89f111c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.564+0000 7fc06e7fc640 1 -- 172.21.15.17:0/596292812 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068007610 con 0x7fc0700989e0 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: dump: 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.586 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 5 monclient: authenticate success, global_id 134105 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 1 -- 172.21.15.17:0/596292812 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06802d050 con 0x7fc0700989e0 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 4 set_mon_vals no callback set 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 1 -- 172.21.15.17:0/596292812 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0680079e0 con 0x7fc0700989e0 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc06e7fc640 10 monclient: dump: 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.588 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 10 monclient: shutdown 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 -- 172.21.15.17:0/596292812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 msgr2=0x7fc070068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 --2- 172.21.15.17:0/596292812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0701328b0 tx=0x7fc068009840 comp rx=0 tx=0).stop 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 -- 172.21.15.17:0/596292812 shutdown_connections 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 --2- 172.21.15.17:0/596292812 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700989e0 0x7fc070068ef0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.17:3300/0 conn(0x7fc0700980a0 0x7fc0700984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 --2- 172.21.15.17:0/596292812 >> v2:172.21.15.67:3300/0 conn(0x7fc0700a58b0 0x7fc070132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 -- 172.21.15.17:0/596292812 >> 172.21.15.17:0/596292812 conn(0x7fc070070ed0 msgr2=0x7fc0700a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 -- 172.21.15.17:0/596292812 shutdown_connections 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.565+0000 7fc0760ef640 1 -- 172.21.15.17:0/596292812 wait complete. 2024-09-15T02:01:08.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.566+0000 7fc0760ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.566+0000 7fc0760ef640 10 monclient: monmap: 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.566+0000 7fc0760ef640 1 Processor -- start 2024-09-15T02:01:08.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.566+0000 7fc0760ef640 1 -- start start 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.566+0000 7fc0760ef640 10 monclient: init 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 0x7fc070130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc0700980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 0x7fc07011b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc0700a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07009b5a0 0x7fc07011bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc07009b5a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0701308b0 con 0x7fc0700a58b0 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc070098dc0 con 0x7fc07009b5a0 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07011cbb0 con 0x7fc0700980a0 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc0760ef640 10 monclient(hunting): authenticate will time out at 1328.563340s 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 0x7fc070130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 0x7fc07011b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 0x7fc070130370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39190/0 (socket says 172.21.15.17:39190) 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 1 -- 172.21.15.17:0/1622140140 learned_addr learned my addr 172.21.15.17:0/1622140140 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): get_auth_request con 0x7fc0700a58b0 auth_method 0 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0700980a0 auth_method 0 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07009b5a0 0x7fc07011bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): get_auth_request con 0x7fc07009b5a0 auth_method 0 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient(hunting): handle_auth_done global_id 104871 payload 824 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 msgr2=0x7fc070130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 0x7fc070130370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07009b5a0 msgr2=0x7fc07011bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07009b5a0 0x7fc07011bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 monclient: found mon.noname-a 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06effd640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064009390 con 0x7fc0700a58b0 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.567+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc0700980a0 0x7fc070130370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc06effd640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 msgr2=0x7fc07011b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc06effd640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 0x7fc07011b7d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc06400e970 tx=0x7fc0640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640095d0 con 0x7fc0700a58b0 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: dump: 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.596 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06400cb50 con 0x7fc0700a58b0 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 5 monclient: authenticate success, global_id 104871 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 10 monclient: _renew_subs 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc068009ae0 con 0x7fc0700a58b0 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064013470 con 0x7fc0700a58b0 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0577fe640 10 monclient: dump: 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 10 monclient: _renew_subs 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0701316d0 con 0x7fc0700a58b0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.570+0000 7fc06dffb640 4 set_mon_vals no callback set 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0760ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0760ef640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 0x7fc07002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc070063b20 con 0x7fc0700989e0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc064013860 con 0x7fc0700a58b0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 0x7fc07002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient: get_auth_request con 0x7fc0700989e0 auth_method 0 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0577fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc05c038740 0x7fc05c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc06404c910 con 0x7fc0700a58b0 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_done global_id 124375 payload 824 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06ffff640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc05c038740 0x7fc05c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06f7fe640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc07011e230 con 0x7fc0700989e0 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.571+0000 7fc06ffff640 10 monclient: get_auth_request con 0x7fc05c038740 auth_method 0 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc06f7fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 msgr2=0x7fc07002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 0x7fc07002aa40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc060009f60 tx=0x7fc0600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060014860 con 0x7fc0700989e0 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 10 monclient: dump: 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc06ffff640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc05c038740 0x7fc05c03abf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc068033000 tx=0x7fc068036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc0600149c0 con 0x7fc0700989e0 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 msgr2=0x7fc07002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 0x7fc07002aa40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc060009f60 tx=0x7fc0600090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 msgr2=0x7fc07002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:08.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.574+0000 7fc0577fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0700989e0 0x7fc07002aa40 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc060009f60 tx=0x7fc0600090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc0760ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc0760ef640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 0x7fc07005e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc0760ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07011e6f0 con 0x7fc07011c250 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 0x7fc07005e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient: get_auth_request con 0x7fc07011c250 auth_method 0 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 10 monclient(hunting): handle_auth_done global_id 124378 payload 824 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.580+0000 7fc06f7fe640 1 -- 172.21.15.17:0/1622140140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc0700724a0 con 0x7fc07011c250 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc06f7fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 msgr2=0x7fc07005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc06f7fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 0x7fc07005e310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc06001b000 tx=0x7fc06001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060024040 con 0x7fc07011c250 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 10 monclient: dump: 2024-09-15T02:01:08.605 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fc0600133a0 con 0x7fc07011c250 2024-09-15T02:01:08.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 msgr2=0x7fc07005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 0x7fc07005e310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc06001b000 tx=0x7fc06001a000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 msgr2=0x7fc07005e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.583+0000 7fc0577fe640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07011c250 0x7fc07005e310 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc06001b000 tx=0x7fc06001a000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:08.607 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.608 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.609 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:08.610 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:08.611 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.612 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.614 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.615 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:08.616 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.617 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.618 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:08.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.620 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.620 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.620 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:08.621 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:08.622 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:08.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:08.622 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:08.622 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:08.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc05c038740 msgr2=0x7fc05c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc05c038740 0x7fc05c03abf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc068033000 tx=0x7fc068036000 comp rx=0 tx=0).stop 2024-09-15T02:01:08.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 10 monclient: shutdown 2024-09-15T02:01:08.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 msgr2=0x7fc07011b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc06ffff640 1 -- 172.21.15.17:0/1622140140 reap_dead start 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 0x7fc07011b7d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc06400e970 tx=0x7fc0640090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 shutdown_connections 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 --2- 172.21.15.17:0/1622140140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc0700a58b0 0x7fc07011b7d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 >> 172.21.15.17:0/1622140140 conn(0x7fc070070ed0 msgr2=0x7fc07009ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 shutdown_connections 2024-09-15T02:01:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.584+0000 7fc0760ef640 1 -- 172.21.15.17:0/1622140140 wait complete. 2024-09-15T02:01:08.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:08.625 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:01:08.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:01:08.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.719+0000 7f302a300640 10 monclient: get_monmap_and_config 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.719+0000 7f302a300640 10 monclient: build_initial_monmap 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.719+0000 7f302a300640 10 monclient: monmap: 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:08.720771+0000 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:08.720771+0000 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:08.723 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:08.723 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:08.723 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:08.723 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.720+0000 7f302a300640 1 Processor -- start 2024-09-15T02:01:08.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.720+0000 7f302a300640 1 -- start start 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.720+0000 7f302a300640 10 monclient: init 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f302406eb30 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f302409e970 0x7f302409bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): picked mon.noname-a con 0x7f302409e970 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f302409c4d0 0x7f3024139b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): picked mon.noname-c con 0x7f302409c4d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302409f6a0 con 0x7f302406eb30 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3024069f40 con 0x7f302409e970 2024-09-15T02:01:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302409ed40 con 0x7f302409c4d0 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302a300640 10 monclient(hunting): authenticate will time out at 1328.717226s 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f302409e970 0x7f302409bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f302409e970 0x7f302409bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42070/0 (socket says 172.21.15.17:42070) 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 -- 172.21.15.17:0/934838633 learned_addr learned my addr 172.21.15.17:0/934838633 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): get_auth_request con 0x7f302409e970 auth_method 0 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302889d640 1 --2- 172.21.15.17:0/934838633 >> v2:172.21.15.67:3300/0 conn(0x7f302409c4d0 0x7f3024139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302889d640 10 monclient(hunting): get_auth_request con 0x7f302409c4d0 auth_method 0 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302889d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302889d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f302889d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient(hunting): handle_auth_done global_id 104874 payload 274 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 -- 172.21.15.17:0/934838633 >> v2:172.21.15.67:3300/0 conn(0x7f302409c4d0 msgr2=0x7f3024139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 --2- 172.21.15.17:0/934838633 >> v2:172.21.15.67:3300/0 conn(0x7f302409c4d0 0x7f3024139b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 -- 172.21.15.17:0/934838633 >> v1:172.21.15.67:6789/0 conn(0x7f302406eb30 legacy=0x7f302406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:08.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:08.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:08.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:08.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:08.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.721+0000 7f30237fe640 1 -- 172.21.15.17:0/934838633 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3024069950 con 0x7f302409e970 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f30237fe640 1 -- 172.21.15.17:0/934838633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409e970 msgr2=0x7f302409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f30237fe640 1 --2- 172.21.15.17:0/934838633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409e970 0x7f302409bf90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3024067140 tx=0x7f30180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa45df09995869bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 1 -- 172.21.15.17:0/934838633 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018032440 con 0x7f302409e970 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: dump: 2024-09-15T02:01:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: _finish_auth 0 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f302a300640 5 monclient: authenticate success, global_id 104874 2024-09-15T02:01:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f302a300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 1 -- 172.21.15.17:0/934838633 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180325a0 con 0x7f302409e970 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f302a300640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f302a300640 4 set_mon_vals no callback set 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 1 -- 172.21.15.17:0/934838633 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018032940 con 0x7f302409e970 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.724+0000 7f3022ffd640 10 monclient: dump: 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 10 monclient: shutdown 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 -- 172.21.15.17:0/934838633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409e970 msgr2=0x7f302409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 --2- 172.21.15.17:0/934838633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409e970 0x7f302409bf90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3024067140 tx=0x7f30180091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 -- 172.21.15.17:0/934838633 shutdown_connections 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 --2- 172.21.15.17:0/934838633 >> v2:172.21.15.67:3300/0 conn(0x7f302409c4d0 0x7f3024139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 --2- 172.21.15.17:0/934838633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409e970 0x7f302409bf90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 -- 172.21.15.17:0/934838633 >> 172.21.15.17:0/934838633 conn(0x7f30240712f0 msgr2=0x7f30240a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 -- 172.21.15.17:0/934838633 shutdown_connections 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 -- 172.21.15.17:0/934838633 wait complete. 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 10 monclient: build_initial_monmap 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 10 monclient: monmap: 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.725+0000 7f302a300640 1 Processor -- start 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 -- start start 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient: init 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f302406eb30 0x7f3024141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): picked mon.noname-b con 0x7f302406eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:08.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): picked mon.noname-a con 0x7f302409c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 0x7f3024132ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): picked mon.noname-c con 0x7f302409e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302409cd10 con 0x7f302409c4d0 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241216e0 con 0x7f302406eb30 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30241201c0 con 0x7f302409e970 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302a300640 10 monclient(hunting): authenticate will time out at 1328.722644s 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 0x7f3024132ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42074/0 (socket says 172.21.15.17:42074) 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 1 -- 172.21.15.17:0/1573513027 learned_addr learned my addr 172.21.15.17:0/1573513027 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): get_auth_request con 0x7f302409e970 auth_method 0 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): get_auth_request con 0x7f302409c4d0 auth_method 0 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f302406eb30 0x7f3024141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): get_auth_request con 0x7f302406eb30 auth_method 0 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f302889d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f30237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.726+0000 7f3023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 10 monclient(hunting): handle_auth_done global_id 134111 payload 824 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 -- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f302406eb30 msgr2=0x7f3024141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f302406eb30 0x7f3024141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 -- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 msgr2=0x7f30241417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 10 monclient: _finish_hunting 0 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 monclient: found mon.noname-c 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f302889d640 1 -- 172.21.15.17:0/1573513027 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3018009450 con 0x7f302409e970 2024-09-15T02:01:08.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.727+0000 7f30237fe640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302889d640 1 -- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 msgr2=0x7f3024132ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302889d640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 0x7f3024132ff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f301c007ea0 tx=0x7f301c0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c014420 con 0x7f302409e970 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: dump: 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.744 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: _finish_auth 0 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c014580 con 0x7f302409e970 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c014940 con 0x7f302409e970 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f300bfff640 10 monclient: dump: 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:08.745 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 5 monclient: authenticate success, global_id 134111 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 10 monclient: _renew_subs 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3024121120 con 0x7f302409e970 2024-09-15T02:01:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f30227fc640 4 set_mon_vals no callback set 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 10 monclient: _renew_subs 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.729+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3024133530 con 0x7f302409e970 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f301c003020 con 0x7f302409e970 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f302a300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f302a300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f302a300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30240724a0 con 0x7f302409e970 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f300bfff640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 0x7f301003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.730+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f301c04ca90 con 0x7f302409e970 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.731+0000 7f3023fff640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 0x7f301003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.731+0000 7f3023fff640 10 monclient: get_auth_request con 0x7f3010038740 auth_method 0 2024-09-15T02:01:08.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.733+0000 7f3023fff640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 0x7f301003abf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3014009a60 tx=0x7f3014006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:08.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.735+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f301c0124f0 con 0x7f302409e970 2024-09-15T02:01:08.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.735+0000 7f300bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:08.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.735+0000 7f300bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:08.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.959+0000 7f302a300640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:01:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.959+0000 7f302a300640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:01:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.959+0000 7f302a300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:08.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.959+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f302406f050 con 0x7f302409e970 2024-09-15T02:01:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f300bfff640 1 -- 172.21.15.17:0/1573513027 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f301c01bb00 con 0x7f302409e970 2024-09-15T02:01:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f300bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:01:08.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f300bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=8,logm=56} crc {auth=688822499,config=3140514009,health=1744959055,kv=1529054990,logm=1518915430}) 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=110003977,config=3140514009,health=2482433086,kv=1529054990,logm=3501329860}) 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=8,logm=56} crc {auth=688822499,config=3140514009,health=1744959055,kv=1529054990,logm=1518915430}) 2024-09-15T02:01:08.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=8,logm=54} crc {auth=2775333670,config=3140514009,health=1744959055,kv=1529054990,logm=3809376356}) 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417269932}) 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741846030}) 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417269932}) 2024-09-15T02:01:08.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765606938}) 2024-09-15T02:01:08.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 msgr2=0x7f301003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 0x7f301003abf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3014009a60 tx=0x7f3014006f50 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 10 monclient: shutdown 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 msgr2=0x7f3024132ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 0x7f3024132ff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f301c007ea0 tx=0x7f301c0097b0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 shutdown_connections 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3010038740 0x7f301003abf0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f302409c4d0 0x7f30241417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f302409e970 0x7f3024132ff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 --2- 172.21.15.17:0/1573513027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f302406eb30 0x7f3024141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 >> 172.21.15.17:0/1573513027 conn(0x7f30240712f0 msgr2=0x7f3024134440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:08.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 shutdown_connections 2024-09-15T02:01:08.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:08.963+0000 7f302a300640 1 -- 172.21.15.17:0/1573513027 wait complete. 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650586829}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133833373}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650586829}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318739348}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350698294}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371972205}) 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350698294}) 2024-09-15T02:01:08.970 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609700995}) 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836171285}) 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2462297259}) 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836171285}) 2024-09-15T02:01:08.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586163782}) 2024-09-15T02:01:08.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:01:08.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413347765}) 2024-09-15T02:01:08.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053413275}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413347765}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464195247}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745827016}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170920599}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745827016}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486174332}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765565976}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3568844579}) 2024-09-15T02:01:08.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765565976}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=990059840}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911936775}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318099359}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911936775}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215308135}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875485897}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598467953}) 2024-09-15T02:01:08.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875485897}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1741087459}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180876666}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363602711}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180876666}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160447073}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186989371}) 2024-09-15T02:01:08.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092629234}) 2024-09-15T02:01:08.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186989371}) 2024-09-15T02:01:08.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211046542}) 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637897139}) 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305831625}) 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637897139}) 2024-09-15T02:01:08.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2280002173}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402871069}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3905114022}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402871069}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=65030935}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470005135}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1354160385}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470005135}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3203259632}) 2024-09-15T02:01:08.989 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159540987}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991373097}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159540987}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966409393}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233220062}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1562250813}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233220062}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648992819}) 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.990 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310718896}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054352775}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310718896}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044644398}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496781805}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3542850985}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496781805}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445515732}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163977219}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=416655290}) 2024-09-15T02:01:08.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163977219}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374997891}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534494071}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143761788}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534494071}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339923587}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948436232}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935764335}) 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948436232}) 2024-09-15T02:01:08.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546437847}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667571324}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859313023}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667571324}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1593614832}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221548287}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979143377}) 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.995 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221548287}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376282704}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105338200}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851857984}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105338200}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=302893838}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587725773}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598176434}) 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587725773}) 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787433610}) 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302741529}) 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449353984}) 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302741529}) 2024-09-15T02:01:08.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:08.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1362390439}) 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255192746}) 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344974110}) 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255192746}) 2024-09-15T02:01:09.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.179+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716913390}) 2024-09-15T02:01:09.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262438715}) 2024-09-15T02:01:09.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3069308031}) 2024-09-15T02:01:09.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262438715}) 2024-09-15T02:01:09.188 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.185+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430049534}) 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986207548}) 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312299649}) 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986207548}) 2024-09-15T02:01:09.208 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.206+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1671402147}) 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63622855}) 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012487725}) 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63622855}) 2024-09-15T02:01:09.209 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.207+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2920961331}) 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455951937}) 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72121851}) 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455951937}) 2024-09-15T02:01:09.211 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266153005}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2776564444}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10566189}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2776564444}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.208+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041066610}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806027184}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680918185}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806027184}) 2024-09-15T02:01:09.212 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.209+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310888033}) 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982695511}) 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374307029}) 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982695511}) 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501498957}) 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.216 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955834187}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534015913}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955834187}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.210+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151067275}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339396795}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012430137}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339396795}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.211+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029372199}) 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.217 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952899110}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446766487}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952899110}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776467933}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.212+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102013762}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558431596}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102013762}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1490365444}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472329364}) 2024-09-15T02:01:09.218 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283635011}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472329364}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.213+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2071818233}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582996574}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120308864}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582996574}) 2024-09-15T02:01:09.219 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.214+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259954295}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076524724}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189774220}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076524724}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616640139}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783037295}) 2024-09-15T02:01:09.225 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003669167}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783037295}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.215+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592011191}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053023095}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3207317883}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053023095}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.216+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395383971}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059107693}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2245190952}) 2024-09-15T02:01:09.226 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059107693}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335879414}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464342310}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638779768}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464342310}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.217+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960882086}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941404242}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902127124}) 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.227 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941404242}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.218+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=132546137}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98478883}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176533056}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98478883}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934572998}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.219+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767641405}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555882210}) 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.228 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767641405}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433116803}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475318890}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655108164}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475318890}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.220+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123363801}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425335533}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925450945}) 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.229 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425335533}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.221+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625786429}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386874309}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133092867}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386874309}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1437587261}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=1720625070,mdsmap=1531151339,mgr=3226792663,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2356627162,mon_config_key=1583035879}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=2997468448,mdsmap=1531151339,mgr=1085909001,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2356627162,mon_config_key=1386283410}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=1720625070,mdsmap=1531151339,mgr=3226792663,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2356627162,mon_config_key=1583035879}) 2024-09-15T02:01:09.230 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.222+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3} crc {logm=3836160096,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2356627162,mon_config_key=2516852880}) 2024-09-15T02:01:09.231 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.228+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.231 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.228+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=2,osdmap=58} crc {monmap=3178194974,osd_metadata=3375641772,osd_snap=1063953654,osdmap=3510293399}) 2024-09-15T02:01:09.231 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.228+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=2,osdmap=59} crc {monmap=3178194974,osd_metadata=3375641772,osd_snap=1063953654,osdmap=1829077726}) 2024-09-15T02:01:09.231 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.229+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:09.231 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.229+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=2,osdmap=58} crc {monmap=3178194974,osd_metadata=3375641772,osd_snap=1063953654,osdmap=3510293399}) 2024-09-15T02:01:09.232 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:09.229+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=2869719389}) 2024-09-15T02:01:09.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:01:09.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:10.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.084+0000 7f966e0dd640 10 monclient: get_monmap_and_config 2024-09-15T02:01:10.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.084+0000 7f966e0dd640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.084+0000 7f966e0dd640 10 monclient: monmap: 2024-09-15T02:01:10.086 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:10.085412+0000 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:10.085412+0000 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:10.087 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 1 Processor -- start 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 1 -- start start 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient: init 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96680a5780 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f966809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.085+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9668069c90 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96680a5b50 con 0x7f96680a5780 2024-09-15T02:01:10.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966809c5f0 con 0x7f9668069c90 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96680a18b0 con 0x7f966809ccf0 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f966e0dd640 10 monclient(hunting): authenticate will time out at 1330.082036s 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f96677fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f96680a5780 0x7f9668069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38656/0 (socket says 172.21.15.17:38656) 2024-09-15T02:01:10.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f96677fe640 1 -- 172.21.15.17:0/4267421733 learned_addr learned my addr 172.21.15.17:0/4267421733 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): get_auth_request con 0x7f966809ccf0 auth_method 0 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient(hunting): handle_auth_done global_id 134114 payload 274 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 1 -- 172.21.15.17:0/4267421733 >> v1:172.21.15.67:6789/0 conn(0x7f9668069c90 legacy=0x7f9668139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 1 -- 172.21.15.17:0/4267421733 >> v1:172.21.15.17:6789/0 conn(0x7f96680a5780 legacy=0x7f9668069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 1 monclient: found mon.noname-b 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.086+0000 7f9666ffd640 1 -- 172.21.15.17:0/4267421733 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966809d210 con 0x7f966809ccf0 2024-09-15T02:01:10.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f9666ffd640 1 -- 172.21.15.17:0/4267421733 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 msgr2=0x7f966809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f9666ffd640 1 --2- 172.21.15.17:0/4267421733 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966809bc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f965c009880 tx=0x7f965c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba4f4662ebf1cdea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 1 -- 172.21.15.17:0/4267421733 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c032440 con 0x7f966809ccf0 2024-09-15T02:01:10.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: dump: 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 1 -- 172.21.15.17:0/4267421733 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c0325a0 con 0x7f966809ccf0 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f966e0dd640 5 monclient: authenticate success, global_id 134114 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f966e0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f966e0dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 1 -- 172.21.15.17:0/4267421733 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c032940 con 0x7f966809ccf0 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f966e0dd640 4 set_mon_vals no callback set 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.089+0000 7f96667fc640 10 monclient: dump: 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 10 monclient: shutdown 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 -- 172.21.15.17:0/4267421733 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 msgr2=0x7f966809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 --2- 172.21.15.17:0/4267421733 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966809bc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f965c009880 tx=0x7f965c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 -- 172.21.15.17:0/4267421733 shutdown_connections 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 --2- 172.21.15.17:0/4267421733 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966809bc90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 -- 172.21.15.17:0/4267421733 >> 172.21.15.17:0/4267421733 conn(0x7f9668070e10 msgr2=0x7f96680a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 -- 172.21.15.17:0/4267421733 shutdown_connections 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 1 -- 172.21.15.17:0/4267421733 wait complete. 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.090+0000 7f966e0dd640 10 monclient: monmap: 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 Processor -- start 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 -- start start 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient: init 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9668069c90 0x7f966813df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9668069c90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-c con 0x7f966809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f96680a5780 0x7f96681414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): picked mon.noname-a con 0x7f96680a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9668122f70 con 0x7f96680a5780 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96681229b0 con 0x7f9668069c90 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9668120440 con 0x7f966809ccf0 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f966e0dd640 10 monclient(hunting): authenticate will time out at 1330.087800s 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9667fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f96680a5780 0x7f96681414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39212/0 (socket says 172.21.15.17:39212) 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 1 -- 172.21.15.17:0/3040551028 learned_addr learned my addr 172.21.15.17:0/3040551028 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 10 monclient(hunting): get_auth_request con 0x7f966809ccf0 auth_method 0 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9667fff640 10 monclient(hunting): get_auth_request con 0x7f96680a5780 auth_method 0 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9667fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.091+0000 7f9667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9667fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9667fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9667fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f96677fe640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9668069c90 0x7f966813df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f96677fe640 10 monclient(hunting): get_auth_request con 0x7f9668069c90 auth_method 0 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f96677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f96677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f96677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient(hunting): handle_auth_done global_id 134117 payload 824 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 -- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9668069c90 msgr2=0x7f966813df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9668069c90 0x7f966813df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 -- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f96680a5780 msgr2=0x7f96681414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f96680a5780 0x7f96681414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 monclient: found mon.noname-c 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.092+0000 7f9666ffd640 1 -- 172.21.15.17:0/3040551028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c009390 con 0x7f966809ccf0 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f9666ffd640 1 -- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 msgr2=0x7f966813d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f9666ffd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f965c009880 tx=0x7f965c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c031990 con 0x7f966809ccf0 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: dump: 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.110 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c031b30 con 0x7f966809ccf0 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 5 monclient: authenticate success, global_id 134117 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 10 monclient: _renew_subs 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96681226d0 con 0x7f966809ccf0 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c03b400 con 0x7f966809ccf0 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f964f7fe640 10 monclient: dump: 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 10 monclient: _renew_subs 2024-09-15T02:01:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9668122d40 con 0x7f966809ccf0 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.094+0000 7f9665ffb640 4 set_mon_vals no callback set 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f966e0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f966e0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f966e0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f965c039040 con 0x7f966809ccf0 2024-09-15T02:01:10.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96680724a0 con 0x7f966809ccf0 2024-09-15T02:01:10.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.095+0000 7f964f7fe640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 0x7f965403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.096+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f965c06c970 con 0x7f966809ccf0 2024-09-15T02:01:10.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.096+0000 7f96677fe640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 0x7f965403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.096+0000 7f96677fe640 10 monclient: get_auth_request con 0x7f9654038740 auth_method 0 2024-09-15T02:01:10.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.098+0000 7f96677fe640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 0x7f965403abf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f965800c890 tx=0x7f9658009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.100+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f965c04c350 con 0x7f966809ccf0 2024-09-15T02:01:10.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.100+0000 7f964f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.100+0000 7f964f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.332+0000 7f966e0dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.332+0000 7f966e0dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.332+0000 7f966e0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.332+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f96680724a0 con 0x7f966809ccf0 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.333+0000 7f964f7fe640 1 -- 172.21.15.17:0/3040551028 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f965c03f080 con 0x7f966809ccf0 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.333+0000 7f964f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.333+0000 7f964f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:10.335 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 msgr2=0x7f965403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 0x7f965403abf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f965800c890 tx=0x7f9658009040 comp rx=0 tx=0).stop 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 10 monclient: shutdown 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 msgr2=0x7f966813d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f965c009880 tx=0x7f965c032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 shutdown_connections 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9654038740 0x7f965403abf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f96680a5780 0x7f96681414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f966809ccf0 0x7f966813d630 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 --2- 172.21.15.17:0/3040551028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9668069c90 0x7f966813df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.334+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 >> 172.21.15.17:0/3040551028 conn(0x7f9668070e10 msgr2=0x7f9668098de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.335+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 shutdown_connections 2024-09-15T02:01:10.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.335+0000 7f966e0dd640 1 -- 172.21.15.17:0/3040551028 wait complete. 2024-09-15T02:01:10.347 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:01:10.347 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:01:10.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:01:10.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:01:10.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 10 monclient: get_monmap_and_config 2024-09-15T02:01:10.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 10 monclient: monmap: 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:10.443874+0000 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:10.443874+0000 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:10.446 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 1 Processor -- start 2024-09-15T02:01:10.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 1 -- start start 2024-09-15T02:01:10.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.443+0000 7fb19c4f8640 10 monclient: init 2024-09-15T02:01:10.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T02:01:10.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb194131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:10.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 0x7fb194131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-b con 0x7fb1940a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb19409f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1940a0040 con 0x7fb19409f1e0 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19409bff0 con 0x7fb194131b20 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940a14a0 con 0x7fb1940a0280 2024-09-15T02:01:10.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19c4f8640 10 monclient(hunting): authenticate will time out at 1330.440329s 2024-09-15T02:01:10.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 0x7fb194131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42096/0 (socket says 172.21.15.17:42096) 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 1 -- 172.21.15.17:0/1891240266 learned_addr learned my addr 172.21.15.17:0/1891240266 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): get_auth_request con 0x7fb194131b20 auth_method 0 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): get_auth_request con 0x7fb1940a0280 auth_method 0 2024-09-15T02:01:10.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient(hunting): handle_auth_done global_id 134120 payload 274 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 -- 172.21.15.17:0/1891240266 >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 msgr2=0x7fb19409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 --2- 172.21.15.17:0/1891240266 >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 -- 172.21.15.17:0/1891240266 >> v1:172.21.15.67:6789/0 conn(0x7fb19409f1e0 legacy=0x7fb194068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 monclient: found mon.noname-b 2024-09-15T02:01:10.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb199a93640 1 -- 172.21.15.17:0/1891240266 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb194132430 con 0x7fb1940a0280 2024-09-15T02:01:10.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.444+0000 7fb19a294640 1 --2- 172.21.15.17:0/1891240266 >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:10.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199a93640 1 -- 172.21.15.17:0/1891240266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 msgr2=0x7fb194131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199a93640 1 --2- 172.21.15.17:0/1891240266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 0x7fb194131ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb184009850 tx=0x7fb1840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca47b7e0abd6685f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 1 -- 172.21.15.17:0/1891240266 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18402adf0 con 0x7fb1940a0280 2024-09-15T02:01:10.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: dump: 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.476 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.476 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.476 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.476 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.476 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 1 -- 172.21.15.17:0/1891240266 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb184032440 con 0x7fb1940a0280 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb19c4f8640 5 monclient: authenticate success, global_id 134120 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb19c4f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 1 -- 172.21.15.17:0/1891240266 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1840327e0 con 0x7fb1940a0280 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb19c4f8640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb19c4f8640 4 set_mon_vals no callback set 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.447+0000 7fb199292640 10 monclient: dump: 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 10 monclient: shutdown 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 -- 172.21.15.17:0/1891240266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 msgr2=0x7fb194131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/1891240266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 0x7fb194131ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb184009850 tx=0x7fb1840090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 -- 172.21.15.17:0/1891240266 shutdown_connections 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/1891240266 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb1940a0280 0x7fb194131ef0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/1891240266 >> v2:172.21.15.17:3300/0 conn(0x7fb194131b20 0x7fb19409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 -- 172.21.15.17:0/1891240266 >> 172.21.15.17:0/1891240266 conn(0x7fb194071270 msgr2=0x7fb1940a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 -- 172.21.15.17:0/1891240266 shutdown_connections 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 1 -- 172.21.15.17:0/1891240266 wait complete. 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.448+0000 7fb19c4f8640 10 monclient: monmap: 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:10.479 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 Processor -- start 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 -- start start 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient: init 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-c con 0x7fb19409f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb1940a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb194131b20 0x7fb194098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): picked mon.noname-b con 0x7fb194131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19411bea0 con 0x7fb1940a0280 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19409ace0 con 0x7fb194131b20 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940a5340 con 0x7fb19409f1e0 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19c4f8640 10 monclient(hunting): authenticate will time out at 1330.445697s 2024-09-15T02:01:10.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42112/0 (socket says 172.21.15.17:42112) 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 learned_addr learned my addr 172.21.15.17:0/2063035646 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39228/0 (socket says 172.21.15.17:39228) 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): get_auth_request con 0x7fb1940a0280 auth_method 0 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): get_auth_request con 0x7fb19409f1e0 auth_method 0 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19aa95640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb194131b20 0x7fb194098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19a294640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19aa95640 10 monclient(hunting): get_auth_request con 0x7fb194131b20 auth_method 0 2024-09-15T02:01:10.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19aa95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19aa95640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.449+0000 7fb19aa95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19aa95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19aa95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19aa95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 msgr2=0x7fb19412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 10 monclient(hunting): handle_auth_done global_id 134123 payload 824 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 msgr2=0x7fb19412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb194131b20 msgr2=0x7fb194098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb194131b20 0x7fb194098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 msgr2=0x7fb19412ed80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb1940a0280 0x7fb19412ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 monclient: found mon.noname-c 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.450+0000 7fb19a294640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190009ae0 con 0x7fb19409f1e0 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19a294640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 msgr2=0x7fb194141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19a294640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb190009fd0 tx=0x7fb19000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb190009d10 con 0x7fb19409f1e0 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: dump: 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19000f040 con 0x7fb19409f1e0 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1900146d0 con 0x7fb19409f1e0 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb18a7fc640 10 monclient: dump: 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 5 monclient: authenticate success, global_id 134123 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 10 monclient: _renew_subs 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb184009390 con 0x7fb19409f1e0 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb198a91640 4 set_mon_vals no callback set 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 10 monclient: _renew_subs 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.452+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb194141f60 con 0x7fb19409f1e0 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb19c4f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb190014b00 con 0x7fb19409f1e0 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 0x7fb194099150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb194063a40 con 0x7fb19412f2c0 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 0x7fb194099150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient: get_auth_request con 0x7fb19412f2c0 auth_method 0 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb18a7fc640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb178038790 0x7fb17803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 10 monclient(hunting): handle_auth_done global_id 104889 payload 824 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb19411c310 con 0x7fb19412f2c0 2024-09-15T02:01:10.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.453+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb19004cf40 con 0x7fb19409f1e0 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.454+0000 7fb19aa95640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb178038790 0x7fb17803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.454+0000 7fb19aa95640 10 monclient: get_auth_request con 0x7fb178038790 auth_method 0 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 msgr2=0x7fb194099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 0x7fb194099150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb184035040 tx=0x7fb184032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184031a30 con 0x7fb19412f2c0 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 10 monclient: dump: 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb184031b90 con 0x7fb19412f2c0 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 msgr2=0x7fb194099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 0x7fb194099150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb184035040 tx=0x7fb184032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 msgr2=0x7fb194099150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb18a7fc640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19412f2c0 0x7fb194099150 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb184035040 tx=0x7fb184032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.456+0000 7fb19aa95640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb178038790 0x7fb17803ac40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb18c007fd0 tx=0x7fb18c007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb19c4f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:10.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 0x7fb1940a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb19c4f8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19412f690 con 0x7fb194099690 2024-09-15T02:01:10.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 0x7fb1940a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient: get_auth_request con 0x7fb194099690 auth_method 0 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 10 monclient(hunting): handle_auth_done global_id 104892 payload 824 2024-09-15T02:01:10.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.462+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb194099a90 con 0x7fb194099690 2024-09-15T02:01:10.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.464+0000 7fb199a93640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 msgr2=0x7fb1940a2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb199a93640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 0x7fb1940a2fc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb184009130 tx=0x7fb18403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184057070 con 0x7fb194099690 2024-09-15T02:01:10.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 10 monclient: dump: 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fb184046600 con 0x7fb194099690 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 msgr2=0x7fb1940a2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 0x7fb1940a2fc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb184009130 tx=0x7fb18403d000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 msgr2=0x7fb1940a2fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb18a7fc640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb194099690 0x7fb1940a2fc0 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb184009130 tx=0x7fb18403d000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:10.501 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:10.502 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.503 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:01:10.503 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.503 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:10.503 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.504 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.505 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:10.506 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.507 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.508 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.509 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.512 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:10.513 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.514 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:10.515 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:10.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb178038790 msgr2=0x7fb17803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb178038790 0x7fb17803ac40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb18c007fd0 tx=0x7fb18c007e10 comp rx=0 tx=0).stop 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 10 monclient: shutdown 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 msgr2=0x7fb194141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb190009fd0 tx=0x7fb19000b040 comp rx=0 tx=0).stop 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19aa95640 1 -- 172.21.15.17:0/2063035646 reap_dead start 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 shutdown_connections 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 --2- 172.21.15.17:0/2063035646 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19409f1e0 0x7fb194141110 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 >> 172.21.15.17:0/2063035646 conn(0x7fb194071270 msgr2=0x7fb19409d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 shutdown_connections 2024-09-15T02:01:10.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.465+0000 7fb19c4f8640 1 -- 172.21.15.17:0/2063035646 wait complete. 2024-09-15T02:01:10.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:10.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:01:10.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.613+0000 7f467fb9d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.613+0000 7f467fb9d640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.613+0000 7f467fb9d640 10 monclient: monmap: 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:10.614105+0000 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:10.614105+0000 2024-09-15T02:01:10.615 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:10.616 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.613+0000 7f467fb9d640 1 Processor -- start 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 -- start start 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient: init 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46780a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4678132040 0x7f46780987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4678132040 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 0x7f46781391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4678136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780a01e0 con 0x7f46780a58b0 2024-09-15T02:01:10.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46780a1580 con 0x7f4678136dd0 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46780a5c80 con 0x7f4678132040 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467fb9d640 10 monclient(hunting): authenticate will time out at 1330.610663s 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 0x7f46781391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 0x7f46781391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42144/0 (socket says 172.21.15.17:42144) 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 1 -- 172.21.15.17:0/1009219642 learned_addr learned my addr 172.21.15.17:0/1009219642 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): get_auth_request con 0x7f4678136dd0 auth_method 0 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467e13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467d138640 1 --2- 172.21.15.17:0/1009219642 >> v2:172.21.15.67:3300/0 conn(0x7f4678132040 0x7f46780987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467d138640 10 monclient(hunting): get_auth_request con 0x7f4678132040 auth_method 0 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.614+0000 7f467d138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient(hunting): handle_auth_done global_id 124393 payload 274 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 1 -- 172.21.15.17:0/1009219642 >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 msgr2=0x7f46781391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 1 --2- 172.21.15.17:0/1009219642 >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 0x7f46781391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 1 -- 172.21.15.17:0/1009219642 >> v1:172.21.15.17:6789/0 conn(0x7f46780a58b0 legacy=0x7f46780980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 1 monclient: found mon.noname-c 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:01:10.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.615+0000 7f467d138640 1 -- 172.21.15.17:0/1009219642 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678132440 con 0x7f4678132040 2024-09-15T02:01:10.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467d138640 1 -- 172.21.15.17:0/1009219642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678132040 msgr2=0x7f46780987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467d138640 1 --2- 172.21.15.17:0/1009219642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678132040 0x7f46780987b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f467809f780 tx=0x7f46680090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68831fe2cf39d26b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 1 -- 172.21.15.17:0/1009219642 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466802ae00 con 0x7f4678132040 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: dump: 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.629 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 1 -- 172.21.15.17:0/1009219642 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4668032440 con 0x7f4678132040 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 5 monclient: authenticate success, global_id 124393 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 1 -- 172.21.15.17:0/1009219642 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680327e0 con 0x7f4678132040 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 4 set_mon_vals no callback set 2024-09-15T02:01:10.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467c937640 10 monclient: dump: 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 10 monclient: shutdown 2024-09-15T02:01:10.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.618+0000 7f467fb9d640 1 -- 172.21.15.17:0/1009219642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678132040 msgr2=0x7f46780987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 --2- 172.21.15.17:0/1009219642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678132040 0x7f46780987b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f467809f780 tx=0x7f46680090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 -- 172.21.15.17:0/1009219642 shutdown_connections 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 --2- 172.21.15.17:0/1009219642 >> v2:172.21.15.17:3300/0 conn(0x7f4678136dd0 0x7f46781391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 --2- 172.21.15.17:0/1009219642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678132040 0x7f46780987b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 -- 172.21.15.17:0/1009219642 >> 172.21.15.17:0/1009219642 conn(0x7f4678070ed0 msgr2=0x7f46780a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 -- 172.21.15.17:0/1009219642 shutdown_connections 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 -- 172.21.15.17:0/1009219642 wait complete. 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 10 monclient: monmap: 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:10.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.619+0000 7f467fb9d640 1 Processor -- start 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 -- start start 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient: init 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 0x7f4678132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f46780a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678132040 0x7f4678133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4678132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678136dd0 0x7f4678133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4678136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4678098dc0 con 0x7f46780a58b0 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780a3540 con 0x7f4678136dd0 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4678136260 con 0x7f4678132040 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467fb9d640 10 monclient(hunting): authenticate will time out at 1330.616524s 2024-09-15T02:01:10.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 0x7f4678132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 0x7f4678132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42148/0 (socket says 172.21.15.17:42148) 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 -- 172.21.15.17:0/2058402144 learned_addr learned my addr 172.21.15.17:0/2058402144 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): get_auth_request con 0x7f46780a58b0 auth_method 0 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678132040 0x7f4678133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): get_auth_request con 0x7f4678132040 auth_method 0 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467e13a640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678136dd0 0x7f4678133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467e13a640 10 monclient(hunting): get_auth_request con 0x7f4678136dd0 auth_method 0 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient(hunting): handle_auth_done global_id 104898 payload 824 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678132040 msgr2=0x7f4678133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678132040 0x7f4678133240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678136dd0 msgr2=0x7f4678133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4678136dd0 0x7f4678133780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 monclient: found mon.noname-a 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.620+0000 7f467d939640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668009390 con 0x7f46780a58b0 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467d939640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 msgr2=0x7f4678132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467d939640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 0x7f4678132d00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f467400b4f0 tx=0x7f46740090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467400f720 con 0x7f46780a58b0 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: dump: 2024-09-15T02:01:10.637 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 5 monclient: authenticate success, global_id 104898 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 10 monclient: _renew_subs 2024-09-15T02:01:10.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678134680 con 0x7f46780a58b0 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467400f8a0 con 0x7f46780a58b0 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467400fc60 con 0x7f46780a58b0 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466dffb640 10 monclient: dump: 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.639 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 10 monclient: _renew_subs 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678133cc0 con 0x7f46780a58b0 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.623+0000 7f466f7fe640 4 set_mon_vals no callback set 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467fb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467fb9d640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 0x7f4678137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4678063b20 con 0x7f4678134d40 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4674011610 con 0x7f46780a58b0 2024-09-15T02:01:10.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 0x7f4678137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient: get_auth_request con 0x7f4678134d40 auth_method 0 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f466dffb640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f465c038770 0x7f465c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 10 monclient(hunting): handle_auth_done global_id 134129 payload 824 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f467404c100 con 0x7f46780a58b0 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467d138640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4678136960 con 0x7f4678134d40 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467e13a640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f465c038770 0x7f465c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.624+0000 7f467e13a640 10 monclient: get_auth_request con 0x7f465c038770 auth_method 0 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.626+0000 7f467d138640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 msgr2=0x7f4678137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.626+0000 7f467d138640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 0x7f4678137210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4668034000 tx=0x7f4668032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668031c20 con 0x7f4678134d40 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 10 monclient: dump: 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.643 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.643 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.643 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4668031d80 con 0x7f4678134d40 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 msgr2=0x7f4678137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 0x7f4678137210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4668034000 tx=0x7f4668032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 msgr2=0x7f4678137210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f466dffb640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678134d40 0x7f4678137210 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4668034000 tx=0x7f4668032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.627+0000 7f467e13a640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f465c038770 0x7f465c03ac20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f46700072f0 tx=0x7f467000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467fb9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:10.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467fb9d640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 0x7f4678137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46780724a0 con 0x7f4678137750 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 0x7f4678137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient: get_auth_request con 0x7f4678137750 auth_method 0 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 10 monclient(hunting): handle_auth_done global_id 134132 payload 824 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.633+0000 7f467d138640 1 -- 172.21.15.17:0/2058402144 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f467809fc90 con 0x7f4678137750 2024-09-15T02:01:10.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f467d138640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 msgr2=0x7f4678137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f467d138640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 0x7f4678137b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4678136760 tx=0x7f466803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668046810 con 0x7f4678137750 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f466dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f466dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.635+0000 7f466dffb640 10 monclient: dump: 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f4668046970 con 0x7f4678137750 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 msgr2=0x7f4678137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 0x7f4678137b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4678136760 tx=0x7f466803d000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 msgr2=0x7f4678137b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f466dffb640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4678137750 0x7f4678137b50 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4678136760 tx=0x7f466803d000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:10.649 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.650 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.651 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:10.652 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.653 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.654 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.656 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.657 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.658 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.659 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:10.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:10.661 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f465c038770 msgr2=0x7f465c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f465c038770 0x7f465c03ac20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f46700072f0 tx=0x7f467000b040 comp rx=0 tx=0).stop 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 10 monclient: shutdown 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 msgr2=0x7f4678132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 --2- 172.21.15.17:0/2058402144 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46780a58b0 0x7f4678132d00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f467400b4f0 tx=0x7f46740090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467e13a640 1 -- 172.21.15.17:0/2058402144 reap_dead start 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467e13a640 1 -- 172.21.15.17:0/2058402144 reap_dead start 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 shutdown_connections 2024-09-15T02:01:10.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 >> 172.21.15.17:0/2058402144 conn(0x7f4678070ed0 msgr2=0x7f4678096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 shutdown_connections 2024-09-15T02:01:10.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.636+0000 7f467fb9d640 1 -- 172.21.15.17:0/2058402144 wait complete. 2024-09-15T02:01:10.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:10.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:01:10.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:01:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.759+0000 7f974839d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.759+0000 7f974839d640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.759+0000 7f974839d640 10 monclient: monmap: 2024-09-15T02:01:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:10.760381+0000 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:10.760381+0000 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:10.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 Processor -- start 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 -- start start 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient: init 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9740097320 0x7f97400976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): picked mon.noname-a con 0x7f9740097320 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f974009a590 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f974012edb0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400985d0 con 0x7f974012edb0 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974009d200 con 0x7f974009a590 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9740097c30 con 0x7f9740097320 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f974839d640 10 monclient(hunting): authenticate will time out at 1330.756806s 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f9746139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9740097320 0x7f97400976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.760+0000 7f9746139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9740097320 0x7f97400976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42162/0 (socket says 172.21.15.17:42162) 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 1 -- 172.21.15.17:0/858429081 learned_addr learned my addr 172.21.15.17:0/858429081 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): get_auth_request con 0x7f9740097320 auth_method 0 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.761+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f974693a640 1 -- 172.21.15.17:0/858429081 >> v1:172.21.15.17:6789/0 conn(0x7f974012edb0 legacy=0x7f97401369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 10 monclient(hunting): handle_auth_done global_id 104901 payload 274 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9745938640 1 -- 172.21.15.17:0/858429081 >> v1:172.21.15.67:6789/0 conn(0x7f974009a590 legacy=0x7f974012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 1 -- 172.21.15.17:0/858429081 >> v1:172.21.15.67:6789/0 conn(0x7f974009a590 legacy=0x7f974012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 1 -- 172.21.15.17:0/858429081 >> v1:172.21.15.17:6789/0 conn(0x7f974012edb0 legacy=0x7f97401369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 1 monclient: found mon.noname-a 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.763+0000 7f9746139640 1 -- 172.21.15.17:0/858429081 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400992f0 con 0x7f9740097320 2024-09-15T02:01:10.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9746139640 1 -- 172.21.15.17:0/858429081 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 msgr2=0x7f97400976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9746139640 1 --2- 172.21.15.17:0/858429081 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f97400976f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f973c009a10 tx=0x7f973c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ee5ad3e76380fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 1 -- 172.21.15.17:0/858429081 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c02f4b0 con 0x7f9740097320 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: dump: 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.775 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 1 -- 172.21.15.17:0/858429081 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c02f610 con 0x7f9740097320 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 5 monclient: authenticate success, global_id 104901 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 4 set_mon_vals no callback set 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 1 -- 172.21.15.17:0/858429081 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c02f9f0 con 0x7f9740097320 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:10.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f9745137640 10 monclient: dump: 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 10 monclient: shutdown 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 -- 172.21.15.17:0/858429081 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 msgr2=0x7f97400976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 --2- 172.21.15.17:0/858429081 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f97400976f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f973c009a10 tx=0x7f973c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 -- 172.21.15.17:0/858429081 shutdown_connections 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 --2- 172.21.15.17:0/858429081 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f97400976f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 -- 172.21.15.17:0/858429081 >> 172.21.15.17:0/858429081 conn(0x7f97400a1000 msgr2=0x7f97400a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 -- 172.21.15.17:0/858429081 shutdown_connections 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.766+0000 7f974839d640 1 -- 172.21.15.17:0/858429081 wait complete. 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.767+0000 7f974839d640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.767+0000 7f974839d640 10 monclient: monmap: 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:10.778 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.767+0000 7f974839d640 1 Processor -- start 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.767+0000 7f974839d640 1 -- start start 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.767+0000 7f974839d640 10 monclient: init 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f974012da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): picked mon.noname-a con 0x7f9740097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:10.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974009a590 0x7f974012dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): picked mon.noname-b con 0x7f974009a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 0x7f9740141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): picked mon.noname-c con 0x7f974012edb0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740122000 con 0x7f9740097320 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974012e6d0 con 0x7f974009a590 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974012e510 con 0x7f974012edb0 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974839d640 10 monclient(hunting): authenticate will time out at 1330.764192s 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f974012da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f974012da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42178/0 (socket says 172.21.15.17:42178) 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 1 -- 172.21.15.17:0/3858940817 learned_addr learned my addr 172.21.15.17:0/3858940817 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): get_auth_request con 0x7f9740097320 auth_method 0 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 0x7f9740141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): get_auth_request con 0x7f974012edb0 auth_method 0 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974009a590 0x7f974012dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): get_auth_request con 0x7f974009a590 auth_method 0 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9745938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient(hunting): handle_auth_done global_id 134135 payload 824 2024-09-15T02:01:10.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974009a590 msgr2=0x7f974012dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f974009a590 0x7f974012dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 msgr2=0x7f974012da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f974012da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 monclient: found mon.noname-c 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f974693a640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c009450 con 0x7f974012edb0 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.768+0000 7f9746139640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9740097320 0x7f974012da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.770+0000 7f974693a640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 msgr2=0x7f9740141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.770+0000 7f974693a640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 0x7f9740141aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f973800c7c0 tx=0x7f9738007f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738005d10 con 0x7f974012edb0 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: dump: 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 5 monclient: authenticate success, global_id 134135 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 monclient: _renew_subs 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738005e90 con 0x7f974012edb0 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974006b2a0 con 0x7f974012edb0 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973800a890 con 0x7f974012edb0 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 10 monclient: dump: 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.785 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f9733fff640 4 set_mon_vals no callback set 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 monclient: _renew_subs 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9740141fe0 con 0x7f974012edb0 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 0x7f974006a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740063c20 con 0x7f9740142200 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.771+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f973800ac80 con 0x7f974012edb0 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 0x7f974006a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient: get_auth_request con 0x7f9740142200 auth_method 0 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f97327fc640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9724038770 0x7f972403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f973804c290 con 0x7f974012edb0 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9745938640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9724038770 0x7f972403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 10 monclient(hunting): handle_auth_done global_id 124402 payload 824 2024-09-15T02:01:10.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9746139640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9740122140 con 0x7f9740142200 2024-09-15T02:01:10.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.772+0000 7f9745938640 10 monclient: get_auth_request con 0x7f9724038770 auth_method 0 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f9746139640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 msgr2=0x7f974006a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f9746139640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 0x7f974006a090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f973c009a10 tx=0x7f973c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f9745938640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9724038770 0x7f972403ac20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9740068e50 tx=0x7f9734009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c02f4b0 con 0x7f9740142200 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f97327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:10.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.774+0000 7f97327fc640 10 monclient: dump: 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f973c02fe20 con 0x7f9740142200 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 msgr2=0x7f974006a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 0x7f974006a090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f973c009a10 tx=0x7f973c032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 msgr2=0x7f974006a090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.775+0000 7f97327fc640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9740142200 0x7f974006a090 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f973c009a10 tx=0x7f973c032000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f974839d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f974839d640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 0x7f9740029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f974839d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974009ad50 con 0x7f97400298c0 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 0x7f9740029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient: get_auth_request con 0x7f97400298c0 auth_method 0 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 10 monclient(hunting): handle_auth_done global_id 124405 payload 824 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.781+0000 7f9746139640 1 -- 172.21.15.17:0/3858940817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97400724a0 con 0x7f97400298c0 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f9746139640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 msgr2=0x7f9740029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f9746139640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 0x7f9740029cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f973c009210 tx=0x7f973c03e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c04c070 con 0x7f97400298c0 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 10 monclient: dump: 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.794 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.794 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.797 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.797 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.797 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.797 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.797 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f973c047d10 con 0x7f97400298c0 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 msgr2=0x7f9740029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 0x7f9740029cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f973c009210 tx=0x7f973c03e000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 msgr2=0x7f9740029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f97327fc640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97400298c0 0x7f9740029cc0 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f973c009210 tx=0x7f973c03e000 comp rx=0 tx=0).stop 2024-09-15T02:01:10.798 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 110, 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:10.799 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:10.800 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:10.801 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:10.802 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.803 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.806 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:10.807 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.808 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.809 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.810 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:10.811 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:10.812 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9724038770 msgr2=0x7f972403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9724038770 0x7f972403ac20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9740068e50 tx=0x7f9734009040 comp rx=0 tx=0).stop 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 10 monclient: shutdown 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 msgr2=0x7f9740141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974693a640 1 -- 172.21.15.17:0/3858940817 reap_dead start 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 0x7f9740141aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f973800c7c0 tx=0x7f9738007f70 comp rx=0 tx=0).stop 2024-09-15T02:01:10.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 shutdown_connections 2024-09-15T02:01:10.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 --2- 172.21.15.17:0/3858940817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f974012edb0 0x7f9740141aa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.784+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 >> 172.21.15.17:0/3858940817 conn(0x7f97400a1000 msgr2=0x7f97400a02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.785+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 shutdown_connections 2024-09-15T02:01:10.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.785+0000 7f974839d640 1 -- 172.21.15.17:0/3858940817 wait complete. 2024-09-15T02:01:10.814 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:10.814 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T02:01:10.814 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:01:10.815 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T02:01:10.815 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:01:10.815 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:01:10.815 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:01:10.831 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:01:10.831 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:01:10.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:10.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.925+0000 7fb019314640 10 monclient: get_monmap_and_config 2024-09-15T02:01:10.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.925+0000 7fb019314640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.925+0000 7fb019314640 10 monclient: monmap: 2024-09-15T02:01:10.927 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.927 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:10.926373+0000 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:10.926373+0000 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:10.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 Processor -- start 2024-09-15T02:01:10.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 -- start start 2024-09-15T02:01:10.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient: init 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): picked mon.noname-b con 0x7fb01409c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb01409c580 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb01409c990 0x7fb01409e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): picked mon.noname-a con 0x7fb01409c990 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01409f790 con 0x7fb01409c580 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb014069d40 con 0x7fb01409c990 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb014068b60 con 0x7fb01409c1b0 2024-09-15T02:01:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb019314640 10 monclient(hunting): authenticate will time out at 1330.922727s 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb0137fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb01409c990 0x7fb01409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb00a7fc640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fb01409c580 legacy=0x7fb01406eb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39284/0 (socket says 172.21.15.17:39284) 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 1 -- 172.21.15.17:0/1690625170 learned_addr learned my addr 172.21.15.17:0/1690625170 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): get_auth_request con 0x7fb01409c1b0 auth_method 0 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb0137fe640 10 monclient(hunting): get_auth_request con 0x7fb01409c990 auth_method 0 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb0137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb0137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb0137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.926+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 10 monclient(hunting): handle_auth_done global_id 134138 payload 274 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 1 -- 172.21.15.17:0/1690625170 >> v2:172.21.15.17:3300/0 conn(0x7fb01409c990 msgr2=0x7fb01409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 1 --2- 172.21.15.17:0/1690625170 >> v2:172.21.15.17:3300/0 conn(0x7fb01409c990 0x7fb01409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 1 -- 172.21.15.17:0/1690625170 >> v1:172.21.15.67:6789/0 conn(0x7fb01409c580 legacy=0x7fb01406eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 1 monclient: found mon.noname-b 2024-09-15T02:01:10.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:10.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.927+0000 7fb012ffd640 1 -- 172.21.15.17:0/1690625170 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014069aa0 con 0x7fb01409c1b0 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.929+0000 7fb012ffd640 1 -- 172.21.15.17:0/1690625170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 msgr2=0x7fb01409a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.929+0000 7fb012ffd640 1 --2- 172.21.15.17:0/1690625170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb004009910 tx=0x7fb0040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeaaa029cd4b9117 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 1 -- 172.21.15.17:0/1690625170 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00402aea0 con 0x7fb01409c1b0 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: dump: 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 1 -- 172.21.15.17:0/1690625170 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004032440 con 0x7fb01409c1b0 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 5 monclient: authenticate success, global_id 134138 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 4 set_mon_vals no callback set 2024-09-15T02:01:10.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 1 -- 172.21.15.17:0/1690625170 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004032820 con 0x7fb01409c1b0 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb0127fc640 10 monclient: dump: 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.941 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 10 monclient: shutdown 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 -- 172.21.15.17:0/1690625170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 msgr2=0x7fb01409a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 --2- 172.21.15.17:0/1690625170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb004009910 tx=0x7fb0040091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 -- 172.21.15.17:0/1690625170 shutdown_connections 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 --2- 172.21.15.17:0/1690625170 >> v2:172.21.15.17:3300/0 conn(0x7fb01409c990 0x7fb01409e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 --2- 172.21.15.17:0/1690625170 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c1b0 0x7fb01409a990 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 -- 172.21.15.17:0/1690625170 >> 172.21.15.17:0/1690625170 conn(0x7fb014071300 msgr2=0x7fb0140a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 -- 172.21.15.17:0/1690625170 shutdown_connections 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.930+0000 7fb019314640 1 -- 172.21.15.17:0/1690625170 wait complete. 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.931+0000 7fb019314640 10 monclient: build_initial_monmap 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.931+0000 7fb019314640 10 monclient: monmap: 2024-09-15T02:01:10.942 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.931+0000 7fb019314640 1 Processor -- start 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.931+0000 7fb019314640 1 -- start start 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.931+0000 7fb019314640 10 monclient: init 2024-09-15T02:01:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): picked mon.noname-a con 0x7fb01409c1b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 0x7fb014132160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): picked mon.noname-b con 0x7fb01409c580 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 0x7fb0141326a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): picked mon.noname-c con 0x7fb01409c990 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01411f840 con 0x7fb01409c1b0 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01411e420 con 0x7fb01409c580 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:10.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140a1210 con 0x7fb01409c990 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb019314640 10 monclient(hunting): authenticate will time out at 1330.928265s 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb00a7fc640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 msgr2=0x7fb014132160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42222/0 (socket says 172.21.15.17:42222) 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 0x7fb0141326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 -- 172.21.15.17:0/4294206172 learned_addr learned my addr 172.21.15.17:0/4294206172 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): get_auth_request con 0x7fb01409c1b0 auth_method 0 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb00a7fc640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 0x7fb014132160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): get_auth_request con 0x7fb01409c990 auth_method 0 2024-09-15T02:01:10.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient(hunting): handle_auth_done global_id 104910 payload 824 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 -- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 msgr2=0x7fb0141326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 0x7fb0141326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 -- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 msgr2=0x7fb014132160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:10.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 0x7fb014132160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 monclient: found mon.noname-a 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb012ffd640 1 -- 172.21.15.17:0/4294206172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004009450 con 0x7fb01409c1b0 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.932+0000 7fb0137fe640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 0x7fb0141326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.934+0000 7fb012ffd640 1 -- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 msgr2=0x7fb014141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.934+0000 7fb012ffd640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb004009970 tx=0x7fb004032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00402aea0 con 0x7fb01409c1b0 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: dump: 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004031c70 con 0x7fb01409c1b0 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 5 monclient: authenticate success, global_id 104910 2024-09-15T02:01:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 10 monclient: _renew_subs 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01411f280 con 0x7fb01409c1b0 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00403c400 con 0x7fb01409c1b0 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb00b7fe640 10 monclient: dump: 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:10.949 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb011ffb640 4 set_mon_vals no callback set 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 10 monclient: _renew_subs 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.935+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb014099870 con 0x7fb01409c1b0 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb00403c830 con 0x7fb01409c1b0 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb019314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb019314640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb019314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0140724a0 con 0x7fb01409c1b0 2024-09-15T02:01:10.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb00b7fe640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 0x7faff403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb00406ce00 con 0x7fb01409c1b0 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb00a7fc640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 0x7faff403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.936+0000 7fb00a7fc640 10 monclient: get_auth_request con 0x7faff40386f0 auth_method 0 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.939+0000 7fb00a7fc640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 0x7faff403aba0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faffc006fd0 tx=0x7faffc006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.940+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb00403b3a0 con 0x7fb01409c1b0 2024-09-15T02:01:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.940+0000 7fb00b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:10.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:10.940+0000 7fb00b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:11.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb019314640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:11.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb019314640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:11.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb019314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:11.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb0140a36e0 con 0x7fb01409c1b0 2024-09-15T02:01:11.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb00b7fe640 1 -- 172.21.15.17:0/4294206172 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fb004037020 con 0x7fb01409c1b0 2024-09-15T02:01:11.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb00b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:11.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.168+0000 7fb00b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:11.171 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:11.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 msgr2=0x7faff403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 0x7faff403aba0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faffc006fd0 tx=0x7faffc006d30 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 10 monclient: shutdown 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 msgr2=0x7fb014141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb004009970 tx=0x7fb004032000 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 shutdown_connections 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faff40386f0 0x7faff403aba0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb01409c990 0x7fb0141326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01409c580 0x7fb014132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 --2- 172.21.15.17:0/4294206172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb01409c1b0 0x7fb014141570 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 >> 172.21.15.17:0/4294206172 conn(0x7fb014071300 msgr2=0x7fb0140a2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 shutdown_connections 2024-09-15T02:01:11.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:11.170+0000 7fb019314640 1 -- 172.21.15.17:0/4294206172 wait complete. 2024-09-15T02:01:14.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:14.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.289+0000 7fab364ef640 10 monclient: get_monmap_and_config 2024-09-15T02:01:14.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.289+0000 7fab364ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:14.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.289+0000 7fab364ef640 10 monclient: monmap: 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:14.290480+0000 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:14.290480+0000 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:14.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 1 Processor -- start 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 1 -- start start 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 10 monclient: init 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.290+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-b con 0x7fab3009c4f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fab3009aed0 0x7fab3006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-a con 0x7fab3009aed0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:14.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab3009c8c0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3006a390 con 0x7fab3009c8c0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3006f780 con 0x7fab3009aed0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab30099f50 con 0x7fab3009c4f0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab364ef640 10 monclient(hunting): authenticate will time out at 1334.287060s 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fab3009aed0 0x7fab3006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab34a8c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fab3009c8c0 legacy=0x7fab3006f070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fab3009aed0 0x7fab3006eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42492/0 (socket says 172.21.15.17:42492) 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40986/0 (socket says 172.21.15.17:40986) 2024-09-15T02:01:14.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 -- 172.21.15.17:0/572614151 learned_addr learned my addr 172.21.15.17:0/572614151 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): get_auth_request con 0x7fab3009aed0 auth_method 0 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): get_auth_request con 0x7fab3009c4f0 auth_method 0 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient(hunting): handle_auth_done global_id 104913 payload 274 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 -- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 msgr2=0x7fab3009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 -- 172.21.15.17:0/572614151 >> v1:172.21.15.67:6789/0 conn(0x7fab3009c8c0 legacy=0x7fab3006f070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2f7fe640 1 -- 172.21.15.17:0/572614151 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3009f790 con 0x7fab3009aed0 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.291+0000 7fab2ffff640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:14.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2f7fe640 1 -- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009aed0 msgr2=0x7fab3006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2f7fe640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009aed0 0x7fab3006eb30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab24009910 tx=0x7fab240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45f513f6a23c110 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 1 -- 172.21.15.17:0/572614151 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2402aea0 con 0x7fab3009aed0 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: dump: 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:14.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: _finish_auth 0 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 1 -- 172.21.15.17:0/572614151 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24032440 con 0x7fab3009aed0 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab364ef640 5 monclient: authenticate success, global_id 104913 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab364ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab364ef640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 1 -- 172.21.15.17:0/572614151 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240327e0 con 0x7fab3009aed0 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab364ef640 4 set_mon_vals no callback set 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:14.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.294+0000 7fab2effd640 10 monclient: dump: 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 10 monclient: shutdown 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:14.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 -- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009aed0 msgr2=0x7fab3006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009aed0 0x7fab3006eb30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab24009910 tx=0x7fab240091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 -- 172.21.15.17:0/572614151 shutdown_connections 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009aed0 0x7fab3006eb30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 --2- 172.21.15.17:0/572614151 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab3009c4f0 0x7fab3009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 -- 172.21.15.17:0/572614151 >> 172.21.15.17:0/572614151 conn(0x7fab30071300 msgr2=0x7fab300a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 -- 172.21.15.17:0/572614151 shutdown_connections 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 1 -- 172.21.15.17:0/572614151 wait complete. 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.295+0000 7fab364ef640 10 monclient: monmap: 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:14.312 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 Processor -- start 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 -- start start 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient: init 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-a con 0x7fab3009c4f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 0x7fab3013d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-b con 0x7fab3009c8c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): picked mon.noname-c con 0x7fab30141970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab30122700 con 0x7fab3009c4f0 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab30122420 con 0x7fab3009c8c0 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3011ff90 con 0x7fab30141970 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab364ef640 10 monclient(hunting): authenticate will time out at 1334.292822s 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:14.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40992/0 (socket says 172.21.15.17:40992) 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42500/0 (socket says 172.21.15.17:42500) 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2f7fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 msgr2=0x7fab3013d080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 1 -- 172.21.15.17:0/605083155 learned_addr learned my addr 172.21.15.17:0/605083155 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 10 monclient(hunting): get_auth_request con 0x7fab30141970 auth_method 0 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab34a8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 10 monclient(hunting): get_auth_request con 0x7fab3009c4f0 auth_method 0 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.296+0000 7fab2f7fe640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 0x7fab3013d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:14.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient(hunting): handle_auth_done global_id 134147 payload 824 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 -- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 msgr2=0x7fab3013d080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 0x7fab3013d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 -- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 msgr2=0x7fab30140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient: _finish_hunting 0 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 monclient: found mon.noname-c 2024-09-15T02:01:14.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab34a8c640 1 -- 172.21.15.17:0/605083155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab20009390 con 0x7fab30141970 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.297+0000 7fab2ffff640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab34a8c640 1 -- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 msgr2=0x7fab3013d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab34a8c640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab2800bb10 tx=0x7fab280097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2800cbc0 con 0x7fab30141970 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: dump: 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.317 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: _finish_auth 0 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2800cd40 con 0x7fab30141970 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 5 monclient: authenticate success, global_id 134147 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28006150 con 0x7fab30141970 2024-09-15T02:01:14.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab17fff640 10 monclient: dump: 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:14.319 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 10 monclient: _renew_subs 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab24009450 con 0x7fab30141970 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab2e7fc640 4 set_mon_vals no callback set 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 10 monclient: _renew_subs 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.299+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3013d5c0 con 0x7fab30141970 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fab28012020 con 0x7fab30141970 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab364ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab364ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab300724a0 con 0x7fab30141970 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.300+0000 7fab17fff640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 0x7fab1c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.301+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fab2804c4b0 con 0x7fab30141970 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.301+0000 7fab2ffff640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 0x7fab1c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.301+0000 7fab2ffff640 10 monclient: get_auth_request con 0x7fab1c0386f0 auth_method 0 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.303+0000 7fab2ffff640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 0x7fab1c03aba0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fab20011580 tx=0x7fab200090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.305+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fab2801d8d0 con 0x7fab30141970 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.305+0000 7fab17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:14.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.305+0000 7fab17fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:14.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab364ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab364ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab30067220 con 0x7fab30141970 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab17fff640 1 -- 172.21.15.17:0/605083155 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fab2801d300 con 0x7fab30141970 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:14.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.533+0000 7fab17fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:14.536 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:14.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 msgr2=0x7fab1c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:14.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 0x7fab1c03aba0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fab20011580 tx=0x7fab200090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 10 monclient: shutdown 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 msgr2=0x7fab3013d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab2800bb10 tx=0x7fab280097d0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 shutdown_connections 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab1c0386f0 0x7fab1c03aba0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fab30141970 0x7fab3013d9f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3009c8c0 0x7fab3013d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 --2- 172.21.15.17:0/605083155 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fab3009c4f0 0x7fab30140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 >> 172.21.15.17:0/605083155 conn(0x7fab30071300 msgr2=0x7fab30098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 shutdown_connections 2024-09-15T02:01:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:14.535+0000 7fab364ef640 1 -- 172.21.15.17:0/605083155 wait complete. 2024-09-15T02:01:17.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:17.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.654+0000 7efcb7f14640 10 monclient: get_monmap_and_config 2024-09-15T02:01:17.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.654+0000 7efcb7f14640 10 monclient: build_initial_monmap 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.654+0000 7efcb7f14640 10 monclient: monmap: 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:17.655468+0000 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:17.655468+0000 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:17.657 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:17.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:17.658 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:17.658 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:17.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.655+0000 7efcb7f14640 1 Processor -- start 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.655+0000 7efcb7f14640 1 -- start start 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.655+0000 7efcb7f14640 10 monclient: init 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.655+0000 7efcb7f14640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-b con 0x7efcb009b850 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 0x7efcb009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-c con 0x7efcb009bc20 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcb009c530 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb009d710 con 0x7efcb009c530 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb0097c20 con 0x7efcb009bc20 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0096e80 con 0x7efcb009b850 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb7f14640 10 monclient(hunting): authenticate will time out at 1337.652116s 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb54af640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 msgr2=0x7efcb009bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb64b1640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7efcb009c530 0x7efcb0135e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:48702/0 (socket says 172.21.15.17:48702) 2024-09-15T02:01:17.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb54af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 0x7efcb009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41008/0 (socket says 172.21.15.17:41008) 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb64b1640 1 -- 172.21.15.17:0/2145622824 learned_addr learned my addr 172.21.15.17:0/2145622824 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): get_auth_request con 0x7efcb009b850 auth_method 0 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_done global_id 134150 payload 274 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 -- 172.21.15.17:0/2145622824 >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 msgr2=0x7efcb009bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:17.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 --2- 172.21.15.17:0/2145622824 >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 0x7efcb009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 -- 172.21.15.17:0/2145622824 >> v1:172.21.15.17:6789/0 conn(0x7efcb009c530 legacy=0x7efcb0135e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient: _finish_hunting 0 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 monclient: found mon.noname-b 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.656+0000 7efcb5cb0640 1 -- 172.21.15.17:0/2145622824 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0096fc0 con 0x7efcb009b850 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb5cb0640 1 -- 172.21.15.17:0/2145622824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 msgr2=0x7efcb0098ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb5cb0640 1 --2- 172.21.15.17:0/2145622824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efcac009850 tx=0x7efcac0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2811ca5ab12823 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 1 -- 172.21.15.17:0/2145622824 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac032440 con 0x7efcb009b850 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: dump: 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:17.669 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: _finish_auth 0 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb7f14640 5 monclient: authenticate success, global_id 134150 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 1 -- 172.21.15.17:0/2145622824 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac0325a0 con 0x7efcb009b850 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb7f14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:17.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb7f14640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb7f14640 4 set_mon_vals no callback set 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 1 -- 172.21.15.17:0/2145622824 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac032980 con 0x7efcb009b850 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.659+0000 7efcb4cae640 10 monclient: dump: 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:17.671 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 10 monclient: shutdown 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 -- 172.21.15.17:0/2145622824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 msgr2=0x7efcb0098ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 --2- 172.21.15.17:0/2145622824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efcac009850 tx=0x7efcac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 -- 172.21.15.17:0/2145622824 shutdown_connections 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 --2- 172.21.15.17:0/2145622824 >> v2:172.21.15.67:3300/0 conn(0x7efcb009bc20 0x7efcb009bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 --2- 172.21.15.17:0/2145622824 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009b850 0x7efcb0098ea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 -- 172.21.15.17:0/2145622824 >> 172.21.15.17:0/2145622824 conn(0x7efcb00a1000 msgr2=0x7efcb00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 -- 172.21.15.17:0/2145622824 shutdown_connections 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 1 -- 172.21.15.17:0/2145622824 wait complete. 2024-09-15T02:01:17.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 10 monclient: build_initial_monmap 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.660+0000 7efcb7f14640 10 monclient: monmap: 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 Processor -- start 2024-09-15T02:01:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 -- start start 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient: init 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-a con 0x7efcb009b850 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-c con 0x7efcb009bc20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 0x7efcb0122df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): picked mon.noname-b con 0x7efcb009c530 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb009d300 con 0x7efcb009b850 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0135ef0 con 0x7efcb009c530 2024-09-15T02:01:17.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb01469a0 con 0x7efcb009bc20 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb7f14640 10 monclient(hunting): authenticate will time out at 1337.657790s 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb64b1640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 msgr2=0x7efcb0122df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41018/0 (socket says 172.21.15.17:41018) 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 1 -- 172.21.15.17:0/4021539515 learned_addr learned my addr 172.21.15.17:0/4021539515 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42506/0 (socket says 172.21.15.17:42506) 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): get_auth_request con 0x7efcb009bc20 auth_method 0 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:17.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb64b1640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 0x7efcb0122df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 10 monclient(hunting): get_auth_request con 0x7efcb009b850 auth_method 0 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb5cb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.661+0000 7efcb54af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb5cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 10 monclient(hunting): handle_auth_done global_id 134153 payload 824 2024-09-15T02:01:17.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 -- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 msgr2=0x7efcb0122df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 0x7efcb0122df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 -- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 msgr2=0x7efcb0145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 10 monclient: _finish_hunting 0 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 monclient: found mon.noname-c 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb54af640 1 -- 172.21.15.17:0/4021539515 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac009390 con 0x7efcb009bc20 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.662+0000 7efcb5cb0640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb54af640 1 -- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 msgr2=0x7efcb01352a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb54af640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efca000b600 tx=0x7efca00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0015070 con 0x7efcb009bc20 2024-09-15T02:01:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: dump: 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:01:17.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0003d90 con 0x7efcb009bc20 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 5 monclient: authenticate success, global_id 134153 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 10 monclient: _renew_subs 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb0125570 con 0x7efcb009bc20 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0006380 con 0x7efcb009bc20 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca5ffb640 10 monclient: dump: 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.679 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 10 monclient: _renew_subs 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efca7fff640 4 set_mon_vals no callback set 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.664+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb01357e0 con 0x7efcb009bc20 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efca0010440 con 0x7efcb009bc20 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efcb7f14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efcb7f14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efcb7f14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb006ed70 con 0x7efcb009bc20 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.665+0000 7efca5ffb640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 0x7efc9403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.666+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7efca004c0b0 con 0x7efcb009bc20 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.666+0000 7efcb5cb0640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 0x7efc9403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.666+0000 7efcb5cb0640 10 monclient: get_auth_request con 0x7efc94038740 auth_method 0 2024-09-15T02:01:17.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.668+0000 7efcb5cb0640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 0x7efc9403abf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7efcac034000 tx=0x7efcac032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:17.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.670+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efcb006ed70 con 0x7efcb009bc20 2024-09-15T02:01:17.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.670+0000 7efca5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:17.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.670+0000 7efca5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:17.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efcb7f14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:17.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efcb7f14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:17.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efcb7f14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:17.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efcb006ed70 con 0x7efcb009bc20 2024-09-15T02:01:17.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efca5ffb640 1 -- 172.21.15.17:0/4021539515 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7efcb006ed70 con 0x7efcb009bc20 2024-09-15T02:01:17.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efca5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:17.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.898+0000 7efca5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:17.901 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:17.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 msgr2=0x7efc9403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 0x7efc9403abf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7efcac034000 tx=0x7efcac032000 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 10 monclient: shutdown 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 msgr2=0x7efcb01352a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efca000b600 tx=0x7efca00091a0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 shutdown_connections 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc94038740 0x7efc9403abf0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb009c530 0x7efcb0122df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efcb009bc20 0x7efcb01352a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 --2- 172.21.15.17:0/4021539515 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efcb009b850 0x7efcb0145ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:17.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 >> 172.21.15.17:0/4021539515 conn(0x7efcb00a1000 msgr2=0x7efcb009f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:17.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 shutdown_connections 2024-09-15T02:01:17.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:17.900+0000 7efcb7f14640 1 -- 172.21.15.17:0/4021539515 wait complete. 2024-09-15T02:01:20.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:21.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.020+0000 7faa4aa80640 10 monclient: get_monmap_and_config 2024-09-15T02:01:21.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.020+0000 7faa4aa80640 10 monclient: build_initial_monmap 2024-09-15T02:01:21.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.020+0000 7faa4aa80640 10 monclient: monmap: 2024-09-15T02:01:21.022 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:21.021334+0000 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:21.021334+0000 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:21.023 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 Processor -- start 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 -- start start 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient: init 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-b con 0x7faa44096c10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa4409d3b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:21.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-c con 0x7faa440976f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa440983b0 con 0x7faa4409d3b0 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa44099260 con 0x7faa440976f0 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4409a460 con 0x7faa44096c10 2024-09-15T02:01:21.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.021+0000 7faa4aa80640 10 monclient(hunting): authenticate will time out at 1341.017914s 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa43fff640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7faa4409d3b0 legacy=0x7faa44096fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4901d640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 msgr2=0x7faa44135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4901d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41026/0 (socket says 172.21.15.17:41026) 2024-09-15T02:01:21.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 learned_addr learned my addr 172.21.15.17:0/2713097597 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa44096c10 auth_method 0 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:21.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.022+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:21.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa44096c10 auth_method 0 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa43fff640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.67:6789/0 conn(0x7faa4409d3b0 legacy=0x7faa44096fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4901d640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 msgr2=0x7faa44135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4901d640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:21.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4901d640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 msgr2=0x7faa44135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa43fff640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.67:6789/0 conn(0x7faa4409d3b0 legacy=0x7faa44096fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4901d640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa44096c10 auth_method 0 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:21.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:21.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:21.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:01:21.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:21.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:21.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:21.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:22.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa43fff640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.67:6789/0 conn(0x7faa4409d3b0 legacy=0x7faa44096fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:22.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4901d640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 msgr2=0x7faa44135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:22.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4901d640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:01:22.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:22.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa44096c10 auth_method 0 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:22.425+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:01:24.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): tick 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 monclient(hunting): continuing hunt 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 msgr2=0x7faa44099f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa44096c10 0x7faa44099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 msgr2=0x7faa44135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.67:3300/0 conn(0x7faa440976f0 0x7faa44135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.67:6789/0 conn(0x7faa4409d3b0 legacy=0x7faa44096fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient: _add_conns ranks=[3,0] 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): picked mon.noname-a con 0x7faa24005870 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa240083d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 -- 172.21.15.17:0/2713097597 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4409a5d0 con 0x7faa240083d0 2024-09-15T02:01:24.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 1 -- 172.21.15.17:0/2713097597 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa44099260 con 0x7faa24005870 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa427fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa24005870 auth_method 0 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:24.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.022+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:24.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.024+0000 7faa43fff640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.17:6789/0 conn(0x7faa240083d0 legacy=0x7faa2400a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:24.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:24.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa24005870 auth_method 0 2024-09-15T02:01:24.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:24.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.223+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:24.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:24.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa24005870 auth_method 0 2024-09-15T02:01:24.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:24.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:24.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:24.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:01:24.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:01:24.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:24.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:24.624+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180364487 0 0) 0x7faa4409a5d0 con 0x7faa240083d0 2024-09-15T02:01:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient(hunting): my global_id is 104922 2024-09-15T02:01:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa300034e0 con 0x7faa240083d0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816905462 0 0) 0x7faa300034e0 con 0x7faa240083d0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 msgr2=0x7faa24007ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 --2- 172.21.15.17:0/2713097597 >> v2:172.21.15.17:3300/0 conn(0x7faa24005870 0x7faa24007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa440983b0 con 0x7faa240083d0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7faa340044f0 con 0x7faa240083d0 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa4aa80640 5 monclient: authenticate success, global_id 104922 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa4aa80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.202+0000 7faa437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient: dump: 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 >> v1:172.21.15.17:6789/0 conn(0x7faa240083d0 legacy=0x7faa2400a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4901d640 1 -- 172.21.15.17:0/2713097597 reap_dead start 2024-09-15T02:01:25.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): picked mon.c con 0x7faa300039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa30004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): picked mon.a con 0x7faa30004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): picked mon.b con 0x7faa30005180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa44099260 con 0x7faa30004780 2024-09-15T02:01:25.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3000c120 con 0x7faa30005180 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3000c2e0 con 0x7faa300039d0 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa437fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4aa80640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa30004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4901d640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 msgr2=0x7faa3000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa300039d0 auth_method 0 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4901d640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa3000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): get_auth_request con 0x7faa30004780 auth_method 0 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient(hunting): handle_auth_done global_id 104922 payload 293 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 msgr2=0x7faa3000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa3000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 msgr2=0x7faa30004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa30004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 monclient: found mon.c 2024-09-15T02:01:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa300034e0 con 0x7faa300039d0 2024-09-15T02:01:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.203+0000 7faa4881c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.206+0000 7faa4881c640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 msgr2=0x7faa30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.206+0000 7faa4881c640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa30003dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faa2c006f60 tx=0x7faa2c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b606237941f03586 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02c680 con 0x7faa300039d0 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: dump: 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c030070 con 0x7faa300039d0 2024-09-15T02:01:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 1 -- 172.21.15.17:0/2713097597 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02ca50 con 0x7faa300039d0 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 4 set_mon_vals no callback set 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa437fe640 10 monclient: dump: 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.223 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 10 monclient: shutdown 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 -- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 msgr2=0x7faa30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa30003dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faa2c006f60 tx=0x7faa2c008040 comp rx=0 tx=0).stop 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 -- 172.21.15.17:0/2713097597 shutdown_connections 2024-09-15T02:01:25.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa30004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa3000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 --2- 172.21.15.17:0/2713097597 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa30003dc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 -- 172.21.15.17:0/2713097597 >> 172.21.15.17:0/2713097597 conn(0x7faa440a1000 msgr2=0x7faa440a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.207+0000 7faa4aa80640 1 -- 172.21.15.17:0/2713097597 shutdown_connections 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.208+0000 7faa4aa80640 1 -- 172.21.15.17:0/2713097597 wait complete. 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.208+0000 7faa4aa80640 10 monclient: build_initial_monmap 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.208+0000 7faa4aa80640 10 monclient: monmap: 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:25.225 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.208+0000 7faa4aa80640 1 Processor -- start 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 -- start start 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient: init 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-c con 0x7faa300039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa44138b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-a con 0x7faa30004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa4413ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): picked mon.noname-b con 0x7faa30005180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4411e490 con 0x7faa30004780 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4411ded0 con 0x7faa30005180 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4411b9f0 con 0x7faa300039d0 2024-09-15T02:01:25.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.209+0000 7faa4aa80640 10 monclient(hunting): authenticate will time out at 1345.205977s 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4901d640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 msgr2=0x7faa4413ca10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa44138b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44486/0 (socket says 172.21.15.17:44486) 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 -- 172.21.15.17:0/3566840306 learned_addr learned my addr 172.21.15.17:0/3566840306 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): get_auth_request con 0x7faa300039d0 auth_method 0 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4901d640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa4413ca10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:25.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): get_auth_request con 0x7faa30004780 auth_method 0 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient(hunting): handle_auth_done global_id 134156 payload 824 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 -- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 msgr2=0x7faa4413ca10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa4413ca10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 -- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 msgr2=0x7faa44138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa44138b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 monclient: found mon.noname-c 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa4881c640 1 -- 172.21.15.17:0/3566840306 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4411dbf0 con 0x7faa300039d0 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.210+0000 7faa43fff640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa44138b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.212+0000 7faa4881c640 1 -- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 msgr2=0x7faa441394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.212+0000 7faa4881c640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c350 tx=0x7faa2c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c036920 con 0x7faa300039d0 2024-09-15T02:01:25.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: dump: 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c00c040 con 0x7faa300039d0 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 5 monclient: authenticate success, global_id 134156 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 10 monclient: _renew_subs 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4411e1b0 con 0x7faa300039d0 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c036cf0 con 0x7faa300039d0 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa40ff9640 10 monclient: dump: 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa42ffd640 4 set_mon_vals no callback set 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 10 monclient: _renew_subs 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.213+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa44139230 con 0x7faa300039d0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faa2c041690 con 0x7faa300039d0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa40ff9640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 0x7faa2403d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faa2c02ce40 con 0x7faa300039d0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa4aa80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa4aa80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa4aa80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa440724a0 con 0x7faa300039d0 2024-09-15T02:01:25.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa43fff640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 0x7faa2403d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.214+0000 7faa43fff640 10 monclient: get_auth_request con 0x7faa2403afd0 auth_method 0 2024-09-15T02:01:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.217+0000 7faa43fff640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 0x7faa2403d480 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faa340046a0 tx=0x7faa34006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.219+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faa2c041e60 con 0x7faa300039d0 2024-09-15T02:01:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.219+0000 7faa40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.219+0000 7faa40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.447+0000 7faa4aa80640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.447+0000 7faa4aa80640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.447+0000 7faa4aa80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.447+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faa44067290 con 0x7faa300039d0 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.448+0000 7faa40ff9640 1 -- 172.21.15.17:0/3566840306 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7faa2c059610 con 0x7faa300039d0 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.448+0000 7faa40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.448+0000 7faa40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:25.450 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 msgr2=0x7faa2403d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 0x7faa2403d480 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faa340046a0 tx=0x7faa34006040 comp rx=0 tx=0).stop 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 10 monclient: shutdown 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 msgr2=0x7faa441394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c350 tx=0x7faa2c037000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 shutdown_connections 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa2403afd0 0x7faa2403d480 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa30004780 0x7faa44138b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa30005180 0x7faa4413ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 --2- 172.21.15.17:0/3566840306 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa300039d0 0x7faa441394a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 >> 172.21.15.17:0/3566840306 conn(0x7faa440a1000 msgr2=0x7faa4409f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 shutdown_connections 2024-09-15T02:01:25.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.450+0000 7faa4aa80640 1 -- 172.21.15.17:0/3566840306 wait complete. 2024-09-15T02:01:25.463 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:01:25.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:01:25.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:01:25.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 10 monclient: get_monmap_and_config 2024-09-15T02:01:25.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 10 monclient: build_initial_monmap 2024-09-15T02:01:25.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 10 monclient: monmap: 2024-09-15T02:01:25.562 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:25.562 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.562 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:25.559995+0000 2024-09-15T02:01:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:25.559995+0000 2024-09-15T02:01:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:25.564 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:25.564 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:25.564 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:25.564 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 1 Processor -- start 2024-09-15T02:01:25.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 1 -- start start 2024-09-15T02:01:25.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.559+0000 7f3b59c17640 10 monclient: init 2024-09-15T02:01:25.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-b con 0x7f3b540708a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 0x7f3b5409b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-c con 0x7f3b5406a510 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:25.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3b5409b8a0 0x7f3b54131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5409b8a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5409da40 con 0x7f3b5409b8a0 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5409bca0 con 0x7f3b5406a510 2024-09-15T02:01:25.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54070c70 con 0x7f3b540708a0 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b59c17640 10 monclient(hunting): authenticate will time out at 1345.556424s 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3b5409b8a0 0x7f3b54131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44498/0 (socket says 172.21.15.17:44498) 2024-09-15T02:01:25.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3b5409b8a0 0x7f3b54131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47990/0 (socket says 172.21.15.17:47990) 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b52ffd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 msgr2=0x7f3b5409b360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 1 -- 172.21.15.17:0/3338152640 learned_addr learned my addr 172.21.15.17:0/3338152640 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): get_auth_request con 0x7f3b540708a0 auth_method 0 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/3338152640 >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 0x7f3b5409b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:25.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): get_auth_request con 0x7f3b5409b8a0 auth_method 0 2024-09-15T02:01:25.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient(hunting): handle_auth_done global_id 104928 payload 274 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 -- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 msgr2=0x7f3b5409ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 -- 172.21.15.17:0/3338152640 >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 msgr2=0x7f3b5409b360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:25.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 --2- 172.21.15.17:0/3338152640 >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 0x7f3b5409b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 monclient: found mon.noname-a 2024-09-15T02:01:25.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:25.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b53fff640 1 -- 172.21.15.17:0/3338152640 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54070f30 con 0x7f3b5409b8a0 2024-09-15T02:01:25.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.560+0000 7f3b537fe640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:25.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b53fff640 1 -- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 msgr2=0x7f3b54131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b53fff640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54131ef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406f450 tx=0x7f3b4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9294b72b78ba96f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 1 -- 172.21.15.17:0/3338152640 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c007760 con 0x7f3b5409b8a0 2024-09-15T02:01:25.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:25.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: dump: 2024-09-15T02:01:25.595 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.595 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.595 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.595 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.595 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.596 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.596 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.596 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.596 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.596 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 1 -- 172.21.15.17:0/3338152640 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c02d050 con 0x7f3b5409b8a0 2024-09-15T02:01:25.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b59c17640 5 monclient: authenticate success, global_id 104928 2024-09-15T02:01:25.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b59c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:25.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 1 -- 172.21.15.17:0/3338152640 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c007af0 con 0x7f3b5409b8a0 2024-09-15T02:01:25.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:25.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b527fc640 10 monclient: dump: 2024-09-15T02:01:25.599 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.599 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.599 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.601 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b59c17640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:25.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.563+0000 7f3b59c17640 4 set_mon_vals no callback set 2024-09-15T02:01:25.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 10 monclient: shutdown 2024-09-15T02:01:25.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 -- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 msgr2=0x7f3b54131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54131ef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406f450 tx=0x7f3b4c009780 comp rx=0 tx=0).stop 2024-09-15T02:01:25.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 -- 172.21.15.17:0/3338152640 shutdown_connections 2024-09-15T02:01:25.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 --2- 172.21.15.17:0/3338152640 >> v2:172.21.15.67:3300/0 conn(0x7f3b5406a510 0x7f3b5409b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54131ef0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 --2- 172.21.15.17:0/3338152640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b540708a0 0x7f3b5409ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 -- 172.21.15.17:0/3338152640 >> 172.21.15.17:0/3338152640 conn(0x7f3b54067ae0 msgr2=0x7f3b540a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 -- 172.21.15.17:0/3338152640 shutdown_connections 2024-09-15T02:01:25.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 1 -- 172.21.15.17:0/3338152640 wait complete. 2024-09-15T02:01:25.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 10 monclient: build_initial_monmap 2024-09-15T02:01:25.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.564+0000 7f3b59c17640 10 monclient: monmap: 2024-09-15T02:01:25.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:25.605 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:25.605 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:25.605 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:25.605 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:25.605 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:25.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:25.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:25.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 Processor -- start 2024-09-15T02:01:25.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 -- start start 2024-09-15T02:01:25.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient: init 2024-09-15T02:01:25.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 0x7f3b5413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-c con 0x7f3b5406a510 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b540708a0 0x7f3b5413cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-b con 0x7f3b540708a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5409b8a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b541226d0 con 0x7f3b5409b8a0 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54122110 con 0x7f3b540708a0 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5411fc30 con 0x7f3b5406a510 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b59c17640 10 monclient(hunting): authenticate will time out at 1345.561864s 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 0x7f3b5413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b53fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54140c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 0x7f3b5413d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44512/0 (socket says 172.21.15.17:44512) 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 1 -- 172.21.15.17:0/1180619579 learned_addr learned my addr 172.21.15.17:0/1180619579 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:25.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 10 monclient(hunting): get_auth_request con 0x7f3b5406a510 auth_method 0 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b53fff640 10 monclient(hunting): get_auth_request con 0x7f3b5409b8a0 auth_method 0 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.565+0000 7f3b52ffd640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b540708a0 msgr2=0x7f3b5413cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b540708a0 0x7f3b5413cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient(hunting): handle_auth_done global_id 134162 payload 824 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b540708a0 msgr2=0x7f3b5413cd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:25.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b540708a0 0x7f3b5413cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 msgr2=0x7f3b54140c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5409b8a0 0x7f3b54140c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 monclient: found mon.noname-c 2024-09-15T02:01:25.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.566+0000 7f3b537fe640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b44009450 con 0x7f3b5406a510 2024-09-15T02:01:25.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b537fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 msgr2=0x7f3b5413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b537fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 0x7f3b5413d6e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400ecb0 tx=0x7f3b440091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44009730 con 0x7f3b5406a510 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: dump: 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.616 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4400cdd0 con 0x7f3b5406a510 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 5 monclient: authenticate success, global_id 134162 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 10 monclient: _renew_subs 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c009a20 con 0x7f3b5406a510 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b440136a0 con 0x7f3b5406a510 2024-09-15T02:01:25.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b3b7fe640 10 monclient: dump: 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 10 monclient: _renew_subs 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b51ffb640 4 set_mon_vals no callback set 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.568+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5413d2d0 con 0x7f3b5406a510 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b59c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b59c17640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 0x7f3b54141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54122590 con 0x7f3b54120f60 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3b44013a50 con 0x7f3b5406a510 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 0x7f3b54141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient: get_auth_request con 0x7f3b54120f60 auth_method 0 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b3b7fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b40038740 0x7f3b4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.569+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3b4404ca70 con 0x7f3b5406a510 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.570+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_done global_id 104934 payload 824 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.570+0000 7f3b52ffd640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b54121fa0 con 0x7f3b54120f60 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.570+0000 7f3b53fff640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b40038740 0x7f3b4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.570+0000 7f3b53fff640 10 monclient: get_auth_request con 0x7f3b40038740 auth_method 0 2024-09-15T02:01:25.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b52ffd640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 msgr2=0x7f3b54141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 0x7f3b54141190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b480050f0 con 0x7f3b54120f60 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 10 monclient: dump: 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3b48005250 con 0x7f3b54120f60 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 msgr2=0x7f3b54141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 0x7f3b54141190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b480091a0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 msgr2=0x7f3b54141190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b3b7fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b54120f60 0x7f3b54141190 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b480091a0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.572+0000 7f3b53fff640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b40038740 0x7f3b4003abf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406a9a0 tx=0x7f3b4c037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b59c17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b59c17640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 0x7f3b541416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b59c17640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54063b00 con 0x7f3b5413e060 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 0x7f3b541416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient: get_auth_request con 0x7f3b5413e060 auth_method 0 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_done global_id 104937 payload 824 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.578+0000 7f3b52ffd640 1 -- 172.21.15.17:0/1180619579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b540724a0 con 0x7f3b5413e060 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b52ffd640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 msgr2=0x7f3b541416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b52ffd640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 0x7f3b541416d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b48015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48010d50 con 0x7f3b5413e060 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 10 monclient: dump: 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f3b48019400 con 0x7f3b5413e060 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 msgr2=0x7f3b541416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 0x7f3b541416d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b48015000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 msgr2=0x7f3b541416d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b3b7fe640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b5413e060 0x7f3b541416d0 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3b5413e750 tx=0x7f3b48015000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.625 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 114, 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:25.626 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:25.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:25.629 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.630 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.632 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.637 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.638 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:25.639 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b40038740 msgr2=0x7f3b4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3b40038740 0x7f3b4003abf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406a9a0 tx=0x7f3b4c037000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 10 monclient: shutdown 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 msgr2=0x7f3b5413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 --2- 172.21.15.17:0/1180619579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3b5406a510 0x7f3b5413d6e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400ecb0 tx=0x7f3b440091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b53fff640 1 -- 172.21.15.17:0/1180619579 reap_dead start 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b53fff640 1 -- 172.21.15.17:0/1180619579 reap_dead start 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 shutdown_connections 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 >> 172.21.15.17:0/1180619579 conn(0x7f3b54067ae0 msgr2=0x7f3b5406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 shutdown_connections 2024-09-15T02:01:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.581+0000 7f3b59c17640 1 -- 172.21.15.17:0/1180619579 wait complete. 2024-09-15T02:01:25.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:01:25.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:01:25.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:01:25.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.737+0000 7f140fe99640 10 monclient: get_monmap_and_config 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.737+0000 7f140fe99640 10 monclient: build_initial_monmap 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.737+0000 7f140fe99640 10 monclient: monmap: 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:25.738571+0000 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:25.738571+0000 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:25.740 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:25.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:25.741 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:25.741 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:25.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 Processor -- start 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 -- start start 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient: init 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-b con 0x7f140809a9f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f140812e420 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f140812e810 0x7f140809bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-a con 0x7f140812e810 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1408098fb0 con 0x7f140812e420 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.738+0000 7f140fe99640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140812e130 con 0x7f140812e810 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140fe99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140809b300 con 0x7f140809a9f0 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140fe99640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140fe99640 10 monclient(hunting): authenticate will time out at 1345.735017s 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f140812e810 0x7f140809bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140d434640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f140812e420 legacy=0x7f140809b850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f140812e810 0x7f140809bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48024/0 (socket says 172.21.15.17:48024) 2024-09-15T02:01:25.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140809adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44518/0 (socket says 172.21.15.17:44518) 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 -- 172.21.15.17:0/2257282504 learned_addr learned my addr 172.21.15.17:0/2257282504 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): get_auth_request con 0x7f140812e810 auth_method 0 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): get_auth_request con 0x7f140809a9f0 auth_method 0 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient(hunting): handle_auth_done global_id 104940 payload 274 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 -- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 msgr2=0x7f140809adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 --2- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140809adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 -- 172.21.15.17:0/2257282504 >> v1:172.21.15.67:6789/0 conn(0x7f140812e420 legacy=0x7f140809b850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 monclient: found mon.noname-a 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.739+0000 7f140e436640 1 -- 172.21.15.17:0/2257282504 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408098cd0 con 0x7f140812e810 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140e436640 1 -- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e810 msgr2=0x7f140809bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140e436640 1 --2- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e810 0x7f140809bf60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1400009f20 tx=0x7f1400009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87ffecbb46589f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 1 -- 172.21.15.17:0/2257282504 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400007610 con 0x7f140812e810 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:25.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: dump: 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 1 -- 172.21.15.17:0/2257282504 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140002d050 con 0x7f140812e810 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 5 monclient: authenticate success, global_id 104940 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 1 -- 172.21.15.17:0/2257282504 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000079a0 con 0x7f140812e810 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140cc33640 10 monclient: dump: 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.754 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 4 set_mon_vals no callback set 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 10 monclient: shutdown 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 -- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e810 msgr2=0x7f140809bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 --2- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e810 0x7f140809bf60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1400009f20 tx=0x7f1400009840 comp rx=0 tx=0).stop 2024-09-15T02:01:25.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 -- 172.21.15.17:0/2257282504 shutdown_connections 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 --2- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e810 0x7f140809bf60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 --2- 172.21.15.17:0/2257282504 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140809adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.742+0000 7f140fe99640 1 -- 172.21.15.17:0/2257282504 >> 172.21.15.17:0/2257282504 conn(0x7f14080a1000 msgr2=0x7f14080a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 1 -- 172.21.15.17:0/2257282504 shutdown_connections 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 1 -- 172.21.15.17:0/2257282504 wait complete. 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 10 monclient: build_initial_monmap 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 10 monclient: monmap: 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:25.756 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 1 Processor -- start 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 1 -- start start 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.743+0000 7f140fe99640 10 monclient: init 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140806e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-c con 0x7f140809a9f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e420 0x7f1408127c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-a con 0x7f140812e420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f140812e810 0x7f1408128150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): picked mon.noname-b con 0x7f140812e810 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140811be00 con 0x7f140812e420 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408142140 con 0x7f140812e810 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140809eee0 con 0x7f140809a9f0 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140fe99640 10 monclient(hunting): authenticate will time out at 1345.740436s 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e420 0x7f1408127c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44528/0 (socket says 172.21.15.17:44528) 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 -- 172.21.15.17:0/877188741 learned_addr learned my addr 172.21.15.17:0/877188741 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): get_auth_request con 0x7f140809a9f0 auth_method 0 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): get_auth_request con 0x7f140812e420 auth_method 0 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140e436640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f140812e810 msgr2=0x7f1408128150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140e436640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f140812e810 0x7f1408128150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient(hunting): handle_auth_done global_id 134168 payload 824 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f140812e810 msgr2=0x7f1408128150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f140812e810 0x7f1408128150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e420 msgr2=0x7f1408127c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e420 0x7f1408127c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient: _finish_hunting 0 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 monclient: found mon.noname-c 2024-09-15T02:01:25.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140dc35640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400009ae0 con 0x7f140809a9f0 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.744+0000 7f140d434640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f140812e420 0x7f1408127c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140dc35640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 msgr2=0x7f140806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140dc35640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140806e930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f140400ea50 tx=0x7f14040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404009600 con 0x7f140809a9f0 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: dump: 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: _finish_auth 0 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140400cc20 con 0x7f140809a9f0 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404013470 con 0x7f140809a9f0 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 5 monclient: authenticate success, global_id 134168 2024-09-15T02:01:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fdffb640 10 monclient: dump: 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 10 monclient: _renew_subs 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f13fffff640 4 set_mon_vals no callback set 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140811c080 con 0x7f140809a9f0 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 10 monclient: _renew_subs 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.747+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140812d330 con 0x7f140809a9f0 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140fe99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140fe99640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 0x7f140812c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140811c500 con 0x7f1408141750 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1404013860 con 0x7f140809a9f0 2024-09-15T02:01:25.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 0x7f140812c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient: get_auth_request con 0x7f1408141750 auth_method 0 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f13fdffb640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f13ec0386f0 0x7f13ec03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 10 monclient(hunting): handle_auth_done global_id 134171 payload 824 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f140d434640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14080724a0 con 0x7f1408141750 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.748+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f140404c920 con 0x7f140809a9f0 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.749+0000 7f140e436640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f13ec0386f0 0x7f13ec03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.749+0000 7f140e436640 10 monclient: get_auth_request con 0x7f13ec0386f0 auth_method 0 2024-09-15T02:01:25.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f140d434640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 msgr2=0x7f140812c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f140d434640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 0x7f140812c300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f800ee00 con 0x7f1408141750 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 10 monclient: dump: 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f13f80123f0 con 0x7f1408141750 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 msgr2=0x7f140812c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 0x7f140812c300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f80090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 msgr2=0x7f140812c300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f13fdffb640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1408141750 0x7f140812c300 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f80090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.751+0000 7f140e436640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f13ec0386f0 0x7f13ec03aba0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1408068cc0 tx=0x7f1400036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140fe99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:25.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140fe99640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 0x7f1408029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1408063c20 con 0x7f14080298c0 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 0x7f1408029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient: get_auth_request con 0x7f14080298c0 auth_method 0 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 10 monclient(hunting): handle_auth_done global_id 134174 payload 824 2024-09-15T02:01:25.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.757+0000 7f140d434640 1 -- 172.21.15.17:0/877188741 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f140809d6c0 con 0x7f14080298c0 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.759+0000 7f140d434640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 msgr2=0x7f1408029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.759+0000 7f140d434640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 0x7f1408029cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.759+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8025070 con 0x7f14080298c0 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 10 monclient: dump: 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:25.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4385 (secure 0 0 0) 0x7f13f801b450 con 0x7f14080298c0 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 msgr2=0x7f1408029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 0x7f1408029cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f8015000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 msgr2=0x7f1408029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:25.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f13fdffb640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f14080298c0 0x7f1408029cc0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f140812d270 tx=0x7f13f8015000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 114, 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:25.787 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:25.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f13ec0386f0 msgr2=0x7f13ec03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f13ec0386f0 0x7f13ec03aba0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1408068cc0 tx=0x7f1400036000 comp rx=0 tx=0).stop 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 10 monclient: shutdown 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 msgr2=0x7f140806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 --2- 172.21.15.17:0/877188741 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f140809a9f0 0x7f140806e930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f140400ea50 tx=0x7f14040090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140e436640 1 -- 172.21.15.17:0/877188741 reap_dead start 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140e436640 1 -- 172.21.15.17:0/877188741 reap_dead start 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 shutdown_connections 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 >> 172.21.15.17:0/877188741 conn(0x7f14080a1000 msgr2=0x7f140812a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 shutdown_connections 2024-09-15T02:01:25.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:25.760+0000 7f140fe99640 1 -- 172.21.15.17:0/877188741 wait complete. 2024-09-15T02:01:25.790 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:01:25.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:01:45.791 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:01:45.791 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:01:45.791 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:01:45.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:01:45.795 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:01:45.795 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:01:45.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.901+0000 7ffbc5a80640 10 monclient: get_monmap_and_config 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.901+0000 7ffbc5a80640 10 monclient: build_initial_monmap 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.901+0000 7ffbc5a80640 10 monclient: monmap: 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:45.903 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:45.902015+0000 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:45.902015+0000 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:45.904 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.901+0000 7ffbc5a80640 1 Processor -- start 2024-09-15T02:01:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 1 -- start start 2024-09-15T02:01:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient: init 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ffbc0068f20 0x7ffbc01321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-a con 0x7ffbc0068f20 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbc00692f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbc0136fe0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc00a1950 con 0x7ffbc00692f0 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc00a1670 con 0x7ffbc0136fe0 2024-09-15T02:01:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc00a0020 con 0x7ffbc0068f20 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbc5a80640 10 monclient(hunting): authenticate will time out at 1365.898560s 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ffbc0068f20 0x7ffbc01321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbb6ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ffbc00692f0 0x7ffbc00696f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44164/0 (socket says 172.21.15.17:44164) 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbb6ffd640 1 -- 172.21.15.17:0/2227777448 learned_addr learned my addr 172.21.15.17:0/2227777448 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ffbc0068f20 0x7ffbc01321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41028/0 (socket says 172.21.15.17:41028) 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): get_auth_request con 0x7ffbc0068f20 auth_method 0 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_done global_id 104946 payload 274 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.902+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/2227777448 >> v1:172.21.15.67:6789/0 conn(0x7ffbc0136fe0 legacy=0x7ffbc01393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/2227777448 >> v1:172.21.15.17:6789/0 conn(0x7ffbc00692f0 legacy=0x7ffbc00696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.903+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/2227777448 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00a1390 con 0x7ffbc0068f20 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.905+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/2227777448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 msgr2=0x7ffbc01321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.905+0000 7ffbbf7fe640 1 --2- 172.21.15.17:0/2227777448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc01321c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00098b0 tx=0x7ffbb00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e9ed0893edad5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 1 -- 172.21.15.17:0/2227777448 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb002ae00 con 0x7ffbc0068f20 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: dump: 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:45.915 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: _finish_auth 0 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 1 -- 172.21.15.17:0/2227777448 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb002f4b0 con 0x7ffbc0068f20 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 5 monclient: authenticate success, global_id 104946 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:45.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 1 -- 172.21.15.17:0/2227777448 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb002f850 con 0x7ffbc0068f20 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 4 set_mon_vals no callback set 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbbeffd640 10 monclient: dump: 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 10 monclient: shutdown 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 -- 172.21.15.17:0/2227777448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 msgr2=0x7ffbc01321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/2227777448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc01321c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00098b0 tx=0x7ffbb00090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 -- 172.21.15.17:0/2227777448 shutdown_connections 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/2227777448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc01321c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 -- 172.21.15.17:0/2227777448 >> 172.21.15.17:0/2227777448 conn(0x7ffbc0070fb0 msgr2=0x7ffbc00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 -- 172.21.15.17:0/2227777448 shutdown_connections 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.906+0000 7ffbc5a80640 1 -- 172.21.15.17:0/2227777448 wait complete. 2024-09-15T02:01:45.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.907+0000 7ffbc5a80640 10 monclient: build_initial_monmap 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.907+0000 7ffbc5a80640 10 monclient: monmap: 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.907+0000 7ffbc5a80640 1 Processor -- start 2024-09-15T02:01:45.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.907+0000 7ffbc5a80640 1 -- start start 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.907+0000 7ffbc5a80640 10 monclient: init 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-a con 0x7ffbc0068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 0x7ffbc0099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-c con 0x7ffbc00692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): picked mon.noname-b con 0x7ffbc0136fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc00a0750 con 0x7ffbc0068f20 2024-09-15T02:01:45.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc009cab0 con 0x7ffbc0136fe0 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc011cd00 con 0x7ffbc00692f0 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbc5a80640 10 monclient(hunting): authenticate will time out at 1365.904363s 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 0x7ffbc0099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41044/0 (socket says 172.21.15.17:41044) 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 0x7ffbc0099be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56172/0 (socket says 172.21.15.17:56172) 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/1758393252 learned_addr learned my addr 172.21.15.17:0/1758393252 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): get_auth_request con 0x7ffbc0068f20 auth_method 0 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:45.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): get_auth_request con 0x7ffbc00692f0 auth_method 0 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:45.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 10 monclient(hunting): get_auth_request con 0x7ffbc0136fe0 auth_method 0 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient(hunting): handle_auth_done global_id 104949 payload 824 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 msgr2=0x7ffbc0099be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 0x7ffbc0099be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 msgr2=0x7ffbc009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 msgr2=0x7ffbc009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 msgr2=0x7ffbc009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbffff640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:45.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.908+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/1758393252 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0009390 con 0x7ffbc0068f20 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbbf7fe640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 msgr2=0x7ffbc0130910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbbf7fe640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00098b0 tx=0x7ffbb0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0032a30 con 0x7ffbc0068f20 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: dump: 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:45.924 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: _finish_auth 0 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 5 monclient: authenticate success, global_id 104949 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb0032bd0 con 0x7ffbc0068f20 2024-09-15T02:01:45.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 10 monclient: _renew_subs 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc011c730 con 0x7ffbc0068f20 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb003b400 con 0x7ffbc0068f20 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbb7fff640 10 monclient: dump: 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:45.926 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 10 monclient: _renew_subs 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbbe7fc640 4 set_mon_vals no callback set 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.911+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc009b770 con 0x7ffbc0068f20 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffbb0039040 con 0x7ffbc0068f20 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbc5a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbc5a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:45.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbc5a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc00724a0 con 0x7ffbc0068f20 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbb7fff640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 0x7ffba003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ffbb006c580 con 0x7ffbc0068f20 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbb6ffd640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 0x7ffba003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.912+0000 7ffbb6ffd640 10 monclient: get_auth_request con 0x7ffba0038770 auth_method 0 2024-09-15T02:01:45.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.915+0000 7ffbb6ffd640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 0x7ffba003ac20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffbc009bfb0 tx=0x7ffba8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:45.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.917+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffbb0039ad0 con 0x7ffbc0068f20 2024-09-15T02:01:45.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.917+0000 7ffbb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:45.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:45.917+0000 7ffbb7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbc5a80640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbc5a80640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbc5a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffbc00724a0 con 0x7ffbc0068f20 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbb7fff640 1 -- 172.21.15.17:0/1758393252 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7ffbb0037020 con 0x7ffbc0068f20 2024-09-15T02:01:46.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:46.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.147+0000 7ffbb7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:46.150 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:46.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 msgr2=0x7ffba003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 0x7ffba003ac20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffbc009bfb0 tx=0x7ffba8007f00 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 10 monclient: shutdown 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 msgr2=0x7ffbc0130910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00098b0 tx=0x7ffbb0032000 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 shutdown_connections 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffba0038770 0x7ffba003ac20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc0136fe0 0x7ffbc009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffbc00692f0 0x7ffbc0099be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 --2- 172.21.15.17:0/1758393252 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffbc0068f20 0x7ffbc0130910 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 >> 172.21.15.17:0/1758393252 conn(0x7ffbc0070fb0 msgr2=0x7ffbc009e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:46.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.149+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 shutdown_connections 2024-09-15T02:01:46.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:46.150+0000 7ffbc5a80640 1 -- 172.21.15.17:0/1758393252 wait complete. 2024-09-15T02:01:49.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 10 monclient: get_monmap_and_config 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 10 monclient: build_initial_monmap 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 10 monclient: monmap: 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:49.270943+0000 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:49.270943+0000 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:49.273 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 1 Processor -- start 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 1 -- start start 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.270+0000 7fd5c7217640 10 monclient: init 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5c0131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5c00a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 0x7fd5c013a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c0137e60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c00a0040 con 0x7fd5c00a0280 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c009bff0 con 0x7fd5c0131b20 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:49.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c00a14a0 con 0x7fd5c0137e60 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c7217640 10 monclient(hunting): authenticate will time out at 1369.267474s 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 0x7fd5c013a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5bffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd5c00a0280 0x7fd5c0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44176/0 (socket says 172.21.15.17:44176) 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5bffff640 1 -- 172.21.15.17:0/3568733413 learned_addr learned my addr 172.21.15.17:0/3568733413 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): get_auth_request con 0x7fd5c0137e60 auth_method 0 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:49.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient(hunting): handle_auth_done global_id 134180 payload 274 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 1 -- 172.21.15.17:0/3568733413 >> v1:172.21.15.67:6789/0 conn(0x7fd5c0131b20 legacy=0x7fd5c009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 1 -- 172.21.15.17:0/3568733413 >> v1:172.21.15.17:6789/0 conn(0x7fd5c00a0280 legacy=0x7fd5c0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient: _finish_hunting 0 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 1 monclient: found mon.noname-b 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:49.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.271+0000 7fd5c57b4640 1 -- 172.21.15.17:0/3568733413 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c009f3b0 con 0x7fd5c0137e60 2024-09-15T02:01:49.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.274+0000 7fd5c57b4640 1 -- 172.21.15.17:0/3568733413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 msgr2=0x7fd5c013a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:49.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.274+0000 7fd5c57b4640 1 --2- 172.21.15.17:0/3568733413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 0x7fd5c013a240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009e90 tx=0x7fd5b8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d09b22d3afadb21a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 1 -- 172.21.15.17:0/3568733413 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8007840 con 0x7fd5c0137e60 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: dump: 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:49.281 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 1 -- 172.21.15.17:0/3568733413 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b802d050 con 0x7fd5c0137e60 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 5 monclient: authenticate success, global_id 134180 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 1 -- 172.21.15.17:0/3568733413 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8007bd0 con 0x7fd5c0137e60 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 4 set_mon_vals no callback set 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5bf7fe640 10 monclient: dump: 2024-09-15T02:01:49.282 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 10 monclient: shutdown 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 -- 172.21.15.17:0/3568733413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 msgr2=0x7fd5c013a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 --2- 172.21.15.17:0/3568733413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 0x7fd5c013a240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009e90 tx=0x7fd5b8009780 comp rx=0 tx=0).stop 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 -- 172.21.15.17:0/3568733413 shutdown_connections 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 --2- 172.21.15.17:0/3568733413 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0137e60 0x7fd5c013a240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 -- 172.21.15.17:0/3568733413 >> 172.21.15.17:0/3568733413 conn(0x7fd5c0071270 msgr2=0x7fd5c00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 -- 172.21.15.17:0/3568733413 shutdown_connections 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.275+0000 7fd5c7217640 1 -- 172.21.15.17:0/3568733413 wait complete. 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.276+0000 7fd5c7217640 10 monclient: build_initial_monmap 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.276+0000 7fd5c7217640 10 monclient: monmap: 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.284 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.276+0000 7fd5c7217640 1 Processor -- start 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.276+0000 7fd5c7217640 1 -- start start 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.276+0000 7fd5c7217640 10 monclient: init 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 0x7fd5c012e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-a con 0x7fd5c00a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 0x7fd5c012ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-c con 0x7fd5c0131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:49.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c0137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c009f070 con 0x7fd5c00a0280 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c012d710 con 0x7fd5c0137e60 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c012d9f0 con 0x7fd5c0131b20 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c7217640 10 monclient(hunting): authenticate will time out at 1369.273299s 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 0x7fd5c012ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 0x7fd5c012e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 0x7fd5c012ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56194/0 (socket says 172.21.15.17:56194) 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 1 -- 172.21.15.17:0/2198949701 learned_addr learned my addr 172.21.15.17:0/2198949701 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:49.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): get_auth_request con 0x7fd5c00a0280 auth_method 0 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): get_auth_request con 0x7fd5c0131b20 auth_method 0 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:49.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 10 monclient(hunting): get_auth_request con 0x7fd5c0137e60 auth_method 0 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient(hunting): handle_auth_done global_id 104952 payload 824 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 msgr2=0x7fd5c012b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 msgr2=0x7fd5c012b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 msgr2=0x7fd5c012ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 0x7fd5c012ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c57b4640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 msgr2=0x7fd5c012b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:49.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient: _finish_hunting 0 2024-09-15T02:01:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 monclient: found mon.noname-a 2024-09-15T02:01:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:49.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.277+0000 7fd5c4fb3640 1 -- 172.21.15.17:0/2198949701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8009a20 con 0x7fd5c00a0280 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c4fb3640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 msgr2=0x7fd5c012e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c4fb3640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 0x7fd5c012e3d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000c4e0 tx=0x7fd5b0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00117a0 con 0x7fd5c00a0280 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: dump: 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.291 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: _finish_auth 0 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b0011920 con 0x7fd5c00a0280 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0011ce0 con 0x7fd5c00a0280 2024-09-15T02:01:49.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 5 monclient: authenticate success, global_id 104952 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 10 monclient: _renew_subs 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c012fa00 con 0x7fd5c00a0280 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5bcff9640 10 monclient: dump: 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:49.293 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5beffd640 4 set_mon_vals no callback set 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 10 monclient: _renew_subs 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.280+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c012c7b0 con 0x7fd5c00a0280 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd5b0012630 con 0x7fd5c00a0280 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5c7217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5c7217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5c7217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5bcff9640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 0x7fd5ac03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c00724a0 con 0x7fd5c00a0280 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.281+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd5b004c1e0 con 0x7fd5c00a0280 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.282+0000 7fd5bffff640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 0x7fd5ac03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.282+0000 7fd5bffff640 10 monclient: get_auth_request con 0x7fd5ac038790 auth_method 0 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.284+0000 7fd5bffff640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 0x7fd5ac03ac40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd5c00a2580 tx=0x7fd5b4004960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.286+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd5b001c460 con 0x7fd5c00a0280 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.286+0000 7fd5bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:49.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.286+0000 7fd5bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:49.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.510+0000 7fd5c7217640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.510+0000 7fd5c7217640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.510+0000 7fd5c7217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.510+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd5c012dc10 con 0x7fd5c00a0280 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.511+0000 7fd5bcff9640 1 -- 172.21.15.17:0/2198949701 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7fd5b001beb0 con 0x7fd5c00a0280 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.511+0000 7fd5bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.511+0000 7fd5bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:49.513 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 msgr2=0x7fd5ac03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 0x7fd5ac03ac40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd5c00a2580 tx=0x7fd5b4004960 comp rx=0 tx=0).stop 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 10 monclient: shutdown 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 msgr2=0x7fd5c012e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 0x7fd5c012e3d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000c4e0 tx=0x7fd5b0007e10 comp rx=0 tx=0).stop 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 shutdown_connections 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd5ac038790 0x7fd5ac03ac40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5c0137e60 0x7fd5c012b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd5c0131b20 0x7fd5c012ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 --2- 172.21.15.17:0/2198949701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd5c00a0280 0x7fd5c012e3d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:49.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 >> 172.21.15.17:0/2198949701 conn(0x7fd5c0071270 msgr2=0x7fd5c0129990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:49.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 shutdown_connections 2024-09-15T02:01:49.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:49.512+0000 7fd5c7217640 1 -- 172.21.15.17:0/2198949701 wait complete. 2024-09-15T02:01:52.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:52.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.633+0000 7f020e6b8640 10 monclient: get_monmap_and_config 2024-09-15T02:01:52.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.633+0000 7f020e6b8640 10 monclient: build_initial_monmap 2024-09-15T02:01:52.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.633+0000 7f020e6b8640 10 monclient: monmap: 2024-09-15T02:01:52.635 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:52.634404+0000 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:52.634404+0000 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:52.636 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 Processor -- start 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 -- start start 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient: init 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f020806e920 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f020806ecf0 0x7f020806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-a con 0x7f020806ecf0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 0x7f0208136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-b con 0x7f0208134b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020809aef0 con 0x7f020806e920 2024-09-15T02:01:52.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020809d720 con 0x7f020806ecf0 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020809b0c0 con 0x7f0208134b50 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.634+0000 7f020e6b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020e6b8640 10 monclient(hunting): authenticate will time out at 1372.631009s 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f020806ecf0 0x7f020806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f0207fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f020806e920 0x7f020812fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44182/0 (socket says 172.21.15.17:44182) 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f0207fff640 1 -- 172.21.15.17:0/1372363813 learned_addr learned my addr 172.21.15.17:0/1372363813 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 0x7f0208136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): get_auth_request con 0x7f020806ecf0 auth_method 0 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): get_auth_request con 0x7f0208134b50 auth_method 0 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient(hunting): handle_auth_done global_id 104955 payload 274 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 -- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 msgr2=0x7f0208136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 0x7f0208136f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 -- 172.21.15.17:0/1372363813 >> v1:172.21.15.17:6789/0 conn(0x7f020806e920 legacy=0x7f020812fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f02077fe640 1 -- 172.21.15.17:0/1372363813 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020809c860 con 0x7f020806ecf0 2024-09-15T02:01:52.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.635+0000 7f020cc55640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 0x7f0208136f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f02077fe640 1 -- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806ecf0 msgr2=0x7f020806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f02077fe640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806ecf0 0x7f020806f0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0208097c60 tx=0x7f01fc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d48249fe198d5f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 1 -- 172.21.15.17:0/1372363813 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc02ae50 con 0x7f020806ecf0 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: dump: 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: _finish_auth 0 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 1 -- 172.21.15.17:0/1372363813 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc032440 con 0x7f020806ecf0 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 1 -- 172.21.15.17:0/1372363813 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0327e0 con 0x7f020806ecf0 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f020e6b8640 5 monclient: authenticate success, global_id 104955 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f020e6b8640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:52.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f020e6b8640 4 set_mon_vals no callback set 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.638+0000 7f0206ffd640 10 monclient: dump: 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:52.649 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 10 monclient: shutdown 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 -- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806ecf0 msgr2=0x7f020806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806ecf0 0x7f020806f0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0208097c60 tx=0x7f01fc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 -- 172.21.15.17:0/1372363813 shutdown_connections 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0208134b50 0x7f0208136f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1372363813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806ecf0 0x7f020806f0f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 -- 172.21.15.17:0/1372363813 >> 172.21.15.17:0/1372363813 conn(0x7f02080a1000 msgr2=0x7f02080a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 -- 172.21.15.17:0/1372363813 shutdown_connections 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 1 -- 172.21.15.17:0/1372363813 wait complete. 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 10 monclient: build_initial_monmap 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.639+0000 7f020e6b8640 10 monclient: monmap: 2024-09-15T02:01:52.650 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 Processor -- start 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 -- start start 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient: init 2024-09-15T02:01:52.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-a con 0x7f020806e920 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f020806ecf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208134b50 0x7f020806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): picked mon.noname-b con 0x7f0208134b50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0208119930 con 0x7f020806e920 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020809bbb0 con 0x7f0208134b50 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020809faa0 con 0x7f020806ecf0 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.640+0000 7f020e6b8640 10 monclient(hunting): authenticate will time out at 1372.636992s 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56202/0 (socket says 172.21.15.17:56202) 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 -- 172.21.15.17:0/1624763586 learned_addr learned my addr 172.21.15.17:0/1624763586 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41064/0 (socket says 172.21.15.17:41064) 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): get_auth_request con 0x7f020806ecf0 auth_method 0 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): get_auth_request con 0x7f020806e920 auth_method 0 2024-09-15T02:01:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f020cc55640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208134b50 0x7f020806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f020cc55640 10 monclient(hunting): get_auth_request con 0x7f0208134b50 auth_method 0 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f020cc55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f020cc55640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f020cc55640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient(hunting): handle_auth_done global_id 134189 payload 824 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 -- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208134b50 msgr2=0x7f020806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208134b50 0x7f020806c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 -- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 msgr2=0x7f020813f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 monclient: found mon.noname-c 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f02077fe640 1 -- 172.21.15.17:0/1624763586 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0200009a20 con 0x7f020806ecf0 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.641+0000 7f0207fff640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f02077fe640 1 -- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 msgr2=0x7f020813fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f02077fe640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009360 tx=0x7f01fc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:52.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc03b8f0 con 0x7f020806ecf0 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: dump: 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:52.656 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: _finish_auth 0 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc03ba50 con 0x7f020806ecf0 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f020e6b8640 5 monclient: authenticate success, global_id 134189 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f020e6b8640 10 monclient: _renew_subs 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc03bdd0 con 0x7f020806ecf0 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f020e6b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc009390 con 0x7f020806ecf0 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f01effff640 10 monclient: dump: 2024-09-15T02:01:52.657 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.643+0000 7f02067fc640 4 set_mon_vals no callback set 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.644+0000 7f020e6b8640 10 monclient: _renew_subs 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.644+0000 7f020e6b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.644+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0208118710 con 0x7f020806ecf0 2024-09-15T02:01:52.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.644+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01fc039040 con 0x7f020806ecf0 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f020e6b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f01effff640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 0x7f01f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f020e6b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f020e6b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02080724a0 con 0x7f020806ecf0 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f01fc06c910 con 0x7f020806ecf0 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f0207fff640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 0x7f01f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.645+0000 7f0207fff640 10 monclient: get_auth_request con 0x7f01f4038740 auth_method 0 2024-09-15T02:01:52.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.648+0000 7f0207fff640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 0x7f01f403abf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f01f8007fd0 tx=0x7f01f8007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:52.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.650+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01fc037080 con 0x7f020806ecf0 2024-09-15T02:01:52.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.650+0000 7f01effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:52.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.650+0000 7f01effff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f020e6b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f020e6b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f020e6b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f02080008d0 con 0x7f020806ecf0 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f01effff640 1 -- 172.21.15.17:0/1624763586 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f01fc03f080 con 0x7f020806ecf0 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f01effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:52.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.879+0000 7f01effff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:52.882 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:52.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 msgr2=0x7f01f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:52.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 0x7f01f403abf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f01f8007fd0 tx=0x7f01f8007f00 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 10 monclient: shutdown 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 msgr2=0x7f020813fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009360 tx=0x7f01fc032000 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 shutdown_connections 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f01f4038740 0x7f01f403abf0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208134b50 0x7f020806c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f020806ecf0 0x7f020813fcd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 --2- 172.21.15.17:0/1624763586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f020806e920 0x7f020813f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 >> 172.21.15.17:0/1624763586 conn(0x7f02080a1000 msgr2=0x7f02081357f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:52.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 shutdown_connections 2024-09-15T02:01:52.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.881+0000 7f020e6b8640 1 -- 172.21.15.17:0/1624763586 wait complete. 2024-09-15T02:01:52.894 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:01:52.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:01:52.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:01:52.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.989+0000 7fed59a34640 10 monclient: get_monmap_and_config 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.989+0000 7fed59a34640 10 monclient: build_initial_monmap 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.989+0000 7fed59a34640 10 monclient: monmap: 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:52.990583+0000 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:52.990583+0000 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:52.992 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:52.993 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:52.993 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:52.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 1 Processor -- start 2024-09-15T02:01:52.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 1 -- start start 2024-09-15T02:01:52.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 10 monclient: init 2024-09-15T02:01:52.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed540a5780 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 0x7fed5409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-b con 0x7fed5409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.990+0000 7fed59a34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed54069c90 0x7fed54139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-c con 0x7fed54069c90 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed540a5b50 con 0x7fed540a5780 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed540a18b0 con 0x7fed54069c90 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5409c5f0 con 0x7fed5409ccf0 2024-09-15T02:01:52.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed59a34640 10 monclient(hunting): authenticate will time out at 1372.987081s 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed52ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fed540a5780 0x7fed54069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54164/0 (socket says 172.21.15.17:54164) 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed52ffd640 1 -- 172.21.15.17:0/154289923 learned_addr learned my addr 172.21.15.17:0/154289923 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 0x7fed5409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request con 0x7fed5409ccf0 auth_method 0 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed537fe640 1 --2- 172.21.15.17:0/154289923 >> v2:172.21.15.67:3300/0 conn(0x7fed54069c90 0x7fed54139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed537fe640 10 monclient(hunting): get_auth_request con 0x7fed54069c90 auth_method 0 2024-09-15T02:01:52.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_done global_id 134192 payload 274 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 -- 172.21.15.17:0/154289923 >> v2:172.21.15.67:3300/0 conn(0x7fed54069c90 msgr2=0x7fed54139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/154289923 >> v2:172.21.15.67:3300/0 conn(0x7fed54069c90 0x7fed54139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 -- 172.21.15.17:0/154289923 >> v1:172.21.15.17:6789/0 conn(0x7fed540a5780 legacy=0x7fed54069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 monclient: found mon.noname-b 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:52.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.991+0000 7fed4a7fc640 1 -- 172.21.15.17:0/154289923 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5409d210 con 0x7fed5409ccf0 2024-09-15T02:01:53.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed4a7fc640 1 -- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 msgr2=0x7fed5409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 0x7fed5409bc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009850 tx=0x7fed3c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3669c41259e8dff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 1 -- 172.21.15.17:0/154289923 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c02adf0 con 0x7fed5409ccf0 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: dump: 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.005 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 5 monclient: authenticate success, global_id 134192 2024-09-15T02:01:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 1 -- 172.21.15.17:0/154289923 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c032440 con 0x7fed5409ccf0 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 4 set_mon_vals no callback set 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 1 -- 172.21.15.17:0/154289923 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0327e0 con 0x7fed5409ccf0 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed527fc640 10 monclient: dump: 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.007 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 10 monclient: shutdown 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 1 -- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 msgr2=0x7fed5409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.994+0000 7fed59a34640 1 --2- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 0x7fed5409bc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009850 tx=0x7fed3c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 -- 172.21.15.17:0/154289923 shutdown_connections 2024-09-15T02:01:53.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 --2- 172.21.15.17:0/154289923 >> v2:172.21.15.67:3300/0 conn(0x7fed54069c90 0x7fed54139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 --2- 172.21.15.17:0/154289923 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed5409ccf0 0x7fed5409bc90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 -- 172.21.15.17:0/154289923 >> 172.21.15.17:0/154289923 conn(0x7fed54070e10 msgr2=0x7fed540a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 -- 172.21.15.17:0/154289923 shutdown_connections 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 1 -- 172.21.15.17:0/154289923 wait complete. 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 10 monclient: build_initial_monmap 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.995+0000 7fed59a34640 10 monclient: monmap: 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:53.009 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 Processor -- start 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 -- start start 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient: init 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-c con 0x7fed54069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5409ccf0 0x7fed5413d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-b con 0x7fed5409ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed540a5780 0x7fed54141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): picked mon.noname-a con 0x7fed540a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54122f90 con 0x7fed540a5780 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541229d0 con 0x7fed5409ccf0 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed541204b0 con 0x7fed54069c90 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed59a34640 10 monclient(hunting): authenticate will time out at 1372.992797s 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed537fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed540a5780 0x7fed54141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56234/0 (socket says 172.21.15.17:56234) 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 1 -- 172.21.15.17:0/385288568 learned_addr learned my addr 172.21.15.17:0/385288568 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 10 monclient(hunting): get_auth_request con 0x7fed54069c90 auth_method 0 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed537fe640 10 monclient(hunting): get_auth_request con 0x7fed540a5780 auth_method 0 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.996+0000 7fed537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5409ccf0 0x7fed5413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request con 0x7fed5409ccf0 auth_method 0 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient(hunting): handle_auth_done global_id 134195 payload 824 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5409ccf0 msgr2=0x7fed5413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5409ccf0 0x7fed5413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed540a5780 msgr2=0x7fed54141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed540a5780 0x7fed54141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 monclient: found mon.noname-c 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed52ffd640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c009390 con 0x7fed54069c90 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.997+0000 7fed537fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed540a5780 0x7fed54141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed52ffd640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 msgr2=0x7fed5413dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed52ffd640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c610 tx=0x7fed44004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44005cb0 con 0x7fed54069c90 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 10 monclient: dump: 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44005e30 con 0x7fed54069c90 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed59a34640 5 monclient: authenticate success, global_id 134195 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed59a34640 10 monclient: _renew_subs 2024-09-15T02:01:53.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed59a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:52.999+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed541226f0 con 0x7fed54069c90 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44007340 con 0x7fed54069c90 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 10 monclient: dump: 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed51ffb640 4 set_mon_vals no callback set 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 10 monclient: _renew_subs 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5413dd30 con 0x7fed54069c90 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 0x7fed54141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54063b20 con 0x7fed54121820 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.000+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fed4401b400 con 0x7fed54069c90 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 0x7fed54141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient: get_auth_request con 0x7fed54121820 auth_method 0 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4b7fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed34038740 0x7fed3403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_done global_id 104964 payload 824 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fed4404c1d0 con 0x7fed54069c90 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed4a7fc640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed54123100 con 0x7fed54121820 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed537fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed34038740 0x7fed3403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.001+0000 7fed537fe640 10 monclient: get_auth_request con 0x7fed34038740 auth_method 0 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4a7fc640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 msgr2=0x7fed54141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 0x7fed54141a50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fed3c034000 tx=0x7fed3c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c031b00 con 0x7fed54121820 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:53.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.003+0000 7fed4b7fe640 10 monclient: dump: 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fed3c031c60 con 0x7fed54121820 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:53.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 msgr2=0x7fed54141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 0x7fed54141a50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fed3c034000 tx=0x7fed3c032000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 msgr2=0x7fed54141a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed4b7fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed54121820 0x7fed54141a50 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fed3c034000 tx=0x7fed3c032000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.004+0000 7fed537fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed34038740 0x7fed3403abf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fed5413e520 tx=0x7fed4c0097b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.009+0000 7fed59a34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.009+0000 7fed59a34640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 0x7fed54141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.009+0000 7fed59a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.009+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54122c10 con 0x7fed5413e920 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 0x7fed54141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient: get_auth_request con 0x7fed5413e920 auth_method 0 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 10 monclient(hunting): handle_auth_done global_id 104967 payload 824 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.010+0000 7fed4a7fc640 1 -- 172.21.15.17:0/385288568 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed54122dd0 con 0x7fed5413e920 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4a7fc640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 msgr2=0x7fed54141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4a7fc640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 0x7fed54141f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fed3c03e000 tx=0x7fed3c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c03c990 con 0x7fed5413e920 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 10 monclient: dump: 2024-09-15T02:01:53.024 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fed3c03caf0 con 0x7fed5413e920 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 msgr2=0x7fed54141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 0x7fed54141f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fed3c03e000 tx=0x7fed3c03c000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 msgr2=0x7fed54141f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.012+0000 7fed4b7fe640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed5413e920 0x7fed54141f90 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fed3c03e000 tx=0x7fed3c03c000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:53.026 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.027 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.028 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:53.029 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:53.030 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:53.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.032 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.033 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.034 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.036 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:53.037 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.040 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:01:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:53.042 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:53.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed34038740 msgr2=0x7fed3403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed34038740 0x7fed3403abf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fed5413e520 tx=0x7fed4c0097b0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 10 monclient: shutdown 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 msgr2=0x7fed5413dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c610 tx=0x7fed44004960 comp rx=0 tx=0).stop 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed537fe640 1 -- 172.21.15.17:0/385288568 reap_dead start 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 shutdown_connections 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 --2- 172.21.15.17:0/385288568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed54069c90 0x7fed5413dfa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 >> 172.21.15.17:0/385288568 conn(0x7fed54070e10 msgr2=0x7fed54098d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 shutdown_connections 2024-09-15T02:01:53.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.013+0000 7fed59a34640 1 -- 172.21.15.17:0/385288568 wait complete. 2024-09-15T02:01:53.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:53.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:01:53.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:01:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.140+0000 7fc9aa4c0640 10 monclient: get_monmap_and_config 2024-09-15T02:01:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.140+0000 7fc9aa4c0640 10 monclient: build_initial_monmap 2024-09-15T02:01:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.140+0000 7fc9aa4c0640 10 monclient: monmap: 2024-09-15T02:01:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:53.141313+0000 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:53.141313+0000 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:53.143 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 Processor -- start 2024-09-15T02:01:53.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 -- start start 2024-09-15T02:01:53.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient: init 2024-09-15T02:01:53.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:53.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc9a4097320 0x7fc9a40976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a4097320 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9a409a590 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9a412edb0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40985d0 con 0x7fc9a409a590 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a409d200 con 0x7fc9a412edb0 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a4097c30 con 0x7fc9a4097320 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9aa4c0640 10 monclient(hunting): authenticate will time out at 1373.137890s 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.141+0000 7fc9a3fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc9a4097320 0x7fc9a40976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a37fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc9a409a590 0x7fc9a412e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54168/0 (socket says 172.21.15.17:54168) 2024-09-15T02:01:53.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3720491835 learned_addr learned my addr 172.21.15.17:0/3720491835 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): get_auth_request con 0x7fc9a4097320 auth_method 0 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_done global_id 104970 payload 274 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3720491835 >> v1:172.21.15.67:6789/0 conn(0x7fc9a412edb0 legacy=0x7fc9a41369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3720491835 >> v1:172.21.15.17:6789/0 conn(0x7fc9a409a590 legacy=0x7fc9a412e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 1 monclient: found mon.noname-a 2024-09-15T02:01:53.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:53.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:53.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.142+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3720491835 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40992f0 con 0x7fc9a4097320 2024-09-15T02:01:53.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3720491835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a4097320 msgr2=0x7fc9a40976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a3fff640 1 --2- 172.21.15.17:0/3720491835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a4097320 0x7fc9a40976f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc994009850 tx=0x7fc9940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cf5fdd800275ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 1 -- 172.21.15.17:0/3720491835 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994032440 con 0x7fc9a4097320 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: dump: 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.155 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 1 -- 172.21.15.17:0/3720491835 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9940325a0 con 0x7fc9a4097320 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 5 monclient: authenticate success, global_id 104970 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 1 -- 172.21.15.17:0/3720491835 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994032940 con 0x7fc9a4097320 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 4 set_mon_vals no callback set 2024-09-15T02:01:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9a2ffd640 10 monclient: dump: 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 10 monclient: shutdown 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3720491835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a4097320 msgr2=0x7fc9a40976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3720491835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a4097320 0x7fc9a40976f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc994009850 tx=0x7fc9940090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3720491835 shutdown_connections 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3720491835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a4097320 0x7fc9a40976f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3720491835 >> 172.21.15.17:0/3720491835 conn(0x7fc9a40a1000 msgr2=0x7fc9a40a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3720491835 shutdown_connections 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.145+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3720491835 wait complete. 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.146+0000 7fc9aa4c0640 10 monclient: build_initial_monmap 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.146+0000 7fc9aa4c0640 10 monclient: monmap: 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:53.158 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.146+0000 7fc9aa4c0640 1 Processor -- start 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.146+0000 7fc9aa4c0640 1 -- start start 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.146+0000 7fc9aa4c0640 10 monclient: init 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:01:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a4097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 0x7fc9a4140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a409a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a412edb0 0x7fc9a411c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a412edb0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a411dfc0 con 0x7fc9a412edb0 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a411fd40 con 0x7fc9a409a590 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a412e760 con 0x7fc9a4097320 2024-09-15T02:01:53.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9aa4c0640 10 monclient(hunting): authenticate will time out at 1373.143438s 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56236/0 (socket says 172.21.15.17:56236) 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3574106057 learned_addr learned my addr 172.21.15.17:0/3574106057 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): get_auth_request con 0x7fc9a4097320 auth_method 0 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a412edb0 0x7fc9a411c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): get_auth_request con 0x7fc9a412edb0 auth_method 0 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 0x7fc9a4140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 10 monclient(hunting): get_auth_request con 0x7fc9a409a590 auth_method 0 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 msgr2=0x7fc9a4140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:01:53.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 msgr2=0x7fc9a4140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 0x7fc9a4140010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 0x7fc9a4140010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient(hunting): handle_auth_done global_id 134198 payload 824 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 msgr2=0x7fc9a4140010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9a409a590 0x7fc9a4140010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a412edb0 msgr2=0x7fc9a411c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a412edb0 0x7fc9a411c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 monclient: found mon.noname-c 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994009390 con 0x7fc9a4097320 2024-09-15T02:01:53.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.147+0000 7fc9a8a5d640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9a412edb0 0x7fc9a411c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9a3fff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 msgr2=0x7fc9a413fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9a3fff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc994009360 tx=0x7fc994032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994000d10 con 0x7fc9a4097320 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: dump: 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.164 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc994031c70 con 0x7fc9a4097320 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 5 monclient: authenticate success, global_id 134198 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 10 monclient: _renew_subs 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a411fa60 con 0x7fc9a4097320 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99403d440 con 0x7fc9a4097320 2024-09-15T02:01:53.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc98bfff640 10 monclient: dump: 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 10 monclient: _renew_subs 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a411eb00 con 0x7fc9a4097320 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.150+0000 7fc9a27fc640 4 set_mon_vals no callback set 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9aa4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 0x7fc9a411d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4063c20 con 0x7fc9a4120c10 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 0x7fc9a411d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient: get_auth_request con 0x7fc9a4120c10 auth_method 0 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc994039040 con 0x7fc9a4097320 2024-09-15T02:01:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_done global_id 134201 payload 824 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9a4120190 con 0x7fc9a4120c10 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc98bfff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc990038790 0x7fc99003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc99406c4f0 con 0x7fc9a4097320 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a8a5d640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc990038790 0x7fc99003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.151+0000 7fc9a8a5d640 10 monclient: get_auth_request con 0x7fc990038790 auth_method 0 2024-09-15T02:01:53.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 msgr2=0x7fc9a411d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 0x7fc9a411d210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc998009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998004e60 con 0x7fc9a4120c10 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc98bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.153+0000 7fc98bfff640 10 monclient: dump: 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc998007170 con 0x7fc9a4120c10 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 msgr2=0x7fc9a411d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 0x7fc9a411d210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc998009040 comp rx=0 tx=0).stop 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 msgr2=0x7fc9a411d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc98bfff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4120c10 0x7fc9a411d210 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc998009040 comp rx=0 tx=0).stop 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.154+0000 7fc9a8a5d640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc990038790 0x7fc99003ac40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc99c007fd0 tx=0x7fc99c0025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.159+0000 7fc9aa4c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:53.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.159+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 0x7fc9a4029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.159+0000 7fc9aa4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.159+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a411fbf0 con 0x7fc9a40298c0 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 0x7fc9a4029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient: get_auth_request con 0x7fc9a40298c0 auth_method 0 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 10 monclient(hunting): handle_auth_done global_id 134204 payload 824 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.160+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9a40724a0 con 0x7fc9a40298c0 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc9a37fe640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 msgr2=0x7fc9a4029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc9a37fe640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 0x7fc9a4029cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc99801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998020810 con 0x7fc9a40298c0 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 10 monclient: dump: 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.174 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fc998020970 con 0x7fc9a40298c0 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 msgr2=0x7fc9a4029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 0x7fc9a4029cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc99801a000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 msgr2=0x7fc9a4029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:53.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.162+0000 7fc98bfff640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a40298c0 0x7fc9a4029cc0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc99800e710 tx=0x7fc99801a000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:53.176 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.177 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:53.179 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:53.187 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:53.188 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:53.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc990038790 msgr2=0x7fc99003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc990038790 0x7fc99003ac40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc99c007fd0 tx=0x7fc99c0025b0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 10 monclient: shutdown 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 msgr2=0x7fc9a413fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9a8a5d640 1 -- 172.21.15.17:0/3574106057 reap_dead start 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc994009360 tx=0x7fc994032000 comp rx=0 tx=0).stop 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 shutdown_connections 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 --2- 172.21.15.17:0/3574106057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc9a4097320 0x7fc9a413fad0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 >> 172.21.15.17:0/3574106057 conn(0x7fc9a40a1000 msgr2=0x7fc9a40a0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 shutdown_connections 2024-09-15T02:01:53.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.163+0000 7fc9aa4c0640 1 -- 172.21.15.17:0/3574106057 wait complete. 2024-09-15T02:01:53.191 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:53.191 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:01:53.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:01:53.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.287+0000 7fe32a19d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:53.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.287+0000 7fe32a19d640 10 monclient: build_initial_monmap 2024-09-15T02:01:53.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.287+0000 7fe32a19d640 10 monclient: monmap: 2024-09-15T02:01:53.289 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:53.288415+0000 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:53.288415+0000 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:53.290 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 Processor -- start 2024-09-15T02:01:53.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 -- start start 2024-09-15T02:01:53.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient: init 2024-09-15T02:01:53.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:53.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe324131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe3240a0280 0x7fe324131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3240a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe324132430 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3240a0040 con 0x7fe324132430 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32409bff0 con 0x7fe324131b20 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3240a14a0 con 0x7fe3240a0280 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe32a19d640 10 monclient(hunting): authenticate will time out at 1373.284876s 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe322ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe3240a0280 0x7fe324131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.288+0000 7fe323fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe324132430 0x7fe324138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54170/0 (socket says 172.21.15.17:54170) 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe323fff640 1 -- 172.21.15.17:0/916747879 learned_addr learned my addr 172.21.15.17:0/916747879 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): get_auth_request con 0x7fe3240a0280 auth_method 0 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient(hunting): handle_auth_done global_id 104976 payload 274 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 1 -- 172.21.15.17:0/916747879 >> v1:172.21.15.67:6789/0 conn(0x7fe324131b20 legacy=0x7fe32409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 1 -- 172.21.15.17:0/916747879 >> v1:172.21.15.17:6789/0 conn(0x7fe324132430 legacy=0x7fe324138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:53.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 1 monclient: found mon.noname-a 2024-09-15T02:01:53.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:53.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:53.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.289+0000 7fe322ffd640 1 -- 172.21.15.17:0/916747879 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32409f3b0 con 0x7fe3240a0280 2024-09-15T02:01:53.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe322ffd640 1 -- 172.21.15.17:0/916747879 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 msgr2=0x7fe324131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe322ffd640 1 --2- 172.21.15.17:0/916747879 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe324131ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe318009850 tx=0x7fe3180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de73c5d38bd14eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 1 -- 172.21.15.17:0/916747879 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318032440 con 0x7fe3240a0280 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: dump: 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.301 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 1 -- 172.21.15.17:0/916747879 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3180325a0 con 0x7fe3240a0280 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 5 monclient: authenticate success, global_id 104976 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 1 -- 172.21.15.17:0/916747879 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318032940 con 0x7fe3240a0280 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 4 set_mon_vals no callback set 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe3227fc640 10 monclient: dump: 2024-09-15T02:01:53.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 10 monclient: shutdown 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 -- 172.21.15.17:0/916747879 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 msgr2=0x7fe324131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 --2- 172.21.15.17:0/916747879 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe324131ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe318009850 tx=0x7fe3180090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 -- 172.21.15.17:0/916747879 shutdown_connections 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 --2- 172.21.15.17:0/916747879 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe324131ef0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 -- 172.21.15.17:0/916747879 >> 172.21.15.17:0/916747879 conn(0x7fe324071270 msgr2=0x7fe3240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 -- 172.21.15.17:0/916747879 shutdown_connections 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.292+0000 7fe32a19d640 1 -- 172.21.15.17:0/916747879 wait complete. 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.293+0000 7fe32a19d640 10 monclient: build_initial_monmap 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.293+0000 7fe32a19d640 10 monclient: monmap: 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:53.304 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.293+0000 7fe32a19d640 1 Processor -- start 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.293+0000 7fe32a19d640 1 -- start start 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.293+0000 7fe32a19d640 10 monclient: init 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3240a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:53.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe324131b20 0x7fe324132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-c con 0x7fe324131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324132430 0x7fe324132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): picked mon.noname-b con 0x7fe324132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240a09b0 con 0x7fe3240a0280 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241315f0 con 0x7fe324132430 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe324133dd0 con 0x7fe324131b20 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe32a19d640 10 monclient(hunting): authenticate will time out at 1373.290323s 2024-09-15T02:01:53.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe324131b20 0x7fe324132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34580/0 (socket says 172.21.15.17:34580) 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 -- 172.21.15.17:0/2516313522 learned_addr learned my addr 172.21.15.17:0/2516313522 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): get_auth_request con 0x7fe3240a0280 auth_method 0 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): get_auth_request con 0x7fe324131b20 auth_method 0 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe323fff640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324132430 0x7fe324132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient(hunting): handle_auth_done global_id 104979 payload 824 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe324131b20 msgr2=0x7fe324132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe324131b20 0x7fe324132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324132430 msgr2=0x7fe324132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324132430 0x7fe324132eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:53.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.294+0000 7fe3237fe640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009390 con 0x7fe3240a0280 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.296+0000 7fe3237fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 msgr2=0x7fe32409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.296+0000 7fe3237fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c610 tx=0x7fe314007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314012840 con 0x7fe3240a0280 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: dump: 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.309 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3140129e0 con 0x7fe3240a0280 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 5 monclient: authenticate success, global_id 104979 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314012da0 con 0x7fe3240a0280 2024-09-15T02:01:53.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 monclient: _renew_subs 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe324135880 con 0x7fe3240a0280 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe30b7fe640 10 monclient: dump: 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:53.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe321ffb640 4 set_mon_vals no callback set 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 monclient: _renew_subs 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324134930 con 0x7fe3240a0280 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.297+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324135e30 con 0x7fe3241333f0 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe314011590 con 0x7fe3240a0280 2024-09-15T02:01:53.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe3241333f0 auth_method 0 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 0x7fe31003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe31404c750 con 0x7fe3240a0280 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe323fff640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 0x7fe31003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.298+0000 7fe323fff640 10 monclient: get_auth_request con 0x7fe310038740 auth_method 0 2024-09-15T02:01:53.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.301+0000 7fe323fff640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 0x7fe31003abf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005ac0 tx=0x7fe31c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:53.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe3241333f0 auth_method 0 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:53.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.499+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:01:53.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:53.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe3241333f0 auth_method 0 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:53.900+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:01:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe3241333f0 auth_method 0 2024-09-15T02:01:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:54.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:54.702+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:01:56.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.294+0000 7fe30a7fc640 10 monclient: tick 2024-09-15T02:01:56.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.294+0000 7fe30a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:56.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.294+0000 7fe30a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:56.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe3241333f0 auth_method 0 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 10 monclient(hunting): handle_auth_done global_id 144100 payload 824 2024-09-15T02:01:56.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.304+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe324134bb0 con 0x7fe3241333f0 2024-09-15T02:01:56.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:56.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe318034000 tx=0x7fe318032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318031ab0 con 0x7fe3241333f0 2024-09-15T02:01:56.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe30b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe30b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.307+0000 7fe30b7fe640 10 monclient: dump: 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.310 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe318031c10 con 0x7fe3241333f0 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe318034000 tx=0x7fe318032000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 msgr2=0x7fe324096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:56.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.308+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3241333f0 0x7fe324096a10 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe318034000 tx=0x7fe318032000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.314+0000 7fe32a19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:56.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.314+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.314+0000 7fe32a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.314+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324063a40 con 0x7fe324096f50 2024-09-15T02:01:56.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe324096f50 auth_method 0 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 10 monclient(hunting): handle_auth_done global_id 144103 payload 824 2024-09-15T02:01:56.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.315+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3241337c0 con 0x7fe324096f50 2024-09-15T02:01:56.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 msgr2=0x7fe324097320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe318009130 tx=0x7fe31803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 msgr2=0x7fe324097320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 msgr2=0x7fe324097320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe318009130 tx=0x7fe31803d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe318009130 tx=0x7fe31803d000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 msgr2=0x7fe324097320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 0x7fe310043520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 msgr2=0x7fe324097320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:56.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324096f50 0x7fe324097320 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310043b30 con 0x7fe310041140 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 0x7fe310043520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient: get_auth_request con 0x7fe310041140 auth_method 0 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.318+0000 7fe322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.319+0000 7fe322ffd640 10 monclient(hunting): handle_auth_done global_id 144106 payload 824 2024-09-15T02:01:56.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.319+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe318031c10 con 0x7fe310041140 2024-09-15T02:01:56.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe322ffd640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 msgr2=0x7fe310043520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:56.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe322ffd640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 0x7fe310043520 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe318009360 tx=0x7fe318046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318053a50 con 0x7fe310041140 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 10 monclient: dump: 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fe31804a7e0 con 0x7fe310041140 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 msgr2=0x7fe310043520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 0x7fe310043520 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe318009360 tx=0x7fe318046000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 msgr2=0x7fe310043520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe30b7fe640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe310041140 0x7fe310043520 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe318009360 tx=0x7fe318046000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.321+0000 7fe323fff640 1 -- 172.21.15.17:0/2516313522 reap_dead start 2024-09-15T02:01:56.330 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:56.348 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:56.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 msgr2=0x7fe31003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 0x7fe31003abf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005ac0 tx=0x7fe31c009040 comp rx=0 tx=0).stop 2024-09-15T02:01:56.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 10 monclient: shutdown 2024-09-15T02:01:56.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 msgr2=0x7fe32409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c610 tx=0x7fe314007ed0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 shutdown_connections 2024-09-15T02:01:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe310038740 0x7fe31003abf0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 --2- 172.21.15.17:0/2516313522 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3240a0280 0x7fe32409b3e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 >> 172.21.15.17:0/2516313522 conn(0x7fe324071270 msgr2=0x7fe324139d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:56.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 shutdown_connections 2024-09-15T02:01:56.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.322+0000 7fe32a19d640 1 -- 172.21.15.17:0/2516313522 wait complete. 2024-09-15T02:01:56.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:56.351 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:01:56.351 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:01:56.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.456+0000 7f213b6c9640 10 monclient: get_monmap_and_config 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.456+0000 7f213b6c9640 10 monclient: build_initial_monmap 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.456+0000 7f213b6c9640 10 monclient: monmap: 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:56.457109+0000 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:56.457109+0000 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:56.459 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.456+0000 7f213b6c9640 1 Processor -- start 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 1 -- start start 2024-09-15T02:01:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient: init 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2134131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21340a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f213409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21340a0040 con 0x7f213409f1e0 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213409bff0 con 0x7f21340a0280 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21340a14a0 con 0x7f2134131b20 2024-09-15T02:01:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f213b6c9640 10 monclient(hunting): authenticate will time out at 1376.453672s 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56268/0 (socket says 172.21.15.17:56268) 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 1 -- 172.21.15.17:0/162261262 learned_addr learned my addr 172.21.15.17:0/162261262 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): get_auth_request con 0x7f2134131b20 auth_method 0 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139c66640 1 --1- 172.21.15.17:0/162261262 >> v1:172.21.15.17:6789/0 conn(0x7f213409f1e0 0x7f2134138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54174/0 (socket says 172.21.15.17:54174) 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.457+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 10 monclient(hunting): handle_auth_done global_id 134210 payload 274 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 1 -- 172.21.15.17:0/162261262 >> v1:172.21.15.67:6789/0 conn(0x7f21340a0280 legacy=0x7f2134131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 1 -- 172.21.15.17:0/162261262 >> v1:172.21.15.17:6789/0 conn(0x7f213409f1e0 legacy=0x7f2134138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 10 monclient: _finish_hunting 0 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 1 monclient: found mon.noname-b 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.458+0000 7f2139465640 1 -- 172.21.15.17:0/162261262 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134132600 con 0x7f2134131b20 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f2139465640 1 -- 172.21.15.17:0/162261262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 msgr2=0x7f213409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f2139465640 1 --2- 172.21.15.17:0/162261262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2130009940 tx=0x7f21300091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12df6878c70a72fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 1 -- 172.21.15.17:0/162261262 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130032440 con 0x7f2134131b20 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:56.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: dump: 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: _finish_auth 0 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:56.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 5 monclient: authenticate success, global_id 134210 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 1 -- 172.21.15.17:0/162261262 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300325a0 con 0x7f2134131b20 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 1 -- 172.21.15.17:0/162261262 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130032940 con 0x7f2134131b20 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 4 set_mon_vals no callback set 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f212bfff640 10 monclient: dump: 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.473 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 10 monclient: shutdown 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 -- 172.21.15.17:0/162261262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 msgr2=0x7f213409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 --2- 172.21.15.17:0/162261262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2130009940 tx=0x7f21300091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 -- 172.21.15.17:0/162261262 shutdown_connections 2024-09-15T02:01:56.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 --2- 172.21.15.17:0/162261262 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409eca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 -- 172.21.15.17:0/162261262 >> 172.21.15.17:0/162261262 conn(0x7f2134071270 msgr2=0x7f21340a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 -- 172.21.15.17:0/162261262 shutdown_connections 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.461+0000 7f213b6c9640 1 -- 172.21.15.17:0/162261262 wait complete. 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.462+0000 7f213b6c9640 10 monclient: build_initial_monmap 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.462+0000 7f213b6c9640 10 monclient: monmap: 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:56.475 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.462+0000 7f213b6c9640 1 Processor -- start 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.462+0000 7f213b6c9640 1 -- start start 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.462+0000 7f213b6c9640 10 monclient: init 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f213409f1e0 0x7f2134099c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-b con 0x7f213409f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-a con 0x7f21340a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): picked mon.noname-c con 0x7f2134131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21340a09e0 con 0x7f21340a0280 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134130b90 con 0x7f213409f1e0 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213409b820 con 0x7f2134131b20 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f213b6c9640 10 monclient(hunting): authenticate will time out at 1376.459445s 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34596/0 (socket says 172.21.15.17:34596) 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56284/0 (socket says 172.21.15.17:56284) 2024-09-15T02:01:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 -- 172.21.15.17:0/3381874839 learned_addr learned my addr 172.21.15.17:0/3381874839 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): get_auth_request con 0x7f21340a0280 auth_method 0 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): get_auth_request con 0x7f2134131b20 auth_method 0 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f213409f1e0 0x7f2134099c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): get_auth_request con 0x7f213409f1e0 auth_method 0 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139465640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient(hunting): handle_auth_done global_id 104982 payload 824 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 -- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 msgr2=0x7f213409a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 -- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f213409f1e0 msgr2=0x7f2134099c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f213409f1e0 0x7f2134099c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient: _finish_hunting 0 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 monclient: found mon.noname-a 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2138c64640 1 -- 172.21.15.17:0/3381874839 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130009450 con 0x7f21340a0280 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.463+0000 7f2139c66640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f2138c64640 1 -- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 msgr2=0x7f213409a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f2138c64640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f212400c5e0 tx=0x7f2124007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2124012840 con 0x7f21340a0280 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: dump: 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.482 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.482 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.482 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.482 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: _finish_auth 0 2024-09-15T02:01:56.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:56.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21240129a0 con 0x7f21340a0280 2024-09-15T02:01:56.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 5 monclient: authenticate success, global_id 104982 2024-09-15T02:01:56.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:56.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2124012d60 con 0x7f21340a0280 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f21297fa640 10 monclient: dump: 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:56.484 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 10 monclient: _renew_subs 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213412c650 con 0x7f21340a0280 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f212b7fe640 4 set_mon_vals no callback set 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 10 monclient: _renew_subs 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.466+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2134131650 con 0x7f21340a0280 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2124011590 con 0x7f21340a0280 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f213b6c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:56.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f213b6c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f213b6c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340724a0 con 0x7f21340a0280 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f21297fa640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 0x7f211803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f212404c6f0 con 0x7f21340a0280 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f2139465640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 0x7f211803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.467+0000 7f2139465640 10 monclient: get_auth_request con 0x7f2118038790 auth_method 0 2024-09-15T02:01:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.470+0000 7f2139465640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 0x7f211803ac40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f21340a2f90 tx=0x7f2130032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.472+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2124010aa0 con 0x7f21340a0280 2024-09-15T02:01:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.472+0000 7f21297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.472+0000 7f21297fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:56.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.700+0000 7f213b6c9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:01:56.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.700+0000 7f213b6c9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:01:56.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.700+0000 7f213b6c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:56.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.700+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f213412cca0 con 0x7f21340a0280 2024-09-15T02:01:56.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.701+0000 7f21297fa640 1 -- 172.21.15.17:0/3381874839 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f2124016080 con 0x7f21340a0280 2024-09-15T02:01:56.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.701+0000 7f21297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:01:56.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.701+0000 7f21297fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:56.706 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.707 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=8,logm=55} crc {auth=4260918319,config=3140514009,health=528802886,kv=1529054990,logm=3896812982}) 2024-09-15T02:01:56.707 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=7,logm=58} crc {auth=1350015372,config=3140514009,health=1449738593,kv=2364430354,logm=1099944549}) 2024-09-15T02:01:56.707 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.708 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=8,logm=55} crc {auth=4260918319,config=3140514009,health=528802886,kv=1529054990,logm=3896812982}) 2024-09-15T02:01:56.708 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=7,logm=54} crc {auth=1646159749,config=3140514009,health=528802886,kv=2364430354,logm=1557320412}) 2024-09-15T02:01:56.708 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.708 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879086168}) 2024-09-15T02:01:56.709 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591386044}) 2024-09-15T02:01:56.710 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.710 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879086168}) 2024-09-15T02:01:56.710 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192821195}) 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 msgr2=0x7f211803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 0x7f211803ac40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f21340a2f90 tx=0x7f2130032000 comp rx=0 tx=0).stop 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 10 monclient: shutdown 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 msgr2=0x7f213409a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f212400c5e0 tx=0x7f2124007ed0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 shutdown_connections 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2118038790 0x7f211803ac40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f21340a0280 0x7f213409a190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2134131b20 0x7f213409a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 --2- 172.21.15.17:0/3381874839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f213409f1e0 0x7f2134099c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 >> 172.21.15.17:0/3381874839 conn(0x7f2134071270 msgr2=0x7f21340a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 shutdown_connections 2024-09-15T02:01:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f213b6c9640 1 -- 172.21.15.17:0/3381874839 wait complete. 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995513630}) 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872086487}) 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995513630}) 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141265509}) 2024-09-15T02:01:56.718 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476650357}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278415880}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476650357}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965909641}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477097025}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=652926186}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477097025}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.707+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287403816}) 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.719 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109406483}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=419873029}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109406483}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108633856}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321620253}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238006061}) 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321620253}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893832268}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713293351}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467123444}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713293351}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2974152416}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491240075}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=674976701}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491240075}) 2024-09-15T02:01:56.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842843142}) 2024-09-15T02:01:56.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274376214}) 2024-09-15T02:01:56.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467831924}) 2024-09-15T02:01:56.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274376214}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492187670}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795949601}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134008381}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795949601}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283868239}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091182592}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553096973}) 2024-09-15T02:01:56.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091182592}) 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639436192}) 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184546730}) 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870538729}) 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184546730}) 2024-09-15T02:01:56.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109941893}) 2024-09-15T02:01:56.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:01:56.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172751737}) 2024-09-15T02:01:56.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793435745}) 2024-09-15T02:01:56.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172751737}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011406871}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=119821166}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367673427}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=119821166}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117530273}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049965368}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509025087}) 2024-09-15T02:01:56.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.733 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049965368}) 2024-09-15T02:01:56.733 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.717+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763589323}) 2024-09-15T02:01:56.733 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425870721}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808560875}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425870721}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.718+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274676137}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913376219}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754323370}) 2024-09-15T02:01:56.736 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913376219}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813431780}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348733729}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2474432324}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348733729}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.719+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066240002}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370347797}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=319033707}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370347797}) 2024-09-15T02:01:56.737 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.720+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4222083703}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086320486}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627450362}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086320486}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.721+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244947590}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538772443}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386352002}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538772443}) 2024-09-15T02:01:56.738 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910488924}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424652816}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3640003184}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424652816}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2108036726}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001745811}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212336030}) 2024-09-15T02:01:56.739 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001745811}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.723+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025560157}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044433305}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065856937}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044433305}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.724+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994228491}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851088051}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983070939}) 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.740 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851088051}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.725+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864671157}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852592802}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768909294}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852592802}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225601775}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693819912}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174766581}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693819912}) 2024-09-15T02:01:56.741 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.726+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771004602}) 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669436209}) 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463269839}) 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669436209}) 2024-09-15T02:01:56.742 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:56.727+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=96767639}) 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954547339}) 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758981421}) 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954547339}) 2024-09-15T02:01:57.258 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.256+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401809203}) 2024-09-15T02:01:57.292 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.292 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956902893}) 2024-09-15T02:01:57.293 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885212422}) 2024-09-15T02:01:57.293 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.293 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956902893}) 2024-09-15T02:01:57.293 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=85600756}) 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64264438}) 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816734899}) 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64264438}) 2024-09-15T02:01:57.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260546593}) 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028006261}) 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248591652}) 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028006261}) 2024-09-15T02:01:57.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866443897}) 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587971390}) 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627304938}) 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587971390}) 2024-09-15T02:01:57.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690575135}) 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063826395}) 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158198137}) 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063826395}) 2024-09-15T02:01:57.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638244116}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728426025}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504831061}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728426025}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2740652472}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661901659}) 2024-09-15T02:01:57.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2163931512}) 2024-09-15T02:01:57.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661901659}) 2024-09-15T02:01:57.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923608979}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=68617650}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336237304}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=68617650}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1472215799}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759330470}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748871034}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759330470}) 2024-09-15T02:01:57.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614030695}) 2024-09-15T02:01:57.726 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:01:57.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624525200}) 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3197429062}) 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624525200}) 2024-09-15T02:01:57.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.820+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594693286}) 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229797655}) 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1347542857}) 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229797655}) 2024-09-15T02:01:57.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.821+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332350067}) 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034536459}) 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749753254}) 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034536459}) 2024-09-15T02:01:57.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.822+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1576338820}) 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448569830}) 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941536631}) 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448569830}) 2024-09-15T02:01:57.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.823+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828649015}) 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154943886}) 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=380950244}) 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154943886}) 2024-09-15T02:01:57.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.824+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812756023}) 2024-09-15T02:01:57.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993483105}) 2024-09-15T02:01:57.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456248192}) 2024-09-15T02:01:57.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993483105}) 2024-09-15T02:01:57.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.825+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139594459}) 2024-09-15T02:01:57.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830030983}) 2024-09-15T02:01:57.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3265073192}) 2024-09-15T02:01:57.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830030983}) 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.826+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038274595}) 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448901191}) 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3597504191}) 2024-09-15T02:01:57.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448901191}) 2024-09-15T02:01:57.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.827+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169682132}) 2024-09-15T02:01:57.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552596767}) 2024-09-15T02:01:57.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175689155}) 2024-09-15T02:01:57.831 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.831 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552596767}) 2024-09-15T02:01:57.831 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.828+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=17169634}) 2024-09-15T02:01:57.831 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.831 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369829619}) 2024-09-15T02:01:57.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1712883040}) 2024-09-15T02:01:57.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369829619}) 2024-09-15T02:01:57.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.829+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146897537}) 2024-09-15T02:01:57.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408242130}) 2024-09-15T02:01:57.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108202824}) 2024-09-15T02:01:57.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408242130}) 2024-09-15T02:01:57.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.830+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444225580}) 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154829645}) 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176282580}) 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154829645}) 2024-09-15T02:01:57.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:57.831+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534122648}) 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.836+0000 7f7322620640 10 monclient: get_monmap_and_config 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.836+0000 7f7322620640 10 monclient: build_initial_monmap 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.836+0000 7f7322620640 10 monclient: monmap: 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:57.837170+0000 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:57.837170+0000 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:57.838 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:57.839 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:57.839 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:57.839 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 Processor -- start 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 -- start start 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient: init 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f731c097320 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f731c09a590 0x7f731c1339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): picked mon.noname-a con 0x7f731c09a590 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f731c133f10 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0985d0 con 0x7f731c097320 2024-09-15T02:01:57.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c09d200 con 0x7f731c133f10 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c0992f0 con 0x7f731c09a590 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.837+0000 7f7322620640 10 monclient(hunting): authenticate will time out at 1377.833864s 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f731bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f731c097320 0x7f731c0976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54182/0 (socket says 172.21.15.17:54182) 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f731c09a590 0x7f731c1339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f731bfff640 1 -- 172.21.15.17:0/2689959215 learned_addr learned my addr 172.21.15.17:0/2689959215 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): get_auth_request con 0x7f731c09a590 auth_method 0 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient(hunting): handle_auth_done global_id 104985 payload 274 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 1 -- 172.21.15.17:0/2689959215 >> v1:172.21.15.67:6789/0 conn(0x7f731c133f10 legacy=0x7f731c1362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 1 -- 172.21.15.17:0/2689959215 >> v1:172.21.15.17:6789/0 conn(0x7f731c097320 legacy=0x7f731c0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 1 monclient: found mon.noname-a 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:57.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.838+0000 7f7313fff640 1 -- 172.21.15.17:0/2689959215 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c09cb90 con 0x7f731c09a590 2024-09-15T02:01:57.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7313fff640 1 -- 172.21.15.17:0/2689959215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 msgr2=0x7f731c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7313fff640 1 --2- 172.21.15.17:0/2689959215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c1339d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7304009850 tx=0x7f73040090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33fbfcf2f97c803 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 1 -- 172.21.15.17:0/2689959215 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304032440 con 0x7f731c09a590 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: dump: 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 1 -- 172.21.15.17:0/2689959215 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73040325a0 con 0x7f731c09a590 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 5 monclient: authenticate success, global_id 104985 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 1 -- 172.21.15.17:0/2689959215 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304032940 con 0x7f731c09a590 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 4 set_mon_vals no callback set 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:57.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f731b7fe640 10 monclient: dump: 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 10 monclient: shutdown 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:57.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 1 -- 172.21.15.17:0/2689959215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 msgr2=0x7f731c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.841+0000 7f7322620640 1 --2- 172.21.15.17:0/2689959215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c1339d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7304009850 tx=0x7f73040090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 -- 172.21.15.17:0/2689959215 shutdown_connections 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 --2- 172.21.15.17:0/2689959215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c1339d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 -- 172.21.15.17:0/2689959215 >> 172.21.15.17:0/2689959215 conn(0x7f731c0a1000 msgr2=0x7f731c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 -- 172.21.15.17:0/2689959215 shutdown_connections 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 -- 172.21.15.17:0/2689959215 wait complete. 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 10 monclient: build_initial_monmap 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 10 monclient: monmap: 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:57.850 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 Processor -- start 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 1 -- start start 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.842+0000 7f7322620640 10 monclient: init 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): picked mon.noname-c con 0x7f731c097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:57.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): picked mon.noname-a con 0x7f731c09a590 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 0x7f731c13a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): picked mon.noname-b con 0x7f731c133f10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c09ad80 con 0x7f731c09a590 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0a06c0 con 0x7f731c133f10 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c06e7b0 con 0x7f731c097320 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7322620640 10 monclient(hunting): authenticate will time out at 1377.839486s 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:57.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34636/0 (socket says 172.21.15.17:34636) 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56290/0 (socket says 172.21.15.17:56290) 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 -- 172.21.15.17:0/3834375942 learned_addr learned my addr 172.21.15.17:0/3834375942 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): get_auth_request con 0x7f731c09a590 auth_method 0 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): get_auth_request con 0x7f731c097320 auth_method 0 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:57.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 0x7f731c13a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 10 monclient(hunting): get_auth_request con 0x7f731c133f10 auth_method 0 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient(hunting): handle_auth_done global_id 104988 payload 824 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 -- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 msgr2=0x7f731c13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 -- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 msgr2=0x7f731c13a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:57.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 0x7f731c13a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient: _finish_hunting 0 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 monclient: found mon.noname-a 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7313fff640 1 -- 172.21.15.17:0/3834375942 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7304009390 con 0x7f731c09a590 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f7320bbd640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 0x7f731c13a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:01:57.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.843+0000 7f731bfff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:57.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7313fff640 1 -- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 msgr2=0x7f731c12cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:57.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7313fff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7304009360 tx=0x7f7304032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:57.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73040318c0 con 0x7f731c09a590 2024-09-15T02:01:57.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:57.857 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: dump: 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: _finish_auth 0 2024-09-15T02:01:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304031a20 con 0x7f731c09a590 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304031dc0 con 0x7f731c09a590 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 5 monclient: authenticate success, global_id 104988 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 10 monclient: _renew_subs 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c1180c0 con 0x7f731c09a590 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f731affd640 4 set_mon_vals no callback set 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:57.859 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7318ff9640 10 monclient: dump: 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:57.860 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 10 monclient: _renew_subs 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.846+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c12d220 con 0x7f731c09a590 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7304039040 con 0x7f731c09a590 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7322620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7322620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:57.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7322620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c12d410 con 0x7f731c09a590 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7318ff9640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 0x7f72fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.847+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f730406c920 con 0x7f731c09a590 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.848+0000 7f731bfff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 0x7f72fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.848+0000 7f731bfff640 10 monclient: get_auth_request con 0x7f72fc038740 auth_method 0 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.850+0000 7f731bfff640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 0x7f72fc03abf0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f730c007fd0 tx=0x7f730c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.852+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f730403b610 con 0x7f731c09a590 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.852+0000 7f7318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:57.862 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:57.852+0000 7f7318ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.078+0000 7f7322620640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.078+0000 7f7322620640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.078+0000 7f7322620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.078+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f731c12d410 con 0x7f731c09a590 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.079+0000 7f7318ff9640 1 -- 172.21.15.17:0/3834375942 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f7304049020 con 0x7f731c09a590 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.079+0000 7f7318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.079+0000 7f7318ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:58.081 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:58.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 msgr2=0x7f72fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 0x7f72fc03abf0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f730c007fd0 tx=0x7f730c007f00 comp rx=0 tx=0).stop 2024-09-15T02:01:58.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 10 monclient: shutdown 2024-09-15T02:01:58.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 msgr2=0x7f731c12cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7304009360 tx=0x7f7304032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 shutdown_connections 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72fc038740 0x7f72fc03abf0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f731c09a590 0x7f731c12cce0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f731c133f10 0x7f731c13a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 --2- 172.21.15.17:0/3834375942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f731c097320 0x7f731c13d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 >> 172.21.15.17:0/3834375942 conn(0x7f731c0a1000 msgr2=0x7f731c132090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 shutdown_connections 2024-09-15T02:01:58.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.081+0000 7f7322620640 1 -- 172.21.15.17:0/3834375942 wait complete. 2024-09-15T02:01:58.093 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:01:58.093 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:01:58.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:01:58.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:01:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.188+0000 7efe304ef640 10 monclient: get_monmap_and_config 2024-09-15T02:01:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.188+0000 7efe304ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.188+0000 7efe304ef640 10 monclient: monmap: 2024-09-15T02:01:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.190 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:58.189482+0000 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:58.189482+0000 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:58.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 1 Processor -- start 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 1 -- start start 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 10 monclient: init 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe2809c500 0x7efe2806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-c con 0x7efe2809c500 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efe2809c8d0 0x7efe2806f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.189+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-a con 0x7efe2809c8d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:58.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe2809d960 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2809ba20 con 0x7efe2809d960 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2809f6c0 con 0x7efe2809c8d0 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2809b740 con 0x7efe2809c500 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe304ef640 10 monclient(hunting): authenticate will time out at 1378.186077s 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efe2809c8d0 0x7efe2806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efe2809c8d0 0x7efe2806f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34640/0 (socket says 172.21.15.17:34640) 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 -- 172.21.15.17:0/590036233 learned_addr learned my addr 172.21.15.17:0/590036233 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): get_auth_request con 0x7efe2809c8d0 auth_method 0 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 1 --2- 172.21.15.17:0/590036233 >> v2:172.21.15.67:3300/0 conn(0x7efe2809c500 0x7efe2806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): get_auth_request con 0x7efe2809c500 auth_method 0 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient(hunting): handle_auth_done global_id 104991 payload 274 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 -- 172.21.15.17:0/590036233 >> v2:172.21.15.67:3300/0 conn(0x7efe2809c500 msgr2=0x7efe2806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 --2- 172.21.15.17:0/590036233 >> v2:172.21.15.67:3300/0 conn(0x7efe2809c500 0x7efe2806ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 -- 172.21.15.17:0/590036233 >> v1:172.21.15.67:6789/0 conn(0x7efe2809d960 legacy=0x7efe2809dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 monclient: found mon.noname-a 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.190+0000 7efe2da8a640 1 -- 172.21.15.17:0/590036233 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2809bd00 con 0x7efe2809c8d0 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2da8a640 1 -- 172.21.15.17:0/590036233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c8d0 msgr2=0x7efe2806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2da8a640 1 --2- 172.21.15.17:0/590036233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c8d0 0x7efe2806f280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efe28068230 tx=0x7efe180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84678d25c32b50b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 1 -- 172.21.15.17:0/590036233 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18032440 con 0x7efe2809c8d0 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: dump: 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.205 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 1 -- 172.21.15.17:0/590036233 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe180325a0 con 0x7efe2809c8d0 2024-09-15T02:01:58.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 1 -- 172.21.15.17:0/590036233 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18032980 con 0x7efe2809c8d0 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 5 monclient: authenticate success, global_id 104991 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 4 set_mon_vals no callback set 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe2d289640 10 monclient: dump: 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.207 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 10 monclient: shutdown 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 1 -- 172.21.15.17:0/590036233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c8d0 msgr2=0x7efe2806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.193+0000 7efe304ef640 1 --2- 172.21.15.17:0/590036233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c8d0 0x7efe2806f280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efe28068230 tx=0x7efe180091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 -- 172.21.15.17:0/590036233 shutdown_connections 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 --2- 172.21.15.17:0/590036233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c8d0 0x7efe2806f280 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 --2- 172.21.15.17:0/590036233 >> v2:172.21.15.67:3300/0 conn(0x7efe2809c500 0x7efe2806ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 -- 172.21.15.17:0/590036233 >> 172.21.15.17:0/590036233 conn(0x7efe280712f0 msgr2=0x7efe280a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 -- 172.21.15.17:0/590036233 shutdown_connections 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 -- 172.21.15.17:0/590036233 wait complete. 2024-09-15T02:01:58.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 10 monclient: monmap: 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.194+0000 7efe304ef640 1 Processor -- start 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 -- start start 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient: init 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 0x7efe2812f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-a con 0x7efe2809c500 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2809c8d0 0x7efe281195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-b con 0x7efe2809c8d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe2809d960 0x7efe28119b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): picked mon.noname-c con 0x7efe2809d960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2809cf00 con 0x7efe2809c500 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2806eb30 con 0x7efe2809c8d0 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2813fcc0 con 0x7efe2809d960 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe304ef640 10 monclient(hunting): authenticate will time out at 1378.191549s 2024-09-15T02:01:58.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 0x7efe2812f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 0x7efe2812f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34646/0 (socket says 172.21.15.17:34646) 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 -- 172.21.15.17:0/1021131676 learned_addr learned my addr 172.21.15.17:0/1021131676 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): get_auth_request con 0x7efe2809c500 auth_method 0 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe2809d960 0x7efe28119b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): get_auth_request con 0x7efe2809d960 auth_method 0 2024-09-15T02:01:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2809c8d0 0x7efe281195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): get_auth_request con 0x7efe2809c8d0 auth_method 0 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2ea8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient(hunting): handle_auth_done global_id 104994 payload 824 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe2809d960 msgr2=0x7efe28119b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efe2809d960 0x7efe28119b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2809c8d0 msgr2=0x7efe281195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2809c8d0 0x7efe281195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 monclient: found mon.noname-a 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.195+0000 7efe2e28b640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18009450 con 0x7efe2809c500 2024-09-15T02:01:58.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe2e28b640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 msgr2=0x7efe2812f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe2e28b640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 0x7efe2812f840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efe2400e900 tx=0x7efe240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24009480 con 0x7efe2809c500 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: dump: 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2400caf0 con 0x7efe2809c500 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 5 monclient: authenticate success, global_id 104994 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 10 monclient: _renew_subs 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2811ae30 con 0x7efe2809c500 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2400ce70 con 0x7efe2809c500 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe1e7fc640 10 monclient: dump: 2024-09-15T02:01:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe2ca88640 4 set_mon_vals no callback set 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 10 monclient: _renew_subs 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.198+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2811ca10 con 0x7efe2809c500 2024-09-15T02:01:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe304ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe304ef640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 0x7efe2811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28063b90 con 0x7efe2812fd80 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 0x7efe2811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efe24013680 con 0x7efe2809c500 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient: get_auth_request con 0x7efe2812fd80 auth_method 0 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe1e7fc640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe0c038790 0x7efe0c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 10 monclient(hunting): handle_auth_done global_id 104997 payload 824 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2da8a640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe280724a0 con 0x7efe2812fd80 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7efe2404c7d0 con 0x7efe2809c500 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2ea8c640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe0c038790 0x7efe0c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.199+0000 7efe2ea8c640 10 monclient: get_auth_request con 0x7efe0c038790 auth_method 0 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe2da8a640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 msgr2=0x7efe2811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe2da8a640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 0x7efe2811a050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efe2809aa60 tx=0x7efe18032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe180319a0 con 0x7efe2812fd80 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.201+0000 7efe1e7fc640 10 monclient: dump: 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efe18031b00 con 0x7efe2812fd80 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 msgr2=0x7efe2811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 0x7efe2811a050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efe2809aa60 tx=0x7efe18032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 msgr2=0x7efe2811a050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe1e7fc640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2812fd80 0x7efe2811a050 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efe2809aa60 tx=0x7efe18032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.202+0000 7efe2ea8c640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe0c038790 0x7efe0c03ac40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe20007fd0 tx=0x7efe200025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.207+0000 7efe304ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.207+0000 7efe304ef640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 0x7efe28097e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe304ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28130150 con 0x7efe2811a590 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 0x7efe28097e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient: get_auth_request con 0x7efe2811a590 auth_method 0 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 10 monclient(hunting): handle_auth_done global_id 105000 payload 824 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.208+0000 7efe2da8a640 1 -- 172.21.15.17:0/1021131676 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efe2811a990 con 0x7efe2811a590 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe2da8a640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 msgr2=0x7efe28097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe2da8a640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 0x7efe28097e70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efe2806ed10 tx=0x7efe1803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1804b070 con 0x7efe2811a590 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.210+0000 7efe1e7fc640 10 monclient: dump: 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.224 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7efe1803c940 con 0x7efe2811a590 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 msgr2=0x7efe28097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 0x7efe28097e70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efe2806ed10 tx=0x7efe1803d000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 msgr2=0x7efe28097e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe1e7fc640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2811a590 0x7efe28097e70 secure :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efe2806ed10 tx=0x7efe1803d000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:58.226 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.227 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.228 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.229 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.231 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.232 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:58.232 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:58.232 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:58.232 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:58.233 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.234 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:58.235 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.236 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.238 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.242 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:58.243 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe0c038790 msgr2=0x7efe0c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efe0c038790 0x7efe0c03ac40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe20007fd0 tx=0x7efe200025b0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 10 monclient: shutdown 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 msgr2=0x7efe2812f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 --2- 172.21.15.17:0/1021131676 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe2809c500 0x7efe2812f840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efe2400e900 tx=0x7efe240090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe2ea8c640 1 -- 172.21.15.17:0/1021131676 reap_dead start 2024-09-15T02:01:58.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe2ea8c640 1 -- 172.21.15.17:0/1021131676 reap_dead start 2024-09-15T02:01:58.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 shutdown_connections 2024-09-15T02:01:58.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 >> 172.21.15.17:0/1021131676 conn(0x7efe280712f0 msgr2=0x7efe280a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 shutdown_connections 2024-09-15T02:01:58.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.211+0000 7efe304ef640 1 -- 172.21.15.17:0/1021131676 wait complete. 2024-09-15T02:01:58.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:58.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:01:58.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:01:58.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.295 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576222869}) 2024-09-15T02:01:58.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3002373545}) 2024-09-15T02:01:58.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576222869}) 2024-09-15T02:01:58.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1708778242}) 2024-09-15T02:01:58.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897546476}) 2024-09-15T02:01:58.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511543967}) 2024-09-15T02:01:58.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897546476}) 2024-09-15T02:01:58.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579346816}) 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727401023}) 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762473052}) 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727401023}) 2024-09-15T02:01:58.304 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984256925}) 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.343+0000 7f730071d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.343+0000 7f730071d640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.343+0000 7f730071d640 10 monclient: monmap: 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:58.344317+0000 2024-09-15T02:01:58.345 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:58.344317+0000 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:58.346 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 Processor -- start 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 -- start start 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient: init 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f72f806eb30 0x7f72f806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): picked mon.noname-a con 0x7f72f806eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72f806f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72f809e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:58.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f809f6a0 con 0x7f72f809e970 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f809ed70 con 0x7f72f806f440 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f8069f40 con 0x7f72f806eb30 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f730071d640 10 monclient(hunting): authenticate will time out at 1378.340775s 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f72f806eb30 0x7f72f806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f72f806eb30 0x7f72f806ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34678/0 (socket says 172.21.15.17:34678) 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 1 -- 172.21.15.17:0/409603337 learned_addr learned my addr 172.21.15.17:0/409603337 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fecba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f72f809e970 0x7f72f8139b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54194/0 (socket says 172.21.15.17:54194) 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request con 0x7f72f806eb30 auth_method 0 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.344+0000 7f72fe4b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_done global_id 105003 payload 274 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 1 -- 172.21.15.17:0/409603337 >> v1:172.21.15.67:6789/0 conn(0x7f72f806f440 legacy=0x7f72f809bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 1 -- 172.21.15.17:0/409603337 >> v1:172.21.15.17:6789/0 conn(0x7f72f809e970 legacy=0x7f72f8139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 1 monclient: found mon.noname-a 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.345+0000 7f72fe4b9640 1 -- 172.21.15.17:0/409603337 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8069950 con 0x7f72f806eb30 2024-09-15T02:01:58.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.347+0000 7f72fe4b9640 1 -- 172.21.15.17:0/409603337 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806eb30 msgr2=0x7f72f806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.347+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/409603337 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806eb30 0x7f72f806ef00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009910 tx=0x7f72f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2df6af1488a102 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 1 -- 172.21.15.17:0/409603337 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4032440 con 0x7f72f806eb30 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: dump: 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.359 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 5 monclient: authenticate success, global_id 105003 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 1 -- 172.21.15.17:0/409603337 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40325a0 con 0x7f72f806eb30 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 4 set_mon_vals no callback set 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 1 -- 172.21.15.17:0/409603337 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4032940 con 0x7f72f806eb30 2024-09-15T02:01:58.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f72fd4b7640 10 monclient: dump: 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 10 monclient: shutdown 2024-09-15T02:01:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 -- 172.21.15.17:0/409603337 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806eb30 msgr2=0x7f72f806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 --2- 172.21.15.17:0/409603337 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806eb30 0x7f72f806ef00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009910 tx=0x7f72f40091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 -- 172.21.15.17:0/409603337 shutdown_connections 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 --2- 172.21.15.17:0/409603337 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806eb30 0x7f72f806ef00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 -- 172.21.15.17:0/409603337 >> 172.21.15.17:0/409603337 conn(0x7f72f80712f0 msgr2=0x7f72f80a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 -- 172.21.15.17:0/409603337 shutdown_connections 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.348+0000 7f730071d640 1 -- 172.21.15.17:0/409603337 wait complete. 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.349+0000 7f730071d640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.349+0000 7f730071d640 10 monclient: monmap: 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:58.362 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.349+0000 7f730071d640 1 Processor -- start 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.349+0000 7f730071d640 1 -- start start 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.349+0000 7f730071d640 10 monclient: init 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72f806eb30 0x7f72f81410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): picked mon.noname-b con 0x7f72f806eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): picked mon.noname-a con 0x7f72f806f440 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f809e970 0x7f72f811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): picked mon.noname-c con 0x7f72f809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f811f450 con 0x7f72f806f440 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81211d0 con 0x7f72f806eb30 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8141fa0 con 0x7f72f809e970 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f730071d640 10 monclient(hunting): authenticate will time out at 1378.346089s 2024-09-15T02:01:58.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f809e970 0x7f72f811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34694/0 (socket says 172.21.15.17:34694) 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 -- 172.21.15.17:0/2625859864 learned_addr learned my addr 172.21.15.17:0/2625859864 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): get_auth_request con 0x7f72f806f440 auth_method 0 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): get_auth_request con 0x7f72f809e970 auth_method 0 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72f806eb30 0x7f72f81410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request con 0x7f72f806eb30 auth_method 0 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient(hunting): handle_auth_done global_id 105006 payload 824 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f809e970 msgr2=0x7f72f811d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f809e970 0x7f72f811d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72f806eb30 msgr2=0x7f72f81410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72f806eb30 0x7f72f81410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 monclient: found mon.noname-a 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fdcb8640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4009450 con 0x7f72f806f440 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fecba640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f809e970 0x7f72f811d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.350+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72f806eb30 0x7f72f81410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72fdcb8640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 msgr2=0x7f72f8133db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72fdcb8640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c5e0 tx=0x7f72e8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8012840 con 0x7f72f806f440 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:58.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72ee7fc640 10 monclient: dump: 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.352+0000 7f72ee7fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80129a0 con 0x7f72f806f440 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 5 monclient: authenticate success, global_id 105006 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 monclient: _renew_subs 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f8120ef0 con 0x7f72f806f440 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8012d60 con 0x7f72f806f440 2024-09-15T02:01:58.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 10 monclient: dump: 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.372 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72fccb6640 4 set_mon_vals no callback set 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 monclient: _renew_subs 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f811ff90 con 0x7f72f806f440 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 0x7f72f811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8063b90 con 0x7f72f81342f0 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.353+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f72e8011590 con 0x7f72f806f440 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 0x7f72f811df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient: get_auth_request con 0x7f72f81342f0 auth_method 0 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72ee7fc640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72dc0387c0 0x7f72dc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f72e804c650 con 0x7f72f806f440 2024-09-15T02:01:58.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_done global_id 134225 payload 824 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fe4b9640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72f81417a0 con 0x7f72f81342f0 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fecba640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72dc0387c0 0x7f72dc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.354+0000 7f72fecba640 10 monclient: get_auth_request con 0x7f72dc0387c0 auth_method 0 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72fe4b9640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 msgr2=0x7f72f811df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 0x7f72f811df20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72f8067140 tx=0x7f72f4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f403ca40 con 0x7f72f81342f0 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 10 monclient: dump: 2024-09-15T02:01:58.376 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f72f403cba0 con 0x7f72f81342f0 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:58.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 msgr2=0x7f72f811df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 0x7f72f811df20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72f8067140 tx=0x7f72f4032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 msgr2=0x7f72f811df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.356+0000 7f72ee7fc640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f81342f0 0x7f72f811df20 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72f8067140 tx=0x7f72f4032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.357+0000 7f72fecba640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72dc0387c0 0x7f72dc03ac70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f72f811feb0 tx=0x7f72f0004e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f730071d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f730071d640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 0x7f72f8096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f730071d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8121080 con 0x7f72f811e460 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 0x7f72f8096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f72fe4b9640 10 monclient: get_auth_request con 0x7f72f811e460 auth_method 0 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f72fe4b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f72fe4b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.362+0000 7f72fe4b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.363+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.363+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.363+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.363+0000 7f72fe4b9640 10 monclient(hunting): handle_auth_done global_id 134228 payload 824 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.363+0000 7f72fe4b9640 1 -- 172.21.15.17:0/2625859864 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f72f8121390 con 0x7f72f811e460 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72fe4b9640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 msgr2=0x7f72f8096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72fe4b9640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 0x7f72f8096a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009210 tx=0x7f72f403c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4049870 con 0x7f72f811e460 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 10 monclient: dump: 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.381 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f72f40499d0 con 0x7f72f811e460 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 msgr2=0x7f72f8096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 0x7f72f8096a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009210 tx=0x7f72f403c000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 msgr2=0x7f72f8096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.365+0000 7f72ee7fc640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f72f811e460 0x7f72f8096a10 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009210 tx=0x7f72f403c000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.383 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.384 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.385 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.386 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.387 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:58.388 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.391 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:58.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.396 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:58.396 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:58.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72dc0387c0 msgr2=0x7f72dc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f72dc0387c0 0x7f72dc03ac70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f72f811feb0 tx=0x7f72f0004e20 comp rx=0 tx=0).stop 2024-09-15T02:01:58.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 10 monclient: shutdown 2024-09-15T02:01:58.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 msgr2=0x7f72f8133db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c5e0 tx=0x7f72e8007ed0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f72fecba640 1 -- 172.21.15.17:0/2625859864 reap_dead start 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 shutdown_connections 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 --2- 172.21.15.17:0/2625859864 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f72f806f440 0x7f72f8133db0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 >> 172.21.15.17:0/2625859864 conn(0x7f72f80712f0 msgr2=0x7f72f809c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 shutdown_connections 2024-09-15T02:01:58.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.366+0000 7f730071d640 1 -- 172.21.15.17:0/2625859864 wait complete. 2024-09-15T02:01:58.399 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:58.399 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:01:58.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:01:58.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.398+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.398+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815614604}) 2024-09-15T02:01:58.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.398+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463465617}) 2024-09-15T02:01:58.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.399+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.399+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815614604}) 2024-09-15T02:01:58.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.399+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639673650}) 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 10 monclient: get_monmap_and_config 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 10 monclient: monmap: 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:58.495015+0000 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:58.495015+0000 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.496 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:58.497 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 1 Processor -- start 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 1 -- start start 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.494+0000 7f9b8f2af640 10 monclient: init 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:01:58.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b880a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b880980a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 0x7f9b881391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b88136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b880a01e0 con 0x7f9b880980a0 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b880a1580 con 0x7f9b88136dd0 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880a5c80 con 0x7f9b880a58b0 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8f2af640 10 monclient(hunting): authenticate will time out at 1378.491479s 2024-09-15T02:01:58.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 0x7f9b881391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56332/0 (socket says 172.21.15.17:56332) 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 -- 172.21.15.17:0/2986777641 learned_addr learned my addr 172.21.15.17:0/2986777641 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): get_auth_request con 0x7f9b880a58b0 auth_method 0 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): get_auth_request con 0x7f9b88136dd0 auth_method 0 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_done global_id 134231 payload 274 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 -- 172.21.15.17:0/2986777641 >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 msgr2=0x7f9b881391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 --2- 172.21.15.17:0/2986777641 >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 0x7f9b881391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 -- 172.21.15.17:0/2986777641 >> v1:172.21.15.67:6789/0 conn(0x7f9b880980a0 legacy=0x7f9b88098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 monclient: found mon.noname-b 2024-09-15T02:01:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:01:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d04b640 1 -- 172.21.15.17:0/2986777641 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b881326c0 con 0x7f9b880a58b0 2024-09-15T02:01:58.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.495+0000 7f9b8d84c640 1 --2- 172.21.15.17:0/2986777641 >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 0x7f9b881391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8d04b640 1 -- 172.21.15.17:0/2986777641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 msgr2=0x7f9b88132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8d04b640 1 --2- 172.21.15.17:0/2986777641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009910 tx=0x7f9b840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c423e689119bb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 1 -- 172.21.15.17:0/2986777641 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84032440 con 0x7f9b880a58b0 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: dump: 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8f2af640 5 monclient: authenticate success, global_id 134231 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8f2af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:01:58.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 1 -- 172.21.15.17:0/2986777641 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b840325a0 con 0x7f9b880a58b0 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8f2af640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b8f2af640 4 set_mon_vals no callback set 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 1 -- 172.21.15.17:0/2986777641 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84032940 con 0x7f9b880a58b0 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.498+0000 7f9b7ffff640 10 monclient: dump: 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.511 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 10 monclient: shutdown 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 -- 172.21.15.17:0/2986777641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 msgr2=0x7f9b88132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/2986777641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009910 tx=0x7f9b840091d0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 -- 172.21.15.17:0/2986777641 shutdown_connections 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/2986777641 >> v2:172.21.15.17:3300/0 conn(0x7f9b88136dd0 0x7f9b881391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/2986777641 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880a58b0 0x7f9b88132040 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 -- 172.21.15.17:0/2986777641 >> 172.21.15.17:0/2986777641 conn(0x7f9b88070ed0 msgr2=0x7f9b880a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 -- 172.21.15.17:0/2986777641 shutdown_connections 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 1 -- 172.21.15.17:0/2986777641 wait complete. 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.499+0000 7f9b8f2af640 10 monclient: monmap: 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:58.513 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 Processor -- start 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 -- start start 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient: init 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 0x7f9b88141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-c con 0x7f9b880980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b880a58b0 0x7f9b88142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b880a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88136dd0 0x7f9b88132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b88136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881364e0 con 0x7f9b880a58b0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b88098dc0 con 0x7f9b88136dd0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880a31a0 con 0x7f9b880980a0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.500+0000 7f9b8f2af640 10 monclient(hunting): authenticate will time out at 1378.496958s 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 0x7f9b88141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b880a58b0 0x7f9b88142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 0x7f9b88141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56340/0 (socket says 172.21.15.17:56340) 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 -- 172.21.15.17:0/1568318049 learned_addr learned my addr 172.21.15.17:0/1568318049 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): get_auth_request con 0x7f9b880980a0 auth_method 0 2024-09-15T02:01:58.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): get_auth_request con 0x7f9b880a58b0 auth_method 0 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient(hunting): handle_auth_done global_id 134234 payload 824 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88136dd0 msgr2=0x7f9b88132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d84c640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88136dd0 0x7f9b88132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88136dd0 0x7f9b88132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b880a58b0 msgr2=0x7f9b88142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b880a58b0 0x7f9b88142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 monclient: found mon.noname-c 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8d04b640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b80009a20 con 0x7f9b880980a0 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.501+0000 7f9b8c84a640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9b880a58b0 0x7f9b88142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:01:58.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8d04b640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 msgr2=0x7f9b88141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8d04b640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 0x7f9b88141c20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009970 tx=0x7f9b84032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840319c0 con 0x7f9b880980a0 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: dump: 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.518 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84031b40 con 0x7f9b880980a0 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 5 monclient: authenticate success, global_id 134234 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 10 monclient: _renew_subs 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8403d440 con 0x7f9b880980a0 2024-09-15T02:01:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b84009450 con 0x7f9b880980a0 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7d7fa640 10 monclient: dump: 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.520 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b7f7fe640 4 set_mon_vals no callback set 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 10 monclient: _renew_subs 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.503+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b88134410 con 0x7f9b880980a0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b8f2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b84039040 con 0x7f9b880980a0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 0x7f9b88133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b88063b20 con 0x7f9b88133060 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b7d7fa640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b6c038740 0x7f9b6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.504+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9b8406c3f0 con 0x7f9b880980a0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 0x7f9b88133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient: get_auth_request con 0x7f9b88133060 auth_method 0 2024-09-15T02:01:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8d84c640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b6c038740 0x7f9b6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8d84c640 10 monclient: get_auth_request con 0x7f9b6c038740 auth_method 0 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_done global_id 144124 payload 824 2024-09-15T02:01:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.505+0000 7f9b8c84a640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b88136290 con 0x7f9b88133060 2024-09-15T02:01:58.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.507+0000 7f9b8c84a640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 msgr2=0x7f9b88133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.507+0000 7f9b8c84a640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 0x7f9b88133430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.507+0000 7f9b8d84c640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b6c038740 0x7f9b6c03abf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9b800099f0 tx=0x7f9b8000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800ce80 con 0x7f9b88133060 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 10 monclient: dump: 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.525 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9b780123f0 con 0x7f9b88133060 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 msgr2=0x7f9b88133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 0x7f9b88133430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78006f50 comp rx=0 tx=0).stop 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 msgr2=0x7f9b88133430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.508+0000 7f9b7d7fa640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133060 0x7f9b88133430 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78006f50 comp rx=0 tx=0).stop 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8f2af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 0x7f9b881371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8f2af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880724a0 con 0x7f9b88133970 2024-09-15T02:01:58.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 0x7f9b881371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient: get_auth_request con 0x7f9b88133970 auth_method 0 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 10 monclient(hunting): handle_auth_done global_id 144127 payload 824 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.514+0000 7f9b8c84a640 1 -- 172.21.15.17:0/1568318049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9b88135b20 con 0x7f9b88133970 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8c84a640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 msgr2=0x7f9b881371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8c84a640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 0x7f9b881371a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7801cd10 con 0x7f9b88133970 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 10 monclient: dump: 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f9b78005870 con 0x7f9b88133970 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 msgr2=0x7f9b881371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 0x7f9b881371a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78015000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 msgr2=0x7f9b881371a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b7d7fa640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88133970 0x7f9b881371a0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800eb30 tx=0x7f9b78015000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:58.529 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 116, 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.530 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.531 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:01:58.532 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:01:58.533 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.534 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:01:58.535 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:01:58.536 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.538 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:01:58.539 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.540 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:01:58.541 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343939941}) 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1848883909}) 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343939941}) 2024-09-15T02:01:58.542 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523766855}) 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b6c038740 msgr2=0x7f9b6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9b6c038740 0x7f9b6c03abf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9b800099f0 tx=0x7f9b8000b040 comp rx=0 tx=0).stop 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 10 monclient: shutdown 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 msgr2=0x7f9b88141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8f2af640 1 --2- 172.21.15.17:0/1568318049 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9b880980a0 0x7f9b88141c20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009970 tx=0x7f9b84032000 comp rx=0 tx=0).stop 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8d84c640 1 -- 172.21.15.17:0/1568318049 reap_dead start 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.517+0000 7f9b8d84c640 1 -- 172.21.15.17:0/1568318049 reap_dead start 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.518+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 shutdown_connections 2024-09-15T02:01:58.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.518+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 >> 172.21.15.17:0/1568318049 conn(0x7f9b88070ed0 msgr2=0x7f9b880a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.518+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 shutdown_connections 2024-09-15T02:01:58.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.518+0000 7f9b8f2af640 1 -- 172.21.15.17:0/1568318049 wait complete. 2024-09-15T02:01:58.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:01:58.545 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T02:01:58.545 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:01:58.545 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T02:01:58.545 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:01:58.546 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:01:58.546 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:01:58.557 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:01:58.557 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:01:58.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067382616}) 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=979786167}) 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067382616}) 2024-09-15T02:01:58.600 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:01:58.597+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971835493}) 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 10 monclient: monmap: 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:01:58.690978+0000 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:01:58.690978+0000 2024-09-15T02:01:58.692 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:01:58.693 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 1 Processor -- start 2024-09-15T02:01:58.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 1 -- start start 2024-09-15T02:01:58.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.690+0000 7fb705d9d640 10 monclient: init 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb700068f20 addr v1:172.21.15.17:6789/0 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7000692f0 0x7fb7000696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7000692f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb700136fe0 addr v1:172.21.15.67:6789/0 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000a1670 con 0x7fb700068f20 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7001328d0 con 0x7fb700136fe0 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7000a0020 con 0x7fb7000692f0 2024-09-15T02:01:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb705d9d640 10 monclient(hunting): authenticate will time out at 1378.687438s 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6ff7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb700068f20 0x7fb7001321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54196/0 (socket says 172.21.15.17:54196) 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/2333680679 learned_addr learned my addr 172.21.15.17:0/2333680679 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 1 --2- 172.21.15.17:0/2333680679 >> v2:172.21.15.67:3300/0 conn(0x7fb7000692f0 0x7fb7000696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): get_auth_request con 0x7fb7000692f0 auth_method 0 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient(hunting): handle_auth_done global_id 144130 payload 274 2024-09-15T02:01:58.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 1 -- 172.21.15.17:0/2333680679 >> v1:172.21.15.67:6789/0 conn(0x7fb700136fe0 legacy=0x7fb7001393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 1 -- 172.21.15.17:0/2333680679 >> v1:172.21.15.17:6789/0 conn(0x7fb700068f20 legacy=0x7fb7001321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 1 monclient: found mon.noname-c 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:01:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.691+0000 7fb6feffd640 1 -- 172.21.15.17:0/2333680679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7000a1390 con 0x7fb7000692f0 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.694+0000 7fb6feffd640 1 -- 172.21.15.17:0/2333680679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000692f0 msgr2=0x7fb7000696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.694+0000 7fb6feffd640 1 --2- 172.21.15.17:0/2333680679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000692f0 0x7fb7000696f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009880 tx=0x7fb6e80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e9e94831fd7b094 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 1 -- 172.21.15.17:0/2333680679 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8032440 con 0x7fb7000692f0 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: dump: 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 5 monclient: authenticate success, global_id 144130 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 1 -- 172.21.15.17:0/2333680679 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e80325a0 con 0x7fb7000692f0 2024-09-15T02:01:58.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 4 set_mon_vals no callback set 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 1 -- 172.21.15.17:0/2333680679 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8032940 con 0x7fb7000692f0 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb6fe7fc640 10 monclient: dump: 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 10 monclient: shutdown 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 -- 172.21.15.17:0/2333680679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000692f0 msgr2=0x7fb7000696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 --2- 172.21.15.17:0/2333680679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000692f0 0x7fb7000696f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009880 tx=0x7fb6e80090f0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 -- 172.21.15.17:0/2333680679 shutdown_connections 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 --2- 172.21.15.17:0/2333680679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000692f0 0x7fb7000696f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 -- 172.21.15.17:0/2333680679 >> 172.21.15.17:0/2333680679 conn(0x7fb700070fb0 msgr2=0x7fb7000a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 -- 172.21.15.17:0/2333680679 shutdown_connections 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.695+0000 7fb705d9d640 1 -- 172.21.15.17:0/2333680679 wait complete. 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.696+0000 7fb705d9d640 10 monclient: build_initial_monmap 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.696+0000 7fb705d9d640 10 monclient: monmap: 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:01:58.708 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.696+0000 7fb705d9d640 1 Processor -- start 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.696+0000 7fb705d9d640 1 -- start start 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.696+0000 7fb705d9d640 10 monclient: init 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb700068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 0x7fb700099c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7000692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb700136fe0 0x7fb70009a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb700136fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:01:58.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70011cd70 con 0x7fb700068f20 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000a0750 con 0x7fb700136fe0 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70009ca40 con 0x7fb7000692f0 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb705d9d640 10 monclient(hunting): authenticate will time out at 1378.693264s 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6feffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 msgr2=0x7fb700099c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6feffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 0x7fb700099c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34722/0 (socket says 172.21.15.17:34722) 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/191776621 learned_addr learned my addr 172.21.15.17:0/191776621 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:01:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb700068f20 auth_method 0 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_done global_id 105015 payload 824 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6fffff640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb700136fe0 0x7fb70009a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 msgr2=0x7fb700099c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 0x7fb700099c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb700136fe0 msgr2=0x7fb70009a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb700136fe0 0x7fb70009a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 monclient: found mon.noname-a 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.697+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/191776621 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009390 con 0x7fb700068f20 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.699+0000 7fb6ff7fe640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 msgr2=0x7fb700130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.699+0000 7fb6ff7fe640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c5e0 tx=0x7fb6f0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6ff7fe640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c5e0 tx=0x7fb6f0004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb6f0005cb0 mon_map magic: 0 v1 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0005cb0 con 0x7fb700068f20 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 10 monclient: dump: 2024-09-15T02:01:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 10 monclient: _finish_auth 0 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 5 monclient: authenticate success, global_id 105015 2024-09-15T02:01:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0005e30 con 0x7fb700068f20 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 10 monclient: _renew_subs 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70009b7b0 con 0x7fb700068f20 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb6fdffb640 4 set_mon_vals no callback set 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 10 monclient: _renew_subs 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.700+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70011c780 con 0x7fb700068f20 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0007340 con 0x7fb700068f20 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 10 monclient: dump: 2024-09-15T02:01:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb6f001b400 con 0x7fb700068f20 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb705d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb705d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb705d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70011cf70 con 0x7fb700068f20 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6f77fe640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 10 monclient: get_auth_request con 0x7fb6e0038740 auth_method 0 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 msgr2=0x7fb6e003abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 msgr2=0x7fb6e003abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.701+0000 7fb6feffd640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:01:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.705+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb6f004c2d0 con 0x7fb700068f20 2024-09-15T02:01:58.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.903+0000 7fb6feffd640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:01:58.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.903+0000 7fb6feffd640 10 monclient: get_auth_request con 0x7fb6e0038740 auth_method 0 2024-09-15T02:01:58.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:58.906+0000 7fb6feffd640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb7000687a0 tx=0x7fb6e8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:01:59.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.323+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6f0007730 con 0x7fb700068f20 2024-09-15T02:01:59.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.323+0000 7fb6f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:01:59.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.323+0000 7fb6f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:01:59.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.553+0000 7fb705d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.553+0000 7fb705d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.553+0000 7fb705d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.553+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb7000724a0 con 0x7fb700068f20 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.554+0000 7fb6f77fe640 1 -- 172.21.15.17:0/191776621 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fb6f001a360 con 0x7fb700068f20 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.554+0000 7fb6f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.554+0000 7fb6f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:01:59.556 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 msgr2=0x7fb6e003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb7000687a0 tx=0x7fb6e8032000 comp rx=0 tx=0).stop 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 10 monclient: shutdown 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 msgr2=0x7fb700130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c5e0 tx=0x7fb6f0004960 comp rx=0 tx=0).stop 2024-09-15T02:01:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 shutdown_connections 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb6e0038740 0x7fb6e003abf0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb700136fe0 0x7fb70009a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb7000692f0 0x7fb700099c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 --2- 172.21.15.17:0/191776621 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb700068f20 0x7fb700130980 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 >> 172.21.15.17:0/191776621 conn(0x7fb700070fb0 msgr2=0x7fb70009e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 shutdown_connections 2024-09-15T02:01:59.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:01:59.556+0000 7fb705d9d640 1 -- 172.21.15.17:0/191776621 wait complete. 2024-09-15T02:02:02.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.675+0000 7fe8c182b640 10 monclient: get_monmap_and_config 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.675+0000 7fe8c182b640 10 monclient: build_initial_monmap 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.675+0000 7fe8c182b640 10 monclient: monmap: 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:02.677288+0000 2024-09-15T02:02:02.678 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:02.677288+0000 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:02.679 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 Processor -- start 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 -- start start 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient: init 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 0x7fe8bc131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8bc098190 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 0x7fe8bc098560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8bc09cfa0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe8bc098aa0 0x7fe8bc067090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8bc098aa0 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8bc09d730 con 0x7fe8bc098aa0 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8bc09e380 con 0x7fe8bc09cfa0 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc1324e0 con 0x7fe8bc098190 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8c182b640 10 monclient(hunting): authenticate will time out at 1382.673659s 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8baffd640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 msgr2=0x7fe8bc131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8baffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 0x7fe8bc131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe8bc098aa0 0x7fe8bc067090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe8bc098aa0 0x7fe8bc067090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34730/0 (socket says 172.21.15.17:34730) 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 1 -- 172.21.15.17:0/623661003 learned_addr learned my addr 172.21.15.17:0/623661003 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): get_auth_request con 0x7fe8bc098aa0 auth_method 0 2024-09-15T02:02:02.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.676+0000 7fe8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 1 --2- 172.21.15.17:0/623661003 >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 0x7fe8bc098560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 10 monclient(hunting): get_auth_request con 0x7fe8bc09cfa0 auth_method 0 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 10 monclient(hunting): handle_auth_done global_id 105018 payload 274 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 -- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 msgr2=0x7fe8bc131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:02.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 --2- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 0x7fe8bc131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 -- 172.21.15.17:0/623661003 >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 msgr2=0x7fe8bc098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:02.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 --2- 172.21.15.17:0/623661003 >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 0x7fe8bc098560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:02:02.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 monclient: found mon.noname-a 2024-09-15T02:02:02.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:02.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8bb7fe640 1 -- 172.21.15.17:0/623661003 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc06eff0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.677+0000 7fe8ba7fc640 1 --2- 172.21.15.17:0/623661003 >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 0x7fe8bc098560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:02:02.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8bb7fe640 1 -- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 msgr2=0x7fe8bc067090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:02.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8bb7fe640 1 --2- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc067090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0a1ea0 tx=0x7fe8b40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e304f0eea6bb6b71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 1 -- 172.21.15.17:0/623661003 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4032440 con 0x7fe8bc098aa0 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: dump: 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:02.695 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 1 -- 172.21.15.17:0/623661003 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b40325a0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 5 monclient: authenticate success, global_id 105018 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 1 -- 172.21.15.17:0/623661003 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4032940 con 0x7fe8bc098aa0 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 4 set_mon_vals no callback set 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8b9ffb640 10 monclient: dump: 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 10 monclient: shutdown 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 -- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 msgr2=0x7fe8bc067090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 --2- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc067090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0a1ea0 tx=0x7fe8b40091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 -- 172.21.15.17:0/623661003 shutdown_connections 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 --2- 172.21.15.17:0/623661003 >> v2:172.21.15.67:3300/0 conn(0x7fe8bc09cfa0 0x7fe8bc098560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 --2- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc067090 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 --2- 172.21.15.17:0/623661003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc098190 0x7fe8bc131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 -- 172.21.15.17:0/623661003 >> 172.21.15.17:0/623661003 conn(0x7fe8bc0a2100 msgr2=0x7fe8bc0a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 -- 172.21.15.17:0/623661003 shutdown_connections 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.680+0000 7fe8c182b640 1 -- 172.21.15.17:0/623661003 wait complete. 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.681+0000 7fe8c182b640 10 monclient: build_initial_monmap 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.681+0000 7fe8c182b640 10 monclient: monmap: 2024-09-15T02:02:02.698 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.681+0000 7fe8c182b640 1 Processor -- start 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.681+0000 7fe8c182b640 1 -- start start 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.681+0000 7fe8c182b640 10 monclient: init 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:02.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc098190 0x7fe8bc141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8bc098190 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8bc098aa0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 0x7fe8bc131670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8bc09cfa0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc121bb0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc141f10 con 0x7fe8bc098190 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc09d900 con 0x7fe8bc09cfa0 2024-09-15T02:02:02.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8c182b640 10 monclient(hunting): authenticate will time out at 1382.679464s 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8bb7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 msgr2=0x7fe8bc131670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8bb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 0x7fe8bc131670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34736/0 (socket says 172.21.15.17:34736) 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 -- 172.21.15.17:0/3341998595 learned_addr learned my addr 172.21.15.17:0/3341998595 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): get_auth_request con 0x7fe8bc098aa0 auth_method 0 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc098190 0x7fe8bc141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): get_auth_request con 0x7fe8bc098190 auth_method 0 2024-09-15T02:02:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient(hunting): handle_auth_done global_id 105021 payload 824 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 -- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 msgr2=0x7fe8bc131670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 0x7fe8bc131670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 -- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc098190 msgr2=0x7fe8bc141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:02.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc098190 0x7fe8bc141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 monclient: found mon.noname-a 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.682+0000 7fe8ba7fc640 1 -- 172.21.15.17:0/3341998595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4009450 con 0x7fe8bc098aa0 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8ba7fc640 1 -- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 msgr2=0x7fe8bc131130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8ba7fc640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d8c0 tx=0x7fe8b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000deb0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: dump: 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:02.703 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b000bae0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 5 monclient: authenticate success, global_id 105021 2024-09-15T02:02:02.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 10 monclient: _renew_subs 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc09f6b0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000bea0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8a2ffd640 10 monclient: dump: 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:02.705 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 10 monclient: _renew_subs 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc121280 con 0x7fe8bc098aa0 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.685+0000 7fe8b97fa640 4 set_mon_vals no callback set 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe8b0014640 con 0x7fe8bc098aa0 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8c182b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:02.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8c182b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8c182b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc11f7b0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8a2ffd640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 0x7fe8a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe8b004cc40 con 0x7fe8bc098aa0 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8baffd640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 0x7fe8a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.686+0000 7fe8baffd640 10 monclient: get_auth_request con 0x7fe8a80386f0 auth_method 0 2024-09-15T02:02:02.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.689+0000 7fe8baffd640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 0x7fe8a803aba0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac003bd0 tx=0x7fe8ac00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:02.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.690+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe8b001e460 con 0x7fe8bc098aa0 2024-09-15T02:02:02.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.690+0000 7fe8a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:02.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.690+0000 7fe8a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8c182b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8c182b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8c182b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe8bc121680 con 0x7fe8bc098aa0 2024-09-15T02:02:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8a2ffd640 1 -- 172.21.15.17:0/3341998595 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fe8b0000ed0 con 0x7fe8bc098aa0 2024-09-15T02:02:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.917+0000 7fe8a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:02.921 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 msgr2=0x7fe8a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 0x7fe8a803aba0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac003bd0 tx=0x7fe8ac00d040 comp rx=0 tx=0).stop 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 10 monclient: shutdown 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 msgr2=0x7fe8bc131130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d8c0 tx=0x7fe8b0008040 comp rx=0 tx=0).stop 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 shutdown_connections 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe8a80386f0 0x7fe8a803aba0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe8bc098aa0 0x7fe8bc131130 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe8bc09cfa0 0x7fe8bc131670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 --2- 172.21.15.17:0/3341998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc098190 0x7fe8bc141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 >> 172.21.15.17:0/3341998595 conn(0x7fe8bc0a2100 msgr2=0x7fe8bc09b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 shutdown_connections 2024-09-15T02:02:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:02.919+0000 7fe8c182b640 1 -- 172.21.15.17:0/3341998595 wait complete. 2024-09-15T02:02:05.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:06.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:06.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:06.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 10 monclient: monmap: 2024-09-15T02:02:06.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:06.043 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:06.041922+0000 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:06.041922+0000 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 1 Processor -- start 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 1 -- start start 2024-09-15T02:02:06.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.040+0000 7fe658d9d640 10 monclient: init 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe654131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6540a0280 0x7fe654131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6540a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe65409f1e0 0x7fe654132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe65409f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:06.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6540a0040 con 0x7fe65409f1e0 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65409bff0 con 0x7fe6540a0280 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540a14a0 con 0x7fe654131b20 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe658d9d640 10 monclient(hunting): authenticate will time out at 1386.038467s 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe65259c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 msgr2=0x7fe65409eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe65259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe65409f1e0 0x7fe654132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe65409f1e0 0x7fe654132430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42078/0 (socket says 172.21.15.17:42078) 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 -- 172.21.15.17:0/1880298949 learned_addr learned my addr 172.21.15.17:0/1880298949 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:06.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): get_auth_request con 0x7fe65409f1e0 auth_method 0 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe651d9b640 1 --2- 172.21.15.17:0/1880298949 >> v2:172.21.15.67:3300/0 conn(0x7fe6540a0280 0x7fe654131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient(hunting): handle_auth_done global_id 105024 payload 274 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 -- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 msgr2=0x7fe65409eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 --2- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 -- 172.21.15.17:0/1880298949 >> v2:172.21.15.67:3300/0 conn(0x7fe6540a0280 msgr2=0x7fe654131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 --2- 172.21.15.17:0/1880298949 >> v2:172.21.15.67:3300/0 conn(0x7fe6540a0280 0x7fe654131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:02:06.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 monclient: found mon.noname-a 2024-09-15T02:02:06.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:06.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.041+0000 7fe652d9d640 1 -- 172.21.15.17:0/1880298949 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654068e80 con 0x7fe65409f1e0 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.044+0000 7fe652d9d640 1 -- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 msgr2=0x7fe654132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.044+0000 7fe652d9d640 1 --2- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654132430 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe65406ab60 tx=0x7fe64c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e7db70b8515166 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 1 -- 172.21.15.17:0/1880298949 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c007610 con 0x7fe65409f1e0 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: dump: 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:06.056 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: _finish_auth 0 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 1 -- 172.21.15.17:0/1880298949 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c02d050 con 0x7fe65409f1e0 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 1 -- 172.21.15.17:0/1880298949 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0079a0 con 0x7fe65409f1e0 2024-09-15T02:02:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 5 monclient: authenticate success, global_id 105024 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 4 set_mon_vals no callback set 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe65159a640 10 monclient: dump: 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:06.059 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:06.059 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:06.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 10 monclient: shutdown 2024-09-15T02:02:06.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 -- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 msgr2=0x7fe654132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 --2- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654132430 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe65406ab60 tx=0x7fe64c009840 comp rx=0 tx=0).stop 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 -- 172.21.15.17:0/1880298949 shutdown_connections 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 --2- 172.21.15.17:0/1880298949 >> v2:172.21.15.67:3300/0 conn(0x7fe6540a0280 0x7fe654131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 --2- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654132430 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 --2- 172.21.15.17:0/1880298949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 -- 172.21.15.17:0/1880298949 >> 172.21.15.17:0/1880298949 conn(0x7fe654071270 msgr2=0x7fe6540a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 -- 172.21.15.17:0/1880298949 shutdown_connections 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.045+0000 7fe658d9d640 1 -- 172.21.15.17:0/1880298949 wait complete. 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.046+0000 7fe658d9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.046+0000 7fe658d9d640 10 monclient: monmap: 2024-09-15T02:02:06.060 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.046+0000 7fe658d9d640 1 Processor -- start 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.046+0000 7fe658d9d640 1 -- start start 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.046+0000 7fe658d9d640 10 monclient: init 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe65409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6540a0280 0x7fe6541416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6540a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65411c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe654131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540a09b0 con 0x7fe65409f1e0 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65412fc10 con 0x7fe6540a0280 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:06.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe654120050 con 0x7fe654131b20 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe658d9d640 10 monclient(hunting): authenticate will time out at 1386.044275s 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe652d9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 msgr2=0x7fe65411c7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe652d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65411c7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42090/0 (socket says 172.21.15.17:42090) 2024-09-15T02:02:06.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 -- 172.21.15.17:0/4113099886 learned_addr learned my addr 172.21.15.17:0/4113099886 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): get_auth_request con 0x7fe65409f1e0 auth_method 0 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:06.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient(hunting): handle_auth_done global_id 105027 payload 824 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 -- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 msgr2=0x7fe65411c7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65411c7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 -- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6540a0280 msgr2=0x7fe6541416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6540a0280 0x7fe6541416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient: _finish_hunting 0 2024-09-15T02:02:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 monclient: found mon.noname-a 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.047+0000 7fe65259c640 1 -- 172.21.15.17:0/4113099886 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c009ae0 con 0x7fe65409f1e0 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.049+0000 7fe65259c640 1 -- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 msgr2=0x7fe654141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.049+0000 7fe65259c640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b600 tx=0x7fe6440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:06.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644015070 con 0x7fe65409f1e0 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: dump: 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:06.068 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:06.069 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644003d90 con 0x7fe65409f1e0 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 5 monclient: authenticate success, global_id 105027 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 10 monclient: _renew_subs 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65411e2b0 con 0x7fe65409f1e0 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644006380 con 0x7fe65409f1e0 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:06.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe6427fc640 10 monclient: dump: 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe650d99640 4 set_mon_vals no callback set 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 10 monclient: _renew_subs 2024-09-15T02:02:06.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.050+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65411f020 con 0x7fe65409f1e0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe644010440 con 0x7fe65409f1e0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe658d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe658d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe658d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe6427fc640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 0x7fe63403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6540724a0 con 0x7fe65409f1e0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe64404bfd0 con 0x7fe65409f1e0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe651d9b640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 0x7fe63403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:06.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.051+0000 7fe651d9b640 10 monclient: get_auth_request con 0x7fe6340386f0 auth_method 0 2024-09-15T02:02:06.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.054+0000 7fe651d9b640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 0x7fe63403aba0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe648006fd0 tx=0x7fe648008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:06.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.056+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe64401a410 con 0x7fe65409f1e0 2024-09-15T02:02:06.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.056+0000 7fe6427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:06.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.056+0000 7fe6427fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:06.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.282+0000 7fe658d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:06.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.282+0000 7fe658d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:06.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.282+0000 7fe658d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:06.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.282+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe65411f020 con 0x7fe65409f1e0 2024-09-15T02:02:06.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:06.283+0000 7fe65259c640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b600 tx=0x7fe6440091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fe65411f020 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 2024-09-15T02:02:07.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.111+0000 7fe6427fc640 1 -- 172.21.15.17:0/4113099886 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fe65411f020 con 0x7fe65409f1e0 2024-09-15T02:02:07.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.111+0000 7fe6427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:07.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.111+0000 7fe6427fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:07.114 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 msgr2=0x7fe63403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 0x7fe63403aba0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe648006fd0 tx=0x7fe648008040 comp rx=0 tx=0).stop 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 10 monclient: shutdown 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 msgr2=0x7fe654141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:07.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b600 tx=0x7fe6440091a0 comp rx=0 tx=0).stop 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 shutdown_connections 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe6340386f0 0x7fe63403aba0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe654131b20 0x7fe65411c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6540a0280 0x7fe6541416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 --2- 172.21.15.17:0/4113099886 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe65409f1e0 0x7fe654141180 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 >> 172.21.15.17:0/4113099886 conn(0x7fe654071270 msgr2=0x7fe6540a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 shutdown_connections 2024-09-15T02:02:07.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:07.114+0000 7fe658d9d640 1 -- 172.21.15.17:0/4113099886 wait complete. 2024-09-15T02:02:10.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.236+0000 7f78f6c0f640 10 monclient: get_monmap_and_config 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.236+0000 7f78f6c0f640 10 monclient: build_initial_monmap 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.236+0000 7f78f6c0f640 10 monclient: monmap: 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:10.239 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:10.238388+0000 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:10.238388+0000 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:10.240 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:10.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 1 Processor -- start 2024-09-15T02:02:10.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 1 -- start start 2024-09-15T02:02:10.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 10 monclient: init 2024-09-15T02:02:10.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:10.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f78f009c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.237+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f78f009c580 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78f009c990 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f009f790 con 0x7f78f009c990 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f006f780 con 0x7f78f009c580 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:10.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0069d40 con 0x7f78f009c1b0 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f6c0f640 10 monclient(hunting): authenticate will time out at 1390.235057s 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f49ab640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 msgr2=0x7f78f009a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f49ab640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f51ac640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f78f009c990 0x7f78f006f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:47964/0 (socket says 172.21.15.17:47964) 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42092/0 (socket says 172.21.15.17:42092) 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78f51ac640 1 -- 172.21.15.17:0/4102782344 learned_addr learned my addr 172.21.15.17:0/4102782344 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78f009c580 auth_method 0 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:02:10.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:10.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:10.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.238+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:10.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.240+0000 7f78f51ac640 1 -- 172.21.15.17:0/4102782344 >> v1:172.21.15.17:6789/0 conn(0x7f78f009c990 legacy=0x7f78f006f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:10.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 msgr2=0x7f78f009a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78f009c580 auth_method 0 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:10.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.439+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 msgr2=0x7f78f009a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78f009c580 auth_method 0 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:02:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:10.840+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:02:11.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 msgr2=0x7f78f009a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:11.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:02:11.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78f009c580 auth_method 0 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:11.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:11.641+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): tick 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 monclient(hunting): continuing hunt 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 msgr2=0x7f78f009a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78f009c1b0 0x7f78f009a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 msgr2=0x7f78f006eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.17:3300/0 conn(0x7f78f009c580 0x7f78f006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 -- 172.21.15.17:0/4102782344 >> v1:172.21.15.17:6789/0 conn(0x7f78f009c990 legacy=0x7f78f006f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient: _add_conns ranks=[4,1] 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 0x7f78d4005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f78d4003720 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78d4006260 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 -- 172.21.15.17:0/4102782344 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f0068b60 con 0x7f78d4006260 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 1 -- 172.21.15.17:0/4102782344 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f006f780 con 0x7f78d4003720 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78ee7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 0x7f78d4005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 10 monclient(hunting): get_auth_request con 0x7f78d4003720 auth_method 0 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 msgr2=0x7f78d4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:13.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 msgr2=0x7f78d4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:13.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 0x7f78d4005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:13.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.238+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 0x7f78d4005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.240+0000 7f78effff640 1 -- 172.21.15.17:0/4102782344 >> v1:172.21.15.67:6789/0 conn(0x7f78d4006260 legacy=0x7f78d4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.309+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470245164 0 0) 0x7f78f0068b60 con 0x7f78d4006260 2024-09-15T02:02:13.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.309+0000 7f78ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.309+0000 7f78ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.309+0000 7f78ef7fe640 10 monclient(hunting): my global_id is 144139 2024-09-15T02:02:13.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.309+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e00034e0 con 0x7f78d4006260 2024-09-15T02:02:13.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121687187 0 0) 0x7f78e00034e0 con 0x7f78d4006260 2024-09-15T02:02:13.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 msgr2=0x7f78d4005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:13.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 --2- 172.21.15.17:0/4102782344 >> v2:172.21.15.67:3300/0 conn(0x7f78d4003720 0x7f78d4005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d400b000 con 0x7f78d4006260 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f78dc00d480 con 0x7f78d4006260 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: dump: 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.320 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 >> v1:172.21.15.67:6789/0 conn(0x7f78d4006260 legacy=0x7f78d4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f51ac640 1 -- 172.21.15.17:0/4102782344 reap_dead start 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 0x7f78e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): picked mon.a con 0x7f78e00039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78e0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): picked mon.b con 0x7f78e0004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 0x7f78e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): picked mon.c con 0x7f78e0005180 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f006f780 con 0x7f78e00039d0 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e000c120 con 0x7f78e0004780 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e000c2e0 con 0x7f78e0005180 2024-09-15T02:02:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78ef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f51ac640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 msgr2=0x7f78e000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f51ac640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 0x7f78e000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 0x7f78e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78e0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): get_auth_request con 0x7f78e00039d0 auth_method 0 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78e0004780 auth_method 0 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78effff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:02:13.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 10 monclient(hunting): handle_auth_done global_id 144139 payload 293 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 msgr2=0x7f78e000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 0x7f78e000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 msgr2=0x7f78e0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.314+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78e0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 1 monclient: found mon.a 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e00034e0 con 0x7f78e00039d0 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f49ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f6c0f640 5 monclient: authenticate success, global_id 144139 2024-09-15T02:02:13.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.315+0000 7f78f6c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:13.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78f49ab640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 msgr2=0x7f78e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78f49ab640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 0x7f78e0003dc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f78e40050d0 tx=0x7f78e4009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9299f7ff2a9f9d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4030410 con 0x7f78e00039d0 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: dump: 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.331 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e402c040 con 0x7f78e00039d0 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78f6c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78f6c0f640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 1 -- 172.21.15.17:0/4102782344 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e40307e0 con 0x7f78e00039d0 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78f6c0f640 4 set_mon_vals no callback set 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.318+0000 7f78ef7fe640 10 monclient: dump: 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 10 monclient: shutdown 2024-09-15T02:02:13.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 -- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 msgr2=0x7f78e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 0x7f78e0003dc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f78e40050d0 tx=0x7f78e4009760 comp rx=0 tx=0).stop 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 -- 172.21.15.17:0/4102782344 shutdown_connections 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e0005180 0x7f78e000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78e0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/4102782344 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e00039d0 0x7f78e0003dc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 -- 172.21.15.17:0/4102782344 >> 172.21.15.17:0/4102782344 conn(0x7f78f0071300 msgr2=0x7f78f00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 -- 172.21.15.17:0/4102782344 shutdown_connections 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 1 -- 172.21.15.17:0/4102782344 wait complete. 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 10 monclient: build_initial_monmap 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.319+0000 7f78f6c0f640 10 monclient: monmap: 2024-09-15T02:02:13.334 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.320+0000 7f78f6c0f640 1 Processor -- start 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 -- start start 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient: init 2024-09-15T02:02:13.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 0x7f78f013d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f78e00039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78f013ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f78e0004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f78e0005180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f009cdb0 con 0x7f78e0005180 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00a25c0 con 0x7f78e0004780 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f49ab640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 msgr2=0x7f78f013d760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f013e330 con 0x7f78e00039d0 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f6c0f640 10 monclient(hunting): authenticate will time out at 1393.318651s 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f49ab640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 0x7f78f013d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38422/0 (socket says 172.21.15.17:38422) 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 1 -- 172.21.15.17:0/267996446 learned_addr learned my addr 172.21.15.17:0/267996446 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): get_auth_request con 0x7f78e0005180 auth_method 0 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78f51ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.321+0000 7f78effff640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78f013ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78effff640 10 monclient(hunting): get_auth_request con 0x7f78e0004780 auth_method 0 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 10 monclient(hunting): handle_auth_done global_id 105030 payload 824 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 -- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 msgr2=0x7f78f013d760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 0x7f78f013d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 -- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 msgr2=0x7f78f013ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78f013ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 monclient: found mon.noname-a 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.322+0000 7f78f51ac640 1 -- 172.21.15.17:0/267996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0121eb0 con 0x7f78e0005180 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f51ac640 1 -- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 msgr2=0x7f78f0140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f51ac640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009f50 tx=0x7f78e8001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8004b20 con 0x7f78e0005180 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: dump: 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e8004cc0 con 0x7f78e0005180 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8007530 con 0x7f78e0005180 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 5 monclient: authenticate success, global_id 105030 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 10 monclient: _renew_subs 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f0122190 con 0x7f78e0005180 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ecff9640 10 monclient: dump: 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.341 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.343 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78ee7fc640 4 set_mon_vals no callback set 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 10 monclient: _renew_subs 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.324+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f013d4f0 con 0x7f78e0005180 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78e8007920 con 0x7f78e0005180 2024-09-15T02:02:13.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f6c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f6c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f6c0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f0122570 con 0x7f78e0005180 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78ecff9640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 0x7f78d403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f78e804b960 con 0x7f78e0005180 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f49ab640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 0x7f78d403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.326+0000 7f78f49ab640 10 monclient: get_auth_request con 0x7f78d4038fe0 auth_method 0 2024-09-15T02:02:13.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.329+0000 7f78f49ab640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 0x7f78d403b490 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f78e4003940 tx=0x7f78e4038000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.331+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f78e8004e30 con 0x7f78e0005180 2024-09-15T02:02:13.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.331+0000 7f78ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.331+0000 7f78ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:13.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78f6c0f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:13.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78f6c0f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:13.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78f6c0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78f00724a0 con 0x7f78e0005180 2024-09-15T02:02:13.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78ecff9640 1 -- 172.21.15.17:0/267996446 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f78e8019310 con 0x7f78e0005180 2024-09-15T02:02:13.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:13.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.561+0000 7f78ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:13.565 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:13.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 msgr2=0x7f78d403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 0x7f78d403b490 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f78e4003940 tx=0x7f78e4038000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 10 monclient: shutdown 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 msgr2=0x7f78f0140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009f50 tx=0x7f78e8001a50 comp rx=0 tx=0).stop 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 shutdown_connections 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f78d4038fe0 0x7f78d403b490 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f78e0005180 0x7f78f0140cb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0004780 0x7f78f013ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 --2- 172.21.15.17:0/267996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f78e00039d0 0x7f78f013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 >> 172.21.15.17:0/267996446 conn(0x7f78f0071300 msgr2=0x7f78f009af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 shutdown_connections 2024-09-15T02:02:13.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.563+0000 7f78f6c0f640 1 -- 172.21.15.17:0/267996446 wait complete. 2024-09-15T02:02:13.576 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:02:13.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:02:13.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 10 monclient: monmap: 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:13.672929+0000 2024-09-15T02:02:13.674 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:13.672929+0000 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 1 Processor -- start 2024-09-15T02:02:13.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 1 -- start start 2024-09-15T02:02:13.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.671+0000 7f273ff9d640 10 monclient: init 2024-09-15T02:02:13.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2738131b20 0x7f273809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2738131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27380a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f273809f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27380a0040 con 0x7f27380a0280 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273809bff0 con 0x7f273809f1e0 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27380a14a0 con 0x7f2738131b20 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273ff9d640 10 monclient(hunting): authenticate will time out at 1393.669436s 2024-09-15T02:02:13.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2738131b20 0x7f273809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273d538640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f27380a0280 0x7f2738131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44092/0 (socket says 172.21.15.17:44092) 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2738131b20 0x7f273809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38428/0 (socket says 172.21.15.17:38428) 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273d538640 1 -- 172.21.15.17:0/3195744877 learned_addr learned my addr 172.21.15.17:0/3195744877 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): get_auth_request con 0x7f2738131b20 auth_method 0 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient(hunting): handle_auth_done global_id 105033 payload 274 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 -- 172.21.15.17:0/3195744877 >> v1:172.21.15.67:6789/0 conn(0x7f273809f1e0 legacy=0x7f2738138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:13.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 -- 172.21.15.17:0/3195744877 >> v1:172.21.15.17:6789/0 conn(0x7f27380a0280 legacy=0x7f2738131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:13.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 monclient: found mon.noname-a 2024-09-15T02:02:13.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.672+0000 7f273dd39640 1 -- 172.21.15.17:0/3195744877 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738132600 con 0x7f2738131b20 2024-09-15T02:02:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.675+0000 7f273dd39640 1 -- 172.21.15.17:0/3195744877 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2738131b20 msgr2=0x7f273809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.675+0000 7f273dd39640 1 --2- 172.21.15.17:0/3195744877 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2738131b20 0x7f273809eca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2734009850 tx=0x7f27340090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2385f542513a52d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 1 -- 172.21.15.17:0/3195744877 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734032440 con 0x7f2738131b20 2024-09-15T02:02:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: dump: 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 1 -- 172.21.15.17:0/3195744877 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27340325a0 con 0x7f2738131b20 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 5 monclient: authenticate success, global_id 105033 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 1 -- 172.21.15.17:0/3195744877 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734032940 con 0x7f2738131b20 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 4 set_mon_vals no callback set 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273cd37640 10 monclient: dump: 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.687 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 10 monclient: shutdown 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 -- 172.21.15.17:0/3195744877 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2738131b20 msgr2=0x7f273809eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 --2- 172.21.15.17:0/3195744877 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2738131b20 0x7f273809eca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2734009850 tx=0x7f27340090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 -- 172.21.15.17:0/3195744877 shutdown_connections 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 --2- 172.21.15.17:0/3195744877 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2738131b20 0x7f273809eca0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 -- 172.21.15.17:0/3195744877 >> 172.21.15.17:0/3195744877 conn(0x7f2738071270 msgr2=0x7f27380a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 -- 172.21.15.17:0/3195744877 shutdown_connections 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.676+0000 7f273ff9d640 1 -- 172.21.15.17:0/3195744877 wait complete. 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 10 monclient: monmap: 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:13.689 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 1 Processor -- start 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 1 -- start start 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 10 monclient: init 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.677+0000 7f273ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 0x7f2738099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-a con 0x7f273809f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27380a0280 0x7f273809a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-c con 0x7f27380a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2738131b20 0x7f273809a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2738131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738130c60 con 0x7f273809f1e0 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380a09b0 con 0x7f2738131b20 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273812cc30 con 0x7f27380a0280 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273ff9d640 10 monclient(hunting): authenticate will time out at 1393.675084s 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273d538640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27380a0280 msgr2=0x7f273809a1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273d538640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27380a0280 0x7f273809a1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 0x7f2738099cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 0x7f2738099cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38444/0 (socket says 172.21.15.17:38444) 2024-09-15T02:02:13.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 -- 172.21.15.17:0/1943644280 learned_addr learned my addr 172.21.15.17:0/1943644280 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): get_auth_request con 0x7f273809f1e0 auth_method 0 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient(hunting): handle_auth_done global_id 105036 payload 824 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27380a0280 msgr2=0x7f273809a1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27380a0280 0x7f273809a1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2738131b20 msgr2=0x7f273809a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2738131b20 0x7f273809a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 monclient: found mon.noname-a 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.678+0000 7f273dd39640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734009390 con 0x7f273809f1e0 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273dd39640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 msgr2=0x7f2738099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273dd39640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 0x7f2738099cb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2734009850 tx=0x7f2734032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340319a0 con 0x7f273809f1e0 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: dump: 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734031b00 con 0x7f273809f1e0 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273ff9d640 5 monclient: authenticate success, global_id 105036 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273ff9d640 10 monclient: _renew_subs 2024-09-15T02:02:13.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273812c670 con 0x7f273809f1e0 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273403b400 con 0x7f273809f1e0 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272dffb640 10 monclient: dump: 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.695 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.680+0000 7f272ffff640 4 set_mon_vals no callback set 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 10 monclient: _renew_subs 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273812b8b0 con 0x7f273809f1e0 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 0x7f273809ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738063a40 con 0x7f273812d390 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2734039040 con 0x7f273809f1e0 2024-09-15T02:02:13.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273d538640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 0x7f273809ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273d538640 10 monclient: get_auth_request con 0x7f273812d390 auth_method 0 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273d538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273d538640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.681+0000 7f273d538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f272dffb640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f271c0386f0 0x7f271c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f273406c950 con 0x7f273809f1e0 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273d538640 10 monclient(hunting): handle_auth_done global_id 105039 payload 824 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273d538640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f273812cac0 con 0x7f273812d390 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273e53a640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f271c0386f0 0x7f271c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.682+0000 7f273e53a640 10 monclient: get_auth_request con 0x7f271c0386f0 auth_method 0 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f273d538640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 msgr2=0x7f273809ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f273d538640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 0x7f273809ac70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f27280047d0 tx=0x7f2728007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728005d70 con 0x7f273812d390 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 10 monclient: dump: 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.700 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f27280070f0 con 0x7f273812d390 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 msgr2=0x7f273809ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 0x7f273809ac70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f27280047d0 tx=0x7f2728007e70 comp rx=0 tx=0).stop 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 msgr2=0x7f273809ac70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f272dffb640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273812d390 0x7f273809ac70 secure :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f27280047d0 tx=0x7f2728007e70 comp rx=0 tx=0).stop 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.684+0000 7f273e53a640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f271c0386f0 0x7f271c03aba0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273ff9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273ff9d640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 0x7f27380a4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380724a0 con 0x7f27380a3d50 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 0x7f27380a4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient: get_auth_request con 0x7f27380a3d50 auth_method 0 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.690+0000 7f273d538640 10 monclient(hunting): handle_auth_done global_id 105042 payload 824 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.691+0000 7f273d538640 1 -- 172.21.15.17:0/1943644280 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f273812c800 con 0x7f27380a3d50 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f273d538640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 msgr2=0x7f27380a4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f273d538640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 0x7f27380a4150 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2728007fd0 tx=0x7f2728016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728021630 con 0x7f27380a3d50 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 10 monclient: dump: 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.703 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f2728021790 con 0x7f27380a3d50 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 msgr2=0x7f27380a4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 0x7f27380a4150 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2728007fd0 tx=0x7f2728016000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 msgr2=0x7f27380a4150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.693+0000 7f272dffb640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27380a3d50 0x7f27380a4150 secure :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2728007fd0 tx=0x7f2728016000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 120, 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:02:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:13.718 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f271c0386f0 msgr2=0x7f271c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f271c0386f0 0x7f271c03aba0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730007e10 comp rx=0 tx=0).stop 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 10 monclient: shutdown 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 msgr2=0x7f2738099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 --2- 172.21.15.17:0/1943644280 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f273809f1e0 0x7f2738099cb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2734009850 tx=0x7f2734032000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273e53a640 1 -- 172.21.15.17:0/1943644280 reap_dead start 2024-09-15T02:02:13.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273e53a640 1 -- 172.21.15.17:0/1943644280 reap_dead start 2024-09-15T02:02:13.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 shutdown_connections 2024-09-15T02:02:13.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 >> 172.21.15.17:0/1943644280 conn(0x7f2738071270 msgr2=0x7f27380a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 shutdown_connections 2024-09-15T02:02:13.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.694+0000 7f273ff9d640 1 -- 172.21.15.17:0/1943644280 wait complete. 2024-09-15T02:02:13.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:02:13.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:02:13.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:02:13.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.814+0000 7fde5d100640 10 monclient: get_monmap_and_config 2024-09-15T02:02:13.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.814+0000 7fde5d100640 10 monclient: build_initial_monmap 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.814+0000 7fde5d100640 10 monclient: monmap: 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:13.816577+0000 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:13.816577+0000 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.818 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:13.819 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:13.819 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:13.819 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:13.819 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:13.819 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 Processor -- start 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 -- start start 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient: init 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde5806eb30 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 0x7fde5809bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-b con 0x7fde5809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde5809c4d0 0x7fde58139b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-c con 0x7fde5809c4d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:13.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5809f6a0 con 0x7fde5806eb30 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde58069f40 con 0x7fde5809c4d0 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5809ed40 con 0x7fde5809e970 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5d100640 10 monclient(hunting): authenticate will time out at 1393.812875s 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 msgr2=0x7fde5809bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.815+0000 7fde5659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 0x7fde5809bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde5809c4d0 0x7fde58139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde56d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde5806eb30 0x7fde5806ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:57880/0 (socket says 172.21.15.17:57880) 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde56d9d640 1 -- 172.21.15.17:0/4114315436 learned_addr learned my addr 172.21.15.17:0/4114315436 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): get_auth_request con 0x7fde5809c4d0 auth_method 0 2024-09-15T02:02:13.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient(hunting): handle_auth_done global_id 144145 payload 274 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 -- 172.21.15.17:0/4114315436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 msgr2=0x7fde5809bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 --2- 172.21.15.17:0/4114315436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 0x7fde5809bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 -- 172.21.15.17:0/4114315436 >> v1:172.21.15.67:6789/0 conn(0x7fde5806eb30 legacy=0x7fde5806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 monclient: found mon.noname-c 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:02:13.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.816+0000 7fde5759e640 1 -- 172.21.15.17:0/4114315436 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58069950 con 0x7fde5809c4d0 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.819+0000 7fde5759e640 1 -- 172.21.15.17:0/4114315436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809c4d0 msgr2=0x7fde58139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.819+0000 7fde5759e640 1 --2- 172.21.15.17:0/4114315436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809c4d0 0x7fde58139b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde50009e60 tx=0x7fde50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1bca371c82e0cfe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 1 -- 172.21.15.17:0/4114315436 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50007840 con 0x7fde5809c4d0 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: dump: 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.832 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 1 -- 172.21.15.17:0/4114315436 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5002d050 con 0x7fde5809c4d0 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 1 -- 172.21.15.17:0/4114315436 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50007bd0 con 0x7fde5809c4d0 2024-09-15T02:02:13.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 5 monclient: authenticate success, global_id 144145 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 4 set_mon_vals no callback set 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde55d9b640 10 monclient: dump: 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.834 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 10 monclient: shutdown 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 -- 172.21.15.17:0/4114315436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809c4d0 msgr2=0x7fde58139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 --2- 172.21.15.17:0/4114315436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809c4d0 0x7fde58139b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde50009e60 tx=0x7fde50009780 comp rx=0 tx=0).stop 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 -- 172.21.15.17:0/4114315436 shutdown_connections 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 --2- 172.21.15.17:0/4114315436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809c4d0 0x7fde58139b70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 --2- 172.21.15.17:0/4114315436 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809e970 0x7fde5809bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 -- 172.21.15.17:0/4114315436 >> 172.21.15.17:0/4114315436 conn(0x7fde580712f0 msgr2=0x7fde580a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 -- 172.21.15.17:0/4114315436 shutdown_connections 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.820+0000 7fde5d100640 1 -- 172.21.15.17:0/4114315436 wait complete. 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.821+0000 7fde5d100640 10 monclient: build_initial_monmap 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.821+0000 7fde5d100640 10 monclient: monmap: 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.821+0000 7fde5d100640 1 Processor -- start 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.821+0000 7fde5d100640 1 -- start start 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.821+0000 7fde5d100640 10 monclient: init 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 0x7fde581410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-a con 0x7fde5806eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:13.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809c4d0 0x7fde58134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-c con 0x7fde5809c4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809e970 0x7fde5811eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): picked mon.noname-b con 0x7fde5809e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde58120640 con 0x7fde5806eb30 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5809cd10 con 0x7fde5809e970 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5809b2a0 con 0x7fde5809c4d0 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5d100640 10 monclient(hunting): authenticate will time out at 1393.819166s 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5659c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809c4d0 msgr2=0x7fde58134f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:13.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809c4d0 0x7fde58134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 0x7fde581410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 0x7fde581410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38466/0 (socket says 172.21.15.17:38466) 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 -- 172.21.15.17:0/685305134 learned_addr learned my addr 172.21.15.17:0/685305134 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): get_auth_request con 0x7fde5806eb30 auth_method 0 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809e970 0x7fde5811eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): get_auth_request con 0x7fde5809e970 auth_method 0 2024-09-15T02:02:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient(hunting): handle_auth_done global_id 105045 payload 824 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809c4d0 msgr2=0x7fde58134f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde5809c4d0 0x7fde58134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809e970 msgr2=0x7fde5811eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde5809e970 0x7fde5811eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:02:13.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 monclient: found mon.noname-a 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.822+0000 7fde56d9d640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde50009a20 con 0x7fde5806eb30 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde56d9d640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 msgr2=0x7fde581410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde56d9d640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 0x7fde581410e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b540 tx=0x7fde480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48011840 con 0x7fde5806eb30 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 10 monclient: dump: 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 10 monclient: _finish_auth 0 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.824+0000 7fde4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde480119a0 con 0x7fde5806eb30 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:13.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 5 monclient: authenticate success, global_id 105045 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48011d60 con 0x7fde5806eb30 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 monclient: _renew_subs 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde58141fe0 con 0x7fde5806eb30 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5559a640 4 set_mon_vals no callback set 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 10 monclient: dump: 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 monclient: _renew_subs 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde58121180 con 0x7fde5806eb30 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 0x7fde5811eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde58122210 con 0x7fde58141620 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.825+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde48010590 con 0x7fde5806eb30 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde4effd640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde380386f0 0x7fde3803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fde4804bdd0 con 0x7fde5806eb30 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 0x7fde5811eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient: get_auth_request con 0x7fde58141620 auth_method 0 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5759e640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde380386f0 0x7fde3803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5759e640 10 monclient: get_auth_request con 0x7fde380386f0 auth_method 0 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 10 monclient(hunting): handle_auth_done global_id 144151 payload 824 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.826+0000 7fde5659c640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde58122750 con 0x7fde58141620 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.828+0000 7fde5659c640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 msgr2=0x7fde5811eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.828+0000 7fde5659c640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 0x7fde5811eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde581209e0 tx=0x7fde400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.828+0000 7fde5759e640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde380386f0 0x7fde3803aba0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fde580a3e30 tx=0x7fde50037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.828+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400050f0 con 0x7fde58141620 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 10 monclient: dump: 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.846 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fde40005250 con 0x7fde58141620 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 msgr2=0x7fde5811eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 0x7fde5811eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde581209e0 tx=0x7fde400091a0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 msgr2=0x7fde5811eff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:13.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.829+0000 7fde4effd640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58141620 0x7fde5811eff0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde581209e0 tx=0x7fde400091a0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5d100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5d100640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 0x7fde5811f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5d100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde58063b90 con 0x7fde58135490 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 0x7fde5811f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient: get_auth_request con 0x7fde58135490 auth_method 0 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:13.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 10 monclient(hunting): handle_auth_done global_id 144154 payload 824 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.835+0000 7fde5659c640 1 -- 172.21.15.17:0/685305134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fde580724a0 con 0x7fde58135490 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde5659c640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 msgr2=0x7fde5811f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde5659c640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 0x7fde5811f530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde40014000 tx=0x7fde40015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40010d50 con 0x7fde58135490 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 10 monclient: dump: 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:13.851 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4385 (secure 0 0 0) 0x7fde40019400 con 0x7fde58135490 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 msgr2=0x7fde5811f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 0x7fde5811f530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde40014000 tx=0x7fde40015000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 msgr2=0x7fde5811f530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:13.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.838+0000 7fde4effd640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde58135490 0x7fde5811f530 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde40014000 tx=0x7fde40015000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 120, 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.853 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.854 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.856 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:13.856 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:13.856 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:13.856 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:13.856 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:13.857 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:13.858 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:13.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:13.861 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:13.864 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:13.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:13.866 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde380386f0 msgr2=0x7fde3803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde380386f0 0x7fde3803aba0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fde580a3e30 tx=0x7fde50037000 comp rx=0 tx=0).stop 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 10 monclient: shutdown 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 msgr2=0x7fde581410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 --2- 172.21.15.17:0/685305134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde5806eb30 0x7fde581410e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b540 tx=0x7fde480090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5759e640 1 -- 172.21.15.17:0/685305134 reap_dead start 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5759e640 1 -- 172.21.15.17:0/685305134 reap_dead start 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 shutdown_connections 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 >> 172.21.15.17:0/685305134 conn(0x7fde580712f0 msgr2=0x7fde5806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 shutdown_connections 2024-09-15T02:02:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:13.839+0000 7fde5d100640 1 -- 172.21.15.17:0/685305134 wait complete. 2024-09-15T02:02:13.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:02:13.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:02:33.870 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:02:33.870 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T02:02:33.871 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:02:33.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:02:33.874 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:02:33.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:02:33.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 10 monclient: monmap: 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:33.981970+0000 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:33.981970+0000 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:33.984 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:33.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 1 Processor -- start 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 1 -- start start 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.981+0000 7f072bd9d640 10 monclient: init 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0724131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07240a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0724137e60 0x7f072413a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0724137e60 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07240a0040 con 0x7f0724131b20 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072409bff0 con 0x7f07240a0280 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07240a14a0 con 0x7f0724137e60 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072bd9d640 10 monclient(hunting): authenticate will time out at 1413.978511s 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f0729b39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f0724131b20 0x7f072409eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52156/0 (socket says 172.21.15.17:52156) 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f0729b39640 1 -- 172.21.15.17:0/814693045 learned_addr learned my addr 172.21.15.17:0/814693045 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:33.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072a33a640 1 --2- 172.21.15.17:0/814693045 >> v2:172.21.15.67:3300/0 conn(0x7f0724137e60 0x7f072413a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072a33a640 10 monclient(hunting): get_auth_request con 0x7f0724137e60 auth_method 0 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072a33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.982+0000 7f072a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient(hunting): handle_auth_done global_id 144157 payload 274 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 1 -- 172.21.15.17:0/814693045 >> v1:172.21.15.67:6789/0 conn(0x7f07240a0280 legacy=0x7f0724131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 1 -- 172.21.15.17:0/814693045 >> v1:172.21.15.17:6789/0 conn(0x7f0724131b20 legacy=0x7f072409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient: _finish_hunting 0 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 1 monclient: found mon.noname-c 2024-09-15T02:02:33.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:02:33.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.983+0000 7f072a33a640 1 -- 172.21.15.17:0/814693045 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072409f3b0 con 0x7f0724137e60 2024-09-15T02:02:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072a33a640 1 -- 172.21.15.17:0/814693045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724137e60 msgr2=0x7f072413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072a33a640 1 --2- 172.21.15.17:0/814693045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724137e60 0x7f072413a240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f071c009850 tx=0x7f071c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d90ebea5c9d57d1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:33.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 1 -- 172.21.15.17:0/814693045 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c032440 con 0x7f0724137e60 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: dump: 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:33.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: _finish_auth 0 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 1 -- 172.21.15.17:0/814693045 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c0325a0 con 0x7f0724137e60 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072bd9d640 5 monclient: authenticate success, global_id 144157 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072bd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072bd9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 1 -- 172.21.15.17:0/814693045 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c032940 con 0x7f0724137e60 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f072bd9d640 4 set_mon_vals no callback set 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.986+0000 7f0728b37640 10 monclient: dump: 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:33.995 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 10 monclient: shutdown 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 -- 172.21.15.17:0/814693045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724137e60 msgr2=0x7f072413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:33.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 --2- 172.21.15.17:0/814693045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724137e60 0x7f072413a240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f071c009850 tx=0x7f071c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 -- 172.21.15.17:0/814693045 shutdown_connections 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 --2- 172.21.15.17:0/814693045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724137e60 0x7f072413a240 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 -- 172.21.15.17:0/814693045 >> 172.21.15.17:0/814693045 conn(0x7f0724071270 msgr2=0x7f07240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 -- 172.21.15.17:0/814693045 shutdown_connections 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 1 -- 172.21.15.17:0/814693045 wait complete. 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 10 monclient: build_initial_monmap 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.987+0000 7f072bd9d640 10 monclient: monmap: 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:33.997 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 Processor -- start 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 -- start start 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient: init 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f07240a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724131b20 0x7f0724133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0724131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0724137e60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072409eed0 con 0x7f07240a0280 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0724135f30 con 0x7f0724131b20 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072409b7c0 con 0x7f0724137e60 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072bd9d640 10 monclient(hunting): authenticate will time out at 1413.984801s 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40920/0 (socket says 172.21.15.17:40920) 2024-09-15T02:02:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:42056/0 (socket says 172.21.15.17:42056) 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 1 -- 172.21.15.17:0/2531460595 learned_addr learned my addr 172.21.15.17:0/2531460595 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 10 monclient(hunting): get_auth_request con 0x7f07240a0280 auth_method 0 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f0729b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 10 monclient(hunting): get_auth_request con 0x7f0724137e60 auth_method 0 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.988+0000 7f072a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f072a33a640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 msgr2=0x7f07241341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:34.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f072a33a640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 msgr2=0x7f07241341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f072a33a640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729338640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724131b20 0x7f0724133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f072a33a640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729338640 10 monclient(hunting): get_auth_request con 0x7f0724131b20 auth_method 0 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729338640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient(hunting): handle_auth_done global_id 105048 payload 824 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 msgr2=0x7f07241341d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724131b20 msgr2=0x7f0724133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724131b20 0x7f0724133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient: _finish_hunting 0 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 monclient: found mon.noname-a 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.989+0000 7f0729b39640 1 -- 172.21.15.17:0/2531460595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c009390 con 0x7f07240a0280 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0729b39640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 msgr2=0x7f072409b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0729b39640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f071400ccd0 tx=0x7f07140053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400a9e0 con 0x7f07240a0280 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: dump: 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:34.002 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071400ab60 con 0x7f07240a0280 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f072bd9d640 5 monclient: authenticate success, global_id 105048 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140070f0 con 0x7f07240a0280 2024-09-15T02:02:34.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f0721ffb640 10 monclient: dump: 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:34.004 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f072bd9d640 10 monclient: _renew_subs 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f072bd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07240a09b0 con 0x7f07240a0280 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.991+0000 7f07237fe640 4 set_mon_vals no callback set 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 10 monclient: _renew_subs 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0724135c80 con 0x7f07240a0280 2024-09-15T02:02:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0714007520 con 0x7f07240a0280 2024-09-15T02:02:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f0721ffb640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 0x7f071003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f071404c840 con 0x7f07240a0280 2024-09-15T02:02:34.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.992+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07240724a0 con 0x7f07240a0280 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.993+0000 7f0729338640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 0x7f071003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.993+0000 7f0729338640 10 monclient: get_auth_request con 0x7f0710038740 auth_method 0 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.995+0000 7f0729338640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 0x7f071003abf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f07180037d0 tx=0x7f0718009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.997+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f071401a570 con 0x7f07240a0280 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.997+0000 7f0721ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:34.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:33.997+0000 7f0721ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:34.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.222+0000 7f072bd9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:34.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f072bd9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f072bd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0724136100 con 0x7f07240a0280 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f0721ffb640 1 -- 172.21.15.17:0/2531460595 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f0714010af0 con 0x7f07240a0280 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f0721ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.223+0000 7f0721ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:34.225 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:34.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 msgr2=0x7f071003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:34.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 0x7f071003abf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f07180037d0 tx=0x7f0718009040 comp rx=0 tx=0).stop 2024-09-15T02:02:34.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 10 monclient: shutdown 2024-09-15T02:02:34.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:34.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 msgr2=0x7f072409b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f071400ccd0 tx=0x7f07140053f0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 shutdown_connections 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0710038740 0x7f071003abf0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0724137e60 0x7f07241341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0724131b20 0x7f0724133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 --2- 172.21.15.17:0/2531460595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f07240a0280 0x7f072409b280 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 >> 172.21.15.17:0/2531460595 conn(0x7f0724071270 msgr2=0x7f07240a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 shutdown_connections 2024-09-15T02:02:34.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:34.225+0000 7f072bd9d640 1 -- 172.21.15.17:0/2531460595 wait complete. 2024-09-15T02:02:37.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.345+0000 7f564599d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.345+0000 7f564599d640 10 monclient: build_initial_monmap 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.345+0000 7f564599d640 10 monclient: monmap: 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:37.346324+0000 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:37.346324+0000 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:37.348 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 Processor -- start 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 -- start start 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient: init 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f564006e920 0x7f564012fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): picked mon.noname-a con 0x7f564006e920 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f564006ecf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f564006f0f0 0x7f5640097360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): picked mon.noname-c con 0x7f564006f0f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564009aef0 con 0x7f564006ecf0 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:37.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564006f5b0 con 0x7f564006e920 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564009d720 con 0x7f564006f0f0 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f564599d640 10 monclient(hunting): authenticate will time out at 1417.342839s 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f563effd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f564006e920 0x7f564012fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f56367fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f564006ecf0 0x7f5640096c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52172/0 (socket says 172.21.15.17:52172) 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f56367fc640 1 -- 172.21.15.17:0/3300301156 learned_addr learned my addr 172.21.15.17:0/3300301156 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f563effd640 10 monclient(hunting): get_auth_request con 0x7f564006e920 auth_method 0 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f563effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f563effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:37.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.346+0000 7f563effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient(hunting): handle_auth_done global_id 105051 payload 274 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 1 -- 172.21.15.17:0/3300301156 >> v2:172.21.15.67:3300/0 conn(0x7f564006f0f0 msgr2=0x7f5640097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 1 --2- 172.21.15.17:0/3300301156 >> v2:172.21.15.67:3300/0 conn(0x7f564006f0f0 0x7f5640097360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 1 -- 172.21.15.17:0/3300301156 >> v1:172.21.15.17:6789/0 conn(0x7f564006ecf0 legacy=0x7f5640096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient: _finish_hunting 0 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 1 monclient: found mon.noname-a 2024-09-15T02:02:37.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:37.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.347+0000 7f563effd640 1 -- 172.21.15.17:0/3300301156 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564009b0c0 con 0x7f564006e920 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563effd640 1 -- 172.21.15.17:0/3300301156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006e920 msgr2=0x7f564012fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563effd640 1 --2- 172.21.15.17:0/3300301156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006e920 0x7f564012fbf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f56400689e0 tx=0x7f56300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e5573b27f014b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 1 -- 172.21.15.17:0/3300301156 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630032440 con 0x7f564006e920 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: dump: 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:37.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 5 monclient: authenticate success, global_id 105051 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 1 -- 172.21.15.17:0/3300301156 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300325a0 con 0x7f564006e920 2024-09-15T02:02:37.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 4 set_mon_vals no callback set 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 1 -- 172.21.15.17:0/3300301156 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630032940 con 0x7f564006e920 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f563e7fc640 10 monclient: dump: 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 10 monclient: shutdown 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 -- 172.21.15.17:0/3300301156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006e920 msgr2=0x7f564012fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 --2- 172.21.15.17:0/3300301156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006e920 0x7f564012fbf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f56400689e0 tx=0x7f56300090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 -- 172.21.15.17:0/3300301156 shutdown_connections 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 --2- 172.21.15.17:0/3300301156 >> v2:172.21.15.67:3300/0 conn(0x7f564006f0f0 0x7f5640097360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 --2- 172.21.15.17:0/3300301156 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006e920 0x7f564012fbf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 -- 172.21.15.17:0/3300301156 >> 172.21.15.17:0/3300301156 conn(0x7f56400a1000 msgr2=0x7f56400a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 -- 172.21.15.17:0/3300301156 shutdown_connections 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.350+0000 7f564599d640 1 -- 172.21.15.17:0/3300301156 wait complete. 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.351+0000 7f564599d640 10 monclient: build_initial_monmap 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.351+0000 7f564599d640 10 monclient: monmap: 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:37.364 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.351+0000 7f564599d640 1 Processor -- start 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.351+0000 7f564599d640 1 -- start start 2024-09-15T02:02:37.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient: init 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f564006e920 0x7f564012ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): picked mon.noname-b con 0x7f564006e920 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): picked mon.noname-c con 0x7f564006ecf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 0x7f56401197a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): picked mon.noname-a con 0x7f564006f0f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564011d610 con 0x7f564006f0f0 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564011cd70 con 0x7f564006e920 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:37.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564011d050 con 0x7f564006ecf0 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f564599d640 10 monclient(hunting): authenticate will time out at 1417.348479s 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 0x7f56401197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:42070/0 (socket says 172.21.15.17:42070) 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 -- 172.21.15.17:0/4264775374 learned_addr learned my addr 172.21.15.17:0/4264775374 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 10 monclient(hunting): get_auth_request con 0x7f564006ecf0 auth_method 0 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f564006e920 0x7f564012ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:37.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): get_auth_request con 0x7f564006f0f0 auth_method 0 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): get_auth_request con 0x7f564006e920 auth_method 0 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 msgr2=0x7f564013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 msgr2=0x7f564013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:37.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f56367fc640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient(hunting): handle_auth_done global_id 105054 payload 824 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 msgr2=0x7f564013cf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f564006e920 msgr2=0x7f564012ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f564006e920 0x7f564012ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 monclient: found mon.noname-a 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.352+0000 7f563f7fe640 1 -- 172.21.15.17:0/4264775374 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009390 con 0x7f564006f0f0 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f563f7fe640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 msgr2=0x7f56401197a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f563f7fe640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 0x7f56401197a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f563800bad0 tx=0x7f5638009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800cbc0 con 0x7f564006f0f0 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: dump: 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.370 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800cd20 con 0x7f564006f0f0 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 5 monclient: authenticate success, global_id 105054 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 10 monclient: _renew_subs 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564011d330 con 0x7f564006f0f0 2024-09-15T02:02:37.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638014660 con 0x7f564006f0f0 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f56377fe640 10 monclient: dump: 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f563dffb640 4 set_mon_vals no callback set 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 10 monclient: _renew_subs 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.355+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564011be10 con 0x7f564006f0f0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5638014a90 con 0x7f564006f0f0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f564599d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f564599d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f564599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56400724a0 con 0x7f564006f0f0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f56377fe640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 0x7f562003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f563804c700 con 0x7f564006f0f0 2024-09-15T02:02:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f563effd640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 0x7f562003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.356+0000 7f563effd640 10 monclient: get_auth_request con 0x7f5620038740 auth_method 0 2024-09-15T02:02:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.359+0000 7f563effd640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 0x7f562003abf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5630031000 tx=0x7f5630032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.361+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f56380112c0 con 0x7f564006f0f0 2024-09-15T02:02:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.361+0000 7f56377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:37.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.361+0000 7f56377fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f564599d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f564599d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f564599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5640067290 con 0x7f564006f0f0 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f56377fe640 1 -- 172.21.15.17:0/4264775374 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f563801b310 con 0x7f564006f0f0 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f56377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:37.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.590+0000 7f56377fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:37.593 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:37.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 msgr2=0x7f562003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:37.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 0x7f562003abf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5630031000 tx=0x7f5630032000 comp rx=0 tx=0).stop 2024-09-15T02:02:37.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 10 monclient: shutdown 2024-09-15T02:02:37.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 msgr2=0x7f56401197a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 0x7f56401197a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f563800bad0 tx=0x7f5638009780 comp rx=0 tx=0).stop 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 shutdown_connections 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5620038740 0x7f562003abf0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f564006f0f0 0x7f56401197a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f564006ecf0 0x7f564013cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 --2- 172.21.15.17:0/4264775374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f564006e920 0x7f564012ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 >> 172.21.15.17:0/4264775374 conn(0x7f56400a1000 msgr2=0x7f5640098a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 shutdown_connections 2024-09-15T02:02:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:37.592+0000 7f564599d640 1 -- 172.21.15.17:0/4264775374 wait complete. 2024-09-15T02:02:40.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:40.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.713+0000 7f1b92294640 10 monclient: get_monmap_and_config 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.713+0000 7f1b92294640 10 monclient: build_initial_monmap 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.713+0000 7f1b92294640 10 monclient: monmap: 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:40.714008+0000 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:40.714008+0000 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:40.716 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:40.716 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:40.716 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.713+0000 7f1b92294640 1 Processor -- start 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 1 -- start start 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient: init 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T02:02:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1b8c0a58b0 0x7f1b8c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b8c0980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b8c136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0a01e0 con 0x7f1b8c0980a0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0a1580 con 0x7f1b8c136dd0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0a5c80 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b92294640 10 monclient(hunting): authenticate will time out at 1420.710548s 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1b8c0a58b0 0x7f1b8c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:40.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1b8c0980a0 0x7f1b8c0984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52184/0 (socket says 172.21.15.17:52184) 2024-09-15T02:02:40.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8b7fe640 1 -- 172.21.15.17:0/3544651036 learned_addr learned my addr 172.21.15.17:0/3544651036 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:40.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): get_auth_request con 0x7f1b8c0a58b0 auth_method 0 2024-09-15T02:02:40.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:40.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:40.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_done global_id 105057 payload 274 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 1 -- 172.21.15.17:0/3544651036 >> v1:172.21.15.67:6789/0 conn(0x7f1b8c136dd0 legacy=0x7f1b8c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 1 -- 172.21.15.17:0/3544651036 >> v1:172.21.15.17:6789/0 conn(0x7f1b8c0980a0 legacy=0x7f1b8c0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.714+0000 7f1b8bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.715+0000 7f1b8bfff640 1 monclient: found mon.noname-a 2024-09-15T02:02:40.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.715+0000 7f1b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:40.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.715+0000 7f1b8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.715+0000 7f1b8bfff640 1 -- 172.21.15.17:0/3544651036 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c098bb0 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.717+0000 7f1b8bfff640 1 -- 172.21.15.17:0/3544651036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0a58b0 msgr2=0x7f1b8c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.717+0000 7f1b8bfff640 1 --2- 172.21.15.17:0/3544651036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c132040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009850 tx=0x7f1b7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bba740ccd4a4bfb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 1 -- 172.21.15.17:0/3544651036 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c032440 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: dump: 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:40.729 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: _finish_auth 0 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 1 -- 172.21.15.17:0/3544651036 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c0325a0 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 5 monclient: authenticate success, global_id 105057 2024-09-15T02:02:40.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 1 -- 172.21.15.17:0/3544651036 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c032940 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b8affd640 10 monclient: dump: 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:40.731 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 4 set_mon_vals no callback set 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 10 monclient: shutdown 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:40.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 -- 172.21.15.17:0/3544651036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0a58b0 msgr2=0x7f1b8c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 --2- 172.21.15.17:0/3544651036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c132040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009850 tx=0x7f1b7c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 -- 172.21.15.17:0/3544651036 shutdown_connections 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 --2- 172.21.15.17:0/3544651036 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c132040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 -- 172.21.15.17:0/3544651036 >> 172.21.15.17:0/3544651036 conn(0x7f1b8c070ed0 msgr2=0x7f1b8c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 -- 172.21.15.17:0/3544651036 shutdown_connections 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.718+0000 7f1b92294640 1 -- 172.21.15.17:0/3544651036 wait complete. 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.719+0000 7f1b92294640 10 monclient: build_initial_monmap 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.719+0000 7f1b92294640 10 monclient: monmap: 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:40.733 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.719+0000 7f1b92294640 1 Processor -- start 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.719+0000 7f1b92294640 1 -- start start 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.719+0000 7f1b92294640 10 monclient: init 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-b con 0x7f1b8c0a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): picked mon.noname-c con 0x7f1b8c136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:40.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c135300 con 0x7f1b8c0980a0 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c135020 con 0x7f1b8c0a58b0 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c135eb0 con 0x7f1b8c136dd0 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b92294640 10 monclient(hunting): authenticate will time out at 1420.716286s 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:40.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40992/0 (socket says 172.21.15.17:40992) 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 -- 172.21.15.17:0/1271538804 learned_addr learned my addr 172.21.15.17:0/1271538804 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): get_auth_request con 0x7f1b8c0980a0 auth_method 0 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 10 monclient(hunting): get_auth_request con 0x7f1b8c136dd0 auth_method 0 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:40.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): get_auth_request con 0x7f1b8c0a58b0 auth_method 0 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 msgr2=0x7f1b8c1335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 msgr2=0x7f1b8c1335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b90831640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:40.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient(hunting): handle_auth_done global_id 105060 payload 824 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 msgr2=0x7f1b8c1335a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b8c0a58b0 msgr2=0x7f1b8c133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 monclient: found mon.noname-a 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.720+0000 7f1b8bfff640 1 -- 172.21.15.17:0/1271538804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c009390 con 0x7f1b8c0980a0 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b8bfff640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 msgr2=0x7f1b8c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b8bfff640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009360 tx=0x7f1b7c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c03c960 con 0x7f1b8c0980a0 2024-09-15T02:02:40.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: dump: 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:40.740 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: _finish_auth 0 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c03cac0 con 0x7f1b8c0980a0 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c03ce80 con 0x7f1b8c0980a0 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b73fff640 10 monclient: dump: 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.741 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 5 monclient: authenticate success, global_id 105060 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 10 monclient: _renew_subs 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b8a7fc640 4 set_mon_vals no callback set 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c133e50 con 0x7f1b8c0980a0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 10 monclient: _renew_subs 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.723+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c09e500 con 0x7f1b8c0980a0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1b7c039040 con 0x7f1b8c0980a0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b73fff640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 0x7f1b7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1b7c042070 con 0x7f1b8c0980a0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b92294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b92294640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b92294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c0724a0 con 0x7f1b8c0980a0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b8b7fe640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 0x7f1b7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.724+0000 7f1b8b7fe640 10 monclient: get_auth_request con 0x7f1b78038740 auth_method 0 2024-09-15T02:02:40.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.727+0000 7f1b8b7fe640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 0x7f1b7803abf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c09ead0 tx=0x7f1b80006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.729+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1b7c04b020 con 0x7f1b8c0980a0 2024-09-15T02:02:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.729+0000 7f1b73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:40.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.729+0000 7f1b73fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b92294640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b92294640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b92294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1b8c135540 con 0x7f1b8c0980a0 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b73fff640 1 -- 172.21.15.17:0/1271538804 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f1b7c037070 con 0x7f1b8c0980a0 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.957+0000 7f1b73fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:40.959 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:40.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 msgr2=0x7f1b7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 0x7f1b7803abf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c09ead0 tx=0x7f1b80006f50 comp rx=0 tx=0).stop 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 10 monclient: shutdown 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 msgr2=0x7f1b8c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009360 tx=0x7f1b7c032000 comp rx=0 tx=0).stop 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 shutdown_connections 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b78038740 0x7f1b7803abf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b8c136dd0 0x7f1b8c1335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b8c0a58b0 0x7f1b8c133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 --2- 172.21.15.17:0/1271538804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b8c0980a0 0x7f1b8c132b20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 >> 172.21.15.17:0/1271538804 conn(0x7f1b8c070ed0 msgr2=0x7f1b8c1391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 shutdown_connections 2024-09-15T02:02:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:40.959+0000 7f1b92294640 1 -- 172.21.15.17:0/1271538804 wait complete. 2024-09-15T02:02:40.971 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:02:40.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:02:40.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.068+0000 7f09cde3f640 10 monclient: get_monmap_and_config 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.068+0000 7f09cde3f640 10 monclient: build_initial_monmap 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.068+0000 7f09cde3f640 10 monclient: monmap: 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:41.069576+0000 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:41.069576+0000 2024-09-15T02:02:41.071 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:41.072 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 1 Processor -- start 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 1 -- start start 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 10 monclient: init 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.069+0000 7f09cde3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-b con 0x7f09c809b180 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09c806eca0 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:41.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09c809d260 0x7f09c8137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-c con 0x7f09c809d260 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c8067970 con 0x7f09c806eca0 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c8099440 con 0x7f09c809d260 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8067ff0 con 0x7f09c809b180 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09cde3f640 10 monclient(hunting): authenticate will time out at 1421.066115s 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c6ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f09c806eca0 0x7f09c809cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:52198/0 (socket says 172.21.15.17:52198) 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c6ffd640 1 -- 172.21.15.17:0/1268731747 learned_addr learned my addr 172.21.15.17:0/1268731747 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09c809b180 auth_method 0 2024-09-15T02:02:41.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 1 -- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 msgr2=0x7f09c809b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 1 -- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 msgr2=0x7f09c809b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1268731747 >> v2:172.21.15.67:3300/0 conn(0x7f09c809d260 0x7f09c8137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): get_auth_request con 0x7f09c809d260 auth_method 0 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient(hunting): handle_auth_done global_id 144169 payload 274 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 -- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 msgr2=0x7f09c809b550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 -- 172.21.15.17:0/1268731747 >> v1:172.21.15.17:6789/0 conn(0x7f09c806eca0 legacy=0x7f09c809cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 monclient: found mon.noname-c 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:02:41.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.070+0000 7f09c7fff640 1 -- 172.21.15.17:0/1268731747 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8099c50 con 0x7f09c809d260 2024-09-15T02:02:41.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.073+0000 7f09c7fff640 1 -- 172.21.15.17:0/1268731747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809d260 msgr2=0x7f09c8137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.073+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809d260 0x7f09c8137780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009e60 tx=0x7f09c0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769e7616e03da991 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 1 -- 172.21.15.17:0/1268731747 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0007760 con 0x7f09c809d260 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: dump: 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.086 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 1 -- 172.21.15.17:0/1268731747 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c002d050 con 0x7f09c809d260 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 5 monclient: authenticate success, global_id 144169 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 4 set_mon_vals no callback set 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 1 -- 172.21.15.17:0/1268731747 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0007af0 con 0x7f09c809d260 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09c67fc640 10 monclient: dump: 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 10 monclient: shutdown 2024-09-15T02:02:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 -- 172.21.15.17:0/1268731747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809d260 msgr2=0x7f09c8137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809d260 0x7f09c8137780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009e60 tx=0x7f09c0009780 comp rx=0 tx=0).stop 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 -- 172.21.15.17:0/1268731747 shutdown_connections 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809d260 0x7f09c8137780 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1268731747 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c809b180 0x7f09c809b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 -- 172.21.15.17:0/1268731747 >> 172.21.15.17:0/1268731747 conn(0x7f09c80a2100 msgr2=0x7f09c80a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 -- 172.21.15.17:0/1268731747 shutdown_connections 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.074+0000 7f09cde3f640 1 -- 172.21.15.17:0/1268731747 wait complete. 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.075+0000 7f09cde3f640 10 monclient: build_initial_monmap 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.075+0000 7f09cde3f640 10 monclient: monmap: 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:41.089 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.075+0000 7f09cde3f640 1 Processor -- start 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.075+0000 7f09cde3f640 1 -- start start 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.075+0000 7f09cde3f640 10 monclient: init 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-c con 0x7f09c806eca0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809b180 0x7f09c81309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-b con 0x7f09c809b180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 0x7f09c811a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): picked mon.noname-a con 0x7f09c809d260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8098060 con 0x7f09c809d260 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c811dae0 con 0x7f09c809b180 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c811ddc0 con 0x7f09c806eca0 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09cde3f640 10 monclient(hunting): authenticate will time out at 1421.072255s 2024-09-15T02:02:41.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:42118/0 (socket says 172.21.15.17:42118) 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 learned_addr learned my addr 172.21.15.17:0/1964838360 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09c806eca0 auth_method 0 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 0x7f09c811a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): get_auth_request con 0x7f09c809d260 auth_method 0 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 msgr2=0x7f09c813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:02:41.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 msgr2=0x7f09c813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809b180 0x7f09c81309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09c809b180 auth_method 0 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient(hunting): handle_auth_done global_id 105063 payload 824 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 msgr2=0x7f09c813cb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09c806eca0 0x7f09c813cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809b180 msgr2=0x7f09c81309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09c809b180 0x7f09c81309c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 monclient: found mon.noname-a 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.076+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009a20 con 0x7f09c809d260 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 msgr2=0x7f09c811a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09c7fff640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 0x7f09c811a550 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f09c0031000 tx=0x7f09c0037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031c10 con 0x7f09c809d260 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: dump: 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.095 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 5 monclient: authenticate success, global_id 105063 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 10 monclient: _renew_subs 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c811bfb0 con 0x7f09c809d260 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0040070 con 0x7f09c809d260 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0045430 con 0x7f09c809d260 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09bf7fe640 10 monclient: dump: 2024-09-15T02:02:41.096 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 10 monclient: _renew_subs 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09c5ffb640 4 set_mon_vals no callback set 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.079+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c813d8b0 con 0x7f09c809d260 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09cde3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 0x7f09c811aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c813daa0 con 0x7f09c811cb80 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f09c0042030 con 0x7f09c809d260 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 0x7f09c811aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient: get_auth_request con 0x7f09c811cb80 auth_method 0 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09bf7fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a8038740 0x7f09a803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f09c006e050 con 0x7f09c809d260 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 105066 payload 824 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09c811df80 con 0x7f09c811cb80 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c6ffd640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a8038740 0x7f09a803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.080+0000 7f09c6ffd640 10 monclient: get_auth_request con 0x7f09a8038740 auth_method 0 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 msgr2=0x7f09c811aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 0x7f09c811aa90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09b800f790 tx=0x7f09b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800fe60 con 0x7f09c811cb80 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.082+0000 7f09bf7fe640 10 monclient: dump: 2024-09-15T02:02:41.099 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f09b800dad0 con 0x7f09c811cb80 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:41.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:41.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 msgr2=0x7f09c811aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 0x7f09c811aa90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09b800f790 tx=0x7f09b8008040 comp rx=0 tx=0).stop 2024-09-15T02:02:41.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 msgr2=0x7f09c811aa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:41.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09bf7fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c811cb80 0x7f09c811aa90 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09b800f790 tx=0x7f09b8008040 comp rx=0 tx=0).stop 2024-09-15T02:02:41.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.083+0000 7f09c6ffd640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a8038740 0x7f09a803abf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007fd0 tx=0x7f09b0007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.088+0000 7f09cde3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.088+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 0x7f09c811afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.088+0000 7f09cde3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.088+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8063870 con 0x7f09c8130f00 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 0x7f09c811afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient: get_auth_request con 0x7f09c8130f00 auth_method 0 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 105069 payload 824 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.089+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09c80724a0 con 0x7f09c8130f00 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09c77fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 msgr2=0x7f09c811afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09c77fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 0x7f09c811afd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f09b801a000 tx=0x7f09b801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8022920 con 0x7f09c8130f00 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 10 monclient: dump: 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f09b8026070 con 0x7f09c8130f00 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 msgr2=0x7f09c811afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 0x7f09c811afd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f09b801a000 tx=0x7f09b801d000 comp rx=0 tx=0).stop 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 msgr2=0x7f09c811afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:41.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.091+0000 7f09bf7fe640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c8130f00 0x7f09c811afd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:41.106 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:41.107 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:41.108 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:41.109 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:41.110 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:41.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:41.113 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:41.115 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:41.116 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.117 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:02:41.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:41.119 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a8038740 msgr2=0x7f09a803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f09a8038740 0x7f09a803abf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007fd0 tx=0x7f09b0007f00 comp rx=0 tx=0).stop 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 10 monclient: shutdown 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 msgr2=0x7f09c811a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 --2- 172.21.15.17:0/1964838360 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09c809d260 0x7f09c811a550 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f09c0031000 tx=0x7f09c0037000 comp rx=0 tx=0).stop 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 reap_dead start 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09c7fff640 1 -- 172.21.15.17:0/1964838360 reap_dead start 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 shutdown_connections 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 >> 172.21.15.17:0/1964838360 conn(0x7f09c80a2100 msgr2=0x7f09c8096a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 shutdown_connections 2024-09-15T02:02:41.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.092+0000 7f09cde3f640 1 -- 172.21.15.17:0/1964838360 wait complete. 2024-09-15T02:02:41.122 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:41.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:02:41.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:02:41.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 10 monclient: get_monmap_and_config 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 10 monclient: build_initial_monmap 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 10 monclient: monmap: 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:41.218855+0000 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:41.218855+0000 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:41.220 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:41.221 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:41.221 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:41.221 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 1 Processor -- start 2024-09-15T02:02:41.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 1 -- start start 2024-09-15T02:02:41.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.218+0000 7f4e650a8640 10 monclient: init 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6009ec30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e6009f000 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e60069590 0x7f4e60069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4e60069590 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6009c6f0 con 0x7f4e6009f000 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6009f3d0 con 0x7f4e60069590 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6006a650 con 0x7f4e6009ec30 2024-09-15T02:02:41.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e650a8640 10 monclient(hunting): authenticate will time out at 1421.215159s 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e60069590 0x7f4e60069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e60069590 0x7f4e60069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41036/0 (socket says 172.21.15.17:41036) 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:42130/0 (socket says 172.21.15.17:42130) 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 -- 172.21.15.17:0/1299493096 learned_addr learned my addr 172.21.15.17:0/1299493096 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): get_auth_request con 0x7f4e60069590 auth_method 0 2024-09-15T02:02:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request con 0x7f4e6009ec30 auth_method 0 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 msgr2=0x7f4e60131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 msgr2=0x7f4e60131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_done global_id 105072 payload 274 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 -- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 msgr2=0x7f4e60131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 -- 172.21.15.17:0/1299493096 >> v1:172.21.15.67:6789/0 conn(0x7f4e6009f000 legacy=0x7f4e60068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient: _finish_hunting 0 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 monclient: found mon.noname-a 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.219+0000 7f4e5f59e640 1 -- 172.21.15.17:0/1299493096 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600a13a0 con 0x7f4e60069590 2024-09-15T02:02:41.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5f59e640 1 -- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e60069590 msgr2=0x7f4e60069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5f59e640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e60069590 0x7f4e60069a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009f20 tx=0x7f4e58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c283fdd2c50d6e5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 1 -- 172.21.15.17:0/1299493096 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58007610 con 0x7f4e60069590 2024-09-15T02:02:41.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: dump: 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: _finish_auth 0 2024-09-15T02:02:41.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e650a8640 5 monclient: authenticate success, global_id 105072 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e650a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 1 -- 172.21.15.17:0/1299493096 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5802d050 con 0x7f4e60069590 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 1 -- 172.21.15.17:0/1299493096 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e580079a0 con 0x7f4e60069590 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e650a8640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e650a8640 4 set_mon_vals no callback set 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.222+0000 7f4e5dd9b640 10 monclient: dump: 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.235 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 10 monclient: shutdown 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 -- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e60069590 msgr2=0x7f4e60069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e60069590 0x7f4e60069a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009f20 tx=0x7f4e58009840 comp rx=0 tx=0).stop 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 -- 172.21.15.17:0/1299493096 shutdown_connections 2024-09-15T02:02:41.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e60069590 0x7f4e60069a10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 --2- 172.21.15.17:0/1299493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009ec30 0x7f4e60131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 -- 172.21.15.17:0/1299493096 >> 172.21.15.17:0/1299493096 conn(0x7f4e60071320 msgr2=0x7f4e600a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 -- 172.21.15.17:0/1299493096 shutdown_connections 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 1 -- 172.21.15.17:0/1299493096 wait complete. 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 10 monclient: build_initial_monmap 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.223+0000 7f4e650a8640 10 monclient: monmap: 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.237 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 Processor -- start 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 -- start start 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient: init 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:41.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4e60069590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 0x7f4e6012eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4e6009ec30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e6009f000 0x7f4e60098e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6009f000 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6011c2a0 con 0x7f4e6009ec30 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60141310 con 0x7f4e6009f000 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6009a8b0 con 0x7f4e60069590 2024-09-15T02:02:41.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e650a8640 10 monclient(hunting): authenticate will time out at 1421.220749s 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 0x7f4e6012eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:42136/0 (socket says 172.21.15.17:42136) 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): get_auth_request con 0x7f4e6009ec30 auth_method 0 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 learned_addr learned my addr 172.21.15.17:0/2109023510 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request con 0x7f4e60069590 auth_method 0 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5f59e640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e6009f000 0x7f4e60098e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5f59e640 10 monclient(hunting): get_auth_request con 0x7f4e6009f000 auth_method 0 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.224+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 msgr2=0x7f4e60140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 msgr2=0x7f4e60140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 10 monclient(hunting): handle_auth_done global_id 105075 payload 824 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 msgr2=0x7f4e60140dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60069590 0x7f4e60140dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e6009f000 msgr2=0x7f4e60098e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e6009f000 0x7f4e60098e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 10 monclient: _finish_hunting 0 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 monclient: found mon.noname-a 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.225+0000 7f4e5e59c640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58009ae0 con 0x7f4e6009ec30 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e5e59c640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 msgr2=0x7f4e6012eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e5e59c640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 0x7f4e6012eea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800c4e0 tx=0x7f4e48007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e480117a0 con 0x7f4e6009ec30 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: dump: 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.243 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: _finish_auth 0 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 5 monclient: authenticate success, global_id 105075 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48011920 con 0x7f4e6009ec30 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48011ce0 con 0x7f4e6009ec30 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 10 monclient: _renew_subs 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6011bfc0 con 0x7f4e6009ec30 2024-09-15T02:02:41.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e56ffd640 10 monclient: dump: 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:41.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e5d59a640 4 set_mon_vals no callback set 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 10 monclient: _renew_subs 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.227+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e60141b10 con 0x7f4e6009ec30 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e650a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60063b20 con 0x7f4e6009b400 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e6009b400 auth_method 0 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4e48012630 con 0x7f4e6009ec30 2024-09-15T02:02:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 0x7f4e4003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.228+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4e4804c260 con 0x7f4e6009ec30 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.229+0000 7f4e5f59e640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 0x7f4e4003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.229+0000 7f4e5f59e640 10 monclient: get_auth_request con 0x7f4e40038790 auth_method 0 2024-09-15T02:02:41.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.231+0000 7f4e5f59e640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 0x7f4e4003ac40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4e6006f780 tx=0x7f4e58036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e6009b400 auth_method 0 2024-09-15T02:02:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.429+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:02:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e6009b400 auth_method 0 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:41.830+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e6009b400 auth_method 0 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:42.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:42.631+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:02:44.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.225+0000 7f4e55ffb640 10 monclient: tick 2024-09-15T02:02:44.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.225+0000 7f4e55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:44.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.225+0000 7f4e55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.233+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.233+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e6009b400 auth_method 0 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.233+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.233+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.233+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.234+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.234+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.234+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.234+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_done global_id 154100 payload 824 2024-09-15T02:02:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.234+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e6009acd0 con 0x7f4e6009b400 2024-09-15T02:02:44.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e50017990 tx=0x7f4e50008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50017e50 con 0x7f4e6009b400 2024-09-15T02:02:44.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.236+0000 7f4e56ffd640 10 monclient: dump: 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.239 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4e5001c070 con 0x7f4e6009b400 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e50017990 tx=0x7f4e50008040 comp rx=0 tx=0).stop 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 msgr2=0x7f4e600298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.237+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e6009b400 0x7f4e600298c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e650a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e650a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6011c150 con 0x7f4e60029e00 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e60029e00 auth_method 0 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.243+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.244+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.244+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.244+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.244+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_done global_id 154103 payload 824 2024-09-15T02:02:44.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.244+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e60130ca0 con 0x7f4e60029e00 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 msgr2=0x7f4e6002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e50017990 tx=0x7f4e50022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 msgr2=0x7f4e6002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 msgr2=0x7f4e6002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e50017990 tx=0x7f4e50022000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e50017990 tx=0x7f4e50022000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 msgr2=0x7f4e6002a200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 0x7f4e400435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 msgr2=0x7f4e6002a200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e60029e00 0x7f4e6002a200 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.246+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40043c00 con 0x7f4e40041210 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 0x7f4e400435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient: get_auth_request con 0x7f4e40041210 auth_method 0 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 10 monclient(hunting): handle_auth_done global_id 154106 payload 824 2024-09-15T02:02:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.247+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e5001c070 con 0x7f4e40041210 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e5ed9d640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 msgr2=0x7f4e400435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e5ed9d640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 0x7f4e400435f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4e50015710 tx=0x7f4e5002b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50034050 con 0x7f4e40041210 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.249+0000 7f4e56ffd640 10 monclient: dump: 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f4e50035040 con 0x7f4e40041210 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 msgr2=0x7f4e400435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 0x7f4e400435f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4e50015710 tx=0x7f4e5002b000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 msgr2=0x7f4e400435f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e56ffd640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e40041210 0x7f4e400435f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e5f59e640 1 -- 172.21.15.17:0/2109023510 reap_dead start 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:44.254 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.255 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.256 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.258 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.262 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.263 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:44.265 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:44.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.266 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:44.266 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 msgr2=0x7f4e4003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 0x7f4e4003ac40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4e6006f780 tx=0x7f4e58036000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 10 monclient: shutdown 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 msgr2=0x7f4e6012eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 0x7f4e6012eea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800c4e0 tx=0x7f4e48007e10 comp rx=0 tx=0).stop 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 shutdown_connections 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e40038790 0x7f4e4003ac40 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 --2- 172.21.15.17:0/2109023510 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e6009ec30 0x7f4e6012eea0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 >> 172.21.15.17:0/2109023510 conn(0x7f4e60071320 msgr2=0x7f4e601311f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 shutdown_connections 2024-09-15T02:02:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.250+0000 7f4e650a8640 1 -- 172.21.15.17:0/2109023510 wait complete. 2024-09-15T02:02:44.268 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:44.268 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:02:44.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.373+0000 7f6184f5d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.373+0000 7f6184f5d640 10 monclient: build_initial_monmap 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.373+0000 7f6184f5d640 10 monclient: monmap: 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:44.374068+0000 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:44.374068+0000 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.375 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:44.376 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:44.376 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:44.376 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:44.376 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:44.376 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.373+0000 7f6184f5d640 1 Processor -- start 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 1 -- start start 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient: init 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 0x7f6180132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f61800a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:44.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61800980a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6180136dd0 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800a01e0 con 0x7f6180136dd0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800a1580 con 0x7f61800980a0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61800a5c80 con 0x7f61800a58b0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f6184f5d640 10 monclient(hunting): authenticate will time out at 1424.370655s 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 0x7f6180132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617ed9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f6180136dd0 0x7f61801391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51494/0 (socket says 172.21.15.17:51494) 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617ed9d640 1 -- 172.21.15.17:0/3939145111 learned_addr learned my addr 172.21.15.17:0/3939145111 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): get_auth_request con 0x7f61800a58b0 auth_method 0 2024-09-15T02:02:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.374+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 10 monclient(hunting): handle_auth_done global_id 154109 payload 274 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 1 -- 172.21.15.17:0/3939145111 >> v1:172.21.15.67:6789/0 conn(0x7f61800980a0 legacy=0x7f6180098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 1 -- 172.21.15.17:0/3939145111 >> v1:172.21.15.17:6789/0 conn(0x7f6180136dd0 legacy=0x7f61801391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 10 monclient: _finish_hunting 0 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 1 monclient: found mon.noname-b 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.375+0000 7f617e59c640 1 -- 172.21.15.17:0/3939145111 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61801326c0 con 0x7f61800a58b0 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.377+0000 7f617e59c640 1 -- 172.21.15.17:0/3939145111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 msgr2=0x7f6180132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.377+0000 7f617e59c640 1 --2- 172.21.15.17:0/3939145111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 0x7f6180132040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6170009880 tx=0x7f61700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f946d602ce0137b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 1 -- 172.21.15.17:0/3939145111 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170032440 con 0x7f61800a58b0 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: dump: 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.387 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: _finish_auth 0 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 1 -- 172.21.15.17:0/3939145111 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61700325a0 con 0x7f61800a58b0 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 5 monclient: authenticate success, global_id 154109 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:44.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 1 -- 172.21.15.17:0/3939145111 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170032940 con 0x7f61800a58b0 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 4 set_mon_vals no callback set 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f617dd9b640 10 monclient: dump: 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 10 monclient: shutdown 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 -- 172.21.15.17:0/3939145111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 msgr2=0x7f6180132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3939145111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 0x7f6180132040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6170009880 tx=0x7f61700090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 -- 172.21.15.17:0/3939145111 shutdown_connections 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3939145111 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f61800a58b0 0x7f6180132040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 -- 172.21.15.17:0/3939145111 >> 172.21.15.17:0/3939145111 conn(0x7f6180070ed0 msgr2=0x7f61800a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 -- 172.21.15.17:0/3939145111 shutdown_connections 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.378+0000 7f6184f5d640 1 -- 172.21.15.17:0/3939145111 wait complete. 2024-09-15T02:02:44.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.379+0000 7f6184f5d640 10 monclient: build_initial_monmap 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.379+0000 7f6184f5d640 10 monclient: monmap: 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.379+0000 7f6184f5d640 1 Processor -- start 2024-09-15T02:02:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.379+0000 7f6184f5d640 1 -- start start 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.379+0000 7f6184f5d640 10 monclient: init 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f61800980a0 0x7f6180132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f61800980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61800a58b0 0x7f6180133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f61800a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 0x7f61801335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6180136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61801351f0 con 0x7f61800980a0 2024-09-15T02:02:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618009ee70 con 0x7f61800a58b0 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6180134e80 con 0x7f6180136dd0 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6184f5d640 10 monclient(hunting): authenticate will time out at 1424.376165s 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f61800980a0 0x7f6180132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 0x7f61801335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f61800980a0 0x7f6180132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54136/0 (socket says 172.21.15.17:54136) 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 1 -- 172.21.15.17:0/3180225019 learned_addr learned my addr 172.21.15.17:0/3180225019 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): get_auth_request con 0x7f61800980a0 auth_method 0 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): get_auth_request con 0x7f6180136dd0 auth_method 0 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6177fff640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61800a58b0 0x7f6180133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6177fff640 10 monclient(hunting): get_auth_request con 0x7f61800a58b0 auth_method 0 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6177fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f6177fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient(hunting): handle_auth_done global_id 154112 payload 824 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61800a58b0 msgr2=0x7f6180133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61800a58b0 0x7f6180133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f61800980a0 msgr2=0x7f6180132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f61800980a0 0x7f6180132b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 monclient: found mon.noname-c 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.380+0000 7f617ed9d640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170009390 con 0x7f6180136dd0 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.382+0000 7f617ed9d640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 msgr2=0x7f61801335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.382+0000 7f617ed9d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 0x7f61801335a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6178005bc0 tx=0x7f61780052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.382+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178007510 con 0x7f6180136dd0 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: dump: 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.396 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6178007690 con 0x7f6180136dd0 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 5 monclient: authenticate success, global_id 154112 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 monclient: _renew_subs 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6180133f90 con 0x7f6180136dd0 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178007a50 con 0x7f6180136dd0 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 10 monclient: dump: 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 monclient: _renew_subs 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f617d59a640 4 set_mon_vals no callback set 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618009e4d0 con 0x7f6180136dd0 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 0x7f6180141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180133e00 con 0x7f6180136130 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.383+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6178012610 con 0x7f6180136dd0 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 0x7f6180141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient: get_auth_request con 0x7f6180136130 auth_method 0 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f61767fc640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6160038790 0x7f616003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f617804c620 con 0x7f6180136dd0 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f6177fff640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6160038790 0x7f616003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 10 monclient(hunting): handle_auth_done global_id 144181 payload 824 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f617e59c640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6180135040 con 0x7f6180136130 2024-09-15T02:02:44.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.384+0000 7f6177fff640 10 monclient: get_auth_request con 0x7f6160038790 auth_method 0 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f617e59c640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 msgr2=0x7f6180141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f617e59c640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 0x7f6180141ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6170009360 tx=0x7f6170032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617003da50 con 0x7f6180136130 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f61767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f61767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.386+0000 7f61767fc640 10 monclient: dump: 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.403 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f6177fff640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6160038790 0x7f616003ac40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f618009eac0 tx=0x7f6168008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6170031420 con 0x7f6180136130 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 msgr2=0x7f6180141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 0x7f6180141ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6170009360 tx=0x7f6170032000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 msgr2=0x7f6180141ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.387+0000 7f61767fc640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180136130 0x7f6180141ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f6184f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 0x7f6180142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f6184f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180063b20 con 0x7f6180142020 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 0x7f6180142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient: get_auth_request con 0x7f6180142020 auth_method 0 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 10 monclient(hunting): handle_auth_done global_id 144184 payload 824 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.393+0000 7f617e59c640 1 -- 172.21.15.17:0/3180225019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6180142960 con 0x7f6180142020 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f617e59c640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 msgr2=0x7f6180142420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f617e59c640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 0x7f6180142420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6170009130 tx=0x7f617003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617003cce0 con 0x7f6180142020 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 10 monclient: dump: 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.406 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f617003ce40 con 0x7f6180142020 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 msgr2=0x7f6180142420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 0x7f6180142420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6170009130 tx=0x7f617003d000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 msgr2=0x7f6180142420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.396+0000 7f61767fc640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180142020 0x7f6180142420 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6170009130 tx=0x7f617003d000 comp rx=0 tx=0).stop 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:44.408 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.409 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.410 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:44.411 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:44.412 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:44.413 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:44.417 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.419 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:44.420 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6160038790 msgr2=0x7f616003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6160038790 0x7f616003ac40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f618009eac0 tx=0x7f6168008040 comp rx=0 tx=0).stop 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 10 monclient: shutdown 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 msgr2=0x7f61801335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f617ed9d640 1 -- 172.21.15.17:0/3180225019 reap_dead start 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 0x7f61801335a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6178005bc0 tx=0x7f61780052b0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 shutdown_connections 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 --2- 172.21.15.17:0/3180225019 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f6180136dd0 0x7f61801335a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 >> 172.21.15.17:0/3180225019 conn(0x7f6180070ed0 msgr2=0x7f61801392b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 shutdown_connections 2024-09-15T02:02:44.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.397+0000 7f6184f5d640 1 -- 172.21.15.17:0/3180225019 wait complete. 2024-09-15T02:02:44.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:44.423 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:02:44.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:02:44.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.517+0000 7fabc0314640 10 monclient: get_monmap_and_config 2024-09-15T02:02:44.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.517+0000 7fabc0314640 10 monclient: build_initial_monmap 2024-09-15T02:02:44.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.517+0000 7fabc0314640 10 monclient: monmap: 2024-09-15T02:02:44.519 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:44.519 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:44.518351+0000 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:44.518351+0000 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:44.520 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 Processor -- start 2024-09-15T02:02:44.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 -- start start 2024-09-15T02:02:44.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient: init 2024-09-15T02:02:44.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:44.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabb80a5780 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabb809ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-b con 0x7fabb809bc90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80a5b50 con 0x7fabb809ccf0 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb809c5f0 con 0x7fabb80a5780 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb809c0f0 con 0x7fabb809bc90 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabc0314640 10 monclient(hunting): authenticate will time out at 1424.514795s 2024-09-15T02:02:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbe8b1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbd8af640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fabb809ccf0 0x7fabb8069c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51502/0 (socket says 172.21.15.17:51502) 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbd8af640 1 -- 172.21.15.17:0/3608067370 learned_addr learned my addr 172.21.15.17:0/3608067370 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbe8b1640 10 monclient(hunting): get_auth_request con 0x7fabb809bc90 auth_method 0 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbe8b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbe8b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.518+0000 7fabbe8b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_done global_id 154115 payload 274 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 1 -- 172.21.15.17:0/3608067370 >> v1:172.21.15.67:6789/0 conn(0x7fabb80a5780 legacy=0x7fabb8069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 1 -- 172.21.15.17:0/3608067370 >> v1:172.21.15.17:6789/0 conn(0x7fabb809ccf0 legacy=0x7fabb8069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient: _finish_hunting 0 2024-09-15T02:02:44.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 1 monclient: found mon.noname-b 2024-09-15T02:02:44.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:44.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.519+0000 7fabbe8b1640 1 -- 172.21.15.17:0/3608067370 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb806a3a0 con 0x7fabb809bc90 2024-09-15T02:02:44.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbe8b1640 1 -- 172.21.15.17:0/3608067370 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 msgr2=0x7fabb813a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbe8b1640 1 --2- 172.21.15.17:0/3608067370 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813a890 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009e90 tx=0x7fabb0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=164407bfbfa1d5fe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 1 -- 172.21.15.17:0/3608067370 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0007840 con 0x7fabb809bc90 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: dump: 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: _finish_auth 0 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 1 -- 172.21.15.17:0/3608067370 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb002d050 con 0x7fabb809bc90 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 5 monclient: authenticate success, global_id 154115 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 4 set_mon_vals no callback set 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 1 -- 172.21.15.17:0/3608067370 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0007bd0 con 0x7fabb809bc90 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabbd0ae640 10 monclient: dump: 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 10 monclient: shutdown 2024-09-15T02:02:44.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 1 -- 172.21.15.17:0/3608067370 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 msgr2=0x7fabb813a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.522+0000 7fabc0314640 1 --2- 172.21.15.17:0/3608067370 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813a890 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009e90 tx=0x7fabb0009780 comp rx=0 tx=0).stop 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 -- 172.21.15.17:0/3608067370 shutdown_connections 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 --2- 172.21.15.17:0/3608067370 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813a890 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 -- 172.21.15.17:0/3608067370 >> 172.21.15.17:0/3608067370 conn(0x7fabb8070e10 msgr2=0x7fabb80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 -- 172.21.15.17:0/3608067370 shutdown_connections 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 -- 172.21.15.17:0/3608067370 wait complete. 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 10 monclient: build_initial_monmap 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 10 monclient: monmap: 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:44.534 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 Processor -- start 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 1 -- start start 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.523+0000 7fabc0314640 10 monclient: init 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-c con 0x7fabb809bc90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-a con 0x7fabb809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb80a5780 0x7fabb8141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): picked mon.noname-b con 0x7fabb80a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8122de0 con 0x7fabb809ccf0 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8122820 con 0x7fabb80a5780 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb81202b0 con 0x7fabb809bc90 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabc0314640 10 monclient(hunting): authenticate will time out at 1424.520336s 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40742/0 (socket says 172.21.15.17:40742) 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54142/0 (socket says 172.21.15.17:54142) 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 -- 172.21.15.17:0/593041205 learned_addr learned my addr 172.21.15.17:0/593041205 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): get_auth_request con 0x7fabb809bc90 auth_method 0 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): get_auth_request con 0x7fabb809ccf0 auth_method 0 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb80a5780 0x7fabb8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): get_auth_request con 0x7fabb80a5780 auth_method 0 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe8b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient(hunting): handle_auth_done global_id 154118 payload 824 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 -- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb80a5780 msgr2=0x7fabb8141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb80a5780 0x7fabb8141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 -- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 msgr2=0x7fabb813d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient: _finish_hunting 0 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 monclient: found mon.noname-c 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbe0b0640 1 -- 172.21.15.17:0/593041205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009a20 con 0x7fabb809bc90 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.524+0000 7fabbd8af640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabbe0b0640 1 -- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 msgr2=0x7fabb813ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabbe0b0640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c610 tx=0x7fabb4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012840 con 0x7fabb809bc90 2024-09-15T02:02:44.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: dump: 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb40129e0 con 0x7fabb809bc90 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 5 monclient: authenticate success, global_id 154118 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 10 monclient: _renew_subs 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb8122540 con 0x7fabb809bc90 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012da0 con 0x7fabb809bc90 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabae7fc640 10 monclient: dump: 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:44.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 10 monclient: _renew_subs 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabbc8ad640 4 set_mon_vals no callback set 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.527+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb813db80 con 0x7fabb809bc90 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fabb4011590 con 0x7fabb809bc90 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabc0314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabc0314640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabc0314640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb80724a0 con 0x7fabb809bc90 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabae7fc640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 0x7fab9c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fabb404be90 con 0x7fabb809bc90 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabbd8af640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 0x7fab9c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.528+0000 7fabbd8af640 10 monclient: get_auth_request con 0x7fab9c038740 auth_method 0 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.531+0000 7fabbd8af640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 0x7fab9c03abf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fabb80a1ad0 tx=0x7faba8006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:44.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.532+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fabb400e890 con 0x7fabb809bc90 2024-09-15T02:02:44.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.532+0000 7fabae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:44.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.532+0000 7fabae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:44.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.758+0000 7fabc0314640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:02:44.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.758+0000 7fabc0314640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:02:44.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.758+0000 7fabc0314640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:44.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.758+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fabb806f510 con 0x7fabb809bc90 2024-09-15T02:02:44.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.760+0000 7fabae7fc640 1 -- 172.21.15.17:0/593041205 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fabb400e290 con 0x7fabb809bc90 2024-09-15T02:02:44.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.760+0000 7fabae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:02:44.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.760+0000 7fabae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 msgr2=0x7fab9c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 0x7fab9c03abf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fabb80a1ad0 tx=0x7faba8006f50 comp rx=0 tx=0).stop 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 10 monclient: shutdown 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 msgr2=0x7fabb813ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c610 tx=0x7fabb4007ed0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 shutdown_connections 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fab9c038740 0x7fab9c03abf0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabb809ccf0 0x7fabb813d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb80a5780 0x7fabb8141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 --2- 172.21.15.17:0/593041205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabb809bc90 0x7fabb813ddf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 >> 172.21.15.17:0/593041205 conn(0x7fabb8070e10 msgr2=0x7fabb809d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 shutdown_connections 2024-09-15T02:02:44.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:44.762+0000 7fabc0314640 1 -- 172.21.15.17:0/593041205 wait complete. 2024-09-15T02:02:44.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=2729602769,config=3140514009,health=1675380437,kv=654750165,logm=1562431315}) 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=8,logm=55} crc {auth=3916897286,config=3140514009,health=612861212,kv=1529054990,logm=849824264}) 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=2729602769,config=3140514009,health=1675380437,kv=654750165,logm=1562431315}) 2024-09-15T02:02:45.017 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.015+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=3373310560,config=3140514009,health=3972976114,kv=1529054990,logm=58792851}) 2024-09-15T02:02:45.018 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.018 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704102281}) 2024-09-15T02:02:45.019 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2420684489}) 2024-09-15T02:02:45.019 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.019 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704102281}) 2024-09-15T02:02:45.019 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.016+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734712792}) 2024-09-15T02:02:45.019 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.020 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321476777}) 2024-09-15T02:02:45.020 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704689793}) 2024-09-15T02:02:45.020 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.020 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321476777}) 2024-09-15T02:02:45.020 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.017+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60580805}) 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248123975}) 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=797051047}) 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248123975}) 2024-09-15T02:02:45.021 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231108274}) 2024-09-15T02:02:45.022 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030821104}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277944117}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.019+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030821104}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262971439}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208498793}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491810930}) 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.023 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208498793}) 2024-09-15T02:02:45.024 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.020+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513483773}) 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319114573}) 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778812984}) 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319114573}) 2024-09-15T02:02:45.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.021+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50723370}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542631160}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489273323}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542631160}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.022+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465604801}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788993538}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255531505}) 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.029 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788993538}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149694308}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081433783}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543904522}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081433783}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=979138727}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979159589}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262259842}) 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979159589}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3437446339}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714458476}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962480434}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714458476}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235627024}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021780903}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838014089}) 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.031 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021780903}) 2024-09-15T02:02:45.032 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573948201}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471757717}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575612377}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471757717}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176339058}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090346608}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170181511}) 2024-09-15T02:02:45.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090346608}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012898221}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349349917}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107049827}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349349917}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717967008}) 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519263278}) 2024-09-15T02:02:45.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=992862085}) 2024-09-15T02:02:45.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519263278}) 2024-09-15T02:02:45.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139120298}) 2024-09-15T02:02:45.040 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.040 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934810670}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827475720}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934810670}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=132347014}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653947149}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624728233}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653947149}) 2024-09-15T02:02:45.041 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32516039}) 2024-09-15T02:02:45.042 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.042 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184398573}) 2024-09-15T02:02:45.042 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478698273}) 2024-09-15T02:02:45.042 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.042 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184398573}) 2024-09-15T02:02:45.043 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900576107}) 2024-09-15T02:02:45.043 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.043 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788271206}) 2024-09-15T02:02:45.043 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668982937}) 2024-09-15T02:02:45.044 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.044 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788271206}) 2024-09-15T02:02:45.044 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2198699989}) 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720045917}) 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663114862}) 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720045917}) 2024-09-15T02:02:45.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517935823}) 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451239098}) 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695103960}) 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451239098}) 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185914932}) 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506928582}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226826696}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506928582}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201196099}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378552586}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930008595}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378552586}) 2024-09-15T02:02:45.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.050+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668183281}) 2024-09-15T02:02:45.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709865027}) 2024-09-15T02:02:45.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263880922}) 2024-09-15T02:02:45.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709865027}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.051+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780613482}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424046039}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539268622}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424046039}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639656348}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053264877}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266483691}) 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053264877}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.052+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285677730}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577149541}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762432465}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577149541}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.053+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372857135}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818068371}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589431499}) 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818068371}) 2024-09-15T02:02:45.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.054+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116018913}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801707108}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3455338716}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801707108}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.055+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930401677}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288646625}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225534648}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288646625}) 2024-09-15T02:02:45.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202350244}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329754624}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467255439}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329754624}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.056+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202148489}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639813033}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2283412647}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639813033}) 2024-09-15T02:02:45.062 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=530052509}) 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400080243}) 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2113935550}) 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400080243}) 2024-09-15T02:02:45.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4001563708}) 2024-09-15T02:02:45.120 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.120 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189807396}) 2024-09-15T02:02:45.120 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3260999689}) 2024-09-15T02:02:45.120 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189807396}) 2024-09-15T02:02:45.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.118+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710748303}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772268635}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033416878}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772268635}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715677369}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481596246}) 2024-09-15T02:02:45.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364553633}) 2024-09-15T02:02:45.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481596246}) 2024-09-15T02:02:45.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960318130}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229246444}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932254061}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229246444}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.120+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=14519975}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620503356}) 2024-09-15T02:02:45.126 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788530068}) 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620503356}) 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299286795}) 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302273464}) 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978075142}) 2024-09-15T02:02:45.127 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302273464}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.122+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287485001}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525145969}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3651142500}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525145969}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.123+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3652318447}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057189222}) 2024-09-15T02:02:45.128 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503865312}) 2024-09-15T02:02:45.129 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.129 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057189222}) 2024-09-15T02:02:45.129 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321608945}) 2024-09-15T02:02:45.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6549502}) 2024-09-15T02:02:45.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888827021}) 2024-09-15T02:02:45.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.124+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6549502}) 2024-09-15T02:02:45.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013523329}) 2024-09-15T02:02:45.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827408295}) 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193238899}) 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827408295}) 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.125+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169970094}) 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290420585}) 2024-09-15T02:02:45.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689278007}) 2024-09-15T02:02:45.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290420585}) 2024-09-15T02:02:45.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.126+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116231139}) 2024-09-15T02:02:45.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248457960}) 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=840239486}) 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248457960}) 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.127+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155071604}) 2024-09-15T02:02:45.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120778444}) 2024-09-15T02:02:45.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887296466}) 2024-09-15T02:02:45.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120778444}) 2024-09-15T02:02:45.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512013896}) 2024-09-15T02:02:45.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84264821}) 2024-09-15T02:02:45.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1731755490}) 2024-09-15T02:02:45.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84264821}) 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3927589027}) 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365341346}) 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3820717765}) 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.151 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365341346}) 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1203358476}) 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612413921}) 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246946034}) 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612413921}) 2024-09-15T02:02:45.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496882197}) 2024-09-15T02:02:45.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235105885}) 2024-09-15T02:02:45.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921945343}) 2024-09-15T02:02:45.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235105885}) 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999816733}) 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184249908}) 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974625694}) 2024-09-15T02:02:45.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184249908}) 2024-09-15T02:02:45.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3366992692}) 2024-09-15T02:02:45.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897246084}) 2024-09-15T02:02:45.155 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1447624899}) 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897246084}) 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016400306}) 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799114820}) 2024-09-15T02:02:45.156 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=638986774}) 2024-09-15T02:02:45.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799114820}) 2024-09-15T02:02:45.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3852501455}) 2024-09-15T02:02:45.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.157 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848705128}) 2024-09-15T02:02:45.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792102304}) 2024-09-15T02:02:45.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848705128}) 2024-09-15T02:02:45.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=435691619}) 2024-09-15T02:02:45.158 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209674629}) 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606994904}) 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209674629}) 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1299137701}) 2024-09-15T02:02:45.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646102932}) 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266002103}) 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646102932}) 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280686899}) 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959311518}) 2024-09-15T02:02:45.161 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394002437}) 2024-09-15T02:02:45.161 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.161 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959311518}) 2024-09-15T02:02:45.161 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866744469}) 2024-09-15T02:02:45.161 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576919974}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237763228}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576919974}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=262349118}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2851669142,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3102201217,mon_config_key=2687124721,monmap=2689118867,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3883770753}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=625753899,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=709213672,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3535454963}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2851669142,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3102201217,mon_config_key=2687124721,monmap=2689118867,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3883770753}) 2024-09-15T02:02:45.162 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=21} crc {logm=2730499182,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2260823826,mon_config_key=1583035879,monmap=3140514009,osd_metadata=1418025409,osd_pg_creating=1863806571,osd_snap=2673976383,osdmap=2859665185}) 2024-09-15T02:02:45.163 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.163 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3194671574}) 2024-09-15T02:02:45.163 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=1944667500}) 2024-09-15T02:02:45.163 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:02:45.163 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3194671574}) 2024-09-15T02:02:45.164 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:02:45.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=1205021129}) 2024-09-15T02:02:45.776 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:02:45.776 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.882+0000 7f4d42a80640 10 monclient: get_monmap_and_config 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.882+0000 7f4d42a80640 10 monclient: build_initial_monmap 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.882+0000 7f4d42a80640 10 monclient: monmap: 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:45.883729+0000 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:45.883729+0000 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:45.886 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:45.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.883+0000 7f4d42a80640 1 Processor -- start 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.883+0000 7f4d42a80640 1 -- start start 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.883+0000 7f4d42a80640 10 monclient: init 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-a con 0x7f4d3c0a5780 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3c09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d3c069ac0 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3c0a5b50 con 0x7f4d3c069ac0 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3c0a18b0 con 0x7f4d3c0a5780 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c09c5f0 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d42a80640 10 monclient(hunting): authenticate will time out at 1425.880270s 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54146/0 (socket says 172.21.15.17:54146) 2024-09-15T02:02:45.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40744/0 (socket says 172.21.15.17:40744) 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 1 -- 172.21.15.17:0/2788039802 learned_addr learned my addr 172.21.15.17:0/2788039802 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): get_auth_request con 0x7f4d3c0a5780 auth_method 0 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request con 0x7f4d3c09ccf0 auth_method 0 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:45.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_done global_id 154121 payload 274 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 -- 172.21.15.17:0/2788039802 >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 msgr2=0x7f4d3c09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/2788039802 >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 -- 172.21.15.17:0/2788039802 >> v1:172.21.15.67:6789/0 conn(0x7f4d3c069ac0 legacy=0x7f4d3c138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 monclient: found mon.noname-b 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d3bfff640 1 -- 172.21.15.17:0/2788039802 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3c09d210 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.884+0000 7f4d4081c640 1 --2- 172.21.15.17:0/2788039802 >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3bfff640 1 -- 172.21.15.17:0/2788039802 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 msgr2=0x7f4d3c069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/2788039802 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009850 tx=0x7f4d280090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b35692972416287 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 1 -- 172.21.15.17:0/2788039802 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2802adf0 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: dump: 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 1 -- 172.21.15.17:0/2788039802 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28032440 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d42a80640 5 monclient: authenticate success, global_id 154121 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d42a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d42a80640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 1 -- 172.21.15.17:0/2788039802 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28032820 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:45.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d42a80640 4 set_mon_vals no callback set 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.887+0000 7f4d3b7fe640 10 monclient: dump: 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 10 monclient: shutdown 2024-09-15T02:02:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 -- 172.21.15.17:0/2788039802 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 msgr2=0x7f4d3c069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 --2- 172.21.15.17:0/2788039802 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009850 tx=0x7f4d280090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 -- 172.21.15.17:0/2788039802 shutdown_connections 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 --2- 172.21.15.17:0/2788039802 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c09ccf0 0x7f4d3c069580 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 --2- 172.21.15.17:0/2788039802 >> v2:172.21.15.17:3300/0 conn(0x7f4d3c0a5780 0x7f4d3c09bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 -- 172.21.15.17:0/2788039802 >> 172.21.15.17:0/2788039802 conn(0x7f4d3c070e10 msgr2=0x7f4d3c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 -- 172.21.15.17:0/2788039802 shutdown_connections 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 1 -- 172.21.15.17:0/2788039802 wait complete. 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 10 monclient: build_initial_monmap 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.888+0000 7f4d42a80640 10 monclient: monmap: 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:45.900 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:45.901 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.889+0000 7f4d42a80640 1 Processor -- start 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.889+0000 7f4d42a80640 1 -- start start 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.889+0000 7f4d42a80640 10 monclient: init 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-a con 0x7f4d3c069ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 0x7f4d3c13d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3c09ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c0a5780 0x7f4d3c141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): picked mon.noname-c con 0x7f4d3c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c120bf0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c1228e0 con 0x7f4d3c09ccf0 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c13e9a0 con 0x7f4d3c0a5780 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d42a80640 10 monclient(hunting): authenticate will time out at 1425.886121s 2024-09-15T02:02:45.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54160/0 (socket says 172.21.15.17:54160) 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 -- 172.21.15.17:0/653505331 learned_addr learned my addr 172.21.15.17:0/653505331 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): get_auth_request con 0x7f4d3c069ac0 auth_method 0 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c0a5780 0x7f4d3c141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): get_auth_request con 0x7f4d3c0a5780 auth_method 0 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4101d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 0x7f4d3c13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient(hunting): handle_auth_done global_id 105087 payload 824 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 -- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c0a5780 msgr2=0x7f4d3c141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c0a5780 0x7f4d3c141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 -- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 msgr2=0x7f4d3c13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 0x7f4d3c13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient: _finish_hunting 0 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 monclient: found mon.noname-a 2024-09-15T02:02:45.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d4081c640 1 -- 172.21.15.17:0/653505331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30009450 con 0x7f4d3c069ac0 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d3bfff640 10 monclient: get_auth_request con 0x7f4d3c09ccf0 auth_method 0 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.890+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 0x7f4d3c13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d4081c640 1 -- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 msgr2=0x7f4d3c13deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d4081c640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000ec80 tx=0x7f4d300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30009730 con 0x7f4d3c069ac0 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 10 monclient: dump: 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:45.908 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 10 monclient: _finish_auth 0 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3000eec0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d42a80640 5 monclient: authenticate success, global_id 105087 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d42a80640 10 monclient: _renew_subs 2024-09-15T02:02:45.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d42a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.892+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d28009390 con 0x7f4d3c069ac0 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30013470 con 0x7f4d3c069ac0 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 10 monclient: dump: 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d3affd640 4 set_mon_vals no callback set 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d42a80640 10 monclient: _renew_subs 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d42a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3c13daa0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.893+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4d300137e0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d42a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d42a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d42a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:45.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3c0724a0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d38ff9640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 0x7f4d2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:45.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4d3004cb40 con 0x7f4d3c069ac0 2024-09-15T02:02:45.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 0x7f4d2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:45.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.894+0000 7f4d3bfff640 10 monclient: get_auth_request con 0x7f4d20038740 auth_method 0 2024-09-15T02:02:45.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.897+0000 7f4d3bfff640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 0x7f4d2003abf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4d28034000 tx=0x7f4d28032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:45.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.899+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4d30023ad0 con 0x7f4d3c069ac0 2024-09-15T02:02:45.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.899+0000 7f4d38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:45.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:45.899+0000 7f4d38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.129+0000 7f4d42a80640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.129+0000 7f4d42a80640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.129+0000 7f4d42a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.129+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4d3c06f510 con 0x7f4d3c069ac0 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.130+0000 7f4d38ff9640 1 -- 172.21.15.17:0/653505331 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f4d300232f0 con 0x7f4d3c069ac0 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.130+0000 7f4d38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.130+0000 7f4d38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:46.132 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 msgr2=0x7f4d2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 0x7f4d2003abf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4d28034000 tx=0x7f4d28032000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 10 monclient: shutdown 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 msgr2=0x7f4d3c13deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000ec80 tx=0x7f4d300091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 shutdown_connections 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4d20038740 0x7f4d2003abf0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4d3c0a5780 0x7f4d3c141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c09ccf0 0x7f4d3c13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 --2- 172.21.15.17:0/653505331 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4d3c069ac0 0x7f4d3c13deb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.131+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 >> 172.21.15.17:0/653505331 conn(0x7f4d3c070e10 msgr2=0x7f4d3c098d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.132+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 shutdown_connections 2024-09-15T02:02:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.132+0000 7f4d42a80640 1 -- 172.21.15.17:0/653505331 wait complete. 2024-09-15T02:02:46.145 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:02:46.145 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:02:46.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:02:46.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.240+0000 7f513e55d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.240+0000 7f513e55d640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.240+0000 7f513e55d640 10 monclient: monmap: 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:46.241310+0000 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:46.241310+0000 2024-09-15T02:02:46.242 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:46.243 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 Processor -- start 2024-09-15T02:02:46.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 -- start start 2024-09-15T02:02:46.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient: init 2024-09-15T02:02:46.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 0x7f5138131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-a con 0x7f513809ec30 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f513809f000 addr v1:172.21.15.17:6789/0 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-b con 0x7f5138069590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513809c6f0 con 0x7f513809f000 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513806a650 con 0x7f513809ec30 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51380a13a0 con 0x7f5138069590 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513e55d640 10 monclient(hunting): authenticate will time out at 1426.237765s 2024-09-15T02:02:46.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40772/0 (socket says 172.21.15.17:40772) 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f51377fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f513809f000 0x7f5138068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51512/0 (socket says 172.21.15.17:51512) 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 1 -- 172.21.15.17:0/2660358470 learned_addr learned my addr 172.21.15.17:0/2660358470 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 10 monclient(hunting): get_auth_request con 0x7f5138069590 auth_method 0 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f5137fff640 1 --2- 172.21.15.17:0/2660358470 >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 0x7f5138131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f513cafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f5137fff640 10 monclient(hunting): get_auth_request con 0x7f513809ec30 auth_method 0 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f5137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f5137fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.241+0000 7f5137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient(hunting): handle_auth_done global_id 154127 payload 274 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 1 -- 172.21.15.17:0/2660358470 >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 msgr2=0x7f5138131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 1 --2- 172.21.15.17:0/2660358470 >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 0x7f5138131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 1 -- 172.21.15.17:0/2660358470 >> v1:172.21.15.17:6789/0 conn(0x7f513809f000 legacy=0x7f5138068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 1 monclient: found mon.noname-b 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f513cafa640 1 -- 172.21.15.17:0/2660358470 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51380a1680 con 0x7f5138069590 2024-09-15T02:02:46.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.242+0000 7f5137fff640 1 --2- 172.21.15.17:0/2660358470 >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 0x7f5138131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513cafa640 1 -- 172.21.15.17:0/2660358470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 msgr2=0x7f51380699e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513cafa640 1 --2- 172.21.15.17:0/2660358470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5130009f20 tx=0x7f5130009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=517a6d1c333a3844 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 1 -- 172.21.15.17:0/2660358470 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5130007610 con 0x7f5138069590 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: dump: 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 1 -- 172.21.15.17:0/2660358470 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513002d050 con 0x7f5138069590 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 5 monclient: authenticate success, global_id 154127 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 1 -- 172.21.15.17:0/2660358470 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51300079a0 con 0x7f5138069590 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 4 set_mon_vals no callback set 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f5136ffd640 10 monclient: dump: 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.256 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 10 monclient: shutdown 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 -- 172.21.15.17:0/2660358470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 msgr2=0x7f51380699e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 --2- 172.21.15.17:0/2660358470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5130009f20 tx=0x7f5130009840 comp rx=0 tx=0).stop 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 -- 172.21.15.17:0/2660358470 shutdown_connections 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 --2- 172.21.15.17:0/2660358470 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f51380699e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 --2- 172.21.15.17:0/2660358470 >> v2:172.21.15.17:3300/0 conn(0x7f513809ec30 0x7f5138131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 -- 172.21.15.17:0/2660358470 >> 172.21.15.17:0/2660358470 conn(0x7f5138071320 msgr2=0x7f51380a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 -- 172.21.15.17:0/2660358470 shutdown_connections 2024-09-15T02:02:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.245+0000 7f513e55d640 1 -- 172.21.15.17:0/2660358470 wait complete. 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.246+0000 7f513e55d640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.246+0000 7f513e55d640 10 monclient: monmap: 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.246+0000 7f513e55d640 1 Processor -- start 2024-09-15T02:02:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.246+0000 7f513e55d640 1 -- start start 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.246+0000 7f513e55d640 10 monclient: init 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f5138140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-c con 0x7f5138069590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513809ec30 0x7f513812ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-b con 0x7f513809ec30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 0x7f5138098cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): picked mon.noname-a con 0x7f513809f000 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513811bf50 con 0x7f513809f000 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5138130c00 con 0x7f513809ec30 2024-09-15T02:02:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513809ad20 con 0x7f5138069590 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513e55d640 10 monclient(hunting): authenticate will time out at 1426.243248s 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f5138140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 0x7f5138098cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f5138140e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40786/0 (socket says 172.21.15.17:40786) 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 0x7f5138098cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54180/0 (socket says 172.21.15.17:54180) 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 -- 172.21.15.17:0/740889820 learned_addr learned my addr 172.21.15.17:0/740889820 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): get_auth_request con 0x7f5138069590 auth_method 0 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): get_auth_request con 0x7f513809f000 auth_method 0 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513809ec30 0x7f513812ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 10 monclient(hunting): get_auth_request con 0x7f513809ec30 auth_method 0 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient(hunting): handle_auth_done global_id 154130 payload 824 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513809ec30 msgr2=0x7f513812ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513809ec30 0x7f513812ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 msgr2=0x7f5138098cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 0x7f5138098cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 monclient: found mon.noname-c 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f5137fff640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128009390 con 0x7f5138069590 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513809ec30 0x7f513812ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.247+0000 7f513cafa640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513809f000 0x7f5138098cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:46.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.249+0000 7f5137fff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 msgr2=0x7f5138140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.249+0000 7f5137fff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f5138140e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f512800ecb0 tx=0x7f51280090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512800cdd0 con 0x7f5138069590 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: dump: 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128013470 con 0x7f5138069590 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 5 monclient: authenticate success, global_id 154130 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 monclient: _renew_subs 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130009ae0 con 0x7f5138069590 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51280137f0 con 0x7f5138069590 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 10 monclient: dump: 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 monclient: _renew_subs 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f51367fc640 4 set_mon_vals no callback set 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513809b3b0 con 0x7f5138069590 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 0x7f51380298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5138063b20 con 0x7f513812f370 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.250+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5128013b60 con 0x7f5138069590 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 0x7f51380298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient: get_auth_request con 0x7f513812f370 auth_method 0 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f511ffff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5124038740 0x7f512403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 10 monclient(hunting): handle_auth_done global_id 105096 payload 824 2024-09-15T02:02:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f512804c960 con 0x7f5138069590 2024-09-15T02:02:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f51377fe640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f513811c3c0 con 0x7f513812f370 2024-09-15T02:02:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f513cafa640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5124038740 0x7f512403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.251+0000 7f513cafa640 10 monclient: get_auth_request con 0x7f5124038740 auth_method 0 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f51377fe640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 msgr2=0x7f51380298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 0x7f51380298c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f513809a2f0 tx=0x7f512c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0074f0 con 0x7f513812f370 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 10 monclient: dump: 2024-09-15T02:02:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f512c007650 con 0x7f513812f370 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 msgr2=0x7f51380298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 0x7f51380298c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f513809a2f0 tx=0x7f512c009040 comp rx=0 tx=0).stop 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 msgr2=0x7f51380298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.253+0000 7f511ffff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f513812f370 0x7f51380298c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.254+0000 7f513cafa640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5124038740 0x7f512403abf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5130030000 tx=0x7f5130036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f513e55d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f513e55d640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 0x7f513802a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f513e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513812f740 con 0x7f5138029e00 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 0x7f513802a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient: get_auth_request con 0x7f5138029e00 auth_method 0 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.259+0000 7f51377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.260+0000 7f51377fe640 10 monclient(hunting): handle_auth_done global_id 105099 payload 824 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.260+0000 7f51377fe640 1 -- 172.21.15.17:0/740889820 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f51380724a0 con 0x7f5138029e00 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f51377fe640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 msgr2=0x7f513802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f51377fe640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 0x7f513802a200 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f513809a2f0 tx=0x7f512c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c00c820 con 0x7f5138029e00 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 10 monclient: dump: 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f512c0072e0 con 0x7f5138029e00 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 msgr2=0x7f513802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 0x7f513802a200 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f513809a2f0 tx=0x7f512c018000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 msgr2=0x7f513802a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f511ffff640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5138029e00 0x7f513802a200 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f513809a2f0 tx=0x7f512c018000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:46.274 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.275 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:46.277 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.278 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:46.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:46.282 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.284 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.285 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:46.286 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5124038740 msgr2=0x7f512403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5124038740 0x7f512403abf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5130030000 tx=0x7f5130036000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 10 monclient: shutdown 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 msgr2=0x7f5138140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513e55d640 1 --2- 172.21.15.17:0/740889820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5138069590 0x7f5138140e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f512800ecb0 tx=0x7f51280090f0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.262+0000 7f513cafa640 1 -- 172.21.15.17:0/740889820 reap_dead start 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.263+0000 7f513cafa640 1 -- 172.21.15.17:0/740889820 reap_dead start 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.263+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 shutdown_connections 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.263+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 >> 172.21.15.17:0/740889820 conn(0x7f5138071320 msgr2=0x7f5138131150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.263+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 shutdown_connections 2024-09-15T02:02:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.263+0000 7f513e55d640 1 -- 172.21.15.17:0/740889820 wait complete. 2024-09-15T02:02:46.289 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:46.289 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:02:46.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:02:46.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.384+0000 7f951611d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:46.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.384+0000 7f951611d640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.384+0000 7f951611d640 10 monclient: monmap: 2024-09-15T02:02:46.386 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:46.385567+0000 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:46.385567+0000 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:46.387 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 1 Processor -- start 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 1 -- start start 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 10 monclient: init 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95101319a0 0x7f9510131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 10 monclient(hunting): picked mon.noname-c con 0x7f95101319a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.385+0000 7f951611d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): picked mon.noname-b con 0x7f95101322b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f951009f000 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95100a1530 con 0x7f951009f000 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9510132680 con 0x7f95101319a0 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951009fca0 con 0x7f95101322b0 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f951611d640 10 monclient(hunting): authenticate will time out at 1426.382070s 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40788/0 (socket says 172.21.15.17:40788) 2024-09-15T02:02:46.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 -- 172.21.15.17:0/3143889768 learned_addr learned my addr 172.21.15.17:0/3143889768 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): get_auth_request con 0x7f95101322b0 auth_method 0 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient(hunting): handle_auth_done global_id 154133 payload 274 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 -- 172.21.15.17:0/3143889768 >> v2:172.21.15.67:3300/0 conn(0x7f95101319a0 msgr2=0x7f9510131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 --2- 172.21.15.17:0/3143889768 >> v2:172.21.15.67:3300/0 conn(0x7f95101319a0 0x7f9510131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 -- 172.21.15.17:0/3143889768 >> v1:172.21.15.67:6789/0 conn(0x7f951009f000 legacy=0x7f9510068e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 monclient: found mon.noname-b 2024-09-15T02:02:46.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.386+0000 7f950effd640 1 -- 172.21.15.17:0/3143889768 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951009f460 con 0x7f95101322b0 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950effd640 1 -- 172.21.15.17:0/3143889768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 msgr2=0x7f951009eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950effd640 1 --2- 172.21.15.17:0/3143889768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009910 tx=0x7f94f80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d000c904872a734 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 1 -- 172.21.15.17:0/3143889768 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8032440 con 0x7f95101322b0 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: dump: 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.396 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 1 -- 172.21.15.17:0/3143889768 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f80325a0 con 0x7f95101322b0 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 5 monclient: authenticate success, global_id 154133 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:46.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 1 -- 172.21.15.17:0/3143889768 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8032980 con 0x7f95101322b0 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 4 set_mon_vals no callback set 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f950e7fc640 10 monclient: dump: 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.398 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 10 monclient: shutdown 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 -- 172.21.15.17:0/3143889768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 msgr2=0x7f951009eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 --2- 172.21.15.17:0/3143889768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009910 tx=0x7f94f80091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 -- 172.21.15.17:0/3143889768 shutdown_connections 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 --2- 172.21.15.17:0/3143889768 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f951009eac0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 --2- 172.21.15.17:0/3143889768 >> v2:172.21.15.67:3300/0 conn(0x7f95101319a0 0x7f9510131d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 -- 172.21.15.17:0/3143889768 >> 172.21.15.17:0/3143889768 conn(0x7f95100a5000 msgr2=0x7f95100a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 -- 172.21.15.17:0/3143889768 shutdown_connections 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.389+0000 7f951611d640 1 -- 172.21.15.17:0/3143889768 wait complete. 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.390+0000 7f951611d640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.390+0000 7f951611d640 10 monclient: monmap: 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.390+0000 7f951611d640 1 Processor -- start 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.390+0000 7f951611d640 1 -- start start 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.390+0000 7f951611d640 10 monclient: init 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:02:46.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f951009f000 0x7f951009ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): picked mon.noname-a con 0x7f951009f000 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95101319a0 0x7f9510130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): picked mon.noname-b con 0x7f95101319a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f9510140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): picked mon.noname-c con 0x7f95101322b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951011c6c0 con 0x7f951009f000 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100691e0 con 0x7f95101319a0 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951011cd30 con 0x7f95101322b0 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f951611d640 10 monclient(hunting): authenticate will time out at 1426.387370s 2024-09-15T02:02:46.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f9510140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f951009f000 0x7f951009ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f9510140fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40796/0 (socket says 172.21.15.17:40796) 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 learned_addr learned my addr 172.21.15.17:0/932516973 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): get_auth_request con 0x7f951009f000 auth_method 0 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): get_auth_request con 0x7f95101322b0 auth_method 0 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95101319a0 0x7f9510130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): get_auth_request con 0x7f95101319a0 auth_method 0 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient(hunting): handle_auth_done global_id 154136 payload 824 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95101319a0 msgr2=0x7f9510130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95101319a0 0x7f9510130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f951009f000 msgr2=0x7f951009ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f951009f000 0x7f951009ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 monclient: found mon.noname-c 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8009450 con 0x7f95101322b0 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.391+0000 7f950f7fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f951009f000 0x7f951009ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 msgr2=0x7f9510140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f950ffff640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f9510140fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f950800b7f0 tx=0x7f9508009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800c6a0 con 0x7f95101322b0 2024-09-15T02:02:46.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: dump: 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 5 monclient: authenticate success, global_id 154136 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 monclient: _renew_subs 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951011dc20 con 0x7f95101322b0 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950800c820 con 0x7f95101322b0 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800cbe0 con 0x7f95101322b0 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f95077fe640 10 monclient: dump: 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 monclient: _renew_subs 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f950dffb640 4 set_mon_vals no callback set 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951011bb30 con 0x7f95101322b0 2024-09-15T02:02:46.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 0x7f9510141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.394+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951011e0a0 con 0x7f9510131040 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 0x7f9510141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient: get_auth_request con 0x7f9510131040 auth_method 0 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9508012070 con 0x7f95101322b0 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 10 monclient(hunting): handle_auth_done global_id 154139 payload 824 2024-09-15T02:02:46.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f95077fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f94f0038740 0x7f94f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950f7fe640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95100724a0 con 0x7f9510131040 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f950804beb0 con 0x7f95101322b0 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950effd640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f94f0038740 0x7f94f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.395+0000 7f950effd640 10 monclient: get_auth_request con 0x7f94f0038740 auth_method 0 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f950f7fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 msgr2=0x7f9510141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f950f7fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 0x7f9510141510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000123f0 con 0x7f9510131040 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 10 monclient: dump: 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.412 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9500012550 con 0x7f9510131040 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.397+0000 7f95077fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.398+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 msgr2=0x7f9510141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.398+0000 7f95077fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 0x7f9510141510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500009120 comp rx=0 tx=0).stop 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.398+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 msgr2=0x7f9510141510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.398+0000 7f95077fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510131040 0x7f9510141510 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500009120 comp rx=0 tx=0).stop 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.398+0000 7f950effd640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f94f0038740 0x7f94f003abf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f95100a0060 tx=0x7f94f8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f951611d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f951611d640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 0x7f9510096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f951611d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9510063b00 con 0x7f9510141a50 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f950f7fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 0x7f9510096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f950f7fe640 10 monclient: get_auth_request con 0x7f9510141a50 auth_method 0 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f950f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f950f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.403+0000 7f950f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.404+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.404+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.404+0000 7f950f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.404+0000 7f950f7fe640 10 monclient(hunting): handle_auth_done global_id 154142 payload 824 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.404+0000 7f950f7fe640 1 -- 172.21.15.17:0/932516973 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9510141e50 con 0x7f9510141a50 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f950f7fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 msgr2=0x7f9510096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f950f7fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 0x7f9510096a10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000059a0 con 0x7f9510141a50 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 10 monclient: dump: 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f9500005b00 con 0x7f9510141a50 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 msgr2=0x7f9510096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.418 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:46.419 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:46.420 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.421 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:46.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:46.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.425 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.428 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:46.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 0x7f9510096a10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500019000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 msgr2=0x7f9510096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.406+0000 7f95077fe640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9510141a50 0x7f9510096a10 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f950000eb30 tx=0x7f9500019000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f94f0038740 msgr2=0x7f94f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f94f0038740 0x7f94f003abf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f95100a0060 tx=0x7f94f8032000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 10 monclient: shutdown 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 msgr2=0x7f9510140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 --2- 172.21.15.17:0/932516973 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f95101322b0 0x7f9510140fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f950800b7f0 tx=0x7f9508009840 comp rx=0 tx=0).stop 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 reap_dead start 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f950ffff640 1 -- 172.21.15.17:0/932516973 reap_dead start 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 shutdown_connections 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 >> 172.21.15.17:0/932516973 conn(0x7f95100a5000 msgr2=0x7f95100a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 shutdown_connections 2024-09-15T02:02:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.407+0000 7f951611d640 1 -- 172.21.15.17:0/932516973 wait complete. 2024-09-15T02:02:46.431 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:46.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:02:46.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.527+0000 7f23ebe9f640 10 monclient: get_monmap_and_config 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.527+0000 7f23ebe9f640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.527+0000 7f23ebe9f640 10 monclient: monmap: 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:46.528213+0000 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:46.528213+0000 2024-09-15T02:02:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:46.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.527+0000 7f23ebe9f640 1 Processor -- start 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 -- start start 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient: init 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23e4068f20 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e40692f0 0x7f23e40696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-c con 0x7f23e40692f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-b con 0x7f23e4136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e40a1950 con 0x7f23e4068f20 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e40a1670 con 0x7f23e40692f0 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e41328d0 con 0x7f23e4136fe0 2024-09-15T02:02:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ebe9f640 10 monclient(hunting): authenticate will time out at 1426.524671s 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40832/0 (socket says 172.21.15.17:40832) 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 1 -- 172.21.15.17:0/879400709 learned_addr learned my addr 172.21.15.17:0/879400709 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): get_auth_request con 0x7f23e4136fe0 auth_method 0 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.528+0000 7f23e943a640 1 --2- 172.21.15.17:0/879400709 >> v2:172.21.15.67:3300/0 conn(0x7f23e40692f0 0x7f23e40696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23e943a640 10 monclient(hunting): get_auth_request con 0x7f23e40692f0 auth_method 0 2024-09-15T02:02:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23e943a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23e943a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23e943a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 10 monclient(hunting): handle_auth_done global_id 154145 payload 274 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 1 -- 172.21.15.17:0/879400709 >> v2:172.21.15.67:3300/0 conn(0x7f23e40692f0 msgr2=0x7f23e40696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 1 --2- 172.21.15.17:0/879400709 >> v2:172.21.15.67:3300/0 conn(0x7f23e40692f0 0x7f23e40696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 1 -- 172.21.15.17:0/879400709 >> v1:172.21.15.67:6789/0 conn(0x7f23e4068f20 legacy=0x7f23e41321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 1 monclient: found mon.noname-b 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.529+0000 7f23ea43c640 1 -- 172.21.15.17:0/879400709 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e40a0020 con 0x7f23e4136fe0 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.531+0000 7f23ea43c640 1 -- 172.21.15.17:0/879400709 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 msgr2=0x7f23e41393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.531+0000 7f23ea43c640 1 --2- 172.21.15.17:0/879400709 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009f50 tx=0x7f23dc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9531cb459b6bb72b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 1 -- 172.21.15.17:0/879400709 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc007610 con 0x7f23e4136fe0 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: dump: 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.541 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 1 -- 172.21.15.17:0/879400709 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc02d050 con 0x7f23e4136fe0 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 5 monclient: authenticate success, global_id 154145 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 4 set_mon_vals no callback set 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 1 -- 172.21.15.17:0/879400709 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0079a0 con 0x7f23e4136fe0 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23e8c39640 10 monclient: dump: 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.543 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 10 monclient: shutdown 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 -- 172.21.15.17:0/879400709 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 msgr2=0x7f23e41393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/879400709 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009f50 tx=0x7f23dc009840 comp rx=0 tx=0).stop 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 -- 172.21.15.17:0/879400709 shutdown_connections 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/879400709 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4136fe0 0x7f23e41393c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/879400709 >> v2:172.21.15.67:3300/0 conn(0x7f23e40692f0 0x7f23e40696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 -- 172.21.15.17:0/879400709 >> 172.21.15.17:0/879400709 conn(0x7f23e4070fb0 msgr2=0x7f23e40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 -- 172.21.15.17:0/879400709 shutdown_connections 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.532+0000 7f23ebe9f640 1 -- 172.21.15.17:0/879400709 wait complete. 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.533+0000 7f23ebe9f640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.533+0000 7f23ebe9f640 10 monclient: monmap: 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.533+0000 7f23ebe9f640 1 Processor -- start 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.533+0000 7f23ebe9f640 1 -- start start 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.533+0000 7f23ebe9f640 10 monclient: init 2024-09-15T02:02:46.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 0x7f23e4099ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-c con 0x7f23e4068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e40692f0 0x7f23e409a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-b con 0x7f23e40692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23e4136fe0 0x7f23e409a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): picked mon.noname-a con 0x7f23e4136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e411c6c0 con 0x7f23e4136fe0 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e409c550 con 0x7f23e40692f0 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e411b870 con 0x7f23e4068f20 2024-09-15T02:02:46.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ebe9f640 10 monclient(hunting): authenticate will time out at 1426.530157s 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 0x7f23e4099ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23e4136fe0 0x7f23e409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 0x7f23e4099ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40844/0 (socket says 172.21.15.17:40844) 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 -- 172.21.15.17:0/1564239609 learned_addr learned my addr 172.21.15.17:0/1564239609 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): get_auth_request con 0x7f23e4068f20 auth_method 0 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): get_auth_request con 0x7f23e4136fe0 auth_method 0 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e40692f0 0x7f23e409a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 10 monclient(hunting): get_auth_request con 0x7f23e40692f0 auth_method 0 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient(hunting): handle_auth_done global_id 154148 payload 824 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e40692f0 msgr2=0x7f23e409a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e40692f0 0x7f23e409a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23e4136fe0 msgr2=0x7f23e409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23e4136fe0 0x7f23e409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 monclient: found mon.noname-c 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e9c3b640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc009ae0 con 0x7f23e4068f20 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e40692f0 0x7f23e409a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.534+0000 7f23ea43c640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23e4136fe0 0x7f23e409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.536+0000 7f23e9c3b640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 msgr2=0x7f23e4099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.536+0000 7f23e9c3b640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 0x7f23e4099ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23e000c610 tx=0x7f23e0007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.536+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0012840 con 0x7f23e4068f20 2024-09-15T02:02:46.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.536+0000 7f23d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: dump: 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 5 monclient: authenticate success, global_id 154148 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e00129e0 con 0x7f23e4068f20 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 10 monclient: _renew_subs 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e409bb40 con 0x7f23e4068f20 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0012da0 con 0x7f23e4068f20 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23d9ffb640 10 monclient: dump: 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.551 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23dbfff640 4 set_mon_vals no callback set 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 10 monclient: _renew_subs 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.537+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e41317c0 con 0x7f23e4068f20 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ebe9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 0x7f23e409aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4063b90 con 0x7f23e402aa40 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f23e0011590 con 0x7f23e4068f20 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23d9ffb640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23c8038790 0x7f23c803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 0x7f23e409aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient: get_auth_request con 0x7f23e402aa40 auth_method 0 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f23e004c0b0 con 0x7f23e4068f20 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ea43c640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23c8038790 0x7f23c803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23ea43c640 10 monclient: get_auth_request con 0x7f23c8038790 auth_method 0 2024-09-15T02:02:46.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 10 monclient(hunting): handle_auth_done global_id 144202 payload 824 2024-09-15T02:02:46.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.538+0000 7f23e943a640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23e402ae10 con 0x7f23e402aa40 2024-09-15T02:02:46.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23e943a640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 msgr2=0x7f23e409aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 0x7f23e409aaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23e409b1a0 tx=0x7f23d4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4014880 con 0x7f23e402aa40 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 10 monclient: dump: 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.556 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23ea43c640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23c8038790 0x7f23c803ac40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f23e40687a0 tx=0x7f23dc036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f23d40149e0 con 0x7f23e402aa40 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 msgr2=0x7f23e409aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 0x7f23e409aaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23e409b1a0 tx=0x7f23d4008040 comp rx=0 tx=0).stop 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 msgr2=0x7f23e409aaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.541+0000 7f23d9ffb640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e402aa40 0x7f23e409aaa0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23e409b1a0 tx=0x7f23d4008040 comp rx=0 tx=0).stop 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23ebe9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 0x7f23e405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23ebe9f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e40a0800 con 0x7f23e405e310 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 0x7f23e405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient: get_auth_request con 0x7f23e405e310 auth_method 0 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.547+0000 7f23e943a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.548+0000 7f23e943a640 10 monclient(hunting): handle_auth_done global_id 144205 payload 824 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.548+0000 7f23e943a640 1 -- 172.21.15.17:0/1564239609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f23e40724a0 con 0x7f23e405e310 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23e943a640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 msgr2=0x7f23e405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23e943a640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 0x7f23e405e710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23d4019000 tx=0x7f23d401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d401ed50 con 0x7f23e405e310 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 10 monclient: dump: 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.559 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f23d4022440 con 0x7f23e405e310 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 msgr2=0x7f23e405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 0x7f23e405e710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23d4019000 tx=0x7f23d401c000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.550+0000 7f23d9ffb640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 msgr2=0x7f23e405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23d9ffb640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e405e310 0x7f23e405e710 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23d4019000 tx=0x7f23d401c000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:02:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 122, 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:02:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:02:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:02:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:02:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:02:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:02:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:02:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:02:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:02:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:02:46.572 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23c8038790 msgr2=0x7f23c803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23c8038790 0x7f23c803ac40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f23e40687a0 tx=0x7f23dc036000 comp rx=0 tx=0).stop 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 10 monclient: shutdown 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 msgr2=0x7f23e4099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 --2- 172.21.15.17:0/1564239609 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23e4068f20 0x7f23e4099ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23e000c610 tx=0x7f23e0007ed0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ea43c640 1 -- 172.21.15.17:0/1564239609 reap_dead start 2024-09-15T02:02:46.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ea43c640 1 -- 172.21.15.17:0/1564239609 reap_dead start 2024-09-15T02:02:46.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 shutdown_connections 2024-09-15T02:02:46.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 >> 172.21.15.17:0/1564239609 conn(0x7f23e4070fb0 msgr2=0x7f23e409dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 shutdown_connections 2024-09-15T02:02:46.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.551+0000 7f23ebe9f640 1 -- 172.21.15.17:0/1564239609 wait complete. 2024-09-15T02:02:46.574 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:02:46.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T02:02:46.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:02:46.574 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T02:02:46.575 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:02:46.575 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:02:46.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:02:46.584 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:02:46.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:02:46.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:46.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.718+0000 7fc1cfe8b640 10 monclient: get_monmap_and_config 2024-09-15T02:02:46.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.718+0000 7fc1cfe8b640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.718+0000 7fc1cfe8b640 10 monclient: monmap: 2024-09-15T02:02:46.720 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:46.719500+0000 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:46.719500+0000 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:46.721 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 Processor -- start 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 -- start start 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient: init 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c809a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c809c2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1c809c690 0x7fc1c806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c809c690 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1c806f070 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c806a5c0 con 0x7fc1c806f070 2024-09-15T02:02:46.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c8099f50 con 0x7fc1c809c690 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c806a300 con 0x7fc1c809c2c0 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cfe8b640 10 monclient(hunting): authenticate will time out at 1426.715966s 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.719+0000 7fc1cdc27640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 msgr2=0x7fc1c809a920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cdc27640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c809a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1c809c690 0x7fc1c806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc1c809c690 0x7fc1c806eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54240/0 (socket says 172.21.15.17:54240) 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 -- 172.21.15.17:0/730941070 learned_addr learned my addr 172.21.15.17:0/730941070 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): get_auth_request con 0x7fc1c809c690 auth_method 0 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient(hunting): handle_auth_done global_id 105108 payload 274 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 -- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 msgr2=0x7fc1c809a920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 --2- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c809a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 -- 172.21.15.17:0/730941070 >> v1:172.21.15.67:6789/0 conn(0x7fc1c806f070 legacy=0x7fc1c809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 monclient: found mon.noname-a 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.720+0000 7fc1cd426640 1 -- 172.21.15.17:0/730941070 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8068ac0 con 0x7fc1c809c690 2024-09-15T02:02:46.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cd426640 1 -- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c809c690 msgr2=0x7fc1c806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cd426640 1 --2- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c809c690 0x7fc1c806eb30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009910 tx=0x7fc1b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79baa7163376951d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 1 -- 172.21.15.17:0/730941070 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8032440 con 0x7fc1c809c690 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: dump: 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.734 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 1 -- 172.21.15.17:0/730941070 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b80325a0 con 0x7fc1c809c690 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 5 monclient: authenticate success, global_id 105108 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 1 -- 172.21.15.17:0/730941070 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8032940 con 0x7fc1c809c690 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 4 set_mon_vals no callback set 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1ccc25640 10 monclient: dump: 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.736 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 10 monclient: shutdown 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c809c690 msgr2=0x7fc1c806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c809c690 0x7fc1c806eb30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009910 tx=0x7fc1b80091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/730941070 shutdown_connections 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c809c690 0x7fc1c806eb30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/730941070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c809a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/730941070 >> 172.21.15.17:0/730941070 conn(0x7fc1c80a2f70 msgr2=0x7fc1c80a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/730941070 shutdown_connections 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.723+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/730941070 wait complete. 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.724+0000 7fc1cfe8b640 10 monclient: build_initial_monmap 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.724+0000 7fc1cfe8b640 10 monclient: monmap: 2024-09-15T02:02:46.737 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.724+0000 7fc1cfe8b640 1 Processor -- start 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.724+0000 7fc1cfe8b640 1 -- start start 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.724+0000 7fc1cfe8b640 10 monclient: init 2024-09-15T02:02:46.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c806f070 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c81411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c809c2c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 0x7fc1c811da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c809c690 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8142160 con 0x7fc1c806f070 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8121370 con 0x7fc1c809c690 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c809cdf0 con 0x7fc1c809c2c0 2024-09-15T02:02:46.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cfe8b640 10 monclient(hunting): authenticate will time out at 1426.721103s 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cd426640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 msgr2=0x7fc1c81411f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cd426640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c81411f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54256/0 (socket says 172.21.15.17:54256) 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 -- 172.21.15.17:0/3241541806 learned_addr learned my addr 172.21.15.17:0/3241541806 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): get_auth_request con 0x7fc1c806f070 auth_method 0 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:46.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient(hunting): handle_auth_done global_id 105111 payload 824 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 -- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 msgr2=0x7fc1c81411f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c81411f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1ce428640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 0x7fc1c811da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 -- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 msgr2=0x7fc1c811da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 0x7fc1c811da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient: _finish_hunting 0 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 monclient: found mon.noname-a 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1cdc27640 1 -- 172.21.15.17:0/3241541806 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009450 con 0x7fc1c806f070 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1ce428640 10 monclient: get_auth_request con 0x7fc1c809c690 auth_method 0 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.725+0000 7fc1ce428640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 0x7fc1c811da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:02:46.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cdc27640 1 -- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 msgr2=0x7fc1c8133f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cdc27640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400b630 tx=0x7fc1c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4003d90 con 0x7fc1c806f070 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cdc27640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400b630 tx=0x7fc1c40091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fc1c4006150 config(0 keys) v1 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 10 monclient: dump: 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.742 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cfe8b640 5 monclient: authenticate success, global_id 105111 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4006150 con 0x7fc1c806f070 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cfe8b640 10 monclient: _renew_subs 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cfe8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c8121930 con 0x7fc1c806f070 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.727+0000 7fc1bffff640 4 set_mon_vals no callback set 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.728+0000 7fc1cfe8b640 10 monclient: _renew_subs 2024-09-15T02:02:46.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.728+0000 7fc1cfe8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.728+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c8121140 con 0x7fc1c806f070 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4006510 con 0x7fc1c806f070 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 10 monclient: dump: 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc1c40103e0 con 0x7fc1c806f070 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cfe8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cfe8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cfe8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c809f050 con 0x7fc1c806f070 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1bdffb640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 0x7fc1ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cd426640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 0x7fc1ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.729+0000 7fc1cd426640 10 monclient: get_auth_request con 0x7fc1ac0386f0 auth_method 0 2024-09-15T02:02:46.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.732+0000 7fc1cd426640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 0x7fc1ac03aba0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc1c806ba90 tx=0x7fc1b8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:46.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:46.733+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc1c404bf90 con 0x7fc1c806f070 2024-09-15T02:02:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.350+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc1c400ed00 con 0x7fc1c806f070 2024-09-15T02:02:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.350+0000 7fc1bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:47.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.350+0000 7fc1bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.582+0000 7fc1cfe8b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.582+0000 7fc1cfe8b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.582+0000 7fc1cfe8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.582+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc1c80724a0 con 0x7fc1c806f070 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.583+0000 7fc1bdffb640 1 -- 172.21.15.17:0/3241541806 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fc1c4017030 con 0x7fc1c806f070 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.583+0000 7fc1bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.583+0000 7fc1bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:47.585 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 msgr2=0x7fc1ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 0x7fc1ac03aba0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc1c806ba90 tx=0x7fc1b8032000 comp rx=0 tx=0).stop 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 10 monclient: shutdown 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 msgr2=0x7fc1c8133f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.584+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400b630 tx=0x7fc1c40091a0 comp rx=0 tx=0).stop 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 shutdown_connections 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1ac0386f0 0x7fc1ac03aba0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c809c690 0x7fc1c811da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc1c809c2c0 0x7fc1c81411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 --2- 172.21.15.17:0/3241541806 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc1c806f070 0x7fc1c8133f50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 >> 172.21.15.17:0/3241541806 conn(0x7fc1c80a2f70 msgr2=0x7fc1c80a0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 shutdown_connections 2024-09-15T02:02:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:47.585+0000 7fc1cfe8b640 1 -- 172.21.15.17:0/3241541806 wait complete. 2024-09-15T02:02:50.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:50.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.705+0000 7f853fe99640 10 monclient: get_monmap_and_config 2024-09-15T02:02:50.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.705+0000 7f853fe99640 10 monclient: build_initial_monmap 2024-09-15T02:02:50.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.705+0000 7f853fe99640 10 monclient: monmap: 2024-09-15T02:02:50.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:50.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:50.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:50.706138+0000 2024-09-15T02:02:50.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:50.706138+0000 2024-09-15T02:02:50.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:50.709 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.709 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:50.709 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:50.709 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:50.709 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:50.710 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:50.710 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.705+0000 7f853fe99640 1 Processor -- start 2024-09-15T02:02:50.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 -- start start 2024-09-15T02:02:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient: init 2024-09-15T02:02:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T02:02:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f85380a58b0 0x7f8538132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-a con 0x7f85380a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:02:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85380980a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8538136dd0 0x7f85381391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-c con 0x7f8538136dd0 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380a01e0 con 0x7f85380980a0 2024-09-15T02:02:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:50.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380a1580 con 0x7f85380a58b0 2024-09-15T02:02:50.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:50.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380a5c80 con 0x7f8538136dd0 2024-09-15T02:02:50.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:50.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853fe99640 10 monclient(hunting): authenticate will time out at 1430.702802s 2024-09-15T02:02:50.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f85380a58b0 0x7f8538132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:50.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f85380a58b0 0x7f8538132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54270/0 (socket says 172.21.15.17:54270) 2024-09-15T02:02:50.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 1 -- 172.21.15.17:0/1550948834 learned_addr learned my addr 172.21.15.17:0/1550948834 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:50.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 10 monclient(hunting): get_auth_request con 0x7f85380a58b0 auth_method 0 2024-09-15T02:02:50.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:50.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:50.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.706+0000 7f853dc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:50.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:50.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:50.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:50.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853e436640 1 --2- 172.21.15.17:0/1550948834 >> v2:172.21.15.67:3300/0 conn(0x7f8538136dd0 0x7f85381391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:50.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853e436640 10 monclient(hunting): get_auth_request con 0x7f8538136dd0 auth_method 0 2024-09-15T02:02:50.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853e436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:50.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853e436640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:50.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853e436640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:50.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient(hunting): handle_auth_done global_id 105114 payload 274 2024-09-15T02:02:50.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 1 -- 172.21.15.17:0/1550948834 >> v2:172.21.15.67:3300/0 conn(0x7f8538136dd0 msgr2=0x7f85381391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:50.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 1 --2- 172.21.15.17:0/1550948834 >> v2:172.21.15.67:3300/0 conn(0x7f8538136dd0 0x7f85381391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 1 -- 172.21.15.17:0/1550948834 >> v1:172.21.15.67:6789/0 conn(0x7f85380980a0 legacy=0x7f85380984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:50.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient: _finish_hunting 0 2024-09-15T02:02:50.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 1 monclient: found mon.noname-a 2024-09-15T02:02:50.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:50.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.707+0000 7f853dc35640 1 -- 172.21.15.17:0/1550948834 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538098bb0 con 0x7f85380a58b0 2024-09-15T02:02:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853dc35640 1 -- 172.21.15.17:0/1550948834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380a58b0 msgr2=0x7f8538132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853dc35640 1 --2- 172.21.15.17:0/1550948834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380a58b0 0x7f8538132040 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8538096b10 tx=0x7f85340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25639ab2c6f4b8bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 1 -- 172.21.15.17:0/1550948834 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534032440 con 0x7f85380a58b0 2024-09-15T02:02:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: dump: 2024-09-15T02:02:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:50.740 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:50.740 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: _finish_auth 0 2024-09-15T02:02:50.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:50.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 1 -- 172.21.15.17:0/1550948834 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85340325a0 con 0x7f85380a58b0 2024-09-15T02:02:50.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 5 monclient: authenticate success, global_id 105114 2024-09-15T02:02:50.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:50.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:50.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:50.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 4 set_mon_vals no callback set 2024-09-15T02:02:50.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 1 -- 172.21.15.17:0/1550948834 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534032940 con 0x7f85380a58b0 2024-09-15T02:02:50.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:50.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:50.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853cc33640 10 monclient: dump: 2024-09-15T02:02:50.742 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:50.743 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:50.743 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.743 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.743 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:50.743 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:50.744 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 10 monclient: shutdown 2024-09-15T02:02:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 -- 172.21.15.17:0/1550948834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380a58b0 msgr2=0x7f8538132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 --2- 172.21.15.17:0/1550948834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380a58b0 0x7f8538132040 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8538096b10 tx=0x7f85340091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 -- 172.21.15.17:0/1550948834 shutdown_connections 2024-09-15T02:02:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 --2- 172.21.15.17:0/1550948834 >> v2:172.21.15.67:3300/0 conn(0x7f8538136dd0 0x7f85381391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 --2- 172.21.15.17:0/1550948834 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380a58b0 0x7f8538132040 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 -- 172.21.15.17:0/1550948834 >> 172.21.15.17:0/1550948834 conn(0x7f8538070ed0 msgr2=0x7f85380a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.710+0000 7f853fe99640 1 -- 172.21.15.17:0/1550948834 shutdown_connections 2024-09-15T02:02:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 1 -- 172.21.15.17:0/1550948834 wait complete. 2024-09-15T02:02:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 10 monclient: build_initial_monmap 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 10 monclient: monmap: 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:50.747 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:50.748 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.748 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:50.748 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:50.748 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:50.748 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 1 Processor -- start 2024-09-15T02:02:50.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 1 -- start start 2024-09-15T02:02:50.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.711+0000 7f853fe99640 10 monclient: init 2024-09-15T02:02:50.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:50.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:02:50.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-a con 0x7f85380980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:50.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 0x7f8538133270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-c con 0x7f85380a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:50.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538136dd0 0x7f85381337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): picked mon.noname-b con 0x7f8538136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:50.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:50.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538130f90 con 0x7f85380980a0 2024-09-15T02:02:50.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:50.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538136370 con 0x7f8538136dd0 2024-09-15T02:02:50.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:50.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8538098dc0 con 0x7f85380a58b0 2024-09-15T02:02:50.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:50.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853fe99640 10 monclient(hunting): authenticate will time out at 1430.708480s 2024-09-15T02:02:50.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853d434640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 msgr2=0x7f8538133270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:50.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853d434640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 0x7f8538133270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:50.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:50.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:54280/0 (socket says 172.21.15.17:54280) 2024-09-15T02:02:50.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 -- 172.21.15.17:0/2433837392 learned_addr learned my addr 172.21.15.17:0/2433837392 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:50.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): get_auth_request con 0x7f85380980a0 auth_method 0 2024-09-15T02:02:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538136dd0 0x7f85381337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:50.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): get_auth_request con 0x7f8538136dd0 auth_method 0 2024-09-15T02:02:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:50.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:50.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:50.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:50.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853e436640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:50.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient(hunting): handle_auth_done global_id 105117 payload 824 2024-09-15T02:02:50.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 -- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 msgr2=0x7f8538133270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:50.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 0x7f8538133270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 -- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538136dd0 msgr2=0x7f85381337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:50.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538136dd0 0x7f85381337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient: _finish_hunting 0 2024-09-15T02:02:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 monclient: found mon.noname-a 2024-09-15T02:02:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.712+0000 7f853dc35640 1 -- 172.21.15.17:0/2433837392 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534009450 con 0x7f85380980a0 2024-09-15T02:02:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853dc35640 1 -- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 msgr2=0x7f8538132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:50.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853dc35640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f853402a790 tx=0x7f8534032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:50.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534031a40 con 0x7f85380980a0 2024-09-15T02:02:50.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:50.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:50.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: dump: 2024-09-15T02:02:50.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:50.760 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:50.760 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.760 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.760 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: _finish_auth 0 2024-09-15T02:02:50.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:50.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534031be0 con 0x7f85380980a0 2024-09-15T02:02:50.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 5 monclient: authenticate success, global_id 105117 2024-09-15T02:02:50.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 10 monclient: _renew_subs 2024-09-15T02:02:50.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8538134720 con 0x7f85380980a0 2024-09-15T02:02:50.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:50.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853403d440 con 0x7f85380980a0 2024-09-15T02:02:50.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:50.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:50.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852dffb640 10 monclient: dump: 2024-09-15T02:02:50.764 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:50.764 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:50.764 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.764 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:50.765 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:50.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 10 monclient: _renew_subs 2024-09-15T02:02:50.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f852ffff640 4 set_mon_vals no callback set 2024-09-15T02:02:50.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.715+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8538134de0 con 0x7f85380980a0 2024-09-15T02:02:50.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8534039040 con 0x7f85380980a0 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f853fe99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f853fe99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f853fe99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85381361d0 con 0x7f85380980a0 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f852dffb640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 0x7f851c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:50.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.716+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f853406c380 con 0x7f85380980a0 2024-09-15T02:02:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.717+0000 7f853d434640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 0x7f851c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.717+0000 7f853d434640 10 monclient: get_auth_request con 0x7f851c0386f0 auth_method 0 2024-09-15T02:02:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.719+0000 7f853d434640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 0x7f851c03aba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8528002ee0 tx=0x7f8528002aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.721+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85381361d0 con 0x7f85380980a0 2024-09-15T02:02:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.721+0000 7f852dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:50.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.721+0000 7f852dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:50.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.948+0000 7f853fe99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:50.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.948+0000 7f853fe99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:50.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.948+0000 7f853fe99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:50.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.948+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f85381361d0 con 0x7f85380980a0 2024-09-15T02:02:50.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.949+0000 7f852dffb640 1 -- 172.21.15.17:0/2433837392 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f85381361d0 con 0x7f85380980a0 2024-09-15T02:02:50.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.949+0000 7f852dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:50.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.949+0000 7f852dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:50.952 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:50.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 msgr2=0x7f851c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:50.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 0x7f851c03aba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8528002ee0 tx=0x7f8528002aa0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 10 monclient: shutdown 2024-09-15T02:02:50.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:50.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 msgr2=0x7f8538132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:50.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f853402a790 tx=0x7f8534032000 comp rx=0 tx=0).stop 2024-09-15T02:02:50.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 shutdown_connections 2024-09-15T02:02:50.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f851c0386f0 0x7f851c03aba0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538136dd0 0x7f85381337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85380a58b0 0x7f8538133270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 --2- 172.21.15.17:0/2433837392 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85380980a0 0x7f8538132d30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:50.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 >> 172.21.15.17:0/2433837392 conn(0x7f8538070ed0 msgr2=0x7f853809afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:50.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 shutdown_connections 2024-09-15T02:02:50.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:50.951+0000 7f853fe99640 1 -- 172.21.15.17:0/2433837392 wait complete. 2024-09-15T02:02:53.965 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:54.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.070+0000 7f33770c9640 10 monclient: get_monmap_and_config 2024-09-15T02:02:54.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.070+0000 7f33770c9640 10 monclient: build_initial_monmap 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.070+0000 7f33770c9640 10 monclient: monmap: 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:54.071647+0000 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:54.071647+0000 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:54.073 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:54.074 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.071+0000 7f33770c9640 1 Processor -- start 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.071+0000 7f33770c9640 1 -- start start 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.071+0000 7f33770c9640 10 monclient: init 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T02:02:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3370067090 0x7f33701321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3370067090 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3370067460 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 0x7f3370139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3370136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337009cbc0 con 0x7f3370067460 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3370067f40 con 0x7f3370067090 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337009e550 con 0x7f3370136f90 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f33770c9640 10 monclient(hunting): authenticate will time out at 1434.068199s 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3375666640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 msgr2=0x7f3370139370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3375666640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 0x7f3370139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3370067090 0x7f33701321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3370067090 0x7f33701321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:38324/0 (socket says 172.21.15.17:38324) 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f336ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3370067460 0x7f3370067830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:56448/0 (socket says 172.21.15.17:56448) 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 -- 172.21.15.17:0/3323484197 learned_addr learned my addr 172.21.15.17:0/3323484197 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): get_auth_request con 0x7f3370067090 auth_method 0 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:54.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient(hunting): handle_auth_done global_id 144214 payload 274 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 -- 172.21.15.17:0/3323484197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 msgr2=0x7f3370139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 --2- 172.21.15.17:0/3323484197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 0x7f3370139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 -- 172.21.15.17:0/3323484197 >> v1:172.21.15.67:6789/0 conn(0x7f3370067460 legacy=0x7f3370067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient: _finish_hunting 0 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 monclient: found mon.noname-c 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:02:54.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.072+0000 7f3374e65640 1 -- 172.21.15.17:0/3323484197 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337009e1e0 con 0x7f3370067090 2024-09-15T02:02:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.075+0000 7f3374e65640 1 -- 172.21.15.17:0/3323484197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 msgr2=0x7f33701321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.075+0000 7f3374e65640 1 --2- 172.21.15.17:0/3323484197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f33701321a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3360009910 tx=0x7f33600091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=837d85dcbdcad9a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 1 -- 172.21.15.17:0/3323484197 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360032440 con 0x7f3370067090 2024-09-15T02:02:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: dump: 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:54.086 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 1 -- 172.21.15.17:0/3323484197 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33600325a0 con 0x7f3370067090 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 5 monclient: authenticate success, global_id 144214 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 10 monclient: get_monmap_and_config success 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 1 -- 172.21.15.17:0/3323484197 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360032940 con 0x7f3370067090 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 4 set_mon_vals no callback set 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:02:54.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f336f7fe640 10 monclient: dump: 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:54.088 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 10 monclient: shutdown 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 -- 172.21.15.17:0/3323484197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 msgr2=0x7f33701321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 --2- 172.21.15.17:0/3323484197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f33701321a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3360009910 tx=0x7f33600091d0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 -- 172.21.15.17:0/3323484197 shutdown_connections 2024-09-15T02:02:54.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 --2- 172.21.15.17:0/3323484197 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370136f90 0x7f3370139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 --2- 172.21.15.17:0/3323484197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f33701321a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 -- 172.21.15.17:0/3323484197 >> 172.21.15.17:0/3323484197 conn(0x7f33700a1ee0 msgr2=0x7f33700a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 -- 172.21.15.17:0/3323484197 shutdown_connections 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.076+0000 7f33770c9640 1 -- 172.21.15.17:0/3323484197 wait complete. 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.077+0000 7f33770c9640 10 monclient: build_initial_monmap 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.077+0000 7f33770c9640 10 monclient: monmap: 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:54.090 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.077+0000 7f33770c9640 1 Processor -- start 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.077+0000 7f33770c9640 1 -- start start 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.077+0000 7f33770c9640 10 monclient: init 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f3370135c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3370067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 0x7f337012eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3370067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3370136f90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370136790 con 0x7f3370136f90 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337009d530 con 0x7f3370067090 2024-09-15T02:02:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33700a0030 con 0x7f3370067460 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f33770c9640 10 monclient(hunting): authenticate will time out at 1434.074401s 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f336ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 msgr2=0x7f337012eaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f336ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 0x7f337012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33620/0 (socket says 172.21.15.17:33620) 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 -- 172.21.15.17:0/3139331179 learned_addr learned my addr 172.21.15.17:0/3139331179 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): get_auth_request con 0x7f3370136f90 auth_method 0 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:54.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f3370135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:54.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:54.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:54.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): get_auth_request con 0x7f3370067090 auth_method 0 2024-09-15T02:02:54.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:54.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:02:54.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3374e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient(hunting): handle_auth_done global_id 105120 payload 824 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 -- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 msgr2=0x7f337012eaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 0x7f337012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 -- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 msgr2=0x7f3370135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f3370135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient: _finish_hunting 0 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 monclient: found mon.noname-a 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:02:54.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.078+0000 7f3375666640 1 -- 172.21.15.17:0/3139331179 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360009450 con 0x7f3370136f90 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f3375666640 1 -- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 msgr2=0x7f3370118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f3375666640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f336800c040 tx=0x7f3368009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336800ce20 con 0x7f3370136f90 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: dump: 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: _finish_auth 0 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 5 monclient: authenticate success, global_id 105120 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 10 monclient: _renew_subs 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3368006360 con 0x7f3370136f90 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33701422b0 con 0x7f3370136f90 2024-09-15T02:02:54.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368006510 con 0x7f3370136f90 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336cff9640 10 monclient: dump: 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:02:54.100 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 10 monclient: _renew_subs 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370119ca0 con 0x7f3370136f90 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.081+0000 7f336effd640 4 set_mon_vals no callback set 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3368012020 con 0x7f3370136f90 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f33770c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f33770c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f33770c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:54.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337011b000 con 0x7f3370136f90 2024-09-15T02:02:54.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f336cff9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 0x7f335c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:54.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f336804c810 con 0x7f3370136f90 2024-09-15T02:02:54.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f3374e65640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 0x7f335c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:54.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.082+0000 7f3374e65640 10 monclient: get_auth_request con 0x7f335c0386f0 auth_method 0 2024-09-15T02:02:54.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.085+0000 7f3374e65640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 0x7f335c03aba0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f337009c510 tx=0x7f3360032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:02:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.087+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f33680132e0 con 0x7f3370136f90 2024-09-15T02:02:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.087+0000 7f336cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:02:54.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.087+0000 7f336cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:02:54.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f33770c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:02:54.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f33770c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:54.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f33770c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:02:54.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f33700724a0 con 0x7f3370136f90 2024-09-15T02:02:54.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f336cff9640 1 -- 172.21.15.17:0/3139331179 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f33680072f0 con 0x7f3370136f90 2024-09-15T02:02:54.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f336cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:02:54.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.311+0000 7f336cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:02:54.314 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:02:54.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 msgr2=0x7f335c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 0x7f335c03aba0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f337009c510 tx=0x7f3360032000 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 10 monclient: shutdown 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 msgr2=0x7f3370118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f336800c040 tx=0x7f3368009730 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 shutdown_connections 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f335c0386f0 0x7f335c03aba0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3370136f90 0x7f3370118540 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3370067460 0x7f337012eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 --2- 172.21.15.17:0/3139331179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370067090 0x7f3370135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 >> 172.21.15.17:0/3139331179 conn(0x7f33700a1ee0 msgr2=0x7f33700a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:02:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 shutdown_connections 2024-09-15T02:02:54.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:54.313+0000 7f33770c9640 1 -- 172.21.15.17:0/3139331179 wait complete. 2024-09-15T02:02:57.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.433+0000 7f0955d1d640 10 monclient: get_monmap_and_config 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.433+0000 7f0955d1d640 10 monclient: build_initial_monmap 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.433+0000 7f0955d1d640 10 monclient: monmap: 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:02:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:02:57.434314+0000 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:02:57.434314+0000 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:02:57.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 Processor -- start 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 -- start start 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient: init 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f095006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:02:57.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f095006f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f095009e970 addr v2:172.21.15.67:3300/0 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095009ed90 con 0x7f095006f440 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095009f6a0 con 0x7f095009e970 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0950069f40 con 0x7f095006eb30 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f0955d1d640 10 monclient(hunting): authenticate will time out at 1437.430838s 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f094f7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 msgr2=0x7f095006ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.434+0000 7f094f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:57.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:38338/0 (socket says 172.21.15.17:38338) 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 learned_addr learned my addr 172.21.15.17:0/977183233 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 10 monclient(hunting): get_auth_request con 0x7f095009e970 auth_method 0 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:57.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.435+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:02:57.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.437+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v1:172.21.15.67:6789/0 conn(0x7f095006f440 legacy=0x7f095009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:02:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.635+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 msgr2=0x7f095006ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.635+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 10 monclient(hunting): get_auth_request con 0x7f095009e970 auth_method 0 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:57.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:57.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:57.636+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:02:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.036+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 msgr2=0x7f095006ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:58.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.036+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:02:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 10 monclient(hunting): get_auth_request con 0x7f095009e970 auth_method 0 2024-09-15T02:02:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:58.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:58.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.037+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:02:58.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.837+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 msgr2=0x7f095006ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:02:58.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.837+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 10 monclient(hunting): get_auth_request con 0x7f095009e970 auth_method 0 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:02:58.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:02:58.839+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:03:00.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): tick 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 monclient(hunting): continuing hunt 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 msgr2=0x7f095006ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f095006eb30 0x7f095006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 msgr2=0x7f095009c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.67:3300/0 conn(0x7f095009e970 0x7f095009c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 -- 172.21.15.17:0/977183233 >> v1:172.21.15.67:6789/0 conn(0x7f095006f440 legacy=0x7f095009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient: _add_conns ranks=[0,3] 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f093c003720 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f093c007440 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 -- 172.21.15.17:0/977183233 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0950069950 con 0x7f093c003720 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 1 -- 172.21.15.17:0/977183233 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095009f6a0 con 0x7f093c007440 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094d7fa640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 10 monclient(hunting): get_auth_request con 0x7f093c007440 auth_method 0 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:00.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:00.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:00.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:00.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.435+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:00.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.437+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> v1:172.21.15.17:6789/0 conn(0x7f093c003720 legacy=0x7f093c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 10 monclient(hunting): get_auth_request con 0x7f093c007440 auth_method 0 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:00.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:00.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:00.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:00.636+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 10 monclient(hunting): get_auth_request con 0x7f093c007440 auth_method 0 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:01.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.037+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.065+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837837141 0 0) 0x7f0950069950 con 0x7f093c003720 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.065+0000 7f094e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.065+0000 7f094e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.065+0000 7f094e7fc640 10 monclient(hunting): my global_id is 105123 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.065+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09380034e0 con 0x7f093c003720 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555385184 0 0) 0x7f09380034e0 con 0x7f093c003720 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 msgr2=0x7f093c0098f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 --2- 172.21.15.17:0/977183233 >> v2:172.21.15.17:3300/0 conn(0x7f093c007440 0x7f093c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c00b000 con 0x7f093c003720 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f0940004380 con 0x7f093c003720 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f0955d1d640 5 monclient: authenticate success, global_id 105123 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f0955d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: dump: 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.078 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 >> v1:172.21.15.17:6789/0 conn(0x7f093c003720 legacy=0x7f093c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 reap_dead start 2024-09-15T02:03:01.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 0x7f0938003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): picked mon.a con 0x7f09380039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f0938004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): picked mon.b con 0x7f0938004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 0x7f093800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): picked mon.c con 0x7f0938005180 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095009f6a0 con 0x7f09380039d0 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c120 con 0x7f0938004780 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c2e0 con 0x7f0938005180 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f0955d1d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094ffff640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 msgr2=0x7f093800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 0x7f0938003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094ffff640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 0x7f093800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): get_auth_request con 0x7f09380039d0 auth_method 0 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094effd640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f0938004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:01.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094effd640 10 monclient(hunting): get_auth_request con 0x7f0938004780 auth_method 0 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.066+0000 7f094effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 10 monclient(hunting): handle_auth_done global_id 105123 payload 293 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 msgr2=0x7f093800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 0x7f093800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 msgr2=0x7f0938004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f0938004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 monclient: found mon.a 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380034e0 con 0x7f09380039d0 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.067+0000 7f094f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094f7fe640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 msgr2=0x7f0938003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094f7fe640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 0x7f0938003dc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0940003940 tx=0x7f0940007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4911737b0d5c8b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940007240 con 0x7f09380039d0 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: dump: 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.084 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f0955d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09400073c0 con 0x7f09380039d0 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f0955d1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f0955d1d640 4 set_mon_vals no callback set 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 1 -- 172.21.15.17:0/977183233 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09400048a0 con 0x7f09380039d0 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.070+0000 7f094e7fc640 10 monclient: dump: 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.085 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 10 monclient: shutdown 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 -- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 msgr2=0x7f0938003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 0x7f0938003dc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0940003940 tx=0x7f0940007c50 comp rx=0 tx=0).stop 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 -- 172.21.15.17:0/977183233 shutdown_connections 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0938005180 0x7f093800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f0938004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 --2- 172.21.15.17:0/977183233 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09380039d0 0x7f0938003dc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 -- 172.21.15.17:0/977183233 >> 172.21.15.17:0/977183233 conn(0x7f09500712f0 msgr2=0x7f09500a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 -- 172.21.15.17:0/977183233 shutdown_connections 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 1 -- 172.21.15.17:0/977183233 wait complete. 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 10 monclient: build_initial_monmap 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.071+0000 7f0955d1d640 10 monclient: monmap: 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:01.087 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.072+0000 7f0955d1d640 1 Processor -- start 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.072+0000 7f0955d1d640 1 -- start start 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.072+0000 7f0955d1d640 10 monclient: init 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 0x7f095011f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f09380039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f095013cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0938004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094f7fe640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 msgr2=0x7f095011f5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0938005180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 0x7f095011f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095009cd10 con 0x7f0938005180 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500a2fe0 con 0x7f0938004780 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0950121bb0 con 0x7f09380039d0 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f0955d1d640 10 monclient(hunting): authenticate will time out at 1441.069214s 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33658/0 (socket says 172.21.15.17:33658) 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 -- 172.21.15.17:0/173768563 learned_addr learned my addr 172.21.15.17:0/173768563 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): get_auth_request con 0x7f0938005180 auth_method 0 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f095013cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): get_auth_request con 0x7f0938004780 auth_method 0 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient(hunting): handle_auth_done global_id 105126 payload 824 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 -- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 msgr2=0x7f095011f5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 0x7f095011f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 -- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 msgr2=0x7f095013cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f095013cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 monclient: found mon.noname-a 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.073+0000 7f094ffff640 1 -- 172.21.15.17:0/173768563 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950121e90 con 0x7f0938005180 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.075+0000 7f094ffff640 1 -- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 msgr2=0x7f09501409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.075+0000 7f094ffff640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f093c0073b0 tx=0x7f0948009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800b2e0 con 0x7f0938005180 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: dump: 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.094 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800f040 con 0x7f0938005180 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 5 monclient: authenticate success, global_id 105126 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948009720 con 0x7f0938005180 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f09377fe640 10 monclient: dump: 2024-09-15T02:03:01.095 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 10 monclient: _renew_subs 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f094d7fa640 4 set_mon_vals no callback set 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095013df00 con 0x7f0938005180 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 10 monclient: _renew_subs 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.076+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095013d830 con 0x7f0938005180 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0948009ad0 con 0x7f0938005180 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f0955d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f0955d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f0955d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0950122250 con 0x7f0938005180 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f09377fe640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 0x7f093c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f094804cb20 con 0x7f0938005180 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f094f7fe640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 0x7f093c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.077+0000 7f094f7fe640 10 monclient: get_auth_request con 0x7f093c039060 auth_method 0 2024-09-15T02:03:01.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.080+0000 7f094f7fe640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 0x7f093c03b510 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0940004560 tx=0x7f0940035000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.082+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f094801d590 con 0x7f0938005180 2024-09-15T02:03:01.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.082+0000 7f09377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.082+0000 7f09377fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:01.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f0955d1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:03:01.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f0955d1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:01.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f0955d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f09500724a0 con 0x7f0938005180 2024-09-15T02:03:01.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f09377fe640 1 -- 172.21.15.17:0/173768563 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f0948027b40 con 0x7f0938005180 2024-09-15T02:03:01.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f09377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:01.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.312+0000 7f09377fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:01.315 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:03:01.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 msgr2=0x7f093c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 0x7f093c03b510 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0940004560 tx=0x7f0940035000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 10 monclient: shutdown 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 msgr2=0x7f09501409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f093c0073b0 tx=0x7f0948009040 comp rx=0 tx=0).stop 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 shutdown_connections 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f093c039060 0x7f093c03b510 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0938005180 0x7f09501409b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0938004780 0x7f095013cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 --2- 172.21.15.17:0/173768563 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09380039d0 0x7f095011f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 >> 172.21.15.17:0/173768563 conn(0x7f09500712f0 msgr2=0x7f095009c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 shutdown_connections 2024-09-15T02:03:01.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.314+0000 7f0955d1d640 1 -- 172.21.15.17:0/173768563 wait complete. 2024-09-15T02:03:01.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:03:01.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:03:01.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 10 monclient: get_monmap_and_config 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 10 monclient: build_initial_monmap 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 10 monclient: monmap: 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:01.424051+0000 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:01.424051+0000 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:01.426 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:01.426 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:01.426 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:01.426 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:01.426 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 1 Processor -- start 2024-09-15T02:03:01.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 1 -- start start 2024-09-15T02:03:01.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.423+0000 7fe9f386f640 10 monclient: init 2024-09-15T02:03:01.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9ec09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-b con 0x7fe9ec09c4f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9ec09aed0 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9ec09c8c0 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec06a390 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec09f790 con 0x7fe9ec09aed0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec099f50 con 0x7fe9ec09c4f0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f386f640 10 monclient(hunting): authenticate will time out at 1441.420508s 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f160b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 msgr2=0x7fe9ec09a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f160b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9ec09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f1e0c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe9ec09c8c0 0x7fe9ec139be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44234/0 (socket says 172.21.15.17:44234) 2024-09-15T02:03:01.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.424+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 learned_addr learned my addr 172.21.15.17:0/2870252992 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:01.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 >> v1:172.21.15.17:6789/0 conn(0x7fe9ec09c8c0 legacy=0x7fe9ec139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180488571 0 0) 0x7fe9ec06a390 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9dffff640 10 monclient(hunting): my global_id is 105129 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.426+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d80034e0 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/2870252992 >> v1:172.21.15.67:6789/0 conn(0x7fe9ec09aed0 legacy=0x7fe9ec06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552875755 0 0) 0x7fe9d80034e0 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 msgr2=0x7fe9ec09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9ec09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 >> v1:172.21.15.67:6789/0 conn(0x7fe9ec09aed0 legacy=0x7fe9ec06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec069d40 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f386f640 5 monclient: authenticate success, global_id 105129 2024-09-15T02:03:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe9e4001aa0 con 0x7fe9ec09c8c0 2024-09-15T02:03:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f386f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: dump: 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.440 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 >> v1:172.21.15.17:6789/0 conn(0x7fe9ec09c8c0 legacy=0x7fe9ec139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8003a80 0x7fe9d8003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): picked mon.b con 0x7fe9d8003a80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9d8004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): picked mon.c con 0x7fe9ec09c4f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 0x7fe9d800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): picked mon.a con 0x7fe9d8004e70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec099f50 con 0x7fe9d8004e70 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8005620 con 0x7fe9d8003a80 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d800bf60 con 0x7fe9ec09c4f0 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9dffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f386f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 msgr2=0x7fe9d8004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f0e0a640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9d8004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 0x7fe9d800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): get_auth_request con 0x7fe9d8004e70 auth_method 0 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8003a80 0x7fe9d8003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): get_auth_request con 0x7fe9d8003a80 auth_method 0 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient(hunting): handle_auth_done global_id 105129 payload 293 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 msgr2=0x7fe9d8004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec09c4f0 0x7fe9d8004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8003a80 msgr2=0x7fe9d8003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8003a80 0x7fe9d8003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 monclient: found mon.a 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80034e0 con 0x7fe9d8004e70 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.427+0000 7fe9f1e0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 msgr2=0x7fe9d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 0x7fe9d800b9d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001810 tx=0x7fe9e400a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300ba6a1d9a3c569 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e402b680 con 0x7fe9d8004e70 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: dump: 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9f386f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e402b800 con 0x7fe9d8004e70 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 1 -- 172.21.15.17:0/2870252992 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e402bbc0 con 0x7fe9d8004e70 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9f386f640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9f386f640 4 set_mon_vals no callback set 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.430+0000 7fe9dffff640 10 monclient: dump: 2024-09-15T02:03:01.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 10 monclient: shutdown 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 -- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 msgr2=0x7fe9d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 --2- 172.21.15.17:0/2870252992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9d8004e70 0x7fe9d800b9d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001810 tx=0x7fe9e400a220 comp rx=0 tx=0).stop 2024-09-15T02:03:01.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/2870252992 reap_dead start 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 -- 172.21.15.17:0/2870252992 shutdown_connections 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 -- 172.21.15.17:0/2870252992 >> 172.21.15.17:0/2870252992 conn(0x7fe9ec071300 msgr2=0x7fe9ec0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 -- 172.21.15.17:0/2870252992 shutdown_connections 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 1 -- 172.21.15.17:0/2870252992 wait complete. 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 10 monclient: build_initial_monmap 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.431+0000 7fe9f386f640 10 monclient: monmap: 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:01.448 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 Processor -- start 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 -- start start 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient: init 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8004e70 0x7fe9ec133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d8004e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-a con 0x7fe9ec1341d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec11df90 0x7fe9ec11e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): picked mon.noname-c con 0x7fe9ec11df90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec09cdb0 con 0x7fe9ec1341d0 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec121090 con 0x7fe9d8004e70 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec11fec0 con 0x7fe9ec11df90 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f386f640 10 monclient(hunting): authenticate will time out at 1441.428764s 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f1e0c640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec11df90 msgr2=0x7fe9ec11e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f1e0c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec11df90 0x7fe9ec11e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33680/0 (socket says 172.21.15.17:33680) 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/3151085007 learned_addr learned my addr 172.21.15.17:0/3151085007 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): get_auth_request con 0x7fe9ec1341d0 auth_method 0 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.432+0000 7fe9f0e0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8004e70 0x7fe9ec133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 10 monclient(hunting): handle_auth_done global_id 105132 payload 824 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec11df90 msgr2=0x7fe9ec11e3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9ec11df90 0x7fe9ec11e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8004e70 msgr2=0x7fe9ec133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8004e70 0x7fe9ec133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 monclient: found mon.noname-a 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec141e70 con 0x7fe9ec1341d0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f160b640 10 monclient: get_auth_request con 0x7fe9d8004e70 auth_method 0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.433+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8004e70 0x7fe9ec133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f0e0a640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 msgr2=0x7fe9ec11da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f0e0a640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800c7c0 tx=0x7fe9e8007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8007300 con 0x7fe9ec1341d0 2024-09-15T02:03:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: dump: 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e8007460 con 0x7fe9ec1341d0 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 5 monclient: authenticate success, global_id 105132 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 10 monclient: _renew_subs 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec120db0 con 0x7fe9ec1341d0 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80059d0 con 0x7fe9ec1341d0 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9dd7fa640 10 monclient: dump: 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.454 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9df7fe640 4 set_mon_vals no callback set 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 10 monclient: _renew_subs 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.435+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec1364c0 con 0x7fe9ec1341d0 2024-09-15T02:03:01.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f386f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f386f640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 0x7fe9ec06bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec063bb0 con 0x7fe9ec11e920 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe9e800a640 con 0x7fe9ec1341d0 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 0x7fe9ec06bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient: get_auth_request con 0x7fe9ec11e920 auth_method 0 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 10 monclient(hunting): handle_auth_done global_id 105135 payload 824 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9dd7fa640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9d0038740 0x7fe9d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9f160b640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9ec121250 con 0x7fe9ec11e920 2024-09-15T02:03:01.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.436+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe9e804bbd0 con 0x7fe9ec1341d0 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.437+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9d0038740 0x7fe9d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.437+0000 7fe9f1e0c640 10 monclient: get_auth_request con 0x7fe9d0038740 auth_method 0 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9f160b640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 msgr2=0x7fe9ec06bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 0x7fe9ec06bae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00096d0 tx=0x7fe9e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0010a90 con 0x7fe9ec11e920 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 10 monclient: dump: 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe9e0010bf0 con 0x7fe9ec11e920 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 msgr2=0x7fe9ec06bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 0x7fe9ec06bae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00096d0 tx=0x7fe9e0008040 comp rx=0 tx=0).stop 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 msgr2=0x7fe9ec06bae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9dd7fa640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec11e920 0x7fe9ec06bae0 secure :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00096d0 tx=0x7fe9e0008040 comp rx=0 tx=0).stop 2024-09-15T02:03:01.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.439+0000 7fe9f1e0c640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9d0038740 0x7fe9d003abf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001c80 tx=0x7fe9e4031000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f386f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f386f640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 0x7fe9ec06c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f386f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec120f40 con 0x7fe9ec06c020 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 0x7fe9ec06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient: get_auth_request con 0x7fe9ec06c020 auth_method 0 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 10 monclient(hunting): handle_auth_done global_id 105138 payload 824 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.445+0000 7fe9f160b640 1 -- 172.21.15.17:0/3151085007 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe9ec0724a0 con 0x7fe9ec06c020 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.447+0000 7fe9f160b640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 msgr2=0x7fe9ec06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.447+0000 7fe9f160b640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 0x7fe9ec06c420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0003d00 tx=0x7fe9e001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00209e0 con 0x7fe9ec06c020 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 10 monclient: dump: 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.464 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7fe9e0020b40 con 0x7fe9ec06c020 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 msgr2=0x7fe9ec06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 0x7fe9ec06c420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0003d00 tx=0x7fe9e001a000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 msgr2=0x7fe9ec06c420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:01.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9dd7fa640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec06c020 0x7fe9ec06c420 secure :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0003d00 tx=0x7fe9e001a000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 126, 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:01.466 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:01.467 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:01.468 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:01.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:01.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:01.472 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.473 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.474 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:01.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:01.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:01.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.481 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.481 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:01.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.481 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:01.482 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9d0038740 msgr2=0x7fe9d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe9d0038740 0x7fe9d003abf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001c80 tx=0x7fe9e4031000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 10 monclient: shutdown 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 msgr2=0x7fe9ec11da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800c7c0 tx=0x7fe9e8007dc0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f1e0c640 1 -- 172.21.15.17:0/3151085007 reap_dead start 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 shutdown_connections 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 --2- 172.21.15.17:0/3151085007 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe9ec1341d0 0x7fe9ec11da50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 >> 172.21.15.17:0/3151085007 conn(0x7fe9ec071300 msgr2=0x7fe9ec0a2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 shutdown_connections 2024-09-15T02:03:01.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.448+0000 7fe9f386f640 1 -- 172.21.15.17:0/3151085007 wait complete. 2024-09-15T02:03:01.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:03:01.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:03:01.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.580+0000 7f46b800f640 10 monclient: get_monmap_and_config 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.580+0000 7f46b800f640 10 monclient: build_initial_monmap 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.580+0000 7f46b800f640 10 monclient: monmap: 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:01.581683+0000 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:01.581683+0000 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.581+0000 7f46b800f640 1 Processor -- start 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.581+0000 7f46b800f640 1 -- start start 2024-09-15T02:03:01.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.581+0000 7f46b800f640 10 monclient: init 2024-09-15T02:03:01.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T02:03:01.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46b0097320 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f46b01339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-b con 0x7f46b009a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46b009a960 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b00985d0 con 0x7f46b009a960 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b009d200 con 0x7f46b0097320 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b00992f0 con 0x7f46b009a590 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b800f640 10 monclient(hunting): authenticate will time out at 1441.578153s 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b55aa640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 msgr2=0x7f46b01339d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b55aa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f46b01339d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b65ac640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f46b009a960 0x7f46b0135f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:44244/0 (socket says 172.21.15.17:44244) 2024-09-15T02:03:01.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.582+0000 7f46b65ac640 1 -- 172.21.15.17:0/4129461217 learned_addr learned my addr 172.21.15.17:0/4129461217 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b65ac640 1 -- 172.21.15.17:0/4129461217 >> v1:172.21.15.17:6789/0 conn(0x7f46b009a960 legacy=0x7f46b0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802215691 0 0) 0x7f46b00985d0 con 0x7f46b009a960 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): my global_id is 105141 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c0034e0 con 0x7f46b009a960 2024-09-15T02:03:01.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b5dab640 1 -- 172.21.15.17:0/4129461217 >> v1:172.21.15.67:6789/0 conn(0x7f46b0097320 legacy=0x7f46b00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353438885 0 0) 0x7f469c0034e0 con 0x7f46b009a960 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 msgr2=0x7f46b01339d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f46b01339d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 >> v1:172.21.15.67:6789/0 conn(0x7f46b0097320 legacy=0x7f46b00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b009cb90 con 0x7f46b009a960 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f46a8001aa0 con 0x7f46b009a960 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: dump: 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.597 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 >> v1:172.21.15.17:6789/0 conn(0x7f46b009a960 legacy=0x7f46b0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f469c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): picked mon.c con 0x7f46b009a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 0x7f469c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): picked mon.a con 0x7f469c0041b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c004e70 0x7f469c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): picked mon.b con 0x7f469c004e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.584+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b00992f0 con 0x7f469c0041b0 2024-09-15T02:03:01.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00bf10 con 0x7f469c004e70 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00c050 con 0x7f46b009a590 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b5dab640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 msgr2=0x7f469c003aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b4da9640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b5dab640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f469c003aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 0x7f469c004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): get_auth_request con 0x7f469c0041b0 auth_method 0 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b65ac640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c004e70 0x7f469c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b65ac640 10 monclient(hunting): get_auth_request con 0x7f469c004e70 auth_method 0 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b65ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b65ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b65ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient(hunting): handle_auth_done global_id 105141 payload 293 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 msgr2=0x7f469c003aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46b009a590 0x7f469c003aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c004e70 msgr2=0x7f469c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c004e70 0x7f469c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 monclient: found mon.a 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 1 -- 172.21.15.17:0/4129461217 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0034e0 con 0x7f469c0041b0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b55aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b800f640 5 monclient: authenticate success, global_id 105141 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.585+0000 7f46b800f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b55aa640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 msgr2=0x7f469c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b55aa640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 0x7f469c004780 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f46a40099f0 tx=0x7f46a4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92a071187c6caae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a4003b80 con 0x7f469c0041b0 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: dump: 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a4003d20 con 0x7f469c0041b0 2024-09-15T02:03:01.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 1 -- 172.21.15.17:0/4129461217 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a402ab80 con 0x7f469c0041b0 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 4 set_mon_vals no callback set 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b4da9640 10 monclient: dump: 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.603 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 10 monclient: shutdown 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 -- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 msgr2=0x7f469c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 --2- 172.21.15.17:0/4129461217 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f469c0041b0 0x7f469c004780 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f46a40099f0 tx=0x7f46a4009060 comp rx=0 tx=0).stop 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b65ac640 1 -- 172.21.15.17:0/4129461217 reap_dead start 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 -- 172.21.15.17:0/4129461217 shutdown_connections 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 -- 172.21.15.17:0/4129461217 >> 172.21.15.17:0/4129461217 conn(0x7f46b00a1000 msgr2=0x7f46b00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 -- 172.21.15.17:0/4129461217 shutdown_connections 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.588+0000 7f46b800f640 1 -- 172.21.15.17:0/4129461217 wait complete. 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.589+0000 7f46b800f640 10 monclient: build_initial_monmap 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.589+0000 7f46b800f640 10 monclient: monmap: 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:01.605 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.589+0000 7f46b800f640 1 Processor -- start 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.589+0000 7f46b800f640 1 -- start start 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.589+0000 7f46b800f640 10 monclient: init 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f469c0041b0 0x7f46b0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-c con 0x7f469c0041b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b011c310 0x7f46b0139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-b con 0x7f46b011c310 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): picked mon.noname-a con 0x7f46b011d790 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b009ad50 con 0x7f46b011d790 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b006eed0 con 0x7f46b011c310 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b5dab640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f469c0041b0 msgr2=0x7f46b0139f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b013ab30 con 0x7f469c0041b0 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b5dab640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f469c0041b0 0x7f46b0139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:01.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b800f640 10 monclient(hunting): authenticate will time out at 1441.586240s 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:33726/0 (socket says 172.21.15.17:33726) 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 learned_addr learned my addr 172.21.15.17:0/110639397 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): get_auth_request con 0x7f46b011d790 auth_method 0 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b55aa640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b011c310 0x7f46b0139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b55aa640 10 monclient(hunting): get_auth_request con 0x7f46b011c310 auth_method 0 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b55aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b55aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b55aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient(hunting): handle_auth_done global_id 105144 payload 824 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f469c0041b0 msgr2=0x7f46b0139f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f469c0041b0 0x7f46b0139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b011c310 msgr2=0x7f46b0139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b011c310 0x7f46b0139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient: _finish_hunting 0 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 monclient: found mon.noname-a 2024-09-15T02:03:01.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.590+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b011e6b0 con 0x7f46b011d790 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 msgr2=0x7f46b013d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b65ac640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f46a8001c80 tx=0x7f46a8001440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8010730 con 0x7f46b011d790 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: dump: 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a8010890 con 0x7f46b011d790 2024-09-15T02:03:01.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b800f640 5 monclient: authenticate success, global_id 105144 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b800f640 10 monclient: _renew_subs 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b800f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b011e990 con 0x7f46b011d790 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8010c50 con 0x7f46b011d790 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.592+0000 7f46a3fff640 4 set_mon_vals no callback set 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.593+0000 7f46a1ffb640 10 monclient: dump: 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.612 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.593+0000 7f46b800f640 10 monclient: _renew_subs 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.593+0000 7f46b800f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:01.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.593+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b0139cf0 con 0x7f46b011d790 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b800f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b800f640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 0x7f46b013ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b0063c20 con 0x7f46b013d9f0 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f46a80133e0 con 0x7f46b011d790 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 0x7f46b013ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46a1ffb640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f469403cb00 0x7f469403efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient: get_auth_request con 0x7f46b013d9f0 auth_method 0 2024-09-15T02:03:01.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f46a804c010 con 0x7f46b011d790 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b55aa640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f469403cb00 0x7f469403efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b55aa640 10 monclient: get_auth_request con 0x7f469403cb00 auth_method 0 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 10 monclient(hunting): handle_auth_done global_id 144232 payload 824 2024-09-15T02:03:01.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.594+0000 7f46b5dab640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46b011ed00 con 0x7f46b013d9f0 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46b5dab640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 msgr2=0x7f46b013ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46b5dab640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 0x7f46b013ddc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0014b0 tx=0x7f46ac007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46b55aa640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f469403cb00 0x7f469403efb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f46a4032000 tx=0x7f46a4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0075f0 con 0x7f46b013d9f0 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 10 monclient: dump: 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.619 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f46ac005780 con 0x7f46b013d9f0 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 msgr2=0x7f46b013ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 0x7f46b013ddc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0014b0 tx=0x7f46ac007d10 comp rx=0 tx=0).stop 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 msgr2=0x7f46b013ddc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.597+0000 7f46a1ffb640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b013d9f0 0x7f46b013ddc0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0014b0 tx=0x7f46ac007d10 comp rx=0 tx=0).stop 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b800f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b800f640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 0x7f46b0117890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b800f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b011e840 con 0x7f46b01174c0 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 0x7f46b0117890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient: get_auth_request con 0x7f46b01174c0 auth_method 0 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:01.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 10 monclient(hunting): handle_auth_done global_id 144235 payload 824 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.603+0000 7f46b5dab640 1 -- 172.21.15.17:0/110639397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46b013e300 con 0x7f46b01174c0 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46b5dab640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 msgr2=0x7f46b0117890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46b5dab640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 0x7f46b0117890 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007fa0 tx=0x7f46ac016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac021630 con 0x7f46b01174c0 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 10 monclient: dump: 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:01.622 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4385 (secure 0 0 0) 0x7f46ac021790 con 0x7f46b01174c0 2024-09-15T02:03:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 msgr2=0x7f46b0117890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 0x7f46b0117890 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007fa0 tx=0x7f46ac016000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 msgr2=0x7f46b0117890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.606+0000 7f46a1ffb640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b01174c0 0x7f46b0117890 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:01.624 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 126, 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.625 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:01.626 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.627 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:01.628 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:01.629 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.631 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:01.635 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:01.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:01.637 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:01.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f469403cb00 msgr2=0x7f469403efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f469403cb00 0x7f469403efb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f46a4032000 tx=0x7f46a4030000 comp rx=0 tx=0).stop 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 10 monclient: shutdown 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 msgr2=0x7f46b013d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f46a8001c80 tx=0x7f46a8001440 comp rx=0 tx=0).stop 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b65ac640 1 -- 172.21.15.17:0/110639397 reap_dead start 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 shutdown_connections 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 --2- 172.21.15.17:0/110639397 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46b011d790 0x7f46b013d4b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 >> 172.21.15.17:0/110639397 conn(0x7f46b00a1000 msgr2=0x7f46b006e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 shutdown_connections 2024-09-15T02:03:01.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:01.607+0000 7f46b800f640 1 -- 172.21.15.17:0/110639397 wait complete. 2024-09-15T02:03:01.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:03:01.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:03:21.640 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:03:21.640 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T02:03:21.641 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:03:21.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:03:21.644 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:03:21.644 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:03:21.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.750+0000 7f8d3a834640 10 monclient: get_monmap_and_config 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.750+0000 7f8d3a834640 10 monclient: build_initial_monmap 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.750+0000 7f8d3a834640 10 monclient: monmap: 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:21.751318+0000 2024-09-15T02:03:21.752 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:21.751318+0000 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:21.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 Processor -- start 2024-09-15T02:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 -- start start 2024-09-15T02:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient: init 2024-09-15T02:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:21.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d340a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d34132040 0x7f8d340987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-a con 0x7f8d34132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-b con 0x7f8d34136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340a01e0 con 0x7f8d340a58b0 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d340a1580 con 0x7f8d34132040 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340a5c80 con 0x7f8d34136dd0 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d3a834640 10 monclient(hunting): authenticate will time out at 1461.747875s 2024-09-15T02:03:21.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.751+0000 7f8d337fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d34132040 0x7f8d340987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d34132040 0x7f8d340987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41956/0 (socket says 172.21.15.17:41956) 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 -- 172.21.15.17:0/254703935 learned_addr learned my addr 172.21.15.17:0/254703935 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): get_auth_request con 0x7f8d34132040 auth_method 0 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 10 monclient(hunting): get_auth_request con 0x7f8d34136dd0 auth_method 0 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:21.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 1 -- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d341391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 1 -- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d341391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d38dd1640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient(hunting): handle_auth_done global_id 105147 payload 274 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 -- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d341391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 -- 172.21.15.17:0/254703935 >> v1:172.21.15.67:6789/0 conn(0x7f8d340a58b0 legacy=0x7f8d340980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient: _finish_hunting 0 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 monclient: found mon.noname-a 2024-09-15T02:03:21.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:21.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.752+0000 7f8d337fe640 1 -- 172.21.15.17:0/254703935 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34132440 con 0x7f8d34132040 2024-09-15T02:03:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d337fe640 1 -- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d34132040 msgr2=0x7f8d340987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d337fe640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d34132040 0x7f8d340987b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009910 tx=0x7f8d280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30d7ff08a62c80e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 1 -- 172.21.15.17:0/254703935 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2802aea0 con 0x7f8d34132040 2024-09-15T02:03:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: dump: 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:21.766 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: _finish_auth 0 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 1 -- 172.21.15.17:0/254703935 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28032440 con 0x7f8d34132040 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d3a834640 5 monclient: authenticate success, global_id 105147 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d3a834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 1 -- 172.21.15.17:0/254703935 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d280327e0 con 0x7f8d34132040 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d3a834640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d3a834640 4 set_mon_vals no callback set 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.755+0000 7f8d32ffd640 10 monclient: dump: 2024-09-15T02:03:21.767 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:21.768 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 10 monclient: shutdown 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 -- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d34132040 msgr2=0x7f8d340987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d34132040 0x7f8d340987b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009910 tx=0x7f8d280091d0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 -- 172.21.15.17:0/254703935 shutdown_connections 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d341391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 --2- 172.21.15.17:0/254703935 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d34132040 0x7f8d340987b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 -- 172.21.15.17:0/254703935 >> 172.21.15.17:0/254703935 conn(0x7f8d34070ed0 msgr2=0x7f8d340a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 -- 172.21.15.17:0/254703935 shutdown_connections 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 1 -- 172.21.15.17:0/254703935 wait complete. 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 10 monclient: build_initial_monmap 2024-09-15T02:03:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.756+0000 7f8d3a834640 10 monclient: monmap: 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 Processor -- start 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 -- start start 2024-09-15T02:03:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient: init 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-a con 0x7f8d340a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d34132040 0x7f8d34141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-b con 0x7f8d34132040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): picked mon.noname-c con 0x7f8d34136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d34134ed0 con 0x7f8d340a58b0 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d341360a0 con 0x7f8d34132040 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:21.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d34134aa0 con 0x7f8d34136dd0 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.757+0000 7f8d3a834640 10 monclient(hunting): authenticate will time out at 1461.753955s 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:41966/0 (socket says 172.21.15.17:41966) 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36900/0 (socket says 172.21.15.17:36900) 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 -- 172.21.15.17:0/2456984431 learned_addr learned my addr 172.21.15.17:0/2456984431 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): get_auth_request con 0x7f8d340a58b0 auth_method 0 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:21.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 10 monclient(hunting): get_auth_request con 0x7f8d34136dd0 auth_method 0 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d34142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d34142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d34132040 0x7f8d34141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:21.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): get_auth_request con 0x7f8d34132040 auth_method 0 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d38dd1640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient(hunting): handle_auth_done global_id 105150 payload 824 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 msgr2=0x7f8d34142300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d34132040 msgr2=0x7f8d34141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d34132040 0x7f8d34141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 monclient: found mon.noname-a 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.758+0000 7f8d33fff640 1 -- 172.21.15.17:0/2456984431 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28009450 con 0x7f8d340a58b0 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d33fff640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 msgr2=0x7f8d34141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d33fff640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b4f0 tx=0x7f8d240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400f720 con 0x7f8d340a58b0 2024-09-15T02:03:21.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 10 monclient: dump: 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:21.776 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 10 monclient: _finish_auth 0 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d3a834640 5 monclient: authenticate success, global_id 105150 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2400f8a0 con 0x7f8d340a58b0 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d3a834640 10 monclient: _renew_subs 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d3a834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.760+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d34135100 con 0x7f8d340a58b0 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400fc60 con 0x7f8d340a58b0 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:21.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 10 monclient: dump: 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d327fc640 4 set_mon_vals no callback set 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 10 monclient: _renew_subs 2024-09-15T02:03:21.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d340a33a0 con 0x7f8d340a58b0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d24011610 con 0x7f8d340a58b0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.761+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d340724a0 con 0x7f8d340a58b0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.762+0000 7f8d1bfff640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 0x7f8d2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.762+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8d2404c1a0 con 0x7f8d340a58b0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.762+0000 7f8d337fe640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 0x7f8d2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:21.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.762+0000 7f8d337fe640 10 monclient: get_auth_request con 0x7f8d20038740 auth_method 0 2024-09-15T02:03:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.765+0000 7f8d337fe640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 0x7f8d2003abf0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8d340a60e0 tx=0x7f8d28032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.766+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d240118c0 con 0x7f8d340a58b0 2024-09-15T02:03:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.766+0000 7f8d1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:21.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.766+0000 7f8d1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:21.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.997+0000 7f8d3a834640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.997+0000 7f8d3a834640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.997+0000 7f8d3a834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.997+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d341394d0 con 0x7f8d340a58b0 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.998+0000 7f8d1bfff640 1 -- 172.21.15.17:0/2456984431 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f8d24017720 con 0x7f8d340a58b0 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.998+0000 7f8d1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:21.998+0000 7f8d1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:22.000 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:03:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 msgr2=0x7f8d2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 0x7f8d2003abf0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8d340a60e0 tx=0x7f8d28032000 comp rx=0 tx=0).stop 2024-09-15T02:03:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 10 monclient: shutdown 2024-09-15T02:03:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:22.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 msgr2=0x7f8d34141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b4f0 tx=0x7f8d240090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 shutdown_connections 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d20038740 0x7f8d2003abf0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d34136dd0 0x7f8d34142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d34132040 0x7f8d34141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 --2- 172.21.15.17:0/2456984431 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d340a58b0 0x7f8d34141880 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 >> 172.21.15.17:0/2456984431 conn(0x7f8d34070ed0 msgr2=0x7f8d340a3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 shutdown_connections 2024-09-15T02:03:22.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:22.000+0000 7f8d3a834640 1 -- 172.21.15.17:0/2456984431 wait complete. 2024-09-15T02:03:25.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:03:25.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.120+0000 7f2a27d5d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:25.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.120+0000 7f2a27d5d640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.120+0000 7f2a27d5d640 10 monclient: monmap: 2024-09-15T02:03:25.122 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.122 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:25.121229+0000 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:25.121229+0000 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:25.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.120+0000 7f2a27d5d640 1 Processor -- start 2024-09-15T02:03:25.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 -- start start 2024-09-15T02:03:25.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient: init 2024-09-15T02:03:25.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2009a5c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2a2009d4b0 0x7f2a2006e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2009d4b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a2006ee60 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2009a990 con 0x7f2a2006ee60 2024-09-15T02:03:25.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2009b0c0 con 0x7f2a2009d4b0 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2009c7f0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a27d5d640 10 monclient(hunting): authenticate will time out at 1465.117836s 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a25af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.121+0000 7f2a252f8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2a2009d4b0 0x7f2a2006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48212/0 (socket says 172.21.15.17:48212) 2024-09-15T02:03:25.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 -- 172.21.15.17:0/3108212633 learned_addr learned my addr 172.21.15.17:0/3108212633 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 10 monclient(hunting): get_auth_request con 0x7f2a2009a5c0 auth_method 0 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a262fa640 1 --1- 172.21.15.17:0/3108212633 >> v1:172.21.15.17:6789/0 conn(0x7f2a2006ee60 0x7f2a20096c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35516/0 (socket says 172.21.15.17:35516) 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): get_auth_request con 0x7f2a2009d4b0 auth_method 0 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 -- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 msgr2=0x7f2a2012fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 -- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 msgr2=0x7f2a2012fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a25af9640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient(hunting): handle_auth_done global_id 105153 payload 274 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 1 -- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 msgr2=0x7f2a2012fc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 1 -- 172.21.15.17:0/3108212633 >> v1:172.21.15.17:6789/0 conn(0x7f2a2006ee60 legacy=0x7f2a20096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 1 monclient: found mon.noname-a 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.122+0000 7f2a252f8640 1 -- 172.21.15.17:0/3108212633 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2009cdb0 con 0x7f2a2009d4b0 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a252f8640 1 -- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009d4b0 msgr2=0x7f2a2006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a252f8640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009d4b0 0x7f2a2006e920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009850 tx=0x7f2a100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e8eb8523da22c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 1 -- 172.21.15.17:0/3108212633 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1002adf0 con 0x7f2a2009d4b0 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:25.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: dump: 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 1 -- 172.21.15.17:0/3108212633 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10032440 con 0x7f2a2009d4b0 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 5 monclient: authenticate success, global_id 105153 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 1 -- 172.21.15.17:0/3108212633 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100327e0 con 0x7f2a2009d4b0 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 4 set_mon_vals no callback set 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a24af7640 10 monclient: dump: 2024-09-15T02:03:25.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 10 monclient: shutdown 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:25.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 1 -- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009d4b0 msgr2=0x7f2a2006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.125+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009d4b0 0x7f2a2006e920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009850 tx=0x7f2a100090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 -- 172.21.15.17:0/3108212633 shutdown_connections 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009d4b0 0x7f2a2006e920 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/3108212633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009a5c0 0x7f2a2012fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 -- 172.21.15.17:0/3108212633 >> 172.21.15.17:0/3108212633 conn(0x7f2a200a1000 msgr2=0x7f2a200a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 -- 172.21.15.17:0/3108212633 shutdown_connections 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 1 -- 172.21.15.17:0/3108212633 wait complete. 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.126+0000 7f2a27d5d640 10 monclient: monmap: 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:25.142 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 Processor -- start 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 -- start start 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient: init 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 0x7f2a2012ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2006ee60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2009a5c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:25.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2009d4b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2011cd90 con 0x7f2a2009a5c0 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2011e480 con 0x7f2a2006ee60 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2006e070 con 0x7f2a2009d4b0 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a27d5d640 10 monclient(hunting): authenticate will time out at 1465.123641s 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35700/0 (socket says 172.21.15.17:35700) 2024-09-15T02:03:25.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 10 monclient(hunting): get_auth_request con 0x7f2a2009d4b0 auth_method 0 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 1 -- 172.21.15.17:0/1496751060 learned_addr learned my addr 172.21.15.17:0/1496751060 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): get_auth_request con 0x7f2a2009a5c0 auth_method 0 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 msgr2=0x7f2a200988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 msgr2=0x7f2a200988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a252f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a262fa640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a25af9640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 0x7f2a2012ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a25af9640 10 monclient(hunting): get_auth_request con 0x7f2a2006ee60 auth_method 0 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a25af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a25af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.127+0000 7f2a25af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 10 monclient(hunting): handle_auth_done global_id 105156 payload 824 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 msgr2=0x7f2a200988f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 msgr2=0x7f2a2012ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 0x7f2a2012ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 monclient: found mon.noname-a 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a252f8640 1 -- 172.21.15.17:0/1496751060 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009390 con 0x7f2a2009a5c0 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.128+0000 7f2a25af9640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 0x7f2a2012ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a252f8640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 msgr2=0x7f2a200983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a252f8640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009850 tx=0x7f2a10032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10031b10 con 0x7f2a2009a5c0 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: dump: 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.147 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10031c70 con 0x7f2a2009a5c0 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1003d440 con 0x7f2a2009a5c0 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 5 monclient: authenticate success, global_id 105156 2024-09-15T02:03:25.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 10 monclient: _renew_subs 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2006f5d0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a17fff640 4 set_mon_vals no callback set 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a15ffb640 10 monclient: dump: 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.149 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.150 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.150 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.150 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.150 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 10 monclient: _renew_subs 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.130+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2012f3a0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2a10039040 con 0x7f2a2009a5c0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a27d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a27d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a27d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a200724a0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a15ffb640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 0x7f2a0403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.131+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f2a1006c170 con 0x7f2a2009a5c0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.132+0000 7f2a25af9640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 0x7f2a0403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.132+0000 7f2a25af9640 10 monclient: get_auth_request con 0x7f2a04038740 auth_method 0 2024-09-15T02:03:25.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.134+0000 7f2a25af9640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 0x7f2a0403abf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006fd0 tx=0x7f2a1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.136+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2a1002f2a0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.136+0000 7f2a15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.136+0000 7f2a15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.361+0000 7f2a27d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.361+0000 7f2a27d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.361+0000 7f2a27d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.361+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2a2011dfa0 con 0x7f2a2009a5c0 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.362+0000 7f2a15ffb640 1 -- 172.21.15.17:0/1496751060 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f2a10037080 con 0x7f2a2009a5c0 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.362+0000 7f2a15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.362+0000 7f2a15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:25.364 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 msgr2=0x7f2a0403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 0x7f2a0403abf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006fd0 tx=0x7f2a1c008040 comp rx=0 tx=0).stop 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 10 monclient: shutdown 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 msgr2=0x7f2a200983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009850 tx=0x7f2a10032000 comp rx=0 tx=0).stop 2024-09-15T02:03:25.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 shutdown_connections 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2a04038740 0x7f2a0403abf0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2a2009a5c0 0x7f2a200983b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2a2009d4b0 0x7f2a200988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 --2- 172.21.15.17:0/1496751060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2006ee60 0x7f2a2012ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 >> 172.21.15.17:0/1496751060 conn(0x7f2a200a1000 msgr2=0x7f2a2009fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 shutdown_connections 2024-09-15T02:03:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.364+0000 7f2a27d5d640 1 -- 172.21.15.17:0/1496751060 wait complete. 2024-09-15T02:03:25.376 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:03:25.376 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:03:25.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.472+0000 7fb62235d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.472+0000 7fb62235d640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.472+0000 7fb62235d640 10 monclient: monmap: 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:25.473287+0000 2024-09-15T02:03:25.474 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:25.473287+0000 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:25.475 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 Processor -- start 2024-09-15T02:03:25.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 -- start start 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient: init 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb61c09c0d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb61c09e970 0x7fb61c09c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c09e970 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb61c06eb30 0x7fb61c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c06eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0699c0 con 0x7fb61c09c0d0 2024-09-15T02:03:25.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c06a5e0 con 0x7fb61c06eb30 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c06a290 con 0x7fb61c09e970 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb62235d640 10 monclient(hunting): authenticate will time out at 1465.469792s 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb6208fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb61c06eb30 0x7fb61c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb61bfff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb61c09c0d0 0x7fb61c09a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35526/0 (socket says 172.21.15.17:35526) 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb61bfff640 1 -- 172.21.15.17:0/4018903203 learned_addr learned my addr 172.21.15.17:0/4018903203 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb6208fa640 10 monclient(hunting): get_auth_request con 0x7fb61c06eb30 auth_method 0 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb6208fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb6208fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.473+0000 7fb6208fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/4018903203 >> v2:172.21.15.67:3300/0 conn(0x7fb61c09e970 0x7fb61c09c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient(hunting): handle_auth_done global_id 105159 payload 274 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 1 -- 172.21.15.17:0/4018903203 >> v2:172.21.15.67:3300/0 conn(0x7fb61c09e970 msgr2=0x7fb61c09c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 1 --2- 172.21.15.17:0/4018903203 >> v2:172.21.15.67:3300/0 conn(0x7fb61c09e970 0x7fb61c09c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 1 -- 172.21.15.17:0/4018903203 >> v1:172.21.15.17:6789/0 conn(0x7fb61c09c0d0 legacy=0x7fb61c09a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 1 monclient: found mon.noname-a 2024-09-15T02:03:25.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.474+0000 7fb6208fa640 1 -- 172.21.15.17:0/4018903203 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c09ed70 con 0x7fb61c06eb30 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb6208fa640 1 -- 172.21.15.17:0/4018903203 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 msgr2=0x7fb61c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb6208fa640 1 --2- 172.21.15.17:0/4018903203 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c06ef20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb61c09f5e0 tx=0x7fb614009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce22fec37e84f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 1 -- 172.21.15.17:0/4018903203 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007840 con 0x7fb61c06eb30 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: dump: 2024-09-15T02:03:25.488 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 1 -- 172.21.15.17:0/4018903203 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61402d050 con 0x7fb61c06eb30 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 5 monclient: authenticate success, global_id 105159 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 4 set_mon_vals no callback set 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 1 -- 172.21.15.17:0/4018903203 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007bd0 con 0x7fb61c06eb30 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb61affd640 10 monclient: dump: 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.490 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 10 monclient: shutdown 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 -- 172.21.15.17:0/4018903203 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 msgr2=0x7fb61c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 --2- 172.21.15.17:0/4018903203 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c06ef20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb61c09f5e0 tx=0x7fb614009780 comp rx=0 tx=0).stop 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 -- 172.21.15.17:0/4018903203 shutdown_connections 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 --2- 172.21.15.17:0/4018903203 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c06ef20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.491 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 --2- 172.21.15.17:0/4018903203 >> v2:172.21.15.67:3300/0 conn(0x7fb61c09e970 0x7fb61c09c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 -- 172.21.15.17:0/4018903203 >> 172.21.15.17:0/4018903203 conn(0x7fb61c0712f0 msgr2=0x7fb61c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 -- 172.21.15.17:0/4018903203 shutdown_connections 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.477+0000 7fb62235d640 1 -- 172.21.15.17:0/4018903203 wait complete. 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.478+0000 7fb62235d640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.478+0000 7fb62235d640 10 monclient: monmap: 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:25.492 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.478+0000 7fb62235d640 1 Processor -- start 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.478+0000 7fb62235d640 1 -- start start 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.478+0000 7fb62235d640 10 monclient: init 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c06eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 0x7fb61c11d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c09c0d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb61c09e970 0x7fb61c11df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.493 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c09e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c134b50 con 0x7fb61c06eb30 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c120da0 con 0x7fb61c09e970 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c09cd80 con 0x7fb61c09c0d0 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb62235d640 10 monclient(hunting): authenticate will time out at 1465.475258s 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 0x7fb61c11d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c133c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35716/0 (socket says 172.21.15.17:35716) 2024-09-15T02:03:25.494 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 -- 172.21.15.17:0/1779446247 learned_addr learned my addr 172.21.15.17:0/1779446247 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): get_auth_request con 0x7fb61c06eb30 auth_method 0 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 10 monclient(hunting): get_auth_request con 0x7fb61c09c0d0 auth_method 0 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 msgr2=0x7fb61c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:03:25.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 msgr2=0x7fb61c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb61c09e970 0x7fb61c11df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 0x7fb61c11d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 10 monclient(hunting): get_auth_request con 0x7fb61c09e970 auth_method 0 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 0x7fb61c11d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient(hunting): handle_auth_done global_id 105162 payload 824 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 msgr2=0x7fb61c11d9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb61c09c0d0 0x7fb61c11d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb61c09e970 msgr2=0x7fb61c11df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb61c09e970 0x7fb61c11df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 monclient: found mon.noname-a 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb61bfff640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614009a20 con 0x7fb61c06eb30 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.479+0000 7fb6208fa640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb61c09e970 0x7fb61c11df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.481+0000 7fb61bfff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 msgr2=0x7fb61c133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.481+0000 7fb61bfff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c133c80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00c5e0 tx=0x7fb60c004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c005cb0 con 0x7fb61c06eb30 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: dump: 2024-09-15T02:03:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60c005e30 con 0x7fb61c06eb30 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 5 monclient: authenticate success, global_id 105162 2024-09-15T02:03:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 monclient: _renew_subs 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c11f3a0 con 0x7fb61c06eb30 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c007340 con 0x7fb61c06eb30 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 10 monclient: dump: 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.499 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 monclient: _renew_subs 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c136450 con 0x7fb61c06eb30 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb61a7fc640 4 set_mon_vals no callback set 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 0x7fb61c11e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c121500 con 0x7fb61c1341c0 2024-09-15T02:03:25.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.482+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb60c01b400 con 0x7fb61c06eb30 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 0x7fb61c11e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient: get_auth_request con 0x7fb61c1341c0 auth_method 0 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb603fff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb608038740 0x7fb60803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb60c04c040 con 0x7fb61c06eb30 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_done global_id 105165 payload 824 2024-09-15T02:03:25.501 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb61c120fe0 con 0x7fb61c1341c0 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb6208fa640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb608038740 0x7fb60803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.483+0000 7fb6208fa640 10 monclient: get_auth_request con 0x7fb608038740 auth_method 0 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 msgr2=0x7fb61c11e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 0x7fb61c11e460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb61000f490 tx=0x7fb610008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610014980 con 0x7fb61c1341c0 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb603fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb603fff640 10 monclient: dump: 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.502 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.485+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb61000c030 con 0x7fb61c1341c0 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 msgr2=0x7fb61c11e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 0x7fb61c11e460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb61000f490 tx=0x7fb610008040 comp rx=0 tx=0).stop 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 msgr2=0x7fb61c11e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb603fff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c1341c0 0x7fb61c11e460 secure :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb61000f490 tx=0x7fb610008040 comp rx=0 tx=0).stop 2024-09-15T02:03:25.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.486+0000 7fb6208fa640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb608038740 0x7fb60803abf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0a3e50 tx=0x7fb614037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb62235d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:25.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb62235d640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 0x7fb61c0a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb62235d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c063b90 con 0x7fb61c11e9a0 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 0x7fb61c0a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb61b7fe640 10 monclient: get_auth_request con 0x7fb61c11e9a0 auth_method 0 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb61b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.491+0000 7fb61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.492+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.492+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.492+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.492+0000 7fb61b7fe640 10 monclient(hunting): handle_auth_done global_id 105168 payload 824 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.492+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb61c0724a0 con 0x7fb61c11e9a0 2024-09-15T02:03:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb61b7fe640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 msgr2=0x7fb61c0a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb61b7fe640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 0x7fb61c0a1990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb610019000 tx=0x7fb61001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610014e80 con 0x7fb61c11e9a0 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 10 monclient: dump: 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.507 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fb610013800 con 0x7fb61c11e9a0 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 msgr2=0x7fb61c0a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 0x7fb61c0a1990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb610019000 tx=0x7fb61001c000 comp rx=0 tx=0).stop 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 msgr2=0x7fb61c0a1990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.494+0000 7fb603fff640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c11e9a0 0x7fb61c0a1990 secure :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb610019000 tx=0x7fb61001c000 comp rx=0 tx=0).stop 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:25.508 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:25.509 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:25.510 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:25.511 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:25.512 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:25.513 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:25.514 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:25.517 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:25.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:25.519 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:25.520 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:25.521 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb608038740 msgr2=0x7fb60803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb608038740 0x7fb60803abf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0a3e50 tx=0x7fb614037000 comp rx=0 tx=0).stop 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 10 monclient: shutdown 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 msgr2=0x7fb61c133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 --2- 172.21.15.17:0/1779446247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb61c06eb30 0x7fb61c133c80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00c5e0 tx=0x7fb60c004960 comp rx=0 tx=0).stop 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb6208fa640 1 -- 172.21.15.17:0/1779446247 reap_dead start 2024-09-15T02:03:25.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb6208fa640 1 -- 172.21.15.17:0/1779446247 reap_dead start 2024-09-15T02:03:25.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 shutdown_connections 2024-09-15T02:03:25.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 >> 172.21.15.17:0/1779446247 conn(0x7fb61c0712f0 msgr2=0x7fb61c134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:25.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 shutdown_connections 2024-09-15T02:03:25.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.495+0000 7fb62235d640 1 -- 172.21.15.17:0/1779446247 wait complete. 2024-09-15T02:03:25.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:25.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:03:25.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 10 monclient: get_monmap_and_config 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 10 monclient: monmap: 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:25.619963+0000 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:25.619963+0000 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 1 Processor -- start 2024-09-15T02:03:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 1 -- start start 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.619+0000 7f71d044f640 10 monclient: init 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71c8131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71c80a0280 0x7f71c8131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-c con 0x7f71c80a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71c8132430 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c80a0040 con 0x7f71c8132430 2024-09-15T02:03:25.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c809bff0 con 0x7f71c8131b20 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c80a14a0 con 0x7f71c80a0280 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71d044f640 10 monclient(hunting): authenticate will time out at 1465.616483s 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71ce9ec640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f71c8132430 0x7f71c8138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35534/0 (socket says 172.21.15.17:35534) 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71ce9ec640 1 -- 172.21.15.17:0/415150835 learned_addr learned my addr 172.21.15.17:0/415150835 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 1 --2- 172.21.15.17:0/415150835 >> v2:172.21.15.67:3300/0 conn(0x7f71c80a0280 0x7f71c8131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): get_auth_request con 0x7f71c80a0280 auth_method 0 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.620+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.622+0000 7f71ce9ec640 1 -- 172.21.15.17:0/415150835 >> v1:172.21.15.17:6789/0 conn(0x7f71c8132430 legacy=0x7f71c8138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71ce1eb640 1 -- 172.21.15.17:0/415150835 >> v1:172.21.15.67:6789/0 conn(0x7f71c8131b20 legacy=0x7f71c809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_done global_id 144247 payload 274 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 1 -- 172.21.15.17:0/415150835 >> v1:172.21.15.67:6789/0 conn(0x7f71c8131b20 legacy=0x7f71c809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 1 -- 172.21.15.17:0/415150835 >> v1:172.21.15.17:6789/0 conn(0x7f71c8132430 legacy=0x7f71c8138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 1 monclient: found mon.noname-c 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:03:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.623+0000 7f71cd9ea640 1 -- 172.21.15.17:0/415150835 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c809f3b0 con 0x7f71c80a0280 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd9ea640 1 -- 172.21.15.17:0/415150835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 msgr2=0x7f71c8131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd9ea640 1 --2- 172.21.15.17:0/415150835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c8131ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009950 tx=0x7f71bc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd6bbfe5645ab854 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 1 -- 172.21.15.17:0/415150835 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc02f4b0 con 0x7f71c80a0280 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: dump: 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 1 -- 172.21.15.17:0/415150835 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc02f610 con 0x7f71c80a0280 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 5 monclient: authenticate success, global_id 144247 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 1 -- 172.21.15.17:0/415150835 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc02f9b0 con 0x7f71c80a0280 2024-09-15T02:03:25.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 4 set_mon_vals no callback set 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71cd1e9640 10 monclient: dump: 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 10 monclient: shutdown 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 1 -- 172.21.15.17:0/415150835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 msgr2=0x7f71c8131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 1 --2- 172.21.15.17:0/415150835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c8131ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009950 tx=0x7f71bc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 1 -- 172.21.15.17:0/415150835 shutdown_connections 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 1 --2- 172.21.15.17:0/415150835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c8131ef0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.626+0000 7f71d044f640 1 -- 172.21.15.17:0/415150835 >> 172.21.15.17:0/415150835 conn(0x7f71c8071270 msgr2=0x7f71c80a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 1 -- 172.21.15.17:0/415150835 shutdown_connections 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 1 -- 172.21.15.17:0/415150835 wait complete. 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 10 monclient: build_initial_monmap 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 10 monclient: monmap: 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:25.632 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:25.633 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 1 Processor -- start 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 1 -- start start 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.627+0000 7f71d044f640 10 monclient: init 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c809b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-b con 0x7f71c80a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 0x7f71c8132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-a con 0x7f71c8131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 0x7f71c8132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): picked mon.noname-c con 0x7f71c8132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c80a4a50 con 0x7f71c8131b20 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c81359e0 con 0x7f71c80a0280 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8138be0 con 0x7f71c8132430 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71d044f640 10 monclient(hunting): authenticate will time out at 1465.624242s 2024-09-15T02:03:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 0x7f71c8132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 0x7f71c8132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 0x7f71c8132970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35752/0 (socket says 172.21.15.17:35752) 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 -- 172.21.15.17:0/1897642943 learned_addr learned my addr 172.21.15.17:0/1897642943 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): get_auth_request con 0x7f71c8131b20 auth_method 0 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 10 monclient(hunting): get_auth_request con 0x7f71c8132430 auth_method 0 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 msgr2=0x7f71c8132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 msgr2=0x7f71c8132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 0x7f71c8132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce9ec640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 0x7f71c8132eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c809b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request con 0x7f71c80a0280 auth_method 0 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71ce1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient(hunting): handle_auth_done global_id 105174 payload 824 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 msgr2=0x7f71c8132eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8132430 0x7f71c8132eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 msgr2=0x7f71c809b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c80a0280 0x7f71c809b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient: _finish_hunting 0 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 monclient: found mon.noname-a 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.628+0000 7f71cd9ea640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc009390 con 0x7f71c8131b20 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.630+0000 7f71cd9ea640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 msgr2=0x7f71c8132970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.630+0000 7f71cd9ea640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 0x7f71c8132970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009360 tx=0x7f71bc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc032ab0 con 0x7f71c8131b20 2024-09-15T02:03:25.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: dump: 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: _finish_auth 0 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 5 monclient: authenticate success, global_id 105174 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc032c10 con 0x7f71c8131b20 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc03d440 con 0x7f71c8131b20 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 monclient: _renew_subs 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8135700 con 0x7f71c8131b20 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71cc9e8640 4 set_mon_vals no callback set 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:25.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 10 monclient: dump: 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 monclient: _renew_subs 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c81363c0 con 0x7f71c8131b20 2024-09-15T02:03:25.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8063a40 con 0x7f71c8134810 2024-09-15T02:03:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.631+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f71bc039040 con 0x7f71c8131b20 2024-09-15T02:03:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 10 monclient: get_auth_request con 0x7f71c8134810 auth_method 0 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ba7fc640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f71ac038740 0x7f71ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f71bc06c210 con 0x7f71c8131b20 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce9ec640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f71ac038740 0x7f71ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.632+0000 7f71ce9ec640 10 monclient: get_auth_request con 0x7f71ac038740 auth_method 0 2024-09-15T02:03:25.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.635+0000 7f71ce9ec640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f71ac038740 0x7f71ac03abf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f71c81360b0 tx=0x7f71c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:25.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 10 monclient: get_auth_request con 0x7f71c8134810 auth_method 0 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:25.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:25.833+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 10 monclient: get_auth_request con 0x7f71c8134810 auth_method 0 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:03:26.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:03:26.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:26.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:26.234+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:03:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient: get_auth_request con 0x7f71c8134810 auth_method 0 2024-09-15T02:03:27.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_done global_id 164099 payload 824 2024-09-15T02:03:27.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.035+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71c80724a0 con 0x7f71c8134810 2024-09-15T02:03:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71c4005e00 tx=0x7f71c4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c4015970 con 0x7f71c8134810 2024-09-15T02:03:27.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 10 monclient: dump: 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.040 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f71c4015ad0 con 0x7f71c8134810 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71c4005e00 tx=0x7f71c4009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 msgr2=0x7f71c802aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:27.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.038+0000 7f71ba7fc640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c8134810 0x7f71c802aa40 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71c4005e00 tx=0x7f71c4009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71d044f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71d044f640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 0x7f71c805e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71d044f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c81337f0 con 0x7f71c81333f0 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 0x7f71c805e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient: get_auth_request con 0x7f71c81333f0 auth_method 0 2024-09-15T02:03:27.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 10 monclient(hunting): handle_auth_done global_id 164102 payload 824 2024-09-15T02:03:27.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.045+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f71c8135b90 con 0x7f71c81333f0 2024-09-15T02:03:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ce1eb640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 msgr2=0x7f71c805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ce1eb640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 0x7f71c805e310 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f71c401a000 tx=0x7f71c401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c401f410 con 0x7f71c81333f0 2024-09-15T02:03:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 10 monclient: dump: 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f71c401f570 con 0x7f71c81333f0 2024-09-15T02:03:27.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 msgr2=0x7f71c805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 0x7f71c805e310 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f71c401a000 tx=0x7f71c401d000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 msgr2=0x7f71c805e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.048+0000 7f71ba7fc640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f71c81333f0 0x7f71c805e310 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f71c401a000 tx=0x7f71c401d000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:27.064 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f71ac038740 msgr2=0x7f71ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f71ac038740 0x7f71ac03abf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f71c81360b0 tx=0x7f71c0009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 10 monclient: shutdown 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 msgr2=0x7f71c8132970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 --2- 172.21.15.17:0/1897642943 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8131b20 0x7f71c8132970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009360 tx=0x7f71bc032000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71ce9ec640 1 -- 172.21.15.17:0/1897642943 reap_dead start 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71ce9ec640 1 -- 172.21.15.17:0/1897642943 reap_dead start 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 shutdown_connections 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 >> 172.21.15.17:0/1897642943 conn(0x7f71c8071270 msgr2=0x7f71c8139d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 shutdown_connections 2024-09-15T02:03:27.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.049+0000 7f71d044f640 1 -- 172.21.15.17:0/1897642943 wait complete. 2024-09-15T02:03:27.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:27.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:03:27.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.172+0000 7faaba457640 10 monclient: get_monmap_and_config 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.172+0000 7faaba457640 10 monclient: build_initial_monmap 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.172+0000 7faaba457640 10 monclient: monmap: 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:27.172990+0000 2024-09-15T02:03:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:27.172990+0000 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:27.175 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.172+0000 7faaba457640 1 Processor -- start 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 1 -- start start 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient: init 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faab4097320 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faab409a590 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faab41374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): picked mon.noname-b con 0x7faab41350f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40985d0 con 0x7faab4097320 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab409d200 con 0x7faab409a590 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40992f0 con 0x7faab41350f0 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faaba457640 10 monclient(hunting): authenticate will time out at 1467.169593s 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faab41374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab3fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7faab4097320 0x7faab40976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35540/0 (socket says 172.21.15.17:35540) 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 learned_addr learned my addr 172.21.15.17:0/1276001176 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 10 monclient(hunting): get_auth_request con 0x7faab41350f0 auth_method 0 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 msgr2=0x7faab41374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:03:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 msgr2=0x7faab41374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faab41374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:27.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.173+0000 7faab89f4640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faab41374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:27.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.175+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 >> v1:172.21.15.17:6789/0 conn(0x7faab4097320 legacy=0x7faab40976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275007602 0 0) 0x7faab40985d0 con 0x7faab4097320 2024-09-15T02:03:27.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): my global_id is 105177 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c0034e0 con 0x7faab4097320 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 1 -- 172.21.15.17:0/1276001176 >> v1:172.21.15.67:6789/0 conn(0x7faab409a590 legacy=0x7faab41339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3273543476 0 0) 0x7faa9c0034e0 con 0x7faab4097320 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 msgr2=0x7faab41374d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faab41374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 >> v1:172.21.15.67:6789/0 conn(0x7faab409a590 legacy=0x7faab41339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab409cb90 con 0x7faab4097320 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7faaa4001720 con 0x7faab4097320 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faaba457640 5 monclient: authenticate success, global_id 105177 2024-09-15T02:03:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faaba457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: dump: 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.188 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 >> v1:172.21.15.17:6789/0 conn(0x7faab4097320 legacy=0x7faab40976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faa9c003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): picked mon.c con 0x7faab41350f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa9c004180 0x7faa9c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): picked mon.a con 0x7faa9c004180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa9c004ec0 0x7faa9c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): picked mon.b con 0x7faa9c004ec0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab40992f0 con 0x7faa9c004180 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c00e120 con 0x7faa9c004ec0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c00e260 con 0x7faab41350f0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faaba457640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faa9c003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): get_auth_request con 0x7faab41350f0 auth_method 0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa9c004180 0x7faa9c004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): get_auth_request con 0x7faa9c004180 auth_method 0 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab89f4640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa9c004ec0 0x7faa9c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab89f4640 10 monclient(hunting): get_auth_request con 0x7faa9c004ec0 auth_method 0 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab89f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab89f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab89f4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:27.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient(hunting): handle_auth_done global_id 105177 payload 293 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa9c004ec0 msgr2=0x7faa9c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa9c004ec0 0x7faa9c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa9c004180 msgr2=0x7faa9c004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa9c004180 0x7faa9c004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 monclient: found mon.c 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c0034e0 con 0x7faab41350f0 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.176+0000 7faab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.179+0000 7faab3fff640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 msgr2=0x7faa9c003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.179+0000 7faab3fff640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faa9c003b00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faaa4001490 tx=0x7faaa4007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c898e8733b29687e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.179+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4005890 con 0x7faab41350f0 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.179+0000 7faab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: dump: 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.193 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa4005a10 con 0x7faab41350f0 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 4 set_mon_vals no callback set 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 1 -- 172.21.15.17:0/1276001176 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4005dd0 con 0x7faab41350f0 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab2ffd640 10 monclient: dump: 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 10 monclient: shutdown 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 -- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 msgr2=0x7faa9c003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 --2- 172.21.15.17:0/1276001176 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab41350f0 0x7faa9c003b00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faaa4001490 tx=0x7faaa4007a50 comp rx=0 tx=0).stop 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faab89f4640 1 -- 172.21.15.17:0/1276001176 reap_dead start 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 -- 172.21.15.17:0/1276001176 shutdown_connections 2024-09-15T02:03:27.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 -- 172.21.15.17:0/1276001176 >> 172.21.15.17:0/1276001176 conn(0x7faab40a1000 msgr2=0x7faab40a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 -- 172.21.15.17:0/1276001176 shutdown_connections 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.180+0000 7faaba457640 1 -- 172.21.15.17:0/1276001176 wait complete. 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.181+0000 7faaba457640 10 monclient: build_initial_monmap 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.181+0000 7faaba457640 10 monclient: monmap: 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:27.196 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.181+0000 7faaba457640 1 Processor -- start 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.181+0000 7faaba457640 1 -- start start 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.181+0000 7faaba457640 10 monclient: init 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 0x7faab413dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): picked mon.noname-a con 0x7faab41350f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 0x7faab406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): picked mon.noname-c con 0x7faab412ed60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab413e1e0 0x7faab406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): picked mon.noname-b con 0x7faab413e1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:27.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4135990 con 0x7faab41350f0 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab409ad50 con 0x7faab413e1e0 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40a0040 con 0x7faab412ed60 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faaba457640 10 monclient(hunting): authenticate will time out at 1467.178376s 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 0x7faab413dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 0x7faab406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 0x7faab413dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35772/0 (socket says 172.21.15.17:35772) 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 0x7faab406d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48286/0 (socket says 172.21.15.17:48286) 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 -- 172.21.15.17:0/2051981482 learned_addr learned my addr 172.21.15.17:0/2051981482 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:27.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): get_auth_request con 0x7faab41350f0 auth_method 0 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): get_auth_request con 0x7faab412ed60 auth_method 0 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab413e1e0 0x7faab406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): get_auth_request con 0x7faab413e1e0 auth_method 0 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab89f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient(hunting): handle_auth_done global_id 105180 payload 824 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 msgr2=0x7faab406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 0x7faab406d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab413e1e0 msgr2=0x7faab406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab413e1e0 0x7faab406dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 monclient: found mon.noname-a 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab3fff640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4117680 con 0x7faab41350f0 2024-09-15T02:03:27.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.182+0000 7faab37fe640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faab412ed60 0x7faab406d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faab3fff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 msgr2=0x7faab413dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faab3fff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 0x7faab413dca0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c5f0 tx=0x7faaa4034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4001240 con 0x7faab41350f0 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: dump: 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 5 monclient: authenticate success, global_id 105180 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa4007510 con 0x7faab41350f0 2024-09-15T02:03:27.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4039430 con 0x7faab41350f0 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faa9bfff640 10 monclient: dump: 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.207 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 10 monclient: _renew_subs 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab41189b0 con 0x7faab41350f0 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faab27fc640 4 set_mon_vals no callback set 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 10 monclient: _renew_subs 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.185+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4117c00 con 0x7faab41350f0 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faaa4039820 con 0x7faab41350f0 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faaba457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faaba457640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 0x7faab406e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4063c20 con 0x7faab406e230 2024-09-15T02:03:27.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faa9bfff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faaa0038740 0x7faaa003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faaa406c930 con 0x7faab41350f0 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 0x7faab406e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient: get_auth_request con 0x7faab406e230 auth_method 0 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab89f4640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faaa0038740 0x7faaa003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab89f4640 10 monclient: get_auth_request con 0x7faaa0038740 auth_method 0 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.186+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.187+0000 7faab37fe640 10 monclient(hunting): handle_auth_done global_id 144259 payload 824 2024-09-15T02:03:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.187+0000 7faab37fe640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faab40724a0 con 0x7faab406e230 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faab37fe640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 msgr2=0x7faab406e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faab37fe640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 0x7faab406e600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faab89f4640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faaa0038740 0x7faaa003abf0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7faab4097e00 tx=0x7faaac00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8001ce0 con 0x7faab406e230 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 10 monclient: dump: 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.210 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faaa80072e0 con 0x7faab406e230 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 msgr2=0x7faab406e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 0x7faab406e600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 msgr2=0x7faab406e600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.189+0000 7faa9bfff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab406e230 0x7faab406e600 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.195+0000 7faaba457640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.195+0000 7faaba457640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 0x7faab4069cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.195+0000 7faaba457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.195+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4117680 con 0x7faab40698d0 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.195+0000 7faab37fe640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 0x7faab4069cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient: get_auth_request con 0x7faab40698d0 auth_method 0 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 10 monclient(hunting): handle_auth_done global_id 144262 payload 824 2024-09-15T02:03:27.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.196+0000 7faab37fe640 1 -- 172.21.15.17:0/2051981482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faab409cfe0 con 0x7faab40698d0 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faab37fe640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 msgr2=0x7faab4069cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faab37fe640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 0x7faab4069cd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa801fc50 con 0x7faab40698d0 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.198+0000 7faa9bfff640 10 monclient: dump: 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.213 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7faaa801fdb0 con 0x7faab40698d0 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 msgr2=0x7faab4069cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 0x7faab4069cd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa801b000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 msgr2=0x7faab4069cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:27.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faa9bfff640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40698d0 0x7faab4069cd0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa801b000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:03:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:27.228 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faaa0038740 msgr2=0x7faaa003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faaa0038740 0x7faaa003abf0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7faab4097e00 tx=0x7faaac00b040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 10 monclient: shutdown 2024-09-15T02:03:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 msgr2=0x7faab413dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 --2- 172.21.15.17:0/2051981482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faab41350f0 0x7faab413dca0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c5f0 tx=0x7faaa4034000 comp rx=0 tx=0).stop 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faab89f4640 1 -- 172.21.15.17:0/2051981482 reap_dead start 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faab89f4640 1 -- 172.21.15.17:0/2051981482 reap_dead start 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 shutdown_connections 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 >> 172.21.15.17:0/2051981482 conn(0x7faab40a1000 msgr2=0x7faab409ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 shutdown_connections 2024-09-15T02:03:27.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.199+0000 7faaba457640 1 -- 172.21.15.17:0/2051981482 wait complete. 2024-09-15T02:03:27.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:27.230 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:03:27.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:03:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.325+0000 7f860da57640 10 monclient: get_monmap_and_config 2024-09-15T02:03:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.325+0000 7f860da57640 10 monclient: build_initial_monmap 2024-09-15T02:03:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.325+0000 7f860da57640 10 monclient: monmap: 2024-09-15T02:03:27.327 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:27.326402+0000 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:27.326402+0000 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:27.328 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 Processor -- start 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 -- start start 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient: init 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f860806eb30 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f860809e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f860809c490 0x7f860813ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): picked mon.noname-c con 0x7f860809c490 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608069f70 con 0x7f860809e970 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860809ed40 con 0x7f860806eb30 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860806ab30 con 0x7f860809c490 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.326+0000 7f860da57640 10 monclient(hunting): authenticate will time out at 1467.322914s 2024-09-15T02:03:27.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f85fffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f860809e970 0x7f860809bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35544/0 (socket says 172.21.15.17:35544) 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f85fffff640 1 -- 172.21.15.17:0/889283324 learned_addr learned my addr 172.21.15.17:0/889283324 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f86077fe640 1 --2- 172.21.15.17:0/889283324 >> v2:172.21.15.67:3300/0 conn(0x7f860809c490 0x7f860813ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f86077fe640 10 monclient(hunting): get_auth_request con 0x7f860809c490 auth_method 0 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f86077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f86077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.327+0000 7f86077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 1 -- 172.21.15.17:0/889283324 >> v1:172.21.15.17:6789/0 conn(0x7f860809e970 legacy=0x7f860809bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480301448 0 0) 0x7f8608069f70 con 0x7f860809e970 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): my global_id is 105183 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f40034e0 con 0x7f860809e970 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 1 -- 172.21.15.17:0/889283324 >> v1:172.21.15.67:6789/0 conn(0x7f860806eb30 legacy=0x7f860806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138256918 0 0) 0x7f85f40034e0 con 0x7f860809e970 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 >> v2:172.21.15.67:3300/0 conn(0x7f860809c490 msgr2=0x7f860813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:27.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 --2- 172.21.15.17:0/889283324 >> v2:172.21.15.67:3300/0 conn(0x7f860809c490 0x7f860813ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 >> v1:172.21.15.67:6789/0 conn(0x7f860806eb30 legacy=0x7f860806ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860809f6c0 con 0x7f860809e970 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f85f0001710 con 0x7f860809e970 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: dump: 2024-09-15T02:03:27.342 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 >> v1:172.21.15.17:6789/0 conn(0x7f860809e970 legacy=0x7f860809bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85f4003b20 0x7f85f4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): picked mon.b con 0x7f85f4003b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85f4004780 0x7f85f4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): picked mon.c con 0x7f85f4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 0x7f85f400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): picked mon.a con 0x7f85f40053b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860806ab30 con 0x7f85f40053b0 2024-09-15T02:03:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f400c380 con 0x7f85f4003b20 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f400c540 con 0x7f85f4004780 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86067fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86077fe640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 0x7f85f400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86077fe640 10 monclient(hunting): get_auth_request con 0x7f85f40053b0 auth_method 0 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85f4004780 0x7f85f4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85f4003b20 0x7f85f4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f86077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 10 monclient(hunting): get_auth_request con 0x7f85f4003b20 auth_method 0 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f8606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 10 monclient(hunting): get_auth_request con 0x7f85f4004780 auth_method 0 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.329+0000 7f85fffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f85fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f85fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f85fffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient(hunting): handle_auth_done global_id 105183 payload 293 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 -- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85f4004780 msgr2=0x7f85f4004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85f4004780 0x7f85f4004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 -- 172.21.15.17:0/889283324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85f4003b20 msgr2=0x7f85f4003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85f4003b20 0x7f85f4003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 monclient: found mon.a 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 -- 172.21.15.17:0/889283324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40034e0 con 0x7f85f40053b0 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f860da57640 5 monclient: authenticate success, global_id 105183 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f860da57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f8606ffd640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85f4003b20 0x7f85f4003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.330+0000 7f86077fe640 1 -- 172.21.15.17:0/889283324 reap_dead start 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86077fe640 1 -- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 msgr2=0x7f85f400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86077fe640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 0x7f85f400baa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8600007440 tx=0x7f8600009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607c2e7e6fb195a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860002e670 con 0x7f85f40053b0 2024-09-15T02:03:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: dump: 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860002e810 con 0x7f85f40053b0 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 1 -- 172.21.15.17:0/889283324 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860002ebd0 con 0x7f85f40053b0 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 4 set_mon_vals no callback set 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f86067fc640 10 monclient: dump: 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 10 monclient: shutdown 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 -- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 msgr2=0x7f85f400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 0x7f85f400baa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8600007440 tx=0x7f8600009710 comp rx=0 tx=0).stop 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 -- 172.21.15.17:0/889283324 shutdown_connections 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 --2- 172.21.15.17:0/889283324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85f40053b0 0x7f85f400baa0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 -- 172.21.15.17:0/889283324 >> 172.21.15.17:0/889283324 conn(0x7f86080712f0 msgr2=0x7f86080a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 -- 172.21.15.17:0/889283324 shutdown_connections 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.333+0000 7f860da57640 1 -- 172.21.15.17:0/889283324 wait complete. 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.334+0000 7f860da57640 10 monclient: build_initial_monmap 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.334+0000 7f860da57640 10 monclient: monmap: 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:27.352 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.334+0000 7f860da57640 1 Processor -- start 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.334+0000 7f860da57640 1 -- start start 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.334+0000 7f860da57640 10 monclient: init 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): picked mon.noname-a con 0x7f8608141080 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 0x7f8608134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): picked mon.noname-b con 0x7f8608133f60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f860811ea70 0x7f860811eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): picked mon.noname-c con 0x7f860811ea70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608134870 con 0x7f8608141080 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860809cd60 con 0x7f8608133f60 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860811fee0 con 0x7f860811ea70 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f860da57640 10 monclient(hunting): authenticate will time out at 1467.331140s 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35794/0 (socket says 172.21.15.17:35794) 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 -- 172.21.15.17:0/2238707643 learned_addr learned my addr 172.21.15.17:0/2238707643 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): get_auth_request con 0x7f8608141080 auth_method 0 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f860811ea70 0x7f860811eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): get_auth_request con 0x7f860811ea70 auth_method 0 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 0x7f8608134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 10 monclient(hunting): get_auth_request con 0x7f8608133f60 auth_method 0 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f86077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient(hunting): handle_auth_done global_id 105186 payload 824 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 -- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f860811ea70 msgr2=0x7f860811eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f860811ea70 0x7f860811eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 -- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 msgr2=0x7f8608134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 0x7f8608134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 monclient: found mon.noname-a 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f8606ffd640 1 -- 172.21.15.17:0/2238707643 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608122370 con 0x7f8608141080 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.335+0000 7f85fffff640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 0x7f8608134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f8606ffd640 1 -- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 msgr2=0x7f8608141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f8606ffd640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f85f800f9d0 tx=0x7f85f8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f80074f0 con 0x7f8608141080 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: dump: 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.357 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: _finish_auth 0 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8007650 con 0x7f8608141080 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f860da57640 5 monclient: authenticate success, global_id 105186 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8005990 con 0x7f8608141080 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f85feffd640 10 monclient: dump: 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f860da57640 10 monclient: _renew_subs 2024-09-15T02:03:27.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f860da57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8608141990 con 0x7f8608141080 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.337+0000 7f8605ffb640 4 set_mon_vals no callback set 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 10 monclient: _renew_subs 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8608122090 con 0x7f8608141080 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85f800d730 con 0x7f8608141080 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.338+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86080724a0 con 0x7f8608141080 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.339+0000 7f85feffd640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 0x7f85e803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.339+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f85f804c760 con 0x7f8608141080 2024-09-15T02:03:27.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.339+0000 7f85fffff640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 0x7f85e803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:27.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.339+0000 7f85fffff640 10 monclient: get_auth_request con 0x7f85e8038740 auth_method 0 2024-09-15T02:03:27.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.341+0000 7f85fffff640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 0x7f85e803abf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f85f00018c0 tx=0x7f85f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:27.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.343+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85f8005d80 con 0x7f8608141080 2024-09-15T02:03:27.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.343+0000 7f85feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:27.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.343+0000 7f85feffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:27.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.568+0000 7f860da57640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.568+0000 7f860da57640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.568+0000 7f860da57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.568+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8608067220 con 0x7f8608141080 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f85feffd640 1 -- 172.21.15.17:0/2238707643 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f85f801ba60 con 0x7f8608141080 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f85feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:03:27.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f85feffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=8,logm=51} crc {auth=1303574637,config=3140514009,health=300358084,kv=1529054990,logm=1606235745}) 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=2120124341,config=3140514009,health=1350913232,kv=1529054990,logm=22566721}) 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=8,logm=51} crc {auth=1303574637,config=3140514009,health=300358084,kv=1529054990,logm=1606235745}) 2024-09-15T02:03:27.571 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.569+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=1303574637,config=3140514010,health=1350913232,kv=1529054990,logm=4181193562}) 2024-09-15T02:03:27.572 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.572 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041811397}) 2024-09-15T02:03:27.572 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907771983}) 2024-09-15T02:03:27.572 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.573 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041811397}) 2024-09-15T02:03:27.573 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.570+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1580388772}) 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 msgr2=0x7f85e803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 0x7f85e803abf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f85f00018c0 tx=0x7f85f0009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 10 monclient: shutdown 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 msgr2=0x7f8608141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f85f800f9d0 tx=0x7f85f8009040 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 shutdown_connections 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85e8038740 0x7f85e803abf0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f860811ea70 0x7f860811eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8608133f60 0x7f8608134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 --2- 172.21.15.17:0/2238707643 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8608141080 0x7f8608141450 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 >> 172.21.15.17:0/2238707643 conn(0x7f86080712f0 msgr2=0x7f860806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:27.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 shutdown_connections 2024-09-15T02:03:27.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:27.571+0000 7f860da57640 1 -- 172.21.15.17:0/2238707643 wait complete. 2024-09-15T02:03:27.576 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.576 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419462891}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34764008}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419462891}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.572+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282264870}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965341903}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2350924513}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965341903}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.573+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869504477}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983467201}) 2024-09-15T02:03:27.577 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165644258}) 2024-09-15T02:03:27.578 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.578 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983467201}) 2024-09-15T02:03:27.578 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.574+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287455842}) 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217323377}) 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684189970}) 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217323377}) 2024-09-15T02:03:27.580 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.575+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3331519812}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211107016}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1098367264}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211107016}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.576+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2528122384}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900065460}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1154523003}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900065460}) 2024-09-15T02:03:27.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.577+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259455148}) 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464832197}) 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904351078}) 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464832197}) 2024-09-15T02:03:27.582 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.578+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624114113}) 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892363996}) 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376063853}) 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892363996}) 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.579+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067161339}) 2024-09-15T02:03:27.585 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114089362}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76064397}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114089362}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.580+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563350508}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688710492}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242971396}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688710492}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.581+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232819431}) 2024-09-15T02:03:27.586 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402259573}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869064013}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402259573}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.582+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738863062}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140510600}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576345450}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140510600}) 2024-09-15T02:03:27.587 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861812125}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691106781}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36589036}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.583+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691106781}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965297847}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722407502}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081877136}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722407502}) 2024-09-15T02:03:27.589 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.584+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411284433}) 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045374604}) 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386436261}) 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045374604}) 2024-09-15T02:03:27.590 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.585+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105572721}) 2024-09-15T02:03:27.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779648969}) 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710552373}) 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779648969}) 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712514136}) 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264547044}) 2024-09-15T02:03:27.593 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603307508}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264547044}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.587+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1656449467}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460997267}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1024361388}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460997267}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2224394847}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782541767}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465033324}) 2024-09-15T02:03:27.594 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782541767}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.588+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036459892}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872993200}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1667655294}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872993200}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.589+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2377703689}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481692612}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098101569}) 2024-09-15T02:03:27.595 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:27.596 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481692612}) 2024-09-15T02:03:27.596 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:27.590+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2634073220}) 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16554580}) 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391988445}) 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16554580}) 2024-09-15T02:03:28.025 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=407998364}) 2024-09-15T02:03:28.026 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678764081}) 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.023+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746755461}) 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678764081}) 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956388298}) 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.027 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497774036}) 2024-09-15T02:03:28.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3857374656}) 2024-09-15T02:03:28.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497774036}) 2024-09-15T02:03:28.028 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.024+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2841069295}) 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467629988}) 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059824922}) 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467629988}) 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.025+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130047582}) 2024-09-15T02:03:28.034 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560749731}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811849125}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560749731}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.026+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=333635020}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617592287}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911404560}) 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.035 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617592287}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.027+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744110783}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004237040}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2595127915}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004237040}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2600332492}) 2024-09-15T02:03:28.036 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066850666}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2462633914}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066850666}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1361315357}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639720259}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943649825}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639720259}) 2024-09-15T02:03:28.037 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.029+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309311365}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114667173}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174334398}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114667173}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.030+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869228488}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755842019}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448948746}) 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.038 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755842019}) 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.031+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159273917}) 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758446965}) 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457423137}) 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758446965}) 2024-09-15T02:03:28.039 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.032+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430620599}) 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=353433542}) 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645289236}) 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=353433542}) 2024-09-15T02:03:28.045 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1001291767}) 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.033+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049064969}) 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434664412}) 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049064969}) 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040690610}) 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.046 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573393263}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296688947}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573393263}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.034+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295398078}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163161014}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3065665412}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163161014}) 2024-09-15T02:03:28.047 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.035+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721416140}) 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843844410}) 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3493493319}) 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843844410}) 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.036+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433500430}) 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.048 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082698172}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2486485125}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082698172}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.037+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27161756}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921287992}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377240320}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921287992}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886470725}) 2024-09-15T02:03:28.049 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735092608}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835718331}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735092608}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.038+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415809095}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293988143}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=653348698}) 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.050 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293988143}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.039+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193603987}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615739041}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071026388}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615739041}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.040+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753002178}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823103708}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767544394}) 2024-09-15T02:03:28.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823103708}) 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299475960}) 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935252389}) 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3823787270}) 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935252389}) 2024-09-15T02:03:28.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.042+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898378453}) 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533370056}) 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167997604}) 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533370056}) 2024-09-15T02:03:28.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.043+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022663170}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422484978}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3400963700}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422484978}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552668165}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563656193}) 2024-09-15T02:03:28.057 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198334542}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563656193}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.044+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021515022}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630639688}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757738877}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630639688}) 2024-09-15T02:03:28.058 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.045+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21473682}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=35482365}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753384900}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=35482365}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.046+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620748253}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449907195}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=767224233}) 2024-09-15T02:03:28.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449907195}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.047+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570353979}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448556427}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849354493}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448556427}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1390847094}) 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.060 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246509793}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381103329}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.048+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246509793}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430819814}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425892672}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2326521271}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425892672}) 2024-09-15T02:03:28.061 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.049+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1210568500}) 2024-09-15T02:03:28.064 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.064 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920446447}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179073176}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920446447}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.057+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614227115}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029386448}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411039846}) 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.065 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029386448}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.058+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110105872}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487679630}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053271072}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487679630}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.059+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215413437}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353295422}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910922762}) 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.066 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353295422}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.060+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=696348739}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558151875}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132426475}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558151875}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404683776}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759813026}) 2024-09-15T02:03:28.067 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577980107}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759813026}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.061+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437626354}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099661719}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263790560}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099661719}) 2024-09-15T02:03:28.068 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.062+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598379799}) 2024-09-15T02:03:28.069 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.069 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2} crc {logm=2820349310,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=565579676,mon_config_key=1583035879,monmap=2689118867}) 2024-09-15T02:03:28.069 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=1} crc {logm=1439263265,mdsmap=1531151339,mgr=1727179341,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2883451744,mon_config_key=1583035879,monmap=662845760}) 2024-09-15T02:03:28.069 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.069 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2} crc {logm=2820349310,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=565579676,mon_config_key=1583035879,monmap=2689118867}) 2024-09-15T02:03:28.070 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.063+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3195782302,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2883451744,mon_config_key=2516852880}) 2024-09-15T02:03:28.072 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.072 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=2222559947}) 2024-09-15T02:03:28.072 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {monmap=1228700967,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3060169243}) 2024-09-15T02:03:28.073 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:03:28.073 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=2222559947}) 2024-09-15T02:03:28.073 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:03:28.070+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,osd_metadata=6,osd_snap=2,osdmap=59} crc {monmap=1228700967,osd_metadata=3375641772,osd_snap=1063953654,osdmap=1097914596}) 2024-09-15T02:03:28.592 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:03:28.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.699+0000 7f619ff9d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.699+0000 7f619ff9d640 10 monclient: build_initial_monmap 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.699+0000 7f619ff9d640 10 monclient: monmap: 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:28.700463+0000 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:28.700463+0000 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:28.702 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 Processor -- start 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 -- start start 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient: init 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 0x7f619809c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-b con 0x7f619809c300 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f619809a920 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f619809acf0 0x7f619809e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-c con 0x7f619809acf0 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6198099f30 con 0x7f619809a920 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619806a5c0 con 0x7f619809acf0 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619809cc10 con 0x7f619809c300 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.700+0000 7f619ff9d640 10 monclient(hunting): authenticate will time out at 1468.696981s 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 0x7f619809c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619d538640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f619809a920 0x7f619806eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35558/0 (socket says 172.21.15.17:35558) 2024-09-15T02:03:28.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619d538640 1 -- 172.21.15.17:0/2851655848 learned_addr learned my addr 172.21.15.17:0/2851655848 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): get_auth_request con 0x7f619809c300 auth_method 0 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619e53a640 1 --2- 172.21.15.17:0/2851655848 >> v2:172.21.15.67:3300/0 conn(0x7f619809acf0 0x7f619809e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient(hunting): handle_auth_done global_id 164111 payload 274 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 -- 172.21.15.17:0/2851655848 >> v2:172.21.15.67:3300/0 conn(0x7f619809acf0 msgr2=0x7f619809e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 --2- 172.21.15.17:0/2851655848 >> v2:172.21.15.67:3300/0 conn(0x7f619809acf0 0x7f619809e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 -- 172.21.15.17:0/2851655848 >> v1:172.21.15.17:6789/0 conn(0x7f619809a920 legacy=0x7f619806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient: _finish_hunting 0 2024-09-15T02:03:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 monclient: found mon.noname-b 2024-09-15T02:03:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.701+0000 7f619dd39640 1 -- 172.21.15.17:0/2851655848 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619809f6e0 con 0x7f619809c300 2024-09-15T02:03:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619dd39640 1 -- 172.21.15.17:0/2851655848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 msgr2=0x7f619809c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619dd39640 1 --2- 172.21.15.17:0/2851655848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 0x7f619809c6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6194009850 tx=0x7f61940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89ed98a273f8c1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 1 -- 172.21.15.17:0/2851655848 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194032440 con 0x7f619809c300 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: dump: 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: _finish_auth 0 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 5 monclient: authenticate success, global_id 164111 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 1 -- 172.21.15.17:0/2851655848 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61940325a0 con 0x7f619809c300 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 4 set_mon_vals no callback set 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 1 -- 172.21.15.17:0/2851655848 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194032940 con 0x7f619809c300 2024-09-15T02:03:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619cd37640 10 monclient: dump: 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 10 monclient: shutdown 2024-09-15T02:03:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 1 -- 172.21.15.17:0/2851655848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 msgr2=0x7f619809c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.704+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2851655848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 0x7f619809c6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6194009850 tx=0x7f61940090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 -- 172.21.15.17:0/2851655848 shutdown_connections 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2851655848 >> v2:172.21.15.67:3300/0 conn(0x7f619809acf0 0x7f619809e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2851655848 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809c300 0x7f619809c6d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 -- 172.21.15.17:0/2851655848 >> 172.21.15.17:0/2851655848 conn(0x7f61980a2f70 msgr2=0x7f61980a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 -- 172.21.15.17:0/2851655848 shutdown_connections 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 -- 172.21.15.17:0/2851655848 wait complete. 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 10 monclient: build_initial_monmap 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 10 monclient: monmap: 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.705+0000 7f619ff9d640 1 Processor -- start 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 -- start start 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient: init 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-c con 0x7f619809a920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619809acf0 0x7f6198131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-b con 0x7f619809acf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f619809c300 0x7f6198097a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): picked mon.noname-a con 0x7f619809c300 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619811dbe0 con 0x7f619809c300 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619809cdf0 con 0x7f619809acf0 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980a2980 con 0x7f619809a920 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619ff9d640 10 monclient(hunting): authenticate will time out at 1468.702583s 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48340/0 (socket says 172.21.15.17:48340) 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f619809c300 0x7f6198097a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 1 -- 172.21.15.17:0/2853527165 learned_addr learned my addr 172.21.15.17:0/2853527165 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): get_auth_request con 0x7f619809a920 auth_method 0 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): get_auth_request con 0x7f619809c300 auth_method 0 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619e53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619d538640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619809acf0 0x7f6198131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619d538640 10 monclient(hunting): get_auth_request con 0x7f619809acf0 auth_method 0 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619d538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619d538640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619d538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.706+0000 7f619dd39640 10 monclient(hunting): handle_auth_done global_id 164114 payload 824 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 -- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619809acf0 msgr2=0x7f6198131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619809acf0 0x7f6198131dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 -- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f619809c300 msgr2=0x7f6198097a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f619809c300 0x7f6198097a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 10 monclient: _finish_hunting 0 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 monclient: found mon.noname-c 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.707+0000 7f619dd39640 1 -- 172.21.15.17:0/2853527165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194009390 con 0x7f619809a920 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619dd39640 1 -- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 msgr2=0x7f6198141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619dd39640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6194009850 tx=0x7f6194032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031990 con 0x7f619809a920 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:03:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: dump: 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: _finish_auth 0 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 5 monclient: authenticate success, global_id 164114 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194031af0 con 0x7f619809a920 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031e70 con 0x7f619809a920 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 10 monclient: _renew_subs 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619811eee0 con 0x7f619809a920 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618ffff640 4 set_mon_vals no callback set 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f618dffb640 10 monclient: dump: 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 10 monclient: _renew_subs 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.709+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619811e070 con 0x7f619809a920 2024-09-15T02:03:28.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6194039070 con 0x7f619809a920 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f619ff9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f619ff9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f619ff9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619811f660 con 0x7f619809a920 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.710+0000 7f618dffb640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 0x7f617c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.711+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f619406cc60 con 0x7f619809a920 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.711+0000 7f619d538640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 0x7f617c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.711+0000 7f619d538640 10 monclient: get_auth_request con 0x7f617c038740 auth_method 0 2024-09-15T02:03:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.713+0000 7f619d538640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 0x7f617c03abf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6188007fd0 tx=0x7f6188009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.715+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f619403c410 con 0x7f619809a920 2024-09-15T02:03:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.715+0000 7f618dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.715+0000 7f618dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:28.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f619ff9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:03:28.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f619ff9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:28.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f619ff9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:28.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f619811f350 con 0x7f619809a920 2024-09-15T02:03:28.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f618dffb640 1 -- 172.21.15.17:0/2853527165 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f6194040020 con 0x7f619809a920 2024-09-15T02:03:28.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f618dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:28.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.943+0000 7f618dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:28.946 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:03:28.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 msgr2=0x7f617c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 0x7f617c03abf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6188007fd0 tx=0x7f6188009040 comp rx=0 tx=0).stop 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 10 monclient: shutdown 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 msgr2=0x7f6198141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6194009850 tx=0x7f6194032000 comp rx=0 tx=0).stop 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 shutdown_connections 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f617c038740 0x7f617c03abf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f619809c300 0x7f6198097a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619809acf0 0x7f6198131dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 --2- 172.21.15.17:0/2853527165 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f619809a920 0x7f6198141370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:28.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 >> 172.21.15.17:0/2853527165 conn(0x7f61980a2f70 msgr2=0x7f619809ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:28.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 shutdown_connections 2024-09-15T02:03:28.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:28.945+0000 7f619ff9d640 1 -- 172.21.15.17:0/2853527165 wait complete. 2024-09-15T02:03:28.957 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:03:28.958 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:03:28.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:03:28.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.053+0000 7efc82f9d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.053+0000 7efc82f9d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.053+0000 7efc82f9d640 10 monclient: monmap: 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:29.054329+0000 2024-09-15T02:03:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:29.054329+0000 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:29.056 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 Processor -- start 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 -- start start 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient: init 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 0x7efc7c09a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c09c0d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-b con 0x7efc7c09c4a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c09c870 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c06a5e0 con 0x7efc7c09c870 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c06a290 con 0x7efc7c09c0d0 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0699c0 con 0x7efc7c09c4a0 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc82f9d640 10 monclient(hunting): authenticate will time out at 1469.050781s 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35830/0 (socket says 172.21.15.17:35830) 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48344/0 (socket says 172.21.15.17:48344) 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 1 -- 172.21.15.17:0/2894000386 learned_addr learned my addr 172.21.15.17:0/2894000386 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 10 monclient(hunting): get_auth_request con 0x7efc7c09c870 auth_method 0 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc8153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 10 monclient(hunting): get_auth_request con 0x7efc7c09c4a0 auth_method 0 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.054+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 0x7efc7c09a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 10 monclient(hunting): get_auth_request con 0x7efc7c09c0d0 auth_method 0 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient(hunting): handle_auth_done global_id 164117 payload 274 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 -- 172.21.15.17:0/2894000386 >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 msgr2=0x7efc7c09a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 0x7efc7c09a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 -- 172.21.15.17:0/2894000386 >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 msgr2=0x7efc7c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 monclient: found mon.noname-b 2024-09-15T02:03:29.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:03:29.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc7bfff640 1 -- 172.21.15.17:0/2894000386 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c09f6e0 con 0x7efc7c09c4a0 2024-09-15T02:03:29.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc8153a640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:29.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.055+0000 7efc80d39640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 0x7efc7c09a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:03:29.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7bfff640 1 -- 172.21.15.17:0/2894000386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 msgr2=0x7efc7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7bfff640 1 --2- 172.21.15.17:0/2894000386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0a3e50 tx=0x7efc640090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471de32e59e584d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 1 -- 172.21.15.17:0/2894000386 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6402adf0 con 0x7efc7c09c4a0 2024-09-15T02:03:29.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: dump: 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 5 monclient: authenticate success, global_id 164117 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 1 -- 172.21.15.17:0/2894000386 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc64032440 con 0x7efc7c09c4a0 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 4 set_mon_vals no callback set 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 1 -- 172.21.15.17:0/2894000386 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640327e0 con 0x7efc7c09c4a0 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc7b7fe640 10 monclient: dump: 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.071 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 10 monclient: shutdown 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 -- 172.21.15.17:0/2894000386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 msgr2=0x7efc7c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 --2- 172.21.15.17:0/2894000386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0a3e50 tx=0x7efc640090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 -- 172.21.15.17:0/2894000386 shutdown_connections 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.17:3300/0 conn(0x7efc7c09c870 0x7efc7c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 --2- 172.21.15.17:0/2894000386 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c4a0 0x7efc7c06eb30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 --2- 172.21.15.17:0/2894000386 >> v2:172.21.15.67:3300/0 conn(0x7efc7c09c0d0 0x7efc7c09a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 -- 172.21.15.17:0/2894000386 >> 172.21.15.17:0/2894000386 conn(0x7efc7c0712f0 msgr2=0x7efc7c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 -- 172.21.15.17:0/2894000386 shutdown_connections 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.058+0000 7efc82f9d640 1 -- 172.21.15.17:0/2894000386 wait complete. 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.059+0000 7efc82f9d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.059+0000 7efc82f9d640 10 monclient: monmap: 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:29.073 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.059+0000 7efc82f9d640 1 Processor -- start 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.059+0000 7efc82f9d640 1 -- start start 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.059+0000 7efc82f9d640 10 monclient: init 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 0x7efc7c13d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c09c0d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c09c4a0 0x7efc7c13d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-b con 0x7efc7c09c4a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 0x7efc7c140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c09c870 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1229c0 con 0x7efc7c09c870 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c122400 con 0x7efc7c09c4a0 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c11fee0 con 0x7efc7c09c0d0 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc82f9d640 10 monclient(hunting): authenticate will time out at 1469.056301s 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 0x7efc7c13d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 0x7efc7c140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 0x7efc7c13d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48350/0 (socket says 172.21.15.17:48350) 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 1 -- 172.21.15.17:0/259136121 learned_addr learned my addr 172.21.15.17:0/259136121 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): get_auth_request con 0x7efc7c09c0d0 auth_method 0 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): get_auth_request con 0x7efc7c09c870 auth_method 0 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc7bfff640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c09c4a0 0x7efc7c13d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc7bfff640 10 monclient(hunting): get_auth_request con 0x7efc7c09c4a0 auth_method 0 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient(hunting): handle_auth_done global_id 105195 payload 824 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 msgr2=0x7efc7c13d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 0x7efc7c13d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c09c4a0 msgr2=0x7efc7c13d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c09c4a0 0x7efc7c13d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 monclient: found mon.noname-a 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc8153a640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009ae0 con 0x7efc7c09c870 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.060+0000 7efc80d39640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7efc7c09c0d0 0x7efc7c13d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.062+0000 7efc8153a640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 msgr2=0x7efc7c140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.062+0000 7efc8153a640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 0x7efc7c140f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7efc74009f40 tx=0x7efc7400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74018070 con 0x7efc7c09c870 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:29.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: dump: 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 5 monclient: authenticate success, global_id 105195 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7400f040 con 0x7efc7c09c870 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 monclient: _renew_subs 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc64009390 con 0x7efc7c09c870 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740092e0 con 0x7efc7c09c870 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc78ff9640 10 monclient: dump: 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.080 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc7affd640 4 set_mon_vals no callback set 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 monclient: _renew_subs 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c13d5c0 con 0x7efc7c09c870 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 0x7efc7c141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.063+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c122880 con 0x7efc7c121250 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efc74003020 con 0x7efc7c09c870 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 0x7efc7c141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient: get_auth_request con 0x7efc7c121250 auth_method 0 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 10 monclient(hunting): handle_auth_done global_id 105198 payload 824 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc80d39640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc7c122b30 con 0x7efc7c121250 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc78ff9640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc5c038740 0x7efc5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7efc7404c800 con 0x7efc7c09c870 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc7bfff640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc5c038740 0x7efc5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.064+0000 7efc7bfff640 10 monclient: get_auth_request con 0x7efc5c038740 auth_method 0 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc80d39640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 msgr2=0x7efc7c141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc80d39640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 0x7efc7c141480 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c014640 con 0x7efc7c121250 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 10 monclient: dump: 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.083 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efc6c0147a0 con 0x7efc7c121250 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 msgr2=0x7efc7c141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 0x7efc7c141480 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c008040 comp rx=0 tx=0).stop 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 msgr2=0x7efc7c141480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.066+0000 7efc78ff9640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c121250 0x7efc7c141480 secure :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c008040 comp rx=0 tx=0).stop 2024-09-15T02:03:29.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.067+0000 7efc7bfff640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc5c038740 0x7efc5c03abf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06a100 tx=0x7efc64032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc82f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:29.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc82f9d640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 0x7efc7c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc82f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c13d810 con 0x7efc7c13e330 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 0x7efc7c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient: get_auth_request con 0x7efc7c13e330 auth_method 0 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 10 monclient(hunting): handle_auth_done global_id 105201 payload 824 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.072+0000 7efc80d39640 1 -- 172.21.15.17:0/259136121 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc7c122230 con 0x7efc7c13e330 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc80d39640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 msgr2=0x7efc7c1419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc80d39640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 0x7efc7c1419c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c02b070 con 0x7efc7c13e330 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 10 monclient: dump: 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.088 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7efc6c023040 con 0x7efc7c13e330 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 msgr2=0x7efc7c1419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 0x7efc7c1419c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c01d000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 msgr2=0x7efc7c1419c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc78ff9640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c13e330 0x7efc7c1419c0 secure :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc6c010da0 tx=0x7efc6c01d000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:29.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:29.090 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.091 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:29.092 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:29.093 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.094 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:29.095 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.096 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:29.098 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.100 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:29.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.102 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:29.103 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc5c038740 msgr2=0x7efc5c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7efc5c038740 0x7efc5c03abf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06a100 tx=0x7efc64032000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 10 monclient: shutdown 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 msgr2=0x7efc7c140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc8153a640 1 -- 172.21.15.17:0/259136121 reap_dead start 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 0x7efc7c140f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7efc74009f40 tx=0x7efc7400b040 comp rx=0 tx=0).stop 2024-09-15T02:03:29.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 shutdown_connections 2024-09-15T02:03:29.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 --2- 172.21.15.17:0/259136121 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efc7c09c870 0x7efc7c140f40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.075+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 >> 172.21.15.17:0/259136121 conn(0x7efc7c0712f0 msgr2=0x7efc7c098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.076+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 shutdown_connections 2024-09-15T02:03:29.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.076+0000 7efc82f9d640 1 -- 172.21.15.17:0/259136121 wait complete. 2024-09-15T02:03:29.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:29.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:03:29.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:03:29.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.201+0000 7fde2871d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.201+0000 7fde2871d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.201+0000 7fde2871d640 10 monclient: monmap: 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:29.202785+0000 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:29.202785+0000 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:29.204 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:29.205 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:29.205 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.202+0000 7fde2871d640 1 Processor -- start 2024-09-15T02:03:29.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.202+0000 7fde2871d640 1 -- start start 2024-09-15T02:03:29.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.202+0000 7fde2871d640 10 monclient: init 2024-09-15T02:03:29.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde20131b20 0x7fde2009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-c con 0x7fde20131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-b con 0x7fde200a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fde2009f1e0 0x7fde20132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-a con 0x7fde2009f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde200a0040 con 0x7fde2009f1e0 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2009bff0 con 0x7fde20131b20 2024-09-15T02:03:29.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde200a14a0 con 0x7fde200a0280 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde2871d640 10 monclient(hunting): authenticate will time out at 1469.199367s 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48364/0 (socket says 172.21.15.17:48364) 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 1 -- 172.21.15.17:0/2825009158 learned_addr learned my addr 172.21.15.17:0/2825009158 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): get_auth_request con 0x7fde200a0280 auth_method 0 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 --2- 172.21.15.17:0/2825009158 >> v2:172.21.15.17:3300/0 conn(0x7fde2009f1e0 0x7fde20132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): get_auth_request con 0x7fde2009f1e0 auth_method 0 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 1 --2- 172.21.15.17:0/2825009158 >> v2:172.21.15.67:3300/0 conn(0x7fde20131b20 0x7fde2009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): get_auth_request con 0x7fde20131b20 auth_method 0 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient(hunting): handle_auth_done global_id 105204 payload 274 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 -- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 msgr2=0x7fde20131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 -- 172.21.15.17:0/2825009158 >> v2:172.21.15.67:3300/0 conn(0x7fde20131b20 msgr2=0x7fde2009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 --2- 172.21.15.17:0/2825009158 >> v2:172.21.15.67:3300/0 conn(0x7fde20131b20 0x7fde2009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 monclient: found mon.noname-a 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde26cba640 1 -- 172.21.15.17:0/2825009158 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20068e80 con 0x7fde2009f1e0 2024-09-15T02:03:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.203+0000 7fde25cb8640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde26cba640 1 -- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 msgr2=0x7fde20132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde26cba640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde20132430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fde2009f710 tx=0x7fde18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e2d484b452f7d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 1 -- 172.21.15.17:0/2825009158 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18007760 con 0x7fde2009f1e0 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: dump: 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.218 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde2871d640 5 monclient: authenticate success, global_id 105204 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 1 -- 172.21.15.17:0/2825009158 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1802d050 con 0x7fde2009f1e0 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde2871d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:29.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde2871d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde2871d640 4 set_mon_vals no callback set 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 1 -- 172.21.15.17:0/2825009158 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18007af0 con 0x7fde2009f1e0 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.206+0000 7fde254b7640 10 monclient: dump: 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.220 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 10 monclient: shutdown 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 -- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 msgr2=0x7fde20132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde20132430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fde2009f710 tx=0x7fde18009780 comp rx=0 tx=0).stop 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 -- 172.21.15.17:0/2825009158 shutdown_connections 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde20132430 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 --2- 172.21.15.17:0/2825009158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde200a0280 0x7fde20131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 --2- 172.21.15.17:0/2825009158 >> v2:172.21.15.67:3300/0 conn(0x7fde20131b20 0x7fde2009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 -- 172.21.15.17:0/2825009158 >> 172.21.15.17:0/2825009158 conn(0x7fde20071270 msgr2=0x7fde200a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 -- 172.21.15.17:0/2825009158 shutdown_connections 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 1 -- 172.21.15.17:0/2825009158 wait complete. 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.207+0000 7fde2871d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient: monmap: 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.222 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 Processor -- start 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 -- start start 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient: init 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-a con 0x7fde2009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde200a0280 0x7fde2011c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-b con 0x7fde200a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:29.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20131b20 0x7fde2011cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): picked mon.noname-c con 0x7fde20131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde200a09b0 con 0x7fde2009f1e0 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2011fb70 con 0x7fde200a0280 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2012fa50 con 0x7fde20131b20 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde2871d640 10 monclient(hunting): authenticate will time out at 1469.204820s 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35884/0 (socket says 172.21.15.17:35884) 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde26cba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20131b20 0x7fde2011cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 1 -- 172.21.15.17:0/1027477106 learned_addr learned my addr 172.21.15.17:0/1027477106 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 10 monclient(hunting): get_auth_request con 0x7fde2009f1e0 auth_method 0 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde264b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde26cba640 10 monclient(hunting): get_auth_request con 0x7fde20131b20 auth_method 0 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde26cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde26cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.208+0000 7fde26cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde26cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient(hunting): handle_auth_done global_id 105207 payload 824 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde25cb8640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde200a0280 0x7fde2011c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20131b20 msgr2=0x7fde2011cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20131b20 0x7fde2011cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde200a0280 msgr2=0x7fde2011c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde200a0280 0x7fde2011c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 monclient: found mon.noname-a 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.209+0000 7fde264b9640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10009450 con 0x7fde2009f1e0 2024-09-15T02:03:29.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde264b9640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 msgr2=0x7fde2011c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde264b9640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00ea20 tx=0x7fde1c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c009600 con 0x7fde2009f1e0 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: dump: 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.227 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c00cc20 con 0x7fde2009f1e0 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c013470 con 0x7fde2009f1e0 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 5 monclient: authenticate success, global_id 105207 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde167fc640 10 monclient: dump: 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.228 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 10 monclient: _renew_subs 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde24cb6640 4 set_mon_vals no callback set 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde18009a20 con 0x7fde2009f1e0 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 10 monclient: _renew_subs 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.211+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2011e6f0 con 0x7fde2009f1e0 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde2871d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde2871d640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 0x7fde20098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde20063a40 con 0x7fde2011d430 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde1c013860 con 0x7fde2009f1e0 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 0x7fde20098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient: get_auth_request con 0x7fde2011d430 auth_method 0 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde167fc640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde04038740 0x7fde0403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 10 monclient(hunting): handle_auth_done global_id 164129 payload 824 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fde1c04c980 con 0x7fde2009f1e0 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.212+0000 7fde25cb8640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde2011fd30 con 0x7fde2011d430 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.213+0000 7fde26cba640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde04038740 0x7fde0403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.213+0000 7fde26cba640 10 monclient: get_auth_request con 0x7fde04038740 auth_method 0 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde25cb8640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 msgr2=0x7fde20098a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde25cb8640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 0x7fde20098a20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde10010900 tx=0x7fde10008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000ecf0 con 0x7fde2011d430 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 10 monclient: dump: 2024-09-15T02:03:29.231 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fde1000ee50 con 0x7fde2011d430 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:29.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 msgr2=0x7fde20098a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 0x7fde20098a20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde10010900 tx=0x7fde10008040 comp rx=0 tx=0).stop 2024-09-15T02:03:29.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 msgr2=0x7fde20098a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde167fc640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde2011d430 0x7fde20098a20 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde10010900 tx=0x7fde10008040 comp rx=0 tx=0).stop 2024-09-15T02:03:29.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.215+0000 7fde26cba640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde04038740 0x7fde0403abf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fde200a2f90 tx=0x7fde18037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde2871d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:29.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde2871d640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 0x7fde20099360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde2871d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde200724a0 con 0x7fde20098f60 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 0x7fde20099360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient: get_auth_request con 0x7fde20098f60 auth_method 0 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 10 monclient(hunting): handle_auth_done global_id 164132 payload 824 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.221+0000 7fde25cb8640 1 -- 172.21.15.17:0/1027477106 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fde2011f950 con 0x7fde20098f60 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde25cb8640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 msgr2=0x7fde20099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde25cb8640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 0x7fde20099360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde1001b000 tx=0x7fde1001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10020040 con 0x7fde20098f60 2024-09-15T02:03:29.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 10 monclient: dump: 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fde10014990 con 0x7fde20098f60 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 msgr2=0x7fde20099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 0x7fde20099360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde1001b000 tx=0x7fde1001c000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 msgr2=0x7fde20099360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde167fc640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde20098f60 0x7fde20099360 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde1001b000 tx=0x7fde1001c000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:29.237 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.238 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.239 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:29.240 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:29.241 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:29.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.243 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.244 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.245 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.248 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:29.249 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde04038740 msgr2=0x7fde0403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.224+0000 7fde2871d640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde04038740 0x7fde0403abf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fde200a2f90 tx=0x7fde18037000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 10 monclient: shutdown 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 msgr2=0x7fde2011c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00ea20 tx=0x7fde1c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde26cba640 1 -- 172.21.15.17:0/1027477106 reap_dead start 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 shutdown_connections 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 --2- 172.21.15.17:0/1027477106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde2009f1e0 0x7fde2011c470 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 >> 172.21.15.17:0/1027477106 conn(0x7fde20071270 msgr2=0x7fde200a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 shutdown_connections 2024-09-15T02:03:29.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.225+0000 7fde2871d640 1 -- 172.21.15.17:0/1027477106 wait complete. 2024-09-15T02:03:29.252 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:29.252 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:03:29.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:03:29.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.347+0000 7fa747f5d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:29.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.347+0000 7fa747f5d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.347+0000 7fa747f5d640 10 monclient: monmap: 2024-09-15T02:03:29.349 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:29.348562+0000 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:29.348562+0000 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:29.350 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 Processor -- start 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 -- start start 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient: init 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa74009c0d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa74009e970 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:29.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa74006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa74006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400699c0 con 0x7fa74009c0d0 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74009ed70 con 0x7fa74009e970 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74006a5e0 con 0x7fa74006eb30 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.348+0000 7fa747f5d640 10 monclient(hunting): authenticate will time out at 1469.344992s 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa74006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa745cf9640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa74009c0d0 0x7fa74009a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35560/0 (socket says 172.21.15.17:35560) 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa745cf9640 1 -- 172.21.15.17:0/3225982343 learned_addr learned my addr 172.21.15.17:0/3225982343 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): get_auth_request con 0x7fa74006eb30 auth_method 0 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient(hunting): handle_auth_done global_id 164135 payload 274 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 1 -- 172.21.15.17:0/3225982343 >> v1:172.21.15.67:6789/0 conn(0x7fa74009e970 legacy=0x7fa74009c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 1 -- 172.21.15.17:0/3225982343 >> v1:172.21.15.17:6789/0 conn(0x7fa74009c0d0 legacy=0x7fa74009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 1 monclient: found mon.noname-b 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:03:29.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.349+0000 7fa7464fa640 1 -- 172.21.15.17:0/3225982343 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74009cbb0 con 0x7fa74006eb30 2024-09-15T02:03:29.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa7464fa640 1 -- 172.21.15.17:0/3225982343 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 msgr2=0x7fa74006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa7464fa640 1 --2- 172.21.15.17:0/3225982343 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa74006ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa738009e90 tx=0x7fa738009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa84a64d4b758e02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 1 -- 172.21.15.17:0/3225982343 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738007840 con 0x7fa74006eb30 2024-09-15T02:03:29.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: dump: 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 1 -- 172.21.15.17:0/3225982343 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73802d050 con 0x7fa74006eb30 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 1 -- 172.21.15.17:0/3225982343 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738007bd0 con 0x7fa74006eb30 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa747f5d640 5 monclient: authenticate success, global_id 164135 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa747f5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa747f5d640 4 set_mon_vals no callback set 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.352+0000 7fa744cf7640 10 monclient: dump: 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.361 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 10 monclient: shutdown 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 -- 172.21.15.17:0/3225982343 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 msgr2=0x7fa74006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 --2- 172.21.15.17:0/3225982343 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa74006ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa738009e90 tx=0x7fa738009780 comp rx=0 tx=0).stop 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 -- 172.21.15.17:0/3225982343 shutdown_connections 2024-09-15T02:03:29.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 --2- 172.21.15.17:0/3225982343 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa74006ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 -- 172.21.15.17:0/3225982343 >> 172.21.15.17:0/3225982343 conn(0x7fa7400712f0 msgr2=0x7fa7400a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 -- 172.21.15.17:0/3225982343 shutdown_connections 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 1 -- 172.21.15.17:0/3225982343 wait complete. 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 10 monclient: build_initial_monmap 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.353+0000 7fa747f5d640 10 monclient: monmap: 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 Processor -- start 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 -- start start 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient: init 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa74006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c0d0 0x7fa74011d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa74009c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009e970 0x7fa74011df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa74009e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740141e50 con 0x7fa74009c0d0 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011fe50 con 0x7fa74009e970 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa740120d40 con 0x7fa74006eb30 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa747f5d640 10 monclient(hunting): authenticate will time out at 1469.350683s 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:48408/0 (socket says 172.21.15.17:48408) 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 1 -- 172.21.15.17:0/2624751596 learned_addr learned my addr 172.21.15.17:0/2624751596 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): get_auth_request con 0x7fa74006eb30 auth_method 0 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c0d0 0x7fa74011d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): get_auth_request con 0x7fa74009c0d0 auth_method 0 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa745cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7464fa640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009e970 0x7fa74011df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7464fa640 10 monclient(hunting): get_auth_request con 0x7fa74009e970 auth_method 0 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7464fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7464fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.354+0000 7fa7464fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 10 monclient(hunting): handle_auth_done global_id 164138 payload 824 2024-09-15T02:03:29.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009e970 msgr2=0x7fa74011df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009e970 0x7fa74011df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c0d0 msgr2=0x7fa74011d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c0d0 0x7fa74011d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 10 monclient: _finish_hunting 0 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 monclient: found mon.noname-c 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa745cf9640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009a20 con 0x7fa74006eb30 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.355+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c0d0 0x7fa74011d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa745cf9640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 msgr2=0x7fa740133c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa745cf9640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c5e0 tx=0x7fa73c004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.368 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c005cb0 con 0x7fa74006eb30 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: dump: 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.369 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: _finish_auth 0 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c005e30 con 0x7fa74006eb30 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c007340 con 0x7fa74006eb30 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa735ffb640 10 monclient: dump: 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.370 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 5 monclient: authenticate success, global_id 164138 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 10 monclient: _renew_subs 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740121300 con 0x7fa74006eb30 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa737fff640 4 set_mon_vals no callback set 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 10 monclient: _renew_subs 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.357+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7401349c0 con 0x7fa74006eb30 2024-09-15T02:03:29.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa73c01b400 con 0x7fa74006eb30 2024-09-15T02:03:29.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa747f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa747f5d640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 0x7fa74011e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740063b90 con 0x7fa740134160 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa735ffb640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa724038790 0x7fa72403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa73c04c1c0 con 0x7fa74006eb30 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 0x7fa74011e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7454f8640 10 monclient: get_auth_request con 0x7fa740134160 auth_method 0 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7454f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7454f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7454f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7464fa640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa724038790 0x7fa72403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.358+0000 7fa7464fa640 10 monclient: get_auth_request con 0x7fa724038790 auth_method 0 2024-09-15T02:03:29.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.359+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.359+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.359+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.359+0000 7fa7454f8640 10 monclient(hunting): handle_auth_done global_id 144286 payload 824 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.359+0000 7fa7454f8640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7400724a0 con 0x7fa740134160 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa7464fa640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa724038790 0x7fa72403ac40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa738034000 tx=0x7fa738037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa7454f8640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 msgr2=0x7fa74011e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 0x7fa74011e460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa73000fa50 tx=0x7fa730007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000dda0 con 0x7fa740134160 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.361+0000 7fa735ffb640 10 monclient: dump: 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.377 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa730014470 con 0x7fa740134160 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 msgr2=0x7fa74011e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 0x7fa74011e460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa73000fa50 tx=0x7fa730007e10 comp rx=0 tx=0).stop 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 msgr2=0x7fa74011e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.362+0000 7fa735ffb640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa740134160 0x7fa74011e460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.367+0000 7fa747f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.367+0000 7fa747f5d640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 0x7fa7400a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa747f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740121490 con 0x7fa74011e9a0 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 0x7fa7400a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient: get_auth_request con 0x7fa74011e9a0 auth_method 0 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 10 monclient(hunting): handle_auth_done global_id 144289 payload 824 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.368+0000 7fa7454f8640 1 -- 172.21.15.17:0/2624751596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa74011f280 con 0x7fa74011e9a0 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.370+0000 7fa7454f8640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 msgr2=0x7fa7400a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.370+0000 7fa7454f8640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 0x7fa7400a1370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa730019000 tx=0x7fa73001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73001edc0 con 0x7fa74011e9a0 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 10 monclient: dump: 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fa73000db90 con 0x7fa74011e9a0 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 msgr2=0x7fa7400a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 0x7fa7400a1370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa730019000 tx=0x7fa73001a000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 msgr2=0x7fa7400a1370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:29.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa735ffb640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74011e9a0 0x7fa7400a1370 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa730019000 tx=0x7fa73001a000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 128, 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:29.394 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa724038790 msgr2=0x7fa72403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa724038790 0x7fa72403ac40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa738034000 tx=0x7fa738037000 comp rx=0 tx=0).stop 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 10 monclient: shutdown 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 msgr2=0x7fa740133c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:29.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c5e0 tx=0x7fa73c004960 comp rx=0 tx=0).stop 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa7464fa640 1 -- 172.21.15.17:0/2624751596 reap_dead start 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 shutdown_connections 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 --2- 172.21.15.17:0/2624751596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74006eb30 0x7fa740133c20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 >> 172.21.15.17:0/2624751596 conn(0x7fa7400712f0 msgr2=0x7fa740134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.371+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 shutdown_connections 2024-09-15T02:03:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:29.372+0000 7fa747f5d640 1 -- 172.21.15.17:0/2624751596 wait complete. 2024-09-15T02:03:29.396 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:03:29.396 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T02:03:29.397 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-15T02:03:29.397 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T02:03:29.397 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:03:29.397 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:03:29.397 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:03:29.412 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:03:29.412 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:03:29.412 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:03:29.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:03:49.413 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:03:49.414 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:03:49.414 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:03:49.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:03:49.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 10 monclient: get_monmap_and_config 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 10 monclient: build_initial_monmap 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 10 monclient: monmap: 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:49.522973+0000 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:49.522973+0000 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:49.525 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:49.525 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:49.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 1 Processor -- start 2024-09-15T02:03:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 1 -- start start 2024-09-15T02:03:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.521+0000 7f0898d54640 10 monclient: init 2024-09-15T02:03:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f089409c4f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-a con 0x7f089409c8c0 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f089406eb30 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089406a390 con 0x7f089406eb30 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089409f790 con 0x7f089409c4f0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0894099f50 con 0x7f089409c8c0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0898d54640 10 monclient(hunting): authenticate will time out at 1489.519525s 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f089259c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f089409a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38126/0 (socket says 172.21.15.17:38126) 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0892d9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f089406eb30 0x7f089406ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34214/0 (socket says 172.21.15.17:34214) 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 learned_addr learned my addr 172.21.15.17:0/1167314843 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f089409c8c0 auth_method 0 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.522+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.524+0000 7f0892d9d640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.17:6789/0 conn(0x7f089406eb30 legacy=0x7f089406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f089409a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f089409c8c0 auth_method 0 2024-09-15T02:03:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:03:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:49.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:49.723+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f089409a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f089409c8c0 auth_method 0 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:50.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.124+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f089409a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f089409c8c0 auth_method 0 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:50.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:03:50.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:50.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:50.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:50.925+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:03:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): tick 2024-09-15T02:03:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 monclient(hunting): continuing hunt 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 msgr2=0x7f089409e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f089409c8c0 0x7f089409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f089409a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.17:6789/0 conn(0x7f089406eb30 legacy=0x7f089406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient: _add_conns ranks=[4,2] 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f0874003700 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f08740061e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0894099f50 con 0x7f0874003700 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874008dc0 con 0x7f08740061e0 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.523+0000 7f088bfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 msgr2=0x7f0874005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:52.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:52.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.524+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 msgr2=0x7f0874005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:52.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:52.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:52.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:52.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:52.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:52.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:52.725+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 msgr2=0x7f0874005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:53.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:53.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:53.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.126+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:03:53.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 msgr2=0x7f0874005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:53.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:53.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:03:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:03:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:03:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:53.927+0000 7f0891d9b640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:03:55.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): tick 2024-09-15T02:03:55.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 monclient(hunting): continuing hunt 2024-09-15T02:03:55.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:03:55.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 msgr2=0x7f0874005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.67:3300/0 conn(0x7f0874003700 0x7f0874005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 1 -- 172.21.15.17:0/1167314843 reap_dead start 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f08740061e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f089409c4f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f0874003b40 0x7f087400c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f0874003b40 addr v2:172.21.15.17:3300/0 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089409b0a0 con 0x7f089409c4f0 2024-09-15T02:03:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0894099f50 con 0x7f0874003b40 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 1 -- 172.21.15.17:0/1167314843 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874008dc0 con 0x7f08740061e0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f088bfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0891d9b640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f087400b270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f0874003b40 0x7f087400c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): get_auth_request con 0x7f0874003b40 auth_method 0 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f0892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient(hunting): handle_auth_done global_id 164141 payload 274 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f0874003b40 msgr2=0x7f087400c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f0874003b40 0x7f087400c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> v1:172.21.15.67:6789/0 conn(0x7f089409c4f0 legacy=0x7f087400b270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient: _finish_hunting 0 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 monclient: found mon.noname-b 2024-09-15T02:03:55.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:55.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.524+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089409f790 con 0x7f08740061e0 2024-09-15T02:03:55.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.527+0000 7f089259c640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.527+0000 7f089259c640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0884003c70 tx=0x7f0884007a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31086e727cf53a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.527+0000 7f089159a640 1 -- 172.21.15.17:0/1167314843 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 1 -- 172.21.15.17:0/1167314843 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088402bce0 con 0x7f08740061e0 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 10 monclient: dump: 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 10 monclient: _finish_auth 0 2024-09-15T02:03:55.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f089159a640 1 -- 172.21.15.17:0/1167314843 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088402be60 con 0x7f08740061e0 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.529+0000 7f0898d54640 5 monclient: authenticate success, global_id 164141 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f089159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f089159a640 1 -- 172.21.15.17:0/1167314843 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884033660 con 0x7f08740061e0 2024-09-15T02:03:55.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f089159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 4 set_mon_vals no callback set 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f089159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f089159a640 10 monclient: dump: 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.535 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 10 monclient: shutdown 2024-09-15T02:03:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 -- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f0874008690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0884003c70 tx=0x7f0884007a10 comp rx=0 tx=0).stop 2024-09-15T02:03:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 -- 172.21.15.17:0/1167314843 shutdown_connections 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 --2- 172.21.15.17:0/1167314843 >> v2:172.21.15.17:3300/0 conn(0x7f0874003b40 0x7f087400c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 --2- 172.21.15.17:0/1167314843 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f0874008690 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.530+0000 7f0898d54640 1 -- 172.21.15.17:0/1167314843 >> 172.21.15.17:0/1167314843 conn(0x7f0894071300 msgr2=0x7f08940a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.531+0000 7f0898d54640 1 -- 172.21.15.17:0/1167314843 shutdown_connections 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.531+0000 7f0898d54640 1 -- 172.21.15.17:0/1167314843 wait complete. 2024-09-15T02:03:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.531+0000 7f0898d54640 10 monclient: build_initial_monmap 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.531+0000 7f0898d54640 10 monclient: monmap: 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:55.542 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:55.543 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 Processor -- start 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 -- start start 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient: init 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:55.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-a con 0x7f0874003b40 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-c con 0x7f08740061e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 0x7f0894140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): picked mon.noname-b con 0x7f089409c4f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089409e600 con 0x7f0874003b40 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941220b0 con 0x7f089409c4f0 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0894122670 con 0x7f08740061e0 2024-09-15T02:03:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0898d54640 10 monclient(hunting): authenticate will time out at 1495.528644s 2024-09-15T02:03:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0892d9d640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 msgr2=0x7f0894140bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0892d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 0x7f0894140bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35804/0 (socket says 172.21.15.17:35804) 2024-09-15T02:03:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55068/0 (socket says 172.21.15.17:55068) 2024-09-15T02:03:55.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 1 -- 172.21.15.17:0/3204059364 learned_addr learned my addr 172.21.15.17:0/3204059364 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 10 monclient(hunting): get_auth_request con 0x7f0874003b40 auth_method 0 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f089259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 10 monclient(hunting): get_auth_request con 0x7f08740061e0 auth_method 0 2024-09-15T02:03:55.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.532+0000 7f0891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f089259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient(hunting): handle_auth_done global_id 164144 payload 824 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 -- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 msgr2=0x7f0894140bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 0x7f0894140bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 -- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 msgr2=0x7f089413d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 monclient: found mon.noname-c 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f0891d9b640 1 -- 172.21.15.17:0/3204059364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884007cf0 con 0x7f08740061e0 2024-09-15T02:03:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.533+0000 7f089259c640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:55.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.535+0000 7f0891d9b640 1 -- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f089413cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.535+0000 7f0891d9b640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f087c006f60 tx=0x7f087c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c01e070 con 0x7f08740061e0 2024-09-15T02:03:55.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:03:55.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: dump: 2024-09-15T02:03:55.558 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.558 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.559 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c00c040 con 0x7f08740061e0 2024-09-15T02:03:55.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0195f0 con 0x7f08740061e0 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 5 monclient: authenticate success, global_id 164144 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 10 monclient: _renew_subs 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089413e1a0 con 0x7f08740061e0 2024-09-15T02:03:55.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0890d99640 4 set_mon_vals no callback set 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f088a7fc640 10 monclient: dump: 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.562 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.563 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.564 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 10 monclient: _renew_subs 2024-09-15T02:03:55.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.536+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089411fc60 con 0x7f08740061e0 2024-09-15T02:03:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f087c022620 con 0x7f08740061e0 2024-09-15T02:03:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f088a7fc640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 0x7f087403bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f087c04d780 con 0x7f08740061e0 2024-09-15T02:03:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f0898d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:55.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f0898d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:55.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f0898d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08941222f0 con 0x7f08740061e0 2024-09-15T02:03:55.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f089259c640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 0x7f087403bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.537+0000 7f089259c640 10 monclient: get_auth_request con 0x7f0874039940 auth_method 0 2024-09-15T02:03:55.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.540+0000 7f089259c640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 0x7f087403bdf0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0884007cc0 tx=0x7f0884035000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.542+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f08941222f0 con 0x7f08740061e0 2024-09-15T02:03:55.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.542+0000 7f088a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:55.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.542+0000 7f088a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:55.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.776+0000 7f0898d54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:03:55.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.776+0000 7f0898d54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:55.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.776+0000 7f0898d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:55.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.776+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f08941222f0 con 0x7f08740061e0 2024-09-15T02:03:55.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.777+0000 7f088a7fc640 1 -- 172.21.15.17:0/3204059364 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f08941222f0 con 0x7f08740061e0 2024-09-15T02:03:55.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.777+0000 7f088a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:03:55.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.777+0000 7f088a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:55.779 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":132,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:03:55.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 msgr2=0x7f087403bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 0x7f087403bdf0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0884007cc0 tx=0x7f0884035000 comp rx=0 tx=0).stop 2024-09-15T02:03:55.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 10 monclient: shutdown 2024-09-15T02:03:55.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:55.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 msgr2=0x7f089413cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f087c006f60 tx=0x7f087c008040 comp rx=0 tx=0).stop 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 shutdown_connections 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0874039940 0x7f087403bdf0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089409c4f0 0x7f0894140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f08740061e0 0x7f089413cd30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 --2- 172.21.15.17:0/3204059364 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0874003b40 0x7f089413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 >> 172.21.15.17:0/3204059364 conn(0x7f0894071300 msgr2=0x7f089406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 shutdown_connections 2024-09-15T02:03:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.779+0000 7f0898d54640 1 -- 172.21.15.17:0/3204059364 wait complete. 2024-09-15T02:03:55.791 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:03:55.791 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:03:55.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:03:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.887+0000 7f85cb4c0640 10 monclient: get_monmap_and_config 2024-09-15T02:03:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.887+0000 7f85cb4c0640 10 monclient: build_initial_monmap 2024-09-15T02:03:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.887+0000 7f85cb4c0640 10 monclient: monmap: 2024-09-15T02:03:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:55.889 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:55.888113+0000 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:55.888113+0000 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.887+0000 7f85cb4c0640 1 Processor -- start 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 1 -- start start 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient: init 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85c40a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85c4132040 0x7f85c40987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f85c4132040 addr v2:172.21.15.67:3300/0 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85c4136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c40a01e0 con 0x7f85c40a58b0 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c40a1580 con 0x7f85c4136dd0 2024-09-15T02:03:55.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c40a5c80 con 0x7f85c4132040 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85cb4c0640 10 monclient(hunting): authenticate will time out at 1495.884723s 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85c9a5d640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f85c4136dd0 legacy=0x7f85c41391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85c8a5b640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f85c4132040 msgr2=0x7f85c40987b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85c925c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f85c40a58b0 0x7f85c40980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60348/0 (socket says 172.21.15.17:60348) 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85c925c640 1 -- 172.21.15.17:0/89236404 learned_addr learned my addr 172.21.15.17:0/89236404 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:03:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.888+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/89236404 >> v2:172.21.15.67:3300/0 conn(0x7f85c4132040 0x7f85c40987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 1 -- 172.21.15.17:0/89236404 >> v1:172.21.15.17:6789/0 conn(0x7f85c40a58b0 legacy=0x7f85c40980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326064269 0 0) 0x7f85c40a01e0 con 0x7f85c40a58b0 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): my global_id is 105219 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b00034e0 con 0x7f85c40a58b0 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 836840097 0 0) 0x7f85b00034e0 con 0x7f85c40a58b0 2024-09-15T02:03:55.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 >> v2:172.21.15.67:3300/0 conn(0x7f85c4132040 msgr2=0x7f85c40987b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 --2- 172.21.15.17:0/89236404 >> v2:172.21.15.67:3300/0 conn(0x7f85c4132040 0x7f85c40987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 >> v1:172.21.15.67:6789/0 conn(0x7f85c4136dd0 legacy=0x7f85c41391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c4132440 con 0x7f85c40a58b0 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: _finish_auth 0 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f85c0001720 con 0x7f85c40a58b0 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85cb4c0640 5 monclient: authenticate success, global_id 105219 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85cb4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:55.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: dump: 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.905 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 >> v1:172.21.15.17:6789/0 conn(0x7f85c40a58b0 legacy=0x7f85c40980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b0003a50 0x7f85b0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): picked mon.c con 0x7f85b0003a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 0x7f85b0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): picked mon.a con 0x7f85b0004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b00099c0 0x7f85b000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): picked mon.b con 0x7f85b00099c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c40a5c80 con 0x7f85b0004780 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b000c390 con 0x7f85b00099c0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b000c550 con 0x7f85b0003a50 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 0x7f85b0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85cb4c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c9a5d640 1 -- 172.21.15.17:0/89236404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b00099c0 msgr2=0x7f85b000bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b0003a50 0x7f85b0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): get_auth_request con 0x7f85b0004780 auth_method 0 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c9a5d640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b00099c0 0x7f85b000bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): get_auth_request con 0x7f85b0003a50 auth_method 0 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_done global_id 105219 payload 293 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 -- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b0003a50 msgr2=0x7f85b0003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85b0003a50 0x7f85b0003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 -- 172.21.15.17:0/89236404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b00099c0 msgr2=0x7f85b000bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b00099c0 0x7f85b000bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient: _finish_hunting 0 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 monclient: found mon.a 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c9a5d640 1 -- 172.21.15.17:0/89236404 reap_dead start 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 1 -- 172.21.15.17:0/89236404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b00034e0 con 0x7f85b0004780 2024-09-15T02:03:55.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 10 monclient: _finish_auth 0 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.891+0000 7f85c8a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85c8a5b640 1 -- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 msgr2=0x7f85b0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 0x7f85b0004bf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098f0 tx=0x7f85b4006ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b37ce934cea36b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003b80 con 0x7f85b0004780 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:55.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: dump: 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85cb4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4003ce0 con 0x7f85b0004780 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85cb4c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85cb4c0640 4 set_mon_vals no callback set 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 1 -- 172.21.15.17:0/89236404 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b402aac0 con 0x7f85b0004780 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.894+0000 7f85bbfff640 10 monclient: dump: 2024-09-15T02:03:55.912 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 10 monclient: shutdown 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 -- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 msgr2=0x7f85b0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 0x7f85b0004bf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098f0 tx=0x7f85b4006ed0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 -- 172.21.15.17:0/89236404 shutdown_connections 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/89236404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85b0004780 0x7f85b0004bf0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 -- 172.21.15.17:0/89236404 >> 172.21.15.17:0/89236404 conn(0x7f85c4070ed0 msgr2=0x7f85c40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 -- 172.21.15.17:0/89236404 shutdown_connections 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.895+0000 7f85cb4c0640 1 -- 172.21.15.17:0/89236404 wait complete. 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient: build_initial_monmap 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient: monmap: 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:55.914 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 1 Processor -- start 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 1 -- start start 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient: init 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:03:55.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b0004780 0x7f85c4130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f85b0004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.896+0000 7f85cb4c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85c4131170 0x7f85c41349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f85c4131170 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 0x7f85c4135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f85c4134f00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c411d400 con 0x7f85c4134f00 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c411cd40 con 0x7f85b0004780 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c4098dc0 con 0x7f85c4131170 2024-09-15T02:03:55.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85cb4c0640 10 monclient(hunting): authenticate will time out at 1495.893056s 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c925c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b0004780 msgr2=0x7f85c4130c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 0x7f85c4135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85c4131170 0x7f85c41349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c925c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b0004780 0x7f85c4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85c4131170 0x7f85c41349c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55094/0 (socket says 172.21.15.17:55094) 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 0x7f85c4135350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35818/0 (socket says 172.21.15.17:35818) 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 1 -- 172.21.15.17:0/2185965574 learned_addr learned my addr 172.21.15.17:0/2185965574 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): get_auth_request con 0x7f85c4131170 auth_method 0 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): get_auth_request con 0x7f85c4134f00 auth_method 0 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c8a5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient(hunting): handle_auth_done global_id 105222 payload 824 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85c4131170 msgr2=0x7f85c41349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85c4131170 0x7f85c41349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b0004780 msgr2=0x7f85c4130c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b0004780 0x7f85c4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient: _finish_hunting 0 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 monclient: found mon.noname-a 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.897+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c40a3700 con 0x7f85c4134f00 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 msgr2=0x7f85c4135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85c9a5d640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 0x7f85c4135350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f85c409f780 tx=0x7f85bc0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc00e890 con 0x7f85c4134f00 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: dump: 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.920 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: _finish_auth 0 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 5 monclient: authenticate success, global_id 105222 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc00ea10 con 0x7f85c4134f00 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 10 monclient: _renew_subs 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c4142080 con 0x7f85c4134f00 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc00edd0 con 0x7f85c4134f00 2024-09-15T02:03:55.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85b97fa640 10 monclient: dump: 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85bb7fe640 4 set_mon_vals no callback set 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 10 monclient: _renew_subs 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.899+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c4136620 con 0x7f85c4134f00 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85cb4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 0x7f85c4135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c4063b20 con 0x7f85c402aa40 2024-09-15T02:03:55.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 0x7f85c4135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85bc006360 con 0x7f85c4134f00 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient: get_auth_request con 0x7f85c402aa40 auth_method 0 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.900+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85c925c640 10 monclient(hunting): handle_auth_done global_id 105225 payload 824 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85c925c640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85c402ae10 con 0x7f85c402aa40 2024-09-15T02:03:55.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85b97fa640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a80386f0 0x7f85a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f85bc04c5b0 con 0x7f85c4134f00 2024-09-15T02:03:55.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a80386f0 0x7f85a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.901+0000 7f85c8a5b640 10 monclient: get_auth_request con 0x7f85a80386f0 auth_method 0 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85c925c640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 msgr2=0x7f85c4135890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85c925c640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 0x7f85c4135890 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c4096df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c00119a0 con 0x7f85c402aa40 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 10 monclient: dump: 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.927 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f85c0011b00 con 0x7f85c402aa40 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 msgr2=0x7f85c4135890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 0x7f85c4135890 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c4096df0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 msgr2=0x7f85c4135890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.903+0000 7f85b97fa640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c402aa40 0x7f85c4135890 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c4096df0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.904+0000 7f85c8a5b640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a80386f0 0x7f85a803aba0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f85b4003fd0 tx=0x7f85b4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85cb4c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 0x7f85c405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85cb4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c411d0e0 con 0x7f85c405e310 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 0x7f85c405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient: get_auth_request con 0x7f85c405e310 auth_method 0 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:55.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 10 monclient(hunting): handle_auth_done global_id 105228 payload 824 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.909+0000 7f85c925c640 1 -- 172.21.15.17:0/2185965574 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f85c4135dd0 con 0x7f85c405e310 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85c925c640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 msgr2=0x7f85c405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85c925c640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 0x7f85c405e710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c001ca10 con 0x7f85c405e310 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 10 monclient: dump: 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f85c001cb70 con 0x7f85c405e310 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:55.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 msgr2=0x7f85c405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 0x7f85c405e710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c0015000 comp rx=0 tx=0).stop 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 msgr2=0x7f85c405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85b97fa640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c405e310 0x7f85c405e710 secure :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001900 tx=0x7f85c0015000 comp rx=0 tx=0).stop 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 132, 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:55.932 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:55.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:55.934 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:55.935 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:55.936 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.937 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:55.939 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:55.940 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:55.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:55.943 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.944 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:55.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:55.946 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:55.947 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a80386f0 msgr2=0x7f85a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f85a80386f0 0x7f85a803aba0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f85b4003fd0 tx=0x7f85b4030000 comp rx=0 tx=0).stop 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 10 monclient: shutdown 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 msgr2=0x7f85c4135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:55.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 --2- 172.21.15.17:0/2185965574 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f85c4134f00 0x7f85c4135350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f85c409f780 tx=0x7f85bc0095a0 comp rx=0 tx=0).stop 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 reap_dead start 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85c9a5d640 1 -- 172.21.15.17:0/2185965574 reap_dead start 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 shutdown_connections 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 >> 172.21.15.17:0/2185965574 conn(0x7f85c4070ed0 msgr2=0x7f85c409b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 shutdown_connections 2024-09-15T02:03:55.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:55.912+0000 7f85cb4c0640 1 -- 172.21.15.17:0/2185965574 wait complete. 2024-09-15T02:03:55.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:03:55.949 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:03:55.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:03:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 10 monclient: get_monmap_and_config 2024-09-15T02:03:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 10 monclient: build_initial_monmap 2024-09-15T02:03:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 10 monclient: monmap: 2024-09-15T02:03:56.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:03:56.046106+0000 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:03:56.046106+0000 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 1 Processor -- start 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 1 -- start start 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.045+0000 7fabd199d640 10 monclient: init 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabcc09ce00 addr v1:172.21.15.67:6789/0 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabcc067b70 addr v1:172.21.15.17:6789/0 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc09e1e0 con 0x7fabcc067b70 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc06efb0 con 0x7fabcc09ce00 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc09e4c0 con 0x7fabcc067090 2024-09-15T02:03:56.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabd199d640 10 monclient(hunting): authenticate will time out at 1496.042472s 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabca7fc640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fabcc09ce00 legacy=0x7fabcc067460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55108/0 (socket says 172.21.15.17:55108) 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fabcc067b70 0x7fabcc139010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:60350/0 (socket says 172.21.15.17:60350) 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 -- 172.21.15.17:0/1848127010 learned_addr learned my addr 172.21.15.17:0/1848127010 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): get_auth_request con 0x7fabcc067090 auth_method 0 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 164150 payload 274 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 -- 172.21.15.17:0/1848127010 >> v1:172.21.15.67:6789/0 conn(0x7fabcc09ce00 legacy=0x7fabcc067460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 -- 172.21.15.17:0/1848127010 >> v1:172.21.15.17:6789/0 conn(0x7fabcc067b70 legacy=0x7fabcc139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient: _finish_hunting 0 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 monclient: found mon.noname-b 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:03:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.046+0000 7fabcaffd640 1 -- 172.21.15.17:0/1848127010 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc06f140 con 0x7fabcc067090 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabcaffd640 1 -- 172.21.15.17:0/1848127010 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 msgr2=0x7fabcc132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabcaffd640 1 --2- 172.21.15.17:0/1848127010 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009940 tx=0x7fabbc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6006cc2d3cb07d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 1 -- 172.21.15.17:0/1848127010 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc032440 con 0x7fabcc067090 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: dump: 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 1 -- 172.21.15.17:0/1848127010 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc0325a0 con 0x7fabcc067090 2024-09-15T02:03:56.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabd199d640 5 monclient: authenticate success, global_id 164150 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabd199d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabd199d640 10 monclient: get_monmap_and_config success 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 1 -- 172.21.15.17:0/1848127010 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc032940 con 0x7fabcc067090 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabd199d640 4 set_mon_vals no callback set 2024-09-15T02:03:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.049+0000 7fabc9ffb640 10 monclient: dump: 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 10 monclient: shutdown 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:56.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 -- 172.21.15.17:0/1848127010 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 msgr2=0x7fabcc132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 --2- 172.21.15.17:0/1848127010 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009940 tx=0x7fabbc0091d0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 -- 172.21.15.17:0/1848127010 shutdown_connections 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 --2- 172.21.15.17:0/1848127010 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc067090 0x7fabcc132140 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 -- 172.21.15.17:0/1848127010 >> 172.21.15.17:0/1848127010 conn(0x7fabcc0a1ee0 msgr2=0x7fabcc0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 -- 172.21.15.17:0/1848127010 shutdown_connections 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.050+0000 7fabd199d640 1 -- 172.21.15.17:0/1848127010 wait complete. 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 10 monclient: build_initial_monmap 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 10 monclient: monmap: 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:03:56.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 1 Processor -- start 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 1 -- start start 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 10 monclient: init 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 10 monclient: _reopen_session rank -1 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:03:56.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.051+0000 7fabd199d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabcc067090 0x7fabcc136780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 0x7fabcc136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc067b70 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 0x7fabcc137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc09ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc1388e0 con 0x7fabcc067b70 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc09fe50 con 0x7fabcc067090 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc12fe70 con 0x7fabcc09ce00 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): _renew_subs 2024-09-15T02:03:56.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabd199d640 10 monclient(hunting): authenticate will time out at 1496.048076s 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 0x7fabcc137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 0x7fabcc136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcaffd640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabcc067090 msgr2=0x7fabcc136780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabcc067090 0x7fabcc136780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 0x7fabcc137200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55124/0 (socket says 172.21.15.17:55124) 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 0x7fabcc136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35860/0 (socket says 172.21.15.17:35860) 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 learned_addr learned my addr 172.21.15.17:0/2823428754 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): get_auth_request con 0x7fabcc09ce00 auth_method 0 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): get_auth_request con 0x7fabcc067b70 auth_method 0 2024-09-15T02:03:56.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:56.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_done global_id 164153 payload 824 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabcc067090 msgr2=0x7fabcc136780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabcc067090 0x7fabcc136780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 msgr2=0x7fabcc136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 0x7fabcc136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 monclient: found mon.noname-c 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:03:56.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc009450 con 0x7fabcc09ce00 2024-09-15T02:03:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.052+0000 7fabca7fc640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fabcc067b70 0x7fabcc136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:03:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.054+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 msgr2=0x7fabcc137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.054+0000 7fabcb7fe640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 0x7fabcc137200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fabc4007fd0 tx=0x7fabc4004ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:56.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.054+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400ce40 con 0x7fabcc09ce00 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: dump: 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.072 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400a640 con 0x7fabcc09ce00 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:03:56.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabd199d640 5 monclient: authenticate success, global_id 164153 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400aa00 con 0x7fabcc09ce00 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabb2ffd640 10 monclient: dump: 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.074 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 monclient: _renew_subs 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc138360 con 0x7fabcc09ce00 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.056+0000 7fabc8ff9640 4 set_mon_vals no callback set 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 monclient: _renew_subs 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc130820 con 0x7fabcc09ce00 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 0x7fabcc132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc118930 con 0x7fabcc137740 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fabc40070f0 con 0x7fabcc09ce00 2024-09-15T02:03:56.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabcaffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 0x7fabcc132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabcaffd640 10 monclient: get_auth_request con 0x7fabcc137740 auth_method 0 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.057+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabb2ffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fabb8038770 0x7fabb803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fabc404c2c0 con 0x7fabcc09ce00 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 164156 payload 824 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabcaffd640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabcc0724a0 con 0x7fabcc137740 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabca7fc640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fabb8038770 0x7fabb803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.058+0000 7fabca7fc640 10 monclient: get_auth_request con 0x7fabb8038770 auth_method 0 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabcaffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 msgr2=0x7fabcc132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabcaffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 0x7fabcc132b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fabcc09c970 tx=0x7fabbc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc032440 con 0x7fabcc137740 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 10 monclient: dump: 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.077 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fabbc032710 con 0x7fabcc137740 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 msgr2=0x7fabcc132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 0x7fabcc132b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fabcc09c970 tx=0x7fabbc032000 comp rx=0 tx=0).stop 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 msgr2=0x7fabcc132b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.060+0000 7fabb2ffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc137740 0x7fabcc132b40 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fabcc09c970 tx=0x7fabbc032000 comp rx=0 tx=0).stop 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.061+0000 7fabca7fc640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fabb8038770 0x7fabb803ac20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fabcc137f60 tx=0x7fabc00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabd199d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:03:56.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabd199d640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 0x7fabcc133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabd199d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc118b50 con 0x7fabcc133080 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 0x7fabcc133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient: get_auth_request con 0x7fabcc133080 auth_method 0 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 164159 payload 824 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.066+0000 7fabcaffd640 1 -- 172.21.15.17:0/2823428754 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fabcc137b10 con 0x7fabcc133080 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabcaffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 msgr2=0x7fabcc133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:03:56.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabcaffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 0x7fabcc133480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02afd0 tx=0x7fabbc03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc04a890 con 0x7fabcc133080 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 10 monclient: dump: 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:03:56.080 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7fabbc048440 con 0x7fabcc133080 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 msgr2=0x7fabcc133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 0x7fabcc133480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02afd0 tx=0x7fabbc03d000 comp rx=0 tx=0).stop 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 msgr2=0x7fabcc133480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabb2ffd640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc133080 0x7fabcc133480 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02afd0 tx=0x7fabbc03d000 comp rx=0 tx=0).stop 2024-09-15T02:03:56.081 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 132, 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:03:56.082 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:03:56.083 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:03:56.084 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:03:56.085 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:56.086 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:03:56.089 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:03:56.090 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:03:56.093 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:03:56.094 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:03:56.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fabb8038770 msgr2=0x7fabb803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fabb8038770 0x7fabb803ac20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fabcc137f60 tx=0x7fabc00090f0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 10 monclient: shutdown 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 msgr2=0x7fabcc137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 1 --2- 172.21.15.17:0/2823428754 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fabcc09ce00 0x7fabcc137200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fabc4007fd0 tx=0x7fabc4004ef0 comp rx=0 tx=0).stop 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 reap_dead start 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabcb7fe640 1 -- 172.21.15.17:0/2823428754 reap_dead start 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.069+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 shutdown_connections 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.070+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 >> 172.21.15.17:0/2823428754 conn(0x7fabcc0a1ee0 msgr2=0x7fabcc131400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.070+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 shutdown_connections 2024-09-15T02:03:56.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:03:56.070+0000 7fabd199d640 1 -- 172.21.15.17:0/2823428754 wait complete. 2024-09-15T02:03:56.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:03:56.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:04:16.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:04:16.098 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:04:16.098 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:04:16.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:04:16.101 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:04:16.101 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:04:16.101 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:04:16.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:04:36.102 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:04:36.103 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:04:36.103 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:04:36.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:04:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 10 monclient: get_monmap_and_config 2024-09-15T02:04:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 10 monclient: build_initial_monmap 2024-09-15T02:04:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 10 monclient: monmap: 2024-09-15T02:04:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:36.211907+0000 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:36.211907+0000 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 1 Processor -- start 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 1 -- start start 2024-09-15T02:04:36.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.211+0000 7f23d4914640 10 monclient: init 2024-09-15T02:04:36.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:36.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T02:04:36.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-a con 0x7f23d0131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-b con 0x7f23d00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23d009f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d00a0040 con 0x7f23d009f1e0 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d009bff0 con 0x7f23d0131b20 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00a14a0 con 0x7f23d00a0280 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23d4914640 10 monclient(hunting): authenticate will time out at 1536.208459s 2024-09-15T02:04:36.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ced9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f23d009f1e0 0x7f23d0068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:41872/0 (socket says 172.21.15.17:41872) 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ced9d640 1 -- 172.21.15.17:0/3943562991 learned_addr learned my addr 172.21.15.17:0/3943562991 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 10 monclient(hunting): get_auth_request con 0x7f23d0131b20 auth_method 0 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request con 0x7f23d00a0280 auth_method 0 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 msgr2=0x7f23d009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 msgr2=0x7f23d009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:36.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.212+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:36.219 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.214+0000 7f23ced9d640 1 -- 172.21.15.17:0/3943562991 >> v1:172.21.15.17:6789/0 conn(0x7f23d009f1e0 legacy=0x7f23d0068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request con 0x7f23d00a0280 auth_method 0 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 10 monclient(hunting): get_auth_request con 0x7f23d0131b20 auth_method 0 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 msgr2=0x7f23d009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:04:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 msgr2=0x7f23d009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:04:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.413+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:04:36.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.498+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948327412 0 0) 0x7f23d00a0040 con 0x7f23d009f1e0 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.498+0000 7f23cd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.498+0000 7f23cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.498+0000 7f23cd59a640 10 monclient(hunting): my global_id is 105234 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23bc0034e0 con 0x7f23d009f1e0 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713951055 0 0) 0x7f23bc0034e0 con 0x7f23d009f1e0 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 msgr2=0x7f23d009eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 --2- 172.21.15.17:0/3943562991 >> v2:172.21.15.17:3300/0 conn(0x7f23d0131b20 0x7f23d009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: _finish_hunting 0 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0132430 con 0x7f23d009f1e0 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: _finish_auth 0 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f23c8001720 con 0x7f23d009f1e0 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23d4914640 5 monclient: authenticate success, global_id 105234 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23d4914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: dump: 2024-09-15T02:04:36.505 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 >> v1:172.21.15.17:6789/0 conn(0x7f23d009f1e0 legacy=0x7f23d0068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23bc003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): picked mon.c con 0x7f23d00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc004150 0x7f23bc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): picked mon.b con 0x7f23bc004150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc004d30 0x7f23bc0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): picked mon.a con 0x7f23bc004d30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d009bff0 con 0x7f23bc004d30 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00a14a0 con 0x7f23bc004150 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc00bd50 con 0x7f23d00a0280 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:36.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23cd59a640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.499+0000 7f23d4914640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23bc003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc004d30 0x7f23bc0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): get_auth_request con 0x7f23d00a0280 auth_method 0 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): get_auth_request con 0x7f23bc004d30 auth_method 0 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc004150 0x7f23bc004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request con 0x7f23bc004150 auth_method 0 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:04:36.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient(hunting): handle_auth_done global_id 105234 payload 293 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc004150 msgr2=0x7f23bc004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc004150 0x7f23bc004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc004d30 msgr2=0x7f23bc0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc004d30 0x7f23bc0051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient: _finish_hunting 0 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 monclient: found mon.c 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0034e0 con 0x7f23d00a0280 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 10 monclient: _finish_auth 0 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:36.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.500+0000 7f23ced9d640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23bc004d30 0x7f23bc0051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:36.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.503+0000 7f23ce59c640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23bc003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.503+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23bc003ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23c00074d0 tx=0x7f23c00097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66717a0fae44e84e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002b9c0 con 0x7f23d00a0280 2024-09-15T02:04:36.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23ce59c640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23bc003ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23c00074d0 tx=0x7f23c00097d0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f23c0033440 mon_map magic: 0 v1 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 10 monclient: dump: 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.517 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c002bb60 con 0x7f23d00a0280 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 4 set_mon_vals no callback set 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 10 monclient: shutdown 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 1 -- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23bc003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23d4914640 1 --2- 172.21.15.17:0/3943562991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23bc003ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23c00074d0 tx=0x7f23c00097d0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23ced9d640 1 -- 172.21.15.17:0/3943562991 reap_dead start 2024-09-15T02:04:36.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 1 -- 172.21.15.17:0/3943562991 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f23c0033440 con 0x7f23d00a0280 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.504+0000 7f23cd59a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 1 -- 172.21.15.17:0/3943562991 shutdown_connections 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 1 -- 172.21.15.17:0/3943562991 >> 172.21.15.17:0/3943562991 conn(0x7f23d0071270 msgr2=0x7f23d00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 1 -- 172.21.15.17:0/3943562991 shutdown_connections 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 1 -- 172.21.15.17:0/3943562991 wait complete. 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 10 monclient: build_initial_monmap 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.505+0000 7f23d4914640 10 monclient: monmap: 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:36.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 Processor -- start 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 -- start start 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient: init 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-c con 0x7f23d00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-b con 0x7f23d012eca0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 0x7f23d0098fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): picked mon.noname-a con 0x7f23d0098b50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d009b0e0 con 0x7f23d0098b50 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d009abb0 con 0x7f23d012eca0 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00a09b0 con 0x7f23d00a0280 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23d4914640 10 monclient(hunting): authenticate will time out at 1536.502631s 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 0x7f23d0098fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37554/0 (socket says 172.21.15.17:37554) 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 1 -- 172.21.15.17:0/2457039031 learned_addr learned my addr 172.21.15.17:0/2457039031 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): get_auth_request con 0x7f23d00a0280 auth_method 0 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): get_auth_request con 0x7f23d0098b50 auth_method 0 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.506+0000 7f23ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request con 0x7f23d012eca0 auth_method 0 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 10 monclient(hunting): handle_auth_done global_id 164162 payload 824 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 msgr2=0x7f23d012f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 msgr2=0x7f23d012f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 msgr2=0x7f23d012f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:36.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 msgr2=0x7f23d0098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 0x7f23d0098fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 10 monclient: _finish_hunting 0 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 monclient: found mon.noname-c 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ce59c640 1 -- 172.21.15.17:0/2457039031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00a4a20 con 0x7f23d00a0280 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.507+0000 7f23ced9d640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 0x7f23d0098fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23ce59c640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23ce59c640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c00074d0 tx=0x7f23c0033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002b9c0 con 0x7f23d00a0280 2024-09-15T02:04:36.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: dump: 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: _finish_auth 0 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c003f660 con 0x7f23d00a0280 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23d4914640 5 monclient: authenticate success, global_id 164162 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23d4914640 10 monclient: _renew_subs 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23d4914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d011bdc0 con 0x7f23d00a0280 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c003fa00 con 0x7f23d00a0280 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23c67fc640 10 monclient: dump: 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.526 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.509+0000 7f23ccd99640 4 set_mon_vals no callback set 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.510+0000 7f23d4914640 10 monclient: _renew_subs 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.510+0000 7f23d4914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.510+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d009b270 con 0x7f23d00a0280 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f23c0003040 con 0x7f23d00a0280 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23d4914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23d4914640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23d4914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d00724a0 con 0x7f23d00a0280 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23c67fc640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 0x7f23b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f23c006d2d0 con 0x7f23d00a0280 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 0x7f23b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.511+0000 7f23cdd9b640 10 monclient: get_auth_request con 0x7f23b0038740 auth_method 0 2024-09-15T02:04:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.514+0000 7f23cdd9b640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 0x7f23b003abf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f23d009a150 tx=0x7f23b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.516+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f23c0031340 con 0x7f23d00a0280 2024-09-15T02:04:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.516+0000 7f23c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:36.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.516+0000 7f23c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23d4914640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:04:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23d4914640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:04:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23d4914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f23d006efc0 con 0x7f23d00a0280 2024-09-15T02:04:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23c67fc640 1 -- 172.21.15.17:0/2457039031 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f23c0038080 con 0x7f23d00a0280 2024-09-15T02:04:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:04:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.748+0000 7f23c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:36.751 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":140,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:04:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 msgr2=0x7f23b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 0x7f23b003abf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f23d009a150 tx=0x7f23b8008040 comp rx=0 tx=0).stop 2024-09-15T02:04:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 10 monclient: shutdown 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 msgr2=0x7f23d0140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c00074d0 tx=0x7f23c0033000 comp rx=0 tx=0).stop 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 shutdown_connections 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f23b0038740 0x7f23b003abf0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f23d0098b50 0x7f23d0098fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d012eca0 0x7f23d012f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 --2- 172.21.15.17:0/2457039031 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23d00a0280 0x7f23d0140fa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 >> 172.21.15.17:0/2457039031 conn(0x7f23d0071270 msgr2=0x7f23d00a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 shutdown_connections 2024-09-15T02:04:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.750+0000 7f23d4914640 1 -- 172.21.15.17:0/2457039031 wait complete. 2024-09-15T02:04:36.763 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:04:36.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:04:36.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:04:36.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.858+0000 7fe8970dd640 10 monclient: get_monmap_and_config 2024-09-15T02:04:36.860 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.859+0000 7fe8970dd640 10 monclient: build_initial_monmap 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.859+0000 7fe8970dd640 10 monclient: monmap: 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:36.859845+0000 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:36.859845+0000 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:36.861 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:36.862 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:36.862 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:36.862 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:36.862 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.859+0000 7fe8970dd640 1 Processor -- start 2024-09-15T02:04:36.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.859+0000 7fe8970dd640 1 -- start start 2024-09-15T02:04:36.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.859+0000 7fe8970dd640 10 monclient: init 2024-09-15T02:04:36.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe89009ec30 addr v1:172.21.15.67:6789/0 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe89009f000 addr v2:172.21.15.67:3300/0 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe890136e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe890132460 con 0x7fe89009ec30 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89009c6f0 con 0x7fe89009f000 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89006a650 con 0x7fe890136e50 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe8970dd640 10 monclient(hunting): authenticate will time out at 1536.856286s 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37556/0 (socket says 172.21.15.17:37556) 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 -- 172.21.15.17:0/3813611725 learned_addr learned my addr 172.21.15.17:0/3813611725 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): get_auth_request con 0x7fe890136e50 auth_method 0 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.866 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 1 --2- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 10 monclient(hunting): get_auth_request con 0x7fe89009f000 auth_method 0 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 1 -- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 msgr2=0x7fe890069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient(hunting): handle_auth_done global_id 164165 payload 274 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 1 -- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 msgr2=0x7fe890069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 1 --2- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 -- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 msgr2=0x7fe890069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe88ffff640 1 --2- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 --2- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 -- 172.21.15.17:0/3813611725 >> v1:172.21.15.67:6789/0 conn(0x7fe89009ec30 legacy=0x7fe890131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient: _finish_hunting 0 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 monclient: found mon.noname-b 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.860+0000 7fe89567a640 1 -- 172.21.15.17:0/3813611725 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900a13a0 con 0x7fe890136e50 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe89567a640 1 -- 172.21.15.17:0/3813611725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 msgr2=0x7fe890139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe89567a640 1 --2- 172.21.15.17:0/3813611725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe878009910 tx=0x7fe8780091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=761146807813e3ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 1 -- 172.21.15.17:0/3813611725 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87802aea0 con 0x7fe890136e50 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: dump: 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 1 -- 172.21.15.17:0/3813611725 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe878032440 con 0x7fe890136e50 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe8970dd640 5 monclient: authenticate success, global_id 164165 2024-09-15T02:04:36.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe8970dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe8970dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe8970dd640 4 set_mon_vals no callback set 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 1 -- 172.21.15.17:0/3813611725 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878032820 con 0x7fe890136e50 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.863+0000 7fe88f7fe640 10 monclient: dump: 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.878 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 10 monclient: shutdown 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 -- 172.21.15.17:0/3813611725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 msgr2=0x7fe890139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 --2- 172.21.15.17:0/3813611725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe878009910 tx=0x7fe8780091d0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 -- 172.21.15.17:0/3813611725 shutdown_connections 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 --2- 172.21.15.17:0/3813611725 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe890136e50 0x7fe890139230 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 --2- 172.21.15.17:0/3813611725 >> v2:172.21.15.67:3300/0 conn(0x7fe89009f000 0x7fe890069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 -- 172.21.15.17:0/3813611725 >> 172.21.15.17:0/3813611725 conn(0x7fe890071320 msgr2=0x7fe8900a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 -- 172.21.15.17:0/3813611725 shutdown_connections 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 1 -- 172.21.15.17:0/3813611725 wait complete. 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 10 monclient: build_initial_monmap 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.864+0000 7fe8970dd640 10 monclient: monmap: 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:36.880 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 Processor -- start 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 -- start start 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient: init 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 0x7fe890097690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe89009ec30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 0x7fe890097bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe89009f000 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890136e50 0x7fe890098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe890136e50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89009bec0 con 0x7fe890136e50 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900a2a10 con 0x7fe89009f000 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89009ade0 con 0x7fe89009ec30 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.865+0000 7fe8970dd640 10 monclient(hunting): authenticate will time out at 1536.861998s 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 0x7fe890097690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 0x7fe890097690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37564/0 (socket says 172.21.15.17:37564) 2024-09-15T02:04:36.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 -- 172.21.15.17:0/2490733677 learned_addr learned my addr 172.21.15.17:0/2490733677 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): get_auth_request con 0x7fe89009ec30 auth_method 0 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890136e50 0x7fe890098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): get_auth_request con 0x7fe890136e50 auth_method 0 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 0x7fe890097bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 10 monclient(hunting): get_auth_request con 0x7fe89009f000 auth_method 0 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe89567a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 msgr2=0x7fe890097bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 msgr2=0x7fe890097bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 0x7fe890097bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:36.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 0x7fe890097bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient(hunting): handle_auth_done global_id 164168 payload 824 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 msgr2=0x7fe890097bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89009f000 0x7fe890097bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890136e50 msgr2=0x7fe890098110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890136e50 0x7fe890098110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient: _finish_hunting 0 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 monclient: found mon.noname-c 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.866+0000 7fe894e79640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe878009450 con 0x7fe89009ec30 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe894e79640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 msgr2=0x7fe890097690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe894e79640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 0x7fe890097690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c610 tx=0x7fe888007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888012840 con 0x7fe89009ec30 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: dump: 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: _finish_auth 0 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 5 monclient: authenticate success, global_id 164168 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880129e0 con 0x7fe89009ec30 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 10 monclient: _renew_subs 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89009b0c0 con 0x7fe89009ec30 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888012da0 con 0x7fe89009ec30 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88cff9640 10 monclient: dump: 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.887 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 10 monclient: _renew_subs 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89012f8c0 con 0x7fe89009ec30 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.868+0000 7fe88effd640 4 set_mon_vals no callback set 2024-09-15T02:04:36.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe8970dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe8970dd640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 0x7fe89002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe890063b20 con 0x7fe890098650 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe888011590 con 0x7fe89009ec30 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 0x7fe89002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient: get_auth_request con 0x7fe890098650 auth_method 0 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 10 monclient(hunting): handle_auth_done global_id 105243 payload 824 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8900724a0 con 0x7fe890098650 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88cff9640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe870038740 0x7fe87003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.869+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe88804bea0 con 0x7fe89009ec30 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.870+0000 7fe89567a640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe870038740 0x7fe87003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.870+0000 7fe89567a640 10 monclient: get_auth_request con 0x7fe870038740 auth_method 0 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 msgr2=0x7fe89002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 0x7fe89002aa40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe880009590 tx=0x7fe88000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800145e0 con 0x7fe890098650 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 10 monclient: dump: 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.890 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe880014740 con 0x7fe890098650 2024-09-15T02:04:36.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 msgr2=0x7fe89002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 0x7fe89002aa40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe880009590 tx=0x7fe88000b040 comp rx=0 tx=0).stop 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 msgr2=0x7fe89002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe88cff9640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe890098650 0x7fe89002aa40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.872+0000 7fe89567a640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe870038740 0x7fe87003abf0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe8900607f0 tx=0x7fe878032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe8970dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe8970dd640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 0x7fe89005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe8970dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89006f0e0 con 0x7fe89005e310 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 0x7fe89005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient: get_auth_request con 0x7fe89005e310 auth_method 0 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:36.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 10 monclient(hunting): handle_auth_done global_id 105246 payload 824 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.878+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe890134010 con 0x7fe89005e310 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.880+0000 7fe88ffff640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 msgr2=0x7fe89005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.880+0000 7fe88ffff640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 0x7fe89005e710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe88001a000 tx=0x7fe88001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880022910 con 0x7fe89005e310 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 10 monclient: dump: 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fe8800143d0 con 0x7fe89005e310 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 msgr2=0x7fe89005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 0x7fe89005e710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe88001a000 tx=0x7fe88001d000 comp rx=0 tx=0).stop 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 msgr2=0x7fe89005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:36.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe88cff9640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe89005e310 0x7fe89005e710 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe88001a000 tx=0x7fe88001d000 comp rx=0 tx=0).stop 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:36.913 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe870038740 msgr2=0x7fe87003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe870038740 0x7fe87003abf0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe8900607f0 tx=0x7fe878032000 comp rx=0 tx=0).stop 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 10 monclient: shutdown 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 msgr2=0x7fe890097690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 --2- 172.21.15.17:0/2490733677 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe89009ec30 0x7fe890097690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c610 tx=0x7fe888007ed0 comp rx=0 tx=0).stop 2024-09-15T02:04:36.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe89567a640 1 -- 172.21.15.17:0/2490733677 reap_dead start 2024-09-15T02:04:36.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe89567a640 1 -- 172.21.15.17:0/2490733677 reap_dead start 2024-09-15T02:04:36.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 shutdown_connections 2024-09-15T02:04:36.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 >> 172.21.15.17:0/2490733677 conn(0x7fe890071320 msgr2=0x7fe89009d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:36.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 shutdown_connections 2024-09-15T02:04:36.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:36.881+0000 7fe8970dd640 1 -- 172.21.15.17:0/2490733677 wait complete. 2024-09-15T02:04:36.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:36.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:04:36.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:04:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.011+0000 7fdfb491d640 10 monclient: get_monmap_and_config 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.011+0000 7fdfb491d640 10 monclient: build_initial_monmap 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.011+0000 7fdfb491d640 10 monclient: monmap: 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:37.012524+0000 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:37.012524+0000 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:37.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 Processor -- start 2024-09-15T02:04:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 -- start start 2024-09-15T02:04:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient: init 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 0x7fdfb009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb0131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfb00a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00a0040 con 0x7fdfb00a0280 2024-09-15T02:04:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb009bff0 con 0x7fdfb0131b20 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00a14a0 con 0x7fdfb009f1e0 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.012+0000 7fdfb491d640 10 monclient(hunting): authenticate will time out at 1537.008976s 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 0x7fdfb009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37568/0 (socket says 172.21.15.17:37568) 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 -- 172.21.15.17:0/536386396 learned_addr learned my addr 172.21.15.17:0/536386396 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): get_auth_request con 0x7fdfb009f1e0 auth_method 0 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): get_auth_request con 0x7fdfb0131b20 auth_method 0 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfadd9b640 1 --1- 172.21.15.17:0/536386396 >> v1:172.21.15.17:6789/0 conn(0x7fdfb00a0280 0x7fdfb0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:41882/0 (socket says 172.21.15.17:41882) 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient(hunting): handle_auth_done global_id 164171 payload 274 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 -- 172.21.15.17:0/536386396 >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 msgr2=0x7fdfb009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 --2- 172.21.15.17:0/536386396 >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 0x7fdfb009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 -- 172.21.15.17:0/536386396 >> v1:172.21.15.17:6789/0 conn(0x7fdfb00a0280 legacy=0x7fdfb0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 monclient: found mon.noname-b 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfaed9d640 1 -- 172.21.15.17:0/536386396 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0132600 con 0x7fdfb009f1e0 2024-09-15T02:04:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.013+0000 7fdfae59c640 1 --2- 172.21.15.17:0/536386396 >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 0x7fdfb009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfaed9d640 1 -- 172.21.15.17:0/536386396 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 msgr2=0x7fdfb0138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfaed9d640 1 --2- 172.21.15.17:0/536386396 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009f20 tx=0x7fdfa8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c792dae55679d265 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 1 -- 172.21.15.17:0/536386396 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8007610 con 0x7fdfb009f1e0 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: dump: 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: _finish_auth 0 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 1 -- 172.21.15.17:0/536386396 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa802d050 con 0x7fdfb009f1e0 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 5 monclient: authenticate success, global_id 164171 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 1 -- 172.21.15.17:0/536386396 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa80079a0 con 0x7fdfb009f1e0 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfad59a640 10 monclient: dump: 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 4 set_mon_vals no callback set 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 10 monclient: shutdown 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 1 -- 172.21.15.17:0/536386396 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 msgr2=0x7fdfb0138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.016+0000 7fdfb491d640 1 --2- 172.21.15.17:0/536386396 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009f20 tx=0x7fdfa8009840 comp rx=0 tx=0).stop 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 -- 172.21.15.17:0/536386396 shutdown_connections 2024-09-15T02:04:37.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 --2- 172.21.15.17:0/536386396 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009f1e0 0x7fdfb0138e60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 --2- 172.21.15.17:0/536386396 >> v2:172.21.15.17:3300/0 conn(0x7fdfb0131b20 0x7fdfb009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 -- 172.21.15.17:0/536386396 >> 172.21.15.17:0/536386396 conn(0x7fdfb0071270 msgr2=0x7fdfb00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 -- 172.21.15.17:0/536386396 shutdown_connections 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 -- 172.21.15.17:0/536386396 wait complete. 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 10 monclient: build_initial_monmap 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 10 monclient: monmap: 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:37.031 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 Processor -- start 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 1 -- start start 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.017+0000 7fdfb491d640 10 monclient: init 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 0x7fdfb012d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 0x7fdfb012d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb0131b20 0x7fdfb012de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb0131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0097a10 con 0x7fdfb009f1e0 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00a09b0 con 0x7fdfb0131b20 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0138fb0 con 0x7fdfb00a0280 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfb491d640 10 monclient(hunting): authenticate will time out at 1537.014405s 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 0x7fdfb012d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 0x7fdfb012d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 0x7fdfb012d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37576/0 (socket says 172.21.15.17:37576) 2024-09-15T02:04:37.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 0x7fdfb012d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49016/0 (socket says 172.21.15.17:49016) 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 -- 172.21.15.17:0/1750214991 learned_addr learned my addr 172.21.15.17:0/1750214991 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): get_auth_request con 0x7fdfb00a0280 auth_method 0 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): get_auth_request con 0x7fdfb009f1e0 auth_method 0 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient(hunting): handle_auth_done global_id 164174 payload 824 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb0131b20 msgr2=0x7fdfb012de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfaed9d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb0131b20 0x7fdfb012de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb0131b20 0x7fdfb012de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 msgr2=0x7fdfb012d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 0x7fdfb012d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 monclient: found mon.noname-c 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfadd9b640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa0009390 con 0x7fdfb00a0280 2024-09-15T02:04:37.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.018+0000 7fdfae59c640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdfb009f1e0 0x7fdfb012d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfadd9b640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 msgr2=0x7fdfb012d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfadd9b640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 0x7fdfb012d910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c610 tx=0x7fdfa4007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4007510 con 0x7fdfb00a0280 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: dump: 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.036 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa40057d0 con 0x7fdfb00a0280 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 5 monclient: authenticate success, global_id 164174 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 10 monclient: _renew_subs 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa8009ae0 con 0x7fdfb00a0280 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4007690 con 0x7fdfb00a0280 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdf9e7fc640 10 monclient: dump: 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfacd99640 4 set_mon_vals no callback set 2024-09-15T02:04:37.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 10 monclient: _renew_subs 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.021+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb012ee30 con 0x7fdfb00a0280 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfb491d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfb491d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 0x7fdfb012e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0063a40 con 0x7fdfb012f530 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdfa401b400 con 0x7fdfb00a0280 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 0x7fdfb012e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient: get_auth_request con 0x7fdfb012f530 auth_method 0 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdf9e7fc640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdf90038790 0x7fdf9003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fdfa404c0d0 con 0x7fdfb00a0280 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 10 monclient(hunting): handle_auth_done global_id 164177 payload 824 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfae59c640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfb00724a0 con 0x7fdfb012f530 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfaed9d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdf90038790 0x7fdf9003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.022+0000 7fdfaed9d640 10 monclient: get_auth_request con 0x7fdf90038790 auth_method 0 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdfae59c640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 msgr2=0x7fdfb012e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdfae59c640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 0x7fdfb012e390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0014940 con 0x7fdfb012f530 2024-09-15T02:04:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.024+0000 7fdf9e7fc640 10 monclient: dump: 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdfa0014aa0 con 0x7fdfb012f530 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 msgr2=0x7fdfb012e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 0x7fdfb012e390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa00090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 msgr2=0x7fdfb012e390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdf9e7fc640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb012f530 0x7fdfb012e390 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa00090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.025+0000 7fdfaed9d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdf90038790 0x7fdf9003ac40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00a5010 tx=0x7fdfa8036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.030+0000 7fdfb491d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.030+0000 7fdfb491d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 0x7fdfb009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.030+0000 7fdfb491d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.030+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb006efc0 con 0x7fdfb009cb40 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 0x7fdfb009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient: get_auth_request con 0x7fdfb009cb40 auth_method 0 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 10 monclient(hunting): handle_auth_done global_id 164180 payload 824 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.031+0000 7fdfae59c640 1 -- 172.21.15.17:0/1750214991 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdfb012fa00 con 0x7fdfb009cb40 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdfae59c640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 msgr2=0x7fdfb009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdfae59c640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 0x7fdfb009cf40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa001c8e0 con 0x7fdfb009cb40 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 10 monclient: dump: 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fdfa001ca40 con 0x7fdfb009cb40 2024-09-15T02:04:37.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 msgr2=0x7fdfb009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 0x7fdfb009cf40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa001c000 comp rx=0 tx=0).stop 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 msgr2=0x7fdfb009cf40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.033+0000 7fdf9e7fc640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb009cb40 0x7fdfb009cf40 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0011f20 tx=0x7fdfa001c000 comp rx=0 tx=0).stop 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:37.047 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:37.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:37.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:37.050 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:37.051 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.054 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:37.056 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:37.057 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:37.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:04:37.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:37.060 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:37.060 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdf90038790 msgr2=0x7fdf9003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdf90038790 0x7fdf9003ac40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00a5010 tx=0x7fdfa8036000 comp rx=0 tx=0).stop 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 10 monclient: shutdown 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 msgr2=0x7fdfb012d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 --2- 172.21.15.17:0/1750214991 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdfb00a0280 0x7fdfb012d910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c610 tx=0x7fdfa4007f00 comp rx=0 tx=0).stop 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfaed9d640 1 -- 172.21.15.17:0/1750214991 reap_dead start 2024-09-15T02:04:37.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfaed9d640 1 -- 172.21.15.17:0/1750214991 reap_dead start 2024-09-15T02:04:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 shutdown_connections 2024-09-15T02:04:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 >> 172.21.15.17:0/1750214991 conn(0x7fdfb0071270 msgr2=0x7fdfb00a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 shutdown_connections 2024-09-15T02:04:37.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.034+0000 7fdfb491d640 1 -- 172.21.15.17:0/1750214991 wait complete. 2024-09-15T02:04:37.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:37.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:04:37.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:04:37.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.158+0000 7fa118b1d640 10 monclient: get_monmap_and_config 2024-09-15T02:04:37.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.158+0000 7fa118b1d640 10 monclient: build_initial_monmap 2024-09-15T02:04:37.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.158+0000 7fa118b1d640 10 monclient: monmap: 2024-09-15T02:04:37.160 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:37.160 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:37.159453+0000 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:37.159453+0000 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:37.161 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 Processor -- start 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 -- start start 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient: init 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa114097320 0x7fa1140976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa114097320 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa11409a590 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa11412edb0 addr v2:172.21.15.67:3300/0 2024-09-15T02:04:37.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1140985d0 con 0x7fa11409a590 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11409d200 con 0x7fa114097320 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa114097c30 con 0x7fa11412edb0 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa118b1d640 10 monclient(hunting): authenticate will time out at 1537.155880s 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.159+0000 7fa11259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa114097320 0x7fa1140976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa111d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa11409a590 0x7fa11412e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:41884/0 (socket says 172.21.15.17:41884) 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa114097320 0x7fa1140976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49024/0 (socket says 172.21.15.17:49024) 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa111d9b640 1 -- 172.21.15.17:0/3079468153 learned_addr learned my addr 172.21.15.17:0/3079468153 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): get_auth_request con 0x7fa114097320 auth_method 0 2024-09-15T02:04:37.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 10 monclient(hunting): get_auth_request con 0x7fa11412edb0 auth_method 0 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient(hunting): handle_auth_done global_id 105255 payload 274 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 -- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 msgr2=0x7fa1141369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:37.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 1 -- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 msgr2=0x7fa1141369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 1 -- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 msgr2=0x7fa1141369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 --2- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 -- 172.21.15.17:0/3079468153 >> v1:172.21.15.17:6789/0 conn(0x7fa11409a590 legacy=0x7fa11412e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient: _finish_hunting 0 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 monclient: found mon.noname-a 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:37.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.160+0000 7fa11259c640 1 -- 172.21.15.17:0/3079468153 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140992f0 con 0x7fa114097320 2024-09-15T02:04:37.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11259c640 1 -- 172.21.15.17:0/3079468153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa114097320 msgr2=0x7fa1140976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11259c640 1 --2- 172.21.15.17:0/3079468153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa114097320 0x7fa1140976f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa114096e70 tx=0x7fa1040090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad60aadc9578412 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 1 -- 172.21.15.17:0/3079468153 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10402ae00 con 0x7fa114097320 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: dump: 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.174 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: _finish_auth 0 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 1 -- 172.21.15.17:0/3079468153 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104032440 con 0x7fa114097320 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 1 -- 172.21.15.17:0/3079468153 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040327e0 con 0x7fa114097320 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 5 monclient: authenticate success, global_id 105255 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 4 set_mon_vals no callback set 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa11159a640 10 monclient: dump: 2024-09-15T02:04:37.175 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 10 monclient: shutdown 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 -- 172.21.15.17:0/3079468153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa114097320 msgr2=0x7fa1140976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3079468153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa114097320 0x7fa1140976f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa114096e70 tx=0x7fa1040090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 -- 172.21.15.17:0/3079468153 shutdown_connections 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3079468153 >> v2:172.21.15.67:3300/0 conn(0x7fa11412edb0 0x7fa1141369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3079468153 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa114097320 0x7fa1140976f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.163+0000 7fa118b1d640 1 -- 172.21.15.17:0/3079468153 >> 172.21.15.17:0/3079468153 conn(0x7fa1140a1000 msgr2=0x7fa1140a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 1 -- 172.21.15.17:0/3079468153 shutdown_connections 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 1 -- 172.21.15.17:0/3079468153 wait complete. 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 10 monclient: build_initial_monmap 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 10 monclient: monmap: 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 1 Processor -- start 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 1 -- start start 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.164+0000 7fa118b1d640 10 monclient: init 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 0x7fa114141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa114097320 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa11409a590 0x7fa11412dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa11409a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa11412edb0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11412e860 con 0x7fa11412edb0 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa114122170 con 0x7fa114097320 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140970f0 con 0x7fa11409a590 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa118b1d640 10 monclient(hunting): authenticate will time out at 1537.161419s 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa11409a590 0x7fa11412dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49036/0 (socket says 172.21.15.17:49036) 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 learned_addr learned my addr 172.21.15.17:0/3061663657 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:37.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): get_auth_request con 0x7fa11412edb0 auth_method 0 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): get_auth_request con 0x7fa11409a590 auth_method 0 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 0x7fa114141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 10 monclient(hunting): get_auth_request con 0x7fa114097320 auth_method 0 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa111d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient(hunting): handle_auth_done global_id 105258 payload 824 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa11409a590 msgr2=0x7fa11412dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa11409a590 0x7fa11412dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 msgr2=0x7fa114141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 msgr2=0x7fa114141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 msgr2=0x7fa114141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 0x7fa114141c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 0x7fa114141c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114097320 0x7fa114141c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 monclient: found mon.noname-a 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.165+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104009390 con 0x7fa11412edb0 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 msgr2=0x7fa1140698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa112d9d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009fd0 tx=0x7fa10c009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00ee30 con 0x7fa11412edb0 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: dump: 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: _finish_auth 0 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 5 monclient: authenticate success, global_id 105258 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c013590 con 0x7fa11412edb0 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 10 monclient: _renew_subs 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:37.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa114120f30 con 0x7fa11412edb0 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c013950 con 0x7fa11412edb0 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa1027fc640 10 monclient: dump: 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:37.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa110d99640 4 set_mon_vals no callback set 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 10 monclient: _renew_subs 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.168+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa114142180 con 0x7fa11412edb0 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa10c01e460 con 0x7fa11412edb0 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa118b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa114063c20 con 0x7fa114142900 2024-09-15T02:04:37.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa1027fc640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 0x7fa0f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa10c04cd20 con 0x7fa11412edb0 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa114142900 auth_method 0 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa111d9b640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 0x7fa0f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa111d9b640 10 monclient: get_auth_request con 0x7fa0f4038740 auth_method 0 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.169+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:37.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.172+0000 7fa111d9b640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 0x7fa0f403abf0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa114122940 tx=0x7fa108007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.370+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.370+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.370+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.370+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:37.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.370+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:04:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa114142900 auth_method 0 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:37.771+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa114142900 auth_method 0 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:38.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:38.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:38.573+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:04:40.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.165+0000 7fa1017fa640 10 monclient: tick 2024-09-15T02:04:40.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.165+0000 7fa1017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:40.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.165+0000 7fa1017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:40.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:40.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa114142900 auth_method 0 2024-09-15T02:04:40.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:40.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:40.175+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 10 monclient: tick 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 msgr2=0x7fa1140298c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa114142900 0x7fa1140298c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.165+0000 7fa1017fa640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec006030 con 0x7fa0ec003500 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa0ec003500 auth_method 0 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 msgr2=0x7fa0ec005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 msgr2=0x7fa0ec005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:04:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.166+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:04:41.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa0ec003500 auth_method 0 2024-09-15T02:04:41.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:04:41.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 10 monclient(hunting): handle_auth_done global_id 174100 payload 824 2024-09-15T02:04:41.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.367+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0ec0061a0 con 0x7fa0ec003500 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 msgr2=0x7fa0ec005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa104034000 tx=0x7fa104032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104047460 con 0x7fa0ec003500 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 10 monclient: dump: 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa1040475c0 con 0x7fa0ec003500 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 msgr2=0x7fa0ec005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa104034000 tx=0x7fa104032000 comp rx=0 tx=0).stop 2024-09-15T02:04:41.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 msgr2=0x7fa0ec005980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:41.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.370+0000 7fa1027fc640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ec003500 0x7fa0ec005980 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa104034000 tx=0x7fa104032000 comp rx=0 tx=0).stop 2024-09-15T02:04:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.376+0000 7fa118b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.376+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 0x7fa114029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.376+0000 7fa118b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.376+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140724a0 con 0x7fa11412e240 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 0x7fa114029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient: get_auth_request con 0x7fa11412e240 auth_method 0 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 10 monclient(hunting): handle_auth_done global_id 174103 payload 824 2024-09-15T02:04:41.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.377+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa11409cfe0 con 0x7fa11412e240 2024-09-15T02:04:41.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa11259c640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 msgr2=0x7fa114029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:41.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa11259c640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 0x7fa114029e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa104035040 tx=0x7fa104040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10403bdc0 con 0x7fa11412e240 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 10 monclient: dump: 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fa1040463c0 con 0x7fa11412e240 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 msgr2=0x7fa114029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 0x7fa114029e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa104035040 tx=0x7fa104040040 comp rx=0 tx=0).stop 2024-09-15T02:04:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 msgr2=0x7fa114029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa1027fc640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa11412e240 0x7fa114029e00 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa104035040 tx=0x7fa104040040 comp rx=0 tx=0).stop 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.380+0000 7fa112d9d640 1 -- 172.21.15.17:0/3061663657 reap_dead start 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:41.401 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 msgr2=0x7fa0f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 0x7fa0f403abf0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa114122940 tx=0x7fa108007f00 comp rx=0 tx=0).stop 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 10 monclient: shutdown 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 msgr2=0x7fa1140698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009fd0 tx=0x7fa10c009870 comp rx=0 tx=0).stop 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 shutdown_connections 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa0f4038740 0x7fa0f403abf0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 --2- 172.21.15.17:0/3061663657 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa11412edb0 0x7fa1140698d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 >> 172.21.15.17:0/3061663657 conn(0x7fa1140a1000 msgr2=0x7fa1140a04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:41.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 shutdown_connections 2024-09-15T02:04:41.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.381+0000 7fa118b1d640 1 -- 172.21.15.17:0/3061663657 wait complete. 2024-09-15T02:04:41.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:41.404 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:04:41.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 10 monclient: get_monmap_and_config 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 10 monclient: build_initial_monmap 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 10 monclient: monmap: 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:41.504126+0000 2024-09-15T02:04:41.505 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:41.504126+0000 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 1 Processor -- start 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 1 -- start start 2024-09-15T02:04:41.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.503+0000 7f3f698ef640 10 monclient: init 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f640a58b0 0x7f3f64132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-a con 0x7f3f640a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f640980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3f64136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640a01e0 con 0x7f3f640980a0 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640a1580 con 0x7f3f640a58b0 2024-09-15T02:04:41.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640a5c80 con 0x7f3f64136dd0 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f698ef640 10 monclient(hunting): authenticate will time out at 1541.500449s 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f640a58b0 0x7f3f64132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:37628/0 (socket says 172.21.15.17:37628) 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 1 -- 172.21.15.17:0/2753549744 learned_addr learned my addr 172.21.15.17:0/2753549744 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3f640a58b0 0x7f3f64132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49044/0 (socket says 172.21.15.17:49044) 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f627fc640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3f640980a0 0x7f3f640984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:41890/0 (socket says 172.21.15.17:41890) 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): get_auth_request con 0x7f3f64136dd0 auth_method 0 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): get_auth_request con 0x7f3f640a58b0 auth_method 0 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_done global_id 105261 payload 274 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 msgr2=0x7f3f641391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:41.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2753549744 >> v1:172.21.15.17:6789/0 conn(0x7f3f640980a0 legacy=0x7f3f640984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 monclient: found mon.noname-a 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2753549744 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64098bb0 con 0x7f3f640a58b0 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.504+0000 7f3f637fe640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640a58b0 msgr2=0x7f3f64132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f62ffd640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640a58b0 0x7f3f64132040 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009850 tx=0x7f3f540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50756aa8d786a8fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:41.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 1 -- 172.21.15.17:0/2753549744 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5402ae80 con 0x7f3f640a58b0 2024-09-15T02:04:41.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: dump: 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: _finish_auth 0 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 1 -- 172.21.15.17:0/2753549744 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54032440 con 0x7f3f640a58b0 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f698ef640 5 monclient: authenticate success, global_id 105261 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f698ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f698ef640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 1 -- 172.21.15.17:0/2753549744 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f540327c0 con 0x7f3f640a58b0 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f698ef640 4 set_mon_vals no callback set 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.507+0000 7f3f61ffb640 10 monclient: dump: 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.520 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 10 monclient: shutdown 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 -- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640a58b0 msgr2=0x7f3f64132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640a58b0 0x7f3f64132040 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009850 tx=0x7f3f540090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 -- 172.21.15.17:0/2753549744 shutdown_connections 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2753549744 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640a58b0 0x7f3f64132040 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 -- 172.21.15.17:0/2753549744 >> 172.21.15.17:0/2753549744 conn(0x7f3f64070ed0 msgr2=0x7f3f640a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 -- 172.21.15.17:0/2753549744 shutdown_connections 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.508+0000 7f3f698ef640 1 -- 172.21.15.17:0/2753549744 wait complete. 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient: build_initial_monmap 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient: monmap: 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 Processor -- start 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 -- start start 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient: init 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-a con 0x7f3f640980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:41.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f640a58b0 0x7f3f64133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3f640a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): picked mon.noname-c con 0x7f3f64136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64130e80 con 0x7f3f640980a0 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64135e70 con 0x7f3f640a58b0 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64098dc0 con 0x7f3f64136dd0 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.509+0000 7f3f698ef640 10 monclient(hunting): authenticate will time out at 1541.505995s 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49046/0 (socket says 172.21.15.17:49046) 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2589723567 learned_addr learned my addr 172.21.15.17:0/2589723567 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): get_auth_request con 0x7f3f640980a0 auth_method 0 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): get_auth_request con 0x7f3f64136dd0 auth_method 0 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f640a58b0 0x7f3f64133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): get_auth_request con 0x7f3f640a58b0 auth_method 0 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient(hunting): handle_auth_done global_id 105264 payload 824 2024-09-15T02:04:41.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 msgr2=0x7f3f641337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f640a58b0 msgr2=0x7f3f64133290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f640a58b0 0x7f3f64133290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 monclient: found mon.noname-a 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2589723567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c009ae0 con 0x7f3f640980a0 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.510+0000 7f3f637fe640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f62ffd640 1 -- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 msgr2=0x7f3f64132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f62ffd640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009850 tx=0x7f3f54032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54031c70 con 0x7f3f640980a0 2024-09-15T02:04:41.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: dump: 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: _finish_auth 0 2024-09-15T02:04:41.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54031dd0 con 0x7f3f640980a0 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 5 monclient: authenticate success, global_id 105264 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 10 monclient: _renew_subs 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f54009390 con 0x7f3f640980a0 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5403d650 con 0x7f3f640980a0 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f4affd640 10 monclient: dump: 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:41.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 10 monclient: _renew_subs 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f64131900 con 0x7f3f640980a0 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.512+0000 7f3f617fa640 4 set_mon_vals no callback set 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.513+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f54039040 con 0x7f3f640980a0 2024-09-15T02:04:41.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.513+0000 7f3f698ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.513+0000 7f3f698ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.513+0000 7f3f698ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.513+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f640724a0 con 0x7f3f640980a0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.514+0000 7f3f4affd640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 0x7f3f5003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.514+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3f5406c810 con 0x7f3f640980a0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.514+0000 7f3f627fc640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 0x7f3f5003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.514+0000 7f3f627fc640 10 monclient: get_auth_request con 0x7f3f50038740 auth_method 0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.516+0000 7f3f627fc640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 0x7f3f5003abf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3f6409f2c0 tx=0x7f3f58006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:41.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.518+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f5402f4d0 con 0x7f3f640980a0 2024-09-15T02:04:41.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.518+0000 7f3f4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:41.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.518+0000 7f3f4affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:41.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.743+0000 7f3f698ef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:04:41.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.743+0000 7f3f698ef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:04:41.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.743+0000 7f3f698ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:41.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.743+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3f6409fc90 con 0x7f3f640980a0 2024-09-15T02:04:41.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.744+0000 7f3f4affd640 1 -- 172.21.15.17:0/2589723567 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3f54030b30 con 0x7f3f640980a0 2024-09-15T02:04:41.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.744+0000 7f3f4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:04:41.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.744+0000 7f3f4affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=2396073569,config=3140514009,health=3930795187,kv=1529054990,logm=568597795}) 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=628995506,config=3140514009,health=4127369059,kv=1529054990,logm=2051001760}) 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=2396073569,config=3140514009,health=3930795187,kv=1529054990,logm=568597795}) 2024-09-15T02:04:41.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.745+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=3536567696,config=3140514009,health=4127369059,kv=1529054990,logm=1637975763}) 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261110271}) 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144006621}) 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261110271}) 2024-09-15T02:04:41.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.746+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082984027}) 2024-09-15T02:04:41.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186037991}) 2024-09-15T02:04:41.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2656274498}) 2024-09-15T02:04:41.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186037991}) 2024-09-15T02:04:41.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.747+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931324454}) 2024-09-15T02:04:41.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 msgr2=0x7f3f5003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 0x7f3f5003abf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3f6409f2c0 tx=0x7f3f58006f50 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 10 monclient: shutdown 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 msgr2=0x7f3f64132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009850 tx=0x7f3f54032000 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 shutdown_connections 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3f50038740 0x7f3f5003abf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3f64136dd0 0x7f3f641337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f640a58b0 0x7f3f64133290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 --2- 172.21.15.17:0/2589723567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3f640980a0 0x7f3f64132d50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 >> 172.21.15.17:0/2589723567 conn(0x7f3f64070ed0 msgr2=0x7f3f64068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:41.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 shutdown_connections 2024-09-15T02:04:41.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f3f698ef640 1 -- 172.21.15.17:0/2589723567 wait complete. 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263882533}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2249521465}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263882533}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.748+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545171534}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.749+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.749+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285901630}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.749+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381970484}) 2024-09-15T02:04:41.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.749+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.749+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285901630}) 2024-09-15T02:04:41.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.750+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181518908}) 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218510321}) 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997979348}) 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218510321}) 2024-09-15T02:04:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808918512}) 2024-09-15T02:04:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883852488}) 2024-09-15T02:04:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3847099003}) 2024-09-15T02:04:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883852488}) 2024-09-15T02:04:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952637933}) 2024-09-15T02:04:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966744639}) 2024-09-15T02:04:41.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752175069}) 2024-09-15T02:04:41.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966744639}) 2024-09-15T02:04:41.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000407352}) 2024-09-15T02:04:41.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529745322}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2450412474}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529745322}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057723916}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763656909}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654757555}) 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763656909}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149412437}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216375378}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514552961}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216375378}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016739671}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742238407}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353248215}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742238407}) 2024-09-15T02:04:41.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267433438}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689901603}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777457785}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689901603}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649840872}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267094743}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4255669436}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267094743}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252849934}) 2024-09-15T02:04:41.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726741335}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=232551638}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726741335}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146918275}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901072446}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717089861}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901072446}) 2024-09-15T02:04:41.770 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678806799}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791406018}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307759311}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791406018}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477156409}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1510989530}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478432711}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1510989530}) 2024-09-15T02:04:41.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511044466}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150723050}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50677970}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150723050}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1058213838}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972883345}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5652325}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972883345}) 2024-09-15T02:04:41.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:41.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704793024}) 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577238975}) 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481785880}) 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577238975}) 2024-09-15T02:04:42.121 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.119+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35649318}) 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884771282}) 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488639329}) 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884771282}) 2024-09-15T02:04:42.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1656869148}) 2024-09-15T02:04:42.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557522943}) 2024-09-15T02:04:42.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765654896}) 2024-09-15T02:04:42.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557522943}) 2024-09-15T02:04:42.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.319+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688114165}) 2024-09-15T02:04:42.323 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.323 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995222805}) 2024-09-15T02:04:42.323 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189838418}) 2024-09-15T02:04:42.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995222805}) 2024-09-15T02:04:42.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.321+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2894322811}) 2024-09-15T02:04:42.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194552357}) 2024-09-15T02:04:42.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493534613}) 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194552357}) 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.753+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479260873}) 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760289702}) 2024-09-15T02:04:42.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128318991}) 2024-09-15T02:04:42.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760289702}) 2024-09-15T02:04:42.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.754+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812265517}) 2024-09-15T02:04:42.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154804912}) 2024-09-15T02:04:42.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3969389980}) 2024-09-15T02:04:42.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154804912}) 2024-09-15T02:04:42.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.755+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168107446}) 2024-09-15T02:04:42.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756600304}) 2024-09-15T02:04:42.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=971489728}) 2024-09-15T02:04:42.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756600304}) 2024-09-15T02:04:42.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.757+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3760636706}) 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977121937}) 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260080573}) 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977121937}) 2024-09-15T02:04:42.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.758+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709350262}) 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241889022}) 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478372174}) 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241889022}) 2024-09-15T02:04:42.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.759+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756314158}) 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404114719}) 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1108746951}) 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404114719}) 2024-09-15T02:04:42.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.760+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832286097}) 2024-09-15T02:04:42.764 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.764 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545637169}) 2024-09-15T02:04:42.764 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499693335}) 2024-09-15T02:04:42.764 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.764 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545637169}) 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.761+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180626726}) 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706029902}) 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=13545836}) 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706029902}) 2024-09-15T02:04:42.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.762+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281573681}) 2024-09-15T02:04:42.765 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:04:42.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466710963}) 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185319100}) 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.763+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466710963}) 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.764+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3857477102}) 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113365051}) 2024-09-15T02:04:42.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=715193676}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113365051}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.765+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756408971}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880955936}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356126646}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880955936}) 2024-09-15T02:04:42.769 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.766+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917849627}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803080115}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=23078179}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803080115}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.767+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3391012509}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053316134}) 2024-09-15T02:04:42.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494323189}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053316134}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.768+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2678895116}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170319647}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811442081}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170319647}) 2024-09-15T02:04:42.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.769+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364204208}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305659173}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062073726}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305659173}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.770+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177191858}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430643995}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406281174}) 2024-09-15T02:04:42.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430643995}) 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.771+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462031749}) 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607023735}) 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1014873234}) 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607023735}) 2024-09-15T02:04:42.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.772+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392521405}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838646888}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280776120}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838646888}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.773+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922584}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173726763}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745752474}) 2024-09-15T02:04:42.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173726763}) 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.774+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163687410}) 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325172357}) 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=175071979}) 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325172357}) 2024-09-15T02:04:42.778 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.775+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456075082}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977249748}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418652091}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977249748}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.776+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284808730}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491205039}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779460025}) 2024-09-15T02:04:42.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491205039}) 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.777+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3238532441}) 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843621237}) 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009180651}) 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843621237}) 2024-09-15T02:04:42.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.778+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852546662}) 2024-09-15T02:04:42.782 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.782 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799298021}) 2024-09-15T02:04:42.782 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399577531}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799298021}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.779+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2372171927}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864507441}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823067094}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864507441}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106423237}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328432514}) 2024-09-15T02:04:42.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087595374}) 2024-09-15T02:04:42.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.780+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.781+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328432514}) 2024-09-15T02:04:42.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.781+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2673576319}) 2024-09-15T02:04:42.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354179412}) 2024-09-15T02:04:42.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117013698}) 2024-09-15T02:04:42.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354179412}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=69892921}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664303348}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089395919}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664303348}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.782+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379519277}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479383557}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944055988}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479383557}) 2024-09-15T02:04:42.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.783+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=999696287}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355159119}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829846021}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355159119}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.784+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054927513}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049080900}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643521191}) 2024-09-15T02:04:42.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049080900}) 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.785+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184925717}) 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309244692}) 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348962713}) 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309244692}) 2024-09-15T02:04:42.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.786+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675667305}) 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532283896}) 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=146925705}) 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532283896}) 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.787+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3811966018}) 2024-09-15T02:04:42.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553054412}) 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70694764}) 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553054412}) 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.788+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307466832}) 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463899152}) 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193959878}) 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463899152}) 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.789+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252065299}) 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226025608}) 2024-09-15T02:04:42.793 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3640100704}) 2024-09-15T02:04:42.793 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.793 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226025608}) 2024-09-15T02:04:42.793 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.790+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1413998507}) 2024-09-15T02:04:42.793 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.791+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.794 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.791+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244085119}) 2024-09-15T02:04:42.794 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.791+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181824217}) 2024-09-15T02:04:42.794 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.791+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.794 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.791+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244085119}) 2024-09-15T02:04:42.794 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.792+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172773256}) 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011915041}) 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635290306}) 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011915041}) 2024-09-15T02:04:42.795 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.793+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913312797}) 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354293246}) 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2439758748}) 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354293246}) 2024-09-15T02:04:42.796 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.794+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004198835}) 2024-09-15T02:04:42.797 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.797 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187387223}) 2024-09-15T02:04:42.797 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091304997}) 2024-09-15T02:04:42.797 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187387223}) 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668408530}) 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4202415071}) 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=887197741}) 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4202415071}) 2024-09-15T02:04:42.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.795+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146769943}) 2024-09-15T02:04:42.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.796+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596010744}) 2024-09-15T02:04:42.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983317235}) 2024-09-15T02:04:42.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596010744}) 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2294972122}) 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024027554}) 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891641685}) 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024027554}) 2024-09-15T02:04:42.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.797+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831590073}) 2024-09-15T02:04:42.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813641714}) 2024-09-15T02:04:42.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922594539}) 2024-09-15T02:04:42.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813641714}) 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.798+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606407306}) 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=1197261307,mdsmap=1531151339,mgr=2913004691,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=116462789,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=457082578}) 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=4} crc {logm=244319120,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=700715359,mon_config_key=199500175,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=1491114145}) 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=1197261307,mdsmap=1531151339,mgr=2913004691,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=116462789,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=457082578}) 2024-09-15T02:04:42.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.799+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=2,osdmap=3} crc {logm=805205205,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=927585707,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_snap=1063953654,osdmap=2157957540}) 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=466592034}) 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=2925323581}) 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=466592034}) 2024-09-15T02:04:42.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:04:42.810+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=1016375983}) 2024-09-15T02:04:42.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.876+0000 7fac9a446640 10 monclient: get_monmap_and_config 2024-09-15T02:04:42.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.876+0000 7fac9a446640 10 monclient: build_initial_monmap 2024-09-15T02:04:42.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.876+0000 7fac9a446640 10 monclient: monmap: 2024-09-15T02:04:42.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:42.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:42.877565+0000 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:42.877565+0000 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:42.879 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.877+0000 7fac9a446640 1 Processor -- start 2024-09-15T02:04:42.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.877+0000 7fac9a446640 1 -- start start 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.877+0000 7fac9a446640 10 monclient: init 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-b con 0x7fac9406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fac9406f440 0x7fac9409bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-a con 0x7fac9406f440 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac9409c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9409f6a0 con 0x7fac9409c4d0 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac94069f40 con 0x7fac9406f440 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94069950 con 0x7fac9406eb30 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac9a446640 10 monclient(hunting): authenticate will time out at 1542.874167s 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fac9406f440 0x7fac9409bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47338/0 (socket says 172.21.15.17:47338) 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): get_auth_request con 0x7fac9406f440 auth_method 0 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 1 -- 172.21.15.17:0/704799536 learned_addr learned my addr 172.21.15.17:0/704799536 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:42.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): get_auth_request con 0x7fac9406eb30 auth_method 0 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient(hunting): handle_auth_done global_id 105267 payload 274 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 -- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 msgr2=0x7fac9406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 -- 172.21.15.17:0/704799536 >> v1:172.21.15.67:6789/0 conn(0x7fac9409c4d0 legacy=0x7fac9409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:42.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient: _finish_hunting 0 2024-09-15T02:04:42.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 monclient: found mon.noname-a 2024-09-15T02:04:42.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:42.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:42.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac937fe640 1 -- 172.21.15.17:0/704799536 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9409ad80 con 0x7fac9406f440 2024-09-15T02:04:42.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.878+0000 7fac93fff640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac937fe640 1 -- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9406f440 msgr2=0x7fac9409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac937fe640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9406f440 0x7fac9409bf90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fac88009850 tx=0x7fac880090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65eadf06a6929 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 1 -- 172.21.15.17:0/704799536 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8802adf0 con 0x7fac9406f440 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: dump: 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:42.890 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: _finish_auth 0 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac9a446640 5 monclient: authenticate success, global_id 105267 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac9a446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 1 -- 172.21.15.17:0/704799536 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac88032440 con 0x7fac9406f440 2024-09-15T02:04:42.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac9a446640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 1 -- 172.21.15.17:0/704799536 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac880327e0 con 0x7fac9406f440 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac9a446640 4 set_mon_vals no callback set 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.881+0000 7fac92ffd640 10 monclient: dump: 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.892 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 10 monclient: shutdown 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 -- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9406f440 msgr2=0x7fac9409bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9406f440 0x7fac9409bf90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fac88009850 tx=0x7fac880090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 -- 172.21.15.17:0/704799536 shutdown_connections 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9406f440 0x7fac9409bf90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 --2- 172.21.15.17:0/704799536 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac9406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 -- 172.21.15.17:0/704799536 >> 172.21.15.17:0/704799536 conn(0x7fac940712f0 msgr2=0x7fac940a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 -- 172.21.15.17:0/704799536 shutdown_connections 2024-09-15T02:04:42.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 1 -- 172.21.15.17:0/704799536 wait complete. 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 10 monclient: build_initial_monmap 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.882+0000 7fac9a446640 10 monclient: monmap: 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:42.894 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 Processor -- start 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 -- start start 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient: init 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-c con 0x7fac9406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9406f440 0x7fac9411d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-b con 0x7fac9406f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 0x7fac9411df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient(hunting): picked mon.noname-a con 0x7fac9409c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.883+0000 7fac9a446640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94141f20 con 0x7fac9409c4d0 2024-09-15T02:04:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94121090 con 0x7fac9406f440 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94120db0 con 0x7fac9406eb30 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac9a446640 10 monclient(hunting): authenticate will time out at 1542.880024s 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 0x7fac9411df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47344/0 (socket says 172.21.15.17:47344) 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 -- 172.21.15.17:0/3611513502 learned_addr learned my addr 172.21.15.17:0/3611513502 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): get_auth_request con 0x7fac9406eb30 auth_method 0 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:42.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): get_auth_request con 0x7fac9409c4d0 auth_method 0 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9406f440 0x7fac9411d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:42.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): get_auth_request con 0x7fac9406f440 auth_method 0 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient(hunting): handle_auth_done global_id 164195 payload 824 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 -- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9406f440 msgr2=0x7fac9411d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9406f440 0x7fac9411d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 -- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 msgr2=0x7fac9411df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 0x7fac9411df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient: _finish_hunting 0 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 monclient: found mon.noname-c 2024-09-15T02:04:42.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac93fff640 1 -- 172.21.15.17:0/3611513502 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84009450 con 0x7fac9406eb30 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.884+0000 7fac989e3640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 0x7fac9411df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac93fff640 1 -- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 msgr2=0x7fac94133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac93fff640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac8400ea30 tx=0x7fac840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84009690 con 0x7fac9406eb30 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: dump: 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.899 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: _finish_auth 0 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8400cc20 con 0x7fac9406eb30 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84013470 con 0x7fac9406eb30 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac9a446640 5 monclient: authenticate success, global_id 164195 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac9a446640 10 monclient: _renew_subs 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac9a446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac88009390 con 0x7fac9406eb30 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac927fc640 4 set_mon_vals no callback set 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.886+0000 7fac7bfff640 10 monclient: dump: 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:42.901 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 10 monclient: _renew_subs 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac94134ac0 con 0x7fac9406eb30 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fac84013820 con 0x7fac9406eb30 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac940724a0 con 0x7fac9406eb30 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.887+0000 7fac7bfff640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 0x7fac8003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.888+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fac8404ca00 con 0x7fac9406eb30 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.888+0000 7fac937fe640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 0x7fac8003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:42.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.888+0000 7fac937fe640 10 monclient: get_auth_request con 0x7fac80038740 auth_method 0 2024-09-15T02:04:42.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.891+0000 7fac937fe640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 0x7fac8003abf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fac9409f9a0 tx=0x7fac88032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:42.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.892+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fac84023aa0 con 0x7fac9406eb30 2024-09-15T02:04:42.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.892+0000 7fac7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:42.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:42.892+0000 7fac7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac9a446640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac9a446640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac9a446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fac94067220 con 0x7fac9406eb30 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac7bfff640 1 -- 172.21.15.17:0/3611513502 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fac84021080 con 0x7fac9406eb30 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.121+0000 7fac7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:43.123 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":140,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 msgr2=0x7fac8003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 0x7fac8003abf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fac9409f9a0 tx=0x7fac88032000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 10 monclient: shutdown 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 msgr2=0x7fac94133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac8400ea30 tx=0x7fac840091d0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 shutdown_connections 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fac80038740 0x7fac8003abf0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fac9409c4d0 0x7fac9411df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9406f440 0x7fac9411d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 --2- 172.21.15.17:0/3611513502 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fac9406eb30 0x7fac94133c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 >> 172.21.15.17:0/3611513502 conn(0x7fac940712f0 msgr2=0x7fac940a1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 shutdown_connections 2024-09-15T02:04:43.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.123+0000 7fac9a446640 1 -- 172.21.15.17:0/3611513502 wait complete. 2024-09-15T02:04:43.136 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:04:43.136 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:04:43.136 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:04:43.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:04:43.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.271+0000 7f9ab2d1d640 10 monclient: get_monmap_and_config 2024-09-15T02:04:43.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.271+0000 7f9ab2d1d640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.271+0000 7f9ab2d1d640 10 monclient: monmap: 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:43.272730+0000 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:43.272730+0000 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:43.274 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:43.275 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:43.275 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:43.275 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:43.275 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.272+0000 7f9ab2d1d640 1 Processor -- start 2024-09-15T02:04:43.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.272+0000 7f9ab2d1d640 1 -- start start 2024-09-15T02:04:43.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.272+0000 7f9ab2d1d640 10 monclient: init 2024-09-15T02:04:43.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9aac068f20 0x7f9aac1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac068f20 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9aac0692f0 0x7f9aac0696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac0692f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9aac12e900 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0a1670 con 0x7f9aac12e900 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0a0020 con 0x7f9aac068f20 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0a1390 con 0x7f9aac0692f0 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab2d1d640 10 monclient(hunting): authenticate will time out at 1543.269176s 2024-09-15T02:04:43.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9aac068f20 0x7f9aac1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab12ba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9aac12e900 0x7f9aac12ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42714/0 (socket says 172.21.15.17:42714) 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab12ba640 1 -- 172.21.15.17:0/2949511767 learned_addr learned my addr 172.21.15.17:0/2949511767 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/2949511767 >> v2:172.21.15.17:3300/0 conn(0x7f9aac068f20 0x7f9aac1321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37026/0 (socket says 172.21.15.17:37026) 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request con 0x7f9aac068f20 auth_method 0 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9aabfff640 1 --2- 172.21.15.17:0/2949511767 >> v2:172.21.15.67:3300/0 conn(0x7f9aac0692f0 0x7f9aac0696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_done global_id 105273 payload 274 2024-09-15T02:04:43.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/2949511767 >> v2:172.21.15.67:3300/0 conn(0x7f9aac0692f0 msgr2=0x7f9aac0696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/2949511767 >> v2:172.21.15.67:3300/0 conn(0x7f9aac0692f0 0x7f9aac0696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/2949511767 >> v1:172.21.15.17:6789/0 conn(0x7f9aac12e900 legacy=0x7f9aac12ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 monclient: found mon.noname-a 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.273+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/2949511767 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac069c30 con 0x7f9aac068f20 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/2949511767 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac068f20 msgr2=0x7f9aac1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/2949511767 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac068f20 0x7f9aac1321c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0687a0 tx=0x7f9aa4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1be7d9a7b967c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 1 -- 172.21.15.17:0/2949511767 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4007840 con 0x7f9aac068f20 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: dump: 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 1 -- 172.21.15.17:0/2949511767 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa402d050 con 0x7f9aac068f20 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 1 -- 172.21.15.17:0/2949511767 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4007bd0 con 0x7f9aac068f20 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9ab2d1d640 5 monclient: authenticate success, global_id 105273 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9ab2d1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9ab2d1d640 4 set_mon_vals no callback set 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.276+0000 7f9aab7fe640 10 monclient: dump: 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.289 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 10 monclient: shutdown 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/2949511767 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac068f20 msgr2=0x7f9aac1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/2949511767 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac068f20 0x7f9aac1321c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0687a0 tx=0x7f9aa4009780 comp rx=0 tx=0).stop 2024-09-15T02:04:43.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/2949511767 shutdown_connections 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/2949511767 >> v2:172.21.15.67:3300/0 conn(0x7f9aac0692f0 0x7f9aac0696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/2949511767 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac068f20 0x7f9aac1321c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/2949511767 >> 172.21.15.17:0/2949511767 conn(0x7f9aac070fb0 msgr2=0x7f9aac0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/2949511767 shutdown_connections 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/2949511767 wait complete. 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 10 monclient: monmap: 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.277+0000 7f9ab2d1d640 1 Processor -- start 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 -- start start 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient: init 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aac068f20 0x7f9aac141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9aac068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 0x7f9aac12daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac0692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9aac12e900 0x7f9aac097e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac12e900 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac130c40 con 0x7f9aac0692f0 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac09b720 con 0x7f9aac068f20 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac141ed0 con 0x7f9aac12e900 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab2d1d640 10 monclient(hunting): authenticate will time out at 1543.274658s 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 0x7f9aac12daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9aac12e900 0x7f9aac097e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 0x7f9aac12daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37040/0 (socket says 172.21.15.17:37040) 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 1 -- 172.21.15.17:0/1573760890 learned_addr learned my addr 172.21.15.17:0/1573760890 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): get_auth_request con 0x7f9aac0692f0 auth_method 0 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): get_auth_request con 0x7f9aac12e900 auth_method 0 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9aabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab12ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aac068f20 0x7f9aac141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request con 0x7f9aac068f20 auth_method 0 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.278+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 10 monclient(hunting): handle_auth_done global_id 105276 payload 824 2024-09-15T02:04:43.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9aac12e900 msgr2=0x7f9aac097e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9aac12e900 0x7f9aac097e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aac068f20 msgr2=0x7f9aac141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aac068f20 0x7f9aac141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 monclient: found mon.noname-a 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.279+0000 7f9aabfff640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa4009a20 con 0x7f9aac0692f0 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aabfff640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 msgr2=0x7f9aac12daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aabfff640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 0x7f9aac12daf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c880 tx=0x7f9aa0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa000ce10 con 0x7f9aac0692f0 2024-09-15T02:04:43.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: dump: 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa0007a40 con 0x7f9aac0692f0 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 5 monclient: authenticate success, global_id 105276 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 10 monclient: _renew_subs 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac099910 con 0x7f9aac0692f0 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0007de0 con 0x7f9aac0692f0 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aa8ff9640 10 monclient: dump: 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 10 monclient: _renew_subs 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac09b4f0 con 0x7f9aac0692f0 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.281+0000 7f9aaaffd640 4 set_mon_vals no callback set 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab2d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 0x7f9aac098360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac063b90 con 0x7f9aac09a4e0 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 0x7f9aac098360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient: get_auth_request con 0x7f9aac09a4e0 auth_method 0 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9aa0011600 con 0x7f9aac0692f0 2024-09-15T02:04:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9aa8ff9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a98038790 0x7f9a9803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_done global_id 105279 payload 824 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aac0724a0 con 0x7f9aac09a4e0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.282+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9aa004c450 con 0x7f9aac0692f0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.283+0000 7f9ab12ba640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a98038790 0x7f9a9803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.283+0000 7f9ab12ba640 10 monclient: get_auth_request con 0x7f9a98038790 auth_method 0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 msgr2=0x7f9aac098360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 0x7f9aac098360 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4034000 tx=0x7f9aa4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4031c10 con 0x7f9aac09a4e0 2024-09-15T02:04:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 10 monclient: dump: 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9aa4031d70 con 0x7f9aac09a4e0 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 msgr2=0x7f9aac098360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 0x7f9aac098360 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4034000 tx=0x7f9aa4037000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 msgr2=0x7f9aac098360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9aa8ff9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac09a4e0 0x7f9aac098360 secure :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4034000 tx=0x7f9aa4037000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.285+0000 7f9ab12ba640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a98038790 0x7f9a9803ac40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c005fd0 tx=0x7f9a9c007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab2d1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 0x7f9aac0988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab2d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac09b890 con 0x7f9aac12e030 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 0x7f9aac0988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient: get_auth_request con 0x7f9aac12e030 auth_method 0 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 10 monclient(hunting): handle_auth_done global_id 105282 payload 824 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.291+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/1573760890 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9aac06f150 con 0x7f9aac12e030 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9ab0ab9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 msgr2=0x7f9aac0988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9ab0ab9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 0x7f9aac0988a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0638e0 tx=0x7f9aa403f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa403cb70 con 0x7f9aac12e030 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9aa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.293+0000 7f9aa8ff9640 10 monclient: dump: 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f9aa404c040 con 0x7f9aac12e030 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 msgr2=0x7f9aac0988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 0x7f9aac0988a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0638e0 tx=0x7f9aa403f040 comp rx=0 tx=0).stop 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 msgr2=0x7f9aac0988a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9aa8ff9640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac12e030 0x7f9aac0988a0 secure :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0638e0 tx=0x7f9aa403f040 comp rx=0 tx=0).stop 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:43.308 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:43.309 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.310 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:43.311 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:43.312 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:43.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.317 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.319 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:04:43.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:43.321 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a98038790 msgr2=0x7f9a9803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9a98038790 0x7f9a9803ac40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c005fd0 tx=0x7f9a9c007f90 comp rx=0 tx=0).stop 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 10 monclient: shutdown 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 msgr2=0x7f9aac12daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 --2- 172.21.15.17:0/1573760890 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9aac0692f0 0x7f9aac12daf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c880 tx=0x7f9aa0008040 comp rx=0 tx=0).stop 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab12ba640 1 -- 172.21.15.17:0/1573760890 reap_dead start 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab12ba640 1 -- 172.21.15.17:0/1573760890 reap_dead start 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 shutdown_connections 2024-09-15T02:04:43.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 >> 172.21.15.17:0/1573760890 conn(0x7f9aac070fb0 msgr2=0x7f9aac130e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 shutdown_connections 2024-09-15T02:04:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.294+0000 7f9ab2d1d640 1 -- 172.21.15.17:0/1573760890 wait complete. 2024-09-15T02:04:43.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:43.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:04:43.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:04:43.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 10 monclient: get_monmap_and_config 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 10 monclient: monmap: 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:43.420993+0000 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:43.420993+0000 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:43.423 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 1 Processor -- start 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 1 -- start start 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.420+0000 7f05d922b640 10 monclient: init 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05d40a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05d4132040 0x7f05d40987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-c con 0x7f05d4132040 addr v2:172.21.15.67:3300/0 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f05d4136dd0 0x7f05d41391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-a con 0x7f05d4136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d40a01e0 con 0x7f05d40a58b0 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d40a1580 con 0x7f05d4136dd0 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d40a5c80 con 0x7f05d4132040 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d922b640 10 monclient(hunting): authenticate will time out at 1543.417307s 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f05d4136dd0 0x7f05d41391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f05d4136dd0 0x7f05d41391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37070/0 (socket says 172.21.15.17:37070) 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 -- 172.21.15.17:0/2710309429 learned_addr learned my addr 172.21.15.17:0/2710309429 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): get_auth_request con 0x7f05d4136dd0 auth_method 0 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient(hunting): handle_auth_done global_id 105285 payload 274 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 -- 172.21.15.17:0/2710309429 >> v2:172.21.15.67:3300/0 conn(0x7f05d4132040 msgr2=0x7f05d40987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d259c640 1 --2- 172.21.15.17:0/2710309429 >> v2:172.21.15.67:3300/0 conn(0x7f05d4132040 0x7f05d40987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 --2- 172.21.15.17:0/2710309429 >> v2:172.21.15.67:3300/0 conn(0x7f05d4132040 0x7f05d40987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 -- 172.21.15.17:0/2710309429 >> v1:172.21.15.67:6789/0 conn(0x7f05d40a58b0 legacy=0x7f05d40980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 monclient: found mon.noname-a 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.421+0000 7f05d359e640 1 -- 172.21.15.17:0/2710309429 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d4132440 con 0x7f05d4136dd0 2024-09-15T02:04:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d359e640 1 -- 172.21.15.17:0/2710309429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4136dd0 msgr2=0x7f05d41391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d359e640 1 --2- 172.21.15.17:0/2710309429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4136dd0 0x7f05d41391b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f05d409f780 tx=0x7f05cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5eb998b65f8c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 1 -- 172.21.15.17:0/2710309429 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc007610 con 0x7f05d4136dd0 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: dump: 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 1 -- 172.21.15.17:0/2710309429 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc02d050 con 0x7f05d4136dd0 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 1 -- 172.21.15.17:0/2710309429 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0079a0 con 0x7f05d4136dd0 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d922b640 5 monclient: authenticate success, global_id 105285 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d922b640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d922b640 4 set_mon_vals no callback set 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.424+0000 7f05d1d9b640 10 monclient: dump: 2024-09-15T02:04:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 10 monclient: shutdown 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 -- 172.21.15.17:0/2710309429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4136dd0 msgr2=0x7f05d41391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 --2- 172.21.15.17:0/2710309429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4136dd0 0x7f05d41391b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f05d409f780 tx=0x7f05cc009840 comp rx=0 tx=0).stop 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 -- 172.21.15.17:0/2710309429 shutdown_connections 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 --2- 172.21.15.17:0/2710309429 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4136dd0 0x7f05d41391b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 --2- 172.21.15.17:0/2710309429 >> v2:172.21.15.67:3300/0 conn(0x7f05d4132040 0x7f05d40987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 -- 172.21.15.17:0/2710309429 >> 172.21.15.17:0/2710309429 conn(0x7f05d4070ed0 msgr2=0x7f05d40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 -- 172.21.15.17:0/2710309429 shutdown_connections 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 1 -- 172.21.15.17:0/2710309429 wait complete. 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.425+0000 7f05d922b640 10 monclient: monmap: 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 Processor -- start 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 -- start start 2024-09-15T02:04:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient: init 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d40a58b0 0x7f05d409b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-c con 0x7f05d40a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-a con 0x7f05d4132040 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d4136dd0 0x7f05d4133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): picked mon.noname-b con 0x7f05d4136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4135e00 con 0x7f05d4132040 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40a3070 con 0x7f05d4136dd0 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4135b20 con 0x7f05d40a58b0 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d922b640 10 monclient(hunting): authenticate will time out at 1543.422740s 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d40a58b0 0x7f05d409b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37084/0 (socket says 172.21.15.17:37084) 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 1 -- 172.21.15.17:0/1796522304 learned_addr learned my addr 172.21.15.17:0/1796522304 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d40a58b0 0x7f05d409b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47350/0 (socket says 172.21.15.17:47350) 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): get_auth_request con 0x7f05d4132040 auth_method 0 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): get_auth_request con 0x7f05d40a58b0 auth_method 0 2024-09-15T02:04:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.426+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d359e640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d4136dd0 0x7f05d4133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d359e640 10 monclient(hunting): get_auth_request con 0x7f05d4136dd0 auth_method 0 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 10 monclient(hunting): handle_auth_done global_id 105288 payload 824 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d40a58b0 msgr2=0x7f05d409b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d40a58b0 0x7f05d409b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d4136dd0 msgr2=0x7f05d4133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d4136dd0 0x7f05d4133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 monclient: found mon.noname-a 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.427+0000 7f05d259c640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc009ae0 con 0x7f05d4132040 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d259c640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 msgr2=0x7f05d4132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d259c640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c880 tx=0x7f05c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c800ce10 con 0x7f05d4132040 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: dump: 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8007a40 con 0x7f05d4132040 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 5 monclient: authenticate success, global_id 105288 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 10 monclient: _renew_subs 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d4131780 con 0x7f05d4132040 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8007de0 con 0x7f05d4132040 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05baffd640 10 monclient: dump: 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d159a640 4 set_mon_vals no callback set 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 10 monclient: _renew_subs 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.429+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d4136980 con 0x7f05d4132040 2024-09-15T02:04:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d922b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d922b640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 0x7f05d41371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4063b20 con 0x7f05d41335a0 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f05c8011600 con 0x7f05d4132040 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 0x7f05d41371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient: get_auth_request con 0x7f05d41335a0 auth_method 0 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05baffd640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05c0038740 0x7f05c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f05c804c8c0 con 0x7f05d4132040 2024-09-15T02:04:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_done global_id 164204 payload 824 2024-09-15T02:04:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.430+0000 7f05d2d9d640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05d4135fc0 con 0x7f05d41335a0 2024-09-15T02:04:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.431+0000 7f05d359e640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05c0038740 0x7f05c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.431+0000 7f05d359e640 10 monclient: get_auth_request con 0x7f05c0038740 auth_method 0 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05d2d9d640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 msgr2=0x7f05d41371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05d2d9d640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 0x7f05d41371a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05c400e7b0 tx=0x7f05c4006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c400cac0 con 0x7f05d41335a0 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 10 monclient: dump: 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f05c400cc20 con 0x7f05d41335a0 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 msgr2=0x7f05d41371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 0x7f05d41371a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05c400e7b0 tx=0x7f05c4006f50 comp rx=0 tx=0).stop 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 msgr2=0x7f05d41371a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05baffd640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41335a0 0x7f05d41371a0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05c400e7b0 tx=0x7f05c4006f50 comp rx=0 tx=0).stop 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.433+0000 7f05d359e640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05c0038740 0x7f05c003abf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f05d40a60e0 tx=0x7f05cc036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d922b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d922b640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 0x7f05d4137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d922b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4131910 con 0x7f05d41376e0 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 0x7f05d4137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient: get_auth_request con 0x7f05d41376e0 auth_method 0 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 10 monclient(hunting): handle_auth_done global_id 164207 payload 824 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.439+0000 7f05d2d9d640 1 -- 172.21.15.17:0/1796522304 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f05d4136200 con 0x7f05d41376e0 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.441+0000 7f05d2d9d640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 msgr2=0x7f05d4137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.441+0000 7f05d2d9d640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 0x7f05d4137ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009fd0 tx=0x7f05c4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4005d60 con 0x7f05d41376e0 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 10 monclient: dump: 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f05c4006050 con 0x7f05d41376e0 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 msgr2=0x7f05d4137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 0x7f05d4137ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009fd0 tx=0x7f05c4015000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 msgr2=0x7f05d4137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05baffd640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f05d41376e0 0x7f05d4137ae0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009fd0 tx=0x7f05c4015000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 6, 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:04:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:43.466 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05c0038740 msgr2=0x7f05c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f05c0038740 0x7f05c003abf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f05d40a60e0 tx=0x7f05cc036000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 10 monclient: shutdown 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 msgr2=0x7f05d4132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c880 tx=0x7f05c8008040 comp rx=0 tx=0).stop 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d359e640 1 -- 172.21.15.17:0/1796522304 reap_dead start 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 shutdown_connections 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 --2- 172.21.15.17:0/1796522304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f05d4132040 0x7f05d4132b20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 >> 172.21.15.17:0/1796522304 conn(0x7f05d4070ed0 msgr2=0x7f05d4138b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 shutdown_connections 2024-09-15T02:04:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.442+0000 7f05d922b640 1 -- 172.21.15.17:0/1796522304 wait complete. 2024-09-15T02:04:43.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:43.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:04:43.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:04:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.564+0000 7f3c56aa8640 10 monclient: get_monmap_and_config 2024-09-15T02:04:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.564+0000 7f3c56aa8640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.564+0000 7f3c56aa8640 10 monclient: monmap: 2024-09-15T02:04:43.566 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:04:43.565694+0000 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:04:43.565694+0000 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:04:43.567 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.565+0000 7f3c56aa8640 1 Processor -- start 2024-09-15T02:04:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.565+0000 7f3c56aa8640 1 -- start start 2024-09-15T02:04:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.565+0000 7f3c56aa8640 10 monclient: init 2024-09-15T02:04:43.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c5009c2c0 addr v1:172.21.15.67:6789/0 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3c5009e970 0x7f3c5009c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5009e970 addr v2:172.21.15.17:3300/0 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c5006eb30 addr v1:172.21.15.17:6789/0 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5009ed80 con 0x7f3c5006eb30 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5006a5c0 con 0x7f3c5009c2c0 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c50099f50 con 0x7f3c5009e970 2024-09-15T02:04:43.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c56aa8640 10 monclient(hunting): authenticate will time out at 1543.562202s 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3c5009e970 0x7f3c5009c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c55045640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3c5006eb30 0x7f3c5006ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42718/0 (socket says 172.21.15.17:42718) 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c55045640 1 -- 172.21.15.17:0/2796574523 learned_addr learned my addr 172.21.15.17:0/2796574523 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request con 0x7f3c5009e970 auth_method 0 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_done global_id 105291 payload 274 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 1 -- 172.21.15.17:0/2796574523 >> v1:172.21.15.67:6789/0 conn(0x7f3c5009c2c0 legacy=0x7f3c5009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 1 -- 172.21.15.17:0/2796574523 >> v1:172.21.15.17:6789/0 conn(0x7f3c5006eb30 legacy=0x7f3c5006ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 1 monclient: found mon.noname-a 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:04:43.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.566+0000 7f3c4ffff640 1 -- 172.21.15.17:0/2796574523 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c50068ac0 con 0x7f3c5009e970 2024-09-15T02:04:43.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4ffff640 1 -- 172.21.15.17:0/2796574523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009e970 msgr2=0x7f3c5009c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4ffff640 1 --2- 172.21.15.17:0/2796574523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009e970 0x7f3c5009c690 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009850 tx=0x7f3c3c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c6072a8aebcbc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 1 -- 172.21.15.17:0/2796574523 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c032440 con 0x7f3c5009e970 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: dump: 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.577 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 1 -- 172.21.15.17:0/2796574523 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c0325a0 con 0x7f3c5009e970 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 1 -- 172.21.15.17:0/2796574523 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c032940 con 0x7f3c5009e970 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c56aa8640 5 monclient: authenticate success, global_id 105291 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c56aa8640 10 monclient: get_monmap_and_config success 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c56aa8640 4 set_mon_vals no callback set 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:04:43.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.569+0000 7f3c4f7fe640 10 monclient: dump: 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 10 monclient: shutdown 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 -- 172.21.15.17:0/2796574523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009e970 msgr2=0x7f3c5009c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/2796574523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009e970 0x7f3c5009c690 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009850 tx=0x7f3c3c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 -- 172.21.15.17:0/2796574523 shutdown_connections 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/2796574523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009e970 0x7f3c5009c690 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 -- 172.21.15.17:0/2796574523 >> 172.21.15.17:0/2796574523 conn(0x7f3c500a2f70 msgr2=0x7f3c500a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 -- 172.21.15.17:0/2796574523 shutdown_connections 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.570+0000 7f3c56aa8640 1 -- 172.21.15.17:0/2796574523 wait complete. 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.571+0000 7f3c56aa8640 10 monclient: build_initial_monmap 2024-09-15T02:04:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.571+0000 7f3c56aa8640 10 monclient: monmap: 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.571+0000 7f3c56aa8640 1 Processor -- start 2024-09-15T02:04:43.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.571+0000 7f3c56aa8640 1 -- start start 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.571+0000 7f3c56aa8640 10 monclient: init 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient: _reopen_session rank -1 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 0x7f3c50133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009c2c0 0x7f3c5011d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5009c2c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5009e970 0x7f3c5011df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5009e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50141ff0 con 0x7f3c5009c2c0 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50120080 con 0x7f3c5009e970 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50120ee0 con 0x7f3c5006eb30 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): _renew_subs 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c56aa8640 10 monclient(hunting): authenticate will time out at 1543.568107s 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009c2c0 0x7f3c5011d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 0x7f3c50133dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009c2c0 0x7f3c5011d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37090/0 (socket says 172.21.15.17:37090) 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): get_auth_request con 0x7f3c5006eb30 auth_method 0 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 1 -- 172.21.15.17:0/4257656318 learned_addr learned my addr 172.21.15.17:0/4257656318 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request con 0x7f3c5009c2c0 auth_method 0 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5009e970 0x7f3c5011df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 10 monclient(hunting): get_auth_request con 0x7f3c5009e970 auth_method 0 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient(hunting): handle_auth_done global_id 164210 payload 824 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5009e970 msgr2=0x7f3c5011df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5009e970 0x7f3c5011df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009c2c0 msgr2=0x7f3c5011d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3c5009c2c0 0x7f3c5011d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient: _finish_hunting 0 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 monclient: found mon.noname-c 2024-09-15T02:04:43.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c54844640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009390 con 0x7f3c5006eb30 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.572+0000 7f3c55045640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5009e970 0x7f3c5011df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c54844640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 msgr2=0x7f3c50133dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c54844640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 0x7f3c50133dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c4e0 tx=0x7f3c44007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440117a0 con 0x7f3c5006eb30 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: dump: 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.586 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: _finish_auth 0 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c56aa8640 5 monclient: authenticate success, global_id 164210 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c56aa8640 10 monclient: _renew_subs 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c56aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c501214a0 con 0x7f3c5006eb30 2024-09-15T02:04:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c44011920 con 0x7f3c5006eb30 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44011ce0 con 0x7f3c5006eb30 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4cff9640 10 monclient: dump: 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.588 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.589 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.574+0000 7f3c4effd640 4 set_mon_vals no callback set 2024-09-15T02:04:43.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 10 monclient: _renew_subs 2024-09-15T02:04:43.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:04:43.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c501365b0 con 0x7f3c5006eb30 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 0x7f3c5011e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50063b90 con 0x7f3c50134300 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.575+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3c44012630 con 0x7f3c5006eb30 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4cff9640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3c34038740 0x7f3c3403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 0x7f3c5011e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3c4404c260 con 0x7f3c5006eb30 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient: get_auth_request con 0x7f3c50134300 auth_method 0 2024-09-15T02:04:43.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c55045640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3c34038740 0x7f3c3403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c55045640 10 monclient: get_auth_request con 0x7f3c34038740 auth_method 0 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_done global_id 174121 payload 824 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.576+0000 7f3c4ffff640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c50134d00 con 0x7f3c50134300 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.578+0000 7f3c4ffff640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 msgr2=0x7f3c5011e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.578+0000 7f3c4ffff640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 0x7f3c5011e460 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c035040 tx=0x7f3c3c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.578+0000 7f3c55045640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3c34038740 0x7f3c3403abf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3c5009cf80 tx=0x7f3c48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c031a70 con 0x7f3c50134300 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 10 monclient: dump: 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3c3c031bd0 con 0x7f3c50134300 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 msgr2=0x7f3c5011e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 0x7f3c5011e460 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c035040 tx=0x7f3c3c032000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 msgr2=0x7f3c5011e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.579+0000 7f3c4cff9640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c50134300 0x7f3c5011e460 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c035040 tx=0x7f3c3c032000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c56aa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 0x7f3c500a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c56aa8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:04:43.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50121630 con 0x7f3c5011e9a0 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 0x7f3c500a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient: get_auth_request con 0x7f3c5011e9a0 auth_method 0 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:04:43.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 10 monclient(hunting): handle_auth_done global_id 174124 payload 824 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.585+0000 7f3c4ffff640 1 -- 172.21.15.17:0/4257656318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c5011f440 con 0x7f3c5011e9a0 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4ffff640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 msgr2=0x7f3c500a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4ffff640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 0x7f3c500a1650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c003080 tx=0x7f3c3c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c045760 con 0x7f3c5011e9a0 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 10 monclient: dump: 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:04:43.598 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f3c3c0458c0 con 0x7f3c5011e9a0 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 msgr2=0x7f3c500a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.588+0000 7f3c4cff9640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 0x7f3c500a1650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c003080 tx=0x7f3c3c03d000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c4cff9640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 msgr2=0x7f3c500a1650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:04:43.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c4cff9640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5011e9a0 0x7f3c500a1650 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c003080 tx=0x7f3c3c03d000 comp rx=0 tx=0).stop 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 140, 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:04:43.600 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.601 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:04:43.603 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:04:43.604 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.605 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.606 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:04:43.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:04:43.610 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:04:43.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:04:43.612 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:04:43.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3c34038740 msgr2=0x7f3c3403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3c34038740 0x7f3c3403abf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3c5009cf80 tx=0x7f3c48009040 comp rx=0 tx=0).stop 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 10 monclient: shutdown 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 msgr2=0x7f3c50133dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 0x7f3c50133dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c4e0 tx=0x7f3c44007e10 comp rx=0 tx=0).stop 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c55045640 1 -- 172.21.15.17:0/4257656318 reap_dead start 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 shutdown_connections 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 --2- 172.21.15.17:0/4257656318 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3c5006eb30 0x7f3c50133dc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 >> 172.21.15.17:0/4257656318 conn(0x7f3c500a2f70 msgr2=0x7f3c50134ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 shutdown_connections 2024-09-15T02:04:43.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:04:43.589+0000 7f3c56aa8640 1 -- 172.21.15.17:0/4257656318 wait complete. 2024-09-15T02:04:43.615 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:04:43.615 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T02:04:43.615 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'b'] 2024-09-15T02:04:43.615 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T02:04:43.615 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:04:43.615 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:04:43.615 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:04:43.627 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:04:43.627 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:04:43.627 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:04:43.627 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T02:04:43.628 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T02:04:43.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:05:03.629 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:05:03.629 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:05:03.629 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T02:05:03.630 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T02:05:03.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:05:03.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:05:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.737+0000 7fb2ce99d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.737+0000 7fb2ce99d640 10 monclient: build_initial_monmap 2024-09-15T02:05:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.737+0000 7fb2ce99d640 10 monclient: monmap: 2024-09-15T02:05:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:03.738366+0000 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:03.738366+0000 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:03.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.738+0000 7fb2ce99d640 1 Processor -- start 2024-09-15T02:05:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.738+0000 7fb2ce99d640 1 -- start start 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.738+0000 7fb2ce99d640 10 monclient: init 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2c809c2c0 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c809e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c806eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c809ed80 con 0x7fb2c809c2c0 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c8099f50 con 0x7fb2c806eb30 2024-09-15T02:05:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c806a300 con 0x7fb2c809e970 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ce99d640 10 monclient(hunting): authenticate will time out at 1563.735257s 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2bffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 msgr2=0x7fb2c809c690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35356/0 (socket says 172.21.15.17:35356) 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2c7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb2c809c2c0 0x7fb2c809a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:37518/0 (socket says 172.21.15.17:37518) 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 learned_addr learned my addr 172.21.15.17:0/1715715387 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2c806eb30 auth_method 0 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:03.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.739+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:03.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.741+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v1:172.21.15.67:6789/0 conn(0x7fb2c809c2c0 legacy=0x7fb2c809a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2bffff640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 msgr2=0x7fb2c809c690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2bffff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2c806eb30 auth_method 0 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:05:03.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:03.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:03.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:03.940+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2bffff640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 msgr2=0x7fb2c809c690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2bffff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2c806eb30 auth_method 0 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:04.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:04.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:04.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:05:04.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:04.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:04.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:04.341+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2bffff640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 msgr2=0x7fb2c809c690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2bffff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2c806eb30 auth_method 0 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:05.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:05.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:05.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:05:05.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:05.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:05.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:05.142+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:06.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): tick 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 monclient(hunting): continuing hunt 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 msgr2=0x7fb2c809c690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2c809e970 0x7fb2c809c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 msgr2=0x7fb2c806ef90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.17:3300/0 conn(0x7fb2c806eb30 0x7fb2c806ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 -- 172.21.15.17:0/1715715387 >> v1:172.21.15.67:6789/0 conn(0x7fb2c809c2c0 legacy=0x7fb2c809a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient: _add_conns ranks=[4,0] 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:06.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a8003720 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2a8006280 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 -- 172.21.15.17:0/1715715387 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c806a5c0 con 0x7fb2a8006280 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 1 -- 172.21.15.17:0/1715715387 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c8099f50 con 0x7fb2a8003720 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c67fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request con 0x7fb2a8003720 auth_method 0 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.739+0000 7fb2c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.740+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.740+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:06.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.740+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:06.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.740+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:06.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.742+0000 7fb2bffff640 1 -- 172.21.15.17:0/1715715387 >> v1:172.21.15.17:6789/0 conn(0x7fb2a8006280 legacy=0x7fb2a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request con 0x7fb2a8003720 auth_method 0 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:06.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:06.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:06.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:06.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:06.940+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:07.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request con 0x7fb2a8003720 auth_method 0 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:07.341+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:08.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request con 0x7fb2a8003720 auth_method 0 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:08.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.143+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:08.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.648+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200325012 0 0) 0x7fb2c806a5c0 con 0x7fb2a8006280 2024-09-15T02:05:08.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.648+0000 7fb2c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.648+0000 7fb2c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:08.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.648+0000 7fb2c77fe640 10 monclient(hunting): my global_id is 105297 2024-09-15T02:05:08.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.648+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b40034e0 con 0x7fb2a8006280 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266933726 0 0) 0x7fb2b40034e0 con 0x7fb2a8006280 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 msgr2=0x7fb2a8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 --2- 172.21.15.17:0/1715715387 >> v2:172.21.15.67:3300/0 conn(0x7fb2a8003720 0x7fb2a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: _finish_hunting 0 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:05:08.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a800b0c0 con 0x7fb2a8006280 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fb2b0004380 con 0x7fb2a8006280 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: dump: 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.663 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 >> v1:172.21.15.17:6789/0 conn(0x7fb2a8006280 legacy=0x7fb2a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 reap_dead start 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 0x7fb2b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): picked mon.b con 0x7fb2b40039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 0x7fb2b4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): picked mon.c con 0x7fb2b4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 0x7fb2b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): picked mon.a con 0x7fb2b4005180 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8099f50 con 0x7fb2b4005180 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b400c120 con 0x7fb2b40039d0 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b400c2e0 con 0x7fb2b4004780 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2c77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2bffff640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 msgr2=0x7fb2b4004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.650+0000 7fb2bffff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 0x7fb2b4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 0x7fb2b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2b4005180 auth_method 0 2024-09-15T02:05:08.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 0x7fb2b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request con 0x7fb2b40039d0 auth_method 0 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:08.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient(hunting): handle_auth_done global_id 105297 payload 293 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 msgr2=0x7fb2b4004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 0x7fb2b4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 msgr2=0x7fb2b4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 0x7fb2b4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient: _finish_hunting 0 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 monclient: found mon.a 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40034e0 con 0x7fb2b4005180 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 10 monclient: _finish_auth 0 2024-09-15T02:05:08.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ccf3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ce99d640 5 monclient: authenticate success, global_id 105297 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 0x7fb2b4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.651+0000 7fb2ce99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2ccf3a640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 msgr2=0x7fb2b400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 0x7fb2b400b9d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8068ce0 tx=0x7fb2c0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40037783c72a1ec5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0030480 con 0x7fb2b4005180 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: dump: 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.668 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c002d030 con 0x7fb2b4005180 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2ce99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 1 -- 172.21.15.17:0/1715715387 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0030870 con 0x7fb2b4005180 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2ce99d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2ce99d640 4 set_mon_vals no callback set 2024-09-15T02:05:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.654+0000 7fb2c77fe640 10 monclient: dump: 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 10 monclient: shutdown 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:08.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 -- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 msgr2=0x7fb2b400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 0x7fb2b400b9d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8068ce0 tx=0x7fb2c0009040 comp rx=0 tx=0).stop 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 -- 172.21.15.17:0/1715715387 shutdown_connections 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4005180 0x7fb2b400b9d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b4004780 0x7fb2b4004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/1715715387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b40039d0 0x7fb2b4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 -- 172.21.15.17:0/1715715387 >> 172.21.15.17:0/1715715387 conn(0x7fb2c80a2f70 msgr2=0x7fb2c80a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 -- 172.21.15.17:0/1715715387 shutdown_connections 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.655+0000 7fb2ce99d640 1 -- 172.21.15.17:0/1715715387 wait complete. 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.656+0000 7fb2ce99d640 10 monclient: build_initial_monmap 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.656+0000 7fb2ce99d640 10 monclient: monmap: 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:08.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.656+0000 7fb2ce99d640 1 Processor -- start 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 -- start start 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient: init 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 0x7fb2c813d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2b40039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:08.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2b4004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 0x7fb2c8140b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2b4005180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8122590 con 0x7fb2b4004780 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2c7fff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 msgr2=0x7fb2c813d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8121fd0 con 0x7fb2b4005180 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c811fa60 con 0x7fb2b40039d0 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2c7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 0x7fb2c813d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:08.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ce99d640 10 monclient(hunting): authenticate will time out at 1568.654629s 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35404/0 (socket says 172.21.15.17:35404) 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 1 -- 172.21.15.17:0/977172255 learned_addr learned my addr 172.21.15.17:0/977172255 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): get_auth_request con 0x7fb2b4004780 auth_method 0 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.657+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 0x7fb2c8140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request con 0x7fb2b4005180 auth_method 0 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2ccf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2ccf3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 10 monclient(hunting): handle_auth_done global_id 105300 payload 824 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 -- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 msgr2=0x7fb2c813d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 0x7fb2c813d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 -- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 msgr2=0x7fb2c8140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 0x7fb2c8140b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 10 monclient: _finish_hunting 0 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 monclient: found mon.noname-a 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2bffff640 1 -- 172.21.15.17:0/977172255 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8121cf0 con 0x7fb2b4004780 2024-09-15T02:05:08.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.658+0000 7fb2ccf3a640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 0x7fb2c8140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2bffff640 1 -- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 msgr2=0x7fb2c813cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2bffff640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c8e0 tx=0x7fb2b0003e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b000ced0 con 0x7fb2b4004780 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: dump: 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.678 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: _finish_auth 0 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b0004cc0 con 0x7fb2b4004780 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00073a0 con 0x7fb2b4004780 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 5 monclient: authenticate success, global_id 105300 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 10 monclient: _renew_subs 2024-09-15T02:05:08.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c81222b0 con 0x7fb2b4004780 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c6ffd640 4 set_mon_vals no callback set 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2c4ff9640 10 monclient: dump: 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:08.680 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 10 monclient: _renew_subs 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.660+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c813d390 con 0x7fb2b4004780 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.661+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb2b000a5e0 con 0x7fb2b4004780 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.661+0000 7fb2ce99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.661+0000 7fb2ce99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.661+0000 7fb2ce99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.662+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c80724a0 con 0x7fb2b4004780 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.662+0000 7fb2c4ff9640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 0x7fb2a803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.662+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb2b004bd40 con 0x7fb2b4004780 2024-09-15T02:05:08.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.662+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 0x7fb2a803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:08.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.662+0000 7fb2c7fff640 10 monclient: get_auth_request con 0x7fb2a8038fe0 auth_method 0 2024-09-15T02:05:08.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.665+0000 7fb2c7fff640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 0x7fb2a803b490 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007b50 tx=0x7fb2b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:08.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.667+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb2b0007790 con 0x7fb2b4004780 2024-09-15T02:05:08.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.667+0000 7fb2c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:08.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.667+0000 7fb2c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:08.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.895+0000 7fb2ce99d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:05:08.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.895+0000 7fb2ce99d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:08.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.895+0000 7fb2ce99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:08.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.895+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb2c813d390 con 0x7fb2b4004780 2024-09-15T02:05:08.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.896+0000 7fb2c4ff9640 1 -- 172.21.15.17:0/977172255 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7fb2c813d390 con 0x7fb2b4004780 2024-09-15T02:05:08.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.896+0000 7fb2c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:08.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.896+0000 7fb2c4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:08.899 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":144,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:05:08.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 msgr2=0x7fb2a803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 0x7fb2a803b490 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007b50 tx=0x7fb2b8009040 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 10 monclient: shutdown 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 msgr2=0x7fb2c813cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c8e0 tx=0x7fb2b0003e10 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 shutdown_connections 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb2a8038fe0 0x7fb2a803b490 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb2b4004780 0x7fb2c813cc50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2b4005180 0x7fb2c8140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 --2- 172.21.15.17:0/977172255 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb2b40039d0 0x7fb2c813d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:08.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.897+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 >> 172.21.15.17:0/977172255 conn(0x7fb2c80a2f70 msgr2=0x7fb2c809fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:08.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.898+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 shutdown_connections 2024-09-15T02:05:08.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:08.898+0000 7fb2ce99d640 1 -- 172.21.15.17:0/977172255 wait complete. 2024-09-15T02:05:08.911 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:05:08.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:05:08.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:05:09.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.005+0000 7f424cac9640 10 monclient: get_monmap_and_config 2024-09-15T02:05:09.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.005+0000 7f424cac9640 10 monclient: build_initial_monmap 2024-09-15T02:05:09.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.005+0000 7f424cac9640 10 monclient: monmap: 2024-09-15T02:05:09.008 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:09.007605+0000 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:09.007605+0000 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:09.009 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 1 Processor -- start 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 1 -- start start 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 10 monclient: init 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.006+0000 7f424cac9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4248096c10 0x7f4248099f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-c con 0x7f4248096c10 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424809d3b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f4248135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f42480976f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480983b0 con 0x7f424809d3b0 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4248099260 con 0x7f4248096c10 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424809a460 con 0x7f42480976f0 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424cac9640 10 monclient(hunting): authenticate will time out at 1569.004078s 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f4246d9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 msgr2=0x7f4248135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f4246d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f4248135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f4245d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f424809d3b0 0x7f4248096fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:50760/0 (socket says 172.21.15.17:50760) 2024-09-15T02:05:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f4245d9b640 1 -- 172.21.15.17:0/516688919 learned_addr learned my addr 172.21.15.17:0/516688919 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 --2- 172.21.15.17:0/516688919 >> v2:172.21.15.67:3300/0 conn(0x7f4248096c10 0x7f4248099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): get_auth_request con 0x7f4248096c10 auth_method 0 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient(hunting): handle_auth_done global_id 174130 payload 274 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 -- 172.21.15.17:0/516688919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 msgr2=0x7f4248135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 --2- 172.21.15.17:0/516688919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f4248135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 -- 172.21.15.17:0/516688919 >> v1:172.21.15.17:6789/0 conn(0x7f424809d3b0 legacy=0x7f4248096fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:09.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient: _finish_hunting 0 2024-09-15T02:05:09.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 monclient: found mon.noname-c 2024-09-15T02:05:09.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:09.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:05:09.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.007+0000 7f424659c640 1 -- 172.21.15.17:0/516688919 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424809a5a0 con 0x7f4248096c10 2024-09-15T02:05:09.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.010+0000 7f424659c640 1 -- 172.21.15.17:0/516688919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4248096c10 msgr2=0x7f4248099f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.010+0000 7f424659c640 1 --2- 172.21.15.17:0/516688919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4248096c10 0x7f4248099f20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4238009850 tx=0x7f42380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ff80ca38c2db0fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 1 -- 172.21.15.17:0/516688919 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238032440 con 0x7f4248096c10 2024-09-15T02:05:09.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: dump: 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: _finish_auth 0 2024-09-15T02:05:09.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 1 -- 172.21.15.17:0/516688919 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42380325a0 con 0x7f4248096c10 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 1 -- 172.21.15.17:0/516688919 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238032940 con 0x7f4248096c10 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 5 monclient: authenticate success, global_id 174130 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 4 set_mon_vals no callback set 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424559a640 10 monclient: dump: 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.022 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 10 monclient: shutdown 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 -- 172.21.15.17:0/516688919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4248096c10 msgr2=0x7f4248099f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 --2- 172.21.15.17:0/516688919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4248096c10 0x7f4248099f20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4238009850 tx=0x7f42380090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 -- 172.21.15.17:0/516688919 shutdown_connections 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 --2- 172.21.15.17:0/516688919 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f4248135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 --2- 172.21.15.17:0/516688919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4248096c10 0x7f4248099f20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 -- 172.21.15.17:0/516688919 >> 172.21.15.17:0/516688919 conn(0x7f42480a1000 msgr2=0x7f42480a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 -- 172.21.15.17:0/516688919 shutdown_connections 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.011+0000 7f424cac9640 1 -- 172.21.15.17:0/516688919 wait complete. 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 10 monclient: build_initial_monmap 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 10 monclient: monmap: 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:09.024 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 1 Processor -- start 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 1 -- start start 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 10 monclient: init 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.012+0000 7f424cac9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:05:09.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 0x7f4248135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-a con 0x7f4248096c10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f424813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-c con 0x7f42480976f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424809d3b0 0x7f424813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f424809d3b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4248098e50 con 0x7f4248096c10 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424812f870 con 0x7f424809d3b0 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4248134400 con 0x7f42480976f0 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424cac9640 10 monclient(hunting): authenticate will time out at 1569.010074s 2024-09-15T02:05:09.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4245d9b640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 msgr2=0x7f424813f3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4245d9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f424813f3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 0x7f4248135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 0x7f4248135110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35412/0 (socket says 172.21.15.17:35412) 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 -- 172.21.15.17:0/3349097831 learned_addr learned my addr 172.21.15.17:0/3349097831 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): get_auth_request con 0x7f4248096c10 auth_method 0 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424809d3b0 0x7f424813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): get_auth_request con 0x7f424809d3b0 auth_method 0 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f4246d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient(hunting): handle_auth_done global_id 105303 payload 824 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 msgr2=0x7f424813f3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f42480976f0 0x7f424813f3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424809d3b0 msgr2=0x7f424813f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424809d3b0 0x7f424813f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient: _finish_hunting 0 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 monclient: found mon.noname-a 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.013+0000 7f424659c640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238009390 con 0x7f4248096c10 2024-09-15T02:05:09.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424659c640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 msgr2=0x7f4248135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424659c640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 0x7f4248135110 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4238009850 tx=0x7f4238032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238031990 con 0x7f4248096c10 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 10 monclient: dump: 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 10 monclient: _finish_auth 0 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424cac9640 5 monclient: authenticate success, global_id 105303 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424cac9640 10 monclient: _renew_subs 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238031af0 con 0x7f4248096c10 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.015+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4248135650 con 0x7f4248096c10 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238031e70 con 0x7f4248096c10 2024-09-15T02:05:09.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 10 monclient: dump: 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 10 monclient: _renew_subs 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42481331c0 con 0x7f4248096c10 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4244d99640 4 set_mon_vals no callback set 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 0x7f424812dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4248063c20 con 0x7f424813fe30 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 0x7f424812dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient: get_auth_request con 0x7f424813fe30 auth_method 0 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4238039040 con 0x7f4248096c10 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.016+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f4245d9b640 10 monclient(hunting): handle_auth_done global_id 105306 payload 824 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f4245d9b640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4248134280 con 0x7f424813fe30 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f42367fc640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42280386f0 0x7f422803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f423806c8a0 con 0x7f4248096c10 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f4246d9d640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42280386f0 0x7f422803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.017+0000 7f4246d9d640 10 monclient: get_auth_request con 0x7f42280386f0 auth_method 0 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f4245d9b640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 msgr2=0x7f424812dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f4245d9b640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 0x7f424812dde0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f423c0047d0 tx=0x7f423c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c005d50 con 0x7f424813fe30 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 10 monclient: dump: 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f423c0070f0 con 0x7f424813fe30 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 msgr2=0x7f424812dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 0x7f424812dde0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f423c0047d0 tx=0x7f423c007f00 comp rx=0 tx=0).stop 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 msgr2=0x7f424812dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.019+0000 7f42367fc640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424813fe30 0x7f424812dde0 secure :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f423c0047d0 tx=0x7f423c007f00 comp rx=0 tx=0).stop 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.020+0000 7f4246d9d640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42280386f0 0x7f422803aba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4248132f50 tx=0x7f42400096a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f424cac9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f424cac9640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 0x7f42480298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f424cac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4248134640 con 0x7f424812e320 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 0x7f42480298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient: get_auth_request con 0x7f424812e320 auth_method 0 2024-09-15T02:05:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.025+0000 7f4245d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.026+0000 7f4245d9b640 10 monclient(hunting): handle_auth_done global_id 105309 payload 824 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.026+0000 7f4245d9b640 1 -- 172.21.15.17:0/3349097831 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f42480724a0 con 0x7f424812e320 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f4245d9b640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 msgr2=0x7f42480298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f4245d9b640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 0x7f42480298c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f423c007fd0 tx=0x7f423c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c021630 con 0x7f424812e320 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:09.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 10 monclient: dump: 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f423c021790 con 0x7f424812e320 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 msgr2=0x7f42480298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 0x7f42480298c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f423c007fd0 tx=0x7f423c016000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 msgr2=0x7f42480298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.028+0000 7f42367fc640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f424812e320 0x7f42480298c0 secure :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f423c007fd0 tx=0x7f423c016000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:05:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 144, 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:09.055 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42280386f0 msgr2=0x7f422803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42280386f0 0x7f422803aba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4248132f50 tx=0x7f42400096a0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 10 monclient: shutdown 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 msgr2=0x7f4248135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 --2- 172.21.15.17:0/3349097831 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4248096c10 0x7f4248135110 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4238009850 tx=0x7f4238032000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f4246d9d640 1 -- 172.21.15.17:0/3349097831 reap_dead start 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f4246d9d640 1 -- 172.21.15.17:0/3349097831 reap_dead start 2024-09-15T02:05:09.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 shutdown_connections 2024-09-15T02:05:09.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 >> 172.21.15.17:0/3349097831 conn(0x7f42480a1000 msgr2=0x7f424812e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:09.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 shutdown_connections 2024-09-15T02:05:09.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.029+0000 7f424cac9640 1 -- 172.21.15.17:0/3349097831 wait complete. 2024-09-15T02:05:09.058 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:05:09.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:05:09.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:05:09.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.152+0000 7fa390034640 10 monclient: get_monmap_and_config 2024-09-15T02:05:09.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.152+0000 7fa390034640 10 monclient: build_initial_monmap 2024-09-15T02:05:09.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.152+0000 7fa390034640 10 monclient: monmap: 2024-09-15T02:05:09.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:09.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:09.154552+0000 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:09.154552+0000 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:09.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 Processor -- start 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 -- start start 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient: init 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 0x7fa38806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): picked mon.noname-b con 0x7fa38806eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa38806f440 0x7fa38809bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): picked mon.noname-c con 0x7fa38806f440 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:09.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa38809c4d0 0x7fa38809d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): picked mon.noname-a con 0x7fa38809c4d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38809f6a0 con 0x7fa38809c4d0 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa388069f40 con 0x7fa38806f440 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa388069950 con 0x7fa38806eb30 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.153+0000 7fa390034640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa390034640 10 monclient(hunting): authenticate will time out at 1569.151029s 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38ddd0640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 msgr2=0x7fa38806ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38ddd0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 0x7fa38806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa38809c4d0 0x7fa38809d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa38809c4d0 0x7fa38809d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35438/0 (socket says 172.21.15.17:35438) 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 -- 172.21.15.17:0/2538945688 learned_addr learned my addr 172.21.15.17:0/2538945688 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): get_auth_request con 0x7fa38809c4d0 auth_method 0 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 1 --2- 172.21.15.17:0/2538945688 >> v2:172.21.15.67:3300/0 conn(0x7fa38806f440 0x7fa38809bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): get_auth_request con 0x7fa38806f440 auth_method 0 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_done global_id 105312 payload 274 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 -- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 msgr2=0x7fa38806ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 0x7fa38806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 -- 172.21.15.17:0/2538945688 >> v2:172.21.15.67:3300/0 conn(0x7fa38806f440 msgr2=0x7fa38809bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/2538945688 >> v2:172.21.15.67:3300/0 conn(0x7fa38806f440 0x7fa38809bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient: _finish_hunting 0 2024-09-15T02:05:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 monclient: found mon.noname-a 2024-09-15T02:05:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.154+0000 7fa38e5d1640 1 -- 172.21.15.17:0/2538945688 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38809ad80 con 0x7fa38809c4d0 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38e5d1640 1 -- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38809c4d0 msgr2=0x7fa38809d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38809c4d0 0x7fa38809d960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa3880716f0 tx=0x7fa380009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6c4e1981480ef1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 1 -- 172.21.15.17:0/2538945688 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380007610 con 0x7fa38809c4d0 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: dump: 2024-09-15T02:05:09.169 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: _finish_auth 0 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 1 -- 172.21.15.17:0/2538945688 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38002d050 con 0x7fa38809c4d0 2024-09-15T02:05:09.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 5 monclient: authenticate success, global_id 105312 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 1 -- 172.21.15.17:0/2538945688 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3800079a0 con 0x7fa38809c4d0 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 4 set_mon_vals no callback set 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa38cdce640 10 monclient: dump: 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.171 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 10 monclient: shutdown 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 1 -- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38809c4d0 msgr2=0x7fa38809d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.157+0000 7fa390034640 1 --2- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38809c4d0 0x7fa38809d960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa3880716f0 tx=0x7fa380009840 comp rx=0 tx=0).stop 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 -- 172.21.15.17:0/2538945688 shutdown_connections 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 --2- 172.21.15.17:0/2538945688 >> v2:172.21.15.67:3300/0 conn(0x7fa38806f440 0x7fa38809bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 --2- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38809c4d0 0x7fa38809d960 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 --2- 172.21.15.17:0/2538945688 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806eb30 0x7fa38806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 -- 172.21.15.17:0/2538945688 >> 172.21.15.17:0/2538945688 conn(0x7fa3880712f0 msgr2=0x7fa3880a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 -- 172.21.15.17:0/2538945688 shutdown_connections 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 -- 172.21.15.17:0/2538945688 wait complete. 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 10 monclient: build_initial_monmap 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 10 monclient: monmap: 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:09.173 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 Processor -- start 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 1 -- start start 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.158+0000 7fa390034640 10 monclient: init 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 0x7fa38813d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): picked mon.noname-a con 0x7fa38806eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806f440 0x7fa38813cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): picked mon.noname-c con 0x7fa38806f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38809c4d0 0x7fa388140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): picked mon.noname-b con 0x7fa38809c4d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3881226c0 con 0x7fa38806eb30 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa388122100 con 0x7fa38809c4d0 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38811fbe0 con 0x7fa38806f440 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa390034640 10 monclient(hunting): authenticate will time out at 1569.156427s 2024-09-15T02:05:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38d5cf640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806f440 msgr2=0x7fa38813cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38d5cf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806f440 0x7fa38813cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 0x7fa38813d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 0x7fa38813d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:35444/0 (socket says 172.21.15.17:35444) 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 -- 172.21.15.17:0/82583200 learned_addr learned my addr 172.21.15.17:0/82583200 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): get_auth_request con 0x7fa38806eb30 auth_method 0 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38809c4d0 0x7fa388140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): get_auth_request con 0x7fa38809c4d0 auth_method 0 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38e5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient(hunting): handle_auth_done global_id 105315 payload 824 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806f440 msgr2=0x7fa38813cd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa38806f440 0x7fa38813cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38809c4d0 msgr2=0x7fa388140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38809c4d0 0x7fa388140c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient: _finish_hunting 0 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 monclient: found mon.noname-a 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.159+0000 7fa38ddd0640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa380009ae0 con 0x7fa38806eb30 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa38ddd0640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 msgr2=0x7fa38813d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa38ddd0640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 0x7fa38813d6d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b630 tx=0x7fa3840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384015070 con 0x7fa38806eb30 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: dump: 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.178 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: _finish_auth 0 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384003d90 con 0x7fa38806eb30 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 5 monclient: authenticate success, global_id 105315 2024-09-15T02:05:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 10 monclient: _renew_subs 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384006380 con 0x7fa38806eb30 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa388121e20 con 0x7fa38806eb30 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37dffb640 10 monclient: dump: 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa37ffff640 4 set_mon_vals no callback set 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 10 monclient: _renew_subs 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.162+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38813d460 con 0x7fa38806eb30 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa390034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa390034640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 0x7fa388141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa384010440 con 0x7fa38806eb30 2024-09-15T02:05:09.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa388063b90 con 0x7fa388120f50 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa37dffb640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa36c038740 0x7fa36c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa38404bfc0 con 0x7fa38806eb30 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 0x7fa388141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient: get_auth_request con 0x7fa388120f50 auth_method 0 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa36c038740 0x7fa36c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.163+0000 7fa38e5d1640 10 monclient: get_auth_request con 0x7fa36c038740 auth_method 0 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.164+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_done global_id 174142 payload 824 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.164+0000 7fa38d5cf640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa388122830 con 0x7fa388120f50 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa38d5cf640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 msgr2=0x7fa388141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa38d5cf640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 0x7fa388141180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa3780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800cac0 con 0x7fa388120f50 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 10 monclient: dump: 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.183 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa38e5d1640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa36c038740 0x7fa36c03abf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa38809f9a0 tx=0x7fa380036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa37800cc20 con 0x7fa388120f50 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 msgr2=0x7fa388141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 0x7fa388141180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa3780090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 msgr2=0x7fa388141180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:09.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.166+0000 7fa37dffb640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa388120f50 0x7fa388141180 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa3780090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa390034640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:09.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa390034640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 0x7fa3881416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:09.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa390034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:09.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa388122560 con 0x7fa38813e050 2024-09-15T02:05:09.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa38d5cf640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 0x7fa3881416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa38d5cf640 10 monclient: get_auth_request con 0x7fa38813e050 auth_method 0 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa38d5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa38d5cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.172+0000 7fa38d5cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.173+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.173+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.173+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.173+0000 7fa38d5cf640 10 monclient(hunting): handle_auth_done global_id 174145 payload 824 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.173+0000 7fa38d5cf640 1 -- 172.21.15.17:0/82583200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa3880724a0 con 0x7fa38813e050 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa38d5cf640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 msgr2=0x7fa3881416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa38d5cf640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 0x7fa3881416c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa378019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378005e70 con 0x7fa38813e050 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa37dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:09.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa37dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.175+0000 7fa37dffb640 10 monclient: dump: 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7fa378006050 con 0x7fa38813e050 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:09.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 msgr2=0x7fa3881416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 0x7fa3881416c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa378019000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 msgr2=0x7fa3881416c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa37dffb640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38813e050 0x7fa3881416c0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa38813e740 tx=0x7fa378019000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 144, 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:09.201 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:09.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa36c038740 msgr2=0x7fa36c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa36c038740 0x7fa36c03abf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa38809f9a0 tx=0x7fa380036000 comp rx=0 tx=0).stop 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 10 monclient: shutdown 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 msgr2=0x7fa38813d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 --2- 172.21.15.17:0/82583200 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa38806eb30 0x7fa38813d6d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b630 tx=0x7fa3840091a0 comp rx=0 tx=0).stop 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa38e5d1640 1 -- 172.21.15.17:0/82583200 reap_dead start 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa38e5d1640 1 -- 172.21.15.17:0/82583200 reap_dead start 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 shutdown_connections 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 >> 172.21.15.17:0/82583200 conn(0x7fa3880712f0 msgr2=0x7fa3880a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 shutdown_connections 2024-09-15T02:05:09.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:09.176+0000 7fa390034640 1 -- 172.21.15.17:0/82583200 wait complete. 2024-09-15T02:05:09.204 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:05:09.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:05:29.204 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:05:29.205 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T02:05:29.205 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:05:29.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:05:29.208 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:05:29.208 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:05:29.209 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:05:29.209 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T02:05:29.209 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T02:05:29.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:05:49.210 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:05:49.211 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:05:49.211 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T02:05:49.211 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T02:05:49.211 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:05:49.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.321+0000 7f8912b5d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.321+0000 7f8912b5d640 10 monclient: build_initial_monmap 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.321+0000 7f8912b5d640 10 monclient: monmap: 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:49.322261+0000 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:49.322261+0000 2024-09-15T02:05:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:49.324 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:49.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 Processor -- start 2024-09-15T02:05:49.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 -- start start 2024-09-15T02:05:49.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient: init 2024-09-15T02:05:49.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:49.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T02:05:49.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:49.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f890c09c1b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f890c09c580 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f890c09c9c0 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c09f790 con 0x7f890c09c9c0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c06f780 con 0x7f890c09c580 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c069d40 con 0x7f890c09c1b0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f8912b5d640 10 monclient(hunting): authenticate will time out at 1609.318860s 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.322+0000 7f890bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34318/0 (socket says 172.21.15.17:34318) 2024-09-15T02:05:49.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 learned_addr learned my addr 172.21.15.17:0/2032285784 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:49.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f890c09c580 auth_method 0 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f890bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89110fa640 1 --1- 172.21.15.17:0/2032285784 >> v1:172.21.15.17:6789/0 conn(0x7f890c09c9c0 0x7f890c06f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:40266/0 (socket says 172.21.15.17:40266) 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f890c09c1b0 auth_method 0 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.323+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.325+0000 7f89110fa640 1 -- 172.21.15.17:0/2032285784 >> v1:172.21.15.17:6789/0 conn(0x7f890c09c9c0 legacy=0x7f890c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.325+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.325+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.325+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.325+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f890c09c1b0 auth_method 0 2024-09-15T02:05:49.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f890c09c580 auth_method 0 2024-09-15T02:05:49.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.526+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:49.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f890c09c1b0 auth_method 0 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:49.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:49.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f890c09c580 auth_method 0 2024-09-15T02:05:49.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:49.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:49.927+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:50.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f890c09c1b0 auth_method 0 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:50.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f890c09c580 auth_method 0 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:50.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:50.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:50.728+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:52.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.322+0000 7f890a7fc640 10 monclient(hunting): tick 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.322+0000 7f890a7fc640 1 monclient(hunting): continuing hunt 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 msgr2=0x7f890c09a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.67:3300/0 conn(0x7f890c09c1b0 0x7f890c09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 -- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 msgr2=0x7f890c06eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 --2- 172.21.15.17:0/2032285784 >> v2:172.21.15.17:3300/0 conn(0x7f890c09c580 0x7f890c06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 -- 172.21.15.17:0/2032285784 >> v1:172.21.15.17:6789/0 conn(0x7f890c09c9c0 legacy=0x7f890c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f88ec0036b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88ec006220 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 -- 172.21.15.17:0/2032285784 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c068b60 con 0x7f88ec006220 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 1 -- 172.21.15.17:0/2032285784 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c06f780 con 0x7f88ec0036b0 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f890a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88ec0036b0 auth_method 0 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:52.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.323+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:52.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.325+0000 7f890bfff640 1 -- 172.21.15.17:0/2032285784 >> v1:172.21.15.67:6789/0 conn(0x7f88ec006220 legacy=0x7f88ec0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88ec0036b0 auth_method 0 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:52.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.524+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88ec0036b0 auth_method 0 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:52.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:52.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:52.925+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:05:53.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:53.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88ec0036b0 auth_method 0 2024-09-15T02:05:53.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:53.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:53.726+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:05:54.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265818789 0 0) 0x7f890c068b60 con 0x7f88ec006220 2024-09-15T02:05:54.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): my global_id is 174148 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f80034e0 con 0x7f88ec006220 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3483021895 0 0) 0x7f88f80034e0 con 0x7f88ec006220 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 msgr2=0x7f88ec005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88ec0036b0 0x7f88ec005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec00b010 con 0x7f88ec006220 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.223 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f88f400c680 con 0x7f88ec006220 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f8912b5d640 5 monclient: authenticate success, global_id 174148 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f8912b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: dump: 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.224 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 >> v1:172.21.15.67:6789/0 conn(0x7f88ec006220 legacy=0x7f88ec0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89110fa640 1 -- 172.21.15.17:0/2032285784 reap_dead start 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 0x7f88f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): picked mon.a con 0x7f88f80039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f8004780 0x7f88f8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): picked mon.c con 0x7f88f8004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f88f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): picked mon.b con 0x7f88f8005250 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c06f780 con 0x7f88f80039d0 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f800c210 con 0x7f88f8005250 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f800c3d0 con 0x7f88f8004780 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f8912b5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 0x7f88f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88f80039d0 auth_method 0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f89108f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.218+0000 7f890bfff640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f8004780 0x7f88f8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f88f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f88f8004780 auth_method 0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): get_auth_request con 0x7f88f8005250 auth_method 0 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient(hunting): handle_auth_done global_id 174148 payload 293 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f8004780 msgr2=0x7f88f8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f8004780 0x7f88f8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 msgr2=0x7f88f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f88f800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 monclient: found mon.a 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f80034e0 con 0x7f88f80039d0 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.229 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.219+0000 7f89108f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.222+0000 7f89108f9640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 msgr2=0x7f88f8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.222+0000 7f89108f9640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 0x7f88f8003dc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006fd0 tx=0x7f88fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6946008943e1dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc035410 con 0x7f88f80039d0 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: dump: 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f8912b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc02f030 con 0x7f88f80039d0 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 1 -- 172.21.15.17:0/2032285784 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0357c0 con 0x7f88f80039d0 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f890b7fe640 10 monclient: dump: 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f8912b5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:54.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.225+0000 7f8912b5d640 4 set_mon_vals no callback set 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 10 monclient: shutdown 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 -- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 msgr2=0x7f88f8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 0x7f88f8003dc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006fd0 tx=0x7f88fc008040 comp rx=0 tx=0).stop 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 -- 172.21.15.17:0/2032285784 shutdown_connections 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f88f800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f8004780 0x7f88f8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2032285784 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f80039d0 0x7f88f8003dc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 -- 172.21.15.17:0/2032285784 >> 172.21.15.17:0/2032285784 conn(0x7f890c071300 msgr2=0x7f890c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 -- 172.21.15.17:0/2032285784 shutdown_connections 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 1 -- 172.21.15.17:0/2032285784 wait complete. 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.226+0000 7f8912b5d640 10 monclient: monmap: 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.227+0000 7f8912b5d640 1 Processor -- start 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.227+0000 7f8912b5d640 1 -- start start 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.227+0000 7f8912b5d640 10 monclient: init 2024-09-15T02:05:54.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f88f80039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f88f8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f890c140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f88f8005250 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c120390 con 0x7f88f8004780 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c122080 con 0x7f88f8005250 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c121da0 con 0x7f88f80039d0 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f8912b5d640 10 monclient(hunting): authenticate will time out at 1614.224439s 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33690/0 (socket says 172.21.15.17:33690) 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 -- 172.21.15.17:0/2266996446 learned_addr learned my addr 172.21.15.17:0/2266996446 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55670/0 (socket says 172.21.15.17:55670) 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 10 monclient(hunting): get_auth_request con 0x7f88f80039d0 auth_method 0 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): get_auth_request con 0x7f88f8004780 auth_method 0 2024-09-15T02:05:54.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f890c140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): get_auth_request con 0x7f88f8005250 auth_method 0 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 msgr2=0x7f890c13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 msgr2=0x7f890c13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:05:54.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89108f9640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f89110fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.228+0000 7f890bfff640 10 monclient(hunting): handle_auth_done global_id 105318 payload 824 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 msgr2=0x7f890c13d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 msgr2=0x7f890c140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f890c140ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 monclient: found mon.noname-a 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.229+0000 7f890bfff640 1 -- 172.21.15.17:0/2266996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c122360 con 0x7f88f8004780 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f890bfff640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 msgr2=0x7f890c13cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f890bfff640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006f60 tx=0x7f88f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4015d40 con 0x7f88f8004780 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: dump: 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400c040 con 0x7f88f8004780 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 5 monclient: authenticate success, global_id 105318 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 10 monclient: _renew_subs 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c13e170 con 0x7f88f8004780 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f40216c0 con 0x7f88f8004780 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8908ff9640 10 monclient: dump: 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f890affd640 4 set_mon_vals no callback set 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 10 monclient: _renew_subs 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.231+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c13d220 con 0x7f88f8004780 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.232+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f88f401e070 con 0x7f88f8004780 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8908ff9640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 0x7f88ec03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8912b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8912b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8912b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f88f404d790 con 0x7f88f8004780 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c0a3090 con 0x7f88f8004780 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f89108f9640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 0x7f88ec03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.233+0000 7f89108f9640 10 monclient: get_auth_request con 0x7f88ec0390c0 auth_method 0 2024-09-15T02:05:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.236+0000 7f89108f9640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 0x7f88ec03b570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006cd0 tx=0x7f88fc039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.238+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f890c0a3090 con 0x7f88f8004780 2024-09-15T02:05:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.238+0000 7f8908ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.238+0000 7f8908ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8912b5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8912b5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8912b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f890c0a3090 con 0x7f88f8004780 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8908ff9640 1 -- 172.21.15.17:0/2266996446 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f890c0a3090 con 0x7f88f8004780 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8908ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.475+0000 7f8908ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:54.477 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 msgr2=0x7f88ec03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 0x7f88ec03b570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006cd0 tx=0x7f88fc039000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 10 monclient: shutdown 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 msgr2=0x7f890c13cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006f60 tx=0x7f88f4008040 comp rx=0 tx=0).stop 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 shutdown_connections 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f88ec0390c0 0x7f88ec03b570 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f88f8004780 0x7f890c13cce0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f8005250 0x7f890c140ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 --2- 172.21.15.17:0/2266996446 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f88f80039d0 0x7f890c13d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 >> 172.21.15.17:0/2266996446 conn(0x7f890c071300 msgr2=0x7f890c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 shutdown_connections 2024-09-15T02:05:54.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.477+0000 7f8912b5d640 1 -- 172.21.15.17:0/2266996446 wait complete. 2024-09-15T02:05:54.489 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:05:54.489 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:05:54.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:05:54.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:54.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 10 monclient: monmap: 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:54.585861+0000 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:54.585861+0000 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:54.587 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 1 Processor -- start 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 1 -- start start 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.585+0000 7faeffb9d640 10 monclient: init 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faef80a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faef8132040 0x7faef80987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-c con 0x7faef8132040 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:54.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faef8136dd0 0x7faef81391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-a con 0x7faef8136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80a01e0 con 0x7faef80a58b0 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80a1580 con 0x7faef8136dd0 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80a5c80 con 0x7faef8132040 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faeffb9d640 10 monclient(hunting): authenticate will time out at 1614.582370s 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd939640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7faef80a58b0 0x7faef80980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55794/0 (socket says 172.21.15.17:55794) 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd939640 1 -- 172.21.15.17:0/1806743013 learned_addr learned my addr 172.21.15.17:0/1806743013 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefe13a640 1 --2- 172.21.15.17:0/1806743013 >> v2:172.21.15.17:3300/0 conn(0x7faef8136dd0 0x7faef81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefe13a640 10 monclient(hunting): get_auth_request con 0x7faef8136dd0 auth_method 0 2024-09-15T02:05:54.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefe13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefe13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefe13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 1 --2- 172.21.15.17:0/1806743013 >> v2:172.21.15.67:3300/0 conn(0x7faef8132040 0x7faef80987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): get_auth_request con 0x7faef8132040 auth_method 0 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.586+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 10 monclient(hunting): handle_auth_done global_id 174154 payload 274 2024-09-15T02:05:54.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 1 -- 172.21.15.17:0/1806743013 >> v2:172.21.15.17:3300/0 conn(0x7faef8136dd0 msgr2=0x7faef81391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 1 --2- 172.21.15.17:0/1806743013 >> v2:172.21.15.17:3300/0 conn(0x7faef8136dd0 0x7faef81391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 1 -- 172.21.15.17:0/1806743013 >> v1:172.21.15.17:6789/0 conn(0x7faef80a58b0 legacy=0x7faef80980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 1 monclient: found mon.noname-c 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:05:54.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.587+0000 7faefd138640 1 -- 172.21.15.17:0/1806743013 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8132440 con 0x7faef8132040 2024-09-15T02:05:54.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefd138640 1 -- 172.21.15.17:0/1806743013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef8132040 msgr2=0x7faef80987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefd138640 1 --2- 172.21.15.17:0/1806743013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef8132040 0x7faef80987b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faef809f780 tx=0x7faee80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a61dbfdf3dcb4b0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 1 -- 172.21.15.17:0/1806743013 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee802ae00 con 0x7faef8132040 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: dump: 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.600 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 1 -- 172.21.15.17:0/1806743013 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee8032440 con 0x7faef8132040 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 5 monclient: authenticate success, global_id 174154 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 1 -- 172.21.15.17:0/1806743013 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee80327e0 con 0x7faef8132040 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 4 set_mon_vals no callback set 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faefc937640 10 monclient: dump: 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.602 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 10 monclient: shutdown 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 1 -- 172.21.15.17:0/1806743013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef8132040 msgr2=0x7faef80987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.590+0000 7faeffb9d640 1 --2- 172.21.15.17:0/1806743013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef8132040 0x7faef80987b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faef809f780 tx=0x7faee80090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 -- 172.21.15.17:0/1806743013 shutdown_connections 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 --2- 172.21.15.17:0/1806743013 >> v2:172.21.15.17:3300/0 conn(0x7faef8136dd0 0x7faef81391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 --2- 172.21.15.17:0/1806743013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef8132040 0x7faef80987b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 -- 172.21.15.17:0/1806743013 >> 172.21.15.17:0/1806743013 conn(0x7faef8070ed0 msgr2=0x7faef80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 -- 172.21.15.17:0/1806743013 shutdown_connections 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 -- 172.21.15.17:0/1806743013 wait complete. 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 10 monclient: monmap: 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 Processor -- start 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 1 -- start start 2024-09-15T02:05:54.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.591+0000 7faeffb9d640 10 monclient: init 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80a58b0 0x7faef8141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-b con 0x7faef80a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faef8132040 0x7faef8141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-c con 0x7faef8132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): picked mon.noname-a con 0x7faef8136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef811bbc0 con 0x7faef8136dd0 2024-09-15T02:05:54.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8134c30 con 0x7faef80a58b0 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef8098dc0 con 0x7faef8132040 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faeffb9d640 10 monclient(hunting): authenticate will time out at 1614.588385s 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faef8132040 0x7faef8141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55694/0 (socket says 172.21.15.17:55694) 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 learned_addr learned my addr 172.21.15.17:0/2040709607 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): get_auth_request con 0x7faef8136dd0 auth_method 0 2024-09-15T02:05:54.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): get_auth_request con 0x7faef8132040 auth_method 0 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd939640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80a58b0 0x7faef8141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd939640 10 monclient(hunting): get_auth_request con 0x7faef80a58b0 auth_method 0 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient(hunting): handle_auth_done global_id 105324 payload 824 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faef8132040 msgr2=0x7faef8141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faef8132040 0x7faef8141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80a58b0 msgr2=0x7faef8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80a58b0 0x7faef8141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 monclient: found mon.noname-a 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.592+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009390 con 0x7faef8136dd0 2024-09-15T02:05:54.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 msgr2=0x7faef8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faefe13a640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7faef0009fa0 tx=0x7faef0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000ee60 con 0x7faef8136dd0 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: dump: 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.609 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef0013590 con 0x7faef8136dd0 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0013950 con 0x7faef8136dd0 2024-09-15T02:05:54.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 5 monclient: authenticate success, global_id 105324 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 10 monclient: _renew_subs 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef80a3350 con 0x7faef8136dd0 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeeffff640 4 set_mon_vals no callback set 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeedffb640 10 monclient: dump: 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.611 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 10 monclient: _renew_subs 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.595+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef8136320 con 0x7faef8136dd0 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faef0013cc0 con 0x7faef8136dd0 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeffb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeffb9d640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 0x7faef81333e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8063b20 con 0x7faef8133010 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeedffb640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faedc038740 0x7faedc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 0x7faef81333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient: get_auth_request con 0x7faef8133010 auth_method 0 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faef004cdb0 con 0x7faef8136dd0 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 10 monclient(hunting): handle_auth_done global_id 105327 payload 824 2024-09-15T02:05:54.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd939640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faef80724a0 con 0x7faef8133010 2024-09-15T02:05:54.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd138640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faedc038740 0x7faedc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.596+0000 7faefd138640 10 monclient: get_auth_request con 0x7faedc038740 auth_method 0 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.598+0000 7faefd939640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 msgr2=0x7faef81333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faefd939640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 0x7faef81333e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faef400d530 tx=0x7faef4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4014910 con 0x7faef8133010 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 10 monclient: dump: 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faef4014a70 con 0x7faef8133010 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:54.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 msgr2=0x7faef81333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 0x7faef81333e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faef400d530 tx=0x7faef4008040 comp rx=0 tx=0).stop 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 msgr2=0x7faef81333e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faeedffb640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133010 0x7faef81333e0 secure :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faef400d530 tx=0x7faef4008040 comp rx=0 tx=0).stop 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.599+0000 7faefd138640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faedc038740 0x7faedc03abf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faef80a60e0 tx=0x7faee8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faeffb9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faeffb9d640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 0x7faef81374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faeffb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef809fc90 con 0x7faef8133920 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 0x7faef81374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient: get_auth_request con 0x7faef8133920 auth_method 0 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 10 monclient(hunting): handle_auth_done global_id 105330 payload 824 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.605+0000 7faefd939640 1 -- 172.21.15.17:0/2040709607 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faef8135010 con 0x7faef8133920 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faefd939640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 msgr2=0x7faef81374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faefd939640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 0x7faef81374c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7faef4019000 tx=0x7faef401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4020930 con 0x7faef8133920 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faeedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faeedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.607+0000 7faeedffb640 10 monclient: dump: 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.619 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7faef4020a90 con 0x7faef8133920 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 msgr2=0x7faef81374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 0x7faef81374c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7faef4019000 tx=0x7faef401c000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 msgr2=0x7faef81374c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeedffb640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8133920 0x7faef81374c0 secure :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7faef4019000 tx=0x7faef401c000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:54.621 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.622 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.623 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:54.624 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:54.625 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.626 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.628 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.630 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.632 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:54.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.635 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:54.635 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faedc038740 msgr2=0x7faedc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faedc038740 0x7faedc03abf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faef80a60e0 tx=0x7faee8032000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 10 monclient: shutdown 2024-09-15T02:05:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.636 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 msgr2=0x7faef8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7faef0009fa0 tx=0x7faef0009900 comp rx=0 tx=0).stop 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faefe13a640 1 -- 172.21.15.17:0/2040709607 reap_dead start 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 shutdown_connections 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 --2- 172.21.15.17:0/2040709607 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faef8136dd0 0x7faef8132ad0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 >> 172.21.15.17:0/2040709607 conn(0x7faef8070ed0 msgr2=0x7faef80a4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 shutdown_connections 2024-09-15T02:05:54.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.608+0000 7faeffb9d640 1 -- 172.21.15.17:0/2040709607 wait complete. 2024-09-15T02:05:54.637 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:54.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:05:54.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.733+0000 7fbcd5c2b640 10 monclient: get_monmap_and_config 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.733+0000 7fbcd5c2b640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.733+0000 7fbcd5c2b640 10 monclient: monmap: 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:54.734377+0000 2024-09-15T02:05:54.735 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:54.734377+0000 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:54.736 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 Processor -- start 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 -- start start 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient: init 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcd009c4f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd009c8c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcd006eb30 0x7fbcd006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd006eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd006a390 con 0x7fbcd009c4f0 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd0099f50 con 0x7fbcd006eb30 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd009f790 con 0x7fbcd009c8c0 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcd5c2b640 10 monclient(hunting): authenticate will time out at 1614.730864s 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.734+0000 7fbcceffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbccf7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbcd009c4f0 0x7fbcd009a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55800/0 (socket says 172.21.15.17:55800) 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33702/0 (socket says 172.21.15.17:33702) 2024-09-15T02:05:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbccf7fe640 1 -- 172.21.15.17:0/538203826 learned_addr learned my addr 172.21.15.17:0/538203826 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): get_auth_request con 0x7fbcd009c8c0 auth_method 0 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient(hunting): handle_auth_done global_id 184103 payload 274 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 -- 172.21.15.17:0/538203826 >> v2:172.21.15.67:3300/0 conn(0x7fbcd006eb30 msgr2=0x7fbcd006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbccffff640 1 --2- 172.21.15.17:0/538203826 >> v2:172.21.15.67:3300/0 conn(0x7fbcd006eb30 0x7fbcd006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 --2- 172.21.15.17:0/538203826 >> v2:172.21.15.67:3300/0 conn(0x7fbcd006eb30 0x7fbcd006ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 -- 172.21.15.17:0/538203826 >> v1:172.21.15.17:6789/0 conn(0x7fbcd009c4f0 legacy=0x7fbcd009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 monclient: found mon.noname-b 2024-09-15T02:05:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.735+0000 7fbcceffd640 1 -- 172.21.15.17:0/538203826 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd009b0a0 con 0x7fbcd009c8c0 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcceffd640 1 -- 172.21.15.17:0/538203826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 msgr2=0x7fbcd009e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcceffd640 1 --2- 172.21.15.17:0/538203826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009850 tx=0x7fbcc40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6af987149ce6e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 1 -- 172.21.15.17:0/538203826 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4032440 con 0x7fbcd009c8c0 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: dump: 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.747 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 5 monclient: authenticate success, global_id 184103 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 1 -- 172.21.15.17:0/538203826 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc40325a0 con 0x7fbcd009c8c0 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 1 -- 172.21.15.17:0/538203826 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4032940 con 0x7fbcd009c8c0 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 4 set_mon_vals no callback set 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcce7fc640 10 monclient: dump: 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.749 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.750 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.750 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.750 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 10 monclient: shutdown 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/538203826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 msgr2=0x7fbcd009e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/538203826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009850 tx=0x7fbcc40090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/538203826 shutdown_connections 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/538203826 >> v2:172.21.15.67:3300/0 conn(0x7fbcd006eb30 0x7fbcd006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/538203826 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd009c8c0 0x7fbcd009e9e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/538203826 >> 172.21.15.17:0/538203826 conn(0x7fbcd0071300 msgr2=0x7fbcd00a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/538203826 shutdown_connections 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.738+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/538203826 wait complete. 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.739+0000 7fbcd5c2b640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.739+0000 7fbcd5c2b640 10 monclient: monmap: 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.739+0000 7fbcd5c2b640 1 Processor -- start 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.739+0000 7fbcd5c2b640 1 -- start start 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.739+0000 7fbcd5c2b640 10 monclient: init 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 0x7fbcd0133ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd009c4f0 0x7fbcd0134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd009c4f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 0x7fbcd011da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd009c8c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0134c40 con 0x7fbcd009c8c0 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd011f3d0 con 0x7fbcd009c4f0 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0120f00 con 0x7fbcd006eb30 2024-09-15T02:05:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcd5c2b640 10 monclient(hunting): authenticate will time out at 1614.736516s 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 0x7fbcd0133ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 0x7fbcd011da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 0x7fbcd0133ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33718/0 (socket says 172.21.15.17:33718) 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 1 -- 172.21.15.17:0/3330493096 learned_addr learned my addr 172.21.15.17:0/3330493096 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request con 0x7fbcd006eb30 auth_method 0 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): get_auth_request con 0x7fbcd009c8c0 auth_method 0 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd009c4f0 0x7fbcd0134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): get_auth_request con 0x7fbcd009c4f0 auth_method 0 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient(hunting): handle_auth_done global_id 105333 payload 824 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 msgr2=0x7fbcd0133ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 0x7fbcd0133ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd009c4f0 msgr2=0x7fbcd0134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd009c4f0 0x7fbcd0134220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 monclient: found mon.noname-a 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4009390 con 0x7fbcd009c8c0 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.740+0000 7fbccf7fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd006eb30 0x7fbcd0133ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:54.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 msgr2=0x7fbcd011da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbccffff640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 0x7fbcd011da50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800ce50 tx=0x7fbcc80077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800e610 con 0x7fbcd009c8c0 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: dump: 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.758 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800e770 con 0x7fbcd009c8c0 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 5 monclient: authenticate success, global_id 105333 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 10 monclient: _renew_subs 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd01211e0 con 0x7fbcd009c8c0 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800eb10 con 0x7fbcd009c8c0 2024-09-15T02:05:54.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcb77fe640 10 monclient: dump: 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 10 monclient: _renew_subs 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbccdffb640 4 set_mon_vals no callback set 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.743+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd011ffa0 con 0x7fbcd009c8c0 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcd5c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 0x7fbcd011e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0063bb0 con 0x7fbcd011df90 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbcc8003020 con 0x7fbcd009c8c0 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 0x7fbcd011e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient: get_auth_request con 0x7fbcd011df90 auth_method 0 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcb77fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcbc038740 0x7fbcbc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbcc8012070 con 0x7fbcd009c8c0 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_done global_id 184109 payload 824 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbccf7fe640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbcd00724a0 con 0x7fbcd011df90 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcceffd640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcbc038740 0x7fbcbc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.744+0000 7fbcceffd640 10 monclient: get_auth_request con 0x7fbcbc038740 auth_method 0 2024-09-15T02:05:54.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbccf7fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 msgr2=0x7fbcd011e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbccf7fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 0x7fbcd011e360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0007320 tx=0x7fbcc0004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000fd50 con 0x7fbcd011df90 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 10 monclient: dump: 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbcc000feb0 con 0x7fbcd011df90 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 msgr2=0x7fbcd011e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 0x7fbcd011e360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0007320 tx=0x7fbcc0004960 comp rx=0 tx=0).stop 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 msgr2=0x7fbcd011e360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcb77fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011df90 0x7fbcd011e360 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0007320 tx=0x7fbcc0004960 comp rx=0 tx=0).stop 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.747+0000 7fbcceffd640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcbc038740 0x7fbcbc03abf0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbcd004b6a0 tx=0x7fbcc4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.766 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbcd5c2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 0x7fbcd00a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbcd5c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0067220 con 0x7fbcd011e8a0 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 0x7fbcd00a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient: get_auth_request con 0x7fbcd011e8a0 auth_method 0 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_done global_id 184112 payload 824 2024-09-15T02:05:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.753+0000 7fbccf7fe640 1 -- 172.21.15.17:0/3330493096 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbcd0121660 con 0x7fbcd011e8a0 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.755+0000 7fbccf7fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 msgr2=0x7fbcd00a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.755+0000 7fbccf7fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 0x7fbcd00a1460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0017000 tx=0x7fbcc0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0021950 con 0x7fbcd011e8a0 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 10 monclient: dump: 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.768 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fbcc0021ab0 con 0x7fbcd011e8a0 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 msgr2=0x7fbcd00a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 0x7fbcd00a1460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0017000 tx=0x7fbcc0018000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 msgr2=0x7fbcd00a1460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcb77fe640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbcd011e8a0 0x7fbcd00a1460 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0017000 tx=0x7fbcc0018000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:05:54.770 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:54.773 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.774 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:54.775 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.776 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.779 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.780 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.781 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:54.782 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcbc038740 msgr2=0x7fbcbc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbcbc038740 0x7fbcbc03abf0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbcd004b6a0 tx=0x7fbcc4032000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 10 monclient: shutdown 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 msgr2=0x7fbcd011da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 --2- 172.21.15.17:0/3330493096 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbcd009c8c0 0x7fbcd011da50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800ce50 tx=0x7fbcc80077b0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 reap_dead start 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbccffff640 1 -- 172.21.15.17:0/3330493096 reap_dead start 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 shutdown_connections 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 >> 172.21.15.17:0/3330493096 conn(0x7fbcd0071300 msgr2=0x7fbcd0134e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 shutdown_connections 2024-09-15T02:05:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.756+0000 7fbcd5c2b640 1 -- 172.21.15.17:0/3330493096 wait complete. 2024-09-15T02:05:54.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:54.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:05:54.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:05:54.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.880+0000 7f603ef5d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:54.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.880+0000 7f603ef5d640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.880+0000 7f603ef5d640 10 monclient: monmap: 2024-09-15T02:05:54.882 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:54.881310+0000 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:54.881310+0000 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:54.883 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 Processor -- start 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 -- start start 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient: init 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7f60380a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f603809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 0x7f6038139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6038069c90 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380a5b50 con 0x7f603809ccf0 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380a18b0 con 0x7f6038069c90 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603809c5f0 con 0x7f60380a5780 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ef5d640 10 monclient(hunting): authenticate will time out at 1614.877778s 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33764/0 (socket says 172.21.15.17:33764) 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 1 -- 172.21.15.17:0/539861721 learned_addr learned my addr 172.21.15.17:0/539861721 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 10 monclient(hunting): get_auth_request con 0x7f60380a5780 auth_method 0 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.881+0000 7f603ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603d4fa640 1 --2- 172.21.15.17:0/539861721 >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 0x7f6038139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient(hunting): handle_auth_done global_id 184115 payload 274 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 1 -- 172.21.15.17:0/539861721 >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 msgr2=0x7f6038139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 1 --2- 172.21.15.17:0/539861721 >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 0x7f6038139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 1 -- 172.21.15.17:0/539861721 >> v1:172.21.15.67:6789/0 conn(0x7f603809ccf0 legacy=0x7f6038069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 1 monclient: found mon.noname-b 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603ccf9640 1 -- 172.21.15.17:0/539861721 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603809d210 con 0x7f60380a5780 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603d4fa640 10 monclient: get_auth_request con 0x7f6038069c90 auth_method 0 2024-09-15T02:05:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.882+0000 7f603d4fa640 1 --2- 172.21.15.17:0/539861721 >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 0x7f6038139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:05:54.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ccf9640 1 -- 172.21.15.17:0/539861721 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 msgr2=0x7f603809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ccf9640 1 --2- 172.21.15.17:0/539861721 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f602c009910 tx=0x7f602c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cef44563d2219fae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 1 -- 172.21.15.17:0/539861721 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c032440 con 0x7f60380a5780 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: dump: 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.898 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 1 -- 172.21.15.17:0/539861721 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c0325a0 con 0x7f60380a5780 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 5 monclient: authenticate success, global_id 184115 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 1 -- 172.21.15.17:0/539861721 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c032980 con 0x7f60380a5780 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 4 set_mon_vals no callback set 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f60377fe640 10 monclient: dump: 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 10 monclient: shutdown 2024-09-15T02:05:54.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 -- 172.21.15.17:0/539861721 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 msgr2=0x7f603809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 --2- 172.21.15.17:0/539861721 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f602c009910 tx=0x7f602c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 -- 172.21.15.17:0/539861721 shutdown_connections 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 --2- 172.21.15.17:0/539861721 >> v2:172.21.15.67:3300/0 conn(0x7f6038069c90 0x7f6038139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 --2- 172.21.15.17:0/539861721 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f603809bc90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 -- 172.21.15.17:0/539861721 >> 172.21.15.17:0/539861721 conn(0x7f6038070e10 msgr2=0x7f60380a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 -- 172.21.15.17:0/539861721 shutdown_connections 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.885+0000 7f603ef5d640 1 -- 172.21.15.17:0/539861721 wait complete. 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.886+0000 7f603ef5d640 10 monclient: build_initial_monmap 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.886+0000 7f603ef5d640 10 monclient: monmap: 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:54.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.886+0000 7f603ef5d640 1 Processor -- start 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.886+0000 7f603ef5d640 1 -- start start 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.886+0000 7f603ef5d640 10 monclient: init 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6038069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:54.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603809ccf0 0x7f603813d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7f603809ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f6038141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7f60380a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6038122cf0 con 0x7f6038069c90 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6038122730 con 0x7f603809ccf0 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6038120250 con 0x7f60380a5780 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ef5d640 10 monclient(hunting): authenticate will time out at 1614.883303s 2024-09-15T02:05:54.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f6038141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55742/0 (socket says 172.21.15.17:55742) 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f6038141270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33780/0 (socket says 172.21.15.17:33780) 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 -- 172.21.15.17:0/3992050835 learned_addr learned my addr 172.21.15.17:0/3992050835 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): get_auth_request con 0x7f6038069c90 auth_method 0 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): get_auth_request con 0x7f60380a5780 auth_method 0 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603809ccf0 0x7f603813d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): get_auth_request con 0x7f603809ccf0 auth_method 0 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient(hunting): handle_auth_done global_id 105336 payload 824 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 msgr2=0x7f6038141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f6038141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603809ccf0 msgr2=0x7f603813d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603809ccf0 0x7f603813d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient: _finish_hunting 0 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 monclient: found mon.noname-a 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603ccf9640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c009450 con 0x7f6038069c90 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.887+0000 7f603d4fa640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f60380a5780 0x7f6038141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ccf9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 msgr2=0x7f603813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ccf9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f602c009210 tx=0x7f602c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c000c40 con 0x7f6038069c90 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: dump: 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: _finish_auth 0 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c031c70 con 0x7f6038069c90 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 5 monclient: authenticate success, global_id 105336 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 10 monclient: _renew_subs 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6038122450 con 0x7f6038069c90 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c03c400 con 0x7f6038069c90 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6034ff9640 10 monclient: dump: 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 10 monclient: _renew_subs 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f6036ffd640 4 set_mon_vals no callback set 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.890+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6038122ac0 con 0x7f6038069c90 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603ef5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603ef5d640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 0x7f60381417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6038063b20 con 0x7f6038121580 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f602c03c830 con 0x7f6038069c90 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6034ff9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f601c038740 0x7f601c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f602c06d060 con 0x7f6038069c90 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 0x7f60381417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient: get_auth_request con 0x7f6038121580 auth_method 0 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603d4fa640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f601c038740 0x7f601c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f603d4fa640 10 monclient: get_auth_request con 0x7f601c038740 auth_method 0 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 10 monclient(hunting): handle_auth_done global_id 174166 payload 824 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.891+0000 7f6037fff640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6038122e60 con 0x7f6038121580 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6037fff640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 msgr2=0x7f60381417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6037fff640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 0x7f60381417b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f603813ed70 tx=0x7f6024007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f603d4fa640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f601c038740 0x7f601c03abf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6030007fd0 tx=0x7f60300077a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602400d910 con 0x7f6038121580 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 10 monclient: dump: 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f602400da70 con 0x7f6038121580 2024-09-15T02:05:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:54.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:54.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 msgr2=0x7f60381417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 0x7f60381417b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f603813ed70 tx=0x7f6024007e10 comp rx=0 tx=0).stop 2024-09-15T02:05:54.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 msgr2=0x7f60381417b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.894+0000 7f6034ff9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6038121580 0x7f60381417b0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f603813ed70 tx=0x7f6024007e10 comp rx=0 tx=0).stop 2024-09-15T02:05:54.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.900+0000 7f603ef5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.900+0000 7f603ef5d640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 0x7f6038141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.900+0000 7f603ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.900+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60381208d0 con 0x7f603813e680 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 0x7f6038141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient: get_auth_request con 0x7f603813e680 auth_method 0 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 10 monclient(hunting): handle_auth_done global_id 174169 payload 824 2024-09-15T02:05:54.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.901+0000 7f6037fff640 1 -- 172.21.15.17:0/3992050835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f60380724a0 con 0x7f603813e680 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.903+0000 7f6037fff640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 msgr2=0x7f6038141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.903+0000 7f6037fff640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 0x7f6038141cf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6024019000 tx=0x7f602401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6024014960 con 0x7f603813e680 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 10 monclient: dump: 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:54.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f6024014ac0 con 0x7f603813e680 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 msgr2=0x7f6038141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 0x7f6038141cf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6024019000 tx=0x7f602401a000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 msgr2=0x7f6038141cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f6034ff9640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f603813e680 0x7f6038141cf0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6024019000 tx=0x7f602401a000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.917 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:54.930 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f601c038740 msgr2=0x7f601c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f601c038740 0x7f601c03abf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6030007fd0 tx=0x7f60300077a0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 10 monclient: shutdown 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 msgr2=0x7f603813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f602c009210 tx=0x7f602c032000 comp rx=0 tx=0).stop 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603d4fa640 1 -- 172.21.15.17:0/3992050835 reap_dead start 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 shutdown_connections 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 --2- 172.21.15.17:0/3992050835 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6038069c90 0x7f603813dd00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 >> 172.21.15.17:0/3992050835 conn(0x7f6038070e10 msgr2=0x7f6038098d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.904+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 shutdown_connections 2024-09-15T02:05:54.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:54.905+0000 7f603ef5d640 1 -- 172.21.15.17:0/3992050835 wait complete. 2024-09-15T02:05:54.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:54.932 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:05:54.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:05:55.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.027+0000 7f30e4e89640 10 monclient: get_monmap_and_config 2024-09-15T02:05:55.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.027+0000 7f30e4e89640 10 monclient: build_initial_monmap 2024-09-15T02:05:55.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.027+0000 7f30e4e89640 10 monclient: monmap: 2024-09-15T02:05:55.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:55.028417+0000 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:55.028417+0000 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:55.030 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:55.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 Processor -- start 2024-09-15T02:05:55.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 -- start start 2024-09-15T02:05:55.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient: init 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30e009c0d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-b con 0x7f30e009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 0x7f30e006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-a con 0x7f30e006eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00699c0 con 0x7f30e009c0d0 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e006a5e0 con 0x7f30e006eb30 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e006a290 con 0x7f30e009e970 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30e4e89640 10 monclient(hunting): authenticate will time out at 1615.024882s 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30ddd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.028+0000 7f30ded9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 0x7f30e006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33784/0 (socket says 172.21.15.17:33784) 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 -- 172.21.15.17:0/3501451090 learned_addr learned my addr 172.21.15.17:0/3501451090 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 0x7f30e006ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55754/0 (socket says 172.21.15.17:55754) 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): get_auth_request con 0x7f30e009e970 auth_method 0 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): get_auth_request con 0x7f30e006eb30 auth_method 0 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_done global_id 184121 payload 274 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 -- 172.21.15.17:0/3501451090 >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 msgr2=0x7f30e006ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 --2- 172.21.15.17:0/3501451090 >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 0x7f30e006ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 -- 172.21.15.17:0/3501451090 >> v1:172.21.15.17:6789/0 conn(0x7f30e009c0d0 legacy=0x7f30e009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 monclient: found mon.noname-b 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.029+0000 7f30ddd9b640 1 -- 172.21.15.17:0/3501451090 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e009ed40 con 0x7f30e009e970 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30ddd9b640 1 -- 172.21.15.17:0/3501451090 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 msgr2=0x7f30e009c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30ddd9b640 1 --2- 172.21.15.17:0/3501451090 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009910 tx=0x7f30d40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6d6e02e8ed805cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 1 -- 172.21.15.17:0/3501451090 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402aea0 con 0x7f30e009e970 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: dump: 2024-09-15T02:05:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: _finish_auth 0 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 1 -- 172.21.15.17:0/3501451090 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d4032440 con 0x7f30e009e970 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 5 monclient: authenticate success, global_id 184121 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 4 set_mon_vals no callback set 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 1 -- 172.21.15.17:0/3501451090 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4032820 con 0x7f30e009e970 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30dd59a640 10 monclient: dump: 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:55.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 10 monclient: shutdown 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 -- 172.21.15.17:0/3501451090 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 msgr2=0x7f30e009c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 --2- 172.21.15.17:0/3501451090 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009910 tx=0x7f30d40091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 -- 172.21.15.17:0/3501451090 shutdown_connections 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 --2- 172.21.15.17:0/3501451090 >> v2:172.21.15.17:3300/0 conn(0x7f30e006eb30 0x7f30e006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 --2- 172.21.15.17:0/3501451090 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e009c4a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 -- 172.21.15.17:0/3501451090 >> 172.21.15.17:0/3501451090 conn(0x7f30e00712f0 msgr2=0x7f30e00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.032+0000 7f30e4e89640 1 -- 172.21.15.17:0/3501451090 shutdown_connections 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 1 -- 172.21.15.17:0/3501451090 wait complete. 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 10 monclient: build_initial_monmap 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 10 monclient: monmap: 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 1 Processor -- start 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 1 -- start start 2024-09-15T02:05:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.033+0000 7f30e4e89640 10 monclient: init 2024-09-15T02:05:55.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:55.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:05:55.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 0x7f30e0141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-b con 0x7f30e006eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 0x7f30e0133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-a con 0x7f30e009c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): picked mon.noname-c con 0x7f30e009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e009cd80 con 0x7f30e009c0d0 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0134c80 con 0x7f30e006eb30 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0121570 con 0x7f30e009e970 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30e4e89640 10 monclient(hunting): authenticate will time out at 1615.030315s 2024-09-15T02:05:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 0x7f30e0133e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33790/0 (socket says 172.21.15.17:33790) 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 0x7f30e0133e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55762/0 (socket says 172.21.15.17:55762) 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 -- 172.21.15.17:0/2119950760 learned_addr learned my addr 172.21.15.17:0/2119950760 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): get_auth_request con 0x7f30e009e970 auth_method 0 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): get_auth_request con 0x7f30e009c0d0 auth_method 0 2024-09-15T02:05:55.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30de59c640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 0x7f30e0141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient(hunting): handle_auth_done global_id 184124 payload 824 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 -- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 msgr2=0x7f30e0141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 0x7f30e0141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 -- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 msgr2=0x7f30e0133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:55.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 0x7f30e0133e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient: _finish_hunting 0 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 monclient: found mon.noname-c 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30ded9d640 1 -- 172.21.15.17:0/2119950760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4009450 con 0x7f30e009e970 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30de59c640 10 monclient: get_auth_request con 0x7f30e006eb30 auth_method 0 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.034+0000 7f30de59c640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 0x7f30e0141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.036+0000 7f30ded9d640 1 -- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 msgr2=0x7f30e011da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.036+0000 7f30ded9d640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009df0 tx=0x7f30d800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8014480 con 0x7f30e009e970 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:55.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: dump: 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: _finish_auth 0 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 5 monclient: authenticate success, global_id 184124 2024-09-15T02:05:55.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d800f040 con 0x7f30e009e970 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8014830 con 0x7f30e009e970 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 10 monclient: _renew_subs 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e011f9f0 con 0x7f30e009e970 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30dcd99640 4 set_mon_vals no callback set 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30ce7fc640 10 monclient: dump: 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 10 monclient: _renew_subs 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.037+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e0120fb0 con 0x7f30e009e970 2024-09-15T02:05:55.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f30d8014be0 con 0x7f30e009e970 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30e4e89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30e4e89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30e4e89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00724a0 con 0x7f30e009e970 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30ce7fc640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 0x7f30c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f30d804cf50 con 0x7f30e009e970 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30de59c640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 0x7f30c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.038+0000 7f30de59c640 10 monclient: get_auth_request con 0x7f30c0038770 auth_method 0 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.041+0000 7f30de59c640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 0x7f30c003ac20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f30d0002750 tx=0x7f30d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.043+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f30d801aaf0 con 0x7f30e009e970 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.043+0000 7f30ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:55.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.043+0000 7f30ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:55.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.273+0000 7f30e4e89640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:05:55.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.273+0000 7f30e4e89640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:05:55.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.273+0000 7f30e4e89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:55.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.273+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f30e011f990 con 0x7f30e009e970 2024-09-15T02:05:55.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f30ce7fc640 1 -- 172.21.15.17:0/2119950760 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f30d8018070 con 0x7f30e009e970 2024-09-15T02:05:55.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f30ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:05:55.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f30ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:55.276 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.276 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=1095133491,config=3140514009,health=514036659,kv=1529054990,logm=642141379}) 2024-09-15T02:05:55.277 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=4282608655,config=3140514009,health=514036659,kv=1529054990,logm=2264902140}) 2024-09-15T02:05:55.277 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.277 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.274+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=1095133491,config=3140514009,health=514036659,kv=1529054990,logm=642141379}) 2024-09-15T02:05:55.277 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.275+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=10,kv=7,logm=50} crc {auth=915623684,config=3140514009,health=54216544,kv=3442201590,logm=707365858}) 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 msgr2=0x7f30c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 0x7f30c003ac20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f30d0002750 tx=0x7f30d0009040 comp rx=0 tx=0).stop 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 10 monclient: shutdown 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 msgr2=0x7f30e011da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:55.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009df0 tx=0x7f30d800b040 comp rx=0 tx=0).stop 2024-09-15T02:05:55.278 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 shutdown_connections 2024-09-15T02:05:55.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f30c0038770 0x7f30c003ac20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.279 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460786322}) 2024-09-15T02:05:55.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f30e009c0d0 0x7f30e0133e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f30e009e970 0x7f30e011da00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 --2- 172.21.15.17:0/2119950760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30e006eb30 0x7f30e0141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:55.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 >> 172.21.15.17:0/2119950760 conn(0x7f30e00712f0 msgr2=0x7f30e0134e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:55.280 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271110492}) 2024-09-15T02:05:55.280 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.280 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460786322}) 2024-09-15T02:05:55.280 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1991552830}) 2024-09-15T02:05:55.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 shutdown_connections 2024-09-15T02:05:55.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:55.276+0000 7f30e4e89640 1 -- 172.21.15.17:0/2119950760 wait complete. 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857452245}) 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4022552723}) 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857452245}) 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.278+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177586213}) 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.283 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304819353}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2832471440}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304819353}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.279+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=41326427}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767888859}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505102632}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767888859}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.280+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046856813}) 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.284 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1949107011}) 2024-09-15T02:05:55.285 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058787503}) 2024-09-15T02:05:55.285 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.285 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1949107011}) 2024-09-15T02:05:55.285 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.281+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781324053}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249139809}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919408354}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249139809}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.282+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88098382}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367733396}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389424783}) 2024-09-15T02:05:55.287 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367733396}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.283+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513033276}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173265197}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3435403565}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173265197}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.284+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962032680}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361663175}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803036840}) 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.288 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361663175}) 2024-09-15T02:05:55.289 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.285+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215963931}) 2024-09-15T02:05:55.290 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.290 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041256137}) 2024-09-15T02:05:55.290 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551985041}) 2024-09-15T02:05:55.290 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.290 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041256137}) 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.286+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=742918661}) 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209039227}) 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680802558}) 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209039227}) 2024-09-15T02:05:55.291 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.287+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645106745}) 2024-09-15T02:05:55.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250623948}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1076728338}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250623948}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183402865}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501279351}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243270185}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.288+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501279351}) 2024-09-15T02:05:55.297 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241009258}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999710178}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507134063}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999710178}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.289+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748700777}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014075551}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1492126711}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014075551}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.290+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645767750}) 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.298 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577242955}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2577665204}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577242955}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.291+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684026491}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829096978}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349004606}) 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.299 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829096978}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264130462}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678004323}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885866878}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678004323}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.292+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237703310}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140634561}) 2024-09-15T02:05:55.301 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096511450}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140634561}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.293+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4198558176}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859108458}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670538030}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859108458}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.294+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988949270}) 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.302 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844476021}) 2024-09-15T02:05:55.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843166689}) 2024-09-15T02:05:55.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844476021}) 2024-09-15T02:05:55.303 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838431382}) 2024-09-15T02:05:55.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.312 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342344080}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.295+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438334406}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342344080}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4183524566}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227587599}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=846150330}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227587599}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.296+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110257228}) 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.313 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556807940}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3477333439}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556807940}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.297+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809150769}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912444945}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806190871}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912444945}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.298+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348893918}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987463810}) 2024-09-15T02:05:55.314 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858380514}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987463810}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.299+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444456652}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847933437}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837334628}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847933437}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470809673}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635972867}) 2024-09-15T02:05:55.315 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=547614527}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635972867}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.300+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406410492}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790627027}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916852443}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790627027}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.301+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858256536}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203901298}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413203950}) 2024-09-15T02:05:55.316 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203901298}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.302+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1891396519}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2009329951}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251621543}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2009329951}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595535793}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942586791}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1622046078}) 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942586791}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.303+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1989469224}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510773336}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2931152620}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510773336}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.304+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=741184259}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4205068292}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407700650}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4205068292}) 2024-09-15T02:05:55.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.305+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376995669}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297788586}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1024430736}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297788586}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822771754}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75994182}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224840907}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75994182}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.306+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130770650}) 2024-09-15T02:05:55.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052760222}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=152452945}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052760222}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.307+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231148369}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931211672}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690087062}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931211672}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.308+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534263365}) 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546427544}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768055252}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546427544}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.309+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905135133}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174760194}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242034006}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174760194}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537919980}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906787805}) 2024-09-15T02:05:55.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011029464}) 2024-09-15T02:05:55.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906787805}) 2024-09-15T02:05:55.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.310+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236451038}) 2024-09-15T02:05:55.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212855840}) 2024-09-15T02:05:55.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.311+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674118560}) 2024-09-15T02:05:55.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212855840}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.312+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482762677}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061690675}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869804740}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061690675}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684859973}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231274897}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202874312}) 2024-09-15T02:05:55.326 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231274897}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.313+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413895563}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051806479}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168310696}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051806479}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.314+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053660005}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800356581}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563103433}) 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800356581}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145859393}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.315+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493749545}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748490910}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493749545}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257364649}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287755207}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550815965}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287755207}) 2024-09-15T02:05:55.328 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:55.316+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817934362}) 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742545885}) 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929037111}) 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742545885}) 2024-09-15T02:05:56.130 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.128+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726304872}) 2024-09-15T02:05:56.131 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.131 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367571866}) 2024-09-15T02:05:56.131 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686462113}) 2024-09-15T02:05:56.131 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.132 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367571866}) 2024-09-15T02:05:56.132 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.129+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243996235}) 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358221666}) 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763742074}) 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358221666}) 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.130+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985280902}) 2024-09-15T02:05:56.133 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.134 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552040931}) 2024-09-15T02:05:56.134 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266749927}) 2024-09-15T02:05:56.134 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.134 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552040931}) 2024-09-15T02:05:56.134 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.131+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893044542}) 2024-09-15T02:05:56.135 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403132789}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1911685443}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403132789}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.132+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2585740555}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063739743}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563011989}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063739743}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.133+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4194366632}) 2024-09-15T02:05:56.136 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.137 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694576800}) 2024-09-15T02:05:56.137 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823111398}) 2024-09-15T02:05:56.137 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.137 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694576800}) 2024-09-15T02:05:56.137 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.134+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915027303}) 2024-09-15T02:05:56.138 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664087143}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494214881}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664087143}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.135+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052203345}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487555083}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=813697650}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487555083}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.136+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155843352}) 2024-09-15T02:05:56.139 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.140 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767377749}) 2024-09-15T02:05:56.140 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=948550032}) 2024-09-15T02:05:56.140 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.140 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767377749}) 2024-09-15T02:05:56.140 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.137+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3510470596}) 2024-09-15T02:05:56.141 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.141 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747156683}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3980286645}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747156683}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.138+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991816359}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185395725}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993888818}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185395725}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=755060988}) 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734257394}) 2024-09-15T02:05:56.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663198751}) 2024-09-15T02:05:56.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.139+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734257394}) 2024-09-15T02:05:56.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027071524}) 2024-09-15T02:05:56.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621793911}) 2024-09-15T02:05:56.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018348388}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621793911}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.140+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242407231}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.141+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.141+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748960925}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.141+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423518088}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.141+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748960925}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000058233}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115617464}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=116817883}) 2024-09-15T02:05:56.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115617464}) 2024-09-15T02:05:56.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.142+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3502629975}) 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221203728}) 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273472554}) 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221203728}) 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.143+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338515341}) 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.148 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460177344}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741408514}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460177344}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.144+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423957578}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129772622}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799100189}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129772622}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.145+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174606912}) 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.149 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554405675}) 2024-09-15T02:05:56.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296866976}) 2024-09-15T02:05:56.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554405675}) 2024-09-15T02:05:56.150 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.146+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183616537}) 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082698027}) 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410195929}) 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082698027}) 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.147+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017809553}) 2024-09-15T02:05:56.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896533146}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1137752351}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896533146}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.148+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744471426}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158212885}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350628607}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158212885}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.149+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351891585}) 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2516059860,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3874443712}) 2024-09-15T02:05:56.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=1} crc {logm=3818184850,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3392345677,mon_config_key=666446000,monmap=1228700967}) 2024-09-15T02:05:56.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2516059860,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3874443712}) 2024-09-15T02:05:56.154 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.150+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=79,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=2} crc {logm=2488180633,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=4030306869,mon_config_key=1386283410}) 2024-09-15T02:05:56.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.157+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.159 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.158+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgrstat=451119839,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=136926027,osdmap=92940000}) 2024-09-15T02:05:56.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.158+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mgrstat=2488649329,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=781602529}) 2024-09-15T02:05:56.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.158+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:05:56.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.158+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgrstat=451119839,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=136926027,osdmap=92940000}) 2024-09-15T02:05:56.160 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:05:56.158+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=2,osd_metadata=5,osd_snap=2,osdmap=56} crc {mgrstat=2829981472,mon_config_key=1583035879,monmap=2689118867,osd_metadata=1418025409,osd_snap=1063953654,osdmap=3082530791}) 2024-09-15T02:05:56.292 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:05:56.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.397+0000 7f5fb4e57640 10 monclient: get_monmap_and_config 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.397+0000 7f5fb4e57640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.397+0000 7f5fb4e57640 10 monclient: monmap: 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:56.398174+0000 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:56.398174+0000 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.399 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:56.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.397+0000 7f5fb4e57640 1 Processor -- start 2024-09-15T02:05:56.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 -- start start 2024-09-15T02:05:56.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient: init 2024-09-15T02:05:56.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fb009c0d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5fb009e970 0x7f5fb009c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb009e970 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fb006eb30 0x7f5fb006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb006eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00699c0 con 0x7f5fb009c0d0 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb006a5e0 con 0x7f5fb009e970 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb006a290 con 0x7f5fb006eb30 2024-09-15T02:05:56.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fb4e57640 10 monclient(hunting): authenticate will time out at 1616.394731s 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5fb009e970 0x7f5fb009c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f5fb009e970 0x7f5fb009c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55766/0 (socket says 172.21.15.17:55766) 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 1 -- 172.21.15.17:0/406495697 learned_addr learned my addr 172.21.15.17:0/406495697 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 10 monclient(hunting): get_auth_request con 0x7f5fb009e970 auth_method 0 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.398+0000 7f5faed9d640 1 --2- 172.21.15.17:0/406495697 >> v2:172.21.15.67:3300/0 conn(0x7f5fb006eb30 0x7f5fb006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): get_auth_request con 0x7f5fb006eb30 auth_method 0 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_done global_id 105345 payload 274 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 1 -- 172.21.15.17:0/406495697 >> v2:172.21.15.67:3300/0 conn(0x7f5fb006eb30 msgr2=0x7f5fb006ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 1 --2- 172.21.15.17:0/406495697 >> v2:172.21.15.67:3300/0 conn(0x7f5fb006eb30 0x7f5fb006ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 1 -- 172.21.15.17:0/406495697 >> v1:172.21.15.67:6789/0 conn(0x7f5fb009c0d0 legacy=0x7f5fb009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 1 monclient: found mon.noname-a 2024-09-15T02:05:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.399+0000 7f5fadd9b640 1 -- 172.21.15.17:0/406495697 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb009ed70 con 0x7f5fb009e970 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fadd9b640 1 -- 172.21.15.17:0/406495697 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009e970 msgr2=0x7f5fb009c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fadd9b640 1 --2- 172.21.15.17:0/406495697 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009e970 0x7f5fb009c4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00a0b00 tx=0x7f5f980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b099269b6eb244 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 1 -- 172.21.15.17:0/406495697 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98032440 con 0x7f5fb009e970 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: dump: 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.411 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 1 -- 172.21.15.17:0/406495697 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f980325a0 con 0x7f5fb009e970 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 1 -- 172.21.15.17:0/406495697 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98032940 con 0x7f5fb009e970 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 5 monclient: authenticate success, global_id 105345 2024-09-15T02:05:56.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 4 set_mon_vals no callback set 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fad59a640 10 monclient: dump: 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 10 monclient: shutdown 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 1 -- 172.21.15.17:0/406495697 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009e970 msgr2=0x7f5fb009c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.402+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/406495697 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009e970 0x7f5fb009c4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00a0b00 tx=0x7f5f980091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 -- 172.21.15.17:0/406495697 shutdown_connections 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/406495697 >> v2:172.21.15.67:3300/0 conn(0x7f5fb006eb30 0x7f5fb006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/406495697 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009e970 0x7f5fb009c4a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 -- 172.21.15.17:0/406495697 >> 172.21.15.17:0/406495697 conn(0x7f5fb00712f0 msgr2=0x7f5fb00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 -- 172.21.15.17:0/406495697 shutdown_connections 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 1 -- 172.21.15.17:0/406495697 wait complete. 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.403+0000 7f5fb4e57640 10 monclient: monmap: 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 Processor -- start 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 -- start start 2024-09-15T02:05:56.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient: init 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 0x7f5fb0133f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb006eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb009c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5fb009e970 0x7f5fb0141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb009cd80 con 0x7f5fb009c0d0 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01211a0 con 0x7f5fb006eb30 2024-09-15T02:05:56.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0122090 con 0x7f5fb009e970 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fb4e57640 10 monclient(hunting): authenticate will time out at 1616.400655s 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55782/0 (socket says 172.21.15.17:55782) 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 1 -- 172.21.15.17:0/745121596 learned_addr learned my addr 172.21.15.17:0/745121596 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): get_auth_request con 0x7f5fb009c0d0 auth_method 0 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5fb009e970 0x7f5fb0141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): get_auth_request con 0x7f5fb009e970 auth_method 0 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fae59c640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 0x7f5fb0133f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fae59c640 10 monclient(hunting): get_auth_request con 0x7f5fb006eb30 auth_method 0 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5faed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.404+0000 7f5fadd9b640 10 monclient(hunting): handle_auth_done global_id 105348 payload 824 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 -- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5fb009e970 msgr2=0x7f5fb0141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5fb009e970 0x7f5fb0141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 -- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 msgr2=0x7f5fb0133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 0x7f5fb0133f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 monclient: found mon.noname-a 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fadd9b640 1 -- 172.21.15.17:0/745121596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009450 con 0x7f5fb009c0d0 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.405+0000 7f5fae59c640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 0x7f5fb0133f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fadd9b640 1 -- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 msgr2=0x7f5fb0141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fadd9b640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009210 tx=0x7f5f98032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980319c0 con 0x7f5fb009c0d0 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: dump: 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.420 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 5 monclient: authenticate success, global_id 105348 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98031b40 con 0x7f5fb009c0d0 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 10 monclient: _renew_subs 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9803d440 con 0x7f5fb009c0d0 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb0122370 con 0x7f5fb009c0d0 2024-09-15T02:05:56.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fa67fc640 10 monclient: dump: 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5facd99640 4 set_mon_vals no callback set 2024-09-15T02:05:56.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 10 monclient: _renew_subs 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.407+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb01344a0 con 0x7f5fb009c0d0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fb4e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5f98039040 con 0x7f5fb009c0d0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fb4e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fb4e57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb00724a0 con 0x7f5fb009c0d0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fa67fc640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 0x7f5f9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.408+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f5f9806c090 con 0x7f5fb009c0d0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.409+0000 7f5fae59c640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 0x7f5f9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.409+0000 7f5fae59c640 10 monclient: get_auth_request con 0x7f5f90038740 auth_method 0 2024-09-15T02:05:56.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.411+0000 7f5fae59c640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 0x7f5f9003abf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0002860 tx=0x7f5fa0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.413+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5f9802f2a0 con 0x7f5fb009c0d0 2024-09-15T02:05:56.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.413+0000 7f5fa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.413+0000 7f5fa67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fb4e57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fb4e57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fb4e57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5fb0067220 con 0x7f5fb009c0d0 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fa67fc640 1 -- 172.21.15.17:0/745121596 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f5f98042610 con 0x7f5fb009c0d0 2024-09-15T02:05:56.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:56.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.641+0000 7f5fa67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:56.644 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 msgr2=0x7f5f9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 0x7f5f9003abf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0002860 tx=0x7f5fa0009040 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 10 monclient: shutdown 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 msgr2=0x7f5fb0141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.643+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009210 tx=0x7f5f98032000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 shutdown_connections 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5f90038740 0x7f5f9003abf0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fb009c0d0 0x7f5fb0141220 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5fb009e970 0x7f5fb0141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 --2- 172.21.15.17:0/745121596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb006eb30 0x7f5fb0133f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 >> 172.21.15.17:0/745121596 conn(0x7f5fb00712f0 msgr2=0x7f5fb006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 shutdown_connections 2024-09-15T02:05:56.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.644+0000 7f5fb4e57640 1 -- 172.21.15.17:0/745121596 wait complete. 2024-09-15T02:05:56.655 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:05:56.655 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:05:56.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:05:56.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:05:56.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 10 monclient: get_monmap_and_config 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 10 monclient: monmap: 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:56.751947+0000 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:56.751947+0000 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:56.753 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:56.754 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 1 Processor -- start 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 1 -- start start 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.751+0000 7f269ea94640 10 monclient: init 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2698131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f26980a0280 0x7f2698131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-a con 0x7f26980a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2698132430 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26980a0040 con 0x7f2698131b20 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269809bff0 con 0x7f2698132430 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26980a14a0 con 0x7f26980a0280 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269ea94640 10 monclient(hunting): authenticate will time out at 1616.748418s 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269c830640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2698131b20 0x7f269809eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55818/0 (socket says 172.21.15.17:55818) 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f269c830640 1 -- 172.21.15.17:0/2528224369 learned_addr learned my addr 172.21.15.17:0/2528224369 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f2697fff640 1 --2- 172.21.15.17:0/2528224369 >> v2:172.21.15.17:3300/0 conn(0x7f26980a0280 0x7f2698131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f2697fff640 10 monclient(hunting): get_auth_request con 0x7f26980a0280 auth_method 0 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f2697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f2697fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.752+0000 7f2697fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f269c830640 1 -- 172.21.15.17:0/2528224369 >> v1:172.21.15.17:6789/0 conn(0x7f2698131b20 legacy=0x7f269809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient(hunting): handle_auth_done global_id 105351 payload 274 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 1 -- 172.21.15.17:0/2528224369 >> v1:172.21.15.67:6789/0 conn(0x7f2698132430 legacy=0x7f2698138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 1 -- 172.21.15.17:0/2528224369 >> v1:172.21.15.17:6789/0 conn(0x7f2698131b20 legacy=0x7f269809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 1 monclient: found mon.noname-a 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.754+0000 7f2697fff640 1 -- 172.21.15.17:0/2528224369 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269809f3b0 con 0x7f26980a0280 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.757+0000 7f2697fff640 1 -- 172.21.15.17:0/2528224369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26980a0280 msgr2=0x7f2698131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.757+0000 7f2697fff640 1 --2- 172.21.15.17:0/2528224369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26980a0280 0x7f2698131ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f26800098d0 tx=0x7f26800090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926aea7908e613b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 1 -- 172.21.15.17:0/2528224369 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680032440 con 0x7f26980a0280 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: dump: 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 1 -- 172.21.15.17:0/2528224369 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26800325a0 con 0x7f26980a0280 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 5 monclient: authenticate success, global_id 105351 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 4 set_mon_vals no callback set 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 1 -- 172.21.15.17:0/2528224369 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680032940 con 0x7f26980a0280 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f26977fe640 10 monclient: dump: 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 10 monclient: shutdown 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 -- 172.21.15.17:0/2528224369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26980a0280 msgr2=0x7f2698131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 --2- 172.21.15.17:0/2528224369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26980a0280 0x7f2698131ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f26800098d0 tx=0x7f26800090f0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 -- 172.21.15.17:0/2528224369 shutdown_connections 2024-09-15T02:05:56.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 --2- 172.21.15.17:0/2528224369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f26980a0280 0x7f2698131ef0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 -- 172.21.15.17:0/2528224369 >> 172.21.15.17:0/2528224369 conn(0x7f2698071270 msgr2=0x7f26980a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 -- 172.21.15.17:0/2528224369 shutdown_connections 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.758+0000 7f269ea94640 1 -- 172.21.15.17:0/2528224369 wait complete. 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.759+0000 7f269ea94640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.759+0000 7f269ea94640 10 monclient: monmap: 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:56.764 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:56.765 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:56.765 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.759+0000 7f269ea94640 1 Processor -- start 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.759+0000 7f269ea94640 1 -- start start 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.759+0000 7f269ea94640 10 monclient: init 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980a0280 0x7f269809b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-b con 0x7f26980a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 0x7f2698132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-c con 0x7f2698131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 0x7f2698132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): picked mon.noname-a con 0x7f2698132430 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698135f20 con 0x7f2698132430 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698135bb0 con 0x7f26980a0280 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2698133da0 con 0x7f2698131b20 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269ea94640 10 monclient(hunting): authenticate will time out at 1616.756259s 2024-09-15T02:05:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 0x7f2698132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 0x7f2698132970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33818/0 (socket says 172.21.15.17:33818) 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 0x7f2698132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 1 -- 172.21.15.17:0/1257351130 learned_addr learned my addr 172.21.15.17:0/1257351130 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): get_auth_request con 0x7f2698131b20 auth_method 0 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): get_auth_request con 0x7f2698132430 auth_method 0 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980a0280 0x7f269809b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): get_auth_request con 0x7f26980a0280 auth_method 0 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient(hunting): handle_auth_done global_id 105357 payload 824 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 msgr2=0x7f2698132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 0x7f2698132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980a0280 msgr2=0x7f269809b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980a0280 0x7f269809b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 monclient: found mon.noname-a 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009390 con 0x7f2698132430 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.760+0000 7f2697fff640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2698131b20 0x7f2698132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.762+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 msgr2=0x7f2698132eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.762+0000 7f269d031640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 0x7f2698132eb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f269000c7e0 tx=0x7f26900025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269000a850 con 0x7f2698132430 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: dump: 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269000a9d0 con 0x7f2698132430 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 5 monclient: authenticate success, global_id 105357 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 10 monclient: _renew_subs 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26981358d0 con 0x7f2698132430 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269000ad90 con 0x7f2698132430 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2694ff9640 10 monclient: dump: 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f2696ffd640 4 set_mon_vals no callback set 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 10 monclient: _renew_subs 2024-09-15T02:05:56.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.763+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2698136590 con 0x7f2698132430 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269ea94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269ea94640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 0x7f26981333f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698063a40 con 0x7f269802aa40 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269d031640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 0x7f2698132eb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f269000c7e0 tx=0x7f26900025b0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f269004c880 osd_map(257..257 src has 229..257) v4 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2690012630 con 0x7f2698132430 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 0x7f26981333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient: get_auth_request con 0x7f269802aa40 auth_method 0 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f2694ff9640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2678038790 0x7f267803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 10 monclient(hunting): handle_auth_done global_id 105360 payload 824 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f269c830640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f269802ae10 con 0x7f269802aa40 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f2697fff640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2678038790 0x7f267803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.764+0000 7f2697fff640 10 monclient: get_auth_request con 0x7f2678038790 auth_method 0 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f269c830640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 msgr2=0x7f26981333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f269c830640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 0x7f26981333f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688005c90 con 0x7f269802aa40 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f2694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f2694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.766+0000 7f2694ff9640 10 monclient: dump: 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2688005df0 con 0x7f269802aa40 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:56.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 msgr2=0x7f26981333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 0x7f26981333f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-09-15T02:05:56.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 msgr2=0x7f26981333f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:56.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2694ff9640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269802aa40 0x7f26981333f0 secure :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-09-15T02:05:56.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.767+0000 7f2697fff640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2678038790 0x7f267803ac40 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2698134bf0 tx=0x7f2680032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.772+0000 7f269ea94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.772+0000 7f269ea94640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 0x7f269805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.772+0000 7f269ea94640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.772+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980a2010 con 0x7f269805e310 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 0x7f269805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient: get_auth_request con 0x7f269805e310 auth_method 0 2024-09-15T02:05:56.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 10 monclient(hunting): handle_auth_done global_id 105363 payload 824 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.773+0000 7f269c830640 1 -- 172.21.15.17:0/1257351130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2698135d70 con 0x7f269805e310 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.775+0000 7f269c830640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 msgr2=0x7f269805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.775+0000 7f269c830640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 0x7f269805e710 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.775+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688021620 con 0x7f269805e310 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:56.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 10 monclient: dump: 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f2688021780 con 0x7f269805e310 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:56.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 msgr2=0x7f269805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 0x7f269805e710 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688018000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 msgr2=0x7f269805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f269805e310 0x7f269805e710 secure :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2688005840 tx=0x7f2688018000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:56.787 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.788 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.789 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:56.790 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.791 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:56.792 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.794 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:56.795 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.798 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:05:56.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:56.800 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2678038790 msgr2=0x7f267803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2678038790 0x7f267803ac40 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2698134bf0 tx=0x7f2680032000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 10 monclient: shutdown 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 msgr2=0x7f2698132eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 --2- 172.21.15.17:0/1257351130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2698132430 0x7f2698132eb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f269000c7e0 tx=0x7f26900025b0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 reap_dead start 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269d031640 1 -- 172.21.15.17:0/1257351130 reap_dead start 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f2694ff9640 1 -- 172.21.15.17:0/1257351130 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f269004c880 con 0x7f2698132430 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 shutdown_connections 2024-09-15T02:05:56.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 >> 172.21.15.17:0/1257351130 conn(0x7f2698071270 msgr2=0x7f2698139bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 shutdown_connections 2024-09-15T02:05:56.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.776+0000 7f269ea94640 1 -- 172.21.15.17:0/1257351130 wait complete. 2024-09-15T02:05:56.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:56.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:05:56.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:05:56.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.898+0000 7fc97fe89640 10 monclient: get_monmap_and_config 2024-09-15T02:05:56.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.898+0000 7fc97fe89640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.899+0000 7fc97fe89640 10 monclient: monmap: 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:56.899809+0000 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:56.899809+0000 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:56.901 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:56.902 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:56.902 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.899+0000 7fc97fe89640 1 Processor -- start 2024-09-15T02:05:56.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.899+0000 7fc97fe89640 1 -- start start 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.899+0000 7fc97fe89640 10 monclient: init 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-b con 0x7fc978097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-a con 0x7fc97809a590 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc97809a960 0x7fc97812ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-c con 0x7fc97809a960 addr v2:172.21.15.67:3300/0 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9780985d0 con 0x7fc97809a590 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97809d200 con 0x7fc97809a960 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc978097c30 con 0x7fc978097320 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97fe89640 10 monclient(hunting): authenticate will time out at 1616.896281s 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33830/0 (socket says 172.21.15.17:33830) 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55826/0 (socket says 172.21.15.17:55826) 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 -- 172.21.15.17:0/3570233783 learned_addr learned my addr 172.21.15.17:0/3570233783 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): get_auth_request con 0x7fc978097320 auth_method 0 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): get_auth_request con 0x7fc97809a590 auth_method 0 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97e426640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.67:3300/0 conn(0x7fc97809a960 0x7fc97812ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97e426640 10 monclient(hunting): get_auth_request con 0x7fc97809a960 auth_method 0 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97e426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97e426640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97e426640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient(hunting): handle_auth_done global_id 184133 payload 274 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 -- 172.21.15.17:0/3570233783 >> v2:172.21.15.67:3300/0 conn(0x7fc97809a960 msgr2=0x7fc97812ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.67:3300/0 conn(0x7fc97809a960 0x7fc97812ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 -- 172.21.15.17:0/3570233783 >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 msgr2=0x7fc97812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:56.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 monclient: found mon.noname-b 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97dc25640 1 -- 172.21.15.17:0/3570233783 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780992f0 con 0x7fc978097320 2024-09-15T02:05:56.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.900+0000 7fc97d424640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:56.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.903+0000 7fc97dc25640 1 -- 172.21.15.17:0/3570233783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 msgr2=0x7fc9780976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.903+0000 7fc97dc25640 1 --2- 172.21.15.17:0/3570233783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc978099a60 tx=0x7fc9740091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d800b87bcea035e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 1 -- 172.21.15.17:0/3570233783 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97402aea0 con 0x7fc978097320 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: dump: 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.926 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 5 monclient: authenticate success, global_id 184133 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 1 -- 172.21.15.17:0/3570233783 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc974032440 con 0x7fc978097320 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 4 set_mon_vals no callback set 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 1 -- 172.21.15.17:0/3570233783 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9740327e0 con 0x7fc978097320 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97cc23640 10 monclient: dump: 2024-09-15T02:05:56.927 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 10 monclient: shutdown 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 -- 172.21.15.17:0/3570233783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 msgr2=0x7fc9780976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 --2- 172.21.15.17:0/3570233783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc978099a60 tx=0x7fc9740091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 -- 172.21.15.17:0/3570233783 shutdown_connections 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.67:3300/0 conn(0x7fc97809a960 0x7fc97812ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 --2- 172.21.15.17:0/3570233783 >> v2:172.21.15.17:3300/0 conn(0x7fc97809a590 0x7fc97812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 --2- 172.21.15.17:0/3570233783 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9780976f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 -- 172.21.15.17:0/3570233783 >> 172.21.15.17:0/3570233783 conn(0x7fc9780a1000 msgr2=0x7fc9780a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 -- 172.21.15.17:0/3570233783 shutdown_connections 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.904+0000 7fc97fe89640 1 -- 172.21.15.17:0/3570233783 wait complete. 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 10 monclient: build_initial_monmap 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 10 monclient: monmap: 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:56.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 1 Processor -- start 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 1 -- start start 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 10 monclient: init 2024-09-15T02:05:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.905+0000 7fc97fe89640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-c con 0x7fc978097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97809a590 0x7fc97806e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-b con 0x7fc97809a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc97809a960 0x7fc97806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): picked mon.noname-a con 0x7fc97809a960 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97809ad50 con 0x7fc97809a960 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97812a5f0 con 0x7fc97809a590 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc978135eb0 con 0x7fc978097320 2024-09-15T02:05:56.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97fe89640 10 monclient(hunting): authenticate will time out at 1616.902117s 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33842/0 (socket says 172.21.15.17:33842) 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 -- 172.21.15.17:0/1248176573 learned_addr learned my addr 172.21.15.17:0/1248176573 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): get_auth_request con 0x7fc978097320 auth_method 0 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc97809a960 0x7fc97806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): get_auth_request con 0x7fc97809a960 auth_method 0 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient(hunting): handle_auth_done global_id 184136 payload 824 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97809a590 0x7fc97806e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97809a590 msgr2=0x7fc97806e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97809a590 0x7fc97806e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc97809a960 msgr2=0x7fc97806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc97809a960 0x7fc97806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient: _finish_hunting 0 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 monclient: found mon.noname-c 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97dc25640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968009390 con 0x7fc978097320 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97d424640 10 monclient: get_auth_request con 0x7fc97809a590 auth_method 0 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97809a590 0x7fc97806e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.906+0000 7fc97e426640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc97809a960 0x7fc97806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97dc25640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 msgr2=0x7fc9781295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97dc25640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc974009970 tx=0x7fc974032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97402aea0 con 0x7fc978097320 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: dump: 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: _finish_auth 0 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:56.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc974000d10 con 0x7fc978097320 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97fe89640 5 monclient: authenticate success, global_id 184136 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97fe89640 10 monclient: _renew_subs 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97fe89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc974009450 con 0x7fc978097320 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97403d440 con 0x7fc978097320 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96dffb640 10 monclient: dump: 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.936 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 10 monclient: _renew_subs 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.908+0000 7fc96ffff640 4 set_mon_vals no callback set 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc978117710 con 0x7fc978097320 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 0x7fc978117930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc978063c20 con 0x7fc978129b20 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc97dc25640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc974009970 tx=0x7fc974032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fc97406c410 osd_map(257..257 src has 229..257) v4 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.909+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc974039040 con 0x7fc978097320 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 0x7fc978117930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient: get_auth_request con 0x7fc978129b20 auth_method 0 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc96dffb640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc95c038740 0x7fc95c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 10 monclient(hunting): handle_auth_done global_id 184139 payload 824 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97d424640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9780724a0 con 0x7fc978129b20 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97e426640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc95c038740 0x7fc95c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.910+0000 7fc97e426640 10 monclient: get_auth_request con 0x7fc95c038740 auth_method 0 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc97d424640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 msgr2=0x7fc978117930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 0x7fc978117930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc968010b20 tx=0x7fc968008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96800edd0 con 0x7fc978129b20 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.912+0000 7fc96dffb640 10 monclient: dump: 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.939 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc96801c400 con 0x7fc978129b20 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 msgr2=0x7fc978117930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc97e426640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc95c038740 0x7fc95c03abf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc9700079f0 tx=0x7fc970009850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 0x7fc978117930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc968010b20 tx=0x7fc968008040 comp rx=0 tx=0).stop 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 msgr2=0x7fc978117930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.913+0000 7fc96dffb640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978129b20 0x7fc978117930 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc968010b20 tx=0x7fc968008040 comp rx=0 tx=0).stop 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.918+0000 7fc97fe89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.918+0000 7fc97fe89640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 0x7fc97806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.918+0000 7fc97fe89640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.918+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97806f4f0 con 0x7fc97806ebb0 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 0x7fc97806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient: get_auth_request con 0x7fc97806ebb0 auth_method 0 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:56.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 10 monclient(hunting): handle_auth_done global_id 184142 payload 824 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.919+0000 7fc97d424640 1 -- 172.21.15.17:0/1248176573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc97809cfe0 con 0x7fc97806ebb0 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc97d424640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 msgr2=0x7fc97806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc97d424640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 0x7fc97806efb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc96801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968006040 con 0x7fc97806ebb0 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 10 monclient: dump: 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:56.942 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fc9680061a0 con 0x7fc97806ebb0 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 msgr2=0x7fc97806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 0x7fc97806efb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc96801c000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 msgr2=0x7fc97806efb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.921+0000 7fc96dffb640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc97806ebb0 0x7fc97806efb0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc96801c000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.944 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:05:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:56.960 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc95c038740 msgr2=0x7fc95c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc95c038740 0x7fc95c03abf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc9700079f0 tx=0x7fc970009850 comp rx=0 tx=0).stop 2024-09-15T02:05:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 10 monclient: shutdown 2024-09-15T02:05:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 msgr2=0x7fc9781295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 --2- 172.21.15.17:0/1248176573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc978097320 0x7fc9781295e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc974009970 tx=0x7fc974032000 comp rx=0 tx=0).stop 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97e426640 1 -- 172.21.15.17:0/1248176573 reap_dead start 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc96dffb640 1 -- 172.21.15.17:0/1248176573 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7fc97406c410 con 0x7fc978097320 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97e426640 1 -- 172.21.15.17:0/1248176573 reap_dead start 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 shutdown_connections 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 >> 172.21.15.17:0/1248176573 conn(0x7fc9780a1000 msgr2=0x7fc97809ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 shutdown_connections 2024-09-15T02:05:56.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:56.922+0000 7fc97fe89640 1 -- 172.21.15.17:0/1248176573 wait complete. 2024-09-15T02:05:56.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:56.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:05:56.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 10 monclient: get_monmap_and_config 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 10 monclient: monmap: 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:57.060030+0000 2024-09-15T02:05:57.061 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:57.060030+0000 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:57.062 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 1 Processor -- start 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 1 -- start start 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.059+0000 7f8fa2cf8640 10 monclient: init 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c0980a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f9c0989b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0a01e0 con 0x7f8f9c0989b0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9c0a1580 con 0x7f8f9c0980a0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c0a5c80 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa2cf8640 10 monclient(hunting): authenticate will time out at 1617.056534s 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33872/0 (socket says 172.21.15.17:33872) 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55836/0 (socket says 172.21.15.17:55836) 2024-09-15T02:05:57.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 -- 172.21.15.17:0/31469920 learned_addr learned my addr 172.21.15.17:0/31469920 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): get_auth_request con 0x7f8f9c0a58b0 auth_method 0 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): get_auth_request con 0x7f8f9c0980a0 auth_method 0 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_done global_id 184145 payload 274 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 -- 172.21.15.17:0/31469920 >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 msgr2=0x7f8f9c098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 --2- 172.21.15.17:0/31469920 >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 -- 172.21.15.17:0/31469920 >> v1:172.21.15.67:6789/0 conn(0x7f8f9c0989b0 legacy=0x7f8f9c068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 monclient: found mon.noname-b 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8fa0a94640 1 -- 172.21.15.17:0/31469920 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c1326c0 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.060+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/31469920 >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.063+0000 7f8fa0a94640 1 -- 172.21.15.17:0/31469920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 msgr2=0x7f8f9c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.063+0000 7f8fa0a94640 1 --2- 172.21.15.17:0/31469920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009f20 tx=0x7f8f8c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d978cb9fad22b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 1 -- 172.21.15.17:0/31469920 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c007610 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: dump: 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.078 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 1 -- 172.21.15.17:0/31469920 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c02d050 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 5 monclient: authenticate success, global_id 184145 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 1 -- 172.21.15.17:0/31469920 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c0079a0 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 4 set_mon_vals no callback set 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8f9b7fe640 10 monclient: dump: 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.080 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 10 monclient: shutdown 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/31469920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 msgr2=0x7f8f9c132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/31469920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009f20 tx=0x7f8f8c009840 comp rx=0 tx=0).stop 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/31469920 shutdown_connections 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/31469920 >> v2:172.21.15.17:3300/0 conn(0x7f8f9c0980a0 0x7f8f9c098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/31469920 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0a58b0 0x7f8f9c132040 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/31469920 >> 172.21.15.17:0/31469920 conn(0x7f8f9c070ed0 msgr2=0x7f8f9c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/31469920 shutdown_connections 2024-09-15T02:05:57.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.064+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/31469920 wait complete. 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.065+0000 7f8fa2cf8640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.065+0000 7f8fa2cf8640 10 monclient: monmap: 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.065+0000 7f8fa2cf8640 1 Processor -- start 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.065+0000 7f8fa2cf8640 1 -- start start 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.065+0000 7f8fa2cf8640 10 monclient: init 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 0x7f8f9c09b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9c0980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c0989b0 0x7f8f9c11bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c0989b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f9c0a58b0 0x7f8f9c130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c0a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c09eb00 con 0x7f8f9c0a58b0 2024-09-15T02:05:57.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c11e450 con 0x7f8f9c0989b0 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c11cea0 con 0x7f8f9c0980a0 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa2cf8640 10 monclient(hunting): authenticate will time out at 1617.062316s 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 0x7f8f9c09b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 0x7f8f9c09b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:33878/0 (socket says 172.21.15.17:33878) 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 -- 172.21.15.17:0/1089676135 learned_addr learned my addr 172.21.15.17:0/1089676135 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): get_auth_request con 0x7f8f9c0980a0 auth_method 0 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f9c0a58b0 0x7f8f9c130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): get_auth_request con 0x7f8f9c0a58b0 auth_method 0 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c0989b0 0x7f8f9c11bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c0989b0 msgr2=0x7f8f9c11bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c0989b0 0x7f8f9c11bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f9c0a58b0 msgr2=0x7f8f9c130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f9c0a58b0 0x7f8f9c130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 monclient: found mon.noname-c 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa0a94640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f90009390 con 0x7f8f9c0980a0 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8f9bfff640 10 monclient: get_auth_request con 0x7f8f9c0989b0 auth_method 0 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8fa1295640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f9c0a58b0 0x7f8f9c130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.066+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c0989b0 0x7f8f9c11bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.068+0000 7f8fa0a94640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 msgr2=0x7f8f9c09b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.068+0000 7f8fa0a94640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 0x7f8f9c09b100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009f20 tx=0x7f8f8c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c007e90 con 0x7f8f9c0980a0 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: dump: 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.087 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c03c840 con 0x7f8f9c0980a0 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 5 monclient: authenticate success, global_id 184148 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 10 monclient: _renew_subs 2024-09-15T02:05:57.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c009ae0 con 0x7f8f9c0980a0 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c03cc00 con 0x7f8f9c0980a0 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f98ff9640 10 monclient: dump: 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.089 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 10 monclient: _renew_subs 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c11e0e0 con 0x7f8f9c0980a0 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.069+0000 7f8f9affd640 4 set_mon_vals no callback set 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa2cf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 0x7f8f9c141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c063b20 con 0x7f8f9c131220 2024-09-15T02:05:57.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8f8c031400 con 0x7f8f9c0980a0 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 0x7f8f9c141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient: get_auth_request con 0x7f8f9c131220 auth_method 0 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f98ff9640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f88038790 0x7f8f8803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8f8c06d350 con 0x7f8f9c0980a0 2024-09-15T02:05:57.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa1295640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f88038790 0x7f8f8803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8fa1295640 10 monclient: get_auth_request con 0x7f8f88038790 auth_method 0 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_done global_id 174184 payload 824 2024-09-15T02:05:57.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.070+0000 7f8f9bfff640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f9c0724a0 con 0x7f8f9c131220 2024-09-15T02:05:57.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f9bfff640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 msgr2=0x7f8f9c141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 0x7f8f9c141360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c097f40 tx=0x7f8f90008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8fa1295640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f88038790 0x7f8f8803ac40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c11e400 tx=0x7f8f94007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9000ecf0 con 0x7f8f9c131220 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 10 monclient: dump: 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8f9000ee50 con 0x7f8f9c131220 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 msgr2=0x7f8f9c141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 0x7f8f9c141360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c097f40 tx=0x7f8f90008040 comp rx=0 tx=0).stop 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 msgr2=0x7f8f9c141360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.073+0000 7f8f98ff9640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c131220 0x7f8f9c141360 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c097f40 tx=0x7f8f90008040 comp rx=0 tx=0).stop 2024-09-15T02:05:57.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8fa2cf8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 0x7f8f9c141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8fa2cf8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c1315f0 con 0x7f8f9c1418a0 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 0x7f8f9c141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient: get_auth_request con 0x7f8f9c1418a0 auth_method 0 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 10 monclient(hunting): handle_auth_done global_id 174187 payload 824 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.079+0000 7f8f9bfff640 1 -- 172.21.15.17:0/1089676135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f9c11e590 con 0x7f8f9c1418a0 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f9bfff640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 msgr2=0x7f8f9c141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f9bfff640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 0x7f8f9c141ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009870 tx=0x7f8f9001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9001c610 con 0x7f8f9c1418a0 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 10 monclient: dump: 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.098 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f8f900273f0 con 0x7f8f9c1418a0 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 msgr2=0x7f8f9c141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 0x7f8f9c141ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009870 tx=0x7f8f9001c000 comp rx=0 tx=0).stop 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 msgr2=0x7f8f9c141ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:05:57.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8f98ff9640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f9c1418a0 0x7f8f9c141ca0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009870 tx=0x7f8f9001c000 comp rx=0 tx=0).stop 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 150, 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:05:57.100 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:57.101 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:57.102 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:57.103 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:05:57.104 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:05:57.108 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:05:57.108 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:57.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:57.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:05:57.110 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:57.111 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:05:57.113 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:05:57.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.115 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:05:57.116 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:05:57.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:05:57.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:05:57.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:05:57.117 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:05:57.117 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:05:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f88038790 msgr2=0x7f8f8803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f88038790 0x7f8f8803ac40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c11e400 tx=0x7f8f94007e10 comp rx=0 tx=0).stop 2024-09-15T02:05:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 10 monclient: shutdown 2024-09-15T02:05:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 msgr2=0x7f8f9c09b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 --2- 172.21.15.17:0/1089676135 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f9c0980a0 0x7f8f9c09b100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009f20 tx=0x7f8f8c036000 comp rx=0 tx=0).stop 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa1295640 1 -- 172.21.15.17:0/1089676135 reap_dead start 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa1295640 1 -- 172.21.15.17:0/1089676135 reap_dead start 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 shutdown_connections 2024-09-15T02:05:57.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.082+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 >> 172.21.15.17:0/1089676135 conn(0x7f8f9c070ed0 msgr2=0x7f8f9c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.083+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 shutdown_connections 2024-09-15T02:05:57.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.083+0000 7f8fa2cf8640 1 -- 172.21.15.17:0/1089676135 wait complete. 2024-09-15T02:05:57.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:05:57.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T02:05:57.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:05:57.122 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T02:05:57.122 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:05:57.122 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:05:57.123 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:05:57.137 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:05:57.137 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:05:57.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:05:57.174 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=110 2024-09-15T02:05:57.174 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 110 -ne 110 ]] 2024-09-15T02:05:57.174 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:05:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:+ export CEPH_ARGS= 2024-09-15T02:05:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:+ CEPH_ARGS= 2024-09-15T02:05:57.175 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph auth del client.foo 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.231+0000 7fe070d1d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.231+0000 7fe070d1d640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.231+0000 7fe070d1d640 10 monclient: monmap: 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:05:57.232259+0000 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:05:57.232259+0000 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.233 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:57.234 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 Processor -- start 2024-09-15T02:05:57.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 -- start start 2024-09-15T02:05:57.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient: init 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe06c131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe06c0a0280 0x7fe06c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fe06c0a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe06c132430 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0a0040 con 0x7fe06c132430 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c09bff0 con 0x7fe06c131b20 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c0a14a0 con 0x7fe06c0a0280 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe070d1d640 10 monclient(hunting): authenticate will time out at 1617.228752s 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe06a59c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fe06c131b20 legacy=0x7fe06c09eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe06c0a0280 0x7fe06c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe06c0a0280 0x7fe06c131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55858/0 (socket says 172.21.15.17:55858) 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 1 -- 172.21.15.17:0/3866956097 learned_addr learned my addr 172.21.15.17:0/3866956097 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): get_auth_request con 0x7fe06c0a0280 auth_method 0 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.232+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe06ad9d640 1 -- 172.21.15.17:0/3866956097 >> v1:172.21.15.17:6789/0 conn(0x7fe06c132430 legacy=0x7fe06c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 10 monclient(hunting): handle_auth_done global_id 105378 payload 274 2024-09-15T02:05:57.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 1 -- 172.21.15.17:0/3866956097 >> v1:172.21.15.67:6789/0 conn(0x7fe06c131b20 legacy=0x7fe06c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:05:57.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 1 -- 172.21.15.17:0/3866956097 >> v1:172.21.15.17:6789/0 conn(0x7fe06c132430 legacy=0x7fe06c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 1 monclient: found mon.noname-a 2024-09-15T02:05:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.235+0000 7fe069d9b640 1 -- 172.21.15.17:0/3866956097 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c09f3b0 con 0x7fe06c0a0280 2024-09-15T02:05:57.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe069d9b640 1 -- 172.21.15.17:0/3866956097 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c0a0280 msgr2=0x7fe06c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe069d9b640 1 --2- 172.21.15.17:0/3866956097 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c0a0280 0x7fe06c131ef0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe054009990 tx=0x7fe0540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ebde696b02124f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 1 -- 172.21.15.17:0/3866956097 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05402ae90 con 0x7fe06c0a0280 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: dump: 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.241 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 1 -- 172.21.15.17:0/3866956097 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05402f4b0 con 0x7fe06c0a0280 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 5 monclient: authenticate success, global_id 105378 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 1 -- 172.21.15.17:0/3866956097 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05402f850 con 0x7fe06c0a0280 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 4 set_mon_vals no callback set 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe06959a640 10 monclient: dump: 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 10 monclient: shutdown 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:57.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 -- 172.21.15.17:0/3866956097 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c0a0280 msgr2=0x7fe06c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3866956097 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c0a0280 0x7fe06c131ef0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe054009990 tx=0x7fe0540091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 -- 172.21.15.17:0/3866956097 shutdown_connections 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3866956097 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c0a0280 0x7fe06c131ef0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 -- 172.21.15.17:0/3866956097 >> 172.21.15.17:0/3866956097 conn(0x7fe06c071270 msgr2=0x7fe06c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 -- 172.21.15.17:0/3866956097 shutdown_connections 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.238+0000 7fe070d1d640 1 -- 172.21.15.17:0/3866956097 wait complete. 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.239+0000 7fe070d1d640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.239+0000 7fe070d1d640 10 monclient: monmap: 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:05:57.244 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.239+0000 7fe070d1d640 1 Processor -- start 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.239+0000 7fe070d1d640 1 -- start start 2024-09-15T02:05:57.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.239+0000 7fe070d1d640 10 monclient: init 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 0x7fe06c139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fe06c0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fe06c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe06c132430 0x7fe06c132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fe06c132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0a09b0 con 0x7fe06c131b20 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c138c70 con 0x7fe06c0a0280 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c131550 con 0x7fe06c132430 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe070d1d640 10 monclient(hunting): authenticate will time out at 1617.236143s 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06a59c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 msgr2=0x7fe06c139540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 0x7fe06c139540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55860/0 (socket says 172.21.15.17:55860) 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe06c132430 0x7fe06c132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 -- 172.21.15.17:0/3265182956 learned_addr learned my addr 172.21.15.17:0/3265182956 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): get_auth_request con 0x7fe06c131b20 auth_method 0 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): get_auth_request con 0x7fe06c132430 auth_method 0 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe06ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient(hunting): handle_auth_done global_id 105381 payload 824 2024-09-15T02:05:57.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 -- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe06c132430 msgr2=0x7fe06c132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe06c132430 0x7fe06c132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 -- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 msgr2=0x7fe06c139540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 0x7fe06c139540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 monclient: found mon.noname-a 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.240+0000 7fe069d9b640 1 -- 172.21.15.17:0/3265182956 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054009450 con 0x7fe06c131b20 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe069d9b640 1 -- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 msgr2=0x7fe06c132970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe069d9b640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe054035040 tx=0x7fe054032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0540329b0 con 0x7fe06c131b20 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe0627fc640 10 monclient: dump: 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.253 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.242+0000 7fe0627fc640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05402ae90 con 0x7fe06c131b20 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 5 monclient: authenticate success, global_id 105381 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 10 monclient: _renew_subs 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c135780 con 0x7fe06c131b20 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054032d80 con 0x7fe06c131b20 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:05:57.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 10 monclient: dump: 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe068d99640 4 set_mon_vals no callback set 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 10 monclient: _renew_subs 2024-09-15T02:05:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c136440 con 0x7fe06c131b20 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.243+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe054039040 con 0x7fe06c131b20 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe0627fc640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 0x7fe04c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe05406c9f0 con 0x7fe06c131b20 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe070d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe070d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe070d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c136630 con 0x7fe06c131b20 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe06a59c640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 0x7fe04c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.244+0000 7fe06a59c640 10 monclient: get_auth_request con 0x7fe04c0386f0 auth_method 0 2024-09-15T02:05:57.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.246+0000 7fe06a59c640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 0x7fe04c03aba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe064006fd0 tx=0x7fe064006d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.248+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe05403f070 con 0x7fe06c131b20 2024-09-15T02:05:57.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.248+0000 7fe0627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.248+0000 7fe0627fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.250+0000 7fb95b71d640 10 monclient: get_monmap_and_config 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.250+0000 7fb95b71d640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.250+0000 7fb95b71d640 10 monclient: monmap: 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:05:57.250942+0000 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:05:57.250942+0000 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.251+0000 7fb95b71d640 1 Processor -- start 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.251+0000 7fb95b71d640 1 -- start start 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.251+0000 7fb95b71d640 10 monclient: init 2024-09-15T02:05:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9540a0d50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb954069880 addr v1:172.21.15.17:6789/0 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb95406a360 0x7fb954139150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-a con 0x7fb95406a360 addr v2:172.21.15.17:3300/0 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9540a02e0 con 0x7fb954069880 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95406b170 con 0x7fb95406a360 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540a19e0 con 0x7fb9540a0d50 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb95b71d640 10 monclient(hunting): authenticate will time out at 1589.720183s 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb95406a360 0x7fb954139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:50710/0 (socket says 172.21.15.67:50710) 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb958cb8640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb954069880 0x7fb954069c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:56528/0 (socket says 172.21.15.67:56528) 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 -- 172.21.15.67:0/3870007759 learned_addr learned my addr 172.21.15.67:0/3870007759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): get_auth_request con 0x7fb9540a0d50 auth_method 0 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): get_auth_request con 0x7fb95406a360 auth_method 0 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient(hunting): handle_auth_done global_id 184154 payload 274 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 -- 172.21.15.67:0/3870007759 >> v2:172.21.15.17:3300/0 conn(0x7fb95406a360 msgr2=0x7fb954139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 --2- 172.21.15.67:0/3870007759 >> v2:172.21.15.17:3300/0 conn(0x7fb95406a360 0x7fb954139150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 -- 172.21.15.67:0/3870007759 >> v1:172.21.15.17:6789/0 conn(0x7fb954069880 legacy=0x7fb954069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 monclient: found mon.noname-b 2024-09-15T02:05:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.252+0000 7fb9594b9640 1 -- 172.21.15.67:0/3870007759 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540a0520 con 0x7fb9540a0d50 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb9594b9640 1 -- 172.21.15.67:0/3870007759 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 msgr2=0x7fb954096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb9594b9640 1 --2- 172.21.15.67:0/3870007759 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb950009910 tx=0x7fb9500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0134974a4527e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 1 -- 172.21.15.67:0/3870007759 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95002aea0 con 0x7fb9540a0d50 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: dump: 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 1 -- 172.21.15.67:0/3870007759 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb950032440 con 0x7fb9540a0d50 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 1 -- 172.21.15.67:0/3870007759 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9500327e0 con 0x7fb9540a0d50 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb94bfff640 10 monclient: dump: 2024-09-15T02:05:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb95b71d640 5 monclient: authenticate success, global_id 184154 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb95b71d640 10 monclient: get_monmap_and_config success 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.255+0000 7fb95b71d640 4 set_mon_vals no callback set 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 10 monclient: shutdown 2024-09-15T02:05:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 -- 172.21.15.67:0/3870007759 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 msgr2=0x7fb954096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 --2- 172.21.15.67:0/3870007759 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb950009910 tx=0x7fb9500091d0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 -- 172.21.15.67:0/3870007759 shutdown_connections 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 --2- 172.21.15.67:0/3870007759 >> v2:172.21.15.17:3300/0 conn(0x7fb95406a360 0x7fb954139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 --2- 172.21.15.67:0/3870007759 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb9540a0d50 0x7fb954096f50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 -- 172.21.15.67:0/3870007759 >> 172.21.15.67:0/3870007759 conn(0x7fb9540716e0 msgr2=0x7fb9540a5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 -- 172.21.15.67:0/3870007759 shutdown_connections 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 1 -- 172.21.15.67:0/3870007759 wait complete. 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 10 monclient: build_initial_monmap 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.256+0000 7fb95b71d640 10 monclient: monmap: 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:05:57.272 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 1 Processor -- start 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 1 -- start start 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 10 monclient: init 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 10 monclient: _reopen_session rank -1 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:05:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 0x7fb954141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-b con 0x7fb954069880 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 0x7fb954135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.257+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-c con 0x7fb95406a360 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 0x7fb954136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9540a0d50 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb954137ad0 con 0x7fb9540a0d50 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95406aac0 con 0x7fb954069880 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540a4820 con 0x7fb95406a360 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): _renew_subs 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb95b71d640 10 monclient(hunting): authenticate will time out at 1589.726028s 2024-09-15T02:05:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb9594b9640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 msgr2=0x7fb954141f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb9594b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 0x7fb954141f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 0x7fb954136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 0x7fb954135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 0x7fb954136530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:37850/0 (socket says 172.21.15.67:37850) 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 1 -- 172.21.15.67:0/2796869117 learned_addr learned my addr 172.21.15.67:0/2796869117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): get_auth_request con 0x7fb9540a0d50 auth_method 0 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): get_auth_request con 0x7fb95406a360 auth_method 0 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:05:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb959cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient(hunting): handle_auth_done global_id 184157 payload 824 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 -- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 msgr2=0x7fb954141f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 0x7fb954141f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 -- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 msgr2=0x7fb954136530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 0x7fb954136530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient: _finish_hunting 0 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 monclient: found mon.noname-c 2024-09-15T02:05:57.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.258+0000 7fb958cb8640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb950009450 con 0x7fb95406a360 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.260+0000 7fb958cb8640 1 -- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 msgr2=0x7fb954135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.260+0000 7fb958cb8640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 0x7fb954135ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb94400b4f0 tx=0x7fb944006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944011840 con 0x7fb95406a360 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: dump: 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.277 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: _finish_auth 0 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9440119c0 con 0x7fb95406a360 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 5 monclient: authenticate success, global_id 184157 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 10 monclient: _renew_subs 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95411c150 con 0x7fb95406a360 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:05:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944011d80 con 0x7fb95406a360 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb9497fa640 10 monclient: dump: 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:05:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb94b7fe640 4 set_mon_vals no callback set 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 10 monclient: _renew_subs 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.261+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb954132380 con 0x7fb95406a360 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb944010590 con 0x7fb95406a360 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb95b71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb95b71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb95b71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb9497fa640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 0x7fb93803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb94404c2c0 con 0x7fb95406a360 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb9594b9640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 0x7fb93803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:05:57.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.262+0000 7fb9594b9640 10 monclient: get_auth_request con 0x7fb938038740 auth_method 0 2024-09-15T02:05:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.264+0000 7fb9594b9640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 0x7fb93803abf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb954072080 tx=0x7fb950032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:05:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.271+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:05:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.271+0000 7fb9497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:05:57.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.271+0000 7fb9497fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:05:57.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe070d1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:05:57.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe070d1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:57.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe070d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:05:57.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe06c0724a0 con 0x7fe06c131b20 2024-09-15T02:05:57.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe0627fc640 1 -- 172.21.15.17:0/3265182956 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fe05403b610 con 0x7fe06c131b20 2024-09-15T02:05:57.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe0627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:05:57.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.478+0000 7fe0627fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:05:57.481 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:05:57.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 msgr2=0x7fe04c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 0x7fe04c03aba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe064006fd0 tx=0x7fe064006d70 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 10 monclient: shutdown 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 msgr2=0x7fe06c132970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe054035040 tx=0x7fe054032000 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 shutdown_connections 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe04c0386f0 0x7fe04c03aba0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe06c131b20 0x7fe06c132970 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe06c132430 0x7fe06c132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 --2- 172.21.15.17:0/3265182956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c0a0280 0x7fe06c139540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:05:57.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 >> 172.21.15.17:0/3265182956 conn(0x7fe06c071270 msgr2=0x7fe06c0a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:05:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 shutdown_connections 2024-09-15T02:05:57.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:05:57.480+0000 7fe070d1d640 1 -- 172.21.15.17:0/3265182956 wait complete. 2024-09-15T02:05:57.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.497+0000 7fb95b71d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:05:57.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.497+0000 7fb95b71d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:05:57.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.497+0000 7fb95b71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:05:57.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:05:57.497+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:06:00.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:00.257+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:00.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:00.257+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:00.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:00.257+0000 7fb93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:00.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:00.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:00.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 10 monclient: build_initial_monmap 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 10 monclient: monmap: 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:00.600914+0000 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:00.600914+0000 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:00.602 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 1 Processor -- start 2024-09-15T02:06:00.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 1 -- start start 2024-09-15T02:06:00.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.600+0000 7fe0aa19d640 10 monclient: init 2024-09-15T02:06:00.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:00.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 0x7fe0a4132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a40a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe0a40980a0 0x7fe0a40984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a40980a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0a40989e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40a01e0 con 0x7fe0a40989e0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40a1580 con 0x7fe0a40980a0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40a5c80 con 0x7fe0a40a58b0 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0aa19d640 10 monclient(hunting): authenticate will time out at 1620.597547s 2024-09-15T02:06:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a37fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 msgr2=0x7fe0a4132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe0a40980a0 0x7fe0a40984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 0x7fe0a4132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe0a40980a0 0x7fe0a40984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55872/0 (socket says 172.21.15.17:55872) 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 1 -- 172.21.15.17:0/2525192530 learned_addr learned my addr 172.21.15.17:0/2525192530 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request con 0x7fe0a40980a0 auth_method 0 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a3fff640 1 --1- 172.21.15.17:0/2525192530 >> v1:172.21.15.17:6789/0 conn(0x7fe0a40989e0 0x7fe0a4068ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:55832/0 (socket says 172.21.15.17:55832) 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:00.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.601+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:00.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.603+0000 7fe0a3fff640 1 -- 172.21.15.17:0/2525192530 >> v1:172.21.15.17:6789/0 conn(0x7fe0a40989e0 legacy=0x7fe0a4068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:00.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_done global_id 105390 payload 274 2024-09-15T02:06:00.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 1 -- 172.21.15.17:0/2525192530 >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 msgr2=0x7fe0a4132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:00.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 1 --2- 172.21.15.17:0/2525192530 >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 0x7fe0a4132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 1 -- 172.21.15.17:0/2525192530 >> v1:172.21.15.17:6789/0 conn(0x7fe0a40989e0 legacy=0x7fe0a4068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 1 monclient: found mon.noname-a 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.604+0000 7fe0a2ffd640 1 -- 172.21.15.17:0/2525192530 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a41326c0 con 0x7fe0a40980a0 2024-09-15T02:06:00.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a2ffd640 1 -- 172.21.15.17:0/2525192530 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 msgr2=0x7fe0a40984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:00.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a2ffd640 1 --2- 172.21.15.17:0/2525192530 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a40984a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40a60e0 tx=0x7fe0980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36ad8abe9c518e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 1 -- 172.21.15.17:0/2525192530 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09802f4b0 con 0x7fe0a40980a0 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: dump: 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 5 monclient: authenticate success, global_id 105390 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 1 -- 172.21.15.17:0/2525192530 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09802f610 con 0x7fe0a40980a0 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 4 set_mon_vals no callback set 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 1 -- 172.21.15.17:0/2525192530 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09802f9b0 con 0x7fe0a40980a0 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0a27fc640 10 monclient: dump: 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:00.612 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 10 monclient: shutdown 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 1 -- 172.21.15.17:0/2525192530 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 msgr2=0x7fe0a40984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.607+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/2525192530 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a40984a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40a60e0 tx=0x7fe0980091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 -- 172.21.15.17:0/2525192530 shutdown_connections 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/2525192530 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a40984a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/2525192530 >> v2:172.21.15.67:3300/0 conn(0x7fe0a40a58b0 0x7fe0a4132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 -- 172.21.15.17:0/2525192530 >> 172.21.15.17:0/2525192530 conn(0x7fe0a4070ed0 msgr2=0x7fe0a40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 -- 172.21.15.17:0/2525192530 shutdown_connections 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 1 -- 172.21.15.17:0/2525192530 wait complete. 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 10 monclient: build_initial_monmap 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.608+0000 7fe0aa19d640 10 monclient: monmap: 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.614 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 Processor -- start 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 -- start start 2024-09-15T02:06:00.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient: init 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a40980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe0a40a58b0 0x7fe0a4130e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a40a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 0x7fe0a411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a4131340 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a411c970 con 0x7fe0a40980a0 2024-09-15T02:06:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a409e9c0 con 0x7fe0a4131340 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4098dc0 con 0x7fe0a40a58b0 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0aa19d640 10 monclient(hunting): authenticate will time out at 1620.605741s 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55884/0 (socket says 172.21.15.17:55884) 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 1 -- 172.21.15.17:0/1506479492 learned_addr learned my addr 172.21.15.17:0/1506479492 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request con 0x7fe0a40980a0 auth_method 0 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:00.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a3fff640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 msgr2=0x7fe0a411b7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a2ffd640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe0a40a58b0 0x7fe0a4130e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a3fff640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 0x7fe0a411b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request con 0x7fe0a40a58b0 auth_method 0 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.609+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_done global_id 105393 payload 824 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe0a40a58b0 msgr2=0x7fe0a4130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe0a40a58b0 0x7fe0a4130e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 msgr2=0x7fe0a411b7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 0x7fe0a411b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 monclient: found mon.noname-a 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.610+0000 7fe0a37fe640 1 -- 172.21.15.17:0/1506479492 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe098009450 con 0x7fe0a40980a0 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0a37fe640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 msgr2=0x7fe0a409b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0a37fe640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe09400b600 tx=0x7fe0940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094015070 con 0x7fe0a40980a0 2024-09-15T02:06:00.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: dump: 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:00.623 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094003d90 con 0x7fe0a40980a0 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094006380 con 0x7fe0a40980a0 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 5 monclient: authenticate success, global_id 105393 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 10 monclient: _renew_subs 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a411cfc0 con 0x7fe0a40980a0 2024-09-15T02:06:00.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0a1ffb640 4 set_mon_vals no callback set 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe08b7fe640 10 monclient: dump: 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:00.625 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 10 monclient: _renew_subs 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.612+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a4142180 con 0x7fe0a40980a0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe094010440 con 0x7fe0a40980a0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe0aa19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe0aa19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe0aa19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a40724a0 con 0x7fe0a40980a0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe08b7fe640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 0x7fe09003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.613+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe09404bfd0 con 0x7fe0a40980a0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.614+0000 7fe0a2ffd640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 0x7fe09003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.614+0000 7fe0a2ffd640 10 monclient: get_auth_request con 0x7fe090038740 auth_method 0 2024-09-15T02:06:00.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.616+0000 7fe0a2ffd640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 0x7fe09003abf0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe098031000 tx=0x7fe098032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:00.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.618+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe094006770 con 0x7fe0a40980a0 2024-09-15T02:06:00.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.618+0000 7fe08b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:00.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.618+0000 7fe08b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.845+0000 7fe0aa19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.845+0000 7fe0aa19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.845+0000 7fe0aa19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.845+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe0a409fc90 con 0x7fe0a40980a0 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.846+0000 7fe08b7fe640 1 -- 172.21.15.17:0/1506479492 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fe094018310 con 0x7fe0a40980a0 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.846+0000 7fe08b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.846+0000 7fe08b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:00.848 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 msgr2=0x7fe09003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 0x7fe09003abf0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe098031000 tx=0x7fe098032000 comp rx=0 tx=0).stop 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 10 monclient: shutdown 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 msgr2=0x7fe0a409b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe09400b600 tx=0x7fe0940091a0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 shutdown_connections 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe090038740 0x7fe09003abf0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4131340 0x7fe0a411b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe0a40a58b0 0x7fe0a4130e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 --2- 172.21.15.17:0/1506479492 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe0a40980a0 0x7fe0a409b140 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 >> 172.21.15.17:0/1506479492 conn(0x7fe0a4070ed0 msgr2=0x7fe0a4096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 shutdown_connections 2024-09-15T02:06:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:00.848+0000 7fe0aa19d640 1 -- 172.21.15.17:0/1506479492 wait complete. 2024-09-15T02:06:01.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:01.257+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:01.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:01.257+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:02.257+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:02.257+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:03.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:03.257+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:03.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:03.257+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:03.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:03.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.967+0000 7f316cf9d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:03.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.967+0000 7f316cf9d640 10 monclient: build_initial_monmap 2024-09-15T02:06:03.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.967+0000 7f316cf9d640 10 monclient: monmap: 2024-09-15T02:06:03.969 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:03.968460+0000 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:03.968460+0000 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:03.970 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 1 Processor -- start 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 1 -- start start 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 10 monclient: init 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T02:06:03.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f31680a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 0x7f3168069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.968+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f316809ccf0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3168069ac0 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680a5b50 con 0x7f3168069ac0 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31680a18b0 con 0x7f316809ccf0 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316809c5f0 con 0x7f31680a5780 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:03.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316cf9d640 10 monclient(hunting): authenticate will time out at 1623.965059s 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 0x7f3168069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 0x7f3168069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49882/0 (socket says 172.21.15.17:49882) 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36892/0 (socket says 172.21.15.17:36892) 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 1 -- 172.21.15.17:0/4080134776 learned_addr learned my addr 172.21.15.17:0/4080134776 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 10 monclient(hunting): get_auth_request con 0x7f316809ccf0 auth_method 0 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f3166d9d640 1 --1- 172.21.15.17:0/4080134776 >> v1:172.21.15.17:6789/0 conn(0x7f3168069ac0 0x7f3168138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39940/0 (socket says 172.21.15.17:39940) 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f315ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): get_auth_request con 0x7f31680a5780 auth_method 0 2024-09-15T02:06:03.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient(hunting): handle_auth_done global_id 184163 payload 274 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 -- 172.21.15.17:0/4080134776 >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 msgr2=0x7f3168069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 --2- 172.21.15.17:0/4080134776 >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 0x7f3168069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 -- 172.21.15.17:0/4080134776 >> v1:172.21.15.17:6789/0 conn(0x7f3168069ac0 legacy=0x7f3168138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient: _finish_hunting 0 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 monclient: found mon.noname-b 2024-09-15T02:06:03.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:03.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:03.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.969+0000 7f316659c640 1 -- 172.21.15.17:0/4080134776 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316809d210 con 0x7f31680a5780 2024-09-15T02:06:03.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316659c640 1 -- 172.21.15.17:0/4080134776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 msgr2=0x7f316809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:03.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316659c640 1 --2- 172.21.15.17:0/4080134776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3158009850 tx=0x7f31580090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d74ee4179b9155c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:03.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 1 -- 172.21.15.17:0/4080134776 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315802adf0 con 0x7f31680a5780 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: dump: 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:03.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: _finish_auth 0 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 1 -- 172.21.15.17:0/4080134776 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158032440 con 0x7f31680a5780 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316cf9d640 5 monclient: authenticate success, global_id 184163 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316cf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316cf9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 1 -- 172.21.15.17:0/4080134776 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580327e0 con 0x7f31680a5780 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f316cf9d640 4 set_mon_vals no callback set 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.972+0000 7f3165d9b640 10 monclient: dump: 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 10 monclient: shutdown 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:03.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 -- 172.21.15.17:0/4080134776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 msgr2=0x7f316809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 --2- 172.21.15.17:0/4080134776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3158009850 tx=0x7f31580090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 -- 172.21.15.17:0/4080134776 shutdown_connections 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 --2- 172.21.15.17:0/4080134776 >> v2:172.21.15.17:3300/0 conn(0x7f316809ccf0 0x7f3168069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 --2- 172.21.15.17:0/4080134776 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f316809bc90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 -- 172.21.15.17:0/4080134776 >> 172.21.15.17:0/4080134776 conn(0x7f3168070e10 msgr2=0x7f31680a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 -- 172.21.15.17:0/4080134776 shutdown_connections 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 1 -- 172.21.15.17:0/4080134776 wait complete. 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 10 monclient: build_initial_monmap 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.973+0000 7f316cf9d640 10 monclient: monmap: 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:03.987 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 Processor -- start 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 -- start start 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient: init 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 0x7f316813dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3168069ac0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:03.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f316809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f3168141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f31680a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3168121620 con 0x7f316809ccf0 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31681225d0 con 0x7f3168069ac0 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31681227d0 con 0x7f31680a5780 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316cf9d640 10 monclient(hunting): authenticate will time out at 1623.970595s 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:03.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316659c640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 msgr2=0x7f316813dda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f316659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 0x7f316813dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49892/0 (socket says 172.21.15.17:49892) 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 1 -- 172.21.15.17:0/3140880608 learned_addr learned my addr 172.21.15.17:0/3140880608 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): get_auth_request con 0x7f316809ccf0 auth_method 0 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f3168141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): get_auth_request con 0x7f31680a5780 auth_method 0 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f315ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.974+0000 7f3166d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 10 monclient(hunting): handle_auth_done global_id 105399 payload 824 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 -- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 msgr2=0x7f3168141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f3168141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 -- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 msgr2=0x7f316813dda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 0x7f316813dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:03.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 monclient: found mon.noname-a 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f315ffff640 1 -- 172.21.15.17:0/3140880608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009390 con 0x7f316809ccf0 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.975+0000 7f3166d9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f3168141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315ffff640 1 -- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 msgr2=0x7f316813d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315ffff640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f315000d840 tx=0x7f3150008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000de30 con 0x7f316809ccf0 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: dump: 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:03.992 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3150009ac0 con 0x7f316809ccf0 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 5 monclient: authenticate success, global_id 105399 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 10 monclient: _renew_subs 2024-09-15T02:06:03.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3168122d90 con 0x7f316809ccf0 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150009e80 con 0x7f316809ccf0 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f315e7fc640 10 monclient: dump: 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:03.994 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f3164d99640 4 set_mon_vals no callback set 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 10 monclient: _renew_subs 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.977+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316813e8a0 con 0x7f316809ccf0 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3150014620 con 0x7f316809ccf0 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f316cf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:03.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f316cf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f316cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680724a0 con 0x7f316809ccf0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f315e7fc640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 0x7f314803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.978+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f315004ce20 con 0x7f316809ccf0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.979+0000 7f316659c640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 0x7f314803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.979+0000 7f316659c640 10 monclient: get_auth_request con 0x7f31480386f0 auth_method 0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.981+0000 7f316659c640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 0x7f314803aba0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f31680a0c50 tx=0x7f3158032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:03.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.983+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3150025ab0 con 0x7f316809ccf0 2024-09-15T02:06:03.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.983+0000 7f315e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:03.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:03.983+0000 7f315e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f316cf9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f316cf9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f316cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f316806f4f0 con 0x7f316809ccf0 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f315e7fc640 1 -- 172.21.15.17:0/3140880608 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f3150025ab0 con 0x7f316809ccf0 2024-09-15T02:06:04.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f315e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:04.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.207+0000 7f315e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:04.210 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:04.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 msgr2=0x7f314803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:04.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 0x7f314803aba0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f31680a0c50 tx=0x7f3158032000 comp rx=0 tx=0).stop 2024-09-15T02:06:04.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 10 monclient: shutdown 2024-09-15T02:06:04.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 msgr2=0x7f316813d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f315000d840 tx=0x7f3150008040 comp rx=0 tx=0).stop 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 shutdown_connections 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f31480386f0 0x7f314803aba0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f316809ccf0 0x7f316813d450 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f31680a5780 0x7f3168141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 --2- 172.21.15.17:0/3140880608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3168069ac0 0x7f316813dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 >> 172.21.15.17:0/3140880608 conn(0x7f3168070e10 msgr2=0x7f316809af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 shutdown_connections 2024-09-15T02:06:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:04.209+0000 7f316cf9d640 1 -- 172.21.15.17:0/3140880608 wait complete. 2024-09-15T02:06:04.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:04.257+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:04.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:04.257+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:05.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:05.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:05.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:06.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:06.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:06.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:06.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:07.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:07.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:07.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:07.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:07.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:07.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.329+0000 7f51812d4640 10 monclient: get_monmap_and_config 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.329+0000 7f51812d4640 10 monclient: build_initial_monmap 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.329+0000 7f51812d4640 10 monclient: monmap: 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:07.330780+0000 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:07.330780+0000 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:07.332 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:07.333 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:07.333 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:07.333 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:07.333 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:07.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.330+0000 7f51812d4640 1 Processor -- start 2024-09-15T02:06:07.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.330+0000 7f51812d4640 1 -- start start 2024-09-15T02:06:07.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.330+0000 7f51812d4640 10 monclient: init 2024-09-15T02:06:07.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:07.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T02:06:07.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:07.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-c con 0x7f517c098210 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-a con 0x7f517c09cd90 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f517c098b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c09d6d0 con 0x7f517c098b20 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c09e170 con 0x7f517c09cd90 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c132430 con 0x7f517c098210 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f51812d4640 10 monclient(hunting): authenticate will time out at 1627.327172s 2024-09-15T02:06:07.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517affd640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 msgr2=0x7f517c131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f517c098b20 0x7f517c067090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:39946/0 (socket says 172.21.15.17:39946) 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49894/0 (socket says 172.21.15.17:49894) 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517b7fe640 1 -- 172.21.15.17:0/3501137159 learned_addr learned my addr 172.21.15.17:0/3501137159 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f517c09cd90 auth_method 0 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.331+0000 7f517a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.333+0000 7f517b7fe640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.17:6789/0 conn(0x7f517c098b20 legacy=0x7f517c067090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.333+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:07.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.333+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:07.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.333+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:07.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.333+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:07.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.532+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 msgr2=0x7f517c131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:07.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.532+0000 7f517affd640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f517c09cd90 auth_method 0 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:07.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.534+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:07.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.933+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 msgr2=0x7f517c131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:07.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.933+0000 7f517affd640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f517c09cd90 auth_method 0 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:07.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:07.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:07.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:07.935+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:08.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:08.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:08.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:08.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:08.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.734+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 msgr2=0x7f517c131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.734+0000 7f517affd640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f517c09cd90 auth_method 0 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:08.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:08.736+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:06:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:09.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:09.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:10.259+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:10.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:10.259+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:10.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:10.259+0000 7fb93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): tick 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 monclient(hunting): continuing hunt 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 msgr2=0x7f517c131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.67:3300/0 conn(0x7f517c098210 0x7f517c131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 -- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 msgr2=0x7f517c0985e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 --2- 172.21.15.17:0/3501137159 >> v2:172.21.15.17:3300/0 conn(0x7f517c09cd90 0x7f517c0985e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.17:6789/0 conn(0x7f517c098b20 legacy=0x7f517c067090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:10.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51680036f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f51680073d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 -- 172.21.15.17:0/3501137159 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c1325a0 con 0x7f51680036f0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 1 -- 172.21.15.17:0/3501137159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c09e170 con 0x7f51680073d0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f5178ff9640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.67:6789/0 conn(0x7f51680036f0 legacy=0x7f5168005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f51680073d0 auth_method 0 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:10.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:10.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:10.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:10.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:10.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.331+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.67:6789/0 conn(0x7f51680036f0 legacy=0x7f5168005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f51680073d0 auth_method 0 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:10.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:10.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:10.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:10.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.532+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f51680073d0 auth_method 0 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.67:6789/0 conn(0x7f51680036f0 legacy=0x7f5168005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:10.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:10.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:10.933+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:11.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.258+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:06:11.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.258+0000 7fb9497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:06:11.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.258+0000 7fb9497fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:11.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.258+0000 7fb93ffff640 10 monclient: tick 2024-09-15T02:06:11.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.258+0000 7fb93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:11.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.418+0000 7fb95b71d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:06:11.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.419+0000 7fb95b71d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:06:11.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.419+0000 7fb95b71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.419+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:06:11.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.420+0000 7fb9497fa640 1 -- 172.21.15.67:0/2796869117 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb9541387b0 con 0x7fb95406a360 2024-09-15T02:06:11.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.420+0000 7fb9497fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T02:06:11.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.420+0000 7fb9497fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 msgr2=0x7fb93803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 0x7fb93803abf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb954072080 tx=0x7fb950032000 comp rx=0 tx=0).stop 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 10 monclient: shutdown 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 msgr2=0x7fb954135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 0x7fb954135ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb94400b4f0 tx=0x7fb944006f50 comp rx=0 tx=0).stop 2024-09-15T02:06:11.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 shutdown_connections 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb938038740 0x7fb93803abf0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb9540a0d50 0x7fb954136530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb95406a360 0x7fb954135ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 --2- 172.21.15.67:0/2796869117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb954069880 0x7fb954141f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 >> 172.21.15.67:0/2796869117 conn(0x7fb9540716e0 msgr2=0x7fb954097820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 shutdown_connections 2024-09-15T02:06:11.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:06:11.422+0000 7fb95b71d640 1 -- 172.21.15.67:0/2796869117 wait complete. 2024-09-15T02:06:11.432 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-15T02:06:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-15T02:06:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:06:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:06:11.433 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:06:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-15T02:06:11.508 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:06:11.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T02:06:11.509 INFO:tasks.workunit.client.0.smithi067.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T02:06:11.509 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T02:06:11.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517affd640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.67:6789/0 conn(0x7f51680036f0 legacy=0x7f5168005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): get_auth_request con 0x7f51680073d0 auth_method 0 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient(hunting): handle_auth_done global_id 184178 payload 274 2024-09-15T02:06:11.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> v1:172.21.15.67:6789/0 conn(0x7f51680036f0 legacy=0x7f5168005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:06:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 1 monclient: found mon.noname-b 2024-09-15T02:06:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.734+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516800b030 con 0x7f51680073d0 2024-09-15T02:06:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.737+0000 7f517a7fc640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.737+0000 7f517a7fc640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5170006ea0 tx=0x7f5170008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e66d6cf21a651273 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 1 -- 172.21.15.17:0/3501137159 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517002cb70 con 0x7f51680073d0 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: dump: 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: _finish_auth 0 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 1 -- 172.21.15.17:0/3501137159 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170030070 con 0x7f51680073d0 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f51812d4640 5 monclient: authenticate success, global_id 184178 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f51812d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f51812d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f51812d4640 4 set_mon_vals no callback set 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 1 -- 172.21.15.17:0/3501137159 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170036400 con 0x7f51680073d0 2024-09-15T02:06:11.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.738+0000 7f5179ffb640 10 monclient: dump: 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:11.742 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 10 monclient: shutdown 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 -- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 msgr2=0x7f5168009860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 --2- 172.21.15.17:0/3501137159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f51680073d0 0x7f5168009860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5170006ea0 tx=0x7f5170008040 comp rx=0 tx=0).stop 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f517b7fe640 1 -- 172.21.15.17:0/3501137159 reap_dead start 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 -- 172.21.15.17:0/3501137159 shutdown_connections 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 -- 172.21.15.17:0/3501137159 >> 172.21.15.17:0/3501137159 conn(0x7f517c0a1ee0 msgr2=0x7f517c0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 -- 172.21.15.17:0/3501137159 shutdown_connections 2024-09-15T02:06:11.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.739+0000 7f51812d4640 1 -- 172.21.15.17:0/3501137159 wait complete. 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.740+0000 7f51812d4640 10 monclient: build_initial_monmap 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.740+0000 7f51812d4640 10 monclient: monmap: 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:11.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.740+0000 7f51812d4640 1 Processor -- start 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.740+0000 7f51812d4640 1 -- start start 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.740+0000 7f51812d4640 10 monclient: init 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 0x7f517c130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-a con 0x7f51680073d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 0x7f517c140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-b con 0x7f517c131120 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): picked mon.noname-c con 0x7f517c141460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:11.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c11f2d0 con 0x7f51680073d0 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c09d4c0 con 0x7f517c131120 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c11cba0 con 0x7f517c141460 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f51812d4640 10 monclient(hunting): authenticate will time out at 1631.737228s 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517a7fc640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 msgr2=0x7f517c140f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 0x7f517c140f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 0x7f517c130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36930/0 (socket says 172.21.15.17:36930) 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 -- 172.21.15.17:0/1869280809 learned_addr learned my addr 172.21.15.17:0/1869280809 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): get_auth_request con 0x7f517c141460 auth_method 0 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): get_auth_request con 0x7f51680073d0 auth_method 0 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient(hunting): handle_auth_done global_id 184181 payload 824 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 -- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 msgr2=0x7f517c140f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 0x7f517c140f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 -- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 msgr2=0x7f517c130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 0x7f517c130be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 monclient: found mon.noname-c 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517b7fe640 1 -- 172.21.15.17:0/1869280809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517000e040 con 0x7f517c141460 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.741+0000 7f517affd640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 0x7f517c130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:11.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f517b7fe640 1 -- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 msgr2=0x7f517c11b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f517b7fe640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f517c09d880 tx=0x7f5174001b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517400cec0 con 0x7f517c141460 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: dump: 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:11.753 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: _finish_auth 0 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174007530 con 0x7f517c141460 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 5 monclient: authenticate success, global_id 184181 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 10 monclient: _renew_subs 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c11eff0 con 0x7f517c141460 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:11.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51740078d0 con 0x7f517c141460 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f5162ffd640 10 monclient: dump: 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:11.755 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 10 monclient: _renew_subs 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51797fa640 4 set_mon_vals no callback set 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.744+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c11ed10 con 0x7f517c141460 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5174013620 con 0x7f517c141460 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f51812d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f51812d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f51812d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c0724a0 con 0x7f517c141460 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f5162ffd640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 0x7f516803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:11.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.745+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f517404c0f0 con 0x7f517c141460 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.746+0000 7f517affd640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 0x7f516803bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.746+0000 7f517affd640 10 monclient: get_auth_request con 0x7f51680396e0 auth_method 0 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.748+0000 7f517affd640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 0x7f516803bb90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f516c009fd0 tx=0x7f516c009440 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.750+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5174007a70 con 0x7f517c141460 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.750+0000 7f5162ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:11.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.750+0000 7f5162ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.983+0000 7f51812d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.983+0000 7f51812d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.983+0000 7f51812d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.983+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f517c0689a0 con 0x7f517c141460 2024-09-15T02:06:11.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.984+0000 7f5162ffd640 1 -- 172.21.15.17:0/1869280809 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f517401a770 con 0x7f517c141460 2024-09-15T02:06:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.984+0000 7f5162ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:11.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.984+0000 7f5162ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:11.986 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:11.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 msgr2=0x7f516803bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 0x7f516803bb90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f516c009fd0 tx=0x7f516c009440 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 10 monclient: shutdown 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 msgr2=0x7f517c11b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.985+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f517c09d880 tx=0x7f5174001b10 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 shutdown_connections 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f51680396e0 0x7f516803bb90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f517c141460 0x7f517c11b730 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517c131120 0x7f517c140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 --2- 172.21.15.17:0/1869280809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f51680073d0 0x7f517c130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 >> 172.21.15.17:0/1869280809 conn(0x7f517c0a1ee0 msgr2=0x7f517c0992c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 shutdown_connections 2024-09-15T02:06:11.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:11.986+0000 7f51812d4640 1 -- 172.21.15.17:0/1869280809 wait complete. 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-15T02:06:11.997 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=0 2024-09-15T02:06:11.998 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:06:11.998 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:06:11.998 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:06:11.999 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:06:11.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:06:12.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:06:12.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.098+0000 7fcf42f1d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.098+0000 7fcf42f1d640 10 monclient: build_initial_monmap 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.098+0000 7fcf42f1d640 10 monclient: monmap: 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:12.099162+0000 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:12.099162+0000 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:12.100 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:12.101 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:12.101 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:12.101 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 Processor -- start 2024-09-15T02:06:12.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 -- start start 2024-09-15T02:06:12.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient: init 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 0x7fcf3c12e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c09a590 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcf3c09a960 0x7fcf3c12ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c09a960 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c0985d0 con 0x7fcf3c09a960 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c09d200 con 0x7fcf3c09a590 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c097c30 con 0x7fcf3c097320 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf42f1d640 10 monclient(hunting): authenticate will time out at 1632.095591s 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf3bfff640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 msgr2=0x7fcf3c12e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf3bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 0x7fcf3c12e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcf3c09a960 0x7fcf3c12ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcf3c09a960 0x7fcf3c12ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49938/0 (socket says 172.21.15.17:49938) 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 1 -- 172.21.15.17:0/1926573067 learned_addr learned my addr 172.21.15.17:0/1926573067 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36936/0 (socket says 172.21.15.17:36936) 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): get_auth_request con 0x7fcf3c09a960 auth_method 0 2024-09-15T02:06:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): get_auth_request con 0x7fcf3c097320 auth_method 0 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.099+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 10 monclient(hunting): handle_auth_done global_id 105420 payload 274 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 -- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 msgr2=0x7fcf3c0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 -- 172.21.15.17:0/1926573067 >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 msgr2=0x7fcf3c12e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1926573067 >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 0x7fcf3c12e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 10 monclient: _finish_hunting 0 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 monclient: found mon.noname-a 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf414ba640 1 -- 172.21.15.17:0/1926573067 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0992f0 con 0x7fcf3c09a960 2024-09-15T02:06:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.100+0000 7fcf40cb9640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:12.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.102+0000 7fcf414ba640 1 -- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a960 msgr2=0x7fcf3c12ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.102+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a960 0x7fcf3c12ebe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c068fc0 tx=0x7fcf34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915938b8abd32f77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 1 -- 172.21.15.17:0/1926573067 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf34007610 con 0x7fcf3c09a960 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: dump: 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.115 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 1 -- 172.21.15.17:0/1926573067 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3402d050 con 0x7fcf3c09a960 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 5 monclient: authenticate success, global_id 105420 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 4 set_mon_vals no callback set 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 1 -- 172.21.15.17:0/1926573067 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf340079a0 con 0x7fcf3c09a960 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:12.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf3b7fe640 10 monclient: dump: 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 10 monclient: shutdown 2024-09-15T02:06:12.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a960 msgr2=0x7fcf3c12ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a960 0x7fcf3c12ebe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c068fc0 tx=0x7fcf34009840 comp rx=0 tx=0).stop 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1926573067 shutdown_connections 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1926573067 >> v2:172.21.15.67:3300/0 conn(0x7fcf3c09a590 0x7fcf3c12e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a960 0x7fcf3c12ebe0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1926573067 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c097320 0x7fcf3c0976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1926573067 >> 172.21.15.17:0/1926573067 conn(0x7fcf3c0a1000 msgr2=0x7fcf3c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1926573067 shutdown_connections 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.103+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1926573067 wait complete. 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.104+0000 7fcf42f1d640 10 monclient: build_initial_monmap 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.104+0000 7fcf42f1d640 10 monclient: monmap: 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.104+0000 7fcf42f1d640 1 Processor -- start 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.104+0000 7fcf42f1d640 1 -- start start 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.104+0000 7fcf42f1d640 10 monclient: init 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:12.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c097320 0x7fcf3c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c097320 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 0x7fcf3c133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c09a590 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 0x7fcf3c1340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c09a960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c0a00a0 con 0x7fcf3c09a590 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c1185b0 con 0x7fcf3c097320 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c117650 con 0x7fcf3c09a960 2024-09-15T02:06:12.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf42f1d640 10 monclient(hunting): authenticate will time out at 1632.101168s 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf40cb9640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c097320 msgr2=0x7fcf3c06ea40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf40cb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c097320 0x7fcf3c06ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 0x7fcf3c133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 0x7fcf3c1340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 0x7fcf3c133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49946/0 (socket says 172.21.15.17:49946) 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 0x7fcf3c1340f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36948/0 (socket says 172.21.15.17:36948) 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 -- 172.21.15.17:0/1050498329 learned_addr learned my addr 172.21.15.17:0/1050498329 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): get_auth_request con 0x7fcf3c09a590 auth_method 0 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): get_auth_request con 0x7fcf3c09a960 auth_method 0 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient(hunting): handle_auth_done global_id 105423 payload 824 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 msgr2=0x7fcf3c1340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 0x7fcf3c1340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c097320 msgr2=0x7fcf3c06ea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c097320 0x7fcf3c06ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 monclient: found mon.noname-a 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf3bfff640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30009390 con 0x7fcf3c09a590 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.105+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcf3c09a960 0x7fcf3c1340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.107+0000 7fcf3bfff640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 msgr2=0x7fcf3c133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.107+0000 7fcf3bfff640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 0x7fcf3c133bb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800b600 tx=0x7fcf280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28015070 con 0x7fcf3c09a590 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: dump: 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: _finish_auth 0 2024-09-15T02:06:12.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 5 monclient: authenticate success, global_id 105423 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28003d90 con 0x7fcf3c09a590 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 monclient: _renew_subs 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf34009ae0 con 0x7fcf3c09a590 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28006380 con 0x7fcf3c09a590 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 10 monclient: dump: 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.125 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 monclient: _renew_subs 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c134ec0 con 0x7fcf3c09a590 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf3affd640 4 set_mon_vals no callback set 2024-09-15T02:06:12.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 0x7fcf3c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c118a30 con 0x7fcf3c134630 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.108+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcf28010440 con 0x7fcf3c09a590 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 0x7fcf3c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient: get_auth_request con 0x7fcf3c134630 auth_method 0 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf38ff9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf200386f0 0x7fcf2003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fcf2804c0c0 con 0x7fcf3c09a590 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_done global_id 105426 payload 824 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf40cb9640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf3c118bd0 con 0x7fcf3c134630 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf200386f0 0x7fcf2003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.109+0000 7fcf414ba640 10 monclient: get_auth_request con 0x7fcf200386f0 auth_method 0 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf40cb9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 msgr2=0x7fcf3c141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf40cb9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 0x7fcf3c141aa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3000ccd0 con 0x7fcf3c134630 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 10 monclient: dump: 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.128 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcf3000ce30 con 0x7fcf3c134630 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.111+0000 7fcf38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.112+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 msgr2=0x7fcf3c141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.112+0000 7fcf38ff9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 0x7fcf3c141aa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf300090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.112+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 msgr2=0x7fcf3c141aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:12.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.112+0000 7fcf38ff9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c134630 0x7fcf3c141aa0 secure :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf300090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.112+0000 7fcf414ba640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf200386f0 0x7fcf2003aba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c099a60 tx=0x7fcf34036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf42f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:12.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 0x7fcf3c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf42f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c063c20 con 0x7fcf3c141fe0 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf40cb9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 0x7fcf3c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf40cb9640 10 monclient: get_auth_request con 0x7fcf3c141fe0 auth_method 0 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf40cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf40cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.117+0000 7fcf40cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.118+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.118+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.118+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.118+0000 7fcf40cb9640 10 monclient(hunting): handle_auth_done global_id 105429 payload 824 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.118+0000 7fcf40cb9640 1 -- 172.21.15.17:0/1050498329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcf3c142920 con 0x7fcf3c141fe0 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf40cb9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 msgr2=0x7fcf3c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf40cb9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 0x7fcf3c1423e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf3001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30023070 con 0x7fcf3c141fe0 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 10 monclient: dump: 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.133 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7fcf30013e20 con 0x7fcf3c141fe0 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 msgr2=0x7fcf3c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 0x7fcf3c1423e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf3001a000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 msgr2=0x7fcf3c1423e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.120+0000 7fcf38ff9640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c141fe0 0x7fcf3c1423e0 secure :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e8f0 tx=0x7fcf3001a000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 154, 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:12.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:12.148 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf200386f0 msgr2=0x7fcf2003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcf200386f0 0x7fcf2003aba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c099a60 tx=0x7fcf34036000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 10 monclient: shutdown 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 msgr2=0x7fcf3c133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 --2- 172.21.15.17:0/1050498329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf3c09a590 0x7fcf3c133bb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800b600 tx=0x7fcf280091a0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf414ba640 1 -- 172.21.15.17:0/1050498329 reap_dead start 2024-09-15T02:06:12.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf414ba640 1 -- 172.21.15.17:0/1050498329 reap_dead start 2024-09-15T02:06:12.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 shutdown_connections 2024-09-15T02:06:12.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 >> 172.21.15.17:0/1050498329 conn(0x7fcf3c0a1000 msgr2=0x7fcf3c069c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:12.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 shutdown_connections 2024-09-15T02:06:12.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.121+0000 7fcf42f1d640 1 -- 172.21.15.17:0/1050498329 wait complete. 2024-09-15T02:06:12.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:06:12.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:06:12.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.246+0000 7f38f979d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.246+0000 7f38f979d640 10 monclient: build_initial_monmap 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.246+0000 7f38f979d640 10 monclient: monmap: 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:12.247264+0000 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:12.247264+0000 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.248 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:12.249 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:12.249 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:12.249 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:12.249 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:12.249 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.246+0000 7f38f979d640 1 Processor -- start 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 -- start start 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient: init 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38f4096c10 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 0x7f38f4096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-c con 0x7f38f409d3b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-b con 0x7f38f4097520 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40983b0 con 0x7f38f4096c10 2024-09-15T02:06:12.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f40998b0 con 0x7f38f409d3b0 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4099260 con 0x7f38f4097520 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f979d640 10 monclient(hunting): authenticate will time out at 1632.243557s 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f2ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f38f4096c10 legacy=0x7f38f4099f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f27fc640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 msgr2=0x7f38f4096fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 0x7f38f4096fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36950/0 (socket says 172.21.15.17:36950) 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 -- 172.21.15.17:0/1800233053 learned_addr learned my addr 172.21.15.17:0/1800233053 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:12.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38f4097520 auth_method 0 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient(hunting): handle_auth_done global_id 184193 payload 274 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 -- 172.21.15.17:0/1800233053 >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 msgr2=0x7f38f4096fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 --2- 172.21.15.17:0/1800233053 >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 0x7f38f4096fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 -- 172.21.15.17:0/1800233053 >> v1:172.21.15.67:6789/0 conn(0x7f38f4096c10 legacy=0x7f38f4099f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:12.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 monclient: found mon.noname-b 2024-09-15T02:06:12.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:12.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:12.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.247+0000 7f38f37fe640 1 -- 172.21.15.17:0/1800233053 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4097920 con 0x7f38f4097520 2024-09-15T02:06:12.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.250+0000 7f38f37fe640 1 -- 172.21.15.17:0/1800233053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 msgr2=0x7f38f4135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.250+0000 7f38f37fe640 1 --2- 172.21.15.17:0/1800233053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009f20 tx=0x7f38ec009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80e224bbfa2ce071 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 1 -- 172.21.15.17:0/1800233053 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec007610 con 0x7f38f4097520 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: dump: 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.258 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 1 -- 172.21.15.17:0/1800233053 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec02d050 con 0x7f38f4097520 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f979d640 5 monclient: authenticate success, global_id 184193 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f979d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f979d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f979d640 4 set_mon_vals no callback set 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 1 -- 172.21.15.17:0/1800233053 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0079a0 con 0x7f38f4097520 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.252+0000 7f38f1ffb640 10 monclient: dump: 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 10 monclient: shutdown 2024-09-15T02:06:12.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 -- 172.21.15.17:0/1800233053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 msgr2=0x7f38f4135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 --2- 172.21.15.17:0/1800233053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009f20 tx=0x7f38ec009840 comp rx=0 tx=0).stop 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 -- 172.21.15.17:0/1800233053 shutdown_connections 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 --2- 172.21.15.17:0/1800233053 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4097520 0x7f38f4135b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 --2- 172.21.15.17:0/1800233053 >> v2:172.21.15.67:3300/0 conn(0x7f38f409d3b0 0x7f38f4096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 -- 172.21.15.17:0/1800233053 >> 172.21.15.17:0/1800233053 conn(0x7f38f40a1000 msgr2=0x7f38f40a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 -- 172.21.15.17:0/1800233053 shutdown_connections 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 1 -- 172.21.15.17:0/1800233053 wait complete. 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 10 monclient: build_initial_monmap 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.253+0000 7f38f979d640 10 monclient: monmap: 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 Processor -- start 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 -- start start 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient: init 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 0x7f38f406dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-a con 0x7f38f4096c10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38f4097520 0x7f38f406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-b con 0x7f38f4097520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): picked mon.noname-c con 0x7f38f409d3b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4134470 con 0x7f38f4096c10 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4133ef0 con 0x7f38f4097520 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4098e50 con 0x7f38f409d3b0 2024-09-15T02:06:12.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f979d640 10 monclient(hunting): authenticate will time out at 1632.250868s 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f27fc640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38f4097520 msgr2=0x7f38f406e160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f2ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 0x7f38f406dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38f4097520 0x7f38f406e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f2ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 0x7f38f406dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49966/0 (socket says 172.21.15.17:49966) 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f2ffd640 1 -- 172.21.15.17:0/987281599 learned_addr learned my addr 172.21.15.17:0/987281599 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): get_auth_request con 0x7f38f4096c10 auth_method 0 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.254+0000 7f38f37fe640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36964/0 (socket says 172.21.15.17:36964) 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38f409d3b0 auth_method 0 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient(hunting): handle_auth_done global_id 184196 payload 824 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38f4097520 msgr2=0x7f38f406e160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:12.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38f4097520 0x7f38f406e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 msgr2=0x7f38f406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 0x7f38f406dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 monclient: found mon.noname-c 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f37fe640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec009ae0 con 0x7f38f409d3b0 2024-09-15T02:06:12.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.255+0000 7f38f2ffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38f4096c10 0x7f38f406dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:12.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f37fe640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 msgr2=0x7f38f406e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f37fe640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38ec033000 tx=0x7f38ec036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec031cd0 con 0x7f38f409d3b0 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: dump: 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.271 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: _finish_auth 0 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec031e50 con 0x7f38f409d3b0 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f979d640 5 monclient: authenticate success, global_id 184196 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec03c670 con 0x7f38f409d3b0 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f979d640 10 monclient: _renew_subs 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f979d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f409f570 con 0x7f38f409d3b0 2024-09-15T02:06:12.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38daffd640 10 monclient: dump: 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38f979d640 10 monclient: _renew_subs 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.257+0000 7f38f17fa640 4 set_mon_vals no callback set 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38f979d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f406a130 con 0x7f38f409d3b0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f38ec03ca60 con 0x7f38f409d3b0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38daffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38e00386f0 0x7f38e003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.258+0000 7f38f979d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f979d640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 0x7f38f41285d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4069c30 con 0x7f38f406a320 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f38ec06d8f0 con 0x7f38f409d3b0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 0x7f38f41285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient: get_auth_request con 0x7f38f406a320 auth_method 0 2024-09-15T02:06:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f2ffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38e00386f0 0x7f38e003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f2ffd640 10 monclient: get_auth_request con 0x7f38e00386f0 auth_method 0 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 10 monclient(hunting): handle_auth_done global_id 184199 payload 824 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.259+0000 7f38f27fc640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38f40724a0 con 0x7f38f406a320 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38f27fc640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 msgr2=0x7f38f41285d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38f27fc640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 0x7f38f41285d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38f2ffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38e00386f0 0x7f38e003aba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f38e40098e0 tx=0x7f38e4009120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8003df0 con 0x7f38f406a320 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.261+0000 7f38daffd640 10 monclient: dump: 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.276 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f38e80050f0 con 0x7f38f406a320 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 msgr2=0x7f38f41285d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 0x7f38f41285d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8009120 comp rx=0 tx=0).stop 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 msgr2=0x7f38f41285d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.262+0000 7f38daffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f406a320 0x7f38f41285d0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8009120 comp rx=0 tx=0).stop 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f979d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f979d640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 0x7f38f4128f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f979d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4063c20 con 0x7f38f4128b10 2024-09-15T02:06:12.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f27fc640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 0x7f38f4128f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.267+0000 7f38f27fc640 10 monclient: get_auth_request con 0x7f38f4128b10 auth_method 0 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 10 monclient(hunting): handle_auth_done global_id 184202 payload 824 2024-09-15T02:06:12.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.268+0000 7f38f27fc640 1 -- 172.21.15.17:0/987281599 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f38f4129450 con 0x7f38f4128b10 2024-09-15T02:06:12.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38f27fc640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 msgr2=0x7f38f4128f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:12.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38f27fc640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 0x7f38f4128f10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:12.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80057c0 con 0x7f38f4128b10 2024-09-15T02:06:12.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:12.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.270+0000 7f38daffd640 10 monclient: dump: 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:12.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f38e801e620 con 0x7f38f4128b10 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 msgr2=0x7f38f4128f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 0x7f38f4128f10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8015000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 msgr2=0x7f38f4128f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38daffd640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f4128b10 0x7f38f4128f10 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38e800b440 tx=0x7f38e8015000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:12.281 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 154, 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.283 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:12.284 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:12.285 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:12.286 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.287 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.289 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.291 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:12.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:12.293 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:12.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:12.295 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38e00386f0 msgr2=0x7f38e003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f38e00386f0 0x7f38e003aba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f38e40098e0 tx=0x7f38e4009120 comp rx=0 tx=0).stop 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 10 monclient: shutdown 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 msgr2=0x7f38f406e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f37fe640 1 -- 172.21.15.17:0/987281599 reap_dead start 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38ec033000 tx=0x7f38ec036000 comp rx=0 tx=0).stop 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 shutdown_connections 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 --2- 172.21.15.17:0/987281599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38f409d3b0 0x7f38f406e6a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 >> 172.21.15.17:0/987281599 conn(0x7f38f40a1000 msgr2=0x7f38f409f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:12.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 shutdown_connections 2024-09-15T02:06:12.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:12.271+0000 7f38f979d640 1 -- 172.21.15.17:0/987281599 wait complete. 2024-09-15T02:06:12.298 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:06:12.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-15T02:06:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:06:12.487 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T02:06:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T02:06:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:06:12.798 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T02:06:12.799 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T02:06:12.799 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:06:13.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T02:06:13.113 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T02:06:13.113 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-15T02:06:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:06:13.428 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T02:06:13.428 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T02:06:13.428 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T02:06:13.684 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=110 2024-09-15T02:06:13.685 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T02:06:13.686 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T02:06:13.686 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T02:06:32.299 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:06:32.299 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:06:32.300 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:06:32.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:06:32.304 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:06:32.304 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:06:32.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.412+0000 7ff9e571d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.412+0000 7ff9e571d640 10 monclient: build_initial_monmap 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.412+0000 7ff9e571d640 10 monclient: monmap: 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:32.413061+0000 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:32.413061+0000 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:32.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.412+0000 7ff9e571d640 1 Processor -- start 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 -- start start 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient: init 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9e0131b20 0x7ff9e009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e0131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e00a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9e009f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00a0040 con 0x7ff9e009f1e0 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e009bff0 con 0x7ff9e0131b20 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00a14a0 con 0x7ff9e00a0280 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9e571d640 10 monclient(hunting): authenticate will time out at 1652.409640s 2024-09-15T02:06:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9e0131b20 0x7ff9e009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9e0131b20 0x7ff9e009eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51192/0 (socket says 172.21.15.17:51192) 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:53772/0 (socket says 172.21.15.17:53772) 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 1 -- 172.21.15.17:0/1434776599 learned_addr learned my addr 172.21.15.17:0/1434776599 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): get_auth_request con 0x7ff9e0131b20 auth_method 0 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9e00a0280 auth_method 0 2024-09-15T02:06:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.413+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 10 monclient(hunting): handle_auth_done global_id 105468 payload 274 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 1 -- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 msgr2=0x7ff9e0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 1 -- 172.21.15.17:0/1434776599 >> v1:172.21.15.67:6789/0 conn(0x7ff9e009f1e0 legacy=0x7ff9e0068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 10 monclient: _finish_hunting 0 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 1 monclient: found mon.noname-a 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9deffd640 1 -- 172.21.15.17:0/1434776599 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0132430 con 0x7ff9e0131b20 2024-09-15T02:06:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.414+0000 7ff9d7fff640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9deffd640 1 -- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e0131b20 msgr2=0x7ff9e009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9deffd640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e0131b20 0x7ff9e009eca0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009910 tx=0x7ff9d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22848111c884bd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 1 -- 172.21.15.17:0/1434776599 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d002aea0 con 0x7ff9e0131b20 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: dump: 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.428 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 1 -- 172.21.15.17:0/1434776599 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d0032440 con 0x7ff9e0131b20 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 1 -- 172.21.15.17:0/1434776599 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0032820 con 0x7ff9e0131b20 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 5 monclient: authenticate success, global_id 105468 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:32.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 4 set_mon_vals no callback set 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9de7fc640 10 monclient: dump: 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:32.430 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 10 monclient: shutdown 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 1 -- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e0131b20 msgr2=0x7ff9e009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.417+0000 7ff9e571d640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e0131b20 0x7ff9e009eca0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009910 tx=0x7ff9d00091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 -- 172.21.15.17:0/1434776599 shutdown_connections 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e00a0280 0x7ff9e0131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 --2- 172.21.15.17:0/1434776599 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e0131b20 0x7ff9e009eca0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 -- 172.21.15.17:0/1434776599 >> 172.21.15.17:0/1434776599 conn(0x7ff9e0071270 msgr2=0x7ff9e00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 -- 172.21.15.17:0/1434776599 shutdown_connections 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 -- 172.21.15.17:0/1434776599 wait complete. 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 10 monclient: build_initial_monmap 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 10 monclient: monmap: 2024-09-15T02:06:32.431 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.418+0000 7ff9e571d640 1 Processor -- start 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 -- start start 2024-09-15T02:06:32.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient: init 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e009f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e00a0280 0x7ff9e012eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e00a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 0x7ff9e0098c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e0131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e01318a0 con 0x7ff9e009f1e0 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e009ac40 con 0x7ff9e00a0280 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00a09b0 con 0x7ff9e0131b20 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9e571d640 10 monclient(hunting): authenticate will time out at 1652.415569s 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51208/0 (socket says 172.21.15.17:51208) 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 -- 172.21.15.17:0/67021157 learned_addr learned my addr 172.21.15.17:0/67021157 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): get_auth_request con 0x7ff9e009f1e0 auth_method 0 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:32.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 0x7ff9e0098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): get_auth_request con 0x7ff9e0131b20 auth_method 0 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9d7fff640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e00a0280 0x7ff9e012eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9e00a0280 auth_method 0 2024-09-15T02:06:32.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient(hunting): handle_auth_done global_id 105471 payload 824 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 -- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 msgr2=0x7ff9e0098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 0x7ff9e0098c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 -- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e00a0280 msgr2=0x7ff9e012eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e00a0280 0x7ff9e012eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient: _finish_hunting 0 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 monclient: found mon.noname-a 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.419+0000 7ff9deffd640 1 -- 172.21.15.17:0/67021157 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8009390 con 0x7ff9e009f1e0 2024-09-15T02:06:32.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.420+0000 7ff9df7fe640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 0x7ff9e0098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9deffd640 1 -- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 msgr2=0x7ff9e01410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9deffd640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009910 tx=0x7ff9d0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d002aea0 con 0x7ff9e009f1e0 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: dump: 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 5 monclient: authenticate success, global_id 105471 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d0000c40 con 0x7ff9e009f1e0 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 10 monclient: _renew_subs 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d0009450 con 0x7ff9e009f1e0 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d003c400 con 0x7ff9e009f1e0 2024-09-15T02:06:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9d6ffd640 10 monclient: dump: 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:32.439 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 10 monclient: _renew_subs 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e009a730 con 0x7ff9e009f1e0 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.422+0000 7ff9ddffb640 4 set_mon_vals no callback set 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff9d003c7b0 con 0x7ff9e009f1e0 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9e571d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9e571d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9e571d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00724a0 con 0x7ff9e009f1e0 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9d6ffd640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 0x7ff9c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff9d006ccc0 con 0x7ff9e009f1e0 2024-09-15T02:06:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.423+0000 7ff9d7fff640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 0x7ff9c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.424+0000 7ff9d7fff640 10 monclient: get_auth_request con 0x7ff9c0038740 auth_method 0 2024-09-15T02:06:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.426+0000 7ff9d7fff640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 0x7ff9c003abf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009360 tx=0x7ff9c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.428+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff9d004ad20 con 0x7ff9e009f1e0 2024-09-15T02:06:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.428+0000 7ff9d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:32.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.428+0000 7ff9d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:32.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.654+0000 7ff9e571d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:32.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9e571d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:32.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9e571d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:32.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9e011c230 con 0x7ff9e009f1e0 2024-09-15T02:06:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9d6ffd640 1 -- 172.21.15.17:0/67021157 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7ff9d0030e40 con 0x7ff9e009f1e0 2024-09-15T02:06:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:32.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.655+0000 7ff9d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:32.657 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 msgr2=0x7ff9c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 0x7ff9c003abf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009360 tx=0x7ff9c8008040 comp rx=0 tx=0).stop 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 10 monclient: shutdown 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 msgr2=0x7ff9e01410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009910 tx=0x7ff9d0032000 comp rx=0 tx=0).stop 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 shutdown_connections 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9c0038740 0x7ff9c003abf0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9e0131b20 0x7ff9e0098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e00a0280 0x7ff9e012eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 --2- 172.21.15.17:0/67021157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9e009f1e0 0x7ff9e01410f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 >> 172.21.15.17:0/67021157 conn(0x7ff9e0071270 msgr2=0x7ff9e00a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 shutdown_connections 2024-09-15T02:06:32.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:32.657+0000 7ff9e571d640 1 -- 172.21.15.17:0/67021157 wait complete. 2024-09-15T02:06:35.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.777+0000 7ffb9aad4640 10 monclient: get_monmap_and_config 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.777+0000 7ffb9aad4640 10 monclient: build_initial_monmap 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.777+0000 7ffb9aad4640 10 monclient: monmap: 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:35.778329+0000 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:35.778329+0000 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.779 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:35.780 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:35.780 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:35.780 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:35.780 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:35.780 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 Processor -- start 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 -- start start 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient: init 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb94131b20 0x7ffb9409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-c con 0x7ffb94131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb9409f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940a0040 con 0x7ffb9409f1e0 2024-09-15T02:06:35.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9409bff0 con 0x7ffb94131b20 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940a14a0 con 0x7ffb940a0280 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.778+0000 7ffb9aad4640 10 monclient(hunting): authenticate will time out at 1655.774980s 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59906/0 (socket says 172.21.15.17:59906) 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 -- 172.21.15.17:0/211500918 learned_addr learned my addr 172.21.15.17:0/211500918 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): get_auth_request con 0x7ffb940a0280 auth_method 0 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:35.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb98870640 1 --2- 172.21.15.17:0/211500918 >> v2:172.21.15.67:3300/0 conn(0x7ffb94131b20 0x7ffb9409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb98870640 10 monclient(hunting): get_auth_request con 0x7ffb94131b20 auth_method 0 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb98870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb98870640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb98870640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient(hunting): handle_auth_done global_id 184235 payload 274 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 -- 172.21.15.17:0/211500918 >> v2:172.21.15.67:3300/0 conn(0x7ffb94131b20 msgr2=0x7ffb9409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 --2- 172.21.15.17:0/211500918 >> v2:172.21.15.67:3300/0 conn(0x7ffb94131b20 0x7ffb9409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 -- 172.21.15.17:0/211500918 >> v1:172.21.15.67:6789/0 conn(0x7ffb9409f1e0 legacy=0x7ffb94068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 monclient: found mon.noname-b 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.779+0000 7ffb93fff640 1 -- 172.21.15.17:0/211500918 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94132430 con 0x7ffb940a0280 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb93fff640 1 -- 172.21.15.17:0/211500918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 msgr2=0x7ffb94131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb93fff640 1 --2- 172.21.15.17:0/211500918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009910 tx=0x7ffb800091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fc3f8f9b70fb191 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 1 -- 172.21.15.17:0/211500918 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8002aea0 con 0x7ffb940a0280 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: dump: 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: _finish_auth 0 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 1 -- 172.21.15.17:0/211500918 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb80032440 con 0x7ffb940a0280 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 5 monclient: authenticate success, global_id 184235 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 1 -- 172.21.15.17:0/211500918 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80032820 con 0x7ffb940a0280 2024-09-15T02:06:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 4 set_mon_vals no callback set 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb937fe640 10 monclient: dump: 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:35.790 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 10 monclient: shutdown 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 1 -- 172.21.15.17:0/211500918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 msgr2=0x7ffb94131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.782+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/211500918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009910 tx=0x7ffb800091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 -- 172.21.15.17:0/211500918 shutdown_connections 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/211500918 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb940a0280 0x7ffb94131ef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/211500918 >> v2:172.21.15.67:3300/0 conn(0x7ffb94131b20 0x7ffb9409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 -- 172.21.15.17:0/211500918 >> 172.21.15.17:0/211500918 conn(0x7ffb94071270 msgr2=0x7ffb940a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 -- 172.21.15.17:0/211500918 shutdown_connections 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 -- 172.21.15.17:0/211500918 wait complete. 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 10 monclient: build_initial_monmap 2024-09-15T02:06:35.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 10 monclient: monmap: 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:35.792 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.783+0000 7ffb9aad4640 1 Processor -- start 2024-09-15T02:06:35.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 -- start start 2024-09-15T02:06:35.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient: init 2024-09-15T02:06:35.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-a con 0x7ffb9409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): picked mon.noname-c con 0x7ffb94131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940a5340 con 0x7ffb9409f1e0 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9411c3b0 con 0x7ffb940a0280 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9411c0d0 con 0x7ffb94131b20 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb9aad4640 10 monclient(hunting): authenticate will time out at 1655.780741s 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59914/0 (socket says 172.21.15.17:59914) 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 1 -- 172.21.15.17:0/1003016844 learned_addr learned my addr 172.21.15.17:0/1003016844 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37876/0 (socket says 172.21.15.17:37876) 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): get_auth_request con 0x7ffb94131b20 auth_method 0 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): get_auth_request con 0x7ffb9409f1e0 auth_method 0 2024-09-15T02:06:35.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb99071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.784+0000 7ffb98870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 10 monclient(hunting): get_auth_request con 0x7ffb940a0280 auth_method 0 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:35.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 10 monclient(hunting): handle_auth_done global_id 184238 payload 824 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 msgr2=0x7ffb9412ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 msgr2=0x7ffb9412ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 msgr2=0x7ffb9412ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb93fff640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 msgr2=0x7ffb94141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 10 monclient: _finish_hunting 0 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 monclient: found mon.noname-c 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:35.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb99071640 1 -- 172.21.15.17:0/1003016844 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80009450 con 0x7ffb94131b20 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.785+0000 7ffb98870640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb99071640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 msgr2=0x7ffb94098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb99071640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00c8c0 tx=0x7ffb8c002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c00aa00 con 0x7ffb94131b20 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: dump: 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:35.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: _finish_auth 0 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 5 monclient: authenticate success, global_id 184238 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 10 monclient: _renew_subs 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c00ab80 con 0x7ffb94131b20 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9411bdf0 con 0x7ffb94131b20 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c012420 con 0x7ffb94131b20 2024-09-15T02:06:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb90ff9640 10 monclient: dump: 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:35.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb927fc640 4 set_mon_vals no callback set 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 10 monclient: _renew_subs 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.787+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb94141db0 con 0x7ffb94131b20 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb9aad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb9aad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffb8c012890 con 0x7ffb94131b20 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb9aad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9411c560 con 0x7ffb94131b20 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.788+0000 7ffb90ff9640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 0x7ffb7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.789+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ffb8c04c990 con 0x7ffb94131b20 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.789+0000 7ffb98870640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 0x7ffb7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.789+0000 7ffb98870640 10 monclient: get_auth_request con 0x7ffb78038740 auth_method 0 2024-09-15T02:06:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.791+0000 7ffb98870640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 0x7ffb7803abf0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ffb88007fa0 tx=0x7ffb88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:35.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.793+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffb9411c560 con 0x7ffb94131b20 2024-09-15T02:06:35.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.793+0000 7ffb90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:35.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:35.793+0000 7ffb90ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:36.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.019+0000 7ffb9aad4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:36.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.019+0000 7ffb9aad4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:36.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.019+0000 7ffb9aad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:36.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.019+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffb9411c560 con 0x7ffb94131b20 2024-09-15T02:06:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.020+0000 7ffb90ff9640 1 -- 172.21.15.17:0/1003016844 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7ffb9411c560 con 0x7ffb94131b20 2024-09-15T02:06:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.020+0000 7ffb90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:36.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.020+0000 7ffb90ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:36.022 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 msgr2=0x7ffb7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 0x7ffb7803abf0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ffb88007fa0 tx=0x7ffb88008040 comp rx=0 tx=0).stop 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 10 monclient: shutdown 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 msgr2=0x7ffb94098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00c8c0 tx=0x7ffb8c002640 comp rx=0 tx=0).stop 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 shutdown_connections 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ffb78038740 0x7ffb7803abf0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:36.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ffb94131b20 0x7ffb94098b40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb940a0280 0x7ffb9412ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 --2- 172.21.15.17:0/1003016844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb9409f1e0 0x7ffb94141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 >> 172.21.15.17:0/1003016844 conn(0x7ffb94071270 msgr2=0x7ffb9409d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 shutdown_connections 2024-09-15T02:06:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:36.022+0000 7ffb9aad4640 1 -- 172.21.15.17:0/1003016844 wait complete. 2024-09-15T02:06:39.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:39.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.141+0000 7f1a32460640 10 monclient: get_monmap_and_config 2024-09-15T02:06:39.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.141+0000 7f1a32460640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.141+0000 7f1a32460640 10 monclient: monmap: 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:39.142468+0000 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:39.142468+0000 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:39.145 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:39.145 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:39.145 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.142+0000 7f1a32460640 1 Processor -- start 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.142+0000 7f1a32460640 1 -- start start 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.142+0000 7f1a32460640 10 monclient: init 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a2c06eb30 0x7f1a2c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c06eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a2c06f440 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c09c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c09e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09ed70 con 0x7f1a2c06f440 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c06ab30 con 0x7f1a2c06eb30 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c09a930 con 0x7f1a2c09e970 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a32460640 10 monclient(hunting): authenticate will time out at 1659.139103s 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c09c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2b7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1a2c06f440 0x7f1a2c09bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43214/0 (socket says 172.21.15.17:43214) 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2b7fe640 1 -- 172.21.15.17:0/3808595008 learned_addr learned my addr 172.21.15.17:0/3808595008 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): get_auth_request con 0x7f1a2c09e970 auth_method 0 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2bfff640 1 --2- 172.21.15.17:0/3808595008 >> v2:172.21.15.67:3300/0 conn(0x7f1a2c06eb30 0x7f1a2c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2bfff640 10 monclient(hunting): get_auth_request con 0x7f1a2c06eb30 auth_method 0 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient(hunting): handle_auth_done global_id 184241 payload 274 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 -- 172.21.15.17:0/3808595008 >> v2:172.21.15.67:3300/0 conn(0x7f1a2c06eb30 msgr2=0x7f1a2c06ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 --2- 172.21.15.17:0/3808595008 >> v2:172.21.15.67:3300/0 conn(0x7f1a2c06eb30 0x7f1a2c06ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 -- 172.21.15.17:0/3808595008 >> v1:172.21.15.17:6789/0 conn(0x7f1a2c06f440 legacy=0x7f1a2c09bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 monclient: found mon.noname-b 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.143+0000 7f1a309fd640 1 -- 172.21.15.17:0/3808595008 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c069f70 con 0x7f1a2c09e970 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a309fd640 1 -- 172.21.15.17:0/3808595008 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 msgr2=0x7f1a2c09c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a309fd640 1 --2- 172.21.15.17:0/3808595008 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c09c490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009e60 tx=0x7f1a24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf13baa6d43a3a76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 1 -- 172.21.15.17:0/3808595008 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007760 con 0x7f1a2c09e970 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: dump: 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.153 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 1 -- 172.21.15.17:0/3808595008 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2402d050 con 0x7f1a2c09e970 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 1 -- 172.21.15.17:0/3808595008 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007af0 con 0x7f1a2c09e970 2024-09-15T02:06:39.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a32460640 5 monclient: authenticate success, global_id 184241 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a32460640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a32460640 4 set_mon_vals no callback set 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.146+0000 7f1a2affd640 10 monclient: dump: 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.155 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 10 monclient: shutdown 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 -- 172.21.15.17:0/3808595008 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 msgr2=0x7f1a2c09c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 --2- 172.21.15.17:0/3808595008 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c09c490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009e60 tx=0x7f1a24009780 comp rx=0 tx=0).stop 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 -- 172.21.15.17:0/3808595008 shutdown_connections 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 --2- 172.21.15.17:0/3808595008 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c09c490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 --2- 172.21.15.17:0/3808595008 >> v2:172.21.15.67:3300/0 conn(0x7f1a2c06eb30 0x7f1a2c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 -- 172.21.15.17:0/3808595008 >> 172.21.15.17:0/3808595008 conn(0x7f1a2c0712f0 msgr2=0x7f1a2c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 -- 172.21.15.17:0/3808595008 shutdown_connections 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 1 -- 172.21.15.17:0/3808595008 wait complete. 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.147+0000 7f1a32460640 10 monclient: monmap: 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:39.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 Processor -- start 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 -- start start 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient: init 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c06eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c06f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c11e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c09e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c09cd60 con 0x7f1a2c06eb30 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c11fda0 con 0x7f1a2c06f440 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c1221f0 con 0x7f1a2c09e970 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.148+0000 7f1a32460640 10 monclient(hunting): authenticate will time out at 1659.144950s 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37882/0 (socket says 172.21.15.17:37882) 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 -- 172.21.15.17:0/3857214185 learned_addr learned my addr 172.21.15.17:0/3857214185 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): get_auth_request con 0x7f1a2c06eb30 auth_method 0 2024-09-15T02:06:39.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c11e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): get_auth_request con 0x7f1a2c09e970 auth_method 0 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f1a2c06f440 auth_method 0 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient(hunting): handle_auth_done global_id 105477 payload 824 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 msgr2=0x7f1a2c11e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c11e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 msgr2=0x7f1a2c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 msgr2=0x7f1a2c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 msgr2=0x7f1a2c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:39.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2b7fe640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 monclient: found mon.noname-a 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a2bfff640 1 -- 172.21.15.17:0/3857214185 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a24009a20 con 0x7f1a2c06eb30 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.149+0000 7f1a309fd640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c11e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a2bfff640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 msgr2=0x7f1a2c140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a2bfff640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d890 tx=0x7f1a1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0098b0 con 0x7f1a2c06eb30 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:39.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: dump: 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c009a10 con 0x7f1a2c06eb30 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a32460640 5 monclient: authenticate success, global_id 105477 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a32460640 10 monclient: _renew_subs 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a32460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c1204c0 con 0x7f1a2c06eb30 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c009dd0 con 0x7f1a2c06eb30 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a13fff640 10 monclient: dump: 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.168 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.152+0000 7f1a32460640 10 monclient: _renew_subs 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.151+0000 7f1a29ffb640 4 set_mon_vals no callback set 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.152+0000 7f1a32460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.152+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c134320 con 0x7f1a2c06eb30 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.152+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1a1c014620 con 0x7f1a2c06eb30 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a32460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a32460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a32460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a13fff640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 0x7f1a1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c0724a0 con 0x7f1a2c06eb30 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1a1c04cd30 con 0x7f1a2c06eb30 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a2b7fe640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 0x7f1a1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.153+0000 7f1a2b7fe640 10 monclient: get_auth_request con 0x7f1a18038740 auth_method 0 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.156+0000 7f1a2b7fe640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 0x7f1a1803abf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006fd0 tx=0x7f1a20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.157+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1a1c025b30 con 0x7f1a2c06eb30 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.157+0000 7f1a13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.157+0000 7f1a13fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:39.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.384+0000 7f1a32460640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:39.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.384+0000 7f1a32460640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:39.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.384+0000 7f1a32460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.384+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a2c134600 con 0x7f1a2c06eb30 2024-09-15T02:06:39.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.385+0000 7f1a13fff640 1 -- 172.21.15.17:0/3857214185 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f1a1c025350 con 0x7f1a2c06eb30 2024-09-15T02:06:39.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.385+0000 7f1a13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:39.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.385+0000 7f1a13fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:39.387 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:39.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 msgr2=0x7f1a1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 0x7f1a1803abf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006fd0 tx=0x7f1a20008040 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 10 monclient: shutdown 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 msgr2=0x7f1a2c140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d890 tx=0x7f1a1c008040 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 shutdown_connections 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1a18038740 0x7f1a1803abf0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1a2c09e970 0x7f1a2c11e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c06f440 0x7f1a2c133de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 --2- 172.21.15.17:0/3857214185 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1a2c06eb30 0x7f1a2c140f90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.386+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 >> 172.21.15.17:0/3857214185 conn(0x7f1a2c0712f0 msgr2=0x7f1a2c09f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.387+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 shutdown_connections 2024-09-15T02:06:39.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.387+0000 7f1a32460640 1 -- 172.21.15.17:0/3857214185 wait complete. 2024-09-15T02:06:39.398 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:06:39.398 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:06:39.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.493+0000 7f9d4b79d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.493+0000 7f9d4b79d640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.493+0000 7f9d4b79d640 10 monclient: monmap: 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:39.494493+0000 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:39.494493+0000 2024-09-15T02:06:39.495 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:39.496 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.494+0000 7f9d4b79d640 1 Processor -- start 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.494+0000 7f9d4b79d640 1 -- start start 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.494+0000 7f9d4b79d640 10 monclient: init 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d4409c1b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d4409a990 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 0x7f9d4406f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4409e9e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d44069aa0 con 0x7f9d4409c1b0 2024-09-15T02:06:39.498 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4409f730 con 0x7f9d4409a990 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4409ada0 con 0x7f9d4409e9e0 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d4b79d640 10 monclient(hunting): authenticate will time out at 1659.491223s 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49539640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9d4409c1b0 0x7f9d4409c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43218/0 (socket says 172.21.15.17:43218) 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49539640 1 -- 172.21.15.17:0/444191817 learned_addr learned my addr 172.21.15.17:0/444191817 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 0x7f9d4406f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request con 0x7f9d4409e9e0 auth_method 0 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 1 -- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 msgr2=0x7f9d4406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:06:39.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 1 -- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 msgr2=0x7f9d4406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:39.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 0x7f9d4406f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:39.500 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.495+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 0x7f9d4406f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d49539640 1 -- 172.21.15.17:0/444191817 >> v1:172.21.15.17:6789/0 conn(0x7f9d4409c1b0 legacy=0x7f9d4409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506428063 0 0) 0x7f9d44069aa0 con 0x7f9d4409c1b0 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient(hunting): my global_id is 105480 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d300034e0 con 0x7f9d4409c1b0 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d48d38640 1 -- 172.21.15.17:0/444191817 >> v1:172.21.15.67:6789/0 conn(0x7f9d4409a990 legacy=0x7f9d4406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1631292783 0 0) 0x7f9d300034e0 con 0x7f9d4409c1b0 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 msgr2=0x7f9d4406f240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 --2- 172.21.15.17:0/444191817 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409e9e0 0x7f9d4406f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 >> v1:172.21.15.67:6789/0 conn(0x7f9d4409a990 legacy=0x7f9d4406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4409af80 con 0x7f9d4409c1b0 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d4b79d640 5 monclient: authenticate success, global_id 105480 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d4b79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f9d40001720 con 0x7f9d4409c1b0 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.497+0000 7f9d3bfff640 10 monclient: dump: 2024-09-15T02:06:39.508 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 >> v1:172.21.15.17:6789/0 conn(0x7f9d4409c1b0 legacy=0x7f9d4409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:06:39.509 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d30003b20 0x7f9d30003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): picked mon.b con 0x7f9d30003b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 0x7f9d30004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): picked mon.a con 0x7f9d30004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d300053b0 0x7f9d3000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): picked mon.c con 0x7f9d300053b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4409ada0 con 0x7f9d30004780 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000c380 con 0x7f9d30003b20 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000c540 con 0x7f9d300053b0 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d3bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:06:39.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d4b79d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 0x7f9d30004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): get_auth_request con 0x7f9d30004780 auth_method 0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d30003b20 0x7f9d30003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d300053b0 0x7f9d3000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): get_auth_request con 0x7f9d30003b20 auth_method 0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request con 0x7f9d300053b0 auth_method 0 2024-09-15T02:06:39.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient(hunting): handle_auth_done global_id 105480 payload 293 2024-09-15T02:06:39.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 -- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d300053b0 msgr2=0x7f9d3000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d300053b0 0x7f9d3000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 -- 172.21.15.17:0/444191817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d30003b20 msgr2=0x7f9d30003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d30003b20 0x7f9d30003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 monclient: found mon.a 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 1 -- 172.21.15.17:0/444191817 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d300034e0 con 0x7f9d30004780 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d48d38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.498+0000 7f9d49d3a640 1 -- 172.21.15.17:0/444191817 reap_dead start 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d48d38640 1 -- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 msgr2=0x7f9d30004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d48d38640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 0x7f9d30004bf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001b00 tx=0x7f9d34007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ca3cc7bd930597 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3402ac50 con 0x7f9d30004780 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: dump: 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.514 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3402adb0 con 0x7f9d30004780 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d4b79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d4b79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 1 -- 172.21.15.17:0/444191817 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34007340 con 0x7f9d30004780 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d4b79d640 4 set_mon_vals no callback set 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.501+0000 7f9d3bfff640 10 monclient: dump: 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 10 monclient: shutdown 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 -- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 msgr2=0x7f9d30004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 0x7f9d30004bf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001b00 tx=0x7f9d34007c40 comp rx=0 tx=0).stop 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 -- 172.21.15.17:0/444191817 shutdown_connections 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/444191817 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d30004780 0x7f9d30004bf0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 -- 172.21.15.17:0/444191817 >> 172.21.15.17:0/444191817 conn(0x7f9d44071300 msgr2=0x7f9d440a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 -- 172.21.15.17:0/444191817 shutdown_connections 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 1 -- 172.21.15.17:0/444191817 wait complete. 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.502+0000 7f9d4b79d640 10 monclient: monmap: 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 Processor -- start 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 -- start start 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient: init 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d30004780 0x7f9d4413de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d30004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 0x7f9d4413d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d44120170 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d44121640 0x7f9d44141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d44121640 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4406f780 con 0x7f9d44120170 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44120b00 con 0x7f9d44121640 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4409cdb0 con 0x7f9d30004780 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d4b79d640 10 monclient(hunting): authenticate will time out at 1659.499858s 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 0x7f9d4413d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d49539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d30004780 0x7f9d4413de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 0x7f9d4413d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37904/0 (socket says 172.21.15.17:37904) 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 1 -- 172.21.15.17:0/526298669 learned_addr learned my addr 172.21.15.17:0/526298669 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 10 monclient(hunting): get_auth_request con 0x7f9d44120170 auth_method 0 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.503+0000 7f9d48d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): get_auth_request con 0x7f9d30004780 auth_method 0 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d44121640 0x7f9d44141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request con 0x7f9d44121640 auth_method 0 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient(hunting): handle_auth_done global_id 105483 payload 824 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d30004780 msgr2=0x7f9d4413de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d30004780 0x7f9d4413de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d44121640 msgr2=0x7f9d44141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d44121640 0x7f9d44141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 monclient: found mon.noname-a 2024-09-15T02:06:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d48d38640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440a2c50 con 0x7f9d44120170 2024-09-15T02:06:39.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.504+0000 7f9d49539640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d30004780 0x7f9d4413de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d48d38640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 msgr2=0x7f9d4413d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d48d38640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 0x7f9d4413d4c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9d3402a740 tx=0x7f9d34030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3403b070 con 0x7f9d44120170 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: dump: 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 5 monclient: authenticate success, global_id 105483 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34005ca0 con 0x7f9d44120170 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 10 monclient: _renew_subs 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340336a0 con 0x7f9d44120170 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d44122560 con 0x7f9d44120170 2024-09-15T02:06:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d397fa640 10 monclient: dump: 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d3b7fe640 4 set_mon_vals no callback set 2024-09-15T02:06:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 10 monclient: _renew_subs 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.506+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4413da00 con 0x7f9d44120170 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d4b79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 0x7f9d44141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44063bb0 con 0x7f9d441418a0 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d34030070 con 0x7f9d44120170 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 0x7f9d44141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient: get_auth_request con 0x7f9d441418a0 auth_method 0 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 10 monclient(hunting): handle_auth_done global_id 105486 payload 824 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d397fa640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d28038790 0x7f9d2803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.507+0000 7f9d49539640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d4413dbf0 con 0x7f9d441418a0 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.508+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9d3406b750 con 0x7f9d44120170 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.508+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d28038790 0x7f9d2803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.508+0000 7f9d49d3a640 10 monclient: get_auth_request con 0x7f9d28038790 auth_method 0 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d49539640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 msgr2=0x7f9d44141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d49539640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 0x7f9d44141c70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d400145b0 con 0x7f9d441418a0 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 10 monclient: dump: 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9d40014710 con 0x7f9d441418a0 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 msgr2=0x7f9d44141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 0x7f9d44141c70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40007a50 comp rx=0 tx=0).stop 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 msgr2=0x7f9d44141c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d397fa640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d441418a0 0x7f9d44141c70 secure :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40007a50 comp rx=0 tx=0).stop 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.510+0000 7f9d49d3a640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d28038790 0x7f9d2803ac40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9d440a3f70 tx=0x7f9d3c009720 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d4b79d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 0x7f9d4406bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d4b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441421b0 con 0x7f9d4406bae0 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 0x7f9d4406bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient: get_auth_request con 0x7f9d4406bae0 auth_method 0 2024-09-15T02:06:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 10 monclient(hunting): handle_auth_done global_id 105489 payload 824 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.516+0000 7f9d49539640 1 -- 172.21.15.17:0/526298669 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d440724a0 con 0x7f9d4406bae0 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.518+0000 7f9d49539640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 msgr2=0x7f9d4406bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.518+0000 7f9d49539640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 0x7f9d4406bee0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40015d40 con 0x7f9d4406bae0 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 10 monclient: dump: 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f9d4002c460 con 0x7f9d4406bae0 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 msgr2=0x7f9d4406bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 0x7f9d4406bee0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40017000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 msgr2=0x7f9d4406bee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d397fa640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d4406bae0 0x7f9d4406bee0 secure :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001900 tx=0x7f9d40017000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.536 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.537 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:39.538 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:39.539 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:39.540 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.541 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:39.542 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.543 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.545 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:39.546 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:39.547 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.549 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:39.550 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:39.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d28038790 msgr2=0x7f9d2803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d28038790 0x7f9d2803ac40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9d440a3f70 tx=0x7f9d3c009720 comp rx=0 tx=0).stop 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 10 monclient: shutdown 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 msgr2=0x7f9d4413d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 --2- 172.21.15.17:0/526298669 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44120170 0x7f9d4413d4c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9d3402a740 tx=0x7f9d34030000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d49d3a640 1 -- 172.21.15.17:0/526298669 reap_dead start 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d49d3a640 1 -- 172.21.15.17:0/526298669 reap_dead start 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 shutdown_connections 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.519+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 >> 172.21.15.17:0/526298669 conn(0x7f9d44071300 msgr2=0x7f9d44098d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.520+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 shutdown_connections 2024-09-15T02:06:39.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.520+0000 7f9d4b79d640 1 -- 172.21.15.17:0/526298669 wait complete. 2024-09-15T02:06:39.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:39.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:06:39.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.648+0000 7f53b8a99640 10 monclient: get_monmap_and_config 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.648+0000 7f53b8a99640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.648+0000 7f53b8a99640 10 monclient: monmap: 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:39.649459+0000 2024-09-15T02:06:39.650 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:39.649459+0000 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:39.651 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 1 Processor -- start 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 1 -- start start 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient: init 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f53b40a5780 0x7f53b409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-a con 0x7f53b40a5780 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53b409ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.649+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53b4069c90 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40a5b50 con 0x7f53b4069c90 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b409c5f0 con 0x7f53b409ccf0 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40a18b0 con 0x7f53b40a5780 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b8a99640 10 monclient(hunting): authenticate will time out at 1659.646047s 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f53b40a5780 0x7f53b409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f53b40a5780 0x7f53b409bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37932/0 (socket says 172.21.15.17:37932) 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 1 -- 172.21.15.17:0/75382674 learned_addr learned my addr 172.21.15.17:0/75382674 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 10 monclient(hunting): get_auth_request con 0x7f53b40a5780 auth_method 0 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b2d9d640 1 --1- 172.21.15.17:0/75382674 >> v1:172.21.15.17:6789/0 conn(0x7f53b4069c90 0x7f53b4139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43234/0 (socket says 172.21.15.17:43234) 2024-09-15T02:06:39.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.650+0000 7f53b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b2d9d640 1 -- 172.21.15.17:0/75382674 >> v1:172.21.15.17:6789/0 conn(0x7f53b4069c90 legacy=0x7f53b4139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient(hunting): handle_auth_done global_id 105492 payload 274 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53abfff640 1 -- 172.21.15.17:0/75382674 >> v1:172.21.15.67:6789/0 conn(0x7f53b409ccf0 legacy=0x7f53b4069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 1 -- 172.21.15.17:0/75382674 >> v1:172.21.15.67:6789/0 conn(0x7f53b409ccf0 legacy=0x7f53b4069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 1 -- 172.21.15.17:0/75382674 >> v1:172.21.15.17:6789/0 conn(0x7f53b4069c90 legacy=0x7f53b4139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 1 monclient: found mon.noname-a 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.652+0000 7f53b259c640 1 -- 172.21.15.17:0/75382674 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b409d210 con 0x7f53b40a5780 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b259c640 1 -- 172.21.15.17:0/75382674 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b40a5780 msgr2=0x7f53b409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b259c640 1 --2- 172.21.15.17:0/75382674 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b40a5780 0x7f53b409bc90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a10 tx=0x7f53a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849a67e4d7ce491c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 1 -- 172.21.15.17:0/75382674 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a402f4b0 con 0x7f53b40a5780 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: dump: 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b8a99640 5 monclient: authenticate success, global_id 105492 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b8a99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 1 -- 172.21.15.17:0/75382674 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a402f610 con 0x7f53b40a5780 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b8a99640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:39.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b8a99640 4 set_mon_vals no callback set 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 1 -- 172.21.15.17:0/75382674 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a402f9b0 con 0x7f53b40a5780 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.655+0000 7f53b1d9b640 10 monclient: dump: 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 10 monclient: shutdown 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 -- 172.21.15.17:0/75382674 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b40a5780 msgr2=0x7f53b409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 --2- 172.21.15.17:0/75382674 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b40a5780 0x7f53b409bc90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a10 tx=0x7f53a40091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 -- 172.21.15.17:0/75382674 shutdown_connections 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 --2- 172.21.15.17:0/75382674 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b40a5780 0x7f53b409bc90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 -- 172.21.15.17:0/75382674 >> 172.21.15.17:0/75382674 conn(0x7f53b4070e10 msgr2=0x7f53b40a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 -- 172.21.15.17:0/75382674 shutdown_connections 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 1 -- 172.21.15.17:0/75382674 wait complete. 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.656+0000 7f53b8a99640 10 monclient: monmap: 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.661 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:39.662 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 Processor -- start 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 -- start start 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient: init 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 0x7f53b413df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b409ccf0 0x7f53b413d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-b con 0x7f53b409ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b40a5780 0x7f53b41414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): picked mon.noname-c con 0x7f53b40a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4122f40 con 0x7f53b4069c90 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4122980 con 0x7f53b409ccf0 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4120460 con 0x7f53b40a5780 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b8a99640 10 monclient(hunting): authenticate will time out at 1659.653764s 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 0x7f53b413df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 0x7f53b413df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37946/0 (socket says 172.21.15.17:37946) 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b40a5780 0x7f53b41414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 1 -- 172.21.15.17:0/305074159 learned_addr learned my addr 172.21.15.17:0/305074159 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): get_auth_request con 0x7f53b4069c90 auth_method 0 2024-09-15T02:06:39.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): get_auth_request con 0x7f53b40a5780 auth_method 0 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53abfff640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b409ccf0 0x7f53b413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.657+0000 7f53b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): get_auth_request con 0x7f53b409ccf0 auth_method 0 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 10 monclient(hunting): handle_auth_done global_id 105498 payload 824 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b40a5780 msgr2=0x7f53b41414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b40a5780 0x7f53b41414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b409ccf0 msgr2=0x7f53b413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b409ccf0 0x7f53b413d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 monclient: found mon.noname-a 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b259c640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4009450 con 0x7f53b4069c90 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.658+0000 7f53b2d9d640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b40a5780 0x7f53b41414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b259c640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 msgr2=0x7f53b413df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b259c640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 0x7f53b413df50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f53a4035040 tx=0x7f53a4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4032a30 con 0x7f53b4069c90 2024-09-15T02:06:39.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: dump: 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.673 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 5 monclient: authenticate success, global_id 105498 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4032bd0 con 0x7f53b4069c90 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 10 monclient: _renew_subs 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b41226a0 con 0x7f53b4069c90 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a403b400 con 0x7f53b4069c90 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53aa7fc640 10 monclient: dump: 2024-09-15T02:06:39.674 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b159a640 4 set_mon_vals no callback set 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 10 monclient: _renew_subs 2024-09-15T02:06:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.660+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b413db40 con 0x7f53b4069c90 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f53a4039040 con 0x7f53b4069c90 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53b8a99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53b8a99640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 0x7f53b4141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4063b20 con 0x7f53b41217d0 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 0x7f53b4141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient: get_auth_request con 0x7f53b41217d0 auth_method 0 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53aa7fc640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5394038790 0x7f539403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f53a406c6d0 con 0x7f53b4069c90 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 10 monclient(hunting): handle_auth_done global_id 184253 payload 824 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53abfff640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b41230b0 con 0x7f53b41217d0 2024-09-15T02:06:39.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.661+0000 7f53b2d9d640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5394038790 0x7f539403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.662+0000 7f53b2d9d640 10 monclient: get_auth_request con 0x7f5394038790 auth_method 0 2024-09-15T02:06:39.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53abfff640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 msgr2=0x7f53b4141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53abfff640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 0x7f53b4141a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a0b70 tx=0x7f539c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c0074f0 con 0x7f53b41217d0 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 10 monclient: dump: 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.679 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53b2d9d640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5394038790 0x7f539403ac40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f53ac002230 tx=0x7f53ac007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f539c007650 con 0x7f53b41217d0 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 msgr2=0x7f53b4141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 0x7f53b4141a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a0b70 tx=0x7f539c009040 comp rx=0 tx=0).stop 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 msgr2=0x7f53b4141a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.664+0000 7f53aa7fc640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b41217d0 0x7f53b4141a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53b8a99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53b8a99640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 0x7f53b4141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53b8a99640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b413dd30 con 0x7f53b413e8d0 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 0x7f53b4141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient: get_auth_request con 0x7f53b413e8d0 auth_method 0 2024-09-15T02:06:39.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 10 monclient(hunting): handle_auth_done global_id 184256 payload 824 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.670+0000 7f53abfff640 1 -- 172.21.15.17:0/305074159 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f53b40724a0 con 0x7f53b413e8d0 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53abfff640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 msgr2=0x7f53b4141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53abfff640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 0x7f53b4141f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a0b70 tx=0x7f539c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c021830 con 0x7f53b413e8d0 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 10 monclient: dump: 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.685 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f539c0072e0 con 0x7f53b413e8d0 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 msgr2=0x7f53b4141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 0x7f53b4141f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a0b70 tx=0x7f539c018000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 msgr2=0x7f53b4141f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53aa7fc640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53b413e8d0 0x7f53b4141f40 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a0b70 tx=0x7f539c018000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:39.686 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.687 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.688 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:39.690 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.691 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.693 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.696 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:39.699 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:39.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5394038790 msgr2=0x7f539403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5394038790 0x7f539403ac40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f53ac002230 tx=0x7f53ac007f90 comp rx=0 tx=0).stop 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 10 monclient: shutdown 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 msgr2=0x7f53b413df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b8a99640 1 --2- 172.21.15.17:0/305074159 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f53b4069c90 0x7f53b413df50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f53a4035040 tx=0x7f53a4032000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.673+0000 7f53b2d9d640 1 -- 172.21.15.17:0/305074159 reap_dead start 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.674+0000 7f53b2d9d640 1 -- 172.21.15.17:0/305074159 reap_dead start 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.674+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 shutdown_connections 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.674+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 >> 172.21.15.17:0/305074159 conn(0x7f53b4070e10 msgr2=0x7f53b4098ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.674+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 shutdown_connections 2024-09-15T02:06:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.674+0000 7f53b8a99640 1 -- 172.21.15.17:0/305074159 wait complete. 2024-09-15T02:06:39.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:39.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:06:39.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:06:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 10 monclient: get_monmap_and_config 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 10 monclient: monmap: 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:39.798064+0000 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:39.798064+0000 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:39.800 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 1 Processor -- start 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 1 -- start start 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.797+0000 7fb0b868b640 10 monclient: init 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0b0130e50 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 0x7fb0b0136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b0132350 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b0137230 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b0131f60 con 0x7fb0b0130e50 2024-09-15T02:06:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b012f8c0 con 0x7fb0b0132350 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b009e130 con 0x7fb0b0137230 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b868b640 10 monclient(hunting): authenticate will time out at 1659.794567s 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59984/0 (socket says 172.21.15.17:59984) 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 -- 172.21.15.17:0/959135719 learned_addr learned my addr 172.21.15.17:0/959135719 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): get_auth_request con 0x7fb0b0137230 auth_method 0 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/959135719 >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 0x7fb0b0136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_done global_id 184259 payload 274 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 -- 172.21.15.17:0/959135719 >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 msgr2=0x7fb0b0136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 --2- 172.21.15.17:0/959135719 >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 0x7fb0b0136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 -- 172.21.15.17:0/959135719 >> v1:172.21.15.67:6789/0 conn(0x7fb0b0130e50 legacy=0x7fb0b0131220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 monclient: found mon.noname-b 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b6c28640 1 -- 172.21.15.17:0/959135719 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b009e410 con 0x7fb0b0137230 2024-09-15T02:06:39.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b5c26640 10 monclient: get_auth_request con 0x7fb0b0132350 auth_method 0 2024-09-15T02:06:39.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.798+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/959135719 >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 0x7fb0b0136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:06:39.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.801+0000 7fb0b6c28640 1 -- 172.21.15.17:0/959135719 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 msgr2=0x7fb0b0139610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.801+0000 7fb0b6c28640 1 --2- 172.21.15.17:0/959135719 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009f20 tx=0x7fb0a8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2aaa5e25876d90 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 1 -- 172.21.15.17:0/959135719 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8007610 con 0x7fb0b0137230 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: dump: 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 5 monclient: authenticate success, global_id 184259 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 1 -- 172.21.15.17:0/959135719 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a802d050 con 0x7fb0b0137230 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 4 set_mon_vals no callback set 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 1 -- 172.21.15.17:0/959135719 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80079a0 con 0x7fb0b0137230 2024-09-15T02:06:39.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b5425640 10 monclient: dump: 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 10 monclient: shutdown 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 -- 172.21.15.17:0/959135719 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 msgr2=0x7fb0b0139610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 --2- 172.21.15.17:0/959135719 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009f20 tx=0x7fb0a8009840 comp rx=0 tx=0).stop 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 -- 172.21.15.17:0/959135719 shutdown_connections 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 --2- 172.21.15.17:0/959135719 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0137230 0x7fb0b0139610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 --2- 172.21.15.17:0/959135719 >> v2:172.21.15.67:3300/0 conn(0x7fb0b0132350 0x7fb0b0136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 -- 172.21.15.17:0/959135719 >> 172.21.15.17:0/959135719 conn(0x7fb0b00a1ee0 msgr2=0x7fb0b00a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 -- 172.21.15.17:0/959135719 shutdown_connections 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.802+0000 7fb0b868b640 1 -- 172.21.15.17:0/959135719 wait complete. 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.803+0000 7fb0b868b640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.803+0000 7fb0b868b640 10 monclient: monmap: 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:39.814 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.803+0000 7fb0b868b640 1 Processor -- start 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.803+0000 7fb0b868b640 1 -- start start 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.803+0000 7fb0b868b640 10 monclient: init 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:06:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b0130e50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b0132350 0x7fb0b00999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b0132350 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0b0137230 0x7fb0b0099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b0137230 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b009d480 con 0x7fb0b0137230 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0136d50 con 0x7fb0b0132350 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b009bcb0 con 0x7fb0b0130e50 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b868b640 10 monclient(hunting): authenticate will time out at 1659.800151s 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:59986/0 (socket says 172.21.15.17:59986) 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 -- 172.21.15.17:0/3065976649 learned_addr learned my addr 172.21.15.17:0/3065976649 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): get_auth_request con 0x7fb0b0130e50 auth_method 0 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0b0137230 0x7fb0b0099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): get_auth_request con 0x7fb0b0137230 auth_method 0 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b0132350 0x7fb0b00999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient(hunting): handle_auth_done global_id 184262 payload 824 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b0132350 msgr2=0x7fb0b00999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b0132350 0x7fb0b00999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0b0137230 msgr2=0x7fb0b0099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb0b0137230 0x7fb0b0099ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 monclient: found mon.noname-c 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b6427640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a8009ae0 con 0x7fb0b0130e50 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b5c26640 10 monclient: get_auth_request con 0x7fb0b0132350 auth_method 0 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.804+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b0132350 0x7fb0b00999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0b6427640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 msgr2=0x7fb0b0136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0b6427640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b5e0 tx=0x7fb0ac0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac00f7a0 con 0x7fb0b0130e50 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 10 monclient: dump: 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.819 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0b868b640 5 monclient: authenticate success, global_id 184262 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac00f920 con 0x7fb0b0130e50 2024-09-15T02:06:39.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.806+0000 7fb0b868b640 10 monclient: _renew_subs 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b0133dd0 con 0x7fb0b0130e50 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac00fce0 con 0x7fb0b0130e50 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 10 monclient: dump: 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.821 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b4c24640 4 set_mon_vals no callback set 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 10 monclient: _renew_subs 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b009c0f0 con 0x7fb0b0130e50 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 0x7fb0b012db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0063870 con 0x7fb0b009a420 2024-09-15T02:06:39.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.807+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb0ac010630 con 0x7fb0b0130e50 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0a67fc640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb094038740 0x7fb09403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb0ac04c2e0 con 0x7fb0b0130e50 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 0x7fb0b012db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient: get_auth_request con 0x7fb0b009a420 auth_method 0 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b6c28640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb094038740 0x7fb09403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b6c28640 10 monclient: get_auth_request con 0x7fb094038740 auth_method 0 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_done global_id 194104 payload 824 2024-09-15T02:06:39.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.808+0000 7fb0b5c26640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0b009a7f0 con 0x7fb0b009a420 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.810+0000 7fb0b5c26640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 msgr2=0x7fb0b012db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.810+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 0x7fb0b012db40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b020 tx=0x7fb0a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0b6c28640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb094038740 0x7fb09403abf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb0b004b6a0 tx=0x7fb0a8036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00089f0 con 0x7fb0b009a420 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:39.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 10 monclient: dump: 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb0a000c040 con 0x7fb0b009a420 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 msgr2=0x7fb0b012db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 0x7fb0b012db40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b020 tx=0x7fb0a0008040 comp rx=0 tx=0).stop 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 msgr2=0x7fb0b012db40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.811+0000 7fb0a67fc640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b009a420 0x7fb0b012db40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b868b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b868b640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 0x7fb0b012e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b868b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00724a0 con 0x7fb0b012e080 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 0x7fb0b012e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient: get_auth_request con 0x7fb0b012e080 auth_method 0 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 10 monclient(hunting): handle_auth_done global_id 194107 payload 824 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.817+0000 7fb0b5c26640 1 -- 172.21.15.17:0/3065976649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb0b00689a0 con 0x7fb0b012e080 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b5c26640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 msgr2=0x7fb0b012e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b5c26640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 0x7fb0b012e480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b020 tx=0x7fb0a001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0020070 con 0x7fb0b012e080 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 10 monclient: dump: 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.829 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fb0a00087e0 con 0x7fb0b012e080 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 msgr2=0x7fb0b012e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 0x7fb0b012e480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b020 tx=0x7fb0a001b000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 msgr2=0x7fb0b012e480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0a67fc640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b012e080 0x7fb0b012e480 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:39.831 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.832 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.833 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:39.834 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.837 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:39.838 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:39.842 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:39.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:39.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:39.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:39.843 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:39.843 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:39.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb094038740 msgr2=0x7fb09403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb094038740 0x7fb09403abf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb0b004b6a0 tx=0x7fb0a8036000 comp rx=0 tx=0).stop 2024-09-15T02:06:39.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 10 monclient: shutdown 2024-09-15T02:06:39.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 msgr2=0x7fb0b0136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b5e0 tx=0x7fb0ac0090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b6c28640 1 -- 172.21.15.17:0/3065976649 reap_dead start 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 shutdown_connections 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 --2- 172.21.15.17:0/3065976649 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb0b0130e50 0x7fb0b0136490 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 >> 172.21.15.17:0/3065976649 conn(0x7fb0b00a1ee0 msgr2=0x7fb0b00a0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 shutdown_connections 2024-09-15T02:06:39.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.820+0000 7fb0b868b640 1 -- 172.21.15.17:0/3065976649 wait complete. 2024-09-15T02:06:39.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:39.845 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:06:39.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:06:39.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.942+0000 7f25140af640 10 monclient: get_monmap_and_config 2024-09-15T02:06:39.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.942+0000 7f25140af640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.942+0000 7f25140af640 10 monclient: monmap: 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:39.943350+0000 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:39.943350+0000 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 Processor -- start 2024-09-15T02:06:39.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 -- start start 2024-09-15T02:06:39.946 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient: init 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f250c09c500 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f250c06f450 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f250c09c8d0 0x7f250c137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): picked mon.noname-c con 0x7f250c09c8d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:39.948 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c09ba20 con 0x7f250c06f450 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c09f6c0 con 0x7f250c09c500 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c09b740 con 0x7f250c09c8d0 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f25140af640 10 monclient(hunting): authenticate will time out at 1659.939903s 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.943+0000 7f251164a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f250c06f450 0x7f250c134760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43238/0 (socket says 172.21.15.17:43238) 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251164a640 1 -- 172.21.15.17:0/3660675076 learned_addr learned my addr 172.21.15.17:0/3660675076 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 1 --2- 172.21.15.17:0/3660675076 >> v2:172.21.15.67:3300/0 conn(0x7f250c09c8d0 0x7f250c137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): get_auth_request con 0x7f250c09c8d0 auth_method 0 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f251264c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f2511e4b640 1 -- 172.21.15.17:0/3660675076 >> v1:172.21.15.67:6789/0 conn(0x7f250c09c500 legacy=0x7f250c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f2511e4b640 1 -- 172.21.15.17:0/3660675076 >> v1:172.21.15.67:6789/0 conn(0x7f250c09c500 legacy=0x7f250c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.944+0000 7f2511e4b640 1 --1- 172.21.15.17:0/3660675076 >> v1:172.21.15.67:6789/0 conn(0x7f250c09c500 0x7f250c06ed40 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251164a640 1 -- 172.21.15.17:0/3660675076 >> v1:172.21.15.17:6789/0 conn(0x7f250c06f450 legacy=0x7f250c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f2510e49640 1 -- 172.21.15.17:0/3660675076 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129445761 0 0) 0x7f250c09ba20 con 0x7f250c06f450 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f2510e49640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f2510e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f2510e49640 10 monclient(hunting): my global_id is 105504 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f2510e49640 1 -- 172.21.15.17:0/3660675076 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f80034e0 con 0x7f250c06f450 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 10 monclient(hunting): handle_auth_done global_id 194110 payload 274 2024-09-15T02:06:39.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 1 -- 172.21.15.17:0/3660675076 >> v1:172.21.15.67:6789/0 conn(0x7f250c09c500 legacy=0x7f250c06ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 1 -- 172.21.15.17:0/3660675076 >> v1:172.21.15.17:6789/0 conn(0x7f250c06f450 legacy=0x7f250c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 1 monclient: found mon.noname-c 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:06:39.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.946+0000 7f251264c640 1 -- 172.21.15.17:0/3660675076 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c09bd00 con 0x7f250c09c8d0 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f251264c640 1 -- 172.21.15.17:0/3660675076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c09c8d0 msgr2=0x7f250c137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f251264c640 1 --2- 172.21.15.17:0/3660675076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c09c8d0 0x7f250c137e90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2504009900 tx=0x7f25040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca42a20d1032deb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 1 -- 172.21.15.17:0/3660675076 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504032440 con 0x7f250c09c8d0 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:39.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: dump: 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 1 -- 172.21.15.17:0/3660675076 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25040325a0 con 0x7f250c09c8d0 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f25140af640 5 monclient: authenticate success, global_id 194110 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f25140af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f25140af640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f25140af640 4 set_mon_vals no callback set 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 1 -- 172.21.15.17:0/3660675076 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504032940 con 0x7f250c09c8d0 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.949+0000 7f2510e49640 10 monclient: dump: 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 10 monclient: shutdown 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 -- 172.21.15.17:0/3660675076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c09c8d0 msgr2=0x7f250c137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 --2- 172.21.15.17:0/3660675076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c09c8d0 0x7f250c137e90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2504009900 tx=0x7f25040090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 -- 172.21.15.17:0/3660675076 shutdown_connections 2024-09-15T02:06:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 --2- 172.21.15.17:0/3660675076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c09c8d0 0x7f250c137e90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 -- 172.21.15.17:0/3660675076 >> 172.21.15.17:0/3660675076 conn(0x7f250c0712f0 msgr2=0x7f250c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 -- 172.21.15.17:0/3660675076 shutdown_connections 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 1 -- 172.21.15.17:0/3660675076 wait complete. 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 10 monclient: build_initial_monmap 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.950+0000 7f25140af640 10 monclient: monmap: 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:39.959 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 Processor -- start 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 -- start start 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient: init 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f250c09c500 0x7f250c068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): picked mon.noname-c con 0x7f250c09c500 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): picked mon.noname-a con 0x7f250c09c8d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 0x7f250c069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): picked mon.noname-b con 0x7f250c0694a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c121dd0 con 0x7f250c09c8d0 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c09cf00 con 0x7f250c0694a0 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c0a2410 con 0x7f250c09c500 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f25140af640 10 monclient(hunting): authenticate will time out at 1659.947720s 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f250c09c500 0x7f250c068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37958/0 (socket says 172.21.15.17:37958) 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 1 -- 172.21.15.17:0/2116327487 learned_addr learned my addr 172.21.15.17:0/2116327487 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): get_auth_request con 0x7f250c09c8d0 auth_method 0 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): get_auth_request con 0x7f250c09c500 auth_method 0 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251164a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f251264c640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 0x7f250c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.951+0000 7f2511e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251264c640 10 monclient(hunting): get_auth_request con 0x7f250c0694a0 auth_method 0 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251264c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251264c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251264c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 10 monclient(hunting): handle_auth_done global_id 105507 payload 824 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 -- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f250c09c500 msgr2=0x7f250c068a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f250c09c500 0x7f250c068a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 -- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 msgr2=0x7f250c069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 0x7f250c069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 10 monclient: _finish_hunting 0 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 monclient: found mon.noname-a 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:39.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251164a640 1 -- 172.21.15.17:0/2116327487 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009390 con 0x7f250c09c8d0 2024-09-15T02:06:39.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.952+0000 7f251264c640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 0x7f250c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:06:39.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f251164a640 1 -- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 msgr2=0x7f250c068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f251164a640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00c7c0 tx=0x7f24fc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00ce90 con 0x7f250c09c8d0 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: dump: 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: _finish_auth 0 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 5 monclient: authenticate success, global_id 105507 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc001d50 con 0x7f250c09c8d0 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc005cb0 con 0x7f250c09c8d0 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 10 monclient: _renew_subs 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c122a90 con 0x7f250c09c8d0 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:39.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2503fff640 4 set_mon_vals no callback set 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f2501ffb640 10 monclient: dump: 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 10 monclient: _renew_subs 2024-09-15T02:06:39.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.954+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c121120 con 0x7f250c09c8d0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f24fc007300 con 0x7f250c09c8d0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f25140af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f2501ffb640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 0x7f24f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f25140af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f25140af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c09ef40 con 0x7f250c09c8d0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.955+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f24fc04bfd0 con 0x7f250c09c8d0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.956+0000 7f2511e4b640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 0x7f24f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.956+0000 7f2511e4b640 10 monclient: get_auth_request con 0x7f24f0038740 auth_method 0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.958+0000 7f2511e4b640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 0x7f24f003abf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2508007fd0 tx=0x7f2508002660 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.960+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f250c09ef40 con 0x7f250c09c8d0 2024-09-15T02:06:39.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.960+0000 7f2501ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:39.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:39.960+0000 7f2501ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:40.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.185+0000 7f25140af640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:06:40.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.185+0000 7f25140af640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:06:40.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.185+0000 7f25140af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:40.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.185+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f250c09ef40 con 0x7f250c09c8d0 2024-09-15T02:06:40.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.185+0000 7f2501ffb640 1 -- 172.21.15.17:0/2116327487 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f250c09ef40 con 0x7f250c09c8d0 2024-09-15T02:06:40.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f2501ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:06:40.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f2501ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=8,logm=55} crc {auth=3606880759,config=3140514009,health=2177675957,kv=1529054990,logm=1109316923}) 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=8,logm=54} crc {auth=542399827,config=3140514009,health=664108966,kv=2154233959,logm=2859643994}) 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=8,logm=55} crc {auth=3606880759,config=3140514009,health=2177675957,kv=1529054990,logm=1109316923}) 2024-09-15T02:06:40.189 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.186+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=8,logm=54} crc {auth=4221300400,config=3140514009,health=261395888,kv=1529054990,logm=752749410}) 2024-09-15T02:06:40.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 msgr2=0x7f24f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:40.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 0x7f24f003abf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2508007fd0 tx=0x7f2508002660 comp rx=0 tx=0).stop 2024-09-15T02:06:40.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 10 monclient: shutdown 2024-09-15T02:06:40.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:40.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927771698}) 2024-09-15T02:06:40.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 msgr2=0x7f250c068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:40.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00c7c0 tx=0x7f24fc007bc0 comp rx=0 tx=0).stop 2024-09-15T02:06:40.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187943792}) 2024-09-15T02:06:40.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 shutdown_connections 2024-09-15T02:06:40.192 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927771698}) 2024-09-15T02:06:40.192 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360631503}) 2024-09-15T02:06:40.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f24f0038740 0x7f24f003abf0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f250c09c8d0 0x7f250c068f60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f250c0694a0 0x7f250c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 --2- 172.21.15.17:0/2116327487 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f250c09c500 0x7f250c068a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 >> 172.21.15.17:0/2116327487 conn(0x7f250c0712f0 msgr2=0x7f250c0a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 shutdown_connections 2024-09-15T02:06:40.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:40.188+0000 7f25140af640 1 -- 172.21.15.17:0/2116327487 wait complete. 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487372421}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4124474828}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487372421}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.189+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159556673}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694366119}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039664523}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694366119}) 2024-09-15T02:06:40.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.191+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=29297910}) 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144766681}) 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618966928}) 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144766681}) 2024-09-15T02:06:40.196 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.192+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043800905}) 2024-09-15T02:06:40.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:06:40.236 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.236 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267295171}) 2024-09-15T02:06:40.236 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141776015}) 2024-09-15T02:06:40.236 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.236 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267295171}) 2024-09-15T02:06:40.237 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.233+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556128047}) 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111881870}) 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603459891}) 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111881870}) 2024-09-15T02:06:40.620 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258170764}) 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834537143}) 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046022476}) 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834537143}) 2024-09-15T02:06:40.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1038359923}) 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463037531}) 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262006655}) 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463037531}) 2024-09-15T02:06:40.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009904755}) 2024-09-15T02:06:40.638 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038923912}) 2024-09-15T02:06:40.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057238317}) 2024-09-15T02:06:40.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038923912}) 2024-09-15T02:06:40.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235182383}) 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436491424}) 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040900314}) 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436491424}) 2024-09-15T02:06:40.894 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.891+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1499161215}) 2024-09-15T02:06:40.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917877103}) 2024-09-15T02:06:40.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750707184}) 2024-09-15T02:06:40.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:40.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917877103}) 2024-09-15T02:06:40.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:40.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1119918515}) 2024-09-15T02:06:41.203 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:06:41.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 10 monclient: get_monmap_and_config 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 10 monclient: monmap: 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:41.310885+0000 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:41.310885+0000 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.314 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 1 Processor -- start 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 1 -- start start 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.309+0000 7f8d62514640 10 monclient: init 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d5c131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-b con 0x7f8d5c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 0x7f8d5c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-a con 0x7f8d5c132430 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0a0040 con 0x7f8d5c131b20 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c09bff0 con 0x7f8d5c132430 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0a14a0 con 0x7f8d5c0a0280 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d62514640 10 monclient(hunting): authenticate will time out at 1661.307448s 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 0x7f8d5c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 0x7f8d5c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37960/0 (socket says 172.21.15.17:37960) 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60000/0 (socket says 172.21.15.17:60000) 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 1 -- 172.21.15.17:0/521942732 learned_addr learned my addr 172.21.15.17:0/521942732 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 10 monclient(hunting): get_auth_request con 0x7f8d5c132430 auth_method 0 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d60ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5bfff640 1 --1- 172.21.15.17:0/521942732 >> v1:172.21.15.17:6789/0 conn(0x7f8d5c131b20 0x7f8d5c09eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43248/0 (socket says 172.21.15.17:43248) 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d5c0a0280 auth_method 0 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_done global_id 184268 payload 274 2024-09-15T02:06:41.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 -- 172.21.15.17:0/521942732 >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 msgr2=0x7f8d5c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:41.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 --2- 172.21.15.17:0/521942732 >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 0x7f8d5c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 -- 172.21.15.17:0/521942732 >> v1:172.21.15.17:6789/0 conn(0x7f8d5c131b20 legacy=0x7f8d5c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:41.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 monclient: found mon.noname-b 2024-09-15T02:06:41.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.310+0000 7f8d5b7fe640 1 -- 172.21.15.17:0/521942732 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c09f3b0 con 0x7f8d5c0a0280 2024-09-15T02:06:41.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5b7fe640 1 -- 172.21.15.17:0/521942732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 msgr2=0x7f8d5c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5b7fe640 1 --2- 172.21.15.17:0/521942732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009850 tx=0x7f8d500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63b95e2b2ec6e7f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 1 -- 172.21.15.17:0/521942732 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5002adf0 con 0x7f8d5c0a0280 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: dump: 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.326 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 1 -- 172.21.15.17:0/521942732 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d50032440 con 0x7f8d5c0a0280 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d62514640 5 monclient: authenticate success, global_id 184268 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d62514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d62514640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 1 -- 172.21.15.17:0/521942732 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500327e0 con 0x7f8d5c0a0280 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d62514640 4 set_mon_vals no callback set 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.313+0000 7f8d5affd640 10 monclient: dump: 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.328 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 10 monclient: shutdown 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 -- 172.21.15.17:0/521942732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 msgr2=0x7f8d5c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 --2- 172.21.15.17:0/521942732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009850 tx=0x7f8d500090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 -- 172.21.15.17:0/521942732 shutdown_connections 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 --2- 172.21.15.17:0/521942732 >> v2:172.21.15.17:3300/0 conn(0x7f8d5c132430 0x7f8d5c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 --2- 172.21.15.17:0/521942732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c0a0280 0x7f8d5c131ef0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 -- 172.21.15.17:0/521942732 >> 172.21.15.17:0/521942732 conn(0x7f8d5c071270 msgr2=0x7f8d5c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 -- 172.21.15.17:0/521942732 shutdown_connections 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 1 -- 172.21.15.17:0/521942732 wait complete. 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.314+0000 7f8d62514640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 10 monclient: monmap: 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 1 Processor -- start 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 1 -- start start 2024-09-15T02:06:41.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 10 monclient: init 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.315+0000 7f8d62514640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d5c0a0280 0x7f8d5c132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-b con 0x7f8d5c0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-a con 0x7f8d5c131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): picked mon.noname-c con 0x7f8d5c132430 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c137f90 con 0x7f8d5c131b20 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c134760 con 0x7f8d5c0a0280 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c138f60 con 0x7f8d5c132430 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d62514640 10 monclient(hunting): authenticate will time out at 1661.313078s 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37968/0 (socket says 172.21.15.17:37968) 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 1 -- 172.21.15.17:0/863694888 learned_addr learned my addr 172.21.15.17:0/863694888 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60004/0 (socket says 172.21.15.17:60004) 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d5c131b20 auth_method 0 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): get_auth_request con 0x7f8d5c132430 auth_method 0 2024-09-15T02:06:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d5c0a0280 0x7f8d5c132900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): get_auth_request con 0x7f8d5c0a0280 auth_method 0 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient(hunting): handle_auth_done global_id 184271 payload 824 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 -- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d5c0a0280 msgr2=0x7f8d5c132900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d5c0a0280 0x7f8d5c132900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 -- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 msgr2=0x7f8d5c132e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 monclient: found mon.noname-c 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d60ab1640 1 -- 172.21.15.17:0/863694888 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50009390 con 0x7f8d5c132430 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.316+0000 7f8d5b7fe640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.318+0000 7f8d60ab1640 1 -- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 msgr2=0x7f8d5c133380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.318+0000 7f8d60ab1640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d54007c00 tx=0x7f8d54009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.318+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400dcc0 con 0x7f8d5c132430 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: dump: 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.335 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5400f040 con 0x7f8d5c132430 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 5 monclient: authenticate success, global_id 184271 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 10 monclient: _renew_subs 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c134270 con 0x7f8d5c132430 2024-09-15T02:06:41.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54015650 con 0x7f8d5c132430 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 10 monclient: dump: 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.337 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 10 monclient: _renew_subs 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d5a7fc640 4 set_mon_vals no callback set 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c134930 con 0x7f8d5c132430 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.319+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d54003020 con 0x7f8d5c132430 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d62514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d62514640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d62514640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c0724a0 con 0x7f8d5c132430 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d43fff640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 0x7f8d4803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8d5404c440 con 0x7f8d5c132430 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d5bfff640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 0x7f8d4803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.320+0000 7f8d5bfff640 10 monclient: get_auth_request con 0x7f8d48038740 auth_method 0 2024-09-15T02:06:41.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.323+0000 7f8d5bfff640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 0x7f8d4803abf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c007310 tx=0x7f8d4c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.324+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d54015a40 con 0x7f8d5c132430 2024-09-15T02:06:41.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.324+0000 7f8d43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.324+0000 7f8d43fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:41.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.522 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436571817}) 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736594951}) 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436571817}) 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.520+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100720012}) 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.523 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286812350}) 2024-09-15T02:06:41.524 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2175342289}) 2024-09-15T02:06:41.524 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.524 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286812350}) 2024-09-15T02:06:41.524 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.521+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090711124}) 2024-09-15T02:06:41.524 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738406687}) 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305180723}) 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738406687}) 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.522+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98434584}) 2024-09-15T02:06:41.525 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220865923}) 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833171079}) 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220865923}) 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.523+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=779000991}) 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.526 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608449760}) 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349476888}) 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608449760}) 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.524+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689717434}) 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.527 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198231819}) 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879083521}) 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198231819}) 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.525+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785543209}) 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.528 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799812261}) 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302012770}) 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799812261}) 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.526+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558752406}) 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.529 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572921495}) 2024-09-15T02:06:41.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38289662}) 2024-09-15T02:06:41.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572921495}) 2024-09-15T02:06:41.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.527+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582525735}) 2024-09-15T02:06:41.530 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224305709}) 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1995820165}) 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224305709}) 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.528+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397083174}) 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.531 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449933924}) 2024-09-15T02:06:41.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3253544373}) 2024-09-15T02:06:41.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449933924}) 2024-09-15T02:06:41.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.529+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2659190003}) 2024-09-15T02:06:41.532 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302252706}) 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710643034}) 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302252706}) 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.530+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538408226}) 2024-09-15T02:06:41.533 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022333048}) 2024-09-15T02:06:41.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215680605}) 2024-09-15T02:06:41.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022333048}) 2024-09-15T02:06:41.534 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.531+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696062106}) 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810710751}) 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915447820}) 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810710751}) 2024-09-15T02:06:41.535 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.532+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757347067}) 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528305002}) 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3050183692}) 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528305002}) 2024-09-15T02:06:41.536 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.533+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3534699631}) 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148318415}) 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685657513}) 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148318415}) 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1695535216}) 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.537 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076390964}) 2024-09-15T02:06:41.538 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1221370524}) 2024-09-15T02:06:41.538 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.538 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076390964}) 2024-09-15T02:06:41.538 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.534+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833287654}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=956381449}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432052752}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=956381449}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.535+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639234285}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195081861}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248657869}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195081861}) 2024-09-15T02:06:41.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.536+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374601866}) 2024-09-15T02:06:41.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168278310}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654532678}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168278310}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.537+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126211283}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192360834}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578479707}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192360834}) 2024-09-15T02:06:41.541 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.538+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681202071}) 2024-09-15T02:06:41.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.559+0000 7f8d62514640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:41.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.559+0000 7f8d62514640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:41.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.559+0000 7f8d62514640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.559+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d5c0980e0 con 0x7f8d5c132430 2024-09-15T02:06:41.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.560+0000 7f8d43fff640 1 -- 172.21.15.17:0/863694888 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f8d54019080 con 0x7f8d5c132430 2024-09-15T02:06:41.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.560+0000 7f8d43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:41.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.560+0000 7f8d43fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:41.563 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 msgr2=0x7f8d4803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 0x7f8d4803abf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c007310 tx=0x7f8d4c007e40 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 10 monclient: shutdown 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 msgr2=0x7f8d5c133380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d54007c00 tx=0x7f8d54009250 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 shutdown_connections 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d48038740 0x7f8d4803abf0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d5c131b20 0x7f8d5c132e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d5c132430 0x7f8d5c133380 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 --2- 172.21.15.17:0/863694888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d5c0a0280 0x7f8d5c132900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 >> 172.21.15.17:0/863694888 conn(0x7f8d5c071270 msgr2=0x7f8d5c131410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 shutdown_connections 2024-09-15T02:06:41.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.562+0000 7f8d62514640 1 -- 172.21.15.17:0/863694888 wait complete. 2024-09-15T02:06:41.576 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:06:41.576 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:06:41.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:06:41.576 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3730344082}) 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1086085895}) 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3730344082}) 2024-09-15T02:06:41.639 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.636+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=458367434}) 2024-09-15T02:06:41.669 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.669 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139142750}) 2024-09-15T02:06:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2553015037}) 2024-09-15T02:06:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139142750}) 2024-09-15T02:06:41.670 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.667+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382554324}) 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=119406117}) 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836367988}) 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=119406117}) 2024-09-15T02:06:41.671 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.668+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810552346}) 2024-09-15T02:06:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605479999}) 2024-09-15T02:06:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480011208}) 2024-09-15T02:06:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.672 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605479999}) 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.669+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865517107}) 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243032383}) 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984059825}) 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243032383}) 2024-09-15T02:06:41.673 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2760005762}) 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649996249}) 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420298118}) 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649996249}) 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.670+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835384884}) 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.674 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916933175}) 2024-09-15T02:06:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725567421}) 2024-09-15T02:06:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916933175}) 2024-09-15T02:06:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.671+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2539576130}) 2024-09-15T02:06:41.675 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213809977}) 2024-09-15T02:06:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776578152}) 2024-09-15T02:06:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213809977}) 2024-09-15T02:06:41.676 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920528491}) 2024-09-15T02:06:41.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f4acc4af640 10 monclient: get_monmap_and_config 2024-09-15T02:06:41.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f4acc4af640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.672+0000 7f4acc4af640 10 monclient: monmap: 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:41.674450+0000 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:41.674450+0000 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:41.677 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.678 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.678 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310037060}) 2024-09-15T02:06:41.678 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899054252}) 2024-09-15T02:06:41.678 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f4acc4af640 1 Processor -- start 2024-09-15T02:06:41.679 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310037060}) 2024-09-15T02:06:41.679 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74265050}) 2024-09-15T02:06:41.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f4acc4af640 1 -- start start 2024-09-15T02:06:41.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.673+0000 7f4acc4af640 10 monclient: init 2024-09-15T02:06:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691710923}) 2024-09-15T02:06:41.680 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493183047}) 2024-09-15T02:06:41.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:06:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ac4131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:41.681 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691710923}) 2024-09-15T02:06:41.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ac40a0280 0x7f4ac4131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.682 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469181455}) 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac40a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ac4132430 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac40a0040 con 0x7f4ac4131b20 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac409bff0 con 0x7f4ac4132430 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac40a14a0 con 0x7f4ac40a0280 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4acc4af640 10 monclient(hunting): authenticate will time out at 1661.671633s 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4aca24b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f4ac4131b20 0x7f4ac409eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43260/0 (socket says 172.21.15.17:43260) 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4aca24b640 1 -- 172.21.15.17:0/1471313716 learned_addr learned my addr 172.21.15.17:0/1471313716 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/1471313716 >> v2:172.21.15.67:3300/0 conn(0x7f4ac40a0280 0x7f4ac4131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4ac9a4a640 10 monclient(hunting): get_auth_request con 0x7f4ac40a0280 auth_method 0 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4ac9a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.684 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.674+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768171878}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369627503}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768171878}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.675+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642756974}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912432013}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301113565}) 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.686 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912432013}) 2024-09-15T02:06:41.687 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=565140247}) 2024-09-15T02:06:41.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.676+0000 7f4aca24b640 1 -- 172.21.15.17:0/1471313716 >> v1:172.21.15.17:6789/0 conn(0x7f4ac4131b20 legacy=0x7f4ac409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4acaa4c640 1 -- 172.21.15.17:0/1471313716 >> v1:172.21.15.67:6789/0 conn(0x7f4ac4132430 legacy=0x7f4ac4138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784467558 0 0) 0x7f4ac40a0040 con 0x7f4ac4131b20 2024-09-15T02:06:41.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): my global_id is 105516 2024-09-15T02:06:41.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab00034e0 con 0x7f4ac4131b20 2024-09-15T02:06:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824345577}) 2024-09-15T02:06:41.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752111597}) 2024-09-15T02:06:41.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824345577}) 2024-09-15T02:06:41.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.690 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842459466}) 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117703723 0 0) 0x7f4ac409bff0 con 0x7f4ac4132430 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 10 monclient(hunting): my global_id is 194122 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac40a0040 con 0x7f4ac4132430 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_done global_id 194119 payload 274 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/1471313716 >> v1:172.21.15.67:6789/0 conn(0x7f4ac4132430 legacy=0x7f4ac4138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/1471313716 >> v1:172.21.15.17:6789/0 conn(0x7f4ac4131b20 legacy=0x7f4ac409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 1 monclient: found mon.noname-c 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:06:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.677+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/1471313716 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac409f3b0 con 0x7f4ac40a0280 2024-09-15T02:06:41.692 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154792285}) 2024-09-15T02:06:41.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770484584}) 2024-09-15T02:06:41.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154792285}) 2024-09-15T02:06:41.693 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.678+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650145391}) 2024-09-15T02:06:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/1471313716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac40a0280 msgr2=0x7f4ac4131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/1471313716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac40a0280 0x7f4ac4131ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009950 tx=0x7f4ab40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beaa9abf6a838a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab402ae10 con 0x7f4ac40a0280 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 10 monclient: dump: 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.699 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4ac9249640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4acc4af640 5 monclient: authenticate success, global_id 194119 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.680+0000 7f4acc4af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab402f4b0 con 0x7f4ac40a0280 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 4 set_mon_vals no callback set 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 1 -- 172.21.15.17:0/1471313716 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab402f830 con 0x7f4ac40a0280 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4ac9249640 10 monclient: dump: 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 10 monclient: shutdown 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 -- 172.21.15.17:0/1471313716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac40a0280 msgr2=0x7f4ac4131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 --2- 172.21.15.17:0/1471313716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac40a0280 0x7f4ac4131ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009950 tx=0x7f4ab40090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 -- 172.21.15.17:0/1471313716 shutdown_connections 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 --2- 172.21.15.17:0/1471313716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac40a0280 0x7f4ac4131ef0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 -- 172.21.15.17:0/1471313716 >> 172.21.15.17:0/1471313716 conn(0x7f4ac4071270 msgr2=0x7f4ac40a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 -- 172.21.15.17:0/1471313716 shutdown_connections 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.681+0000 7f4acc4af640 1 -- 172.21.15.17:0/1471313716 wait complete. 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient: monmap: 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 Processor -- start 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 -- start start 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient: init 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac40a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ac4131b20 0x7f4ac41351d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac4131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac4132430 0x7f4ac4132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac4132430 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac41358a0 con 0x7f4ac40a0280 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4133d70 con 0x7f4ac4132430 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4133ee0 con 0x7f4ac4131b20 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.682+0000 7f4acc4af640 10 monclient(hunting): authenticate will time out at 1661.679970s 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37996/0 (socket says 172.21.15.17:37996) 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 -- 172.21.15.17:0/4068450044 learned_addr learned my addr 172.21.15.17:0/4068450044 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): get_auth_request con 0x7f4ac40a0280 auth_method 0 2024-09-15T02:06:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient(hunting): handle_auth_done global_id 105519 payload 824 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4acaa4c640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac4132430 0x7f4ac4132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ac4131b20 msgr2=0x7f4ac41351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4ac4131b20 0x7f4ac41351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac4132430 msgr2=0x7f4ac4132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ac4132430 0x7f4ac4132970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 monclient: found mon.noname-a 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.683+0000 7f4aca24b640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4009390 con 0x7f4ac40a0280 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aca24b640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 msgr2=0x7f4ac4138b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aca24b640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c790 tx=0x7f4ac0007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac000ce60 con 0x7f4ac40a0280 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: dump: 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 5 monclient: authenticate success, global_id 105519 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0005aa0 con 0x7f4ac40a0280 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 10 monclient: _renew_subs 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac41349a0 con 0x7f4ac40a0280 2024-09-15T02:06:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0005e60 con 0x7f4ac40a0280 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4aba7fc640 10 monclient: dump: 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4ac8a48640 4 set_mon_vals no callback set 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 10 monclient: _renew_subs 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.685+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac4136560 con 0x7f4ac40a0280 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4aca24b640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c790 tx=0x7f4ac0007c00 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f4ac0007300 mgrmap(e 5) v1 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4ac0007300 con 0x7f4ac40a0280 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4acc4af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4acc4af640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 0x7f4ac4133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4063a40 con 0x7f4ac4132eb0 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 0x7f4ac4133280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4aba7fc640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4aa8038740 0x7f4aa803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 10 monclient: get_auth_request con 0x7f4ac4132eb0 auth_method 0 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.686+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_done global_id 105522 payload 824 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ac41337c0 con 0x7f4ac4132eb0 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4acaa4c640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4aa8038740 0x7f4aa803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f4acaa4c640 10 monclient: get_auth_request con 0x7f4aa8038740 auth_method 0 2024-09-15T02:06:41.712 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.712 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283706195}) 2024-09-15T02:06:41.712 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815039793}) 2024-09-15T02:06:41.712 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.712 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283706195}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.687+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670615300}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724995197}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405118761}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724995197}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.688+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492619696}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444366473}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3251537805}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444366473}) 2024-09-15T02:06:41.713 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532329966}) 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 msgr2=0x7f4ac4133280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 0x7f4ac4133280 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4ac409f710 tx=0x7f4ab4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.689+0000 7f4acaa4c640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4aa8038740 0x7f4aa803abf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4abc001920 tx=0x7f4abc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab403e660 con 0x7f4ac4132eb0 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 10 monclient: dump: 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.715 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921211100}) 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418524964}) 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921211100}) 2024-09-15T02:06:41.716 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063852973}) 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4ab4032440 con 0x7f4ac4132eb0 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 msgr2=0x7f4ac4133280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 0x7f4ac4133280 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4ac409f710 tx=0x7f4ab4032000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 msgr2=0x7f4ac4133280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f4aba7fc640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4132eb0 0x7f4ac4133280 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4ac409f710 tx=0x7f4ab4032000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886760753}) 2024-09-15T02:06:41.720 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.690+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1896204665}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886760753}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264556276}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358787238}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=658236477}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358787238}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.691+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702223701}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173442605}) 2024-09-15T02:06:41.721 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254303073}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173442605}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.692+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1302090201}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244759903}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153365547}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244759903}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781232413}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297353678}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.693+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1559212712}) 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.722 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297353678}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40025557}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054733064}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591016593}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054733064}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.694+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444414527}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898331671}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032706121}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898331671}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.695+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722631005}) 2024-09-15T02:06:41.723 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922544595}) 2024-09-15T02:06:41.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126195646}) 2024-09-15T02:06:41.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922544595}) 2024-09-15T02:06:41.724 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3292483888}) 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4acc4af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4acc4af640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 0x7f4ac4096e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4acc4af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4135a10 con 0x7f4ac4096a10 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 0x7f4ac4096e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient: get_auth_request con 0x7f4ac4096a10 auth_method 0 2024-09-15T02:06:41.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 10 monclient(hunting): handle_auth_done global_id 105525 payload 824 2024-09-15T02:06:41.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.696+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/4068450044 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4ac40724a0 con 0x7f4ac4096a10 2024-09-15T02:06:41.727 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156657191}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39003444}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156657191}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828374672}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756957637}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706049271}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756957637}) 2024-09-15T02:06:41.728 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.697+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959251666}) 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372368530}) 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1249189266}) 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372368530}) 2024-09-15T02:06:41.729 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.698+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825527092}) 2024-09-15T02:06:41.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4ac9a4a640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 msgr2=0x7f4ac4096e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4ac9a4a640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 0x7f4ac4096e10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009950 tx=0x7f4ab403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab403ee50 con 0x7f4ac4096a10 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 10 monclient: dump: 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.731 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.731 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070200699}) 2024-09-15T02:06:41.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934628892}) 2024-09-15T02:06:41.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070200699}) 2024-09-15T02:06:41.732 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3262841027}) 2024-09-15T02:06:41.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f4ab403b3a0 con 0x7f4ac4096a10 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 msgr2=0x7f4ac4096e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 0x7f4ac4096e10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009950 tx=0x7f4ab403b000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 msgr2=0x7f4ac4096e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.699+0000 7f4aba7fc640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac4096a10 0x7f4ac4096e10 secure :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009950 tx=0x7f4ab403b000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:41.733 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.734 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.735 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:41.736 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:41.737 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:41.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:41.739 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:41.740 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.742 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.743 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.744 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.745 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:41.746 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:41.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4aa8038740 msgr2=0x7f4aa803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4aa8038740 0x7f4aa803abf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4abc001920 tx=0x7f4abc009040 comp rx=0 tx=0).stop 2024-09-15T02:06:41.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 10 monclient: shutdown 2024-09-15T02:06:41.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 msgr2=0x7f4ac4138b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c790 tx=0x7f4ac0007c00 comp rx=0 tx=0).stop 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acaa4c640 1 -- 172.21.15.17:0/4068450044 reap_dead start 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4aba7fc640 1 -- 172.21.15.17:0/4068450044 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f4ac004c220 con 0x7f4ac40a0280 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 shutdown_connections 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 --2- 172.21.15.17:0/4068450044 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4ac40a0280 0x7f4ac4138b30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 >> 172.21.15.17:0/4068450044 conn(0x7f4ac4071270 msgr2=0x7f4ac4130d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 shutdown_connections 2024-09-15T02:06:41.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f4acc4af640 1 -- 172.21.15.17:0/4068450044 wait complete. 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369755873}) 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=532996135}) 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369755873}) 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.700+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099311179}) 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735468336}) 2024-09-15T02:06:41.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239512351}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735468336}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.701+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3013504162}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608517109}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393947466}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608517109}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.702+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042297922}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342230868}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012351482}) 2024-09-15T02:06:41.756 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342230868}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.703+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1077306166}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153039024}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083542261}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153039024}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.704+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909556936}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541777039}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000471146}) 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.757 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541777039}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.705+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1740475913}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407718460}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820929692}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407718460}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.706+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845405911}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466785937}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640435429}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466785937}) 2024-09-15T02:06:41.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.708+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988908394}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584612308}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274345452}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584612308}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.709+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=171344292}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937410131}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1837779152}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937410131}) 2024-09-15T02:06:41.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.710+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184830746}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974321080}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004292950}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974321080}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.711+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531191476}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034395082}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797609139}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034395082}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.712+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=839020522}) 2024-09-15T02:06:41.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427179981}) 2024-09-15T02:06:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943670671}) 2024-09-15T02:06:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427179981}) 2024-09-15T02:06:41.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.713+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019467797}) 2024-09-15T02:06:41.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:41.762 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:06:41.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698712485}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416966511}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698712485}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.714+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781251843}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669076841}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291793371}) 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.765 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669076841}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909929087}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976194485}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373965276}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976194485}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.715+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=259456308}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=1} crc {logm=676405005,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=4178741443,mon_config_key=551709721}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3} crc {logm=1798770486,mdsmap=1531151339,mgr=3946839590,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2507939938,mon_config_key=2516852880}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=1} crc {logm=676405005,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=4178741443,mon_config_key=551709721}) 2024-09-15T02:06:41.766 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.716+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=73,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=1} crc {logm=908055595,mdsmap=3585761405,mgr=4173831893,mgr_metadata=2029235354,mgrstat=3145770102,mon_config_key=1583035879,monmap=662845760}) 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mon_config_key=3445516029,monmap=3178194974,osd_metadata=3003250900,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3134355883}) 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mon_config_key=4169721880,monmap=3178194974,osd_metadata=611002287,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=1213237024}) 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mon_config_key=3445516029,monmap=3178194974,osd_metadata=3003250900,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3134355883}) 2024-09-15T02:06:41.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:06:41.722+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mon_config_key=199500175,monmap=2689118867,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=646062518}) 2024-09-15T02:06:41.863 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 10 monclient: monmap: 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:41.863140+0000 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:41.863140+0000 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:41.864 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:41.865 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:41.865 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 1 Processor -- start 2024-09-15T02:06:41.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 1 -- start start 2024-09-15T02:06:41.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.861+0000 7fbbac39d640 10 monclient: init 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbba40708a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 0x7fbba406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-c con 0x7fbba4097870 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 0x7fbba41374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-b con 0x7fbba4135110 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba406d7d0 con 0x7fbba40708a0 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba4096b40 con 0x7fbba4097870 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4097c40 con 0x7fbba4135110 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbac39d640 10 monclient(hunting): authenticate will time out at 1661.859712s 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa139640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbba40708a0 0x7fbba406a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:43276/0 (socket says 172.21.15.17:43276) 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa139640 1 -- 172.21.15.17:0/3666589374 learned_addr learned my addr 172.21.15.17:0/3666589374 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 1 --2- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 0x7fbba41374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): get_auth_request con 0x7fbba4135110 auth_method 0 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.862+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_done global_id 184274 payload 274 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbba9938640 1 --2- 172.21.15.17:0/3666589374 >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 0x7fbba406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 1 -- 172.21.15.17:0/3666589374 >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 msgr2=0x7fbba406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 1 --2- 172.21.15.17:0/3666589374 >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 0x7fbba406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 1 -- 172.21.15.17:0/3666589374 >> v1:172.21.15.17:6789/0 conn(0x7fbba40708a0 legacy=0x7fbba406a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 1 monclient: found mon.noname-b 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbbaa93a640 1 -- 172.21.15.17:0/3666589374 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba406e5a0 con 0x7fbba4135110 2024-09-15T02:06:41.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbba9938640 10 monclient: get_auth_request con 0x7fbba4097870 auth_method 0 2024-09-15T02:06:41.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.863+0000 7fbba9938640 1 --2- 172.21.15.17:0/3666589374 >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 0x7fbba406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.865+0000 7fbbaa93a640 1 -- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 msgr2=0x7fbba41374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.865+0000 7fbbaa93a640 1 --2- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 0x7fbba41374f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009e60 tx=0x7fbb9c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e77763db809fef83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 1 -- 172.21.15.17:0/3666589374 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c007840 con 0x7fbba4135110 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: dump: 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 5 monclient: authenticate success, global_id 184274 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 1 -- 172.21.15.17:0/3666589374 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9c02d050 con 0x7fbba4135110 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 4 set_mon_vals no callback set 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 1 -- 172.21.15.17:0/3666589374 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c007bd0 con 0x7fbba4135110 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbba9137640 10 monclient: dump: 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.880 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 10 monclient: shutdown 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 -- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 msgr2=0x7fbba41374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 --2- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 0x7fbba41374f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009e60 tx=0x7fbb9c009780 comp rx=0 tx=0).stop 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 -- 172.21.15.17:0/3666589374 shutdown_connections 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 --2- 172.21.15.17:0/3666589374 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4135110 0x7fbba41374f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 --2- 172.21.15.17:0/3666589374 >> v2:172.21.15.67:3300/0 conn(0x7fbba4097870 0x7fbba406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 -- 172.21.15.17:0/3666589374 >> 172.21.15.17:0/3666589374 conn(0x7fbba4067ae0 msgr2=0x7fbba40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 -- 172.21.15.17:0/3666589374 shutdown_connections 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.866+0000 7fbbac39d640 1 -- 172.21.15.17:0/3666589374 wait complete. 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient: build_initial_monmap 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient: monmap: 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:41.881 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 1 Processor -- start 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 1 -- start start 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient: init 2024-09-15T02:06:41.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-c con 0x7fbba40708a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbba4097870 0x7fbba412f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-b con 0x7fbba4097870 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.867+0000 7fbbac39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbba4135110 0x7fbba413ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): picked mon.noname-a con 0x7fbba4135110 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba406e8d0 con 0x7fbba4135110 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba409c8e0 con 0x7fbba4097870 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba411a7b0 con 0x7fbba40708a0 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbac39d640 10 monclient(hunting): authenticate will time out at 1661.865042s 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbba4135110 0x7fbba413ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60038/0 (socket says 172.21.15.17:60038) 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 -- 172.21.15.17:0/2214784123 learned_addr learned my addr 172.21.15.17:0/2214784123 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): get_auth_request con 0x7fbba40708a0 auth_method 0 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): get_auth_request con 0x7fbba4135110 auth_method 0 2024-09-15T02:06:41.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbba9938640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbba4097870 0x7fbba412f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbba9938640 10 monclient(hunting): get_auth_request con 0x7fbba4097870 auth_method 0 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbba9938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbba9938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbba9938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient(hunting): handle_auth_done global_id 184277 payload 824 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbba4097870 msgr2=0x7fbba412f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbba4097870 0x7fbba412f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbba4135110 msgr2=0x7fbba413ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbba4135110 0x7fbba413ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient: _finish_hunting 0 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 monclient: found mon.noname-c 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.868+0000 7fbbaa139640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c009a20 con 0x7fbba40708a0 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbaa139640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 msgr2=0x7fbba4118910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbaa139640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c5e0 tx=0x7fbba0004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0005cb0 con 0x7fbba40708a0 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:41.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: dump: 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:41.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba0005e30 con 0x7fbba40708a0 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 5 monclient: authenticate success, global_id 184277 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 10 monclient: _renew_subs 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba409bcf0 con 0x7fbba40708a0 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0007340 con 0x7fbba40708a0 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbb9a7fc640 10 monclient: dump: 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbba8936640 4 set_mon_vals no callback set 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 10 monclient: _renew_subs 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.870+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba412fd60 con 0x7fbba40708a0 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbbac39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbbac39d640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 0x7fbba413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.890 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4063b00 con 0x7fbba4119820 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbba001b400 con 0x7fbba40708a0 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 0x7fbba413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient: get_auth_request con 0x7fbba4119820 auth_method 0 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.871+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbb9a7fc640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb88038740 0x7fbb8803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbba9938640 10 monclient(hunting): handle_auth_done global_id 184280 payload 824 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbba004c1c0 con 0x7fbba40708a0 2024-09-15T02:06:41.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbba9938640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbba40724a0 con 0x7fbba4119820 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbbaa93a640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb88038740 0x7fbb8803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.872+0000 7fbbaa93a640 10 monclient: get_auth_request con 0x7fbb88038740 auth_method 0 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbba9938640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 msgr2=0x7fbba413efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbba9938640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 0x7fbba413efd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb94008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94014980 con 0x7fbba4119820 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 10 monclient: dump: 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.892 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbb9400c030 con 0x7fbba4119820 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 msgr2=0x7fbba413efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 0x7fbba413efd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb94008040 comp rx=0 tx=0).stop 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 msgr2=0x7fbba413efd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbb9a7fc640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4119820 0x7fbba413efd0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb94008040 comp rx=0 tx=0).stop 2024-09-15T02:06:41.893 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.874+0000 7fbbaa93a640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb88038740 0x7fbb8803abf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbba406e0a0 tx=0x7fbb9c037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbbac39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:41.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbbac39d640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 0x7fbba413f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbbac39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba40711e0 con 0x7fbba4118260 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 0x7fbba413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient: get_auth_request con 0x7fbba4118260 auth_method 0 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 10 monclient(hunting): handle_auth_done global_id 184283 payload 824 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.880+0000 7fbba9938640 1 -- 172.21.15.17:0/2214784123 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbba411aac0 con 0x7fbba4118260 2024-09-15T02:06:41.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbba9938640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 msgr2=0x7fbba413f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbba9938640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 0x7fbba413f510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb9401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94022070 con 0x7fbba4118260 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 10 monclient: dump: 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:41.897 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fbb94013800 con 0x7fbba4118260 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 msgr2=0x7fbba413f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 0x7fbba413f510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb9401d000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 msgr2=0x7fbba413f510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbb9a7fc640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba4118260 0x7fbba413f510 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400afd0 tx=0x7fbb9401d000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:41.898 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:41.899 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:41.900 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:41.901 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:41.902 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:41.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.909 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:06:41.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:41.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:41.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:41.911 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:41.911 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:41.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb88038740 msgr2=0x7fbb8803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbb88038740 0x7fbb8803abf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbba406e0a0 tx=0x7fbb9c037000 comp rx=0 tx=0).stop 2024-09-15T02:06:41.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 10 monclient: shutdown 2024-09-15T02:06:41.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:41.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 msgr2=0x7fbba4118910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c5e0 tx=0x7fbba0004960 comp rx=0 tx=0).stop 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbaa93a640 1 -- 172.21.15.17:0/2214784123 reap_dead start 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 shutdown_connections 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 --2- 172.21.15.17:0/2214784123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbba40708a0 0x7fbba4118910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.883+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 >> 172.21.15.17:0/2214784123 conn(0x7fbba4067ae0 msgr2=0x7fbba4130330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.884+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 shutdown_connections 2024-09-15T02:06:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:41.884+0000 7fbbac39d640 1 -- 172.21.15.17:0/2214784123 wait complete. 2024-09-15T02:06:41.913 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:41.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:06:41.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:06:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 10 monclient: build_initial_monmap 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 10 monclient: monmap: 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:42.011180+0000 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:42.011180+0000 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:42.012 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:42.013 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:42.013 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 1 Processor -- start 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 1 -- start start 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.009+0000 7f4fa139d640 10 monclient: init 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f9c096c10 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:42.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f9c09d3b0 0x7f4f9c096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9c09d3b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4f9c097520 0x7f4f9c135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c097520 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c0983b0 con 0x7f4f9c096c10 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c099260 con 0x7f4f9c097520 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c097920 con 0x7f4f9c09d3b0 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4fa139d640 10 monclient(hunting): authenticate will time out at 1662.007503s 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4f9c097520 0x7f4f9c135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4f9c097520 0x7f4f9c135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38046/0 (socket says 172.21.15.17:38046) 2024-09-15T02:06:42.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/3610981161 learned_addr learned my addr 172.21.15.17:0/3610981161 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f9c097520 auth_method 0 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/3610981161 >> v2:172.21.15.67:3300/0 conn(0x7f4f9c09d3b0 0x7f4f9c096fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f9c09d3b0 auth_method 0 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_done global_id 105531 payload 274 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/3610981161 >> v2:172.21.15.67:3300/0 conn(0x7f4f9c09d3b0 msgr2=0x7f4f9c096fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 --2- 172.21.15.17:0/3610981161 >> v2:172.21.15.67:3300/0 conn(0x7f4f9c09d3b0 0x7f4f9c096fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/3610981161 >> v1:172.21.15.67:6789/0 conn(0x7f4f9c096c10 legacy=0x7f4f9c099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 monclient: found mon.noname-a 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.010+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/3610981161 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c097a60 con 0x7f4f9c097520 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/3610981161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c097520 msgr2=0x7f4f9c135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f9b7fe640 1 --2- 172.21.15.17:0/3610981161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c097520 0x7f4f9c135b80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c065ca0 tx=0x7f4f94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4daf20db1270e696 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 1 -- 172.21.15.17:0/3610981161 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f94007610 con 0x7f4f9c097520 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: dump: 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: _finish_auth 0 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 1 -- 172.21.15.17:0/3610981161 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9402d050 con 0x7f4f9c097520 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4fa139d640 5 monclient: authenticate success, global_id 105531 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4fa139d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4fa139d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:42.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4fa139d640 4 set_mon_vals no callback set 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 1 -- 172.21.15.17:0/3610981161 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940079a0 con 0x7f4f9c097520 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.013+0000 7f4f99ffb640 10 monclient: dump: 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 10 monclient: shutdown 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 -- 172.21.15.17:0/3610981161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c097520 msgr2=0x7f4f9c135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 --2- 172.21.15.17:0/3610981161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c097520 0x7f4f9c135b80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c065ca0 tx=0x7f4f94009840 comp rx=0 tx=0).stop 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 -- 172.21.15.17:0/3610981161 shutdown_connections 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 --2- 172.21.15.17:0/3610981161 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c097520 0x7f4f9c135b80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 --2- 172.21.15.17:0/3610981161 >> v2:172.21.15.67:3300/0 conn(0x7f4f9c09d3b0 0x7f4f9c096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 -- 172.21.15.17:0/3610981161 >> 172.21.15.17:0/3610981161 conn(0x7f4f9c0a1000 msgr2=0x7f4f9c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 -- 172.21.15.17:0/3610981161 shutdown_connections 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.014+0000 7f4fa139d640 1 -- 172.21.15.17:0/3610981161 wait complete. 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient: build_initial_monmap 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient: monmap: 2024-09-15T02:06:42.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 Processor -- start 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 -- start start 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient: init 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:42.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 0x7f4f9c1337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9c096c10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c097520 0x7f4f9c133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9c097520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 0x7f4f9c134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c09d3b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c118870 con 0x7f4f9c09d3b0 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c118ac0 con 0x7f4f9c097520 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c098e50 con 0x7f4f9c096c10 2024-09-15T02:06:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.015+0000 7f4fa139d640 10 monclient(hunting): authenticate will time out at 1662.012941s 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 0x7f4f9c134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 0x7f4f9c134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38060/0 (socket says 172.21.15.17:38060) 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/1138888501 learned_addr learned my addr 172.21.15.17:0/1138888501 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f9c09d3b0 auth_method 0 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 0x7f4f9c1337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): get_auth_request con 0x7f4f9c096c10 auth_method 0 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c097520 0x7f4f9c133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f9c097520 auth_method 0 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient(hunting): handle_auth_done global_id 184286 payload 824 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c097520 msgr2=0x7f4f9c133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c097520 0x7f4f9c133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 msgr2=0x7f4f9c134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 0x7f4f9c134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient: _finish_hunting 0 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 monclient: found mon.noname-c 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9affd640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94009ae0 con 0x7f4f9c096c10 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.016+0000 7f4f9b7fe640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4f9c09d3b0 0x7f4f9c134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f9affd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 msgr2=0x7f4f9c1337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f9affd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 0x7f4f9c1337b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c540 tx=0x7f4f8c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0117a0 con 0x7f4f9c096c10 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: dump: 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.035 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: _finish_auth 0 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c011920 con 0x7f4f9c096c10 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 5 monclient: authenticate success, global_id 184286 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 10 monclient: _renew_subs 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c0a0990 con 0x7f4f9c096c10 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c011ce0 con 0x7f4f9c096c10 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f92ffd640 10 monclient: dump: 2024-09-15T02:06:42.036 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 10 monclient: _renew_subs 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4f98ff9640 4 set_mon_vals no callback set 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.018+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c1352e0 con 0x7f4f9c096c10 2024-09-15T02:06:42.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4fa139d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4f8c012630 con 0x7f4f9c096c10 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4fa139d640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 0x7f4f9c12f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c118c50 con 0x7f4f9c134770 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4f92ffd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4f7c038740 0x7f4f7c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.019+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 0x7f4f9c12f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient: get_auth_request con 0x7f4f9c134770 auth_method 0 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4f8c04c1a0 con 0x7f4f9c096c10 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9b7fe640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4f7c038740 0x7f4f7c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9b7fe640 10 monclient: get_auth_request con 0x7f4f7c038740 auth_method 0 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_done global_id 194134 payload 824 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.020+0000 7f4f9a7fc640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f9c118620 con 0x7f4f9c134770 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f9a7fc640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 msgr2=0x7f4f9c12f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 0x7f4f9c12f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4f84011bf0 tx=0x7f4f840090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f9b7fe640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4f7c038740 0x7f4f7c03abf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c12ea50 tx=0x7f4f94036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84014440 con 0x7f4f9c134770 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.022+0000 7f4f92ffd640 10 monclient: dump: 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4f840145a0 con 0x7f4f9c134770 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 msgr2=0x7f4f9c12f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 0x7f4f9c12f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4f84011bf0 tx=0x7f4f840090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 msgr2=0x7f4f9c12f6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:42.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.023+0000 7f4f92ffd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c134770 0x7f4f9c12f6b0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4f84011bf0 tx=0x7f4f840090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.028+0000 7f4fa139d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.028+0000 7f4fa139d640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 0x7f4f9c12fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.028+0000 7f4fa139d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.028+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c063c20 con 0x7f4f9c12fbf0 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 0x7f4f9c12fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient: get_auth_request con 0x7f4f9c12fbf0 auth_method 0 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 10 monclient(hunting): handle_auth_done global_id 194137 payload 824 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.029+0000 7f4f9a7fc640 1 -- 172.21.15.17:0/1138888501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4f9c0724a0 con 0x7f4f9c12fbf0 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.031+0000 7f4f9a7fc640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 msgr2=0x7f4f9c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f9a7fc640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 0x7f4f9c12fff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f8401b000 tx=0x7f4f8401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84028400 con 0x7f4f9c12fbf0 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 10 monclient: dump: 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f4f8401c3e0 con 0x7f4f9c12fbf0 2024-09-15T02:06:42.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 msgr2=0x7f4f9c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 0x7f4f9c12fff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f8401b000 tx=0x7f4f8401c000 comp rx=0 tx=0).stop 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 msgr2=0x7f4f9c12fff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f92ffd640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c12fbf0 0x7f4f9c12fff0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f8401b000 tx=0x7f4f8401c000 comp rx=0 tx=0).stop 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 156, 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:42.057 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:42.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4f7c038740 msgr2=0x7f4f7c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4f7c038740 0x7f4f7c03abf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c12ea50 tx=0x7f4f94036000 comp rx=0 tx=0).stop 2024-09-15T02:06:42.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 10 monclient: shutdown 2024-09-15T02:06:42.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 msgr2=0x7f4f9c1337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4fa139d640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 0x7f4f9c1337b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c540 tx=0x7f4f8c007e10 comp rx=0 tx=0).stop 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.032+0000 7f4f9b7fe640 1 -- 172.21.15.17:0/1138888501 reap_dead start 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.033+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 shutdown_connections 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.033+0000 7f4fa139d640 1 --2- 172.21.15.17:0/1138888501 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4f9c096c10 0x7f4f9c1337b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.033+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 >> 172.21.15.17:0/1138888501 conn(0x7f4f9c0a1000 msgr2=0x7f4f9c09f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.033+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 shutdown_connections 2024-09-15T02:06:42.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.033+0000 7f4fa139d640 1 -- 172.21.15.17:0/1138888501 wait complete. 2024-09-15T02:06:42.059 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:06:42.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T02:06:42.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:06:42.060 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T02:06:42.060 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:06:42.060 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:06:42.060 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:06:42.073 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:06:42.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:06:42.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:42.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.166+0000 7f3835b5d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.166+0000 7f3835b5d640 10 monclient: build_initial_monmap 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.166+0000 7f3835b5d640 10 monclient: monmap: 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:42.168131+0000 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:42.168131+0000 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:42.170 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:42.170 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 Processor -- start 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 -- start start 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient: init 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 0x7f383009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f3830131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:42.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38300a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f383009f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300a0040 con 0x7f38300a0280 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383009bff0 con 0x7f3830131b20 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300a14a0 con 0x7f383009f1e0 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f3835b5d640 10 monclient(hunting): authenticate will time out at 1662.164705s 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382f7fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 msgr2=0x7f383009eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 0x7f383009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:42.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382effd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f38300a0280 legacy=0x7f3830131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60078/0 (socket says 172.21.15.17:60078) 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 1 -- 172.21.15.17:0/1271977769 learned_addr learned my addr 172.21.15.17:0/1271977769 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): get_auth_request con 0x7f383009f1e0 auth_method 0 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.167+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 10 monclient(hunting): handle_auth_done global_id 184289 payload 274 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 1 -- 172.21.15.17:0/1271977769 >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 msgr2=0x7f383009eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 1 --2- 172.21.15.17:0/1271977769 >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 0x7f383009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 1 -- 172.21.15.17:0/1271977769 >> v1:172.21.15.67:6789/0 conn(0x7f38300a0280 legacy=0x7f3830131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:42.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:42.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 1 monclient: found mon.noname-b 2024-09-15T02:06:42.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:42.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.168+0000 7f382ffff640 1 -- 172.21.15.17:0/1271977769 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830132600 con 0x7f383009f1e0 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.170+0000 7f382ffff640 1 -- 172.21.15.17:0/1271977769 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 msgr2=0x7f3830138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.170+0000 7f382ffff640 1 --2- 172.21.15.17:0/1271977769 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3828009f20 tx=0x7f3828009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b11c921ed10e73f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 1 -- 172.21.15.17:0/1271977769 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828007610 con 0x7f383009f1e0 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: dump: 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 1 -- 172.21.15.17:0/1271977769 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382802d050 con 0x7f383009f1e0 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 1 -- 172.21.15.17:0/1271977769 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38280079a0 con 0x7f383009f1e0 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 5 monclient: authenticate success, global_id 184289 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:42.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 4 set_mon_vals no callback set 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f382e7fc640 10 monclient: dump: 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.182 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 10 monclient: shutdown 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 -- 172.21.15.17:0/1271977769 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 msgr2=0x7f3830138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1271977769 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3828009f20 tx=0x7f3828009840 comp rx=0 tx=0).stop 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 -- 172.21.15.17:0/1271977769 shutdown_connections 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1271977769 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f383009f1e0 0x7f3830138e60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1271977769 >> v2:172.21.15.67:3300/0 conn(0x7f3830131b20 0x7f383009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 -- 172.21.15.17:0/1271977769 >> 172.21.15.17:0/1271977769 conn(0x7f3830071270 msgr2=0x7f38300a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 -- 172.21.15.17:0/1271977769 shutdown_connections 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.171+0000 7f3835b5d640 1 -- 172.21.15.17:0/1271977769 wait complete. 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.172+0000 7f3835b5d640 10 monclient: build_initial_monmap 2024-09-15T02:06:42.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.172+0000 7f3835b5d640 10 monclient: monmap: 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.172+0000 7f3835b5d640 1 Processor -- start 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.172+0000 7f3835b5d640 1 -- start start 2024-09-15T02:06:42.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.172+0000 7f3835b5d640 10 monclient: init 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 0x7f383009b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f383009f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f38300a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f3830131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383012ede0 con 0x7f38300a0280 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830097da0 con 0x7f383009f1e0 2024-09-15T02:06:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300977e0 con 0x7f3830131b20 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f3835b5d640 10 monclient(hunting): authenticate will time out at 1662.170338s 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382f7fe640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 msgr2=0x7f383009b4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 0x7f383009b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:42.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60088/0 (socket says 172.21.15.17:60088) 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38076/0 (socket says 172.21.15.17:38076) 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 -- 172.21.15.17:0/1462280231 learned_addr learned my addr 172.21.15.17:0/1462280231 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): get_auth_request con 0x7f3830131b20 auth_method 0 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): get_auth_request con 0x7f38300a0280 auth_method 0 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient(hunting): handle_auth_done global_id 184292 payload 824 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 -- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 msgr2=0x7f383009b4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 0x7f383009b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 -- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 msgr2=0x7f383012d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 monclient: found mon.noname-c 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382ffff640 1 -- 172.21.15.17:0/1462280231 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828009ae0 con 0x7f3830131b20 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.173+0000 7f382effd640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f382ffff640 1 -- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 msgr2=0x7f383012d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f382ffff640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3828009f20 tx=0x7f3828036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828031da0 con 0x7f3830131b20 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: dump: 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: _finish_auth 0 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382803c420 con 0x7f3830131b20 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 5 monclient: authenticate success, global_id 184292 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 10 monclient: _renew_subs 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3830097500 con 0x7f3830131b20 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382803c7e0 con 0x7f3830131b20 2024-09-15T02:06:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f38177fe640 10 monclient: dump: 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:42.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 10 monclient: _renew_subs 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f382dffb640 4 set_mon_vals no callback set 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.176+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383012f1f0 con 0x7f3830131b20 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f382803cbd0 con 0x7f3830131b20 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f38177fe640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 0x7f381c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f382806d050 con 0x7f3830131b20 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f3835b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f3835b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f3835b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38300979a0 con 0x7f3830131b20 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f382f7fe640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 0x7f381c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:42.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.177+0000 7f382f7fe640 10 monclient: get_auth_request con 0x7f381c0386f0 auth_method 0 2024-09-15T02:06:42.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.180+0000 7f382f7fe640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 0x7f381c03aba0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f38200038f0 tx=0x7f3820006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:42.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.181+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3828049080 con 0x7f3830131b20 2024-09-15T02:06:42.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.181+0000 7f38177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:42.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.181+0000 7f38177fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:42.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.407+0000 7f3835b5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:42.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.407+0000 7f3835b5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:42.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.407+0000 7f3835b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:42.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.407+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f38300724a0 con 0x7f3830131b20 2024-09-15T02:06:42.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.408+0000 7f38177fe640 1 -- 172.21.15.17:0/1462280231 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f3828040070 con 0x7f3830131b20 2024-09-15T02:06:42.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.408+0000 7f38177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:42.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.408+0000 7f38177fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:42.411 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 msgr2=0x7f381c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 0x7f381c03aba0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f38200038f0 tx=0x7f3820006d80 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 10 monclient: shutdown 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 msgr2=0x7f383012d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3828009f20 tx=0x7f3828036000 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 shutdown_connections 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f381c0386f0 0x7f381c03aba0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38300a0280 0x7f383012d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3830131b20 0x7f383012d610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 --2- 172.21.15.17:0/1462280231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383009f1e0 0x7f383009b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 >> 172.21.15.17:0/1462280231 conn(0x7f3830071270 msgr2=0x7f38300a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:42.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 shutdown_connections 2024-09-15T02:06:42.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:42.410+0000 7f3835b5d640 1 -- 172.21.15.17:0/1462280231 wait complete. 2024-09-15T02:06:45.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.529+0000 7fd37431d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.529+0000 7fd37431d640 10 monclient: build_initial_monmap 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.529+0000 7fd37431d640 10 monclient: monmap: 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:45.531241+0000 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:45.531241+0000 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:45.533 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:45.533 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:45.533 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:45.533 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 Processor -- start 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 -- start start 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient: init 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd36c096c10 0x7fd36c099f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c096c10 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 0x7fd36c096fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c09d3b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd36c097520 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:45.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c099260 con 0x7fd36c097520 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36c098440 con 0x7fd36c096c10 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c097960 con 0x7fd36c09d3b0 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd37431d640 10 monclient(hunting): authenticate will time out at 1665.527815s 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd3728ba640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fd36c097520 legacy=0x7fd36c12e630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd3720b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd36c096c10 0x7fd36c099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.530+0000 7fd3718b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 0x7fd36c096fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd36c096c10 0x7fd36c099f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49058/0 (socket says 172.21.15.17:49058) 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 -- 172.21.15.17:0/594417942 learned_addr learned my addr 172.21.15.17:0/594417942 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): get_auth_request con 0x7fd36c096c10 auth_method 0 2024-09-15T02:06:45.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): get_auth_request con 0x7fd36c09d3b0 auth_method 0 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:45.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient(hunting): handle_auth_done global_id 105540 payload 274 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 -- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 msgr2=0x7fd36c096fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 0x7fd36c096fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 -- 172.21.15.17:0/594417942 >> v1:172.21.15.67:6789/0 conn(0x7fd36c097520 legacy=0x7fd36c12e630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient: _finish_hunting 0 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 monclient: found mon.noname-a 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3720b9640 1 -- 172.21.15.17:0/594417942 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c09a460 con 0x7fd36c096c10 2024-09-15T02:06:45.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.531+0000 7fd3718b8640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 0x7fd36c096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3720b9640 1 -- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c096c10 msgr2=0x7fd36c099f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3720b9640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c096c10 0x7fd36c099f20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd36800bbc0 tx=0x7fd36800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5326f772fa22cb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 1 -- 172.21.15.17:0/594417942 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368030050 con 0x7fd36c096c10 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: dump: 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:45.543 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: _finish_auth 0 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 1 -- 172.21.15.17:0/594417942 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368032710 con 0x7fd36c096c10 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 5 monclient: authenticate success, global_id 105540 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:45.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 1 -- 172.21.15.17:0/594417942 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368032ab0 con 0x7fd36c096c10 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 4 set_mon_vals no callback set 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd3710b7640 10 monclient: dump: 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 10 monclient: shutdown 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 -- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c096c10 msgr2=0x7fd36c099f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c096c10 0x7fd36c099f20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd36800bbc0 tx=0x7fd36800b460 comp rx=0 tx=0).stop 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 -- 172.21.15.17:0/594417942 shutdown_connections 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c09d3b0 0x7fd36c096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 --2- 172.21.15.17:0/594417942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c096c10 0x7fd36c099f20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.534+0000 7fd37431d640 1 -- 172.21.15.17:0/594417942 >> 172.21.15.17:0/594417942 conn(0x7fd36c0a1000 msgr2=0x7fd36c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 1 -- 172.21.15.17:0/594417942 shutdown_connections 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 1 -- 172.21.15.17:0/594417942 wait complete. 2024-09-15T02:06:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 10 monclient: build_initial_monmap 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 10 monclient: monmap: 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 1 Processor -- start 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 1 -- start start 2024-09-15T02:06:45.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.535+0000 7fd37431d640 10 monclient: init 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 0x7fd36c141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-c con 0x7fd36c096c10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c097520 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 0x7fd36c12e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c09d3b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c121f50 con 0x7fd36c097520 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c1206c0 con 0x7fd36c09d3b0 2024-09-15T02:06:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c098e50 con 0x7fd36c096c10 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd37431d640 10 monclient(hunting): authenticate will time out at 1665.533320s 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 0x7fd36c141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49070/0 (socket says 172.21.15.17:49070) 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 1 -- 172.21.15.17:0/155065348 learned_addr learned my addr 172.21.15.17:0/155065348 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): get_auth_request con 0x7fd36c096c10 auth_method 0 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): get_auth_request con 0x7fd36c097520 auth_method 0 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3728ba640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 msgr2=0x7fd36c12e020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3728ba640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 0x7fd36c12e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient(hunting): handle_auth_done global_id 184298 payload 824 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 msgr2=0x7fd36c12e020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 0x7fd36c12e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 msgr2=0x7fd36c12dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient: _finish_hunting 0 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 monclient: found mon.noname-c 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3720b9640 1 -- 172.21.15.17:0/155065348 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c009390 con 0x7fd36c096c10 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.536+0000 7fd3718b8640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3720b9640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 msgr2=0x7fd36c141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3720b9640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 0x7fd36c141400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd368034000 tx=0x7fd36803a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36803e860 con 0x7fd36c096c10 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: dump: 2024-09-15T02:06:45.551 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36803e9e0 con 0x7fd36c096c10 2024-09-15T02:06:45.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36803eda0 con 0x7fd36c096c10 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 5 monclient: authenticate success, global_id 184298 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3627fc640 10 monclient: dump: 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:45.553 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 10 monclient: _renew_subs 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36800b700 con 0x7fd36c096c10 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd3708b6640 4 set_mon_vals no callback set 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 10 monclient: _renew_subs 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.539+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36c142230 con 0x7fd36c096c10 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd368034600 con 0x7fd36c096c10 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd37431d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd37431d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd37431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:45.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36c0724a0 con 0x7fd36c096c10 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd3627fc640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 0x7fd35003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd36806d1d0 con 0x7fd36c096c10 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd3718b8640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 0x7fd35003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.540+0000 7fd3718b8640 10 monclient: get_auth_request con 0x7fd3500386f0 auth_method 0 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.543+0000 7fd3718b8640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 0x7fd35003aba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd35c011640 tx=0x7fd35c0090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.544+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd36803db80 con 0x7fd36c096c10 2024-09-15T02:06:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.544+0000 7fd3627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:45.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.544+0000 7fd3627fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:45.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd37431d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd37431d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd37431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd36c122190 con 0x7fd36c096c10 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd3627fc640 1 -- 172.21.15.17:0/155065348 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fd368042020 con 0x7fd36c096c10 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd3627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.772+0000 7fd3627fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:45.775 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 msgr2=0x7fd35003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 0x7fd35003aba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd35c011640 tx=0x7fd35c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 10 monclient: shutdown 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 msgr2=0x7fd36c141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 0x7fd36c141400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd368034000 tx=0x7fd36803a000 comp rx=0 tx=0).stop 2024-09-15T02:06:45.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 shutdown_connections 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd3500386f0 0x7fd35003aba0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd36c097520 0x7fd36c12dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c09d3b0 0x7fd36c12e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 --2- 172.21.15.17:0/155065348 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd36c096c10 0x7fd36c141400 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 >> 172.21.15.17:0/155065348 conn(0x7fd36c0a1000 msgr2=0x7fd36c06bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 shutdown_connections 2024-09-15T02:06:45.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:45.774+0000 7fd37431d640 1 -- 172.21.15.17:0/155065348 wait complete. 2024-09-15T02:06:48.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:48.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 10 monclient: get_monmap_and_config 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 10 monclient: build_initial_monmap 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 10 monclient: monmap: 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:48.896143+0000 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:48.896143+0000 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:48.897 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:48.898 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:48.898 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:48.898 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:48.898 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.899 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 1 Processor -- start 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 1 -- start start 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.894+0000 7f00e446f640 10 monclient: init 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 0x7f00dc132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-c con 0x7f00dc0a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f00dc0980a0 0x7f00dc0984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-a con 0x7f00dc0980a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00dc0989e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc0a01e0 con 0x7f00dc0989e0 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:48.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc0a1580 con 0x7f00dc0980a0 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc0a5c80 con 0x7f00dc0a58b0 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e446f640 10 monclient(hunting): authenticate will time out at 1668.892742s 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e220b640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 msgr2=0x7f00dc132040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f00dc0980a0 0x7f00dc0984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e220b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 0x7f00dc132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f00dc0980a0 0x7f00dc0984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49076/0 (socket says 172.21.15.17:49076) 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 1 -- 172.21.15.17:0/4225190572 learned_addr learned my addr 172.21.15.17:0/4225190572 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 10 monclient(hunting): get_auth_request con 0x7f00dc0980a0 auth_method 0 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:48.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:48.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.895+0000 7f00e1a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e2a0c640 1 -- 172.21.15.17:0/4225190572 >> v1:172.21.15.17:6789/0 conn(0x7f00dc0989e0 legacy=0x7f00dc068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_done global_id 105546 payload 274 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 1 -- 172.21.15.17:0/4225190572 >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 msgr2=0x7f00dc132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 1 --2- 172.21.15.17:0/4225190572 >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 0x7f00dc132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 1 -- 172.21.15.17:0/4225190572 >> v1:172.21.15.17:6789/0 conn(0x7f00dc0989e0 legacy=0x7f00dc068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient: _finish_hunting 0 2024-09-15T02:06:48.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 1 monclient: found mon.noname-a 2024-09-15T02:06:48.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:48.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:48.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.898+0000 7f00e1a0a640 1 -- 172.21.15.17:0/4225190572 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc1326c0 con 0x7f00dc0980a0 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1a0a640 1 -- 172.21.15.17:0/4225190572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc0980a0 msgr2=0x7f00dc0984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1a0a640 1 --2- 172.21.15.17:0/4225190572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc0980a0 0x7f00dc0984a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0a60e0 tx=0x7f00cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c065fd874e3ab3dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 1 -- 172.21.15.17:0/4225190572 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc02f4b0 con 0x7f00dc0980a0 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:48.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: dump: 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: _finish_auth 0 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:48.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 1 -- 172.21.15.17:0/4225190572 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc02f610 con 0x7f00dc0980a0 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 5 monclient: authenticate success, global_id 105546 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 1 -- 172.21.15.17:0/4225190572 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc02f9f0 con 0x7f00dc0980a0 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 4 set_mon_vals no callback set 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e1209640 10 monclient: dump: 2024-09-15T02:06:48.912 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:48.913 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 10 monclient: shutdown 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 -- 172.21.15.17:0/4225190572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc0980a0 msgr2=0x7f00dc0984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 --2- 172.21.15.17:0/4225190572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc0980a0 0x7f00dc0984a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0a60e0 tx=0x7f00cc0091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 -- 172.21.15.17:0/4225190572 shutdown_connections 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 --2- 172.21.15.17:0/4225190572 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc0980a0 0x7f00dc0984a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 --2- 172.21.15.17:0/4225190572 >> v2:172.21.15.67:3300/0 conn(0x7f00dc0a58b0 0x7f00dc132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 -- 172.21.15.17:0/4225190572 >> 172.21.15.17:0/4225190572 conn(0x7f00dc070ed0 msgr2=0x7f00dc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.901+0000 7f00e446f640 1 -- 172.21.15.17:0/4225190572 shutdown_connections 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 1 -- 172.21.15.17:0/4225190572 wait complete. 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 10 monclient: build_initial_monmap 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 10 monclient: monmap: 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.915 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 1 Processor -- start 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 1 -- start start 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.902+0000 7f00e446f640 10 monclient: init 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 0x7f00dc09b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-b con 0x7f00dc0980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 0x7f00dc130e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-c con 0x7f00dc0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): picked mon.noname-a con 0x7f00dc131340 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc11c970 con 0x7f00dc131340 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc09e9c0 con 0x7f00dc0980a0 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc098dc0 con 0x7f00dc0a58b0 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e446f640 10 monclient(hunting): authenticate will time out at 1668.900295s 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49090/0 (socket says 172.21.15.17:49090) 2024-09-15T02:06:48.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 -- 172.21.15.17:0/434305981 learned_addr learned my addr 172.21.15.17:0/434305981 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 0x7f00dc130e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e220b640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 msgr2=0x7f00dc09b140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): get_auth_request con 0x7f00dc131340 auth_method 0 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e220b640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 0x7f00dc09b140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): get_auth_request con 0x7f00dc0a58b0 auth_method 0 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:48.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient(hunting): handle_auth_done global_id 105549 payload 824 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 -- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 msgr2=0x7f00dc130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 0x7f00dc130e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 -- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 msgr2=0x7f00dc09b140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 0x7f00dc09b140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient: _finish_hunting 0 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 monclient: found mon.noname-a 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e2a0c640 1 -- 172.21.15.17:0/434305981 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc009450 con 0x7f00dc131340 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.903+0000 7f00e1a0a640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 0x7f00dc130e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e2a0c640 1 -- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 msgr2=0x7f00dc11b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e2a0c640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f00d400c840 tx=0x7f00d40018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4001950 con 0x7f00dc131340 2024-09-15T02:06:48.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: dump: 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:48.921 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 5 monclient: authenticate success, global_id 105549 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 10 monclient: _renew_subs 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d400a640 con 0x7f00dc131340 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc11cfc0 con 0x7f00dc131340 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d400aa00 con 0x7f00dc131340 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:48.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e0a08640 4 set_mon_vals no callback set 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00d27fc640 10 monclient: dump: 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:48.923 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 10 monclient: _renew_subs 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.906+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc142180 con 0x7f00dc131340 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00e446f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00e446f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00e446f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc0724a0 con 0x7f00dc131340 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f00d40070f0 con 0x7f00dc131340 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00d27fc640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 0x7f00c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:48.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f00d404c140 con 0x7f00dc131340 2024-09-15T02:06:48.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.907+0000 7f00e220b640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 0x7f00c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:48.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.908+0000 7f00e220b640 10 monclient: get_auth_request con 0x7f00c0038740 auth_method 0 2024-09-15T02:06:48.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.910+0000 7f00e220b640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 0x7f00c003abf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f00d80038f0 tx=0x7f00d8006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:48.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.912+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f00d4010d70 con 0x7f00dc131340 2024-09-15T02:06:48.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.912+0000 7f00d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:48.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:48.912+0000 7f00d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:49.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00e446f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:49.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00e446f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:49.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00e446f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:49.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f00dc09fc90 con 0x7f00dc131340 2024-09-15T02:06:49.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00d27fc640 1 -- 172.21.15.17:0/434305981 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f00d4010770 con 0x7f00dc131340 2024-09-15T02:06:49.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:49.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.144+0000 7f00d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:49.150 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 msgr2=0x7f00c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 0x7f00c003abf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f00d80038f0 tx=0x7f00d8006d80 comp rx=0 tx=0).stop 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 10 monclient: shutdown 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 msgr2=0x7f00dc11b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f00d400c840 tx=0x7f00d40018c0 comp rx=0 tx=0).stop 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 shutdown_connections 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f00c0038740 0x7f00c003abf0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:49.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f00dc131340 0x7f00dc11b7d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:49.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f00dc0a58b0 0x7f00dc130e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:49.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.146+0000 7f00e446f640 1 --2- 172.21.15.17:0/434305981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00dc0980a0 0x7f00dc09b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:49.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.147+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 >> 172.21.15.17:0/434305981 conn(0x7f00dc070ed0 msgr2=0x7f00dc096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:49.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.147+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 shutdown_connections 2024-09-15T02:06:49.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:49.147+0000 7f00e446f640 1 -- 172.21.15.17:0/434305981 wait complete. 2024-09-15T02:06:52.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:06:52.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.266+0000 7fed8431d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:52.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.266+0000 7fed8431d640 10 monclient: build_initial_monmap 2024-09-15T02:06:52.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.266+0000 7fed8431d640 10 monclient: monmap: 2024-09-15T02:06:52.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:52.269 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:52.268612+0000 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:52.268612+0000 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:52.270 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.267+0000 7fed8431d640 1 Processor -- start 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.267+0000 7fed8431d640 1 -- start start 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.267+0000 7fed8431d640 10 monclient: init 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed7c09c500 addr v1:172.21.15.17:6789/0 2024-09-15T02:06:52.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-b con 0x7fed7c06f450 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed7c09c8d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c09ba20 con 0x7fed7c09c500 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c09f6c0 con 0x7fed7c09c8d0 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c09b740 con 0x7fed7c06f450 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed8431d640 10 monclient(hunting): authenticate will time out at 1672.265349s 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed820b9640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fed7c09c500 0x7fed7c06ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:56942/0 (socket says 172.21.15.17:56942) 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 learned_addr learned my addr 172.21.15.17:0/3157834131 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:06:52.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed7c09c8d0 legacy=0x7fed7c136cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed7c06f450 auth_method 0 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:52.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:52.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.268+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:52.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.270+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.17:6789/0 conn(0x7fed7c09c500 legacy=0x7fed7c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed7c09c8d0 legacy=0x7fed7c136cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed7c06f450 auth_method 0 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:06:52.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:52.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:52.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.469+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed7c09c8d0 legacy=0x7fed7c136cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed7c06f450 auth_method 0 2024-09-15T02:06:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:06:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:52.870+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:53.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed7c09c8d0 legacy=0x7fed7c136cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:53.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:53.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed7c06f450 auth_method 0 2024-09-15T02:06:53.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:53.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:06:55.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): tick 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 monclient(hunting): continuing hunt 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 msgr2=0x7fed7c134760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed7c06f450 0x7fed7c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed7c09c8d0 legacy=0x7fed7c136cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.17:6789/0 conn(0x7fed7c09c500 legacy=0x7fed7c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient: _add_conns ranks=[4,3] 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fed60003730 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:55.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fed600061d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c09b740 con 0x7fed600061d0 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed60008d40 con 0x7fed60003730 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed6ffff640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 msgr2=0x7fed60005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:55.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:55.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:55.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:55.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.269+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 msgr2=0x7fed60005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:55.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:06:55.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:55.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:55.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.470+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:06:55.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 msgr2=0x7fed60005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:55.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:55.871+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 msgr2=0x7fed60005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:06:56.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:06:56.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:06:56.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:06:56.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:56.672+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): tick 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 monclient(hunting): continuing hunt 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 msgr2=0x7fed60005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.67:3300/0 conn(0x7fed60003730 0x7fed60005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 msgr2=0x7fed60008680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 reap_dead start 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fed600061d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 0x7fed60009130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fed60003b90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed60009670 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c09bd00 con 0x7fed60009670 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c09b740 con 0x7fed600061d0 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 1 -- 172.21.15.17:0/3157834131 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed60008d40 con 0x7fed60003b90 2024-09-15T02:06:58.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed6ffff640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> v2:172.21.15.17:3300/0 conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 0x7fed60009130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed828ba640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed60009670 legacy=0x7fed6000ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed820b9640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed820b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed820b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed820b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed60003b90 auth_method 0 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.268+0000 7fed818b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient(hunting): handle_auth_done global_id 105552 payload 274 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 msgr2=0x7fed60009130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 0x7fed60009130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> v1:172.21.15.67:6789/0 conn(0x7fed60009670 legacy=0x7fed6000ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 1 monclient: found mon.noname-a 2024-09-15T02:06:58.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:58.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c09f6c0 con 0x7fed600061d0 2024-09-15T02:06:58.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.269+0000 7fed818b8640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 0x7fed60009130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.273+0000 7fed820b9640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed600061d0 msgr2=0x7fed60008680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.273+0000 7fed820b9640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed600061d0 0x7fed60008680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fed7c09bbb0 tx=0x7fed78007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548dcd558111788b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.273+0000 7fed810b7640 1 -- 172.21.15.17:0/3157834131 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 1 -- 172.21.15.17:0/3157834131 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78005890 con 0x7fed600061d0 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: dump: 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.279 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 1 -- 172.21.15.17:0/3157834131 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed780059f0 con 0x7fed600061d0 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 1 -- 172.21.15.17:0/3157834131 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78005df0 con 0x7fed600061d0 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed8431d640 5 monclient: authenticate success, global_id 105552 2024-09-15T02:06:58.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed810b7640 10 monclient: dump: 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed8431d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:58.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.275+0000 7fed8431d640 4 set_mon_vals no callback set 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 10 monclient: shutdown 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed600061d0 msgr2=0x7fed60008680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed600061d0 0x7fed60008680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fed7c09bbb0 tx=0x7fed78007e00 comp rx=0 tx=0).stop 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- 172.21.15.17:0/3157834131 shutdown_connections 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed60003b90 0x7fed60009130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 --2- 172.21.15.17:0/3157834131 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed600061d0 0x7fed60008680 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- 172.21.15.17:0/3157834131 >> 172.21.15.17:0/3157834131 conn(0x7fed7c0712f0 msgr2=0x7fed7c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- 172.21.15.17:0/3157834131 shutdown_connections 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- 172.21.15.17:0/3157834131 wait complete. 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 10 monclient: build_initial_monmap 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 10 monclient: monmap: 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:58.282 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 Processor -- start 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 1 -- start start 2024-09-15T02:06:58.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.276+0000 7fed8431d640 10 monclient: init 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 0x7fed7c13f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-b con 0x7fed60003b90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-c con 0x7fed600061d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 0x7fed7c1300a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): picked mon.noname-a con 0x7fed60009670 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c1305e0 con 0x7fed60009670 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c09cf00 con 0x7fed60003b90 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c098ca0 con 0x7fed600061d0 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed8431d640 10 monclient(hunting): authenticate will time out at 1678.274197s 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed820b9640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 msgr2=0x7fed7c13f100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed820b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 0x7fed7c13f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 0x7fed7c1300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36428/0 (socket says 172.21.15.17:36428) 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 -- 172.21.15.17:0/937751452 learned_addr learned my addr 172.21.15.17:0/937751452 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:58.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): get_auth_request con 0x7fed600061d0 auth_method 0 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): get_auth_request con 0x7fed60009670 auth_method 0 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient(hunting): handle_auth_done global_id 184307 payload 824 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 -- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 msgr2=0x7fed7c13f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 0x7fed7c13f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 -- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 msgr2=0x7fed7c1300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 0x7fed7c1300a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 monclient: found mon.noname-c 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed818b8640 1 -- 172.21.15.17:0/937751452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7000e040 con 0x7fed600061d0 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.277+0000 7fed828ba640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 0x7fed7c1300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed818b8640 1 -- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 msgr2=0x7fed7c12fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed818b8640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed70018800 tx=0x7fed70008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70027070 con 0x7fed600061d0 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: dump: 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7001f040 con 0x7fed600061d0 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 5 monclient: authenticate success, global_id 184307 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 10 monclient: _renew_subs 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c09ba20 con 0x7fed600061d0 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7001e050 con 0x7fed600061d0 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6e7fc640 10 monclient: dump: 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 10 monclient: _renew_subs 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed6ffff640 4 set_mon_vals no callback set 2024-09-15T02:06:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.280+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c11b930 con 0x7fed600061d0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fed70023610 con 0x7fed600061d0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed8431d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed8431d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed8431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c11d350 con 0x7fed600061d0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed6e7fc640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 0x7fed6003c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.281+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fed7004e1c0 con 0x7fed600061d0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.282+0000 7fed820b9640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 0x7fed6003c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.282+0000 7fed820b9640 10 monclient: get_auth_request con 0x7fed60039d90 auth_method 0 2024-09-15T02:06:58.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.284+0000 7fed820b9640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 0x7fed6003c240 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fed60003b00 tx=0x7fed78030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.286+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fed700190f0 con 0x7fed600061d0 2024-09-15T02:06:58.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.286+0000 7fed6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.286+0000 7fed6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed8431d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed8431d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed8431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fed7c0724a0 con 0x7fed600061d0 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed6e7fc640 1 -- 172.21.15.17:0/937751452 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7fed70022780 con 0x7fed600061d0 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.516+0000 7fed6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:58.519 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 msgr2=0x7fed6003c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 0x7fed6003c240 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fed60003b00 tx=0x7fed78030000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 10 monclient: shutdown 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 msgr2=0x7fed7c12fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed70018800 tx=0x7fed70008040 comp rx=0 tx=0).stop 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 shutdown_connections 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fed60039d90 0x7fed6003c240 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fed60009670 0x7fed7c1300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fed600061d0 0x7fed7c12fb60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 --2- 172.21.15.17:0/937751452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed60003b90 0x7fed7c13f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 >> 172.21.15.17:0/937751452 conn(0x7fed7c0712f0 msgr2=0x7fed7c132a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 shutdown_connections 2024-09-15T02:06:58.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.518+0000 7fed8431d640 1 -- 172.21.15.17:0/937751452 wait complete. 2024-09-15T02:06:58.531 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:06:58.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:06:58.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.627+0000 7f16ac79d640 10 monclient: get_monmap_and_config 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.627+0000 7f16ac79d640 10 monclient: build_initial_monmap 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.627+0000 7f16ac79d640 10 monclient: monmap: 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.629 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:58.628316+0000 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:58.628316+0000 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 Processor -- start 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 -- start start 2024-09-15T02:06:58.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient: init 2024-09-15T02:06:58.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:58.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:06:58.632 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f16a406eb30 0x7f16a406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-a con 0x7f16a406eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 0x7f16a409bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-b con 0x7f16a406f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16a409c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a409f6a0 con 0x7f16a409c4d0 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a4069f40 con 0x7f16a406eb30 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4069950 con 0x7f16a406f440 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16ac79d640 10 monclient(hunting): authenticate will time out at 1678.624775s 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f16a406eb30 0x7f16a406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 0x7f16a409bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f16a406eb30 0x7f16a406ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:45574/0 (socket says 172.21.15.17:45574) 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 0x7f16a409bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36434/0 (socket says 172.21.15.17:36434) 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aad3a640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16a409c4d0 legacy=0x7f16a409d960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 1 -- 172.21.15.17:0/4171483526 learned_addr learned my addr 172.21.15.17:0/4171483526 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 10 monclient(hunting): get_auth_request con 0x7f16a406eb30 auth_method 0 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16aa539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 10 monclient(hunting): get_auth_request con 0x7f16a406f440 auth_method 0 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.628+0000 7f16a9d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient(hunting): handle_auth_done global_id 105558 payload 274 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 1 -- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 msgr2=0x7f16a409bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 1 --2- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 0x7f16a409bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 1 -- 172.21.15.17:0/4171483526 >> v1:172.21.15.67:6789/0 conn(0x7f16a409c4d0 legacy=0x7f16a409d960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 1 monclient: found mon.noname-a 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.629+0000 7f16aa539640 1 -- 172.21.15.17:0/4171483526 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a409ad80 con 0x7f16a406eb30 2024-09-15T02:06:58.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16aa539640 1 -- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a406eb30 msgr2=0x7f16a406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16aa539640 1 --2- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a406eb30 0x7f16a406ef00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009f20 tx=0x7f16a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be89b25f0a173ebe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 1 -- 172.21.15.17:0/4171483526 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0007610 con 0x7f16a406eb30 2024-09-15T02:06:58.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: dump: 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 1 -- 172.21.15.17:0/4171483526 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a002d050 con 0x7f16a406eb30 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 5 monclient: authenticate success, global_id 105558 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 1 -- 172.21.15.17:0/4171483526 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a00079e0 con 0x7f16a406eb30 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 4 set_mon_vals no callback set 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16a9537640 10 monclient: dump: 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.643 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 10 monclient: shutdown 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.632+0000 7f16ac79d640 1 -- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a406eb30 msgr2=0x7f16a406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 --2- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a406eb30 0x7f16a406ef00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009f20 tx=0x7f16a0009840 comp rx=0 tx=0).stop 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 -- 172.21.15.17:0/4171483526 shutdown_connections 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 --2- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406f440 0x7f16a409bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 --2- 172.21.15.17:0/4171483526 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a406eb30 0x7f16a406ef00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 -- 172.21.15.17:0/4171483526 >> 172.21.15.17:0/4171483526 conn(0x7f16a40712f0 msgr2=0x7f16a40a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 -- 172.21.15.17:0/4171483526 shutdown_connections 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 -- 172.21.15.17:0/4171483526 wait complete. 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 10 monclient: build_initial_monmap 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 10 monclient: monmap: 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:58.645 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 Processor -- start 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 1 -- start start 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.633+0000 7f16ac79d640 10 monclient: init 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 0x7f16a4133ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-c con 0x7f16a406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a406f440 0x7f16a4134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-b con 0x7f16a406f440 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a409c4d0 0x7f16a411d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): picked mon.noname-a con 0x7f16a409c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:58.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a409cd10 con 0x7f16a409c4d0 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4134bd0 con 0x7f16a406f440 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4121710 con 0x7f16a406eb30 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16ac79d640 10 monclient(hunting): authenticate will time out at 1678.630445s 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 0x7f16a4133ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 0x7f16a4133ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36438/0 (socket says 172.21.15.17:36438) 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a409c4d0 0x7f16a411d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 -- 172.21.15.17:0/774003443 learned_addr learned my addr 172.21.15.17:0/774003443 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): get_auth_request con 0x7f16a406eb30 auth_method 0 2024-09-15T02:06:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16a9d38640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a406f440 msgr2=0x7f16a4134220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): get_auth_request con 0x7f16a409c4d0 auth_method 0 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16a9d38640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a406f440 0x7f16a4134220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient(hunting): handle_auth_done global_id 184313 payload 824 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a406f440 msgr2=0x7f16a4134220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a406f440 0x7f16a4134220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a409c4d0 msgr2=0x7f16a411d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a409c4d0 0x7f16a411d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 monclient: found mon.noname-c 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aa539640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0009ae0 con 0x7f16a406eb30 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.634+0000 7f16aad3a640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a409c4d0 0x7f16a411d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16aa539640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 msgr2=0x7f16a4133ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16aa539640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 0x7f16a4133ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009f80 tx=0x7f16a0036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a003b830 con 0x7f16a406eb30 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: dump: 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a003b990 con 0x7f16a406eb30 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a003bd50 con 0x7f16a406eb30 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f169a7fc640 10 monclient: dump: 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.653 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 5 monclient: authenticate success, global_id 184313 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 10 monclient: _renew_subs 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16a8d36640 4 set_mon_vals no callback set 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a4120e00 con 0x7f16a406eb30 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 10 monclient: _renew_subs 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.637+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a4136510 con 0x7f16a406eb30 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16ac79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16ac79d640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 0x7f16a411e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4063b90 con 0x7f16a411df20 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f16a0003020 con 0x7f16a406eb30 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 0x7f16a411e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient: get_auth_request con 0x7f16a411df20 auth_method 0 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f169a7fc640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16880386f0 0x7f168803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f16a002ae40 con 0x7f16a406eb30 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 10 monclient(hunting): handle_auth_done global_id 105564 payload 824 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16a9d38640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a40724a0 con 0x7f16a411df20 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16aad3a640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16880386f0 0x7f168803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.638+0000 7f16aad3a640 10 monclient: get_auth_request con 0x7f16880386f0 auth_method 0 2024-09-15T02:06:58.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f16a9d38640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 msgr2=0x7f16a411e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f16a9d38640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 0x7f16a411e2f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f16940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400ee10 con 0x7f16a411df20 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.640+0000 7f169a7fc640 10 monclient: dump: 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f169400ccd0 con 0x7f16a411df20 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 msgr2=0x7f16a411e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 0x7f16a411e2f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f16940090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 msgr2=0x7f16a411e2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f169a7fc640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411df20 0x7f16a411e2f0 secure :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f16940090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.641+0000 7f16aad3a640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16880386f0 0x7f168803aba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f169c0038f0 tx=0x7f169c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.646+0000 7f16ac79d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.646+0000 7f16ac79d640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 0x7f16a4096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.646+0000 7f16ac79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.646+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4120f90 con 0x7f16a411e830 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 0x7f16a4096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient: get_auth_request con 0x7f16a411e830 auth_method 0 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 10 monclient(hunting): handle_auth_done global_id 105567 payload 824 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.647+0000 7f16a9d38640 1 -- 172.21.15.17:0/774003443 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f16a41214e0 con 0x7f16a411e830 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f16a9d38640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 msgr2=0x7f16a4096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f16a9d38640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 0x7f16a4096a10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f1694019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694005e70 con 0x7f16a411e830 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 10 monclient: dump: 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f169401b800 con 0x7f16a411e830 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 msgr2=0x7f16a4096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 0x7f16a4096a10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f1694019000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 msgr2=0x7f16a4096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.649+0000 7f169a7fc640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16a411e830 0x7f16a4096a10 secure :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f169400e820 tx=0x7f1694019000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 160, 2024-09-15T02:06:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:58.678 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:58.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16880386f0 msgr2=0x7f168803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f16880386f0 0x7f168803aba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f169c0038f0 tx=0x7f169c009060 comp rx=0 tx=0).stop 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 10 monclient: shutdown 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 msgr2=0x7f16a4133ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 --2- 172.21.15.17:0/774003443 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f16a406eb30 0x7f16a4133ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009f80 tx=0x7f16a0036000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16aad3a640 1 -- 172.21.15.17:0/774003443 reap_dead start 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16aad3a640 1 -- 172.21.15.17:0/774003443 reap_dead start 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 shutdown_connections 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 >> 172.21.15.17:0/774003443 conn(0x7f16a40712f0 msgr2=0x7f16a409b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 shutdown_connections 2024-09-15T02:06:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.650+0000 7f16ac79d640 1 -- 172.21.15.17:0/774003443 wait complete. 2024-09-15T02:06:58.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:06:58.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:06:58.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:06:58.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.776+0000 7f028badd640 10 monclient: get_monmap_and_config 2024-09-15T02:06:58.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.776+0000 7f028badd640 10 monclient: build_initial_monmap 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.776+0000 7f028badd640 10 monclient: monmap: 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:06:58.777776+0000 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:06:58.777776+0000 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:06:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:06:58.780 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:06:58.780 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.777+0000 7f028badd640 1 Processor -- start 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.777+0000 7f028badd640 1 -- start start 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.777+0000 7f028badd640 10 monclient: init 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 0x7f028409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): picked mon.noname-c con 0x7f02840a5780 addr v2:172.21.15.67:3300/0 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): picked mon.noname-b con 0x7f028409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 0x7f0284069f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): picked mon.noname-a con 0x7f0284069ac0 addr v2:172.21.15.17:3300/0 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028406a460 con 0x7f0284069ac0 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02840a18b0 con 0x7f02840a5780 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840a5b50 con 0x7f028409ccf0 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028badd640 10 monclient(hunting): authenticate will time out at 1678.774352s 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289879640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 msgr2=0x7f028409bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 0x7f0284069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289879640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 0x7f028409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36450/0 (socket says 172.21.15.17:36450) 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 -- 172.21.15.17:0/3371263256 learned_addr learned my addr 172.21.15.17:0/3371263256 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): get_auth_request con 0x7f028409ccf0 auth_method 0 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): get_auth_request con 0x7f0284069ac0 auth_method 0 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient(hunting): handle_auth_done global_id 184316 payload 274 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 -- 172.21.15.17:0/3371263256 >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 msgr2=0x7f028409bc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 --2- 172.21.15.17:0/3371263256 >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 0x7f028409bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 -- 172.21.15.17:0/3371263256 >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 msgr2=0x7f0284069f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 --2- 172.21.15.17:0/3371263256 >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 0x7f0284069f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 monclient: found mon.noname-b 2024-09-15T02:06:58.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:06:58.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f0289078640 1 -- 172.21.15.17:0/3371263256 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028409c5f0 con 0x7f028409ccf0 2024-09-15T02:06:58.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.778+0000 7f028a07a640 1 --2- 172.21.15.17:0/3371263256 >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 0x7f0284069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:58.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0289078640 1 -- 172.21.15.17:0/3371263256 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 msgr2=0x7f0284069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0289078640 1 --2- 172.21.15.17:0/3371263256 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f028406a5b0 tx=0x7f02780091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97aae5b44b1b256 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 1 -- 172.21.15.17:0/3371263256 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278032440 con 0x7f028409ccf0 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: dump: 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.793 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 1 -- 172.21.15.17:0/3371263256 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02780325a0 con 0x7f028409ccf0 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f028badd640 5 monclient: authenticate success, global_id 184316 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f028badd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f028badd640 10 monclient: get_monmap_and_config success 2024-09-15T02:06:58.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 1 -- 172.21.15.17:0/3371263256 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278032980 con 0x7f028409ccf0 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f028badd640 4 set_mon_vals no callback set 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.781+0000 7f0288877640 10 monclient: dump: 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.795 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 10 monclient: shutdown 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 -- 172.21.15.17:0/3371263256 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 msgr2=0x7f0284069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 --2- 172.21.15.17:0/3371263256 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f028406a5b0 tx=0x7f02780091d0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 -- 172.21.15.17:0/3371263256 shutdown_connections 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 --2- 172.21.15.17:0/3371263256 >> v2:172.21.15.17:3300/0 conn(0x7f0284069ac0 0x7f0284069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 --2- 172.21.15.17:0/3371263256 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284069580 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 --2- 172.21.15.17:0/3371263256 >> v2:172.21.15.67:3300/0 conn(0x7f02840a5780 0x7f028409bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 -- 172.21.15.17:0/3371263256 >> 172.21.15.17:0/3371263256 conn(0x7f0284070e10 msgr2=0x7f02840a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 -- 172.21.15.17:0/3371263256 shutdown_connections 2024-09-15T02:06:58.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 1 -- 172.21.15.17:0/3371263256 wait complete. 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 10 monclient: build_initial_monmap 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.782+0000 7f028badd640 10 monclient: monmap: 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:06:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 Processor -- start 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 -- start start 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient: init 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient: _reopen_session rank -1 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 0x7f028409a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): picked mon.noname-a con 0x7f0284069ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): picked mon.noname-c con 0x7f028409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840a5780 0x7f028411b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): picked mon.noname-b con 0x7f02840a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02841421b0 con 0x7f0284069ac0 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028409b040 con 0x7f02840a5780 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028411dce0 con 0x7f028409ccf0 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): _renew_subs 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028badd640 10 monclient(hunting): authenticate will time out at 1678.779803s 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 0x7f028409a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:36452/0 (socket says 172.21.15.17:36452) 2024-09-15T02:06:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028a07a640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840a5780 msgr2=0x7f028411b6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 1 -- 172.21.15.17:0/907827556 learned_addr learned my addr 172.21.15.17:0/907827556 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 0x7f028409a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:45634/0 (socket says 172.21.15.17:45634) 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 10 monclient(hunting): get_auth_request con 0x7f028409ccf0 auth_method 0 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 10 monclient(hunting): get_auth_request con 0x7f0284069ac0 auth_method 0 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f028a07a640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840a5780 0x7f028411b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.783+0000 7f0289879640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient(hunting): handle_auth_done global_id 105573 payload 824 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 msgr2=0x7f0284130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840a5780 msgr2=0x7f028411b6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840a5780 0x7f028411b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient: _finish_hunting 0 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 monclient: found mon.noname-a 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289879640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c009a20 con 0x7f0284069ac0 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.784+0000 7f0289078640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f028409ccf0 0x7f0284130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0289879640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 msgr2=0x7f028409a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0289879640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 0x7f028409a490 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f028000b5b0 tx=0x7f02800090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280003d90 con 0x7f0284069ac0 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: dump: 2024-09-15T02:06:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: _finish_auth 0 2024-09-15T02:06:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0280006150 con 0x7f0284069ac0 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 5 monclient: authenticate success, global_id 105573 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 10 monclient: _renew_subs 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0278009450 con 0x7f0284069ac0 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280006510 con 0x7f0284069ac0 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f0275ffb640 10 monclient: dump: 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 10 monclient: _renew_subs 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f02777fe640 4 set_mon_vals no callback set 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.786+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028411e320 con 0x7f0284069ac0 2024-09-15T02:06:58.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f02800103e0 con 0x7f0284069ac0 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f028badd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f028badd640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 0x7f028411bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028411cea0 con 0x7f0284141820 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0275ffb640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f02680386f0 0x7f026803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f028004bdf0 con 0x7f0284069ac0 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 0x7f028411bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient: get_auth_request con 0x7f0284141820 auth_method 0 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.787+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.788+0000 7f0289078640 10 monclient(hunting): handle_auth_done global_id 184322 payload 824 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.788+0000 7f028a07a640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f02680386f0 0x7f026803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.788+0000 7f0289078640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f028409b710 con 0x7f0284141820 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.788+0000 7f028a07a640 10 monclient: get_auth_request con 0x7f02680386f0 auth_method 0 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0289078640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 msgr2=0x7f028411bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0289078640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 0x7f028411bc10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0278009420 tx=0x7f0278032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278032440 con 0x7f0284141820 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:58.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 10 monclient: dump: 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f028a07a640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f02680386f0 0x7f026803aba0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f028411cb10 tx=0x7f027c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0278031e50 con 0x7f0284141820 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 msgr2=0x7f028411bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 0x7f028411bc10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0278009420 tx=0x7f0278032000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 msgr2=0x7f028411bc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:58.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.790+0000 7f0275ffb640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284141820 0x7f028411bc10 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0278009420 tx=0x7f0278032000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f028badd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f028badd640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 0x7f028411c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f028badd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028411e4b0 con 0x7f0284131100 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 0x7f028411c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient: get_auth_request con 0x7f0284131100 auth_method 0 2024-09-15T02:06:58.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 10 monclient(hunting): handle_auth_done global_id 184325 payload 824 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.796+0000 7f0289078640 1 -- 172.21.15.17:0/907827556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02840724a0 con 0x7f0284131100 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.798+0000 7f0289078640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 msgr2=0x7f028411c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.798+0000 7f0289078640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 0x7f028411c150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f02840638f0 tx=0x7f027803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278046c60 con 0x7f0284131100 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:06:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 10 monclient: dump: 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f027803b3d0 con 0x7f0284131100 2024-09-15T02:06:58.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 msgr2=0x7f028411c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 0x7f028411c150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f02840638f0 tx=0x7f027803d000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 msgr2=0x7f028411c150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f0275ffb640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0284131100 0x7f028411c150 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f02840638f0 tx=0x7f027803d000 comp rx=0 tx=0).stop 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:58.814 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 160, 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.815 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.816 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:06:58.817 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:06:58.818 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:06:58.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:06:58.822 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:06:58.823 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.824 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.825 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.826 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:06:58.827 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f02680386f0 msgr2=0x7f026803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f02680386f0 0x7f026803aba0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f028411cb10 tx=0x7f027c00b040 comp rx=0 tx=0).stop 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 10 monclient: shutdown 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 msgr2=0x7f028409a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 --2- 172.21.15.17:0/907827556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0284069ac0 0x7f028409a490 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f028000b5b0 tx=0x7f02800090f0 comp rx=0 tx=0).stop 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028a07a640 1 -- 172.21.15.17:0/907827556 reap_dead start 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028a07a640 1 -- 172.21.15.17:0/907827556 reap_dead start 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 shutdown_connections 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 >> 172.21.15.17:0/907827556 conn(0x7f0284070e10 msgr2=0x7f02840a2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 shutdown_connections 2024-09-15T02:06:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:06:58.799+0000 7f028badd640 1 -- 172.21.15.17:0/907827556 wait complete. 2024-09-15T02:06:58.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:06:58.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:07:18.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:07:18.831 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:07:18.831 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:07:18.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:07:18.835 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:07:18.836 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:07:18.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:18.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.942+0000 7f5ef7668640 10 monclient: get_monmap_and_config 2024-09-15T02:07:18.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.942+0000 7f5ef7668640 10 monclient: build_initial_monmap 2024-09-15T02:07:18.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.942+0000 7f5ef7668640 10 monclient: monmap: 2024-09-15T02:07:18.944 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:18.944 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:18.943615+0000 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:18.943615+0000 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:18.945 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:18.946 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:18.946 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:18.946 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.943+0000 7f5ef7668640 1 Processor -- start 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.943+0000 7f5ef7668640 1 -- start start 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.943+0000 7f5ef7668640 10 monclient: init 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 0x7f5ef009a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef009c0d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ef009c4a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef009c870 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00699c0 con 0x7f5ef009c4a0 2024-09-15T02:07:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef006a5e0 con 0x7f5ef009c870 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef006a290 con 0x7f5ef009c0d0 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef7668640 10 monclient(hunting): authenticate will time out at 1698.940223s 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:42088/0 (socket says 172.21.15.17:42088) 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 0x7f5ef009a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 -- 172.21.15.17:0/2134665003 learned_addr learned my addr 172.21.15.17:0/2134665003 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 10 monclient(hunting): get_auth_request con 0x7f5ef009c870 auth_method 0 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): get_auth_request con 0x7f5ef009c0d0 auth_method 0 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 -- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 msgr2=0x7f5ef009e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 -- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 msgr2=0x7f5ef009e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 --2- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5c05640 1 --2- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient(hunting): handle_auth_done global_id 184328 payload 274 2024-09-15T02:07:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 -- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 msgr2=0x7f5ef009e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 --2- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 -- 172.21.15.17:0/2134665003 >> v1:172.21.15.67:6789/0 conn(0x7f5ef009c4a0 legacy=0x7f5ef006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient: _finish_hunting 0 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 monclient: found mon.noname-b 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.944+0000 7f5ef5404640 1 -- 172.21.15.17:0/2134665003 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef009f6e0 con 0x7f5ef009c0d0 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef5404640 1 -- 172.21.15.17:0/2134665003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 msgr2=0x7f5ef009a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:18.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef5404640 1 --2- 172.21.15.17:0/2134665003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 0x7f5ef009a920 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009850 tx=0x7f5eec0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15a382ebed42f411 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:18.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 1 -- 172.21.15.17:0/2134665003 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec02adf0 con 0x7f5ef009c0d0 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: dump: 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:18.958 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 1 -- 172.21.15.17:0/2134665003 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec032440 con 0x7f5ef009c0d0 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef7668640 5 monclient: authenticate success, global_id 184328 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef7668640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 1 -- 172.21.15.17:0/2134665003 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec032820 con 0x7f5ef009c0d0 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef7668640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ef7668640 4 set_mon_vals no callback set 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.947+0000 7f5ee3fff640 10 monclient: dump: 2024-09-15T02:07:18.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 10 monclient: shutdown 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 -- 172.21.15.17:0/2134665003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 msgr2=0x7f5ef009a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:18.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 --2- 172.21.15.17:0/2134665003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 0x7f5ef009a920 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009850 tx=0x7f5eec0090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 -- 172.21.15.17:0/2134665003 shutdown_connections 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 --2- 172.21.15.17:0/2134665003 >> v2:172.21.15.67:3300/0 conn(0x7f5ef009c870 0x7f5ef009e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 --2- 172.21.15.17:0/2134665003 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c0d0 0x7f5ef009a920 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 -- 172.21.15.17:0/2134665003 >> 172.21.15.17:0/2134665003 conn(0x7f5ef00712f0 msgr2=0x7f5ef00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 -- 172.21.15.17:0/2134665003 shutdown_connections 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.948+0000 7f5ef7668640 1 -- 172.21.15.17:0/2134665003 wait complete. 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.949+0000 7f5ef7668640 10 monclient: build_initial_monmap 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.949+0000 7f5ef7668640 10 monclient: monmap: 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:18.961 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.949+0000 7f5ef7668640 1 Processor -- start 2024-09-15T02:07:18.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.949+0000 7f5ef7668640 1 -- start start 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.949+0000 7f5ef7668640 10 monclient: init 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 0x7f5ef01413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef009c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef009c4a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef009c870 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:18.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01422e0 con 0x7f5ef009c0d0 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef009cd80 con 0x7f5ef009c4a0 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00a2a70 con 0x7f5ef009c870 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef7668640 10 monclient(hunting): authenticate will time out at 1698.946453s 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 0x7f5ef01413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 0x7f5ef01413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34008/0 (socket says 172.21.15.17:34008) 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 1 -- 172.21.15.17:0/1700381977 learned_addr learned my addr 172.21.15.17:0/1700381977 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): get_auth_request con 0x7f5ef009c0d0 auth_method 0 2024-09-15T02:07:18.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44480/0 (socket says 172.21.15.17:44480) 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): get_auth_request con 0x7f5ef009c870 auth_method 0 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 10 monclient(hunting): get_auth_request con 0x7f5ef009c4a0 auth_method 0 2024-09-15T02:07:18.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 msgr2=0x7f5ef01340b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 msgr2=0x7f5ef01340b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef4c03640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:18.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient(hunting): handle_auth_done global_id 184331 payload 824 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 msgr2=0x7f5ef01340b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 msgr2=0x7f5ef01413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 0x7f5ef01413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient: _finish_hunting 0 2024-09-15T02:07:18.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 monclient: found mon.noname-c 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.950+0000 7f5ef5c05640 1 -- 172.21.15.17:0/1700381977 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec009390 con 0x7f5ef009c870 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef5c05640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 msgr2=0x7f5ef01345f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef5c05640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8007b60 tx=0x7f5ee8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:18.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800dc60 con 0x7f5ef009c870 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: dump: 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.969 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: _finish_auth 0 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:18.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 5 monclient: authenticate success, global_id 184331 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee800f040 con 0x7f5ef009c870 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 10 monclient: _renew_subs 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef011f750 con 0x7f5ef009c870 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8015400 con 0x7f5ef009c870 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:18.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee17fa640 10 monclient: dump: 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 10 monclient: _renew_subs 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:18.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ee37fe640 4 set_mon_vals no callback set 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.953+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef0121280 con 0x7f5ef009c870 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ee8003020 con 0x7f5ef009c870 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef7668640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef7668640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef7668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef00724a0 con 0x7f5ef009c870 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ee17fa640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 0x7f5ed403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f5ee804c430 con 0x7f5ef009c870 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef5404640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 0x7f5ed403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:18.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.954+0000 7f5ef5404640 10 monclient: get_auth_request con 0x7f5ed4038790 auth_method 0 2024-09-15T02:07:18.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.957+0000 7f5ef5404640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 0x7f5ed403ac40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009130 tx=0x7f5eec032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:18.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.959+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ee8013ae0 con 0x7f5ef009c870 2024-09-15T02:07:18.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.959+0000 7f5ee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:18.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:18.959+0000 7f5ee17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ef7668640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ef7668640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ef7668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5ef009eb40 con 0x7f5ef009c870 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ee17fa640 1 -- 172.21.15.17:0/1700381977 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f5ee8019070 con 0x7f5ef009c870 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:19.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.184+0000 7f5ee17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:19.187 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:19.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 msgr2=0x7f5ed403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 0x7f5ed403ac40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009130 tx=0x7f5eec032000 comp rx=0 tx=0).stop 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 10 monclient: shutdown 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 msgr2=0x7f5ef01345f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.186+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8007b60 tx=0x7f5ee8009250 comp rx=0 tx=0).stop 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 shutdown_connections 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5ed4038790 0x7f5ed403ac40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:19.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ef009c870 0x7f5ef01345f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:19.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef009c4a0 0x7f5ef01340b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:19.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 --2- 172.21.15.17:0/1700381977 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ef009c0d0 0x7f5ef01413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:19.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 >> 172.21.15.17:0/1700381977 conn(0x7f5ef00712f0 msgr2=0x7f5ef00a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:19.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 shutdown_connections 2024-09-15T02:07:19.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:19.187+0000 7f5ef7668640 1 -- 172.21.15.17:0/1700381977 wait complete. 2024-09-15T02:07:22.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:22.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 10 monclient: get_monmap_and_config 2024-09-15T02:07:22.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 10 monclient: build_initial_monmap 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 10 monclient: monmap: 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:22.307938+0000 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:22.307938+0000 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:22.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 1 Processor -- start 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 1 -- start start 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.307+0000 7f7df404f640 10 monclient: init 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T02:07:22.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7dec0a58b0 0x7f7dec132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec0a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 0x7f7dec0984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec0980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec0989e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0a01e0 con 0x7f7dec0a58b0 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0a1580 con 0x7f7dec0989e0 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0a5c80 con 0x7f7dec0980a0 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:22.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df404f640 10 monclient(hunting): authenticate will time out at 1702.304609s 2024-09-15T02:07:22.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7dec0a58b0 0x7f7dec132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 0x7f7dec0984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f7dec0a58b0 0x7f7dec132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34012/0 (socket says 172.21.15.17:34012) 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 0x7f7dec0984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44484/0 (socket says 172.21.15.17:44484) 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 1 -- 172.21.15.17:0/1827552523 learned_addr learned my addr 172.21.15.17:0/1827552523 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): get_auth_request con 0x7f7dec0a58b0 auth_method 0 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): get_auth_request con 0x7f7dec0980a0 auth_method 0 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df25ec640 1 --2- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df25ec640 10 monclient(hunting): get_auth_request con 0x7f7dec0989e0 auth_method 0 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df25ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df25ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.308+0000 7f7df25ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 10 monclient(hunting): handle_auth_done global_id 105579 payload 274 2024-09-15T02:07:22.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 -- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 msgr2=0x7f7dec0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 --2- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 0x7f7dec0984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df25ec640 1 -- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 msgr2=0x7f7dec068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 -- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 msgr2=0x7f7dec068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df25ec640 1 -- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 msgr2=0x7f7dec068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df25ec640 1 --2- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df25ec640 1 --2- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 --2- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 10 monclient: _finish_hunting 0 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 monclient: found mon.noname-a 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.309+0000 7f7df1deb640 1 -- 172.21.15.17:0/1827552523 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec1326c0 con 0x7f7dec0a58b0 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.311+0000 7f7df1deb640 1 -- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0a58b0 msgr2=0x7f7dec132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df1deb640 1 --2- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0a58b0 0x7f7dec132040 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009910 tx=0x7f7de80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5336e80f2a4c39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 1 -- 172.21.15.17:0/1827552523 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8032440 con 0x7f7dec0a58b0 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: dump: 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.324 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: _finish_auth 0 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 1 -- 172.21.15.17:0/1827552523 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de80325a0 con 0x7f7dec0a58b0 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 1 -- 172.21.15.17:0/1827552523 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8032940 con 0x7f7dec0a58b0 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 5 monclient: authenticate success, global_id 105579 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 4 set_mon_vals no callback set 2024-09-15T02:07:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df0de9640 10 monclient: dump: 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 10 monclient: shutdown 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 -- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0a58b0 msgr2=0x7f7dec132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 --2- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0a58b0 0x7f7dec132040 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009910 tx=0x7f7de80091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 -- 172.21.15.17:0/1827552523 shutdown_connections 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 --2- 172.21.15.17:0/1827552523 >> v2:172.21.15.67:3300/0 conn(0x7f7dec0989e0 0x7f7dec068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 --2- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0980a0 0x7f7dec0984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 --2- 172.21.15.17:0/1827552523 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0a58b0 0x7f7dec132040 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 -- 172.21.15.17:0/1827552523 >> 172.21.15.17:0/1827552523 conn(0x7f7dec070ed0 msgr2=0x7f7dec0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.312+0000 7f7df404f640 1 -- 172.21.15.17:0/1827552523 shutdown_connections 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 1 -- 172.21.15.17:0/1827552523 wait complete. 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 10 monclient: build_initial_monmap 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 10 monclient: monmap: 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:22.327 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 1 Processor -- start 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 1 -- start start 2024-09-15T02:07:22.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.313+0000 7f7df404f640 10 monclient: init 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec0980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 0x7f7dec11dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec11e150 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec11eae0 con 0x7f7dec0980a0 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec098dc0 con 0x7f7dec11e150 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec09eb40 con 0x7f7dec0a58b0 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df404f640 10 monclient(hunting): authenticate will time out at 1702.310462s 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 0x7f7dec11dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 0x7f7dec11dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44494/0 (socket says 172.21.15.17:44494) 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34014/0 (socket says 172.21.15.17:34014) 2024-09-15T02:07:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 1 -- 172.21.15.17:0/984729130 learned_addr learned my addr 172.21.15.17:0/984729130 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): get_auth_request con 0x7f7dec0a58b0 auth_method 0 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): get_auth_request con 0x7f7dec0980a0 auth_method 0 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:22.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 10 monclient(hunting): get_auth_request con 0x7f7dec11e150 auth_method 0 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df15ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 msgr2=0x7f7dec11e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 msgr2=0x7f7dec11e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient(hunting): handle_auth_done global_id 105582 payload 824 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df25ec640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:22.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 msgr2=0x7f7dec11dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 0x7f7dec11dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 msgr2=0x7f7dec11e5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient: _finish_hunting 0 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 monclient: found mon.noname-a 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.314+0000 7f7df1deb640 1 -- 172.21.15.17:0/984729130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de8009450 con 0x7f7dec0980a0 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df1deb640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 msgr2=0x7f7dec1312b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df1deb640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7de8031000 tx=0x7f7de8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8031a40 con 0x7f7dec0980a0 2024-09-15T02:07:22.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: dump: 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:22.334 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: _finish_auth 0 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 5 monclient: authenticate success, global_id 105582 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de8031bc0 con 0x7f7dec0980a0 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 10 monclient: _renew_subs 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec121220 con 0x7f7dec0980a0 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de803d440 con 0x7f7dec0980a0 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:22.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7dddffb640 10 monclient: dump: 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7ddffff640 4 set_mon_vals no callback set 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 10 monclient: _renew_subs 2024-09-15T02:07:22.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.317+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec11f760 con 0x7f7dec0980a0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7de8039040 con 0x7f7dec0980a0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df404f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df404f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df404f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec0724a0 con 0x7f7dec0980a0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7dddffb640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 0x7f7dd003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f7de806b520 con 0x7f7dec0980a0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df15ea640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 0x7f7dd003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.318+0000 7f7df15ea640 10 monclient: get_auth_request con 0x7f7dd0038740 auth_method 0 2024-09-15T02:07:22.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.321+0000 7f7df15ea640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 0x7f7dd003abf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009d40 tx=0x7f7de00090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:22.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.323+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7de8038830 con 0x7f7dec0980a0 2024-09-15T02:07:22.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.323+0000 7f7dddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:22.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.323+0000 7f7dddffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7df404f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7df404f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7df404f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7dec09fc90 con 0x7f7dec0980a0 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7dddffb640 1 -- 172.21.15.17:0/984729130 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f7de8037080 con 0x7f7dec0980a0 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7dddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.551+0000 7f7dddffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:22.553 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:22.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 msgr2=0x7f7dd003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:22.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 0x7f7dd003abf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009d40 tx=0x7f7de00090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 10 monclient: shutdown 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 msgr2=0x7f7dec1312b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7de8031000 tx=0x7f7de8032000 comp rx=0 tx=0).stop 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 shutdown_connections 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7dd0038740 0x7f7dd003abf0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dec11e150 0x7f7dec11e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7dec0a58b0 0x7f7dec11dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 --2- 172.21.15.17:0/984729130 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7dec0980a0 0x7f7dec1312b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 >> 172.21.15.17:0/984729130 conn(0x7f7dec070ed0 msgr2=0x7f7dec09e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.553+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 shutdown_connections 2024-09-15T02:07:22.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:22.554+0000 7f7df404f640 1 -- 172.21.15.17:0/984729130 wait complete. 2024-09-15T02:07:25.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:25.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.673+0000 7fecf869a640 10 monclient: get_monmap_and_config 2024-09-15T02:07:25.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.673+0000 7fecf869a640 10 monclient: build_initial_monmap 2024-09-15T02:07:25.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.673+0000 7fecf869a640 10 monclient: monmap: 2024-09-15T02:07:25.675 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:25.674452+0000 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:25.674452+0000 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 Processor -- start 2024-09-15T02:07:25.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 -- start start 2024-09-15T02:07:25.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient: init 2024-09-15T02:07:25.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecf006e920 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fecf006ecf0 0x7fecf0134a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-a con 0x7fecf006ecf0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf006f110 0x7fecf0136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-c con 0x7fecf006f110 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf009c860 con 0x7fecf006e920 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf009d090 con 0x7fecf006ecf0 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.674+0000 7fecf869a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf009d4b0 con 0x7fecf006f110 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf869a640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:25.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf869a640 10 monclient(hunting): authenticate will time out at 1705.671009s 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fecf006ecf0 0x7fecf0134a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fecf006ecf0 0x7fecf0134a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49224/0 (socket says 172.21.15.17:49224) 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 -- 172.21.15.17:0/1135230788 learned_addr learned my addr 172.21.15.17:0/1135230788 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): get_auth_request con 0x7fecf006ecf0 auth_method 0 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:25.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf6c37640 1 --2- 172.21.15.17:0/1135230788 >> v2:172.21.15.67:3300/0 conn(0x7fecf006f110 0x7fecf0136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf6c37640 10 monclient(hunting): get_auth_request con 0x7fecf006f110 auth_method 0 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf6c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf6c37640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf6c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient(hunting): handle_auth_done global_id 105585 payload 274 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 -- 172.21.15.17:0/1135230788 >> v2:172.21.15.67:3300/0 conn(0x7fecf006f110 msgr2=0x7fecf0136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 --2- 172.21.15.17:0/1135230788 >> v2:172.21.15.67:3300/0 conn(0x7fecf006f110 0x7fecf0136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 -- 172.21.15.17:0/1135230788 >> v1:172.21.15.67:6789/0 conn(0x7fecf006e920 legacy=0x7fecf012fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:25.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient: _finish_hunting 0 2024-09-15T02:07:25.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 monclient: found mon.noname-a 2024-09-15T02:07:25.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:25.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.675+0000 7fecf5c35640 1 -- 172.21.15.17:0/1135230788 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf009cad0 con 0x7fecf006ecf0 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5c35640 1 -- 172.21.15.17:0/1135230788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 msgr2=0x7fecf0134a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5c35640 1 --2- 172.21.15.17:0/1135230788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf0134a50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fecf009ac60 tx=0x7fece00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fffedaf569c37c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 1 -- 172.21.15.17:0/1135230788 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0032440 con 0x7fecf006ecf0 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: dump: 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:25.687 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: _finish_auth 0 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 1 -- 172.21.15.17:0/1135230788 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece00325a0 con 0x7fecf006ecf0 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 1 -- 172.21.15.17:0/1135230788 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0032980 con 0x7fecf006ecf0 2024-09-15T02:07:25.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf869a640 5 monclient: authenticate success, global_id 105585 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf869a640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf869a640 4 set_mon_vals no callback set 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.678+0000 7fecf5434640 10 monclient: dump: 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:25.689 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 10 monclient: shutdown 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 -- 172.21.15.17:0/1135230788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 msgr2=0x7fecf0134a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 --2- 172.21.15.17:0/1135230788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf0134a50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fecf009ac60 tx=0x7fece00091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 -- 172.21.15.17:0/1135230788 shutdown_connections 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 --2- 172.21.15.17:0/1135230788 >> v2:172.21.15.67:3300/0 conn(0x7fecf006f110 0x7fecf0136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 --2- 172.21.15.17:0/1135230788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf0134a50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 -- 172.21.15.17:0/1135230788 >> 172.21.15.17:0/1135230788 conn(0x7fecf00a1000 msgr2=0x7fecf00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 -- 172.21.15.17:0/1135230788 shutdown_connections 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 1 -- 172.21.15.17:0/1135230788 wait complete. 2024-09-15T02:07:25.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 10 monclient: build_initial_monmap 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.679+0000 7fecf869a640 10 monclient: monmap: 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 Processor -- start 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 -- start start 2024-09-15T02:07:25.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient: init 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 0x7fecf012f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-b con 0x7fecf006e920 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-a con 0x7fecf006ecf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 0x7fecf00988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): picked mon.noname-c con 0x7fecf006f110 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006db90 con 0x7fecf006ecf0 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf011ecc0 con 0x7fecf006e920 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0099e20 con 0x7fecf006f110 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf869a640 10 monclient(hunting): authenticate will time out at 1705.676573s 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49226/0 (socket says 172.21.15.17:49226) 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 0x7fecf00988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 1 -- 172.21.15.17:0/1619174645 learned_addr learned my addr 172.21.15.17:0/1619174645 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): get_auth_request con 0x7fecf006ecf0 auth_method 0 2024-09-15T02:07:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): get_auth_request con 0x7fecf006f110 auth_method 0 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf5c35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:25.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6c37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6436640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 0x7fecf012f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6436640 10 monclient(hunting): get_auth_request con 0x7fecf006e920 auth_method 0 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6436640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.680+0000 7fecf6436640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 10 monclient(hunting): handle_auth_done global_id 105588 payload 824 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 -- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 msgr2=0x7fecf00988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 0x7fecf00988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 -- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 msgr2=0x7fecf012f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 0x7fecf012f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 10 monclient: _finish_hunting 0 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 monclient: found mon.noname-a 2024-09-15T02:07:25.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf5c35640 1 -- 172.21.15.17:0/1619174645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0009450 con 0x7fecf006ecf0 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf6c37640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 0x7fecf00988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.681+0000 7fecf6436640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 0x7fecf012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf5c35640 1 -- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 msgr2=0x7fecf00983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf5c35640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fece0035040 tx=0x7fece0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0031a40 con 0x7fecf006ecf0 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: dump: 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:25.696 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: _finish_auth 0 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece0031be0 con 0x7fecf006ecf0 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 5 monclient: authenticate success, global_id 105588 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 10 monclient: _renew_subs 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf011e000 con 0x7fecf006ecf0 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece003d440 con 0x7fecf006ecf0 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fece67fc640 10 monclient: dump: 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:25.698 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf4c33640 4 set_mon_vals no callback set 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 10 monclient: _renew_subs 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.683+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf012e500 con 0x7fecf006ecf0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fece0039040 con 0x7fecf006ecf0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fece67fc640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 0x7fecd403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fecf869a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fecf869a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fecf869a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fece006c530 con 0x7fecf006ecf0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf00724a0 con 0x7fecf006ecf0 2024-09-15T02:07:25.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.684+0000 7fecf6436640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 0x7fecd403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:25.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.685+0000 7fecf6436640 10 monclient: get_auth_request con 0x7fecd4038740 auth_method 0 2024-09-15T02:07:25.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.687+0000 7fecf6436640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 0x7fecd403abf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fecec002860 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:25.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.689+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fece0042070 con 0x7fecf006ecf0 2024-09-15T02:07:25.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.689+0000 7fece67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:25.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.689+0000 7fece67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:25.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fecf869a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:25.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fecf869a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:25.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fecf869a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:25.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fecf0067290 con 0x7fecf006ecf0 2024-09-15T02:07:25.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fece67fc640 1 -- 172.21.15.17:0/1619174645 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fece0037030 con 0x7fecf006ecf0 2024-09-15T02:07:25.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fece67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:25.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.920+0000 7fece67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:25.923 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 msgr2=0x7fecd403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 0x7fecd403abf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fecec002860 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 10 monclient: shutdown 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 msgr2=0x7fecf00983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fece0035040 tx=0x7fece0032000 comp rx=0 tx=0).stop 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 shutdown_connections 2024-09-15T02:07:25.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fecd4038740 0x7fecd403abf0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecf006ecf0 0x7fecf00983b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecf006f110 0x7fecf00988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 --2- 172.21.15.17:0/1619174645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf006e920 0x7fecf012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 >> 172.21.15.17:0/1619174645 conn(0x7fecf00a1000 msgr2=0x7fecf01329d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 shutdown_connections 2024-09-15T02:07:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:25.922+0000 7fecf869a640 1 -- 172.21.15.17:0/1619174645 wait complete. 2024-09-15T02:07:25.935 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:07:25.935 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:07:25.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.030+0000 7f6426cc0640 10 monclient: get_monmap_and_config 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.030+0000 7f6426cc0640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.030+0000 7f6426cc0640 10 monclient: monmap: 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:26.031445+0000 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:26.031445+0000 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:26.033 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 Processor -- start 2024-09-15T02:07:26.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 -- start start 2024-09-15T02:07:26.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient: init 2024-09-15T02:07:26.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 0x7f642009c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f642009c520 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f642006ee50 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f642006f250 0x7f642009e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f642006f250 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642009b4e0 con 0x7f642006ee50 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642009ce30 con 0x7f642009c520 2024-09-15T02:07:26.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642009b760 con 0x7f642006f250 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.031+0000 7f6426cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6426cc0640 10 monclient(hunting): authenticate will time out at 1706.028008s 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 0x7f642009c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 0x7f642009c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49232/0 (socket says 172.21.15.17:49232) 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 1 -- 172.21.15.17:0/4221697094 learned_addr learned my addr 172.21.15.17:0/4221697094 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): get_auth_request con 0x7f642009c520 auth_method 0 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 --2- 172.21.15.17:0/4221697094 >> v2:172.21.15.67:3300/0 conn(0x7f642006f250 0x7f642009e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): get_auth_request con 0x7f642006f250 auth_method 0 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient(hunting): handle_auth_done global_id 204106 payload 274 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 -- 172.21.15.17:0/4221697094 >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 msgr2=0x7f642009c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 --2- 172.21.15.17:0/4221697094 >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 0x7f642009c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 -- 172.21.15.17:0/4221697094 >> v1:172.21.15.17:6789/0 conn(0x7f642006ee50 legacy=0x7f642009d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 monclient: found mon.noname-c 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:07:26.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.032+0000 7f642525d640 1 -- 172.21.15.17:0/4221697094 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642006f690 con 0x7f642006f250 2024-09-15T02:07:26.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f642525d640 1 -- 172.21.15.17:0/4221697094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642006f250 msgr2=0x7f642009e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f642525d640 1 --2- 172.21.15.17:0/4221697094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642006f250 0x7f642009e070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642006ed40 tx=0x7f6418009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a80f8478856b4c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 1 -- 172.21.15.17:0/4221697094 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418007760 con 0x7f642006f250 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: dump: 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.048 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 1 -- 172.21.15.17:0/4221697094 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641802d050 con 0x7f642006f250 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f6426cc0640 5 monclient: authenticate success, global_id 204106 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f6426cc0640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:26.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 1 -- 172.21.15.17:0/4221697094 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418007b30 con 0x7f642006f250 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f6426cc0640 4 set_mon_vals no callback set 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.035+0000 7f641f7fe640 10 monclient: dump: 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.050 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 10 monclient: shutdown 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 -- 172.21.15.17:0/4221697094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642006f250 msgr2=0x7f642009e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 --2- 172.21.15.17:0/4221697094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642006f250 0x7f642009e070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642006ed40 tx=0x7f6418009780 comp rx=0 tx=0).stop 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 -- 172.21.15.17:0/4221697094 shutdown_connections 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 --2- 172.21.15.17:0/4221697094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642006f250 0x7f642009e070 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 --2- 172.21.15.17:0/4221697094 >> v2:172.21.15.17:3300/0 conn(0x7f642009c520 0x7f642009c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 -- 172.21.15.17:0/4221697094 >> 172.21.15.17:0/4221697094 conn(0x7f64200a2f70 msgr2=0x7f64200a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 -- 172.21.15.17:0/4221697094 shutdown_connections 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 1 -- 172.21.15.17:0/4221697094 wait complete. 2024-09-15T02:07:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.036+0000 7f6426cc0640 10 monclient: monmap: 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 1 Processor -- start 2024-09-15T02:07:26.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 1 -- start start 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 10 monclient: init 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 0x7f642013eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f642006ee50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.037+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f642006f250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642009c520 0x7f642011c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f642009c520 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642011e590 con 0x7f642006f250 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642009cf90 con 0x7f642009c520 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420098b00 con 0x7f642006ee50 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6426cc0640 10 monclient(hunting): authenticate will time out at 1706.034059s 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 0x7f642013eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 0x7f642013eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40898/0 (socket says 172.21.15.17:40898) 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49246/0 (socket says 172.21.15.17:49246) 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 1 -- 172.21.15.17:0/3409400114 learned_addr learned my addr 172.21.15.17:0/3409400114 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): get_auth_request con 0x7f642006ee50 auth_method 0 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): get_auth_request con 0x7f642006f250 auth_method 0 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642009c520 0x7f642011c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 10 monclient(hunting): get_auth_request con 0x7f642009c520 auth_method 0 2024-09-15T02:07:26.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient(hunting): handle_auth_done global_id 105594 payload 824 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 msgr2=0x7f642013eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 0x7f642013eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642009c520 msgr2=0x7f642011c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642009c520 0x7f642011c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 monclient: found mon.noname-a 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f641ffff640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418009a20 con 0x7f642006f250 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f6424a5c640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f642006ee50 0x7f642013eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.038+0000 7f642525d640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642009c520 0x7f642011c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641ffff640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 msgr2=0x7f6420132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641ffff640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f640c00b490 tx=0x7f640c006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00f720 con 0x7f642006f250 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: dump: 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.057 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00f880 con 0x7f642006f250 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f6426cc0640 5 monclient: authenticate success, global_id 105594 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f6426cc0640 10 monclient: _renew_subs 2024-09-15T02:07:26.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f6426cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642013f9f0 con 0x7f642006f250 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00fc40 con 0x7f642006f250 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641cff9640 10 monclient: dump: 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.059 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.040+0000 7f641effd640 4 set_mon_vals no callback set 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 10 monclient: _renew_subs 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642011fb80 con 0x7f642006f250 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 0x7f642011caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420063b90 con 0x7f6420132fa0 2024-09-15T02:07:26.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f641ffff640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f640c00b490 tx=0x7f640c006f50 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f640c04bb40 osd_map(257..257 src has 229..257) v4 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 0x7f642011caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient: get_auth_request con 0x7f6420132fa0 auth_method 0 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f640c011610 con 0x7f642006f250 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.041+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f6424a5c640 10 monclient(hunting): handle_auth_done global_id 105597 payload 824 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f641cff9640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6404038740 0x7f640403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f6424a5c640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64200724a0 con 0x7f6420132fa0 2024-09-15T02:07:26.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f642525d640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6404038740 0x7f640403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.042+0000 7f642525d640 10 monclient: get_auth_request con 0x7f6404038740 auth_method 0 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f6424a5c640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 msgr2=0x7f642011caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f6424a5c640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 0x7f642011caf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f64140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400fed0 con 0x7f6420132fa0 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 10 monclient: dump: 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.062 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6414014860 con 0x7f6420132fa0 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 msgr2=0x7f642011caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 0x7f642011caf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f64140091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 msgr2=0x7f642011caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.044+0000 7f641cff9640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6420132fa0 0x7f642011caf0 secure :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f64140091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.045+0000 7f642525d640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6404038740 0x7f640403abf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6418034000 tx=0x7f6418037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6426cc0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6426cc0640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 0x7f6420097210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6426cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420133370 con 0x7f642011d030 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 0x7f6420097210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient: get_auth_request con 0x7f642011d030 auth_method 0 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 10 monclient(hunting): handle_auth_done global_id 105600 payload 824 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.050+0000 7f6424a5c640 1 -- 172.21.15.17:0/3409400114 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f642011d430 con 0x7f642011d030 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.053+0000 7f6424a5c640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 msgr2=0x7f6420097210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.053+0000 7f6424a5c640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 0x7f6420097210 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f641401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414024040 con 0x7f642011d030 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 10 monclient: dump: 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.067 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f64140133a0 con 0x7f642011d030 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 msgr2=0x7f6420097210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 0x7f6420097210 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f641401c000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 msgr2=0x7f6420097210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642011d030 0x7f6420097210 secure :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6414009fa0 tx=0x7f641401c000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:26.083 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:26.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6404038740 msgr2=0x7f640403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6404038740 0x7f640403abf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6418034000 tx=0x7f6418037000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 10 monclient: shutdown 2024-09-15T02:07:26.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 msgr2=0x7f6420132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 --2- 172.21.15.17:0/3409400114 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f642006f250 0x7f6420132a60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f640c00b490 tx=0x7f640c006f50 comp rx=0 tx=0).stop 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f642525d640 1 -- 172.21.15.17:0/3409400114 reap_dead start 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f642525d640 1 -- 172.21.15.17:0/3409400114 reap_dead start 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f641cff9640 1 -- 172.21.15.17:0/3409400114 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f640c04bb40 con 0x7f642006f250 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 shutdown_connections 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 >> 172.21.15.17:0/3409400114 conn(0x7f64200a2f70 msgr2=0x7f64200a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 shutdown_connections 2024-09-15T02:07:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.055+0000 7f6426cc0640 1 -- 172.21.15.17:0/3409400114 wait complete. 2024-09-15T02:07:26.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:26.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:07:26.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.181+0000 7f1346edd640 10 monclient: get_monmap_and_config 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.181+0000 7f1346edd640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.181+0000 7f1346edd640 10 monclient: monmap: 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:26.182455+0000 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:26.182455+0000 2024-09-15T02:07:26.183 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:26.184 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 1 Processor -- start 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 1 -- start start 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 10 monclient: init 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.182+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1340067090 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1340067460 0x7f1340067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-c con 0x7f1340067460 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1340136f90 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134009cbc0 con 0x7f1340067090 2024-09-15T02:07:26.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134009e550 con 0x7f1340136f90 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13401328b0 con 0x7f1340067460 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1346edd640 10 monclient(hunting): authenticate will time out at 1706.179165s 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1344c79640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1340067090 0x7f13401321a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58850/0 (socket says 172.21.15.17:58850) 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f1344c79640 1 -- 172.21.15.17:0/396528941 learned_addr learned my addr 172.21.15.17:0/396528941 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 1 --2- 172.21.15.17:0/396528941 >> v2:172.21.15.67:3300/0 conn(0x7f1340067460 0x7f1340067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): get_auth_request con 0x7f1340067460 auth_method 0 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.183+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f1344c79640 1 -- 172.21.15.17:0/396528941 >> v1:172.21.15.17:6789/0 conn(0x7f1340067090 legacy=0x7f13401321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254810875 0 0) 0x7f134009cbc0 con 0x7f1340067090 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 10 monclient(hunting): my global_id is 105603 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13300034e0 con 0x7f1340067090 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f134547a640 1 -- 172.21.15.17:0/396528941 >> v1:172.21.15.67:6789/0 conn(0x7f1340136f90 legacy=0x7f1340139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1048640954 0 0) 0x7f13300034e0 con 0x7f1340067090 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 10 monclient(hunting): handle_auth_done global_id 204112 payload 274 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 1 -- 172.21.15.17:0/396528941 >> v1:172.21.15.67:6789/0 conn(0x7f1340136f90 legacy=0x7f1340139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 1 -- 172.21.15.17:0/396528941 >> v1:172.21.15.17:6789/0 conn(0x7f1340067090 legacy=0x7f13401321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 1 monclient: found mon.noname-c 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133ffff640 1 -- 172.21.15.17:0/396528941 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134009e1e0 con 0x7f1340067460 2024-09-15T02:07:26.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.185+0000 7f133f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133ffff640 1 -- 172.21.15.17:0/396528941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067460 msgr2=0x7f1340067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133ffff640 1 --2- 172.21.15.17:0/396528941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067460 0x7f1340067830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f132c009950 tx=0x7f132c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120b7cc1182926f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334001720 con 0x7f1340067460 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: dump: 2024-09-15T02:07:26.194 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c02acd0 con 0x7f1340067460 2024-09-15T02:07:26.195 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f1346edd640 5 monclient: authenticate success, global_id 204112 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f1346edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f1346edd640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 1 -- 172.21.15.17:0/396528941 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c032670 con 0x7f1340067460 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f1346edd640 4 set_mon_vals no callback set 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.188+0000 7f133f7fe640 10 monclient: dump: 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.196 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 10 monclient: shutdown 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 -- 172.21.15.17:0/396528941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067460 msgr2=0x7f1340067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 --2- 172.21.15.17:0/396528941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067460 0x7f1340067830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f132c009950 tx=0x7f132c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 -- 172.21.15.17:0/396528941 shutdown_connections 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 --2- 172.21.15.17:0/396528941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067460 0x7f1340067830 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 -- 172.21.15.17:0/396528941 >> 172.21.15.17:0/396528941 conn(0x7f13400a1ee0 msgr2=0x7f13400a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.197 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 -- 172.21.15.17:0/396528941 shutdown_connections 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 1 -- 172.21.15.17:0/396528941 wait complete. 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.189+0000 7f1346edd640 10 monclient: monmap: 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:26.198 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 Processor -- start 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 -- start start 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient: init 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067090 0x7f1340118770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-b con 0x7f1340067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 0x7f1340118cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-a con 0x7f1340067460 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340136f90 0x7f13401191f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): picked mon.noname-c con 0x7f1340136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.199 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134011ba10 con 0x7f1340067460 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134011a460 con 0x7f1340067090 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1340119f20 con 0x7f1340136f90 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.190+0000 7f1346edd640 10 monclient(hunting): authenticate will time out at 1706.186953s 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 0x7f1340118cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 0x7f1340118cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49276/0 (socket says 172.21.15.17:49276) 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 -- 172.21.15.17:0/861336650 learned_addr learned my addr 172.21.15.17:0/861336650 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): get_auth_request con 0x7f1340067460 auth_method 0 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.200 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340136f90 0x7f13401191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067090 0x7f1340118770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): get_auth_request con 0x7f1340136f90 auth_method 0 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): get_auth_request con 0x7f1340067090 auth_method 0 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.201 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient(hunting): handle_auth_done global_id 105606 payload 824 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340136f90 msgr2=0x7f13401191f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340136f90 0x7f13401191f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067090 msgr2=0x7f1340118770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067090 0x7f1340118770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.202 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 monclient: found mon.noname-a 2024-09-15T02:07:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f133ffff640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c009390 con 0x7f1340067460 2024-09-15T02:07:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f134547a640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340136f90 0x7f13401191f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:26.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.191+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340067090 0x7f1340118770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:26.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133ffff640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 msgr2=0x7f1340118cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133ffff640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 0x7f1340118cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f132c009950 tx=0x7f132c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c03b820 con 0x7f1340067460 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: dump: 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.207 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c03b9a0 con 0x7f1340067460 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f1346edd640 5 monclient: authenticate success, global_id 105606 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c03bd60 con 0x7f1340067460 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133cff9640 10 monclient: dump: 2024-09-15T02:07:26.208 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.193+0000 7f133effd640 4 set_mon_vals no callback set 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 monclient: _renew_subs 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134009d530 con 0x7f1340067460 2024-09-15T02:07:26.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 monclient: _renew_subs 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134011b700 con 0x7f1340067460 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 0x7f1340132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1340063870 con 0x7f1340119730 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.194+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f132c039040 con 0x7f1340067460 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 0x7f1340132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient: get_auth_request con 0x7f1340119730 auth_method 0 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f133cff9640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1324038770 0x7f132403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f132c02ae40 con 0x7f1340067460 2024-09-15T02:07:26.211 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 10 monclient(hunting): handle_auth_done global_id 184349 payload 824 2024-09-15T02:07:26.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f1344c79640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13400724a0 con 0x7f1340119730 2024-09-15T02:07:26.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f134547a640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1324038770 0x7f132403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.195+0000 7f134547a640 10 monclient: get_auth_request con 0x7f1324038770 auth_method 0 2024-09-15T02:07:26.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f1344c79640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 msgr2=0x7f1340132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 0x7f1340132ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1334001550 tx=0x7f1334001e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334012560 con 0x7f1340119730 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 10 monclient: dump: 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.213 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f13340126c0 con 0x7f1340119730 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 msgr2=0x7f1340132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 0x7f1340132ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1334001550 tx=0x7f1334001e90 comp rx=0 tx=0).stop 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 msgr2=0x7f1340132ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.197+0000 7f133cff9640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340119730 0x7f1340132ad0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1334001550 tx=0x7f1334001e90 comp rx=0 tx=0).stop 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.198+0000 7f134547a640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1324038770 0x7f132403ac20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f134011a310 tx=0x7f1338009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1346edd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1346edd640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 0x7f1340133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1346edd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.214 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13400689a0 con 0x7f1340133010 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 0x7f1340133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient: get_auth_request con 0x7f1340133010 auth_method 0 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.203+0000 7f1344c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.204+0000 7f1344c79640 10 monclient(hunting): handle_auth_done global_id 184352 payload 824 2024-09-15T02:07:26.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.204+0000 7f1344c79640 1 -- 172.21.15.17:0/861336650 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f134011aab0 con 0x7f1340133010 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f1344c79640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 msgr2=0x7f1340133410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f1344c79640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 0x7f1340133410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1334001850 tx=0x7f1334015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334020470 con 0x7f1340133010 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:26.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 10 monclient: dump: 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f13340205d0 con 0x7f1340133010 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 msgr2=0x7f1340133410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 0x7f1340133410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1334001850 tx=0x7f1334015000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 msgr2=0x7f1340133410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.206+0000 7f133cff9640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1340133010 0x7f1340133410 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1334001850 tx=0x7f1334015000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:26.218 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.219 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:26.221 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.222 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:26.223 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.226 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.229 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:26.230 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1324038770 msgr2=0x7f132403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1324038770 0x7f132403ac20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f134011a310 tx=0x7f1338009040 comp rx=0 tx=0).stop 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 10 monclient: shutdown 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 msgr2=0x7f1340118cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 --2- 172.21.15.17:0/861336650 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1340067460 0x7f1340118cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f132c009950 tx=0x7f132c032000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f134547a640 1 -- 172.21.15.17:0/861336650 reap_dead start 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f134547a640 1 -- 172.21.15.17:0/861336650 reap_dead start 2024-09-15T02:07:26.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 shutdown_connections 2024-09-15T02:07:26.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 >> 172.21.15.17:0/861336650 conn(0x7f13400a1ee0 msgr2=0x7f1340097ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 shutdown_connections 2024-09-15T02:07:26.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.207+0000 7f1346edd640 1 -- 172.21.15.17:0/861336650 wait complete. 2024-09-15T02:07:26.233 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:26.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:07:26.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:07:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 10 monclient: get_monmap_and_config 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 10 monclient: monmap: 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:26.331069+0000 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:26.331069+0000 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:26.332 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:26.333 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:26.333 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 1 Processor -- start 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 1 -- start start 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.330+0000 7f2ef9280640 10 monclient: init 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T02:07:26.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 0x7f2ef409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef4131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ef40a0280 0x7f2ef4131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef40a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ef409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40a0040 con 0x7f2ef409f1e0 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef409bff0 con 0x7f2ef4131b20 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40a14a0 con 0x7f2ef40a0280 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef9280640 10 monclient(hunting): authenticate will time out at 1706.327375s 2024-09-15T02:07:26.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 0x7f2ef409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef37fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f2ef409f1e0 0x7f2ef4068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58862/0 (socket says 172.21.15.17:58862) 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 0x7f2ef409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49286/0 (socket says 172.21.15.17:49286) 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef37fe640 1 -- 172.21.15.17:0/2686915273 learned_addr learned my addr 172.21.15.17:0/2686915273 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f2ef4131b20 auth_method 0 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/2686915273 >> v2:172.21.15.67:3300/0 conn(0x7f2ef40a0280 0x7f2ef4131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request con 0x7f2ef40a0280 auth_method 0 2024-09-15T02:07:26.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.331+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_done global_id 204121 payload 274 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 1 -- 172.21.15.17:0/2686915273 >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 msgr2=0x7f2ef409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/2686915273 >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 0x7f2ef409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 1 -- 172.21.15.17:0/2686915273 >> v1:172.21.15.17:6789/0 conn(0x7f2ef409f1e0 legacy=0x7f2ef4068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 1 monclient: found mon.noname-c 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:07:26.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.332+0000 7f2ef27fc640 1 -- 172.21.15.17:0/2686915273 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4132430 con 0x7f2ef40a0280 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef27fc640 1 -- 172.21.15.17:0/2686915273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef40a0280 msgr2=0x7f2ef4131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/2686915273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef40a0280 0x7f2ef4131ef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40a5010 tx=0x7f2ee80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e42902b6f04a1d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 1 -- 172.21.15.17:0/2686915273 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee802ae00 con 0x7f2ef40a0280 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: dump: 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.346 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 1 -- 172.21.15.17:0/2686915273 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee8032440 con 0x7f2ef40a0280 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 1 -- 172.21.15.17:0/2686915273 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80327e0 con 0x7f2ef40a0280 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef1ffb640 10 monclient: dump: 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.348 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 5 monclient: authenticate success, global_id 204121 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 4 set_mon_vals no callback set 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 10 monclient: shutdown 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 1 -- 172.21.15.17:0/2686915273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef40a0280 msgr2=0x7f2ef4131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.335+0000 7f2ef9280640 1 --2- 172.21.15.17:0/2686915273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef40a0280 0x7f2ef4131ef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40a5010 tx=0x7f2ee80090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 -- 172.21.15.17:0/2686915273 shutdown_connections 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 --2- 172.21.15.17:0/2686915273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef40a0280 0x7f2ef4131ef0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 --2- 172.21.15.17:0/2686915273 >> v2:172.21.15.17:3300/0 conn(0x7f2ef4131b20 0x7f2ef409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 -- 172.21.15.17:0/2686915273 >> 172.21.15.17:0/2686915273 conn(0x7f2ef4071270 msgr2=0x7f2ef40a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 -- 172.21.15.17:0/2686915273 shutdown_connections 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 1 -- 172.21.15.17:0/2686915273 wait complete. 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.336+0000 7f2ef9280640 10 monclient: monmap: 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 Processor -- start 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 -- start start 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient: init 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 0x7f2ef4141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef40a0280 0x7f2ef412eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef40a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4131b20 0x7f2ef412f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef4131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef409aec0 con 0x7f2ef409f1e0 2024-09-15T02:07:26.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef411c110 con 0x7f2ef4131b20 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40a09b0 con 0x7f2ef40a0280 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef9280640 10 monclient(hunting): authenticate will time out at 1706.333534s 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 0x7f2ef4141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef40a0280 0x7f2ef412eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 0x7f2ef4141240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49296/0 (socket says 172.21.15.17:49296) 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 -- 172.21.15.17:0/3646498439 learned_addr learned my addr 172.21.15.17:0/3646498439 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f2ef409f1e0 auth_method 0 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request con 0x7f2ef40a0280 auth_method 0 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4131b20 0x7f2ef412f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): get_auth_request con 0x7f2ef4131b20 auth_method 0 2024-09-15T02:07:26.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient(hunting): handle_auth_done global_id 105612 payload 824 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef40a0280 msgr2=0x7f2ef412eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2ef40a0280 0x7f2ef412eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4131b20 msgr2=0x7f2ef412f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4131b20 0x7f2ef412f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 monclient: found mon.noname-a 2024-09-15T02:07:26.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.337+0000 7f2ef2ffd640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8009390 con 0x7f2ef409f1e0 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef2ffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 msgr2=0x7f2ef4141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef2ffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 0x7f2ef4141240 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400ea20 tx=0x7f2ee40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4009600 con 0x7f2ef409f1e0 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: dump: 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.355 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee400cc20 con 0x7f2ef409f1e0 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 5 monclient: authenticate success, global_id 105612 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 10 monclient: _renew_subs 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef41317b0 con 0x7f2ef409f1e0 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4013470 con 0x7f2ef409f1e0 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2edaffd640 10 monclient: dump: 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.357 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 10 monclient: _renew_subs 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef409b510 con 0x7f2ef409f1e0 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.340+0000 7f2ef17fa640 4 set_mon_vals no callback set 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef9280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef9280640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 0x7f2ef4099220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4063a40 con 0x7f2ef4098e50 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2ee4013860 con 0x7f2ef409f1e0 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 0x7f2ef4099220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient: get_auth_request con 0x7f2ef4098e50 auth_method 0 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2edaffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ee0038740 0x7f2ee003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f2ee404c9f0 con 0x7f2ef409f1e0 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_done global_id 204127 payload 824 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ef409aa00 con 0x7f2ef4098e50 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef37fe640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ee0038740 0x7f2ee003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.341+0000 7f2ef37fe640 10 monclient: get_auth_request con 0x7f2ee0038740 auth_method 0 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 msgr2=0x7f2ef4099220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 0x7f2ef4099220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8034000 tx=0x7f2ee8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031c20 con 0x7f2ef4098e50 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2ef37fe640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ee0038740 0x7f2ee003abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006d70 tx=0x7f2eec0052b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 10 monclient: dump: 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.360 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2ee8031d80 con 0x7f2ef4098e50 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 msgr2=0x7f2ef4099220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 0x7f2ef4099220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8034000 tx=0x7f2ee8032000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 msgr2=0x7f2ef4099220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.344+0000 7f2edaffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4098e50 0x7f2ef4099220 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8034000 tx=0x7f2ee8032000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.350+0000 7f2ef9280640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.350+0000 7f2ef9280640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.350+0000 7f2ef9280640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.350+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40724a0 con 0x7f2ef4099760 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.350+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.351+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 msgr2=0x7f2ef4132a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.351+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 msgr2=0x7f2ef4132a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:07:26.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.351+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:26.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.351+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient: get_auth_request con 0x7f2ef4099760 auth_method 0 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 10 monclient(hunting): handle_auth_done global_id 204130 payload 824 2024-09-15T02:07:26.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.552+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2ef4132140 con 0x7f2ef4099760 2024-09-15T02:07:26.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2ef27fc640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 msgr2=0x7f2ef4132a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2ef27fc640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2ef411bcc0 tx=0x7f2ee803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8046810 con 0x7f2ef4099760 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 10 monclient: dump: 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.557 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f2ee803b3d0 con 0x7f2ef4099760 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 msgr2=0x7f2ef4132a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2ef411bcc0 tx=0x7f2ee803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 msgr2=0x7f2ef4132a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:26.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.555+0000 7f2edaffd640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef4099760 0x7f2ef4132a00 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2ef411bcc0 tx=0x7f2ee803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:26.571 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ee0038740 msgr2=0x7f2ee003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2ee0038740 0x7f2ee003abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006d70 tx=0x7f2eec0052b0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 10 monclient: shutdown 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 msgr2=0x7f2ef4141240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 --2- 172.21.15.17:0/3646498439 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2ef409f1e0 0x7f2ef4141240 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400ea20 tx=0x7f2ee40090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef37fe640 1 -- 172.21.15.17:0/3646498439 reap_dead start 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef37fe640 1 -- 172.21.15.17:0/3646498439 reap_dead start 2024-09-15T02:07:26.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 shutdown_connections 2024-09-15T02:07:26.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 >> 172.21.15.17:0/3646498439 conn(0x7f2ef4071270 msgr2=0x7f2ef40a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 shutdown_connections 2024-09-15T02:07:26.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.556+0000 7f2ef9280640 1 -- 172.21.15.17:0/3646498439 wait complete. 2024-09-15T02:07:26.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:26.573 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:07:26.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:07:26.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.668+0000 7f46e6a57640 10 monclient: get_monmap_and_config 2024-09-15T02:07:26.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.668+0000 7f46e6a57640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.668+0000 7f46e6a57640 10 monclient: monmap: 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:26.669784+0000 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:26.669784+0000 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:26.671 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.669+0000 7f46e6a57640 1 Processor -- start 2024-09-15T02:07:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.669+0000 7f46e6a57640 1 -- start start 2024-09-15T02:07:26.674 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.669+0000 7f46e6a57640 10 monclient: init 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 0x7f46e006a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-b con 0x7f46e009e570 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46e006dce0 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46e0133ff0 0x7f46e01363d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-c con 0x7f46e0133ff0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e009e940 con 0x7f46e006dce0 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e006b450 con 0x7f46e0133ff0 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e006ca70 con 0x7f46e009e570 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e6a57640 10 monclient(hunting): authenticate will time out at 1706.666298s 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46df7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f46e006dce0 0x7f46e006a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58866/0 (socket says 172.21.15.17:58866) 2024-09-15T02:07:26.676 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46df7fe640 1 -- 172.21.15.17:0/129148360 learned_addr learned my addr 172.21.15.17:0/129148360 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 --2- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 0x7f46e006a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): get_auth_request con 0x7f46e009e570 auth_method 0 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 1 --2- 172.21.15.17:0/129148360 >> v2:172.21.15.67:3300/0 conn(0x7f46e0133ff0 0x7f46e01363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): get_auth_request con 0x7f46e0133ff0 auth_method 0 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient(hunting): handle_auth_done global_id 184358 payload 274 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 -- 172.21.15.17:0/129148360 >> v2:172.21.15.67:3300/0 conn(0x7f46e0133ff0 msgr2=0x7f46e01363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 --2- 172.21.15.17:0/129148360 >> v2:172.21.15.67:3300/0 conn(0x7f46e0133ff0 0x7f46e01363d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 -- 172.21.15.17:0/129148360 >> v1:172.21.15.17:6789/0 conn(0x7f46e006dce0 legacy=0x7f46e006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 monclient: found mon.noname-b 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.679 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.670+0000 7f46dffff640 1 -- 172.21.15.17:0/129148360 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e006b770 con 0x7f46e009e570 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.673+0000 7f46dffff640 1 -- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 msgr2=0x7f46e006a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.673+0000 7f46dffff640 1 --2- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 0x7f46e006a1c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009850 tx=0x7f46d00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a957244d0ac6a026 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 1 -- 172.21.15.17:0/129148360 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d002adf0 con 0x7f46e009e570 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:26.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: dump: 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.690 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.691 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 1 -- 172.21.15.17:0/129148360 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0032440 con 0x7f46e009e570 2024-09-15T02:07:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 5 monclient: authenticate success, global_id 184358 2024-09-15T02:07:26.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 1 -- 172.21.15.17:0/129148360 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00327e0 con 0x7f46e009e570 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 4 set_mon_vals no callback set 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46deffd640 10 monclient: dump: 2024-09-15T02:07:26.693 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 10 monclient: shutdown 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 -- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 msgr2=0x7f46e006a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 --2- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 0x7f46e006a1c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009850 tx=0x7f46d00090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 -- 172.21.15.17:0/129148360 shutdown_connections 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 --2- 172.21.15.17:0/129148360 >> v2:172.21.15.67:3300/0 conn(0x7f46e0133ff0 0x7f46e01363d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 --2- 172.21.15.17:0/129148360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e009e570 0x7f46e006a1c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 -- 172.21.15.17:0/129148360 >> 172.21.15.17:0/129148360 conn(0x7f46e0067090 msgr2=0x7f46e0067490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 -- 172.21.15.17:0/129148360 shutdown_connections 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.674+0000 7f46e6a57640 1 -- 172.21.15.17:0/129148360 wait complete. 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.675+0000 7f46e6a57640 10 monclient: build_initial_monmap 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.675+0000 7f46e6a57640 10 monclient: monmap: 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:26.696 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.675+0000 7f46e6a57640 1 Processor -- start 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.675+0000 7f46e6a57640 1 -- start start 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.675+0000 7f46e6a57640 10 monclient: init 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:07:26.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e006dce0 0x7f46e011b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-b con 0x7f46e006dce0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 0x7f46e011ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-a con 0x7f46e009e570 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 0x7f46e011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): picked mon.noname-c con 0x7f46e0133ff0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e011d570 con 0x7f46e009e570 2024-09-15T02:07:26.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e009d0b0 con 0x7f46e006dce0 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e006be70 con 0x7f46e0133ff0 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e6a57640 10 monclient(hunting): authenticate will time out at 1706.672291s 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 0x7f46e011ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 0x7f46e011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 0x7f46e011ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49302/0 (socket says 172.21.15.17:49302) 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 1 -- 172.21.15.17:0/1267628745 learned_addr learned my addr 172.21.15.17:0/1267628745 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): get_auth_request con 0x7f46e0133ff0 auth_method 0 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): get_auth_request con 0x7f46e009e570 auth_method 0 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e006dce0 0x7f46e011b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): get_auth_request con 0x7f46e006dce0 auth_method 0 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:26.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient(hunting): handle_auth_done global_id 184361 payload 824 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 -- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e006dce0 msgr2=0x7f46e011b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e006dce0 0x7f46e011b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 -- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 msgr2=0x7f46e011ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 0x7f46e011ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient: _finish_hunting 0 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 monclient: found mon.noname-c 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.676+0000 7f46e4ff4640 1 -- 172.21.15.17:0/1267628745 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0009390 con 0x7f46e0133ff0 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.678+0000 7f46e4ff4640 1 -- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 msgr2=0x7f46e011bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.678+0000 7f46e4ff4640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 0x7f46e011bf70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009e50 tx=0x7f46d800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8014480 con 0x7f46e0133ff0 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: dump: 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:26.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d800f040 con 0x7f46e0133ff0 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8014830 con 0x7f46e0133ff0 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:26.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46c7fff640 10 monclient: dump: 2024-09-15T02:07:26.706 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:26.706 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:26.706 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.706 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:26.706 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 5 monclient: authenticate success, global_id 184361 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 10 monclient: _renew_subs 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e011eab0 con 0x7f46e0133ff0 2024-09-15T02:07:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46de7fc640 4 set_mon_vals no callback set 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 10 monclient: _renew_subs 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.679+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e011dbc0 con 0x7f46e0133ff0 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f46d8014ba0 con 0x7f46e0133ff0 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46e6a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46e6a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46e6a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e011de40 con 0x7f46e0133ff0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46c7fff640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 0x7f46cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f46d804d640 con 0x7f46e0133ff0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46dffff640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 0x7f46cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.680+0000 7f46dffff640 10 monclient: get_auth_request con 0x7f46cc038740 auth_method 0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.683+0000 7f46dffff640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 0x7f46cc03abf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f46e00a1600 tx=0x7f46d0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:26.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.684+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f46e011de40 con 0x7f46e0133ff0 2024-09-15T02:07:26.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.684+0000 7f46c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:26.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.684+0000 7f46c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.909+0000 7f46e6a57640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:07:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.909+0000 7f46e6a57640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:07:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.909+0000 7f46e6a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.909+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f46e011de40 con 0x7f46e0133ff0 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=975316838,config=3140514009,health=4281930021,kv=1529054990,logm=448688420}) 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=4249445756,config=55734535,health=145741386,kv=1529054990,logm=305324907}) 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=975316838,config=3140514009,health=4281930021,kv=1529054990,logm=448688420}) 2024-09-15T02:07:26.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=7,logm=54} crc {auth=503690097,config=3140514009,health=145741386,kv=3442201590,logm=1892834362}) 2024-09-15T02:07:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f46c7fff640 1 -- 172.21.15.17:0/1267628745 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f46e011de40 con 0x7f46e0133ff0 2024-09-15T02:07:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f46c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:07:26.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.911+0000 7f46c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224347151}) 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2166341735}) 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224347151}) 2024-09-15T02:07:26.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103967433}) 2024-09-15T02:07:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 msgr2=0x7f46cc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 0x7f46cc03abf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f46e00a1600 tx=0x7f46d0032000 comp rx=0 tx=0).stop 2024-09-15T02:07:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 10 monclient: shutdown 2024-09-15T02:07:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:26.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 msgr2=0x7f46e011bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 0x7f46e011bf70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009e50 tx=0x7f46d800b040 comp rx=0 tx=0).stop 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 shutdown_connections 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f46cc038740 0x7f46cc03abf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f46e009e570 0x7f46e011ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f46e0133ff0 0x7f46e011bf70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 --2- 172.21.15.17:0/1267628745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e006dce0 0x7f46e011b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.913+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 >> 172.21.15.17:0/1267628745 conn(0x7f46e0067090 msgr2=0x7f46e009c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 shutdown_connections 2024-09-15T02:07:26.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f46e6a57640 1 -- 172.21.15.17:0/1267628745 wait complete. 2024-09-15T02:07:26.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229900683}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515102297}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229900683}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369204891}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232761757}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745235286}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232761757}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3739780110}) 2024-09-15T02:07:26.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115055313}) 2024-09-15T02:07:26.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398907292}) 2024-09-15T02:07:26.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115055313}) 2024-09-15T02:07:26.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93653635}) 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550250464}) 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778988083}) 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550250464}) 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394272800}) 2024-09-15T02:07:26.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.922 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261423461}) 2024-09-15T02:07:26.922 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=144988355}) 2024-09-15T02:07:26.922 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.922 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261423461}) 2024-09-15T02:07:26.922 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899743797}) 2024-09-15T02:07:26.923 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.923 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212812599}) 2024-09-15T02:07:26.923 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159110060}) 2024-09-15T02:07:26.923 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.923 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212812599}) 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701100400}) 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538268335}) 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=141948517}) 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538268335}) 2024-09-15T02:07:26.924 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133993366}) 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105205507}) 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266456066}) 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105205507}) 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597033654}) 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2392637383}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189121210}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2392637383}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.923+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336785809}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978631338}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229954535}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978631338}) 2024-09-15T02:07:26.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.924+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4058780270}) 2024-09-15T02:07:26.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:07:26.929 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696382077}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=891436063}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696382077}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.925+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425238673}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054821622}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3798680813}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054821622}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.926+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091753471}) 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.930 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753735792}) 2024-09-15T02:07:26.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869675908}) 2024-09-15T02:07:26.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753735792}) 2024-09-15T02:07:26.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.927+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222706273}) 2024-09-15T02:07:26.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921313625}) 2024-09-15T02:07:26.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247000279}) 2024-09-15T02:07:26.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921313625}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.928+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=137535569}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120587518}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632415573}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120587518}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.929+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244238966}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289850553}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269391084}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289850553}) 2024-09-15T02:07:26.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.930+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319736476}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476864904}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1320043868}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476864904}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.931+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569883382}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1620747189}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096390098}) 2024-09-15T02:07:26.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1620747189}) 2024-09-15T02:07:26.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031711279}) 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799563448}) 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914565077}) 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799563448}) 2024-09-15T02:07:26.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1157889403}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768989318}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992144711}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768989318}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568074542}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674600642}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3168053821}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674600642}) 2024-09-15T02:07:26.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1529732832}) 2024-09-15T02:07:26.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898500559}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3450045666}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898500559}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9392834}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670542536}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3765113105}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670542536}) 2024-09-15T02:07:26.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814377522}) 2024-09-15T02:07:26.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696777207}) 2024-09-15T02:07:26.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186413122}) 2024-09-15T02:07:26.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.938+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696777207}) 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945779679}) 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16509773}) 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766619798}) 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16509773}) 2024-09-15T02:07:26.942 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792545591}) 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440028354}) 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1544785774}) 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440028354}) 2024-09-15T02:07:26.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475072163}) 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510958231}) 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615609095}) 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510958231}) 2024-09-15T02:07:26.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1634498523}) 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052642406}) 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1427610178}) 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052642406}) 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=665562629}) 2024-09-15T02:07:26.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291574606}) 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826201844}) 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291574606}) 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517330374}) 2024-09-15T02:07:26.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549878564}) 2024-09-15T02:07:26.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278594359}) 2024-09-15T02:07:26.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549878564}) 2024-09-15T02:07:26.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205899376}) 2024-09-15T02:07:26.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629592489}) 2024-09-15T02:07:26.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670253291}) 2024-09-15T02:07:26.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.946+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.946+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629592489}) 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.946+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4114269346}) 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948300928}) 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422970150}) 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948300928}) 2024-09-15T02:07:26.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046924356}) 2024-09-15T02:07:26.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245947478}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668987151}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245947478}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308939651}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331306441}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98699595}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331306441}) 2024-09-15T02:07:26.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.949+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079344773}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545610681}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3817571615}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545610681}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.950+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394145555}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870965300}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2610404221}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870965300}) 2024-09-15T02:07:26.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.951+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317930412}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198817159}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=248670712}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198817159}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.952+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066033516}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881302907}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2348369048}) 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881302907}) 2024-09-15T02:07:26.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.953+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43994113}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374366076}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751136891}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374366076}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.954+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987982989}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171972120}) 2024-09-15T02:07:26.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169893469}) 2024-09-15T02:07:26.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171972120}) 2024-09-15T02:07:26.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.955+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843256944}) 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594937777}) 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1992887156}) 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594937777}) 2024-09-15T02:07:26.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.956+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910511103}) 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806737394}) 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738129147}) 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806737394}) 2024-09-15T02:07:26.960 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.957+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2476449590}) 2024-09-15T02:07:26.961 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.961 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903290016}) 2024-09-15T02:07:26.961 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584497232}) 2024-09-15T02:07:26.961 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903290016}) 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.958+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624397420}) 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821175980}) 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787682069}) 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821175980}) 2024-09-15T02:07:26.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.959+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216156765}) 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201127956}) 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374700785}) 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201127956}) 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262329421}) 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.964 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892664356}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543702187}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892664356}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395413730}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028092938}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=48279449}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028092938}) 2024-09-15T02:07:26.965 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.962+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3306102058}) 2024-09-15T02:07:26.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25427753}) 2024-09-15T02:07:26.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011904543}) 2024-09-15T02:07:26.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25427753}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.963+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652178211}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251632888}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236851836}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251632888}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.964+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610336154}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161755751}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=77127472}) 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161755751}) 2024-09-15T02:07:26.969 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872686952}) 2024-09-15T02:07:26.970 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180965649}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769248417}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180965649}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.967+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042586959}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053016312}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780235505}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053016312}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.968+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611078515}) 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471265166}) 2024-09-15T02:07:26.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354022930}) 2024-09-15T02:07:26.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471265166}) 2024-09-15T02:07:26.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.969+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328501849}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276980967}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859163575}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276980967}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.970+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323165605}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483840140}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349816143}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483840140}) 2024-09-15T02:07:26.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.971+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390677912}) 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596843925}) 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731180823}) 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596843925}) 2024-09-15T02:07:26.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.972+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446648129}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918851558}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=336691150}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918851558}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.973+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527719144}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258949666}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3176957825}) 2024-09-15T02:07:26.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258949666}) 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.974+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496269942}) 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=292138675}) 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037578676}) 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=292138675}) 2024-09-15T02:07:26.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.975+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407447430}) 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662408845}) 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212987548}) 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662408845}) 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.976+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680845344}) 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491762358}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080652055}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491762358}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.977+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3003372116}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552793550}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25448985}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552793550}) 2024-09-15T02:07:26.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.978+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4239121911}) 2024-09-15T02:07:26.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186401135}) 2024-09-15T02:07:26.983 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808717944}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186401135}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467026424}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618309125}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167174931}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618309125}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.980+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3986258398}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833337982}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716163343}) 2024-09-15T02:07:26.984 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833337982}) 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.981+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250792617}) 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388071956}) 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030661126}) 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388071956}) 2024-09-15T02:07:26.985 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.982+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704990671}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756267413}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165551118}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756267413}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485448820}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167669734}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750454428}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167669734}) 2024-09-15T02:07:26.987 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.983+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966397219}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408839199}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3958365836}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408839199}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.984+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514259443}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152743475}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4294070626}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152743475}) 2024-09-15T02:07:26.988 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.985+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064121258}) 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101653305}) 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543009368}) 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101653305}) 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239551782}) 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.991 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588579824}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010481698}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588579824}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.986+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2200879453}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950413143}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540218471}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950413143}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.987+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152931339}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244613964}) 2024-09-15T02:07:26.992 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055096520}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244613964}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509570566}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140942530}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033583985}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140942530}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.988+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025687996}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320219743}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675352780}) 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.993 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320219743}) 2024-09-15T02:07:26.994 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.989+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906634457}) 2024-09-15T02:07:26.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943617447}) 2024-09-15T02:07:26.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798150378}) 2024-09-15T02:07:26.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.996 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943617447}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.990+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290314667}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232293742}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1167697148}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232293742}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780531002}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407837395}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52414954}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407837395}) 2024-09-15T02:07:26.997 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.991+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689211032}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487675077}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403399173}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487675077}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.992+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009118248}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101036305}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248268081}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101036305}) 2024-09-15T02:07:26.998 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.993+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611978002}) 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=896551801,mdsmap=1531151339,mgr=2208818414,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2983066093,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=1441233093}) 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=1059915425,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3608543741,mon_config_key=1583035879,monmap=3140514009,osd_metadata=1149731095,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3573084706}) 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=896551801,mdsmap=1531151339,mgr=2208818414,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2983066093,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=1441233093}) 2024-09-15T02:07:26.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.994+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=8} crc {logm=1882209822,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3608543741,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1149731095,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=904814557}) 2024-09-15T02:07:27.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:27.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3240628586}) 2024-09-15T02:07:27.001 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=46} crc {osdmap=1500654915}) 2024-09-15T02:07:27.001 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:07:27.001 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3240628586}) 2024-09-15T02:07:27.001 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:07:26.999+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=1472191856}) 2024-09-15T02:07:27.929 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:07:27.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:28.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.035+0000 7ff80299d640 10 monclient: get_monmap_and_config 2024-09-15T02:07:28.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.035+0000 7ff80299d640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.035+0000 7ff80299d640 10 monclient: monmap: 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:28.036753+0000 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:28.036753+0000 2024-09-15T02:07:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:28.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.036+0000 7ff80299d640 1 Processor -- start 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.036+0000 7ff80299d640 1 -- start start 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.036+0000 7ff80299d640 10 monclient: init 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7fc09ce00 0x7ff7fc067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc09ce00 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7fc0679a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:28.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc09e1e0 con 0x7ff7fc0679a0 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7fc09e4c0 con 0x7ff7fc09ce00 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc09cbf0 con 0x7ff7fc067090 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff80299d640 10 monclient(hunting): authenticate will time out at 1708.033349s 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40970/0 (socket says 172.21.15.17:40970) 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2872835732 learned_addr learned my addr 172.21.15.17:0/2872835732 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request con 0x7ff7fc067090 auth_method 0 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fb7fe640 1 --2- 172.21.15.17:0/2872835732 >> v2:172.21.15.67:3300/0 conn(0x7ff7fc09ce00 0x7ff7fc067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff7fc09ce00 auth_method 0 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_done global_id 184364 payload 274 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2872835732 >> v2:172.21.15.67:3300/0 conn(0x7ff7fc09ce00 msgr2=0x7ff7fc067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 --2- 172.21.15.17:0/2872835732 >> v2:172.21.15.67:3300/0 conn(0x7ff7fc09ce00 0x7ff7fc067460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2872835732 >> v1:172.21.15.67:6789/0 conn(0x7ff7fc0679a0 legacy=0x7ff7fc12e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 monclient: found mon.noname-b 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.037+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2872835732 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc067e00 con 0x7ff7fc067090 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2872835732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 msgr2=0x7ff7fc132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7fbfff640 1 --2- 172.21.15.17:0/2872835732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009910 tx=0x7ff7ec0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e438757ddc74eb02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 1 -- 172.21.15.17:0/2872835732 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec02aea0 con 0x7ff7fc067090 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: dump: 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.052 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 1 -- 172.21.15.17:0/2872835732 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec032440 con 0x7ff7fc067090 2024-09-15T02:07:28.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 1 -- 172.21.15.17:0/2872835732 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec0327e0 con 0x7ff7fc067090 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff80299d640 5 monclient: authenticate success, global_id 184364 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff80299d640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff80299d640 4 set_mon_vals no callback set 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.040+0000 7ff7faffd640 10 monclient: dump: 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.054 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 10 monclient: shutdown 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 -- 172.21.15.17:0/2872835732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 msgr2=0x7ff7fc132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 --2- 172.21.15.17:0/2872835732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009910 tx=0x7ff7ec0091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 -- 172.21.15.17:0/2872835732 shutdown_connections 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 --2- 172.21.15.17:0/2872835732 >> v2:172.21.15.67:3300/0 conn(0x7ff7fc09ce00 0x7ff7fc067460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 --2- 172.21.15.17:0/2872835732 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc132140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 -- 172.21.15.17:0/2872835732 >> 172.21.15.17:0/2872835732 conn(0x7ff7fc0a1ee0 msgr2=0x7ff7fc0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 -- 172.21.15.17:0/2872835732 shutdown_connections 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.041+0000 7ff80299d640 1 -- 172.21.15.17:0/2872835732 wait complete. 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 10 monclient: monmap: 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 1 Processor -- start 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 1 -- start start 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 10 monclient: init 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.042+0000 7ff80299d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc0679a0 0x7ff7fc06aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc0679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:28.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 0x7ff7fc06af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc09ce00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc09d5c0 con 0x7ff7fc09ce00 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc134cc0 con 0x7ff7fc0679a0 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc06dfb0 con 0x7ff7fc067090 2024-09-15T02:07:28.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff80299d640 10 monclient(hunting): authenticate will time out at 1708.039059s 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40974/0 (socket says 172.21.15.17:40974) 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2819194250 learned_addr learned my addr 172.21.15.17:0/2819194250 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 0x7ff7fc06af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request con 0x7ff7fc067090 auth_method 0 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): get_auth_request con 0x7ff7fc09ce00 auth_method 0 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fb7fe640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc0679a0 0x7ff7fc06aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff7fc0679a0 auth_method 0 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient(hunting): handle_auth_done global_id 184367 payload 824 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc0679a0 msgr2=0x7ff7fc06aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc0679a0 0x7ff7fc06aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 msgr2=0x7ff7fc06af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 0x7ff7fc06af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 monclient: found mon.noname-c 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2819194250 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec009450 con 0x7ff7fc067090 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.043+0000 7ff800f3a640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 0x7ff7fc06af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.045+0000 7ff7fbfff640 1 -- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 msgr2=0x7ff7fc1311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.045+0000 7ff7fbfff640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009970 tx=0x7ff7ec032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec02aea0 con 0x7ff7fc067090 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: dump: 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.063 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec006b30 con 0x7ff7fc067090 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 5 monclient: authenticate success, global_id 184367 2024-09-15T02:07:28.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 10 monclient: _renew_subs 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc06e290 con 0x7ff7fc067090 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec03c400 con 0x7ff7fc067090 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7e3fff640 10 monclient: dump: 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 10 monclient: _renew_subs 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff7fa7fc640 4 set_mon_vals no callback set 2024-09-15T02:07:28.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.046+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc06c930 con 0x7ff7fc067090 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff7ec03c830 con 0x7ff7fc067090 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff80299d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff80299d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff80299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc0724a0 con 0x7ff7fc067090 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff7e3fff640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 0x7ff7e803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff7ec06cb50 con 0x7ff7fc067090 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff7fb7fe640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 0x7ff7e803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.047+0000 7ff7fb7fe640 10 monclient: get_auth_request con 0x7ff7e8038770 auth_method 0 2024-09-15T02:07:28.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.050+0000 7ff7fb7fe640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 0x7ff7e803ac20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc06c020 tx=0x7ff7f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.051+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff7ec03b3a0 con 0x7ff7fc067090 2024-09-15T02:07:28.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.051+0000 7ff7e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.051+0000 7ff7e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff80299d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff80299d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff80299d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff7fc06f3d0 con 0x7ff7fc067090 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff7e3fff640 1 -- 172.21.15.17:0/2819194250 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7ff7ec037080 con 0x7ff7fc067090 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff7e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.281+0000 7ff7e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:28.283 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:28.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 msgr2=0x7ff7e803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 0x7ff7e803ac20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc06c020 tx=0x7ff7f0008040 comp rx=0 tx=0).stop 2024-09-15T02:07:28.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 10 monclient: shutdown 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 msgr2=0x7ff7fc1311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009970 tx=0x7ff7ec032000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 shutdown_connections 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff7e8038770 0x7ff7e803ac20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff7fc09ce00 0x7ff7fc06af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc0679a0 0x7ff7fc06aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 --2- 172.21.15.17:0/2819194250 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff7fc067090 0x7ff7fc1311b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 >> 172.21.15.17:0/2819194250 conn(0x7ff7fc0a1ee0 msgr2=0x7ff7fc0a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 shutdown_connections 2024-09-15T02:07:28.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.283+0000 7ff80299d640 1 -- 172.21.15.17:0/2819194250 wait complete. 2024-09-15T02:07:28.295 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:07:28.295 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:07:28.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:07:28.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.391+0000 7f9c206d4640 10 monclient: get_monmap_and_config 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.391+0000 7f9c206d4640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.391+0000 7f9c206d4640 10 monclient: monmap: 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:28.392330+0000 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:28.392330+0000 2024-09-15T02:07:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:28.394 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 Processor -- start 2024-09-15T02:07:28.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 -- start start 2024-09-15T02:07:28.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient: init 2024-09-15T02:07:28.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 0x7f9c1809a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1809c1b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 0x7f9c1806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c1809c580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c1809c990 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1809f790 con 0x7f9c1809c990 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1806f780 con 0x7f9c1809c1b0 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18069d40 con 0x7f9c1809c580 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c206d4640 10 monclient(hunting): authenticate will time out at 1708.388785s 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1e470640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 0x7f9c1809a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1dc6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 0x7f9c1806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1ec71640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9c1809c990 0x7f9c1806f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:58876/0 (socket says 172.21.15.17:58876) 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1ec71640 1 -- 172.21.15.17:0/1429034511 learned_addr learned my addr 172.21.15.17:0/1429034511 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1e470640 10 monclient(hunting): get_auth_request con 0x7f9c1809c1b0 auth_method 0 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1e470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1e470640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1e470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request con 0x7f9c1809c580 auth_method 0 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.392+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_done global_id 184370 payload 274 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/1429034511 >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 msgr2=0x7f9c1809a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/1429034511 >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 0x7f9c1809a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/1429034511 >> v1:172.21.15.17:6789/0 conn(0x7f9c1809c990 legacy=0x7f9c1806f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 1 monclient: found mon.noname-b 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/1429034511 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c18068b60 con 0x7f9c1809c580 2024-09-15T02:07:28.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.393+0000 7f9c1e470640 1 --2- 172.21.15.17:0/1429034511 >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 0x7f9c1809a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/1429034511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 msgr2=0x7f9c1806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/1429034511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 0x7f9c1806eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009850 tx=0x7f9c080090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d06f4c1534420a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 1 -- 172.21.15.17:0/1429034511 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0802adf0 con 0x7f9c1809c580 2024-09-15T02:07:28.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: dump: 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.407 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 5 monclient: authenticate success, global_id 184370 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 1 -- 172.21.15.17:0/1429034511 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08032440 con 0x7f9c1809c580 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 4 set_mon_vals no callback set 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 1 -- 172.21.15.17:0/1429034511 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080327e0 con 0x7f9c1809c580 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c1d46e640 10 monclient: dump: 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 10 monclient: shutdown 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 -- 172.21.15.17:0/1429034511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 msgr2=0x7f9c1806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 --2- 172.21.15.17:0/1429034511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 0x7f9c1806eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009850 tx=0x7f9c080090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 -- 172.21.15.17:0/1429034511 shutdown_connections 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 --2- 172.21.15.17:0/1429034511 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c580 0x7f9c1806eb30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 --2- 172.21.15.17:0/1429034511 >> v2:172.21.15.17:3300/0 conn(0x7f9c1809c1b0 0x7f9c1809a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 -- 172.21.15.17:0/1429034511 >> 172.21.15.17:0/1429034511 conn(0x7f9c18071300 msgr2=0x7f9c180a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 -- 172.21.15.17:0/1429034511 shutdown_connections 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.396+0000 7f9c206d4640 1 -- 172.21.15.17:0/1429034511 wait complete. 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.397+0000 7f9c206d4640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.397+0000 7f9c206d4640 10 monclient: monmap: 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:28.410 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.397+0000 7f9c206d4640 1 Processor -- start 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.397+0000 7f9c206d4640 1 -- start start 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.397+0000 7f9c206d4640 10 monclient: init 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:07:28.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 0x7f9c1813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c1809c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 0x7f9c1813d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1809c580 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1809c990 0x7f9c181411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c1809c990 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18122c50 con 0x7f9c1809c580 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18122690 con 0x7f9c1809c990 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c181201b0 con 0x7f9c1809c1b0 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c206d4640 10 monclient(hunting): authenticate will time out at 1708.394075s 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 0x7f9c1813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 0x7f9c1813d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 0x7f9c1813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:40994/0 (socket says 172.21.15.17:40994) 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 0x7f9c1813d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49338/0 (socket says 172.21.15.17:49338) 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 -- 172.21.15.17:0/849365701 learned_addr learned my addr 172.21.15.17:0/849365701 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): get_auth_request con 0x7f9c1809c1b0 auth_method 0 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request con 0x7f9c1809c580 auth_method 0 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient(hunting): handle_auth_done global_id 184373 payload 824 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1809c990 msgr2=0x7f9c181411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1809c990 0x7f9c181411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 msgr2=0x7f9c1813d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 0x7f9c1813d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 monclient: found mon.noname-c 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1e470640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14009390 con 0x7f9c1809c1b0 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.398+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1809c580 0x7f9c1813d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c1e470640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 msgr2=0x7f9c1813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c1e470640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 0x7f9c1813dc60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400ecb0 tx=0x7f9c140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400cdd0 con 0x7f9c1809c1b0 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: dump: 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.415 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14013470 con 0x7f9c1809c1b0 2024-09-15T02:07:28.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 5 monclient: authenticate success, global_id 184373 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14013810 con 0x7f9c1809c1b0 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 10 monclient: _renew_subs 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c08009390 con 0x7f9c1809c1b0 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c0e7fc640 10 monclient: dump: 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.417 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c1cc6d640 4 set_mon_vals no callback set 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 10 monclient: _renew_subs 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.400+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1813d850 con 0x7f9c1809c1b0 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c206d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c206d4640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 0x7f9c18141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18122b10 con 0x7f9c181214e0 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9c14013b80 con 0x7f9c1809c1b0 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 0x7f9c18141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient: get_auth_request con 0x7f9c181214e0 auth_method 0 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.401+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_done global_id 105627 payload 824 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c18122520 con 0x7f9c181214e0 2024-09-15T02:07:28.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c0e7fc640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9bfc038790 0x7f9bfc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9c1404caa0 con 0x7f9c1809c1b0 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c1ec71640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9bfc038790 0x7f9bfc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.402+0000 7f9c1ec71640 10 monclient: get_auth_request con 0x7f9bfc038790 auth_method 0 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 msgr2=0x7f9c18141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 0x7f9c18141710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9c08034000 tx=0x7f9c08032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0803c440 con 0x7f9c181214e0 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 10 monclient: dump: 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.420 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c1ec71640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9bfc038790 0x7f9bfc03ac40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9c1813e1e0 tx=0x7f9c10004f30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9c0803c5a0 con 0x7f9c181214e0 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.404+0000 7f9c0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.405+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 msgr2=0x7f9c18141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.405+0000 7f9c0e7fc640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 0x7f9c18141710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9c08034000 tx=0x7f9c08032000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.405+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 msgr2=0x7f9c18141710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.405+0000 7f9c0e7fc640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c181214e0 0x7f9c18141710 secure :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9c08034000 tx=0x7f9c08032000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c206d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:28.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c206d4640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 0x7f9c18141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c206d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1813daa0 con 0x7f9c1813e5e0 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 0x7f9c18141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient: get_auth_request con 0x7f9c1813e5e0 auth_method 0 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.410+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.411+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.411+0000 7f9c1dc6f640 10 monclient(hunting): handle_auth_done global_id 105630 payload 824 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.411+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/849365701 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9c180724a0 con 0x7f9c1813e5e0 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c1dc6f640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 msgr2=0x7f9c18141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c1dc6f640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 0x7f9c18141c50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806ab50 tx=0x7f9c0803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080469c0 con 0x7f9c1813e5e0 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 10 monclient: dump: 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f9c08046b20 con 0x7f9c1813e5e0 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 msgr2=0x7f9c18141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 0x7f9c18141c50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806ab50 tx=0x7f9c0803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 msgr2=0x7f9c18141c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.413+0000 7f9c0e7fc640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9c1813e5e0 0x7f9c18141c50 secure :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806ab50 tx=0x7f9c0803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:28.426 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:28.427 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:28.428 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:28.429 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:28.430 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.431 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.433 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.435 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.437 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.438 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 6 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.440 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:28.440 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9bfc038790 msgr2=0x7f9bfc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9bfc038790 0x7f9bfc03ac40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9c1813e1e0 tx=0x7f9c10004f30 comp rx=0 tx=0).stop 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 10 monclient: shutdown 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 msgr2=0x7f9c1813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 --2- 172.21.15.17:0/849365701 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9c1809c1b0 0x7f9c1813dc60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400ecb0 tx=0x7f9c140090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c1ec71640 1 -- 172.21.15.17:0/849365701 reap_dead start 2024-09-15T02:07:28.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c1ec71640 1 -- 172.21.15.17:0/849365701 reap_dead start 2024-09-15T02:07:28.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 shutdown_connections 2024-09-15T02:07:28.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 >> 172.21.15.17:0/849365701 conn(0x7f9c18071300 msgr2=0x7f9c18098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 shutdown_connections 2024-09-15T02:07:28.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.414+0000 7f9c206d4640 1 -- 172.21.15.17:0/849365701 wait complete. 2024-09-15T02:07:28.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:28.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:07:28.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:07:28.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 10 monclient: get_monmap_and_config 2024-09-15T02:07:28.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 10 monclient: monmap: 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:28.539019+0000 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:28.539019+0000 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:28.540 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:28.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 1 Processor -- start 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 1 -- start start 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.538+0000 7fafb335d640 10 monclient: init 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-b con 0x7fafac0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafac0980a0 0x7fafac098470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-c con 0x7fafac0980a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-a con 0x7fafac0989b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac0a01e0 con 0x7fafac0989b0 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac0a1580 con 0x7fafac0980a0 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac0a5c80 con 0x7fafac0a58b0 2024-09-15T02:07:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb335d640 10 monclient(hunting): authenticate will time out at 1708.535499s 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41000/0 (socket says 172.21.15.17:41000) 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 -- 172.21.15.17:0/852004591 learned_addr learned my addr 172.21.15.17:0/852004591 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49364/0 (socket says 172.21.15.17:49364) 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): get_auth_request con 0x7fafac0a58b0 auth_method 0 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): get_auth_request con 0x7fafac0989b0 auth_method 0 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.67:3300/0 conn(0x7fafac0980a0 0x7fafac098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): get_auth_request con 0x7fafac0980a0 auth_method 0 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb08f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient(hunting): handle_auth_done global_id 184376 payload 274 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 -- 172.21.15.17:0/852004591 >> v2:172.21.15.67:3300/0 conn(0x7fafac0980a0 msgr2=0x7fafac098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.67:3300/0 conn(0x7fafac0980a0 0x7fafac098470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 -- 172.21.15.17:0/852004591 >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 msgr2=0x7fafac068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 monclient: found mon.noname-b 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb10f9640 1 -- 172.21.15.17:0/852004591 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac1326c0 con 0x7fafac0a58b0 2024-09-15T02:07:28.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.539+0000 7fafb18fa640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb10f9640 1 -- 172.21.15.17:0/852004591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 msgr2=0x7fafac132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb10f9640 1 --2- 172.21.15.17:0/852004591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fafac0a5ff0 tx=0x7fafa80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=113216d09a279d0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 1 -- 172.21.15.17:0/852004591 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8032440 con 0x7fafac0a58b0 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:28.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: dump: 2024-09-15T02:07:28.556 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.556 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.556 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.556 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.556 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb335d640 5 monclient: authenticate success, global_id 184376 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 1 -- 172.21.15.17:0/852004591 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa80325a0 con 0x7fafac0a58b0 2024-09-15T02:07:28.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb335d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb335d640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 1 -- 172.21.15.17:0/852004591 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8032980 con 0x7fafac0a58b0 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafb335d640 4 set_mon_vals no callback set 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.542+0000 7fafa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafa3fff640 10 monclient: dump: 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.558 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 10 monclient: shutdown 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 -- 172.21.15.17:0/852004591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 msgr2=0x7fafac132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.559 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 --2- 172.21.15.17:0/852004591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fafac0a5ff0 tx=0x7fafa80091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 -- 172.21.15.17:0/852004591 shutdown_connections 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.67:3300/0 conn(0x7fafac0980a0 0x7fafac098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 --2- 172.21.15.17:0/852004591 >> v2:172.21.15.17:3300/0 conn(0x7fafac0989b0 0x7fafac068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 --2- 172.21.15.17:0/852004591 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac132040 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 -- 172.21.15.17:0/852004591 >> 172.21.15.17:0/852004591 conn(0x7fafac070ed0 msgr2=0x7fafac0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 -- 172.21.15.17:0/852004591 shutdown_connections 2024-09-15T02:07:28.560 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.543+0000 7fafb335d640 1 -- 172.21.15.17:0/852004591 wait complete. 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 10 monclient: monmap: 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:28.561 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 1 Processor -- start 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 1 -- start start 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 10 monclient: init 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.544+0000 7fafb335d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 0x7fafac09afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-a con 0x7fafac0980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:28.562 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafac0989b0 0x7fafac130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-b con 0x7fafac0989b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): picked mon.noname-c con 0x7fafac0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac11df50 con 0x7fafac0980a0 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac09e9c0 con 0x7fafac0989b0 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac11dc70 con 0x7fafac0a58b0 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb335d640 10 monclient(hunting): authenticate will time out at 1708.541074s 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 0x7fafac09afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41012/0 (socket says 172.21.15.17:41012) 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 0x7fafac09afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49368/0 (socket says 172.21.15.17:49368) 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 learned_addr learned my addr 172.21.15.17:0/1129853554 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): get_auth_request con 0x7fafac0a58b0 auth_method 0 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): get_auth_request con 0x7fafac0980a0 auth_method 0 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb08f8640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafac0989b0 0x7fafac130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient(hunting): handle_auth_done global_id 184379 payload 824 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafac0989b0 msgr2=0x7fafac130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafac0989b0 0x7fafac130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 msgr2=0x7fafac09afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 0x7fafac09afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 monclient: found mon.noname-c 2024-09-15T02:07:28.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa4009a20 con 0x7fafac0a58b0 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb08f8640 10 monclient: get_auth_request con 0x7fafac0989b0 auth_method 0 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb08f8640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafac0989b0 0x7fafac130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.545+0000 7fafb10f9640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fafac0980a0 0x7fafac09afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 msgr2=0x7fafac141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb18fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fafa400efc0 tx=0x7fafa400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa400ce70 con 0x7fafac0a58b0 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: dump: 2024-09-15T02:07:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb335d640 5 monclient: authenticate success, global_id 184379 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa400f040 con 0x7fafac0a58b0 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb335d640 10 monclient: _renew_subs 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8009450 con 0x7fafac0a58b0 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40148e0 con 0x7fafac0a58b0 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa17fa640 10 monclient: dump: 2024-09-15T02:07:28.568 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 10 monclient: _renew_subs 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac11cd80 con 0x7fafac0a58b0 2024-09-15T02:07:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.547+0000 7fafa2ffd640 4 set_mon_vals no callback set 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 0x7fafac1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac063b20 con 0x7fafac11bca0 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 0x7fafac1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fafa4014cd0 con 0x7fafac0a58b0 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient: get_auth_request con 0x7fafac11bca0 auth_method 0 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.548+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafb10f9640 10 monclient(hunting): handle_auth_done global_id 184382 payload 824 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafb10f9640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafac11c9c0 con 0x7fafac11bca0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafa17fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf90038740 0x7faf9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fafa404d670 con 0x7fafac0a58b0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafb08f8640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf90038740 0x7faf9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.549+0000 7fafb08f8640 10 monclient: get_auth_request con 0x7faf90038740 auth_method 0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafb10f9640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 msgr2=0x7fafac1416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafb10f9640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 0x7fafac1416d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fafac09c270 tx=0x7fafa8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8032440 con 0x7fafac11bca0 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 10 monclient: dump: 2024-09-15T02:07:28.571 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fafa8046440 con 0x7fafac11bca0 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 msgr2=0x7fafac1416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 0x7fafac1416d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fafac09c270 tx=0x7fafa8032000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 msgr2=0x7fafac1416d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.551+0000 7fafa17fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac11bca0 0x7fafac1416d0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fafac09c270 tx=0x7fafa8032000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.552+0000 7fafb08f8640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf90038740 0x7faf9003abf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf9c006e70 tx=0x7faf9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb335d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:28.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb335d640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 0x7fafac141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb335d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac098e30 con 0x7fafac131090 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 0x7fafac141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient: get_auth_request con 0x7fafac131090 auth_method 0 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 10 monclient(hunting): handle_auth_done global_id 184385 payload 824 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.557+0000 7fafb10f9640 1 -- 172.21.15.17:0/1129853554 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fafac0724a0 con 0x7fafac131090 2024-09-15T02:07:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb10f9640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 msgr2=0x7fafac141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb10f9640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 0x7fafac141c10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009210 tx=0x7fafa803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa804c070 con 0x7fafac131090 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 10 monclient: dump: 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.576 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fafa803c800 con 0x7fafac131090 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 msgr2=0x7fafac141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 0x7fafac141c10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009210 tx=0x7fafa803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 msgr2=0x7fafac141c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafa17fa640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac131090 0x7fafac141c10 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009210 tx=0x7fafa803d000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.578 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.579 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.580 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.581 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:28.581 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:28.581 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:28.581 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:28.582 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:28.583 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:28.583 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.588 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.589 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.590 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.591 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.592 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.593 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:28.594 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf90038740 msgr2=0x7faf9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf90038740 0x7faf9003abf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf9c006e70 tx=0x7faf9c008040 comp rx=0 tx=0).stop 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 10 monclient: shutdown 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 msgr2=0x7fafac141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb335d640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fafa400efc0 tx=0x7fafa400b040 comp rx=0 tx=0).stop 2024-09-15T02:07:28.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.560+0000 7fafb18fa640 1 -- 172.21.15.17:0/1129853554 reap_dead start 2024-09-15T02:07:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.561+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 shutdown_connections 2024-09-15T02:07:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.561+0000 7fafb335d640 1 --2- 172.21.15.17:0/1129853554 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fafac0a58b0 0x7fafac141190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.561+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 >> 172.21.15.17:0/1129853554 conn(0x7fafac070ed0 msgr2=0x7fafac096e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.561+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 shutdown_connections 2024-09-15T02:07:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.561+0000 7fafb335d640 1 -- 172.21.15.17:0/1129853554 wait complete. 2024-09-15T02:07:28.596 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:28.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:07:28.597 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:07:28.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 10 monclient: get_monmap_and_config 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 10 monclient: monmap: 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:28.693067+0000 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:28.693067+0000 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:28.695 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 1 Processor -- start 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 1 -- start start 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.692+0000 7f55f886f640 10 monclient: init 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T02:07:28.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-a con 0x7f55f40a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55f40980a0 0x7f55f40984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-c con 0x7f55f40980a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-b con 0x7f55f40989e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f40a01e0 con 0x7f55f40a58b0 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f40a1580 con 0x7f55f40980a0 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40a5c80 con 0x7f55f40989e0 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f886f640 10 monclient(hunting): authenticate will time out at 1708.689500s 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49380/0 (socket says 172.21.15.17:49380) 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41038/0 (socket says 172.21.15.17:41038) 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 1 -- 172.21.15.17:0/1015076249 learned_addr learned my addr 172.21.15.17:0/1015076249 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): get_auth_request con 0x7f55f40a58b0 auth_method 0 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): get_auth_request con 0x7f55f40989e0 auth_method 0 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_done global_id 184388 payload 274 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.67:3300/0 conn(0x7f55f40980a0 0x7f55f40984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 -- 172.21.15.17:0/1015076249 >> v2:172.21.15.67:3300/0 conn(0x7f55f40980a0 msgr2=0x7f55f40984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.67:3300/0 conn(0x7f55f40980a0 0x7f55f40984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 -- 172.21.15.17:0/1015076249 >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 msgr2=0x7f55f4132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.693+0000 7f55f2d9d640 1 monclient: found mon.noname-b 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.694+0000 7f55f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.694+0000 7f55f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.694+0000 7f55f2d9d640 1 -- 172.21.15.17:0/1015076249 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f41326c0 con 0x7f55f40989e0 2024-09-15T02:07:28.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.694+0000 7f55f259c640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:28.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.696+0000 7f55f2d9d640 1 -- 172.21.15.17:0/1015076249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 msgr2=0x7f55f4068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.696+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/1015076249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55f41328b0 tx=0x7f55ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13a0e4f181aa1818 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.696+0000 7f55f159a640 1 -- 172.21.15.17:0/1015076249 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec007760 con 0x7f55f40989e0 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: dump: 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.709 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 1 -- 172.21.15.17:0/1015076249 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec02d050 con 0x7f55f40989e0 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 5 monclient: authenticate success, global_id 184388 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 1 -- 172.21.15.17:0/1015076249 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec007af0 con 0x7f55f40989e0 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:28.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 4 set_mon_vals no callback set 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f159a640 10 monclient: dump: 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 10 monclient: shutdown 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 -- 172.21.15.17:0/1015076249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 msgr2=0x7f55f4068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 --2- 172.21.15.17:0/1015076249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55f41328b0 tx=0x7f55ec009780 comp rx=0 tx=0).stop 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 -- 172.21.15.17:0/1015076249 shutdown_connections 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.67:3300/0 conn(0x7f55f40980a0 0x7f55f40984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 --2- 172.21.15.17:0/1015076249 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40989e0 0x7f55f4068ef0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 --2- 172.21.15.17:0/1015076249 >> v2:172.21.15.17:3300/0 conn(0x7f55f40a58b0 0x7f55f4132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 -- 172.21.15.17:0/1015076249 >> 172.21.15.17:0/1015076249 conn(0x7f55f4070ed0 msgr2=0x7f55f40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 -- 172.21.15.17:0/1015076249 shutdown_connections 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.697+0000 7f55f886f640 1 -- 172.21.15.17:0/1015076249 wait complete. 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.698+0000 7f55f886f640 10 monclient: build_initial_monmap 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.698+0000 7f55f886f640 10 monclient: monmap: 2024-09-15T02:07:28.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.698+0000 7f55f886f640 1 Processor -- start 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.698+0000 7f55f886f640 1 -- start start 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.698+0000 7f55f886f640 10 monclient: init 2024-09-15T02:07:28.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 0x7f55f4130ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-c con 0x7f55f40980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40a58b0 0x7f55f4140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-b con 0x7f55f40a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f55f411dd10 0x7f55f4141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): picked mon.noname-a con 0x7f55f411dd10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4098dc0 con 0x7f55f411dd10 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f411f250 con 0x7f55f40a58b0 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f411ef10 con 0x7f55f40980a0 2024-09-15T02:07:28.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f886f640 10 monclient(hunting): authenticate will time out at 1708.695130s 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 0x7f55f4130ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 0x7f55f4130ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:41040/0 (socket says 172.21.15.17:41040) 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 -- 172.21.15.17:0/907452401 learned_addr learned my addr 172.21.15.17:0/907452401 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): get_auth_request con 0x7f55f40980a0 auth_method 0 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f55f411dd10 0x7f55f4141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): get_auth_request con 0x7f55f411dd10 auth_method 0 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40a58b0 0x7f55f4140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 10 monclient(hunting): get_auth_request con 0x7f55f40a58b0 auth_method 0 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient(hunting): handle_auth_done global_id 184391 payload 824 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40a58b0 msgr2=0x7f55f4140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40a58b0 0x7f55f4140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f55f411dd10 msgr2=0x7f55f4141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f55f411dd10 0x7f55f4141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient: _finish_hunting 0 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 monclient: found mon.noname-c 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f259c640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4009450 con 0x7f55f40980a0 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.699+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40a58b0 0x7f55f4140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f259c640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 msgr2=0x7f55f4130ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f259c640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 0x7f55f4130ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55e400ec80 tx=0x7f55e40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4009730 con 0x7f55f40980a0 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: dump: 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.718 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: _finish_auth 0 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e400eec0 con 0x7f55f40980a0 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f886f640 5 monclient: authenticate success, global_id 184391 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f886f640 10 monclient: _renew_subs 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4013470 con 0x7f55f40980a0 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f886f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec009a20 con 0x7f55f40980a0 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55e27fc640 10 monclient: dump: 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.719 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.701+0000 7f55f0d99640 4 set_mon_vals no callback set 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 10 monclient: _renew_subs 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f411ecb0 con 0x7f55f40980a0 2024-09-15T02:07:28.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 0x7f55f411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4063b20 con 0x7f55f40989e0 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.702+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f55e4013820 con 0x7f55f40980a0 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 0x7f55f411b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient: get_auth_request con 0x7f55f40989e0 auth_method 0 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55e27fc640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f55d40387c0 0x7f55d403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f55e404c9f0 con 0x7f55f40980a0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_done global_id 204151 payload 824 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f55d40387c0 0x7f55d403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f1d9b640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55f40724a0 con 0x7f55f40989e0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.703+0000 7f55f2d9d640 10 monclient: get_auth_request con 0x7f55d40387c0 auth_method 0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55f1d9b640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 msgr2=0x7f55f411b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 0x7f55f411b7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e800ee50 con 0x7f55f40989e0 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55e27fc640 10 monclient: dump: 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.722 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.705+0000 7f55f2d9d640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f55d40387c0 0x7f55d403ac70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f55f40a60e0 tx=0x7f55ec037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f55e800c020 con 0x7f55f40989e0 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 msgr2=0x7f55f411b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 0x7f55f411b7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e8008040 comp rx=0 tx=0).stop 2024-09-15T02:07:28.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 msgr2=0x7f55f411b7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.706+0000 7f55e27fc640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f40989e0 0x7f55f411b7d0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e8008040 comp rx=0 tx=0).stop 2024-09-15T02:07:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.711+0000 7f55f886f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.711+0000 7f55f886f640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 0x7f55f411c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.711+0000 7f55f886f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:28.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.711+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f411f050 con 0x7f55f411bd10 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 0x7f55f411c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient: get_auth_request con 0x7f55f411bd10 auth_method 0 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 10 monclient(hunting): handle_auth_done global_id 204154 payload 824 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.712+0000 7f55f1d9b640 1 -- 172.21.15.17:0/907452401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55f411c650 con 0x7f55f411bd10 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.714+0000 7f55f1d9b640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 msgr2=0x7f55f411c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.714+0000 7f55f1d9b640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 0x7f55f411c110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:28.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8021a50 con 0x7f55f411bd10 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 10 monclient: dump: 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:28.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7f55e8021bb0 con 0x7f55f411bd10 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 msgr2=0x7f55f411c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 0x7f55f411c110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e801b000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 msgr2=0x7f55f411c110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55e27fc640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f411bd10 0x7f55f411c110 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e780 tx=0x7f55e801b000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:28.728 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 162, 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.729 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.730 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:28.731 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:28.732 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.733 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.734 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:28.735 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.736 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:28.737 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.739 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:28.740 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f55d40387c0 msgr2=0x7f55d403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f55d40387c0 0x7f55d403ac70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f55f40a60e0 tx=0x7f55ec037000 comp rx=0 tx=0).stop 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 10 monclient: shutdown 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 msgr2=0x7f55f4130ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 --2- 172.21.15.17:0/907452401 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f55f40980a0 0x7f55f4130ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55e400ec80 tx=0x7f55e40091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f2d9d640 1 -- 172.21.15.17:0/907452401 reap_dead start 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f2d9d640 1 -- 172.21.15.17:0/907452401 reap_dead start 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 shutdown_connections 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 >> 172.21.15.17:0/907452401 conn(0x7f55f4070ed0 msgr2=0x7f55f4096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 shutdown_connections 2024-09-15T02:07:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:28.715+0000 7f55f886f640 1 -- 172.21.15.17:0/907452401 wait complete. 2024-09-15T02:07:28.743 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:07:28.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T02:07:28.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-15T02:07:28.743 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T02:07:28.743 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:07:28.743 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:07:28.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:07:28.755 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:07:28.755 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:07:28.755 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:07:28.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:07:48.756 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:07:48.757 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:07:48.757 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:07:48.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.863+0000 7faa93678640 10 monclient: get_monmap_and_config 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.863+0000 7faa93678640 10 monclient: build_initial_monmap 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.863+0000 7faa93678640 10 monclient: monmap: 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:48.864387+0000 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:48.864387+0000 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:48.865 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:48.866 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 Processor -- start 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 -- start start 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient: init 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa8c097320 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:48.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faa8c09a590 0x7faa8c1339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c09a590 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 0x7faa8c1362f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c133f10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0985d0 con 0x7faa8c097320 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c09d200 con 0x7faa8c09a590 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0992f0 con 0x7faa8c133f10 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa93678640 10 monclient(hunting): authenticate will time out at 1728.860930s 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.864+0000 7faa91c15640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 msgr2=0x7faa8c1362f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa91c15640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 0x7faa8c1362f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:48.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7faa8c09a590 0x7faa8c1339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa91414640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7faa8c097320 0x7faa8c0976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46682/0 (socket says 172.21.15.17:46682) 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa91414640 1 -- 172.21.15.17:0/1475505655 learned_addr learned my addr 172.21.15.17:0/1475505655 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): get_auth_request con 0x7faa8c09a590 auth_method 0 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient(hunting): handle_auth_done global_id 105645 payload 274 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 -- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 msgr2=0x7faa8c1362f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 --2- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 0x7faa8c1362f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 -- 172.21.15.17:0/1475505655 >> v1:172.21.15.17:6789/0 conn(0x7faa8c097320 legacy=0x7faa8c0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient: _finish_hunting 0 2024-09-15T02:07:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 monclient: found mon.noname-a 2024-09-15T02:07:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.865+0000 7faa90c13640 1 -- 172.21.15.17:0/1475505655 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c09cb90 con 0x7faa8c09a590 2024-09-15T02:07:48.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa90c13640 1 -- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c09a590 msgr2=0x7faa8c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:48.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa90c13640 1 --2- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c09a590 0x7faa8c1339d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009850 tx=0x7faa7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee3a75d0c76bf0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:48.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 1 -- 172.21.15.17:0/1475505655 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c032440 con 0x7faa8c09a590 2024-09-15T02:07:48.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: dump: 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 1 -- 172.21.15.17:0/1475505655 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c0325a0 con 0x7faa8c09a590 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 1 -- 172.21.15.17:0/1475505655 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c032940 con 0x7faa8c09a590 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 5 monclient: authenticate success, global_id 105645 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 4 set_mon_vals no callback set 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa83fff640 10 monclient: dump: 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.878 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 10 monclient: shutdown 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 1 -- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c09a590 msgr2=0x7faa8c1339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 1 --2- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c09a590 0x7faa8c1339d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009850 tx=0x7faa7c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 1 -- 172.21.15.17:0/1475505655 shutdown_connections 2024-09-15T02:07:48.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 1 --2- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c133f10 0x7faa8c1362f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.868+0000 7faa93678640 1 --2- 172.21.15.17:0/1475505655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c09a590 0x7faa8c1339d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 1 -- 172.21.15.17:0/1475505655 >> 172.21.15.17:0/1475505655 conn(0x7faa8c0a1000 msgr2=0x7faa8c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 1 -- 172.21.15.17:0/1475505655 shutdown_connections 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 1 -- 172.21.15.17:0/1475505655 wait complete. 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 10 monclient: build_initial_monmap 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 10 monclient: monmap: 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.869+0000 7faa93678640 1 Processor -- start 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 -- start start 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient: init 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c097320 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 0x7faa8c132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c09a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:48.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c133f10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c11fd70 con 0x7faa8c133f10 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1203c0 con 0x7faa8c097320 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c11e7c0 con 0x7faa8c09a590 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa93678640 10 monclient(hunting): authenticate will time out at 1728.866633s 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa90c13640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 msgr2=0x7faa8c132c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa90c13640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 0x7faa8c132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:38440/0 (socket says 172.21.15.17:38440) 2024-09-15T02:07:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 1 -- 172.21.15.17:0/2723855451 learned_addr learned my addr 172.21.15.17:0/2723855451 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): get_auth_request con 0x7faa8c133f10 auth_method 0 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91c15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91414640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91414640 10 monclient(hunting): get_auth_request con 0x7faa8c097320 auth_method 0 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91414640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.870+0000 7faa91414640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91414640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 msgr2=0x7faa8c141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:07:48.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91414640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 msgr2=0x7faa8c141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91414640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91414640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 10 monclient(hunting): handle_auth_done global_id 105648 payload 824 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 msgr2=0x7faa8c132c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 0x7faa8c132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 msgr2=0x7faa8c141b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 10 monclient: _finish_hunting 0 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 monclient: found mon.noname-a 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.871+0000 7faa91c15640 1 -- 172.21.15.17:0/2723855451 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c009390 con 0x7faa8c133f10 2024-09-15T02:07:48.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa91c15640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 msgr2=0x7faa8c11c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa91c15640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faa84009fc0 tx=0x7faa84009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84006150 con 0x7faa8c133f10 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: dump: 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:48.885 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: _finish_auth 0 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa840062f0 con 0x7faa8c133f10 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 5 monclient: authenticate success, global_id 105648 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 10 monclient: _renew_subs 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c09ad50 con 0x7faa8c133f10 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840064a0 con 0x7faa8c133f10 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa817fa640 10 monclient: dump: 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa837fe640 4 set_mon_vals no callback set 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 10 monclient: _renew_subs 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:48.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.873+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c11e240 con 0x7faa8c133f10 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.874+0000 7faa93678640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.874+0000 7faa93678640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.874+0000 7faa93678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.874+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0724a0 con 0x7faa8c133f10 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.874+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faa84012020 con 0x7faa8c133f10 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.875+0000 7faa817fa640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 0x7faa7003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.875+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faa8404c7f0 con 0x7faa8c133f10 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.875+0000 7faa91414640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 0x7faa7003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:48.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.875+0000 7faa91414640 10 monclient: get_auth_request con 0x7faa70038740 auth_method 0 2024-09-15T02:07:48.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.877+0000 7faa91414640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 0x7faa7003abf0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7faa88006fd0 tx=0x7faa88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:48.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.879+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faa84013510 con 0x7faa8c133f10 2024-09-15T02:07:48.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.879+0000 7faa817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:48.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:48.879+0000 7faa817fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:49.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.111+0000 7faa93678640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:49.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.111+0000 7faa93678640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:49.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.111+0000 7faa93678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:49.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.111+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faa8c09cfe0 con 0x7faa8c133f10 2024-09-15T02:07:49.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.112+0000 7faa817fa640 1 -- 172.21.15.17:0/2723855451 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7faa84007ae0 con 0x7faa8c133f10 2024-09-15T02:07:49.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.112+0000 7faa817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:49.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.112+0000 7faa817fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:49.114 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 msgr2=0x7faa7003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 0x7faa7003abf0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7faa88006fd0 tx=0x7faa88008040 comp rx=0 tx=0).stop 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 10 monclient: shutdown 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 msgr2=0x7faa8c11c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faa84009fc0 tx=0x7faa84009730 comp rx=0 tx=0).stop 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 shutdown_connections 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faa70038740 0x7faa7003abf0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faa8c133f10 0x7faa8c11c740 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faa8c09a590 0x7faa8c132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 --2- 172.21.15.17:0/2723855451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa8c097320 0x7faa8c141b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:49.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 >> 172.21.15.17:0/2723855451 conn(0x7faa8c0a1000 msgr2=0x7faa8c0a04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:49.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 shutdown_connections 2024-09-15T02:07:49.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:49.114+0000 7faa93678640 1 -- 172.21.15.17:0/2723855451 wait complete. 2024-09-15T02:07:52.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:07:52.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.233+0000 7f9b15109640 10 monclient: get_monmap_and_config 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.233+0000 7f9b15109640 10 monclient: build_initial_monmap 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.233+0000 7f9b15109640 10 monclient: monmap: 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:52.234142+0000 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:52.234142+0000 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:52.235 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:52.236 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:52.236 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:52.236 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:52.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.233+0000 7f9b15109640 1 Processor -- start 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.233+0000 7f9b15109640 1 -- start start 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient: init 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b1009c1b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1009e9e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b1006eb30 addr v1:172.21.15.17:6789/0 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1009f790 con 0x7f9b1006eb30 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b10068b60 con 0x7f9b1009c1b0 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b10069d40 con 0x7f9b1009e9e0 2024-09-15T02:07:52.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b15109640 10 monclient(hunting): authenticate will time out at 1732.230628s 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0f59e640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9b1006eb30 0x7f9b1006ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:46686/0 (socket says 172.21.15.17:46686) 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9b1009e9e0 auth_method 0 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0f59e640 1 -- 172.21.15.17:0/528962359 learned_addr learned my addr 172.21.15.17:0/528962359 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:52.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.234+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.236+0000 7f9b0f59e640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.17:6789/0 conn(0x7f9b1006eb30 legacy=0x7f9b1006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.237+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 legacy=0x7f9b1009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:52.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9b1009e9e0 auth_method 0 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:52.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.435+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9b1009e9e0 auth_method 0 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:07:52.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:52.836+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:07:53.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:53.639 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9b1009e9e0 auth_method 0 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:53.640 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.637+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:07:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.931+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 legacy=0x7f9b1009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T02:07:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.931+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 legacy=0x7f9b1009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:07:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.931+0000 7f9b0ed9d640 1 --1- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 0x7f9b1009a990 :-1 s=OPENED pgs=47 cs=1 l=1).handle_message read tag failed 2024-09-15T02:07:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.931+0000 7f9b0ed9d640 1 --1- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 0x7f9b1009a990 :-1 s=OPENED pgs=47 cs=1 l=1).fault on lossy channel, failing 2024-09-15T02:07:53.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:53.931+0000 7f9b0dd9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.67:6789/0 2024-09-15T02:07:54.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080876944 0 0) 0x7f9b1009f790 con 0x7f9b1006eb30 2024-09-15T02:07:54.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 10 monclient(hunting): my global_id is 105651 2024-09-15T02:07:54.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af80034e0 con 0x7f9b1006eb30 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465727046 0 0) 0x7f9af80034e0 con 0x7f9b1006eb30 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 msgr2=0x7f9b1009c580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> v2:172.21.15.17:3300/0 conn(0x7f9b1009e9e0 0x7f9b1009c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.67:6789/0 conn(0x7f9b1009c1b0 legacy=0x7f9b1009a990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.250+0000 7f9b0dd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10069aa0 con 0x7f9b1006eb30 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f9b08001aa0 con 0x7f9b1006eb30 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b15109640 5 monclient: authenticate success, global_id 105651 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b15109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: dump: 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.257 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 >> v1:172.21.15.17:6789/0 conn(0x7f9b1006eb30 legacy=0x7f9b1006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.b con 0x7f9af80039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004780 0x7f9af8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004780 msgr2=0x7f9af8004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.c con 0x7f9af8004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af8005250 0x7f9af800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004780 0x7f9af8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.a con 0x7f9af8005250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10069d40 con 0x7f9af8005250 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800c210 con 0x7f9af80039d0 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800c3d0 con 0x7f9af8004780 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b15109640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): get_auth_request con 0x7f9af80039d0 auth_method 0 2024-09-15T02:07:54.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af8005250 0x7f9af800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request con 0x7f9af8005250 auth_method 0 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 10 monclient(hunting): handle_auth_done global_id 105651 payload 293 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004780 msgr2=0x7f9af8004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004780 0x7f9af8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.251+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af8005250 msgr2=0x7f9af800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af8005250 0x7f9af800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 monclient: found mon.b 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af80034e0 con 0x7f9af80039d0 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af8005250 0x7f9af800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:07:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0f59e640 1 -- 172.21.15.17:0/528962359 reap_dead start 2024-09-15T02:07:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 msgr2=0x7f9af8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:07:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 msgr2=0x7f9af8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:07:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f9b00007e10 tx=0x7f9b00001ba0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:07:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.252+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 10 monclient: get_auth_request con 0x7f9af80039d0 auth_method 0 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 0 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient: ms_handle_reset current mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 msgr2=0x7f9af8003dc0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af80039d0 0x7f9af8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9af80099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.c con 0x7f9af8004bf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 0x7f9af800a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.a con 0x7f9af800a090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800ab80 0x7f9af800aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): picked mon.b con 0x7f9af800ab80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800b7f0 con 0x7f9af800a090 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800b960 con 0x7f9af800ab80 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800bb20 con 0x7f9af8004bf0 2024-09-15T02:07:54.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0dd9b640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 msgr2=0x7f9af80099c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.454+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9af80099c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 0x7f9af800a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9af800a090 auth_method 0 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800ab80 0x7f9af800aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request con 0x7f9af800ab80 auth_method 0 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:07:54.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient(hunting): handle_auth_done global_id 105651 payload 293 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 msgr2=0x7f9af80099c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9af80099c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800ab80 msgr2=0x7f9af800aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800ab80 0x7f9af800aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 monclient: found mon.a 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f9b1009f790 con 0x7f9af800a090 2024-09-15T02:07:54.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.455+0000 7f9b0e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0e59c640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 msgr2=0x7f9af800a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 0x7f9af800a460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9b04006ea0 tx=0x7f9b04008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fba7fed1b60ebf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04013e00 con 0x7f9af800a090 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 10 monclient: dump: 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.466 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 1 -- 172.21.15.17:0/528962359 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04011c50 con 0x7f9af800a090 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b15109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b0dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b15109640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.457+0000 7f9b15109640 4 set_mon_vals no callback set 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 10 monclient: shutdown 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 -- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 msgr2=0x7f9af800a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 0x7f9af800a460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9b04006ea0 tx=0x7f9b04008040 comp rx=0 tx=0).stop 2024-09-15T02:07:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 -- 172.21.15.17:0/528962359 shutdown_connections 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800a090 0x7f9af800a460 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800ab80 0x7f9af800aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 --2- 172.21.15.17:0/528962359 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9af80099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 -- 172.21.15.17:0/528962359 >> 172.21.15.17:0/528962359 conn(0x7f9b10071300 msgr2=0x7f9b100a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 -- 172.21.15.17:0/528962359 shutdown_connections 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.458+0000 7f9b15109640 1 -- 172.21.15.17:0/528962359 wait complete. 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.459+0000 7f9b15109640 10 monclient: build_initial_monmap 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.459+0000 7f9b15109640 10 monclient: monmap: 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:54.468 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 Processor -- start 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 -- start start 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient: init 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:07:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9b10141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-c con 0x7f9af8004bf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 0x7f9b10133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-b con 0x7f9af800a090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): picked mon.noname-a con 0x7f9af800ab80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1009cdb0 con 0x7f9af800ab80 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10134c80 con 0x7f9af800a090 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10121340 con 0x7f9af8004bf0 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b0ed9d640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 msgr2=0x7f9b10141340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b15109640 10 monclient(hunting): authenticate will time out at 1734.456907s 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.460+0000 7f9b0ed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9b10141340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60164/0 (socket says 172.21.15.17:60164) 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 -- 172.21.15.17:0/311785884 learned_addr learned my addr 172.21.15.17:0/311785884 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request con 0x7f9af800ab80 auth_method 0 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 0x7f9b10133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request con 0x7f9af800a090 auth_method 0 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient(hunting): handle_auth_done global_id 105654 payload 824 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 -- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 msgr2=0x7f9b10141340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9b10141340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 -- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 msgr2=0x7f9b10133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 0x7f9b10133f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 monclient: found mon.noname-a 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0f59e640 1 -- 172.21.15.17:0/311785884 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1011f560 con 0x7f9af800ab80 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.461+0000 7f9b0e59c640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 0x7f9b10133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.463+0000 7f9b0f59e640 1 -- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 msgr2=0x7f9b1011dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.463+0000 7f9b0f59e640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9b08001c80 tx=0x7f9b08009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0801d070 con 0x7f9af800ab80 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:54.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: dump: 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0800ad30 con 0x7f9af800ab80 2024-09-15T02:07:54.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 5 monclient: authenticate success, global_id 105654 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 10 monclient: _renew_subs 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b101210c0 con 0x7f9af800ab80 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0800aea0 con 0x7f9af800ab80 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9af6ffd640 10 monclient: dump: 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 10 monclient: _renew_subs 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b10141880 con 0x7f9af800ab80 2024-09-15T02:07:54.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.464+0000 7f9b0d59a640 4 set_mon_vals no callback set 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b08020020 con 0x7f9af800ab80 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b15109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b15109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9af6ffd640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 0x7f9afc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b15109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b10142200 con 0x7f9af800ab80 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9b08054bf0 con 0x7f9af800ab80 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 0x7f9afc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.465+0000 7f9b0ed9d640 10 monclient: get_auth_request con 0x7f9afc038720 auth_method 0 2024-09-15T02:07:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.468+0000 7f9b0ed9d640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 0x7f9afc03abd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9b00000ea0 tx=0x7f9b00001ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.470+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9b08023420 con 0x7f9af800ab80 2024-09-15T02:07:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.470+0000 7f9af6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.470+0000 7f9af6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9b15109640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9b15109640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9b15109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b100724a0 con 0x7f9af800ab80 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9af6ffd640 1 -- 172.21.15.17:0/311785884 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f9b08032080 con 0x7f9af800ab80 2024-09-15T02:07:54.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9af6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:07:54.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.700+0000 7f9af6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:54.703 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":166,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 msgr2=0x7f9afc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 0x7f9afc03abd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9b00000ea0 tx=0x7f9b00001ba0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 10 monclient: shutdown 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 msgr2=0x7f9b1011dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.702+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9b08001c80 tx=0x7f9b08009040 comp rx=0 tx=0).stop 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 shutdown_connections 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9afc038720 0x7f9afc03abd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9af800ab80 0x7f9b1011dad0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9af800a090 0x7f9b10133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 --2- 172.21.15.17:0/311785884 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9af8004bf0 0x7f9b10141340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 >> 172.21.15.17:0/311785884 conn(0x7f9b10071300 msgr2=0x7f9b100a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 shutdown_connections 2024-09-15T02:07:54.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.703+0000 7f9b15109640 1 -- 172.21.15.17:0/311785884 wait complete. 2024-09-15T02:07:54.715 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:07:54.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:07:54.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:07:54.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.810+0000 7fbe2739d640 10 monclient: get_monmap_and_config 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.810+0000 7fbe2739d640 10 monclient: build_initial_monmap 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.810+0000 7fbe2739d640 10 monclient: monmap: 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:54.811240+0000 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:54.811240+0000 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.812 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:54.813 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.810+0000 7fbe2739d640 1 Processor -- start 2024-09-15T02:07:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 -- start start 2024-09-15T02:07:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient: init 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe20131b20 0x7fbe2009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe20131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe200a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2009f1e0 0x7fbe20132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2009f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe200a0040 con 0x7fbe20131b20 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2009bff0 con 0x7fbe2009f1e0 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200a14a0 con 0x7fbe200a0280 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe2739d640 10 monclient(hunting): authenticate will time out at 1734.807709s 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe24938640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 msgr2=0x7fbe20131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe24938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe20131b20 0x7fbe2009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbe20131b20 0x7fbe2009eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60176/0 (socket says 172.21.15.17:60176) 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 1 -- 172.21.15.17:0/1705598299 learned_addr learned my addr 172.21.15.17:0/1705598299 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): get_auth_request con 0x7fbe20131b20 auth_method 0 2024-09-15T02:07:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.811+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe2593a640 1 --2- 172.21.15.17:0/1705598299 >> v2:172.21.15.67:3300/0 conn(0x7fbe2009f1e0 0x7fbe20132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe2593a640 10 monclient(hunting): get_auth_request con 0x7fbe2009f1e0 auth_method 0 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe2593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe2593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe2593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 10 monclient(hunting): handle_auth_done global_id 105657 payload 274 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 -- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 msgr2=0x7fbe20131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 --2- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 -- 172.21.15.17:0/1705598299 >> v2:172.21.15.67:3300/0 conn(0x7fbe2009f1e0 msgr2=0x7fbe20132430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 --2- 172.21.15.17:0/1705598299 >> v2:172.21.15.67:3300/0 conn(0x7fbe2009f1e0 0x7fbe20132430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 monclient: found mon.noname-a 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.812+0000 7fbe25139640 1 -- 172.21.15.17:0/1705598299 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20068e80 con 0x7fbe20131b20 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.814+0000 7fbe25139640 1 -- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 msgr2=0x7fbe2009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.814+0000 7fbe25139640 1 --2- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2009eca0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe200a2f90 tx=0x7fbe100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46be7e87edd17a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 1 -- 172.21.15.17:0/1705598299 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10032440 con 0x7fbe20131b20 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: dump: 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.827 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 1 -- 172.21.15.17:0/1705598299 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe100325a0 con 0x7fbe20131b20 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 5 monclient: authenticate success, global_id 105657 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 1 -- 172.21.15.17:0/1705598299 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10032940 con 0x7fbe20131b20 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 4 set_mon_vals no callback set 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe17fff640 10 monclient: dump: 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.829 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 10 monclient: shutdown 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 -- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 msgr2=0x7fbe2009eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2009eca0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe200a2f90 tx=0x7fbe100091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 -- 172.21.15.17:0/1705598299 shutdown_connections 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1705598299 >> v2:172.21.15.67:3300/0 conn(0x7fbe2009f1e0 0x7fbe20132430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1705598299 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2009eca0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 -- 172.21.15.17:0/1705598299 >> 172.21.15.17:0/1705598299 conn(0x7fbe20071270 msgr2=0x7fbe200a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 -- 172.21.15.17:0/1705598299 shutdown_connections 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.815+0000 7fbe2739d640 1 -- 172.21.15.17:0/1705598299 wait complete. 2024-09-15T02:07:54.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.816+0000 7fbe2739d640 10 monclient: build_initial_monmap 2024-09-15T02:07:54.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.816+0000 7fbe2739d640 10 monclient: monmap: 2024-09-15T02:07:54.831 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:54.831 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:54.831 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.816+0000 7fbe2739d640 1 Processor -- start 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.816+0000 7fbe2739d640 1 -- start start 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.816+0000 7fbe2739d640 10 monclient: init 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2009f1e0 0x7fbe20145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2009f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20130920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe200a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe20131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200a09b0 con 0x7fbe20131b20 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2009e750 con 0x7fbe2009f1e0 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2006e260 con 0x7fbe200a0280 2024-09-15T02:07:54.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2739d640 10 monclient(hunting): authenticate will time out at 1734.813270s 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe24938640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 msgr2=0x7fbe20130920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe24938640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20130920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60190/0 (socket says 172.21.15.17:60190) 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 learned_addr learned my addr 172.21.15.17:0/1969491158 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): get_auth_request con 0x7fbe20131b20 auth_method 0 2024-09-15T02:07:54.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2009f1e0 0x7fbe20145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): get_auth_request con 0x7fbe2009f1e0 auth_method 0 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient(hunting): handle_auth_done global_id 105660 payload 824 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 msgr2=0x7fbe20130920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe200a0280 0x7fbe20130920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2009f1e0 msgr2=0x7fbe20145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2009f1e0 0x7fbe20145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 monclient: found mon.noname-a 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.817+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10009450 con 0x7fbe20131b20 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.819+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 msgr2=0x7fbe2006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.819+0000 7fbe2593a640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2006abb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fa0 tx=0x7fbe18009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1800ee60 con 0x7fbe20131b20 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:54.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: dump: 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe18013590 con 0x7fbe20131b20 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18013950 con 0x7fbe20131b20 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 5 monclient: authenticate success, global_id 105660 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 monclient: _renew_subs 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2006e540 con 0x7fbe20131b20 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe157fa640 10 monclient: dump: 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe177fe640 4 set_mon_vals no callback set 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 monclient: _renew_subs 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20145a80 con 0x7fbe20131b20 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 0x7fbe2006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.820+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20063a40 con 0x7fbe2006d370 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 0x7fbe2006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe18013cc0 con 0x7fbe20131b20 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient: get_auth_request con 0x7fbe2006d370 auth_method 0 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe157fa640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe040386f0 0x7fbe0403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 10 monclient(hunting): handle_auth_done global_id 105663 payload 824 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe25139640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe200724a0 con 0x7fbe2006d370 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbe1804cdb0 con 0x7fbe20131b20 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe24938640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe040386f0 0x7fbe0403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.821+0000 7fbe24938640 10 monclient: get_auth_request con 0x7fbe040386f0 auth_method 0 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe25139640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 msgr2=0x7fbe2006b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe25139640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 0x7fbe2006b0f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fbe10034000 tx=0x7fbe10032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10031c00 con 0x7fbe2006d370 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.823+0000 7fbe157fa640 10 monclient: dump: 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.843 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbe10031d60 con 0x7fbe2006d370 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 msgr2=0x7fbe2006b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 0x7fbe2006b0f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fbe10034000 tx=0x7fbe10032000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 msgr2=0x7fbe2006b0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe157fa640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe2006d370 0x7fbe2006b0f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.824+0000 7fbe24938640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe040386f0 0x7fbe0403aba0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbe201315a0 tx=0x7fbe1c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe2739d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:54.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 0x7fbe2006b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe2739d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200a3cc0 con 0x7fbe20130e60 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 0x7fbe2006b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient: get_auth_request con 0x7fbe20130e60 auth_method 0 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 10 monclient(hunting): handle_auth_done global_id 105666 payload 824 2024-09-15T02:07:54.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.830+0000 7fbe25139640 1 -- 172.21.15.17:0/1969491158 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbe2006efc0 con 0x7fbe20130e60 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe25139640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 msgr2=0x7fbe2006b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe25139640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 0x7fbe2006b630 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fbe20071b40 tx=0x7fbe1003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe100468f0 con 0x7fbe20130e60 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.832+0000 7fbe157fa640 10 monclient: dump: 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.848 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7fbe100319f0 con 0x7fbe20130e60 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 msgr2=0x7fbe2006b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 0x7fbe2006b630 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fbe20071b40 tx=0x7fbe1003d000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 msgr2=0x7fbe2006b630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:54.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe157fa640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20130e60 0x7fbe2006b630 secure :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fbe20071b40 tx=0x7fbe1003d000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 166, 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:07:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:54.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:54.867 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:54.867 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe040386f0 msgr2=0x7fbe0403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe040386f0 0x7fbe0403aba0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbe201315a0 tx=0x7fbe1c006d80 comp rx=0 tx=0).stop 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 10 monclient: shutdown 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 msgr2=0x7fbe2006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 --2- 172.21.15.17:0/1969491158 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe20131b20 0x7fbe2006abb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fa0 tx=0x7fbe18009900 comp rx=0 tx=0).stop 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 reap_dead start 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2593a640 1 -- 172.21.15.17:0/1969491158 reap_dead start 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 shutdown_connections 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 >> 172.21.15.17:0/1969491158 conn(0x7fbe20071270 msgr2=0x7fbe2009cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 shutdown_connections 2024-09-15T02:07:54.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.833+0000 7fbe2739d640 1 -- 172.21.15.17:0/1969491158 wait complete. 2024-09-15T02:07:54.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:07:54.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:07:54.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:07:54.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.965+0000 7f8f6719d640 10 monclient: get_monmap_and_config 2024-09-15T02:07:54.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.965+0000 7f8f6719d640 10 monclient: build_initial_monmap 2024-09-15T02:07:54.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.966+0000 7f8f6719d640 10 monclient: monmap: 2024-09-15T02:07:54.967 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:07:54.966831+0000 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:07:54.966831+0000 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:07:54.968 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.966+0000 7f8f6719d640 1 Processor -- start 2024-09-15T02:07:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.966+0000 7f8f6719d640 1 -- start start 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.966+0000 7f8f6719d640 10 monclient: init 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f600a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f60132040 0x7f8f600987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f60132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f60136dd0 0x7f8f601391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f60136dd0 addr v2:172.21.15.67:3300/0 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f600a01e0 con 0x7f8f600a58b0 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f600a1580 con 0x7f8f60132040 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f600a5c80 con 0x7f8f60136dd0 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6719d640 10 monclient(hunting): authenticate will time out at 1734.963321s 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f60132040 0x7f8f600987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8f60132040 0x7f8f600987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60216/0 (socket says 172.21.15.17:60216) 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 -- 172.21.15.17:0/1631589061 learned_addr learned my addr 172.21.15.17:0/1631589061 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request con 0x7f8f60132040 auth_method 0 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6573a640 1 --2- 172.21.15.17:0/1631589061 >> v2:172.21.15.67:3300/0 conn(0x7f8f60136dd0 0x7f8f601391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6573a640 10 monclient(hunting): get_auth_request con 0x7f8f60136dd0 auth_method 0 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6573a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f6573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_done global_id 105669 payload 274 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 -- 172.21.15.17:0/1631589061 >> v2:172.21.15.67:3300/0 conn(0x7f8f60136dd0 msgr2=0x7f8f601391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/1631589061 >> v2:172.21.15.67:3300/0 conn(0x7f8f60136dd0 0x7f8f601391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 -- 172.21.15.17:0/1631589061 >> v1:172.21.15.67:6789/0 conn(0x7f8f600a58b0 legacy=0x7f8f600980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 monclient: found mon.noname-a 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.967+0000 7f8f5ffff640 1 -- 172.21.15.17:0/1631589061 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f60132440 con 0x7f8f60132040 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5ffff640 1 -- 172.21.15.17:0/1631589061 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f60132040 msgr2=0x7f8f600987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/1631589061 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f60132040 0x7f8f600987b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8f6006f830 tx=0x7f8f540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2ac0f9391df2d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 1 -- 172.21.15.17:0/1631589061 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54032440 con 0x7f8f60132040 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: dump: 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.979 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 1 -- 172.21.15.17:0/1631589061 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540325a0 con 0x7f8f60132040 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f6719d640 5 monclient: authenticate success, global_id 105669 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f6719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:07:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 1 -- 172.21.15.17:0/1631589061 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54032980 con 0x7f8f60132040 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f5f7fe640 10 monclient: dump: 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.981 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f6719d640 10 monclient: get_monmap_and_config success 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.970+0000 7f8f6719d640 4 set_mon_vals no callback set 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 10 monclient: shutdown 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 -- 172.21.15.17:0/1631589061 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f60132040 msgr2=0x7f8f600987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 --2- 172.21.15.17:0/1631589061 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f60132040 0x7f8f600987b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8f6006f830 tx=0x7f8f540091d0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 -- 172.21.15.17:0/1631589061 shutdown_connections 2024-09-15T02:07:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 --2- 172.21.15.17:0/1631589061 >> v2:172.21.15.67:3300/0 conn(0x7f8f60136dd0 0x7f8f601391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 --2- 172.21.15.17:0/1631589061 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f60132040 0x7f8f600987b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 -- 172.21.15.17:0/1631589061 >> 172.21.15.17:0/1631589061 conn(0x7f8f60070ed0 msgr2=0x7f8f600a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 -- 172.21.15.17:0/1631589061 shutdown_connections 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 1 -- 172.21.15.17:0/1631589061 wait complete. 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 10 monclient: build_initial_monmap 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.971+0000 7f8f6719d640 10 monclient: monmap: 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 1 Processor -- start 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 1 -- start start 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 10 monclient: init 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 10 monclient: _reopen_session rank -1 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 0x7f8f60132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f600a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f60132040 0x7f8f60133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.972+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f60132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:07:54.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60136dd0 0x7f8f60133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f60136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60098dc0 con 0x7f8f600a58b0 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600a3540 con 0x7f8f60136dd0 2024-09-15T02:07:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f5ffff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f60132040 msgr2=0x7f8f60133240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3301/0 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60136260 con 0x7f8f60132040 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): _renew_subs 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f6719d640 10 monclient(hunting): authenticate will time out at 1734.969049s 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f5ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f60132040 0x7f8f60133240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 0x7f8f60132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 0x7f8f60132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60218/0 (socket says 172.21.15.17:60218) 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 -- 172.21.15.17:0/348201441 learned_addr learned my addr 172.21.15.17:0/348201441 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): get_auth_request con 0x7f8f600a58b0 auth_method 0 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient(hunting): handle_auth_done global_id 105672 payload 824 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f60132040 msgr2=0x7f8f60133240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8f60132040 0x7f8f60133240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60136dd0 msgr2=0x7f8f60133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60136dd0 0x7f8f60133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient: _finish_hunting 0 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 monclient: found mon.noname-a 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.973+0000 7f8f64f39640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54009450 con 0x7f8f600a58b0 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f64f39640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 msgr2=0x7f8f60132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f64f39640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 0x7f8f60132d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000b4f0 tx=0x7f8f500090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5000f720 con 0x7f8f600a58b0 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: dump: 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.988 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: _finish_auth 0 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 5 monclient: authenticate success, global_id 105672 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000f880 con 0x7f8f600a58b0 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 10 monclient: _renew_subs 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f60134680 con 0x7f8f600a58b0 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5000fc40 con 0x7f8f600a58b0 2024-09-15T02:07:54.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5cff9640 10 monclient: dump: 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 10 monclient: _renew_subs 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f60133cc0 con 0x7f8f600a58b0 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.975+0000 7f8f5effd640 4 set_mon_vals no callback set 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8f50011610 con 0x7f8f600a58b0 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f6719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f6719d640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 0x7f8f60137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60063b20 con 0x7f8f60134d40 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5cff9640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f4c038770 0x7f8f4c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8f5004c160 con 0x7f8f600a58b0 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 0x7f8f60137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5ffff640 10 monclient: get_auth_request con 0x7f8f60134d40 auth_method 0 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.976+0000 7f8f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f6573a640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f4c038770 0x7f8f4c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f6573a640 10 monclient: get_auth_request con 0x7f8f4c038770 auth_method 0 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_done global_id 204169 payload 824 2024-09-15T02:07:54.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.977+0000 7f8f5ffff640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f60136960 con 0x7f8f60134d40 2024-09-15T02:07:54.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.979+0000 7f8f6573a640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f4c038770 0x7f8f4c03ac20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8f58007460 tx=0x7f8f58009a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.979+0000 7f8f5ffff640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 msgr2=0x7f8f60137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.979+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 0x7f8f60137210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f600a60e0 tx=0x7f8f54032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54032440 con 0x7f8f60134d40 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 10 monclient: dump: 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.995 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8f54032960 con 0x7f8f60134d40 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 msgr2=0x7f8f60137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 0x7f8f60137210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f600a60e0 tx=0x7f8f54032000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 msgr2=0x7f8f60137210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.980+0000 7f8f5cff9640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60134d40 0x7f8f60137210 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f600a60e0 tx=0x7f8f54032000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f6719d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:07:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f6719d640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 0x7f8f60137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f6719d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600724a0 con 0x7f8f60137750 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 0x7f8f60137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient: get_auth_request con 0x7f8f60137750 auth_method 0 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.986+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.987+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_done global_id 204172 payload 824 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.987+0000 7f8f5ffff640 1 -- 172.21.15.17:0/348201441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f6009fc90 con 0x7f8f60137750 2024-09-15T02:07:54.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5ffff640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 msgr2=0x7f8f60137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5ffff640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 0x7f8f60137b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009210 tx=0x7f8f5403c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54044a10 con 0x7f8f60137750 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 10 monclient: dump: 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:07:54.998 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f8f54044b70 con 0x7f8f60137750 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.989+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 msgr2=0x7f8f60137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f5cff9640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 0x7f8f60137b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009210 tx=0x7f8f5403c000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f5cff9640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 msgr2=0x7f8f60137b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f5cff9640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f60137750 0x7f8f60137b50 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009210 tx=0x7f8f5403c000 comp rx=0 tx=0).stop 2024-09-15T02:07:54.999 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 166, 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:07:55.000 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:07:55.001 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:07:55.002 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:07:55.003 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:07:55.004 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:55.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:55.007 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.008 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:07:55.009 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:07:55.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.011 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:07:55.012 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f4c038770 msgr2=0x7f8f4c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8f4c038770 0x7f8f4c03ac20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8f58007460 tx=0x7f8f58009a10 comp rx=0 tx=0).stop 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 10 monclient: shutdown 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 msgr2=0x7f8f60132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 --2- 172.21.15.17:0/348201441 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8f600a58b0 0x7f8f60132d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000b4f0 tx=0x7f8f500090f0 comp rx=0 tx=0).stop 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6573a640 1 -- 172.21.15.17:0/348201441 reap_dead start 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6573a640 1 -- 172.21.15.17:0/348201441 reap_dead start 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 shutdown_connections 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 >> 172.21.15.17:0/348201441 conn(0x7f8f60070ed0 msgr2=0x7f8f60096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:07:55.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 shutdown_connections 2024-09-15T02:07:55.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:07:54.990+0000 7f8f6719d640 1 -- 172.21.15.17:0/348201441 wait complete. 2024-09-15T02:07:55.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:07:55.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:08:15.017 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T02:08:15.017 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T02:08:15.017 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:08:15.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:08:15.022 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:08:15.022 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:08:15.023 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:08:15.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:08:35.024 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:08:35.024 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:08:35.025 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:08:35.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:35.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 10 monclient: get_monmap_and_config 2024-09-15T02:08:35.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 10 monclient: build_initial_monmap 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 10 monclient: monmap: 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:35.134910+0000 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:35.134910+0000 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:35.136 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:35.137 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:35.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 1 Processor -- start 2024-09-15T02:08:35.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 1 -- start start 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.134+0000 7f8951509640 10 monclient: init 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): picked mon.noname-c con 0x7f894c131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f894c0a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): picked mon.noname-a con 0x7f894c09f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c0a0040 con 0x7f894c0a0280 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894c09bff0 con 0x7f894c09f1e0 2024-09-15T02:08:35.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894c0a14a0 con 0x7f894c131b20 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f8951509640 10 monclient(hunting): authenticate will time out at 1775.131275s 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:36050/0 (socket says 172.21.15.17:36050) 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 learned_addr learned my addr 172.21.15.17:0/1223406992 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f894c09f1e0 auth_method 0 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f894c131b20 auth_method 0 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:35.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.137+0000 7f894a7fc640 1 -- 172.21.15.17:0/1223406992 >> v1:172.21.15.67:6789/0 conn(0x7f894c0a0280 legacy=0x7f894c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f894c09f1e0 auth_method 0 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:35.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f894c131b20 auth_method 0 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:35.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f894c09f1e0 auth_method 0 2024-09-15T02:08:35.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f894c131b20 auth_method 0 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:35.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:35.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:35.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:35.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f894c09f1e0 auth_method 0 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f894c131b20 auth_method 0 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.538+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:36.539+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:08:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): tick 2024-09-15T02:08:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 monclient(hunting): continuing hunt 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 msgr2=0x7f894c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.67:3300/0 conn(0x7f894c131b20 0x7f894c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 -- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 msgr2=0x7f894c138e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 --2- 172.21.15.17:0/1223406992 >> v2:172.21.15.17:3300/0 conn(0x7f894c09f1e0 0x7f894c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 -- 172.21.15.17:0/1223406992 >> v1:172.21.15.67:6789/0 conn(0x7f894c0a0280 legacy=0x7f894c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient: _add_conns ranks=[2,0] 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f89380036f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8938006240 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 -- 172.21.15.17:0/1223406992 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c132600 con 0x7f8938006240 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 1 -- 172.21.15.17:0/1223406992 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c09bff0 con 0x7f89380036f0 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f8948ff9640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89380036f0 auth_method 0 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:38.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.135+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:38.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.137+0000 7f894a7fc640 1 -- 172.21.15.17:0/1223406992 >> v1:172.21.15.17:6789/0 conn(0x7f8938006240 legacy=0x7f89380086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:38.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:38.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89380036f0 auth_method 0 2024-09-15T02:08:38.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:38.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:38.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:38.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:38.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:38.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:38.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.336+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89380036f0 auth_method 0 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:38.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:38.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:38.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:38.737+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89380036f0 auth_method 0 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:39.538+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:08:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.050+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806938018 0 0) 0x7f894c132600 con 0x7f8938006240 2024-09-15T02:08:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.050+0000 7f8949ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.050+0000 7f8949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.050+0000 7f8949ffb640 10 monclient(hunting): my global_id is 105675 2024-09-15T02:08:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.050+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89340034e0 con 0x7f8938006240 2024-09-15T02:08:40.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2137358685 0 0) 0x7f89340034e0 con 0x7f8938006240 2024-09-15T02:08:40.055 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 msgr2=0x7f8938005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89380036f0 0x7f8938005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: _finish_hunting 0 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893800b010 con 0x7f8938006240 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: _finish_auth 0 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f8940001770 con 0x7f8938006240 2024-09-15T02:08:40.056 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8951509640 5 monclient: authenticate success, global_id 105675 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8951509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: dump: 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:40.057 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 >> v1:172.21.15.17:6789/0 conn(0x7f8938006240 legacy=0x7f89380086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 1 -- 172.21.15.17:0/1223406992 reap_dead start 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 0x7f8934003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): picked mon.c con 0x7f89340039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:40.058 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f8934004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): picked mon.b con 0x7f8934004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 0x7f893400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): picked mon.a con 0x7f8934005250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c09bff0 con 0x7f8934005250 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893400c210 con 0x7f8934004780 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893400c3d0 con 0x7f89340039d0 2024-09-15T02:08:40.059 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8949ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f8951509640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 0x7f8934003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89340039d0 auth_method 0 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 0x7f893400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f8934005250 auth_method 0 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:40.060 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894a7fc640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f8934004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894a7fc640 10 monclient(hunting): get_auth_request con 0x7f8934004780 auth_method 0 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.051+0000 7f894b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:40.061 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:40.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:40.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 10 monclient(hunting): handle_auth_done global_id 105675 payload 293 2024-09-15T02:08:40.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 msgr2=0x7f8934004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:40.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f8934004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 msgr2=0x7f893400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 0x7f893400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 10 monclient: _finish_hunting 0 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 monclient: found mon.c 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89340034e0 con 0x7f89340039d0 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 10 monclient: _finish_auth 0 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:40.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.052+0000 7f894b7fe640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 0x7f893400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:40.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f894affd640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 msgr2=0x7f8934003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f894affd640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 0x7f8934003dc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f893c006ea0 tx=0x7f893c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2989707ad649e0d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c035410 con 0x7f89340039d0 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: dump: 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c02f030 con 0x7f89340039d0 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8951509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8951509640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 1 -- 172.21.15.17:0/1223406992 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c0357e0 con 0x7f89340039d0 2024-09-15T02:08:40.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8951509640 4 set_mon_vals no callback set 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.055+0000 7f8949ffb640 10 monclient: dump: 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 10 monclient: shutdown 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 -- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 msgr2=0x7f8934003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 0x7f8934003dc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f893c006ea0 tx=0x7f893c008040 comp rx=0 tx=0).stop 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 -- 172.21.15.17:0/1223406992 shutdown_connections 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8934005250 0x7f893400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f8934004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.075 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 --2- 172.21.15.17:0/1223406992 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f89340039d0 0x7f8934003dc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 -- 172.21.15.17:0/1223406992 >> 172.21.15.17:0/1223406992 conn(0x7f894c071270 msgr2=0x7f894c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 -- 172.21.15.17:0/1223406992 shutdown_connections 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.056+0000 7f8951509640 1 -- 172.21.15.17:0/1223406992 wait complete. 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.057+0000 7f8951509640 10 monclient: build_initial_monmap 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.057+0000 7f8951509640 10 monclient: monmap: 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:40.076 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.057+0000 7f8951509640 1 Processor -- start 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.057+0000 7f8951509640 1 -- start start 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.057+0000 7f8951509640 10 monclient: init 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): picked mon.noname-a con 0x7f89340039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f894c12ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): picked mon.noname-b con 0x7f8934004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.077 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): picked mon.noname-c con 0x7f8934005250 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c11c5a0 con 0x7f89340039d0 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c0a09b0 con 0x7f8934004780 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c09e1e0 con 0x7f8934005250 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f8951509640 10 monclient(hunting): authenticate will time out at 1780.054433s 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:36132/0 (socket says 172.21.15.17:36132) 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 1 -- 172.21.15.17:0/788262332 learned_addr learned my addr 172.21.15.17:0/788262332 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): get_auth_request con 0x7f89340039d0 auth_method 0 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 10 monclient(hunting): get_auth_request con 0x7f8934005250 auth_method 0 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f894c12ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): get_auth_request con 0x7f8934004780 auth_method 0 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:40.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 msgr2=0x7f894c098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 msgr2=0x7f894c098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894b7fe640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 10 monclient(hunting): handle_auth_done global_id 105678 payload 824 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 msgr2=0x7f894c098c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.058+0000 7f894affd640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 msgr2=0x7f894c12ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f894c12ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 10 monclient: _finish_hunting 0 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 1 monclient: found mon.noname-a 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894affd640 1 -- 172.21.15.17:0/788262332 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c09b340 con 0x7f89340039d0 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.059+0000 7f894a7fc640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f894c12ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f894affd640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 msgr2=0x7f894c140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f894affd640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f893c035410 tx=0x7f893c03a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c035800 con 0x7f89340039d0 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:40.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: dump: 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: _finish_auth 0 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c03f650 con 0x7f89340039d0 2024-09-15T02:08:40.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c03f9f0 con 0x7f89340039d0 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8932ffd640 10 monclient: dump: 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:40.083 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 5 monclient: authenticate success, global_id 105678 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 10 monclient: _renew_subs 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c141780 con 0x7f89340039d0 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f89497fa640 4 set_mon_vals no callback set 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 10 monclient: _renew_subs 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.061+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c09a770 con 0x7f89340039d0 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.062+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f893c04a3b0 con 0x7f89340039d0 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8951509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:40.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8951509640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8951509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c0724a0 con 0x7f89340039d0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8932ffd640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 0x7f893803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f893c06e740 con 0x7f89340039d0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f894a7fc640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 0x7f893803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.063+0000 7f894a7fc640 10 monclient: get_auth_request con 0x7f8938039160 auth_method 0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.066+0000 7f894a7fc640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 0x7f893803b610 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8940001920 tx=0x7f8940007940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.067+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f893c03d530 con 0x7f89340039d0 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.067+0000 7f8932ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.067+0000 7f8932ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8951509640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8951509640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8951509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f894c140ec0 con 0x7f89340039d0 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8932ffd640 1 -- 172.21.15.17:0/788262332 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f893c033e60 con 0x7f89340039d0 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8932ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:40.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.299+0000 7f8932ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:40.302 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":170,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 msgr2=0x7f893803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 0x7f893803b610 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8940001920 tx=0x7f8940007940 comp rx=0 tx=0).stop 2024-09-15T02:08:40.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 10 monclient: shutdown 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 msgr2=0x7f894c140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f893c035410 tx=0x7f893c03a000 comp rx=0 tx=0).stop 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 shutdown_connections 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8938039160 0x7f893803b610 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8934005250 0x7f894c098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934004780 0x7f894c12ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 --2- 172.21.15.17:0/788262332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f89340039d0 0x7f894c140920 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 >> 172.21.15.17:0/788262332 conn(0x7f894c071270 msgr2=0x7f894c1307a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 shutdown_connections 2024-09-15T02:08:40.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:40.301+0000 7f8951509640 1 -- 172.21.15.17:0/788262332 wait complete. 2024-09-15T02:08:43.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.422+0000 7faf2a660640 10 monclient: get_monmap_and_config 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.422+0000 7faf2a660640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.422+0000 7faf2a660640 10 monclient: monmap: 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:43.422946+0000 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:43.422946+0000 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:43.425 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.422+0000 7faf2a660640 1 Processor -- start 2024-09-15T02:08:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 1 -- start start 2024-09-15T02:08:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient: init 2024-09-15T02:08:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-b con 0x7faf24131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf240a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf2409f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240a0040 con 0x7faf2409f1e0 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2409bff0 con 0x7faf240a0280 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf240a14a0 con 0x7faf24131b20 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf2a660640 10 monclient(hunting): authenticate will time out at 1783.419557s 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf28bfd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7faf2409f1e0 0x7faf24138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33226/0 (socket says 172.21.15.17:33226) 2024-09-15T02:08:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57198/0 (socket says 172.21.15.17:57198) 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf28bfd640 1 -- 172.21.15.17:0/950319969 learned_addr learned my addr 172.21.15.17:0/950319969 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 10 monclient(hunting): get_auth_request con 0x7faf24131b20 auth_method 0 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 msgr2=0x7faf2409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 msgr2=0x7faf2409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.423+0000 7faf23fff640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.425+0000 7faf28bfd640 1 -- 172.21.15.17:0/950319969 >> v1:172.21.15.17:6789/0 conn(0x7faf2409f1e0 legacy=0x7faf24138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879115983 0 0) 0x7faf240a0040 con 0x7faf2409f1e0 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): my global_id is 105681 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf0c0034e0 con 0x7faf2409f1e0 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 -- 172.21.15.17:0/950319969 >> v1:172.21.15.67:6789/0 conn(0x7faf240a0280 legacy=0x7faf24131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 184071352 0 0) 0x7faf0c0034e0 con 0x7faf2409f1e0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 msgr2=0x7faf2409eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf2409eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 >> v1:172.21.15.67:6789/0 conn(0x7faf240a0280 legacy=0x7faf24131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf24132600 con 0x7faf2409f1e0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7faf1c001aa0 con 0x7faf2409f1e0 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf2a660640 5 monclient: authenticate success, global_id 105681 2024-09-15T02:08:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf2a660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: dump: 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.438 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 >> v1:172.21.15.17:6789/0 conn(0x7faf2409f1e0 legacy=0x7faf24138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf0c003b00 0x7faf0c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): picked mon.b con 0x7faf0c003b00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf0c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): picked mon.a con 0x7faf0c004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf0c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): picked mon.c con 0x7faf24131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240a14a0 con 0x7faf0c004780 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0055f0 con 0x7faf0c003b00 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c00bf40 con 0x7faf24131b20 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf2a660640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf0c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf0c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf0c003b00 0x7faf0c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): get_auth_request con 0x7faf0c004780 auth_method 0 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): get_auth_request con 0x7faf24131b20 auth_method 0 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): get_auth_request con 0x7faf0c003b00 auth_method 0 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient(hunting): handle_auth_done global_id 105681 payload 293 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 msgr2=0x7faf0c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf0c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf0c003b00 msgr2=0x7faf0c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf0c003b00 0x7faf0c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 monclient: found mon.a 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 1 -- 172.21.15.17:0/950319969 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c0034e0 con 0x7faf0c004780 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.426+0000 7faf28bfd640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf24131b20 0x7faf0c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf237fe640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 msgr2=0x7faf0c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf237fe640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf0c004bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf18001b00 tx=0x7faf18007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12541985d55f4730 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1802ac60 con 0x7faf0c004780 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: dump: 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf2a660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1802adc0 con 0x7faf0c004780 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 1 -- 172.21.15.17:0/950319969 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18007340 con 0x7faf0c004780 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf2a660640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf2a660640 4 set_mon_vals no callback set 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.429+0000 7faf22ffd640 10 monclient: dump: 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.445 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 10 monclient: shutdown 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 -- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 msgr2=0x7faf0c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 --2- 172.21.15.17:0/950319969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf0c004bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf18001b00 tx=0x7faf18007c40 comp rx=0 tx=0).stop 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf28bfd640 1 -- 172.21.15.17:0/950319969 reap_dead start 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 -- 172.21.15.17:0/950319969 shutdown_connections 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 -- 172.21.15.17:0/950319969 >> 172.21.15.17:0/950319969 conn(0x7faf24071270 msgr2=0x7faf240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 -- 172.21.15.17:0/950319969 shutdown_connections 2024-09-15T02:08:43.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.430+0000 7faf2a660640 1 -- 172.21.15.17:0/950319969 wait complete. 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.431+0000 7faf2a660640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.431+0000 7faf2a660640 10 monclient: monmap: 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:43.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.431+0000 7faf2a660640 1 Processor -- start 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.431+0000 7faf2a660640 1 -- start start 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.431+0000 7faf2a660640 10 monclient: init 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-a con 0x7faf0c004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-c con 0x7faf241416a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf241305c0 0x7faf24099f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): picked mon.noname-b con 0x7faf241305c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240a09b0 con 0x7faf0c004780 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2411d480 con 0x7faf241305c0 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf24142000 con 0x7faf241416a0 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf2a660640 10 monclient(hunting): authenticate will time out at 1783.428239s 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47748/0 (socket says 172.21.15.17:47748) 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 -- 172.21.15.17:0/113731698 learned_addr learned my addr 172.21.15.17:0/113731698 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): get_auth_request con 0x7faf0c004780 auth_method 0 2024-09-15T02:08:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 10 monclient(hunting): get_auth_request con 0x7faf241416a0 auth_method 0 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf241305c0 0x7faf24099f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 msgr2=0x7faf24130080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 msgr2=0x7faf24130080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): get_auth_request con 0x7faf241305c0 auth_method 0 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf237fe640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf28bfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient(hunting): handle_auth_done global_id 105684 payload 824 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 msgr2=0x7faf24130080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf241305c0 msgr2=0x7faf24099f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf241305c0 0x7faf24099f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 monclient: found mon.noname-a 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.432+0000 7faf23fff640 1 -- 172.21.15.17:0/113731698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2411b740 con 0x7faf0c004780 2024-09-15T02:08:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.434+0000 7faf23fff640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 msgr2=0x7faf24141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.434+0000 7faf23fff640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7faf14010ad0 tx=0x7faf14008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1400eca0 con 0x7faf0c004780 2024-09-15T02:08:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: dump: 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1400ee00 con 0x7faf0c004780 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 5 monclient: authenticate success, global_id 105684 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf140166b0 con 0x7faf0c004780 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 10 monclient: _renew_subs 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf240a4170 con 0x7faf0c004780 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf0bfff640 10 monclient: dump: 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf227fc640 4 set_mon_vals no callback set 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 10 monclient: _renew_subs 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.435+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2411c2b0 con 0x7faf0c004780 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faf14016aa0 con 0x7faf0c004780 2024-09-15T02:08:43.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf2a660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf2a660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf2a660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf240724a0 con 0x7faf0c004780 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf0bfff640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 0x7faf1003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faf1404d010 con 0x7faf0c004780 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf237fe640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 0x7faf1003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.436+0000 7faf237fe640 10 monclient: get_auth_request con 0x7faf10038740 auth_method 0 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.439+0000 7faf237fe640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 0x7faf1003abf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faf1802f000 tx=0x7faf18030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.440+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faf1407ee40 con 0x7faf0c004780 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.440+0000 7faf0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.440+0000 7faf0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf2a660640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf2a660640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf2a660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faf24131fa0 con 0x7faf0c004780 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf0bfff640 1 -- 172.21.15.17:0/113731698 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7faf14026530 con 0x7faf0c004780 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.673+0000 7faf0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:43.676 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:43.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 msgr2=0x7faf1003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 0x7faf1003abf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faf1802f000 tx=0x7faf18030000 comp rx=0 tx=0).stop 2024-09-15T02:08:43.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 10 monclient: shutdown 2024-09-15T02:08:43.677 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 msgr2=0x7faf24141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7faf14010ad0 tx=0x7faf14008040 comp rx=0 tx=0).stop 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 shutdown_connections 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faf10038740 0x7faf1003abf0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf241305c0 0x7faf24099f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faf241416a0 0x7faf24130080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 --2- 172.21.15.17:0/113731698 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faf0c004780 0x7faf24141160 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.675+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 >> 172.21.15.17:0/113731698 conn(0x7faf24071270 msgr2=0x7faf240a4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.676+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 shutdown_connections 2024-09-15T02:08:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.676+0000 7faf2a660640 1 -- 172.21.15.17:0/113731698 wait complete. 2024-09-15T02:08:43.688 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:08:43.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:08:43.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:08:43.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 10 monclient: get_monmap_and_config 2024-09-15T02:08:43.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 10 monclient: monmap: 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:43.784857+0000 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:43.784857+0000 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:43.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 1 Processor -- start 2024-09-15T02:08:43.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 1 -- start start 2024-09-15T02:08:43.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.784+0000 7f8117ea8640 10 monclient: init 2024-09-15T02:08:43.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f811006e920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f811006ecf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f811006f0c0 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811009aef0 con 0x7f811006f0c0 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811006f540 con 0x7f811006ecf0 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8110097a70 con 0x7f811006e920 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8117ea8640 10 monclient(hunting): authenticate will time out at 1783.781444s 2024-09-15T02:08:43.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57226/0 (socket says 172.21.15.17:57226) 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 -- 172.21.15.17:0/1219586611 learned_addr learned my addr 172.21.15.17:0/1219586611 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 10 monclient(hunting): get_auth_request con 0x7f811006e920 auth_method 0 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 msgr2=0x7f811012fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 msgr2=0x7f811012fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.785+0000 7f8115c44640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8116445640 1 -- 172.21.15.17:0/1219586611 >> v1:172.21.15.17:6789/0 conn(0x7f811006f0c0 legacy=0x7f8110097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8115443640 1 -- 172.21.15.17:0/1219586611 >> v1:172.21.15.67:6789/0 conn(0x7f811006ecf0 legacy=0x7f8110096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171050337 0 0) 0x7f811009aef0 con 0x7f811006f0c0 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8114c42640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8114c42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8114c42640 10 monclient(hunting): my global_id is 105687 2024-09-15T02:08:43.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.787+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0034e0 con 0x7f811006f0c0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421732116 0 0) 0x7f811006f540 con 0x7f811006ecf0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): my global_id is 204184 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811009aef0 con 0x7f811006ecf0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1542672669 0 0) 0x7f80fc0034e0 con 0x7f811006f0c0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 msgr2=0x7f811012fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f811012fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 >> v1:172.21.15.67:6789/0 conn(0x7f811006ecf0 legacy=0x7f8110096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:08:43.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811009d720 con 0x7f811006f0c0 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8117ea8640 5 monclient: authenticate success, global_id 105687 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f8108001770 con 0x7f811006f0c0 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8117ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: dump: 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 >> v1:172.21.15.17:6789/0 conn(0x7f811006f0c0 legacy=0x7f8110097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc003c10 0x7f80fc004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): picked mon.b con 0x7f80fc003c10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f80fc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): picked mon.a con 0x7f80fc004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f80fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): picked mon.c con 0x7f811006e920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110097a70 con 0x7f80fc004780 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0054b0 con 0x7f80fc003c10 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bf10 con 0x7f811006e920 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8114c42640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8117ea8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f80fc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): get_auth_request con 0x7f80fc004780 auth_method 0 2024-09-15T02:08:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f80fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc003c10 0x7f80fc004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): get_auth_request con 0x7f811006e920 auth_method 0 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): get_auth_request con 0x7f80fc003c10 auth_method 0 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8116445640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient(hunting): handle_auth_done global_id 105687 payload 293 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 msgr2=0x7f80fc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811006e920 0x7f80fc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc003c10 msgr2=0x7f80fc004000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc003c10 0x7f80fc004000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 monclient: found mon.a 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 1 -- 172.21.15.17:0/1219586611 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0034e0 con 0x7f80fc004780 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.788+0000 7f8115443640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8115443640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 msgr2=0x7f80fc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8115443640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f80fc004bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8100007d80 tx=0x7f8100001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c488ea185f614d05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810002ade0 con 0x7f80fc004780 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: dump: 2024-09-15T02:08:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81000070f0 con 0x7f80fc004780 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8117ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8117ea8640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 1 -- 172.21.15.17:0/1219586611 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000074b0 con 0x7f80fc004780 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8117ea8640 4 set_mon_vals no callback set 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.791+0000 7f8114c42640 10 monclient: dump: 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.809 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 10 monclient: shutdown 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 -- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 msgr2=0x7f80fc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 --2- 172.21.15.17:0/1219586611 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f80fc004bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8100007d80 tx=0x7f8100001b50 comp rx=0 tx=0).stop 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8116445640 1 -- 172.21.15.17:0/1219586611 reap_dead start 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 -- 172.21.15.17:0/1219586611 shutdown_connections 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 -- 172.21.15.17:0/1219586611 >> 172.21.15.17:0/1219586611 conn(0x7f81100a1000 msgr2=0x7f81100a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 -- 172.21.15.17:0/1219586611 shutdown_connections 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 1 -- 172.21.15.17:0/1219586611 wait complete. 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.792+0000 7f8117ea8640 10 monclient: monmap: 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 Processor -- start 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 -- start start 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient: init 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f811012ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.811 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f80fc004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811012f0f0 0x7f811013f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f811012f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 0x7f8110119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f811013f580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811009bbb0 con 0x7f80fc004780 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811011d2a0 con 0x7f811012f0f0 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811011aae0 con 0x7f811013f580 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8117ea8640 10 monclient(hunting): authenticate will time out at 1783.789631s 2024-09-15T02:08:43.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f811012ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f811012ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47758/0 (socket says 172.21.15.17:47758) 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 1 -- 172.21.15.17:0/1274210362 learned_addr learned my addr 172.21.15.17:0/1274210362 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): get_auth_request con 0x7f80fc004780 auth_method 0 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 0x7f8110119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 10 monclient(hunting): get_auth_request con 0x7f811013f580 auth_method 0 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115443640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811012f0f0 0x7f811013f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 msgr2=0x7f8110119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 msgr2=0x7f8110119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 0x7f8110119670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115443640 10 monclient(hunting): get_auth_request con 0x7f811012f0f0 auth_method 0 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115443640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115443640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8116445640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 0x7f8110119670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.793+0000 7f8115c44640 10 monclient(hunting): handle_auth_done global_id 105690 payload 824 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 msgr2=0x7f8110119670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f811013f580 0x7f8110119670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811012f0f0 msgr2=0x7f811013f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811012f0f0 0x7f811013f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 monclient: found mon.noname-a 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.794+0000 7f8115c44640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811012f4e0 con 0x7f80fc004780 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8115c44640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 msgr2=0x7f811012ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8115c44640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f811012ebb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f810c007e30 tx=0x7f810c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c016400 con 0x7f80fc004780 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: dump: 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c016560 con 0x7f80fc004780 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c016900 con 0x7f80fc004780 2024-09-15T02:08:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 5 monclient: authenticate success, global_id 105690 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 10 monclient: _renew_subs 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811013f9c0 con 0x7f80fc004780 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8107fff640 4 set_mon_vals no callback set 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8105ffb640 10 monclient: dump: 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 10 monclient: _renew_subs 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.796+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811011cfc0 con 0x7f80fc004780 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f810c016cb0 con 0x7f80fc004780 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8117ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8117ea8640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 0x7f8110029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110063c20 con 0x7f81100298c0 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 0x7f8110029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient: get_auth_request con 0x7f81100298c0 auth_method 0 2024-09-15T02:08:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8105ffb640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80f4038740 0x7f80f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f810c04d660 con 0x7f80fc004780 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 10 monclient(hunting): handle_auth_done global_id 105693 payload 824 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.797+0000 7f8115443640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81100724a0 con 0x7f81100298c0 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.798+0000 7f8116445640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80f4038740 0x7f80f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.798+0000 7f8116445640 10 monclient: get_auth_request con 0x7f80f4038740 auth_method 0 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8115443640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 msgr2=0x7f8110029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8115443640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 0x7f8110029c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8100032000 tx=0x7f8100030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100044070 con 0x7f81100298c0 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 10 monclient: dump: 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8100040470 con 0x7f81100298c0 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 msgr2=0x7f8110029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 0x7f8110029c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8100032000 tx=0x7f8100030000 comp rx=0 tx=0).stop 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 msgr2=0x7f8110029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8105ffb640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81100298c0 0x7f8110029c90 secure :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8100032000 tx=0x7f8100030000 comp rx=0 tx=0).stop 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.800+0000 7f8116445640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80f4038740 0x7f80f403abf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8110069740 tx=0x7f8108007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8117ea8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8117ea8640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 0x7f811002a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8117ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811006c470 con 0x7f811002a1d0 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 0x7f811002a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient: get_auth_request con 0x7f811002a1d0 auth_method 0 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 10 monclient(hunting): handle_auth_done global_id 105696 payload 824 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.806+0000 7f8115443640 1 -- 172.21.15.17:0/1274210362 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8110067290 con 0x7f811002a1d0 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8115443640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 msgr2=0x7f811002a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8115443640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 0x7f811002a5d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f81000014b0 tx=0x7f810003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810003a040 con 0x7f811002a1d0 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 10 monclient: dump: 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f81000339e0 con 0x7f811002a1d0 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 msgr2=0x7f811002a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 0x7f811002a5d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f81000014b0 tx=0x7f810003e000 comp rx=0 tx=0).stop 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 msgr2=0x7f811002a5d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8105ffb640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f811002a1d0 0x7f811002a5d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.827 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:43.828 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:43.830 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:43.831 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:43.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:43.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:43.839 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80f4038740 msgr2=0x7f80f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f80f4038740 0x7f80f403abf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8110069740 tx=0x7f8108007980 comp rx=0 tx=0).stop 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 10 monclient: shutdown 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 msgr2=0x7f811012ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 --2- 172.21.15.17:0/1274210362 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80fc004780 0x7f811012ebb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f810c007e30 tx=0x7f810c0091a0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8116445640 1 -- 172.21.15.17:0/1274210362 reap_dead start 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8116445640 1 -- 172.21.15.17:0/1274210362 reap_dead start 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 shutdown_connections 2024-09-15T02:08:43.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 >> 172.21.15.17:0/1274210362 conn(0x7f81100a1000 msgr2=0x7f8110132220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:43.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 shutdown_connections 2024-09-15T02:08:43.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.809+0000 7f8117ea8640 1 -- 172.21.15.17:0/1274210362 wait complete. 2024-09-15T02:08:43.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:43.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:08:43.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:08:43.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.938+0000 7fede5c68640 10 monclient: get_monmap_and_config 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.938+0000 7fede5c68640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.938+0000 7fede5c68640 10 monclient: monmap: 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:43.939179+0000 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:43.939179+0000 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:43.940 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:43.941 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:43.941 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:43.941 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:43.941 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 Processor -- start 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 -- start start 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient: init 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede00976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-b con 0x7fede0097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fede009a590 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fede009a960 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede012f4c0 con 0x7fede009a960 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00985d0 con 0x7fede009a590 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede009d200 con 0x7fede0097320 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7fede5c68640 10 monclient(hunting): authenticate will time out at 1783.935650s 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede00976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fede009a960 0x7fede012edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33248/0 (socket says 172.21.15.17:33248) 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 learned_addr learned my addr 172.21.15.17:0/2164554585 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 10 monclient(hunting): get_auth_request con 0x7fede0097320 auth_method 0 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede00976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.939+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede00976f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.941+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 >> v1:172.21.15.17:6789/0 conn(0x7fede009a960 legacy=0x7fede012edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 1 -- 172.21.15.17:0/2164554585 >> v1:172.21.15.67:6789/0 conn(0x7fede009a590 legacy=0x7fede012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413555113 0 0) 0x7fede012f4c0 con 0x7fede009a960 2024-09-15T02:08:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): my global_id is 105699 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc80034e0 con 0x7fede009a960 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733191969 0 0) 0x7fede00985d0 con 0x7fede009a590 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): my global_id is 204190 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede012f4c0 con 0x7fede009a590 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149503717 0 0) 0x7fedc80034e0 con 0x7fede009a960 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede00976f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede00976f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 >> v1:172.21.15.67:6789/0 conn(0x7fede009a590 legacy=0x7fede012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0097c30 con 0x7fede009a960 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fedd8001aa0 con 0x7fede009a960 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fede5c68640 5 monclient: authenticate success, global_id 105699 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fede5c68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: dump: 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.954 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 >> v1:172.21.15.17:6789/0 conn(0x7fede009a960 legacy=0x7fede012edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:08:43.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedc8003c10 0x7fedc8004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): picked mon.b con 0x7fedc8003c10 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedc8004780 0x7fedc8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): picked mon.a con 0x7fedc8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fedc800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): picked mon.c con 0x7fede0097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede009d200 con 0x7fedc8004780 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8005550 con 0x7fedc8003c10 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc800bf10 con 0x7fede0097320 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fedde7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7fede5c68640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fedc800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedc8004780 0x7fedc8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): get_auth_request con 0x7fede0097320 auth_method 0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): get_auth_request con 0x7fedc8004780 auth_method 0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedc8003c10 0x7fedc8004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): get_auth_request con 0x7fedc8003c10 auth_method 0 2024-09-15T02:08:43.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient(hunting): handle_auth_done global_id 105699 payload 293 2024-09-15T02:08:43.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedc8003c10 msgr2=0x7fedc8004000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedc8003c10 0x7fedc8004000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedc8004780 msgr2=0x7fedc8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedc8004780 0x7fedc8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.942+0000 7feddffff640 1 monclient: found mon.c 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc80034e0 con 0x7fede0097320 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddffff640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.943+0000 7feddeffd640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fedc8004780 0x7fedc8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fedc800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7feddffff640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fedc800b9d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fedd8001810 tx=0x7fedd800a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76dd143b7d440dde server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd802b630 con 0x7fede0097320 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: dump: 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fede5c68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd802b7b0 con 0x7fede0097320 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fede5c68640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fede5c68640 4 set_mon_vals no callback set 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 1 -- 172.21.15.17:0/2164554585 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd802bb70 con 0x7fede0097320 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.945+0000 7fedde7fc640 10 monclient: dump: 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 10 monclient: shutdown 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 -- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fedc800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 --2- 172.21.15.17:0/2164554585 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fedc800b9d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fedd8001810 tx=0x7fedd800a200 comp rx=0 tx=0).stop 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7feddffff640 1 -- 172.21.15.17:0/2164554585 reap_dead start 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 -- 172.21.15.17:0/2164554585 shutdown_connections 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 -- 172.21.15.17:0/2164554585 >> 172.21.15.17:0/2164554585 conn(0x7fede00a1000 msgr2=0x7fede00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 -- 172.21.15.17:0/2164554585 shutdown_connections 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.946+0000 7fede5c68640 1 -- 172.21.15.17:0/2164554585 wait complete. 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient: build_initial_monmap 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient: monmap: 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:43.963 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 1 Processor -- start 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 1 -- start start 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient: init 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-c con 0x7fede0097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede013ff70 0x7fede011d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-b con 0x7fede013ff70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 0x7fede011d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.947+0000 7fede5c68640 10 monclient(hunting): picked mon.noname-a con 0x7fede0140340 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede011f010 con 0x7fede0140340 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede009ad50 con 0x7fede013ff70 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede006ef30 con 0x7fede0097320 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7fede5c68640 10 monclient(hunting): authenticate will time out at 1783.944024s 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 0x7fede011d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57270/0 (socket says 172.21.15.17:57270) 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 learned_addr learned my addr 172.21.15.17:0/2879443731 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): get_auth_request con 0x7fede0140340 auth_method 0 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 10 monclient(hunting): get_auth_request con 0x7fede0097320 auth_method 0 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddeffd640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede013ff70 0x7fede011d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient(hunting): handle_auth_done global_id 105702 payload 824 2024-09-15T02:08:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 msgr2=0x7fede013fa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede0097320 0x7fede013fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede013ff70 msgr2=0x7fede011d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede013ff70 0x7fede011d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 monclient: found mon.noname-a 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.948+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede011eb60 con 0x7fede0140340 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 msgr2=0x7fede011d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7feddffff640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 0x7fede011d6c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005760 tx=0x7fedd8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd803c070 con 0x7fede0140340 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: dump: 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: _finish_auth 0 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8001a60 con 0x7fede0140340 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 5 monclient: authenticate success, global_id 105702 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 10 monclient: _renew_subs 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede011f6b0 con 0x7fede0140340 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8007360 con 0x7fede0140340 2024-09-15T02:08:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedc77fe640 10 monclient: dump: 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:43.973 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:43.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fedddffb640 4 set_mon_vals no callback set 2024-09-15T02:08:43.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 10 monclient: _renew_subs 2024-09-15T02:08:43.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:43.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.950+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede0120610 con 0x7fede0140340 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7fede5c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7fede5c68640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0063c20 con 0x7fede00298c0 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fedd8031070 con 0x7fede0140340 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 10 monclient: get_auth_request con 0x7fede00298c0 auth_method 0 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.951+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.952+0000 7fedc77fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedcc038740 0x7fedcc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.952+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fedd806b500 con 0x7fede0140340 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.952+0000 7feddeffd640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedcc038740 0x7fedcc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.952+0000 7feddeffd640 10 monclient: get_auth_request con 0x7fedcc038740 auth_method 0 2024-09-15T02:08:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:43.954+0000 7feddeffd640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedcc038740 0x7fedcc03abf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fedd40014e0 tx=0x7fedd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:44.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:44.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 10 monclient: get_auth_request con 0x7fede00298c0 auth_method 0 2024-09-15T02:08:44.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:44.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:44.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:44.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:08:44.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:44.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:44.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.152+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 10 monclient: get_auth_request con 0x7fede00298c0 auth_method 0 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:08:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:08:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:44.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:44.553+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:45.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient: get_auth_request con 0x7fede00298c0 auth_method 0 2024-09-15T02:08:45.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:45.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 10 monclient(hunting): handle_auth_done global_id 214100 payload 824 2024-09-15T02:08:45.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.354+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fede0120800 con 0x7fede00298c0 2024-09-15T02:08:45.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0016e70 con 0x7fede00298c0 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 10 monclient: dump: 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.359 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fedd0022420 con 0x7fede00298c0 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0008040 comp rx=0 tx=0).stop 2024-09-15T02:08:45.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 msgr2=0x7fede0029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:45.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.357+0000 7fedc77fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede00298c0 0x7fede0029c90 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0008040 comp rx=0 tx=0).stop 2024-09-15T02:08:45.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7fede5c68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:45.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7fede5c68640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 0x7fede002a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7fede5c68640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0120da0 con 0x7fede002a1d0 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 0x7fede002a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient: get_auth_request con 0x7fede002a1d0 auth_method 0 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 10 monclient(hunting): handle_auth_done global_id 214103 payload 824 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.364+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fede00724a0 con 0x7fede002a1d0 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7feddf7fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 msgr2=0x7fede002a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7feddf7fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 0x7fede002a5d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0025000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0034070 con 0x7fede002a1d0 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 10 monclient: dump: 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.372 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fedd0021d80 con 0x7fede002a1d0 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 msgr2=0x7fede002a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 0x7fede002a5d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0025000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 msgr2=0x7fede002a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.367+0000 7fedc77fe640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fede002a1d0 0x7fede002a5d0 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006e00 tx=0x7fedd0025000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:45.374 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 4, 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.375 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.376 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.377 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:45.379 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.380 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:45.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.382 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.383 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:45.385 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:45.386 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.387 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.389 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.390 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:45.390 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedcc038740 msgr2=0x7fedcc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fedcc038740 0x7fedcc03abf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fedd40014e0 tx=0x7fedd4009040 comp rx=0 tx=0).stop 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 10 monclient: shutdown 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 msgr2=0x7fede011d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 --2- 172.21.15.17:0/2879443731 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fede0140340 0x7fede011d6c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005760 tx=0x7fedd8031000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 reap_dead start 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7feddffff640 1 -- 172.21.15.17:0/2879443731 reap_dead start 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 shutdown_connections 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 >> 172.21.15.17:0/2879443731 conn(0x7fede00a1000 msgr2=0x7fede006e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 shutdown_connections 2024-09-15T02:08:45.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.368+0000 7fede5c68640 1 -- 172.21.15.17:0/2879443731 wait complete. 2024-09-15T02:08:45.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:45.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:08:45.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:08:45.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.497+0000 7f3ab4c6f640 10 monclient: get_monmap_and_config 2024-09-15T02:08:45.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.498+0000 7f3ab4c6f640 10 monclient: build_initial_monmap 2024-09-15T02:08:45.499 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.498+0000 7f3ab4c6f640 10 monclient: monmap: 2024-09-15T02:08:45.499 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:45.499 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:45.498849+0000 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:45.498849+0000 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:45.500 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.498+0000 7f3ab4c6f640 1 Processor -- start 2024-09-15T02:08:45.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.498+0000 7f3ab4c6f640 1 -- start start 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.498+0000 7f3ab4c6f640 10 monclient: init 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3ab01325e0 0x7f3ab0131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ab01325e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 0x7f3ab009ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ab00a0700 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ab0068e80 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab006a650 con 0x7f3ab0068e80 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab01329b0 con 0x7f3ab01325e0 2024-09-15T02:08:45.503 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab00a19b0 con 0x7f3ab00a0700 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3ab4c6f640 10 monclient(hunting): authenticate will time out at 1785.495209s 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f3ab01325e0 0x7f3ab0131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aaed9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f3ab0068e80 0x7f3ab0069320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33250/0 (socket says 172.21.15.17:33250) 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aaed9d640 1 -- 172.21.15.17:0/1093934904 learned_addr learned my addr 172.21.15.17:0/1093934904 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): get_auth_request con 0x7f3ab01325e0 auth_method 0 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 1 --2- 172.21.15.17:0/1093934904 >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 0x7f3ab009ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 10 monclient(hunting): get_auth_request con 0x7f3ab00a0700 auth_method 0 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient(hunting): handle_auth_done global_id 105705 payload 274 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 -- 172.21.15.17:0/1093934904 >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 msgr2=0x7f3ab009ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 --2- 172.21.15.17:0/1093934904 >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 0x7f3ab009ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 -- 172.21.15.17:0/1093934904 >> v1:172.21.15.17:6789/0 conn(0x7f3ab0068e80 legacy=0x7f3ab0069320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient: _finish_hunting 0 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 monclient: found mon.noname-a 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aae59c640 1 -- 172.21.15.17:0/1093934904 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab00a0280 con 0x7f3ab01325e0 2024-09-15T02:08:45.506 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.499+0000 7f3aa5d9b640 1 --2- 172.21.15.17:0/1093934904 >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 0x7f3ab009ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aae59c640 1 -- 172.21.15.17:0/1093934904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab01325e0 msgr2=0x7f3ab0131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aae59c640 1 --2- 172.21.15.17:0/1093934904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab01325e0 0x7f3ab0131d50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ab004b910 tx=0x7f3aa00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5920240f35bfbd00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 1 -- 172.21.15.17:0/1093934904 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa002add0 con 0x7f3ab01325e0 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: dump: 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: _finish_auth 0 2024-09-15T02:08:45.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 1 -- 172.21.15.17:0/1093934904 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa0032440 con 0x7f3ab01325e0 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 1 -- 172.21.15.17:0/1093934904 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00327e0 con 0x7f3ab01325e0 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3ab4c6f640 5 monclient: authenticate success, global_id 105705 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3ab4c6f640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3ab4c6f640 4 set_mon_vals no callback set 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.502+0000 7f3aadd9b640 10 monclient: dump: 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 10 monclient: shutdown 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/1093934904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab01325e0 msgr2=0x7f3ab0131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/1093934904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab01325e0 0x7f3ab0131d50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ab004b910 tx=0x7f3aa00090f0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/1093934904 shutdown_connections 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/1093934904 >> v2:172.21.15.67:3300/0 conn(0x7f3ab00a0700 0x7f3ab009ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/1093934904 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab01325e0 0x7f3ab0131d50 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/1093934904 >> 172.21.15.17:0/1093934904 conn(0x7f3ab0071320 msgr2=0x7f3ab00a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/1093934904 shutdown_connections 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.503+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/1093934904 wait complete. 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 10 monclient: build_initial_monmap 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 10 monclient: monmap: 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:45.515 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 1 Processor -- start 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 1 -- start start 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 10 monclient: init 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.504+0000 7f3ab4c6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 0x7f3ab0140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ab0068e80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab00a0700 0x7f3ab012fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ab00a0700 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 0x7f3ab0099b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ab01325e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.516 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab0141de0 con 0x7f3ab0068e80 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab009b6a0 con 0x7f3ab00a0700 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab009bec0 con 0x7f3ab01325e0 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3ab4c6f640 10 monclient(hunting): authenticate will time out at 1785.501084s 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 0x7f3ab0140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 0x7f3ab0140ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47818/0 (socket says 172.21.15.17:47818) 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 0x7f3ab0099b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 1 -- 172.21.15.17:0/545643860 learned_addr learned my addr 172.21.15.17:0/545643860 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): get_auth_request con 0x7f3ab0068e80 auth_method 0 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): get_auth_request con 0x7f3ab01325e0 auth_method 0 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab00a0700 0x7f3ab012fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): get_auth_request con 0x7f3ab00a0700 auth_method 0 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient(hunting): handle_auth_done global_id 214106 payload 824 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab00a0700 msgr2=0x7f3ab012fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab00a0700 0x7f3ab012fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 msgr2=0x7f3ab0140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 0x7f3ab0140ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 monclient: found mon.noname-c 2024-09-15T02:08:45.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aaed9d640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa0009390 con 0x7f3ab01325e0 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.505+0000 7f3aae59c640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3ab0068e80 0x7f3ab0140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aaed9d640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 msgr2=0x7f3ab0099b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aaed9d640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 0x7f3ab0099b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800cbc0 tx=0x7f3aa80052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800a850 con 0x7f3ab01325e0 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 10 monclient: dump: 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.520 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa800a9d0 con 0x7f3ab01325e0 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800ad90 con 0x7f3ab01325e0 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3ab4c6f640 5 monclient: authenticate success, global_id 214106 2024-09-15T02:08:45.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3ab4c6f640 10 monclient: _renew_subs 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab011d0f0 con 0x7f3ab01325e0 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3aa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3aa6ffd640 10 monclient: dump: 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.522 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.507+0000 7f3aad59a640 4 set_mon_vals no callback set 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 10 monclient: _renew_subs 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab0141420 con 0x7f3ab01325e0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 0x7f3ab009a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab0063b00 con 0x7f3ab011beb0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.508+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3aa8007300 con 0x7f3ab01325e0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 0x7f3ab009a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient: get_auth_request con 0x7f3ab011beb0 auth_method 0 2024-09-15T02:08:45.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aa6ffd640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a900387c0 0x7f3a9003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3aa804c630 con 0x7f3ab01325e0 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aa5d9b640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a900387c0 0x7f3a9003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aa5d9b640 10 monclient: get_auth_request con 0x7f3a900387c0 auth_method 0 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 10 monclient(hunting): handle_auth_done global_id 204199 payload 824 2024-09-15T02:08:45.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.509+0000 7f3aae59c640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ab011d5a0 con 0x7f3ab011beb0 2024-09-15T02:08:45.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.511+0000 7f3aae59c640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 msgr2=0x7f3ab009a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.511+0000 7f3aae59c640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 0x7f3ab009a0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0035040 tx=0x7f3aa0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa5d9b640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a900387c0 0x7f3a9003ac70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006e70 tx=0x7f3a98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa003c650 con 0x7f3ab011beb0 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 10 monclient: dump: 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.527 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3aa003c7b0 con 0x7f3ab011beb0 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 msgr2=0x7f3ab009a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 0x7f3ab009a0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0035040 tx=0x7f3aa0032000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 msgr2=0x7f3ab009a0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.512+0000 7f3aa6ffd640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab011beb0 0x7f3ab009a0b0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0035040 tx=0x7f3aa0032000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3ab4c6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 0x7f3ab009a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3ab4c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00724a0 con 0x7f3ab0130230 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 0x7f3ab009a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient: get_auth_request con 0x7f3ab0130230 auth_method 0 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 10 monclient(hunting): handle_auth_done global_id 204202 payload 824 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.518+0000 7f3aae59c640 1 -- 172.21.15.17:0/545643860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3ab006f0e0 con 0x7f3ab0130230 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aae59c640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 msgr2=0x7f3ab009a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aae59c640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 0x7f3ab009a5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0043000 tx=0x7f3aa003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0057070 con 0x7f3ab0130230 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 10 monclient: dump: 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f3aa0053430 con 0x7f3ab0130230 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 msgr2=0x7f3ab009a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 0x7f3ab009a5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0043000 tx=0x7f3aa003d000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 msgr2=0x7f3ab009a5f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aa6ffd640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab0130230 0x7f3ab009a5f0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0043000 tx=0x7f3aa003d000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 5, 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:45.544 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:45.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a900387c0 msgr2=0x7f3a9003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3a900387c0 0x7f3a9003ac70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006e70 tx=0x7f3a98008040 comp rx=0 tx=0).stop 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 10 monclient: shutdown 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 msgr2=0x7f3ab0099b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 0x7f3ab0099b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800cbc0 tx=0x7f3aa80052b0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.521+0000 7f3aaed9d640 1 -- 172.21.15.17:0/545643860 reap_dead start 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.522+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 shutdown_connections 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.522+0000 7f3ab4c6f640 1 --2- 172.21.15.17:0/545643860 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3ab01325e0 0x7f3ab0099b70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.522+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 >> 172.21.15.17:0/545643860 conn(0x7f3ab0071320 msgr2=0x7f3ab0132b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.522+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 shutdown_connections 2024-09-15T02:08:45.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.522+0000 7f3ab4c6f640 1 -- 172.21.15.17:0/545643860 wait complete. 2024-09-15T02:08:45.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:45.547 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:08:45.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 10 monclient: get_monmap_and_config 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 10 monclient: build_initial_monmap 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 10 monclient: monmap: 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:45.642026+0000 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:45.642026+0000 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:45.643 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:45.644 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:45.644 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:45.644 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:45.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 1 Processor -- start 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 1 -- start start 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.641+0000 7f01bf79d640 10 monclient: init 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01b80a58b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b8132040 0x7f01b80987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-b con 0x7f01b8132040 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f01b8136dd0 0x7f01b81391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-a con 0x7f01b8136dd0 addr v2:172.21.15.17:3300/0 2024-09-15T02:08:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80a01e0 con 0x7f01b80a58b0 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b80a1580 con 0x7f01b8136dd0 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80a5c80 con 0x7f01b8132040 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bf79d640 10 monclient(hunting): authenticate will time out at 1785.638496s 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f01b8136dd0 0x7f01b81391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b8132040 0x7f01b80987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f01b8136dd0 0x7f01b81391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47820/0 (socket says 172.21.15.17:47820) 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 -- 172.21.15.17:0/4092663586 learned_addr learned my addr 172.21.15.17:0/4092663586 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): get_auth_request con 0x7f01b8136dd0 auth_method 0 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): get_auth_request con 0x7f01b8132040 auth_method 0 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_done global_id 105711 payload 274 2024-09-15T02:08:45.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 -- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b8132040 msgr2=0x7f01b80987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 --2- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b8132040 0x7f01b80987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 -- 172.21.15.17:0/4092663586 >> v1:172.21.15.67:6789/0 conn(0x7f01b80a58b0 legacy=0x7f01b80980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient: _finish_hunting 0 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 monclient: found mon.noname-a 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.642+0000 7f01bdd3a640 1 -- 172.21.15.17:0/4092663586 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8132440 con 0x7f01b8136dd0 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01bdd3a640 1 -- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 msgr2=0x7f01b81391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01bdd3a640 1 --2- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b81391b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009f50 tx=0x7f01b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5562f1305042248 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 1 -- 172.21.15.17:0/4092663586 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0007610 con 0x7f01b8136dd0 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: dump: 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.654 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: _finish_auth 0 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 1 -- 172.21.15.17:0/4092663586 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b002d050 con 0x7f01b8136dd0 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 1 -- 172.21.15.17:0/4092663586 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00079a0 con 0x7f01b8136dd0 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01bf79d640 5 monclient: authenticate success, global_id 105711 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01bf79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01bf79d640 4 set_mon_vals no callback set 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:45.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.645+0000 7f01affff640 10 monclient: dump: 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 10 monclient: shutdown 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:45.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 -- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 msgr2=0x7f01b81391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 --2- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b81391b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009f50 tx=0x7f01b0009840 comp rx=0 tx=0).stop 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 -- 172.21.15.17:0/4092663586 shutdown_connections 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 --2- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b81391b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 --2- 172.21.15.17:0/4092663586 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b8132040 0x7f01b80987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 -- 172.21.15.17:0/4092663586 >> 172.21.15.17:0/4092663586 conn(0x7f01b8070ed0 msgr2=0x7f01b80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 -- 172.21.15.17:0/4092663586 shutdown_connections 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.646+0000 7f01bf79d640 1 -- 172.21.15.17:0/4092663586 wait complete. 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient: build_initial_monmap 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient: monmap: 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:45.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 Processor -- start 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 -- start start 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient: init 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-c con 0x7f01b80a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b8132040 0x7f01b8132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-b con 0x7f01b8132040 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): picked mon.noname-a con 0x7f01b8136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8136200 con 0x7f01b8136dd0 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8098dc0 con 0x7f01b8132040 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80a3070 con 0x7f01b80a58b0 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.647+0000 7f01bf79d640 10 monclient(hunting): authenticate will time out at 1785.643959s 2024-09-15T02:08:45.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47822/0 (socket says 172.21.15.17:47822) 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57336/0 (socket says 172.21.15.17:57336) 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 -- 172.21.15.17:0/3249118695 learned_addr learned my addr 172.21.15.17:0/3249118695 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): get_auth_request con 0x7f01b8136dd0 auth_method 0 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): get_auth_request con 0x7f01b80a58b0 auth_method 0 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b8132040 0x7f01b8132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): get_auth_request con 0x7f01b8132040 auth_method 0 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bcd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:45.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient(hunting): handle_auth_done global_id 105714 payload 824 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 -- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 msgr2=0x7f01b809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 -- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b8132040 msgr2=0x7f01b8132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b8132040 0x7f01b8132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient: _finish_hunting 0 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 monclient: found mon.noname-a 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bdd3a640 1 -- 172.21.15.17:0/3249118695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009ae0 con 0x7f01b8136dd0 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.648+0000 7f01bd539640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bdd3a640 1 -- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 msgr2=0x7f01b8133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bdd3a640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009880 tx=0x7f01b0036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0007170 con 0x7f01b8136dd0 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: dump: 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.664 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: _finish_auth 0 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0032ce0 con 0x7f01b8136dd0 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 5 monclient: authenticate success, global_id 105714 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 10 monclient: _renew_subs 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b81367c0 con 0x7f01b8136dd0 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b003b640 con 0x7f01b8136dd0 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01ad7fa640 10 monclient: dump: 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:45.665 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01af7fe640 4 set_mon_vals no callback set 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 10 monclient: _renew_subs 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.650+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b8133e30 con 0x7f01b8136dd0 2024-09-15T02:08:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01b0003030 con 0x7f01b8136dd0 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01ad7fa640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 0x7f019c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f01b006d4c0 con 0x7f01b8136dd0 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01bf79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01bf79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01bf79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.651+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b80724a0 con 0x7f01b8136dd0 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.652+0000 7f01bd539640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 0x7f019c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:45.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.652+0000 7f01bd539640 10 monclient: get_auth_request con 0x7f019c038740 auth_method 0 2024-09-15T02:08:45.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.654+0000 7f01bd539640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 0x7f019c03abf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009e40 tx=0x7f01b4006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:45.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.656+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01b0049080 con 0x7f01b8136dd0 2024-09-15T02:08:45.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.656+0000 7f01ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:45.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.656+0000 7f01ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.883+0000 7f01bf79d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:08:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.883+0000 7f01bf79d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:08:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.883+0000 7f01bf79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:45.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.883+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f01b809fc90 con 0x7f01b8136dd0 2024-09-15T02:08:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f01ad7fa640 1 -- 172.21.15.17:0/3249118695 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f01b005f080 con 0x7f01b8136dd0 2024-09-15T02:08:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f01ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:08:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f01ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=1493299555,config=3140514009,health=1557968630,kv=1529054990,logm=3955909124}) 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=10,kv=8,logm=50} crc {auth=302210786,config=3140514009,health=3830066417,kv=1529054990,logm=3058289399}) 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=1493299555,config=3140514009,health=1557968630,kv=1529054990,logm=3955909124}) 2024-09-15T02:08:45.886 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.884+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=1416486026,config=3140514009,health=1480848848,kv=1529054990,logm=334274768}) 2024-09-15T02:08:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 msgr2=0x7f019c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 0x7f019c03abf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009e40 tx=0x7f01b4006f50 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 10 monclient: shutdown 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 msgr2=0x7f01b8133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009880 tx=0x7f01b0036000 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 shutdown_connections 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f019c038740 0x7f019c03abf0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f01b8136dd0 0x7f01b8133040 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b8132040 0x7f01b8132b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 --2- 172.21.15.17:0/3249118695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f01b80a58b0 0x7f01b809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 >> 172.21.15.17:0/3249118695 conn(0x7f01b8070ed0 msgr2=0x7f01b809afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 shutdown_connections 2024-09-15T02:08:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:45.886+0000 7f01bf79d640 1 -- 172.21.15.17:0/3249118695 wait complete. 2024-09-15T02:08:45.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876158515}) 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=441153130}) 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876158515}) 2024-09-15T02:08:45.899 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.897+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278663564}) 2024-09-15T02:08:45.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846304697}) 2024-09-15T02:08:45.900 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3055556797}) 2024-09-15T02:08:45.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846304697}) 2024-09-15T02:08:45.901 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382211117}) 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239458362}) 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199351564}) 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239458362}) 2024-09-15T02:08:45.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531134007}) 2024-09-15T02:08:45.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397977426}) 2024-09-15T02:08:45.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2288167156}) 2024-09-15T02:08:45.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397977426}) 2024-09-15T02:08:45.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3150543045}) 2024-09-15T02:08:45.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910317294}) 2024-09-15T02:08:45.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967109425}) 2024-09-15T02:08:45.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910317294}) 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247645555}) 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395959194}) 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196585138}) 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395959194}) 2024-09-15T02:08:45.906 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1922245608}) 2024-09-15T02:08:45.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.962 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016149667}) 2024-09-15T02:08:45.963 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.960+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1598388981}) 2024-09-15T02:08:45.963 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.963 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016149667}) 2024-09-15T02:08:45.963 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.961+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389665866}) 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761420635}) 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966201079}) 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761420635}) 2024-09-15T02:08:45.967 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:45.965+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155393213}) 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462909760}) 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1439847819}) 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462909760}) 2024-09-15T02:08:46.030 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.028+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=176250276}) 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162837729}) 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330976557}) 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162837729}) 2024-09-15T02:08:46.119 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.117+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910184518}) 2024-09-15T02:08:46.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426436840}) 2024-09-15T02:08:46.122 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3008053707}) 2024-09-15T02:08:46.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426436840}) 2024-09-15T02:08:46.123 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.121+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405583644}) 2024-09-15T02:08:46.614 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.614 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499949263}) 2024-09-15T02:08:46.614 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=565492910}) 2024-09-15T02:08:46.615 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.615 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499949263}) 2024-09-15T02:08:46.615 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.612+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973158617}) 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334191498}) 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2669763644}) 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334191498}) 2024-09-15T02:08:46.616 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.613+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492971513}) 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927091998}) 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4091959022}) 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927091998}) 2024-09-15T02:08:46.617 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.614+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529158887}) 2024-09-15T02:08:46.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296730906}) 2024-09-15T02:08:46.618 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651031683}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296730906}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.615+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1271787887}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463542106}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590600680}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463542106}) 2024-09-15T02:08:46.619 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.616+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598054507}) 2024-09-15T02:08:46.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919876153}) 2024-09-15T02:08:46.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073370129}) 2024-09-15T02:08:46.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.621 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919876153}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.617+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244928563}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441748282}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333922960}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441748282}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.618+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20387968}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478518016}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363459297}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478518016}) 2024-09-15T02:08:46.622 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.619+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954609096}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981916857}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456247090}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981916857}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.620+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=132122318}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557415955}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835258379}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557415955}) 2024-09-15T02:08:46.625 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.621+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570360805}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306860218}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5404914}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306860218}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.622+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460310934}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828562470}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155743134}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828562470}) 2024-09-15T02:08:46.626 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.623+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380465657}) 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423334949}) 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481927632}) 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423334949}) 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.624+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949929133}) 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.628 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241065104}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1530303748}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241065104}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.625+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4138469060}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=18328444}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950642110}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=18328444}) 2024-09-15T02:08:46.629 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.626+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813626622}) 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528211955}) 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3571771101}) 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528211955}) 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.627+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096412690}) 2024-09-15T02:08:46.631 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202900486}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487404355}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202900486}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.628+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824520633}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026428514}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188884037}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026428514}) 2024-09-15T02:08:46.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.629+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549942992}) 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780840916}) 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880682108}) 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780840916}) 2024-09-15T02:08:46.633 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.630+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362006113}) 2024-09-15T02:08:46.899 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:08:46.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:46.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.902 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241802298}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413526788}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241802298}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.899+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972342010}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796378511}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997434459}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796378511}) 2024-09-15T02:08:46.903 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.900+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251019841}) 2024-09-15T02:08:46.904 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.901+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.901+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461353064}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.901+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138426035}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461353064}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.902+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304366744}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588179419}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747056698}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588179419}) 2024-09-15T02:08:46.905 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.903+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2117228329}) 2024-09-15T02:08:46.907 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.907 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501846758}) 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1458832935}) 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501846758}) 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.904+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112684453}) 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.908 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279765206}) 2024-09-15T02:08:46.909 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272732572}) 2024-09-15T02:08:46.909 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.909 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279765206}) 2024-09-15T02:08:46.909 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.905+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875470323}) 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409709891}) 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306294587}) 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409709891}) 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.906+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109640237}) 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.912 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642352900}) 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3414290694}) 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642352900}) 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.907+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030160534}) 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.913 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571283588}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3660602652}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571283588}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.908+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916395806}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747456075}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915545509}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747456075}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.909+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606112662}) 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.914 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468570554}) 2024-09-15T02:08:46.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890865513}) 2024-09-15T02:08:46.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468570554}) 2024-09-15T02:08:46.915 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.910+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186152645}) 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243369077}) 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753955396}) 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243369077}) 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.911+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154939266}) 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.918 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047702834}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614104671}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047702834}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.912+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201412829}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145181086}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945398766}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145181086}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.913+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597816171}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122840984}) 2024-09-15T02:08:46.919 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1659632359}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122840984}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.914+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=417359379}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729982244}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084937164}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729982244}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062110585}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.915+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308643283}) 2024-09-15T02:08:46.920 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742152029}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308643283}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4203694436}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274812899}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722304064}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274812899}) 2024-09-15T02:08:46.921 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.916+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720369261}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958028900}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152316987}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958028900}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.917+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494455375}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3600484351}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662767992}) 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.925 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3600484351}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.918+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894980662}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593324689}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007359685}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593324689}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.919+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79453828}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517037945}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005254284}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517037945}) 2024-09-15T02:08:46.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.920+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438820837}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671809927}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827915106}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671809927}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.921+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907163266}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032021316}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432991950}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032021316}) 2024-09-15T02:08:46.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.922+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009792573}) 2024-09-15T02:08:46.934 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.934 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575758180}) 2024-09-15T02:08:46.934 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362883458}) 2024-09-15T02:08:46.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575758180}) 2024-09-15T02:08:46.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.932+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264801459}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610214821}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770868447}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610214821}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.933+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184310887}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718280081}) 2024-09-15T02:08:46.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907860428}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718280081}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.934+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954219079}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267827558}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484173278}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267827558}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.935+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866477124}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489538279}) 2024-09-15T02:08:46.940 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889984044}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489538279}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.936+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470586356}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726773699}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1882169789}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726773699}) 2024-09-15T02:08:46.941 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.937+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961086936}) 2024-09-15T02:08:46.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086015054}) 2024-09-15T02:08:46.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2891297417}) 2024-09-15T02:08:46.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086015054}) 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.939+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570992586}) 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95610613}) 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103319273}) 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95610613}) 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.940+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976790123}) 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418136721}) 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3685952713}) 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418136721}) 2024-09-15T02:08:46.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.941+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282229187}) 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048262135}) 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442350698}) 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048262135}) 2024-09-15T02:08:46.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.942+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229401691}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204320017}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029833624}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204320017}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.943+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504943989}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770906757}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=501620752}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770906757}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.944+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790319786}) 2024-09-15T02:08:46.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814271009}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4000065791}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814271009}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.945+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752164987}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096808068}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779937165}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096808068}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.947+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535374019}) 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728162991}) 2024-09-15T02:08:46.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690797422}) 2024-09-15T02:08:46.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728162991}) 2024-09-15T02:08:46.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.948+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968513750}) 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666190988}) 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512263277}) 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666190988}) 2024-09-15T02:08:46.968 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.966+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218750520}) 2024-09-15T02:08:46.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21447792}) 2024-09-15T02:08:46.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204203096}) 2024-09-15T02:08:46.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:46.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21447792}) 2024-09-15T02:08:46.982 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:46.979+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021311514}) 2024-09-15T02:08:47.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 10 monclient: get_monmap_and_config 2024-09-15T02:08:47.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 10 monclient: monmap: 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:47.019906+0000 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:47.019906+0000 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:47.021 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 1 Processor -- start 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 1 -- start start 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.019+0000 7fcd7b39d640 10 monclient: init 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd74131b20 0x7fcd7409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd74131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcd740a0280 0x7fcd74131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd740a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd7409f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd740a0040 con 0x7fcd7409f1e0 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7409bff0 con 0x7fcd740a0280 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd740a14a0 con 0x7fcd74131b20 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd7b39d640 10 monclient(hunting): authenticate will time out at 1787.016442s 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcd740a0280 0x7fcd74131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fcd740a0280 0x7fcd74131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47828/0 (socket says 172.21.15.17:47828) 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 -- 172.21.15.17:0/4156362944 learned_addr learned my addr 172.21.15.17:0/4156362944 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): get_auth_request con 0x7fcd740a0280 auth_method 0 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd79139640 1 --2- 172.21.15.17:0/4156362944 >> v2:172.21.15.67:3300/0 conn(0x7fcd74131b20 0x7fcd7409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd79139640 10 monclient(hunting): get_auth_request con 0x7fcd74131b20 auth_method 0 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd79139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd79139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd79139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient(hunting): handle_auth_done global_id 105717 payload 274 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 -- 172.21.15.17:0/4156362944 >> v2:172.21.15.67:3300/0 conn(0x7fcd74131b20 msgr2=0x7fcd7409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 --2- 172.21.15.17:0/4156362944 >> v2:172.21.15.67:3300/0 conn(0x7fcd74131b20 0x7fcd7409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 -- 172.21.15.17:0/4156362944 >> v1:172.21.15.67:6789/0 conn(0x7fcd7409f1e0 legacy=0x7fcd74068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 monclient: found mon.noname-a 2024-09-15T02:08:47.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.020+0000 7fcd78938640 1 -- 172.21.15.17:0/4156362944 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd74132430 con 0x7fcd740a0280 2024-09-15T02:08:47.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd78938640 1 -- 172.21.15.17:0/4156362944 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd740a0280 msgr2=0x7fcd74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.036 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd78938640 1 --2- 172.21.15.17:0/4156362944 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd740a0280 0x7fcd74131ef0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fcd7409f710 tx=0x7fcd680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94e7e663b61c866 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 1 -- 172.21.15.17:0/4156362944 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68032440 con 0x7fcd740a0280 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: dump: 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.037 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd7b39d640 5 monclient: authenticate success, global_id 105717 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd7b39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 1 -- 172.21.15.17:0/4156362944 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd680325a0 con 0x7fcd740a0280 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd7b39d640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd7b39d640 4 set_mon_vals no callback set 2024-09-15T02:08:47.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 1 -- 172.21.15.17:0/4156362944 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68032940 con 0x7fcd740a0280 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.023+0000 7fcd67fff640 10 monclient: dump: 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 10 monclient: shutdown 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 -- 172.21.15.17:0/4156362944 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd740a0280 msgr2=0x7fcd74131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/4156362944 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd740a0280 0x7fcd74131ef0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fcd7409f710 tx=0x7fcd680091d0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 -- 172.21.15.17:0/4156362944 shutdown_connections 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/4156362944 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd740a0280 0x7fcd74131ef0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/4156362944 >> v2:172.21.15.67:3300/0 conn(0x7fcd74131b20 0x7fcd7409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 -- 172.21.15.17:0/4156362944 >> 172.21.15.17:0/4156362944 conn(0x7fcd74071270 msgr2=0x7fcd740a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 -- 172.21.15.17:0/4156362944 shutdown_connections 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.024+0000 7fcd7b39d640 1 -- 172.21.15.17:0/4156362944 wait complete. 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.025+0000 7fcd7b39d640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.025+0000 7fcd7b39d640 10 monclient: monmap: 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.025+0000 7fcd7b39d640 1 Processor -- start 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.025+0000 7fcd7b39d640 1 -- start start 2024-09-15T02:08:47.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.025+0000 7fcd7b39d640 10 monclient: init 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd7409f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 0x7fcd7412eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd740a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd74131b20 0x7fcd74098d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd74131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7411bfc0 con 0x7fcd7409f1e0 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd740a09b0 con 0x7fcd740a0280 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74141fb0 con 0x7fcd74131b20 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7b39d640 10 monclient(hunting): authenticate will time out at 1787.022121s 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47832/0 (socket says 172.21.15.17:47832) 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 -- 172.21.15.17:0/2992202750 learned_addr learned my addr 172.21.15.17:0/2992202750 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): get_auth_request con 0x7fcd7409f1e0 auth_method 0 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd74131b20 0x7fcd74098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): get_auth_request con 0x7fcd74131b20 auth_method 0 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 0x7fcd7412eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd7993a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 10 monclient(hunting): get_auth_request con 0x7fcd740a0280 auth_method 0 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient(hunting): handle_auth_done global_id 105720 payload 824 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 -- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd74131b20 msgr2=0x7fcd74098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd74131b20 0x7fcd74098d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 -- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 msgr2=0x7fcd7412eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 0x7fcd7412eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 monclient: found mon.noname-a 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd79139640 1 -- 172.21.15.17:0/2992202750 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68009450 con 0x7fcd7409f1e0 2024-09-15T02:08:47.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.026+0000 7fcd78938640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 0x7fcd7412eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd79139640 1 -- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 msgr2=0x7fcd741410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd79139640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d890 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000de20 con 0x7fcd7409f1e0 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: dump: 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.046 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7000bae0 con 0x7fcd7409f1e0 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000be80 con 0x7fcd7409f1e0 2024-09-15T02:08:47.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd7b39d640 5 monclient: authenticate success, global_id 105720 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd657fa640 10 monclient: dump: 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.048 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 10 monclient: _renew_subs 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.028+0000 7fcd677fe640 4 set_mon_vals no callback set 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd740a5360 con 0x7fcd7409f1e0 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 10 monclient: _renew_subs 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7409b430 con 0x7fcd7409f1e0 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.029+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcd70014640 con 0x7fcd7409f1e0 2024-09-15T02:08:47.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd7b39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd7b39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd7b39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7411c430 con 0x7fcd7409f1e0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd657fa640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 0x7fcd5803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fcd7004cd00 con 0x7fcd7409f1e0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd78938640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 0x7fcd5803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.030+0000 7fcd78938640 10 monclient: get_auth_request con 0x7fcd58038740 auth_method 0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.033+0000 7fcd78938640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 0x7fcd5803abf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcd740a13a0 tx=0x7fcd68032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.034+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcd7001eb40 con 0x7fcd7409f1e0 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.034+0000 7fcd657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.034+0000 7fcd657fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351424227}) 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218268832}) 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351424227}) 2024-09-15T02:08:47.051 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.041+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397177211}) 2024-09-15T02:08:47.091 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.091 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788718806}) 2024-09-15T02:08:47.092 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136868497}) 2024-09-15T02:08:47.092 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.092 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788718806}) 2024-09-15T02:08:47.092 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010262431}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879884968}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389803403}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879884968}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.090+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975202510}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264016868}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=293942473}) 2024-09-15T02:08:47.093 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.094 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264016868}) 2024-09-15T02:08:47.094 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.091+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183808955}) 2024-09-15T02:08:47.094 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.094 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716517750}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426435873}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716517750}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.092+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236276083}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743931857}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700182054}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743931857}) 2024-09-15T02:08:47.095 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.093+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243665224}) 2024-09-15T02:08:47.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd7b39d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:47.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd7b39d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:47.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd7b39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd740724a0 con 0x7fcd7409f1e0 2024-09-15T02:08:47.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd657fa640 1 -- 172.21.15.17:0/2992202750 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fcd7001dbb0 con 0x7fcd7409f1e0 2024-09-15T02:08:47.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:47.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.265+0000 7fcd657fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:47.267 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 msgr2=0x7fcd5803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 0x7fcd5803abf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcd740a13a0 tx=0x7fcd68032000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 10 monclient: shutdown 2024-09-15T02:08:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 msgr2=0x7fcd741410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d890 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 shutdown_connections 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd58038740 0x7fcd5803abf0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd74131b20 0x7fcd74098d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd740a0280 0x7fcd7412eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 --2- 172.21.15.17:0/2992202750 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd7409f1e0 0x7fcd741410d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 >> 172.21.15.17:0/2992202750 conn(0x7fcd74071270 msgr2=0x7fcd7409d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 shutdown_connections 2024-09-15T02:08:47.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.267+0000 7fcd7b39d640 1 -- 172.21.15.17:0/2992202750 wait complete. 2024-09-15T02:08:47.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:08:47.280 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:08:47.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:08:47.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:08:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.415+0000 7f1572e3f640 10 monclient: get_monmap_and_config 2024-09-15T02:08:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.415+0000 7f1572e3f640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.415+0000 7f1572e3f640 10 monclient: monmap: 2024-09-15T02:08:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:47.416454+0000 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:47.416454+0000 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:47.418 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 1 Processor -- start 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 1 -- start start 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 10 monclient: init 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.416+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-b con 0x7f156c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f156c06f440 0x7f156c09bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-c con 0x7f156c06f440 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f156c09c490 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c069c60 con 0x7f156c09c490 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c06a920 con 0x7f156c06f440 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c09f690 con 0x7f156c06eb30 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1572e3f640 10 monclient(hunting): authenticate will time out at 1787.413121s 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57374/0 (socket says 172.21.15.17:57374) 2024-09-15T02:08:47.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 -- 172.21.15.17:0/427505424 learned_addr learned my addr 172.21.15.17:0/427505424 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): get_auth_request con 0x7f156c06eb30 auth_method 0 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f15713dc640 1 --1- 172.21.15.17:0/427505424 >> v1:172.21.15.17:6789/0 conn(0x7f156c09c490 0x7f156c09d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33258/0 (socket says 172.21.15.17:33258) 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f156bfff640 1 --2- 172.21.15.17:0/427505424 >> v2:172.21.15.67:3300/0 conn(0x7f156c06f440 0x7f156c09bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f156bfff640 10 monclient(hunting): get_auth_request con 0x7f156c06f440 auth_method 0 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f156bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f156bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f156bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient(hunting): handle_auth_done global_id 214118 payload 274 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 -- 172.21.15.17:0/427505424 >> v2:172.21.15.67:3300/0 conn(0x7f156c06f440 msgr2=0x7f156c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 --2- 172.21.15.17:0/427505424 >> v2:172.21.15.67:3300/0 conn(0x7f156c06f440 0x7f156c09bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 -- 172.21.15.17:0/427505424 >> v1:172.21.15.17:6789/0 conn(0x7f156c09c490 legacy=0x7f156c09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 monclient: found mon.noname-b 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.417+0000 7f1570bdb640 1 -- 172.21.15.17:0/427505424 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c09ad40 con 0x7f156c06eb30 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f1570bdb640 1 -- 172.21.15.17:0/427505424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 msgr2=0x7f156c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f1570bdb640 1 --2- 172.21.15.17:0/427505424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1560009910 tx=0x7f15600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88dc6d685be9bb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 1 -- 172.21.15.17:0/427505424 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156002aea0 con 0x7f156c06eb30 2024-09-15T02:08:47.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: dump: 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 1 -- 172.21.15.17:0/427505424 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1560032440 con 0x7f156c06eb30 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 1 -- 172.21.15.17:0/427505424 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15600327e0 con 0x7f156c06eb30 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f1572e3f640 5 monclient: authenticate success, global_id 214118 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f1572e3f640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f1572e3f640 4 set_mon_vals no callback set 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.420+0000 7f156b7fe640 10 monclient: dump: 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.428 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 10 monclient: shutdown 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 -- 172.21.15.17:0/427505424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 msgr2=0x7f156c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 --2- 172.21.15.17:0/427505424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1560009910 tx=0x7f15600091d0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 -- 172.21.15.17:0/427505424 shutdown_connections 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 --2- 172.21.15.17:0/427505424 >> v2:172.21.15.67:3300/0 conn(0x7f156c06f440 0x7f156c09bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 --2- 172.21.15.17:0/427505424 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c06ef00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 -- 172.21.15.17:0/427505424 >> 172.21.15.17:0/427505424 conn(0x7f156c0712f0 msgr2=0x7f156c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 -- 172.21.15.17:0/427505424 shutdown_connections 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 -- 172.21.15.17:0/427505424 wait complete. 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 10 monclient: monmap: 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.430 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.421+0000 7f1572e3f640 1 Processor -- start 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 -- start start 2024-09-15T02:08:47.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient: init 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-c con 0x7f156c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 0x7f156c133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-a con 0x7f156c06f440 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156c09c490 0x7f156c11e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): picked mon.noname-b con 0x7f156c09c490 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c1222f0 con 0x7f156c06f440 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c09cd80 con 0x7f156c09c490 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c11fe10 con 0x7f156c06eb30 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1572e3f640 10 monclient(hunting): authenticate will time out at 1787.418765s 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57384/0 (socket says 172.21.15.17:57384) 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 1 -- 172.21.15.17:0/2379438425 learned_addr learned my addr 172.21.15.17:0/2379438425 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): get_auth_request con 0x7f156c06eb30 auth_method 0 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 0x7f156c133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): get_auth_request con 0x7f156c06f440 auth_method 0 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.422+0000 7f156bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f15713dc640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156c09c490 0x7f156c11e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f15713dc640 10 monclient(hunting): get_auth_request con 0x7f156c09c490 auth_method 0 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f15713dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f15713dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f15713dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 10 monclient(hunting): handle_auth_done global_id 105723 payload 824 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 msgr2=0x7f156c141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156c09c490 msgr2=0x7f156c11e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156c09c490 0x7f156c11e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 monclient: found mon.noname-a 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f156bfff640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560009450 con 0x7f156c06f440 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.423+0000 7f1570bdb640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f156c06eb30 0x7f156c141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f156bfff640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 msgr2=0x7f156c133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f156bfff640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 0x7f156c133ee0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f155800d8c0 tx=0x7f1558008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155800b8d0 con 0x7f156c06f440 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: dump: 2024-09-15T02:08:47.440 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155800ba30 con 0x7f156c06f440 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 5 monclient: authenticate success, global_id 105723 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 10 monclient: _renew_subs 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156c120a20 con 0x7f156c06f440 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155800bdf0 con 0x7f156c06f440 2024-09-15T02:08:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1568ff9640 10 monclient: dump: 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 10 monclient: _renew_subs 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f156affd640 4 set_mon_vals no callback set 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.425+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c134420 con 0x7f156c06f440 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1572e3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1558014640 con 0x7f156c06f440 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1572e3f640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 0x7f156c11eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c1221b0 con 0x7f156c1415c0 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 0x7f156c11eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient: get_auth_request con 0x7f156c1415c0 auth_method 0 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1568ff9640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1550038740 0x7f155003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f155804cb90 con 0x7f156c06f440 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 10 monclient(hunting): handle_auth_done global_id 105726 payload 824 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f1570bdb640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f156c1211c0 con 0x7f156c1415c0 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f15713dc640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1550038740 0x7f155003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.426+0000 7f15713dc640 10 monclient: get_auth_request con 0x7f1550038740 auth_method 0 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1570bdb640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 msgr2=0x7f156c11eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1570bdb640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 0x7f156c11eee0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1560009420 tx=0x7f1560032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156002aea0 con 0x7f156c1415c0 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 10 monclient: dump: 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f156003c650 con 0x7f156c1415c0 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 msgr2=0x7f156c11eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f15713dc640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1550038740 0x7f155003abf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f156c122680 tx=0x7f1564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 0x7f156c11eee0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1560009420 tx=0x7f1560032000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 msgr2=0x7f156c11eee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.429+0000 7f1568ff9640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c1415c0 0x7f156c11eee0 secure :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1560009420 tx=0x7f1560032000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1572e3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1572e3f640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 0x7f156c096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1572e3f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c063b90 con 0x7f156c11f420 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 0x7f156c096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient: get_auth_request con 0x7f156c11f420 auth_method 0 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 10 monclient(hunting): handle_auth_done global_id 105729 payload 824 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.435+0000 7f1570bdb640 1 -- 172.21.15.17:0/2379438425 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f156c11f820 con 0x7f156c11f420 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1570bdb640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 msgr2=0x7f156c096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1570bdb640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 0x7f156c096a10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f156c09cf10 tx=0x7f156003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560053410 con 0x7f156c11f420 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1568ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.437+0000 7f1568ff9640 10 monclient: dump: 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f156003b3d0 con 0x7f156c11f420 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 msgr2=0x7f156c096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 0x7f156c096a10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f156c09cf10 tx=0x7f156003d000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 msgr2=0x7f156c096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1568ff9640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c11f420 0x7f156c096a10 secure :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f156c09cf10 tx=0x7f156003d000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:47.452 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.453 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.454 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.455 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.456 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:47.457 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.458 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.460 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:47.461 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.462 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.463 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 7 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.464 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.465 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.466 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:47.466 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1550038740 msgr2=0x7f155003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1550038740 0x7f155003abf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f156c122680 tx=0x7f1564009040 comp rx=0 tx=0).stop 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 10 monclient: shutdown 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 msgr2=0x7f156c133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 0x7f156c133ee0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f155800d8c0 tx=0x7f1558008040 comp rx=0 tx=0).stop 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f15713dc640 1 -- 172.21.15.17:0/2379438425 reap_dead start 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 shutdown_connections 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 --2- 172.21.15.17:0/2379438425 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f156c06f440 0x7f156c133ee0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 >> 172.21.15.17:0/2379438425 conn(0x7f156c0712f0 msgr2=0x7f156c09e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 shutdown_connections 2024-09-15T02:08:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.438+0000 7f1572e3f640 1 -- 172.21.15.17:0/2379438425 wait complete. 2024-09-15T02:08:47.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:47.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:08:47.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:08:47.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 10 monclient: get_monmap_and_config 2024-09-15T02:08:47.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 10 monclient: monmap: 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:47.564991+0000 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:47.564991+0000 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:47.566 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 1 Processor -- start 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 1 -- start start 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.564+0000 7fb4aecdd640 10 monclient: init 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a80980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4a80989e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80a01e0 con 0x7fb4a80989e0 2024-09-15T02:08:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a80a1580 con 0x7fb4a80a58b0 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a80a5c80 con 0x7fb4a80980a0 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aecdd640 10 monclient(hunting): authenticate will time out at 1787.561458s 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47880/0 (socket says 172.21.15.17:47880) 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 1 -- 172.21.15.17:0/1236785693 learned_addr learned my addr 172.21.15.17:0/1236785693 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): get_auth_request con 0x7fb4a80a58b0 auth_method 0 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57394/0 (socket says 172.21.15.17:57394) 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): get_auth_request con 0x7fb4a80980a0 auth_method 0 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_done global_id 214124 payload 274 2024-09-15T02:08:47.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 -- 172.21.15.17:0/1236785693 >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 msgr2=0x7fb4a8132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 --2- 172.21.15.17:0/1236785693 >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 -- 172.21.15.17:0/1236785693 >> v1:172.21.15.67:6789/0 conn(0x7fb4a80989e0 legacy=0x7fb4a8068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 monclient: found mon.noname-b 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4a7fff640 1 -- 172.21.15.17:0/1236785693 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a81326c0 con 0x7fb4a80980a0 2024-09-15T02:08:47.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.565+0000 7fb4aca79640 1 --2- 172.21.15.17:0/1236785693 >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a7fff640 1 -- 172.21.15.17:0/1236785693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 msgr2=0x7fb4a80984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a7fff640 1 --2- 172.21.15.17:0/1236785693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb494009880 tx=0x7fb4940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cbbbf9ba153ff49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 1 -- 172.21.15.17:0/1236785693 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49402aea0 con 0x7fb4a80980a0 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: dump: 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.579 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 1 -- 172.21.15.17:0/1236785693 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494032440 con 0x7fb4a80980a0 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 1 -- 172.21.15.17:0/1236785693 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4940327e0 con 0x7fb4a80980a0 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4aecdd640 5 monclient: authenticate success, global_id 214124 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4aecdd640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4aecdd640 4 set_mon_vals no callback set 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.568+0000 7fb4a77fe640 10 monclient: dump: 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 10 monclient: shutdown 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 -- 172.21.15.17:0/1236785693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 msgr2=0x7fb4a80984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/1236785693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb494009880 tx=0x7fb4940090f0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 -- 172.21.15.17:0/1236785693 shutdown_connections 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/1236785693 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80980a0 0x7fb4a80984a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/1236785693 >> v2:172.21.15.17:3300/0 conn(0x7fb4a80a58b0 0x7fb4a8132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 -- 172.21.15.17:0/1236785693 >> 172.21.15.17:0/1236785693 conn(0x7fb4a8070ed0 msgr2=0x7fb4a80a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 -- 172.21.15.17:0/1236785693 shutdown_connections 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.569+0000 7fb4aecdd640 1 -- 172.21.15.17:0/1236785693 wait complete. 2024-09-15T02:08:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.570+0000 7fb4aecdd640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.570+0000 7fb4aecdd640 10 monclient: monmap: 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.570+0000 7fb4aecdd640 1 Processor -- start 2024-09-15T02:08:47.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.570+0000 7fb4aecdd640 1 -- start start 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.570+0000 7fb4aecdd640 10 monclient: init 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a80980a0 0x7fb4a809aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a80980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 0x7fb4a8130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a80a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 0x7fb4a811b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a81311a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a811e060 con 0x7fb4a80a58b0 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a811cc50 con 0x7fb4a80980a0 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8098dc0 con 0x7fb4a81311a0 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aecdd640 10 monclient(hunting): authenticate will time out at 1787.567148s 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 0x7fb4a8130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 0x7fb4a811b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 0x7fb4a8130c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47896/0 (socket says 172.21.15.17:47896) 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 1 -- 172.21.15.17:0/3246652796 learned_addr learned my addr 172.21.15.17:0/3246652796 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): get_auth_request con 0x7fb4a80a58b0 auth_method 0 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): get_auth_request con 0x7fb4a81311a0 auth_method 0 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a80980a0 0x7fb4a809aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 10 monclient(hunting): get_auth_request con 0x7fb4a80980a0 auth_method 0 2024-09-15T02:08:47.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient(hunting): handle_auth_done global_id 214127 payload 824 2024-09-15T02:08:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a80980a0 msgr2=0x7fb4a809aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a80980a0 0x7fb4a809aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 msgr2=0x7fb4a8130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 0x7fb4a8130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 monclient: found mon.noname-c 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4ad27a640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c009450 con 0x7fb4a81311a0 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a80980a0 0x7fb4a809aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.571+0000 7fb4a7fff640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb4a80a58b0 0x7fb4a8130c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4ad27a640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 msgr2=0x7fb4a811b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4ad27a640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 0x7fb4a811b7d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000c7e0 tx=0x7fb4a00025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000a850 con 0x7fb4a81311a0 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 10 monclient: dump: 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.589 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4aecdd640 5 monclient: authenticate success, global_id 214127 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a000a9d0 con 0x7fb4a81311a0 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4aecdd640 10 monclient: _renew_subs 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4aecdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb494009390 con 0x7fb4a81311a0 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.573+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000ad90 con 0x7fb4a81311a0 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a4ff9640 10 monclient: dump: 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.590 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 10 monclient: _renew_subs 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a67fc640 4 set_mon_vals no callback set 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81420c0 con 0x7fb4a81311a0 2024-09-15T02:08:47.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 0x7fb4a811bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a811df20 con 0x7fb4a80989e0 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb4a0012630 con 0x7fb4a81311a0 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 0x7fb4a811bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient: get_auth_request con 0x7fb4a80989e0 auth_method 0 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.574+0000 7fb4a4ff9640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb48c038740 0x7fb48c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb4a004c8b0 con 0x7fb4a81311a0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4aca79640 10 monclient(hunting): handle_auth_done global_id 214130 payload 824 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4aca79640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4a811ced0 con 0x7fb4a80989e0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4a7fff640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb48c038740 0x7fb48c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.575+0000 7fb4a7fff640 10 monclient: get_auth_request con 0x7fb48c038740 auth_method 0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4aca79640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 msgr2=0x7fb4a811bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 0x7fb4a811bd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00fd60 con 0x7fb4a80989e0 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 10 monclient: dump: 2024-09-15T02:08:47.593 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb49c00fec0 con 0x7fb4a80989e0 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 msgr2=0x7fb4a811bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 0x7fb4a811bd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c008040 comp rx=0 tx=0).stop 2024-09-15T02:08:47.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a7fff640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb48c038740 0x7fb48c03abf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb494009130 tx=0x7fb494032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 msgr2=0x7fb4a811bd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.577+0000 7fb4a4ff9640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a80989e0 0x7fb4a811bd10 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c008040 comp rx=0 tx=0).stop 2024-09-15T02:08:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aecdd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 0x7fb4a809c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aecdd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8063b20 con 0x7fb4a811c250 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 0x7fb4a809c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient: get_auth_request con 0x7fb4a811c250 auth_method 0 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.583+0000 7fb4aca79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.584+0000 7fb4aca79640 10 monclient(hunting): handle_auth_done global_id 214133 payload 824 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.584+0000 7fb4aca79640 1 -- 172.21.15.17:0/3246652796 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4a811c650 con 0x7fb4a811c250 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4aca79640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 msgr2=0x7fb4a809c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4aca79640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 0x7fb4a809c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c022070 con 0x7fb4a811c250 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 10 monclient: dump: 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.598 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4396 (secure 0 0 0) 0x7fb49c0253b0 con 0x7fb4a811c250 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 msgr2=0x7fb4a809c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 0x7fb4a809c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c01c000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 msgr2=0x7fb4a809c550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.586+0000 7fb4a4ff9640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a811c250 0x7fb4a809c550 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb49c011990 tx=0x7fb49c01c000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:47.599 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:47.600 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.601 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:47.602 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.603 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.605 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:47.606 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.610 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:47.611 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945693877}) 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1009563532}) 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945693877}) 2024-09-15T02:08:47.612 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:08:47.586+0000 7f742dbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465283095}) 2024-09-15T02:08:47.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb48c038740 msgr2=0x7fb48c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb48c038740 0x7fb48c03abf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb494009130 tx=0x7fb494032000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 10 monclient: shutdown 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 msgr2=0x7fb4a811b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4ad27a640 1 -- 172.21.15.17:0/3246652796 reap_dead start 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 0x7fb4a811b7d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000c7e0 tx=0x7fb4a00025b0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 shutdown_connections 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 --2- 172.21.15.17:0/3246652796 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4a81311a0 0x7fb4a811b7d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 >> 172.21.15.17:0/3246652796 conn(0x7fb4a8070ed0 msgr2=0x7fb4a80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 shutdown_connections 2024-09-15T02:08:47.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.587+0000 7fb4aecdd640 1 -- 172.21.15.17:0/3246652796 wait complete. 2024-09-15T02:08:47.615 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:47.615 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:08:47.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.710+0000 7f1c26900640 10 monclient: get_monmap_and_config 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.710+0000 7f1c26900640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.710+0000 7f1c26900640 10 monclient: monmap: 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:47.711561+0000 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:47.711561+0000 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:47.713 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 1 Processor -- start 2024-09-15T02:08:47.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 1 -- start start 2024-09-15T02:08:47.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient: init 2024-09-15T02:08:47.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c200a5780 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c2009ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c2009bc90 0x7f1c2013a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2009bc90 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.711+0000 7f1c26900640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c200a5b50 con 0x7f1c2009ccf0 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c26900640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2009c5f0 con 0x7f1c200a5780 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c26900640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c200a18b0 con 0x7f1c2009bc90 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c26900640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c26900640 10 monclient(hunting): authenticate will time out at 1787.708022s 2024-09-15T02:08:47.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c1f7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1c2009ccf0 0x7f1c20069c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33266/0 (socket says 172.21.15.17:33266) 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c1f7fe640 1 -- 172.21.15.17:0/1264406356 learned_addr learned my addr 172.21.15.17:0/1264406356 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 1 --2- 172.21.15.17:0/1264406356 >> v2:172.21.15.67:3300/0 conn(0x7f1c2009bc90 0x7f1c2013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): get_auth_request con 0x7f1c2009bc90 auth_method 0 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_done global_id 204223 payload 274 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 1 -- 172.21.15.17:0/1264406356 >> v1:172.21.15.67:6789/0 conn(0x7f1c200a5780 legacy=0x7f1c20069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 1 -- 172.21.15.17:0/1264406356 >> v1:172.21.15.17:6789/0 conn(0x7f1c2009ccf0 legacy=0x7f1c20069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 1 monclient: found mon.noname-c 2024-09-15T02:08:47.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:08:47.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.712+0000 7f1c24e9d640 1 -- 172.21.15.17:0/1264406356 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2006a3a0 con 0x7f1c2009bc90 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.715+0000 7f1c24e9d640 1 -- 172.21.15.17:0/1264406356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2009bc90 msgr2=0x7f1c2013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.715+0000 7f1c24e9d640 1 --2- 172.21.15.17:0/1264406356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2009bc90 0x7f1c2013a890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c18009e60 tx=0x7f1c18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a4491cfee715bc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 1 -- 172.21.15.17:0/1264406356 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18007630 con 0x7f1c2009bc90 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: dump: 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.727 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 1 -- 172.21.15.17:0/1264406356 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1802d050 con 0x7f1c2009bc90 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 1 -- 172.21.15.17:0/1264406356 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180079c0 con 0x7f1c2009bc90 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c1effd640 10 monclient: dump: 2024-09-15T02:08:47.728 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 5 monclient: authenticate success, global_id 204223 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 4 set_mon_vals no callback set 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 10 monclient: shutdown 2024-09-15T02:08:47.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 -- 172.21.15.17:0/1264406356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2009bc90 msgr2=0x7f1c2013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 --2- 172.21.15.17:0/1264406356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2009bc90 0x7f1c2013a890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c18009e60 tx=0x7f1c18009780 comp rx=0 tx=0).stop 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 -- 172.21.15.17:0/1264406356 shutdown_connections 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 --2- 172.21.15.17:0/1264406356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2009bc90 0x7f1c2013a890 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 -- 172.21.15.17:0/1264406356 >> 172.21.15.17:0/1264406356 conn(0x7f1c20070e10 msgr2=0x7f1c200a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 -- 172.21.15.17:0/1264406356 shutdown_connections 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.716+0000 7f1c26900640 1 -- 172.21.15.17:0/1264406356 wait complete. 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.717+0000 7f1c26900640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.717+0000 7f1c26900640 10 monclient: monmap: 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:47.730 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.717+0000 7f1c26900640 1 Processor -- start 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.717+0000 7f1c26900640 1 -- start start 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.717+0000 7f1c26900640 10 monclient: init 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2009bc90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:47.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1c2009ccf0 0x7f1c2013d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2009ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c200a5780 0x7f1c201411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): picked mon.noname-b con 0x7f1c200a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20122a90 con 0x7f1c2009bc90 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c201226b0 con 0x7f1c200a5780 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c201209a0 con 0x7f1c2009ccf0 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c26900640 10 monclient(hunting): authenticate will time out at 1787.714272s 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47904/0 (socket says 172.21.15.17:47904) 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1c2009ccf0 0x7f1c2013d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 -- 172.21.15.17:0/3693114653 learned_addr learned my addr 172.21.15.17:0/3693114653 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): get_auth_request con 0x7f1c2009bc90 auth_method 0 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request con 0x7f1c2009ccf0 auth_method 0 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c200a5780 0x7f1c201411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): get_auth_request con 0x7f1c200a5780 auth_method 0 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c24e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_done global_id 105738 payload 824 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1c2009ccf0 msgr2=0x7f1c2013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1c2009ccf0 0x7f1c2013d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c200a5780 msgr2=0x7f1c201411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c200a5780 0x7f1c201411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 monclient: found mon.noname-a 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1ffff640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c18009a20 con 0x7f1c2009bc90 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.718+0000 7f1c1f7fe640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1c2009ccf0 0x7f1c2013d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c1ffff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 msgr2=0x7f1c2013dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c1ffff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c5e0 tx=0x7f1c10007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10012840 con 0x7f1c2009bc90 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:08:47.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c100129a0 con 0x7f1c2009bc90 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10012d60 con 0x7f1c2009bc90 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 5 monclient: authenticate success, global_id 105738 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 10 monclient: _renew_subs 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c20120190 con 0x7f1c2009bc90 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.737 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c1e7fc640 4 set_mon_vals no callback set 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 10 monclient: _renew_subs 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.721+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2013e690 con 0x7f1c2009bc90 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c26900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c26900640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 0x7f1c20141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20122570 con 0x7f1c20121500 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1c10011590 con 0x7f1c2009bc90 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c07fff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1c0c038740 0x7f1c0c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 0x7f1c20141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient: get_auth_request con 0x7f1c20121500 auth_method 0 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1c1004c7d0 con 0x7f1c2009bc90 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c24e9d640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1c0c038740 0x7f1c0c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c24e9d640 10 monclient: get_auth_request con 0x7f1c0c038740 auth_method 0 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_done global_id 204229 payload 824 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.722+0000 7f1c1f7fe640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c20122d70 con 0x7f1c20121500 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.724+0000 7f1c1f7fe640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 msgr2=0x7f1c20141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.724+0000 7f1c1f7fe640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 0x7f1c20141730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c24e9d640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1c0c038740 0x7f1c0c03abf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1c20068e70 tx=0x7f1c18037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400fcf0 con 0x7f1c20121500 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.740 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1c1400fe50 con 0x7f1c20121500 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 msgr2=0x7f1c20141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 0x7f1c20141730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14007f00 comp rx=0 tx=0).stop 2024-09-15T02:08:47.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 msgr2=0x7f1c20141730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.725+0000 7f1c07fff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20121500 0x7f1c20141730 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14007f00 comp rx=0 tx=0).stop 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.731+0000 7f1c26900640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.731+0000 7f1c26900640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 0x7f1c2011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.731+0000 7f1c26900640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.731+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20063b20 con 0x7f1c20141c70 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 0x7f1c2011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient: get_auth_request con 0x7f1c20141c70 auth_method 0 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_done global_id 204232 payload 824 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.732+0000 7f1c1f7fe640 1 -- 172.21.15.17:0/3693114653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1c20142070 con 0x7f1c20141c70 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.734+0000 7f1c1f7fe640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 msgr2=0x7f1c2011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.734+0000 7f1c1f7fe640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 0x7f1c2011b6d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14021880 con 0x7f1c20141c70 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T02:08:47.745 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f1c1401c3e0 con 0x7f1c20141c70 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:47.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 msgr2=0x7f1c2011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 0x7f1c2011b6d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14018000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 msgr2=0x7f1c2011b6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.735+0000 7f1c07fff640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c20141c70 0x7f1c2011b6d0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c14007fd0 tx=0x7f1c14018000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 174, 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:08:47.747 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 7, 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.748 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.749 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:08:47.750 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:08:47.751 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:08:47.752 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.753 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.754 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.755 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:08:47.756 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:08:47.757 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.758 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:08:47.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:08:47.760 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:08:47.760 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1c0c038740 msgr2=0x7f1c0c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1c0c038740 0x7f1c0c03abf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1c20068e70 tx=0x7f1c18037000 comp rx=0 tx=0).stop 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 10 monclient: shutdown 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 msgr2=0x7f1c2013dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c5e0 tx=0x7f1c10007ed0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c24e9d640 1 -- 172.21.15.17:0/3693114653 reap_dead start 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 shutdown_connections 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 --2- 172.21.15.17:0/3693114653 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c2009bc90 0x7f1c2013dc80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 >> 172.21.15.17:0/3693114653 conn(0x7f1c20070e10 msgr2=0x7f1c2009a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 shutdown_connections 2024-09-15T02:08:47.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.736+0000 7f1c26900640 1 -- 172.21.15.17:0/3693114653 wait complete. 2024-09-15T02:08:47.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:08:47.762 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T02:08:47.763 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:08:47.763 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T02:08:47.763 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:08:47.763 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:08:47.763 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:08:47.780 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:08:47.780 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:08:47.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 10 monclient: get_monmap_and_config 2024-09-15T02:08:47.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 10 monclient: monmap: 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:47.914978+0000 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:47.914978+0000 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:47.916 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:47.917 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 1 Processor -- start 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 1 -- start start 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.914+0000 7fa99342b640 10 monclient: init 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa98c131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa98c132430 0x7fa98c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c132430 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:47.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c0a0040 con 0x7fa98c131b20 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c09bff0 con 0x7fa98c132430 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0a14a0 con 0x7fa98c0a0280 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa99342b640 10 monclient(hunting): authenticate will time out at 1787.911534s 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57468/0 (socket says 172.21.15.17:57468) 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 -- 172.21.15.17:0/3190696846 learned_addr learned my addr 172.21.15.17:0/3190696846 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): get_auth_request con 0x7fa98c0a0280 auth_method 0 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient(hunting): handle_auth_done global_id 214139 payload 274 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9919c8640 1 --2- 172.21.15.17:0/3190696846 >> v2:172.21.15.67:3300/0 conn(0x7fa98c132430 0x7fa98c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 -- 172.21.15.17:0/3190696846 >> v2:172.21.15.67:3300/0 conn(0x7fa98c132430 msgr2=0x7fa98c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 --2- 172.21.15.17:0/3190696846 >> v2:172.21.15.67:3300/0 conn(0x7fa98c132430 0x7fa98c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 -- 172.21.15.17:0/3190696846 >> v1:172.21.15.67:6789/0 conn(0x7fa98c131b20 legacy=0x7fa98c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 monclient: found mon.noname-b 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.915+0000 7fa9909c6640 1 -- 172.21.15.17:0/3190696846 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c09f3b0 con 0x7fa98c0a0280 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa9909c6640 1 -- 172.21.15.17:0/3190696846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 msgr2=0x7fa98c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa9909c6640 1 --2- 172.21.15.17:0/3190696846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009910 tx=0x7fa97c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab560f4810b4ffc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 1 -- 172.21.15.17:0/3190696846 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c032440 con 0x7fa98c0a0280 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:08:47.927 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 10 monclient: dump: 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.918+0000 7fa983fff640 1 -- 172.21.15.17:0/3190696846 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c0325a0 con 0x7fa98c0a0280 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 5 monclient: authenticate success, global_id 214139 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa983fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa983fff640 1 -- 172.21.15.17:0/3190696846 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c032940 con 0x7fa98c0a0280 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 4 set_mon_vals no callback set 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa983fff640 10 monclient: dump: 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 10 monclient: shutdown 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 -- 172.21.15.17:0/3190696846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 msgr2=0x7fa98c131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 --2- 172.21.15.17:0/3190696846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009910 tx=0x7fa97c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 -- 172.21.15.17:0/3190696846 shutdown_connections 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 --2- 172.21.15.17:0/3190696846 >> v2:172.21.15.67:3300/0 conn(0x7fa98c132430 0x7fa98c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 --2- 172.21.15.17:0/3190696846 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c131ef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 -- 172.21.15.17:0/3190696846 >> 172.21.15.17:0/3190696846 conn(0x7fa98c071270 msgr2=0x7fa98c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 -- 172.21.15.17:0/3190696846 shutdown_connections 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.919+0000 7fa99342b640 1 -- 172.21.15.17:0/3190696846 wait complete. 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.920+0000 7fa99342b640 10 monclient: build_initial_monmap 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.920+0000 7fa99342b640 10 monclient: monmap: 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.920+0000 7fa99342b640 1 Processor -- start 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.920+0000 7fa99342b640 1 -- start start 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.920+0000 7fa99342b640 10 monclient: init 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 0x7fa98c132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 0x7fa98c132e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): picked mon.noname-a con 0x7fa98c132430 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0a09b0 con 0x7fa98c132430 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c134300 con 0x7fa98c131b20 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c138ff0 con 0x7fa98c0a0280 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa99342b640 10 monclient(hunting): authenticate will time out at 1787.917251s 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9919c8640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 msgr2=0x7fa98c132e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9919c8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 0x7fa98c132e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57470/0 (socket says 172.21.15.17:57470) 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 -- 172.21.15.17:0/603963568 learned_addr learned my addr 172.21.15.17:0/603963568 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): get_auth_request con 0x7fa98c0a0280 auth_method 0 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 0x7fa98c132900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 10 monclient(hunting): get_auth_request con 0x7fa98c131b20 auth_method 0 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient(hunting): handle_auth_done global_id 214142 payload 824 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 -- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 msgr2=0x7fa98c132900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 0x7fa98c132900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 -- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 msgr2=0x7fa98c132e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 0x7fa98c132e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient: _finish_hunting 0 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 monclient: found mon.noname-c 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9911c7640 1 -- 172.21.15.17:0/603963568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c009450 con 0x7fa98c0a0280 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.921+0000 7fa9909c6640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 0x7fa98c132900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9911c7640 1 -- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 msgr2=0x7fa98c09b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:47.935 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9911c7640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c610 tx=0x7fa988007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988012840 con 0x7fa98c0a0280 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.923+0000 7fa9817fa640 10 monclient: dump: 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.936 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 10 monclient: _finish_auth 0 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 5 monclient: authenticate success, global_id 214142 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9880129e0 con 0x7fa98c0a0280 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _renew_subs 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988012da0 con 0x7fa98c0a0280 2024-09-15T02:08:47.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c1358b0 con 0x7fa98c0a0280 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 10 monclient: dump: 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:47.938 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9837fe640 4 set_mon_vals no callback set 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _renew_subs 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c138660 con 0x7fa98c0a0280 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa988011590 con 0x7fa98c0a0280 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.924+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c0724a0 con 0x7fa98c0a0280 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.925+0000 7fa9817fa640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 0x7fa97003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.925+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa98804c130 con 0x7fa98c0a0280 2024-09-15T02:08:47.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.925+0000 7fa9909c6640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 0x7fa97003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:47.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.925+0000 7fa9909c6640 10 monclient: get_auth_request con 0x7fa970038740 auth_method 0 2024-09-15T02:08:47.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.927+0000 7fa9909c6640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 0x7fa97003abf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa98c09f710 tx=0x7fa97c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:47.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.929+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa9880198f0 con 0x7fa98c0a0280 2024-09-15T02:08:47.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.929+0000 7fa9817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:47.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:47.929+0000 7fa9817fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:48.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.155+0000 7fa99342b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa99342b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa99342b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa98c0980e0 con 0x7fa98c0a0280 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa9817fa640 1 -- 172.21.15.17:0/603963568 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fa988019310 con 0x7fa98c0a0280 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa9817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.156+0000 7fa9817fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:48.158 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:48.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 msgr2=0x7fa97003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 0x7fa97003abf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa98c09f710 tx=0x7fa97c032000 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 10 monclient: shutdown 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 msgr2=0x7fa98c09b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c610 tx=0x7fa988007ed0 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 shutdown_connections 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa970038740 0x7fa97003abf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa98c132430 0x7fa98c132e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c131b20 0x7fa98c132900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 --2- 172.21.15.17:0/603963568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa98c0a0280 0x7fa98c09b490 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 >> 172.21.15.17:0/603963568 conn(0x7fa98c071270 msgr2=0x7fa98c131280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:48.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 shutdown_connections 2024-09-15T02:08:48.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:48.158+0000 7fa99342b640 1 -- 172.21.15.17:0/603963568 wait complete. 2024-09-15T02:08:51.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:51.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.277+0000 7fd78764f640 10 monclient: get_monmap_and_config 2024-09-15T02:08:51.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.277+0000 7fd78764f640 10 monclient: build_initial_monmap 2024-09-15T02:08:51.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.277+0000 7fd78764f640 10 monclient: monmap: 2024-09-15T02:08:51.279 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:51.278556+0000 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:51.278556+0000 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:51.280 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:51.281 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:51.281 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.278+0000 7fd78764f640 1 Processor -- start 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.278+0000 7fd78764f640 1 -- start start 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.278+0000 7fd78764f640 10 monclient: init 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T02:08:51.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-b con 0x7fd78006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd78006f440 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd78009e970 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78009ed90 con 0x7fd78006f440 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78009f6a0 con 0x7fd78009e970 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd780069f40 con 0x7fd78006eb30 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd78764f640 10 monclient(hunting): authenticate will time out at 1791.275194s 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd784bea640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fd78006f440 legacy=0x7fd78009bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:51.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57496/0 (socket says 172.21.15.17:57496) 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 -- 172.21.15.17:0/1952620861 learned_addr learned my addr 172.21.15.17:0/1952620861 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): get_auth_request con 0x7fd78006eb30 auth_method 0 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient(hunting): handle_auth_done global_id 214145 payload 274 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 -- 172.21.15.17:0/1952620861 >> v1:172.21.15.67:6789/0 conn(0x7fd78009e970 legacy=0x7fd780139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 -- 172.21.15.17:0/1952620861 >> v1:172.21.15.17:6789/0 conn(0x7fd78006f440 legacy=0x7fd78009bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient: _finish_hunting 0 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 monclient: found mon.noname-b 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.279+0000 7fd7853eb640 1 -- 172.21.15.17:0/1952620861 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780069950 con 0x7fd78006eb30 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd7853eb640 1 -- 172.21.15.17:0/1952620861 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 msgr2=0x7fd78006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd7853eb640 1 --2- 172.21.15.17:0/1952620861 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd770009940 tx=0x7fd7700091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fafc3acf9f275f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 1 -- 172.21.15.17:0/1952620861 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770032440 con 0x7fd78006eb30 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: dump: 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.292 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: _finish_auth 0 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd78764f640 5 monclient: authenticate success, global_id 214145 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd78764f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 1 -- 172.21.15.17:0/1952620861 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7700325a0 con 0x7fd78006eb30 2024-09-15T02:08:51.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd78764f640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd78764f640 4 set_mon_vals no callback set 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 1 -- 172.21.15.17:0/1952620861 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770032940 con 0x7fd78006eb30 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.282+0000 7fd777fff640 10 monclient: dump: 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:51.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 10 monclient: shutdown 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 -- 172.21.15.17:0/1952620861 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 msgr2=0x7fd78006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 --2- 172.21.15.17:0/1952620861 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd770009940 tx=0x7fd7700091d0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 -- 172.21.15.17:0/1952620861 shutdown_connections 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 --2- 172.21.15.17:0/1952620861 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd78006ef00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 -- 172.21.15.17:0/1952620861 >> 172.21.15.17:0/1952620861 conn(0x7fd7800712f0 msgr2=0x7fd7800a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 -- 172.21.15.17:0/1952620861 shutdown_connections 2024-09-15T02:08:51.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 1 -- 172.21.15.17:0/1952620861 wait complete. 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 10 monclient: build_initial_monmap 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.283+0000 7fd78764f640 10 monclient: monmap: 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:51.296 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 Processor -- start 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 -- start start 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient: init 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-c con 0x7fd78006eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 0x7fd780133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-a con 0x7fd78006f440 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78009e970 0x7fd7801342e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): picked mon.noname-b con 0x7fd78009e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:51.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd780121810 con 0x7fd78006f440 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78011ee50 con 0x7fd78009e970 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78009cd10 con 0x7fd78006eb30 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd78764f640 10 monclient(hunting): authenticate will time out at 1791.280935s 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd784bea640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 msgr2=0x7fd780133da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.284+0000 7fd784bea640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 0x7fd780133da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57504/0 (socket says 172.21.15.17:57504) 2024-09-15T02:08:51.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 -- 172.21.15.17:0/2624598005 learned_addr learned my addr 172.21.15.17:0/2624598005 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): get_auth_request con 0x7fd78006eb30 auth_method 0 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd785bec640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78009e970 0x7fd7801342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd785bec640 10 monclient(hunting): get_auth_request con 0x7fd78009e970 auth_method 0 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd785bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd785bec640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd785bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient(hunting): handle_auth_done global_id 214148 payload 824 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 -- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78009e970 msgr2=0x7fd7801342e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78009e970 0x7fd7801342e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 -- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 msgr2=0x7fd780133da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 0x7fd780133da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient: _finish_hunting 0 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 monclient: found mon.noname-c 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.285+0000 7fd7853eb640 1 -- 172.21.15.17:0/2624598005 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd770009450 con 0x7fd78006eb30 2024-09-15T02:08:51.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7853eb640 1 -- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 msgr2=0x7fd780141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7853eb640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd770009940 tx=0x7fd770032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77003c960 con 0x7fd78006eb30 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: dump: 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:51.301 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:51.304 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.304 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.304 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:51.304 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: _finish_auth 0 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77003cac0 con 0x7fd78006eb30 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 5 monclient: authenticate success, global_id 214148 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 10 monclient: _renew_subs 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780134c40 con 0x7fd78006eb30 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:51.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77003ce80 con 0x7fd78006eb30 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7757fa640 10 monclient: dump: 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:51.306 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 10 monclient: _renew_subs 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7801211c0 con 0x7fd78006eb30 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.287+0000 7fd7777fe640 4 set_mon_vals no callback set 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd770039040 con 0x7fd78006eb30 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd78764f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd78764f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd78764f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7800724a0 con 0x7fd78006eb30 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd7757fa640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 0x7fd76403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.288+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd77006c590 con 0x7fd78006eb30 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.289+0000 7fd784bea640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 0x7fd76403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:51.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.289+0000 7fd784bea640 10 monclient: get_auth_request con 0x7fd7640386f0 auth_method 0 2024-09-15T02:08:51.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.291+0000 7fd784bea640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 0x7fd76403aba0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd78011f7e0 tx=0x7fd77c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:51.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.293+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd770043360 con 0x7fd78006eb30 2024-09-15T02:08:51.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.293+0000 7fd7757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:51.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.293+0000 7fd7757fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:51.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd78764f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:51.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd78764f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:51.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd78764f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:51.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd7800724a0 con 0x7fd78006eb30 2024-09-15T02:08:51.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd7757fa640 1 -- 172.21.15.17:0/2624598005 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fd770040080 con 0x7fd78006eb30 2024-09-15T02:08:51.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd7757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:51.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.522+0000 7fd7757fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:51.524 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 msgr2=0x7fd76403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 0x7fd76403aba0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd78011f7e0 tx=0x7fd77c006d80 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 10 monclient: shutdown 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 msgr2=0x7fd780141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd770009940 tx=0x7fd770032000 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 shutdown_connections 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd7640386f0 0x7fd76403aba0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd78006f440 0x7fd780133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78009e970 0x7fd7801342e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 --2- 172.21.15.17:0/2624598005 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd78006eb30 0x7fd780141540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:51.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 >> 172.21.15.17:0/2624598005 conn(0x7fd7800712f0 msgr2=0x7fd78009d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:51.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 shutdown_connections 2024-09-15T02:08:51.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:51.524+0000 7fd78764f640 1 -- 172.21.15.17:0/2624598005 wait complete. 2024-09-15T02:08:54.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.644+0000 7f66e2860640 10 monclient: get_monmap_and_config 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.644+0000 7f66e2860640 10 monclient: build_initial_monmap 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.644+0000 7f66e2860640 10 monclient: monmap: 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:54.645453+0000 2024-09-15T02:08:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:54.645453+0000 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:54.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.645+0000 7f66e2860640 1 Processor -- start 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.645+0000 7f66e2860640 1 -- start start 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.645+0000 7f66e2860640 10 monclient: init 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T02:08:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66dc131b20 0x7f66dc09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-c con 0x7f66dc131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66dc0a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66dc09f1e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0a0040 con 0x7f66dc0a0280 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc09bff0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66db7fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f66dc0a0280 legacy=0x7f66dc131ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66dc0a14a0 con 0x7f66dc131b20 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e2860640 10 monclient(hunting): authenticate will time out at 1794.642176s 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66dc131b20 0x7f66dc09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e0dfd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66dc09f1e0 0x7f66dc138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:51180/0 (socket says 172.21.15.17:51180) 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66e0dfd640 1 -- 172.21.15.17:0/2963970748 learned_addr learned my addr 172.21.15.17:0/2963970748 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): get_auth_request con 0x7f66dc131b20 auth_method 0 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient(hunting): handle_auth_done global_id 204241 payload 274 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 1 -- 172.21.15.17:0/2963970748 >> v1:172.21.15.67:6789/0 conn(0x7f66dc09f1e0 legacy=0x7f66dc138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 1 -- 172.21.15.17:0/2963970748 >> v1:172.21.15.17:6789/0 conn(0x7f66dc0a0280 legacy=0x7f66dc131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 1 monclient: found mon.noname-c 2024-09-15T02:08:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:08:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.646+0000 7f66dbfff640 1 -- 172.21.15.17:0/2963970748 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc132600 con 0x7f66dc131b20 2024-09-15T02:08:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.649+0000 7f66dbfff640 1 -- 172.21.15.17:0/2963970748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 msgr2=0x7f66dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.649+0000 7f66dbfff640 1 --2- 172.21.15.17:0/2963970748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc09eca0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009850 tx=0x7f66cc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4007e70b8d84f41a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 1 -- 172.21.15.17:0/2963970748 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc032440 con 0x7f66dc131b20 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: dump: 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: _finish_auth 0 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 1 -- 172.21.15.17:0/2963970748 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc0325a0 con 0x7f66dc131b20 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 1 -- 172.21.15.17:0/2963970748 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc032940 con 0x7f66dc131b20 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 5 monclient: authenticate success, global_id 204241 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 10 monclient: get_monmap_and_config success 2024-09-15T02:08:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 4 set_mon_vals no callback set 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66daffd640 10 monclient: dump: 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:54.658 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 10 monclient: shutdown 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 -- 172.21.15.17:0/2963970748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 msgr2=0x7f66dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 --2- 172.21.15.17:0/2963970748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc09eca0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009850 tx=0x7f66cc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 -- 172.21.15.17:0/2963970748 shutdown_connections 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 --2- 172.21.15.17:0/2963970748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc09eca0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 -- 172.21.15.17:0/2963970748 >> 172.21.15.17:0/2963970748 conn(0x7f66dc071270 msgr2=0x7f66dc0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.650+0000 7f66e2860640 1 -- 172.21.15.17:0/2963970748 shutdown_connections 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.651+0000 7f66e2860640 1 -- 172.21.15.17:0/2963970748 wait complete. 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.651+0000 7f66e2860640 10 monclient: build_initial_monmap 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.651+0000 7f66e2860640 10 monclient: monmap: 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.651+0000 7f66e2860640 1 Processor -- start 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 -- start start 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient: init 2024-09-15T02:08:54.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-c con 0x7f66dc09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 0x7f66dc134510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-a con 0x7f66dc0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc134a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): picked mon.noname-b con 0x7f66dc131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:08:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc137e10 con 0x7f66dc0a0280 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc138fd0 con 0x7f66dc131b20 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1399e0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e2860640 10 monclient(hunting): authenticate will time out at 1794.648581s 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66db7fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 msgr2=0x7f66dc134510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66db7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 0x7f66dc134510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55444/0 (socket says 172.21.15.17:55444) 2024-09-15T02:08:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 1 -- 172.21.15.17:0/1615718756 learned_addr learned my addr 172.21.15.17:0/1615718756 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): get_auth_request con 0x7f66dc09f1e0 auth_method 0 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc134a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): get_auth_request con 0x7f66dc131b20 auth_method 0 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66e0dfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.652+0000 7f66dbfff640 10 monclient(hunting): handle_auth_done global_id 214151 payload 824 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 -- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 msgr2=0x7f66dc134a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc134a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 -- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 msgr2=0x7f66dc134510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 0x7f66dc134510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 monclient: found mon.noname-c 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:08:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.653+0000 7f66dbfff640 1 -- 172.21.15.17:0/1615718756 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc009390 con 0x7f66dc09f1e0 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66dbfff640 1 -- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 msgr2=0x7f66dc138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66dbfff640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009360 tx=0x7f66cc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc0318c0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: dump: 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.667 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: _finish_auth 0 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc031a20 con 0x7f66dc09f1e0 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 5 monclient: authenticate success, global_id 214151 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:08:54.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 10 monclient: _renew_subs 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc031de0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc136000 con 0x7f66dc09f1e0 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66c3fff640 10 monclient: dump: 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66d9ffb640 4 set_mon_vals no callback set 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 10 monclient: _renew_subs 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:54.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.655+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc136bd0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f66cc043030 con 0x7f66dc09f1e0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66e2860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66e2860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66e2860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc0724a0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66c3fff640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 0x7f66c803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f66cc041070 con 0x7f66dc09f1e0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66db7fe640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 0x7f66c803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.656+0000 7f66db7fe640 10 monclient: get_auth_request con 0x7f66c80386f0 auth_method 0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.659+0000 7f66db7fe640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 0x7f66c803aba0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f66d0003940 tx=0x7f66d0006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.661+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f66cc044380 con 0x7f66dc09f1e0 2024-09-15T02:08:54.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.661+0000 7f66c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:08:54.672 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.661+0000 7f66c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:08:54.888 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66e2860640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66e2860640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66e2860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f66dc0724a0 con 0x7f66dc09f1e0 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66c3fff640 1 -- 172.21.15.17:0/1615718756 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f66cc037020 con 0x7f66dc09f1e0 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:08:54.889 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.887+0000 7f66c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:08:54.890 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 msgr2=0x7f66c803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 0x7f66c803aba0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f66d0003940 tx=0x7f66d0006d80 comp rx=0 tx=0).stop 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 10 monclient: shutdown 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 msgr2=0x7f66dc138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:08:54.891 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009360 tx=0x7f66cc032000 comp rx=0 tx=0).stop 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 shutdown_connections 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f66c80386f0 0x7f66c803aba0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f66dc0a0280 0x7f66dc134510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66dc131b20 0x7f66dc134a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 --2- 172.21.15.17:0/1615718756 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f66dc09f1e0 0x7f66dc138a90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 >> 172.21.15.17:0/1615718756 conn(0x7f66dc071270 msgr2=0x7f66dc09e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 shutdown_connections 2024-09-15T02:08:54.892 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:54.889+0000 7f66e2860640 1 -- 172.21.15.17:0/1615718756 wait complete. 2024-09-15T02:08:57.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:08:58.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.009+0000 7f4b72617640 10 monclient: get_monmap_and_config 2024-09-15T02:08:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.009+0000 7f4b72617640 10 monclient: build_initial_monmap 2024-09-15T02:08:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.009+0000 7f4b72617640 10 monclient: monmap: 2024-09-15T02:08:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:08:58.011 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:08:58.010160+0000 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:08:58.010160+0000 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:08:58.012 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:08:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:08:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:08:58.013 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:08:58.013 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:08:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 Processor -- start 2024-09-15T02:08:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 -- start start 2024-09-15T02:08:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient: init 2024-09-15T02:08:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient: _reopen_session rank -1 2024-09-15T02:08:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T02:08:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:58.020 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6c0a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:08:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b6c0980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:08:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:08:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-b con 0x7f4b6c136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:08:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0a01e0 con 0x7f4b6c0980a0 2024-09-15T02:08:58.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a1580 con 0x7f4b6c0a58b0 2024-09-15T02:08:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): start opening mon connection 2024-09-15T02:08:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a5c80 con 0x7f4b6c136dd0 2024-09-15T02:08:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): _renew_subs 2024-09-15T02:08:58.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b72617640 10 monclient(hunting): authenticate will time out at 1798.006737s 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6b7fe640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f4b6c0980a0 legacy=0x7f4b6c0984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55450/0 (socket says 172.21.15.17:55450) 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 learned_addr learned my addr 172.21.15.17:0/1102796628 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b6c136dd0 auth_method 0 2024-09-15T02:08:58.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b70bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request con 0x7f4b6c0a58b0 auth_method 0 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.010+0000 7f4b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:08:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.011+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:08:58.212 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.211+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.17:6789/0 conn(0x7f4b6c0980a0 legacy=0x7f4b6c0984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b6c136dd0 auth_method 0 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:58.215 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.216 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request con 0x7f4b6c0a58b0 auth_method 0 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.217 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.212+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:08:58.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.612+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.17:6789/0 conn(0x7f4b6c0980a0 legacy=0x7f4b6c0984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b6c136dd0 auth_method 0 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request con 0x7f4b6c0a58b0 auth_method 0 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:58.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:58.613+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:08:59.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.413+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.17:6789/0 conn(0x7f4b6c0980a0 legacy=0x7f4b6c0984a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:08:59.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:59.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b6c136dd0 auth_method 0 2024-09-15T02:08:59.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:59.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request con 0x7f4b6c0a58b0 auth_method 0 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:08:59.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:08:59.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:08:59.414+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): tick 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 monclient(hunting): continuing hunt 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 msgr2=0x7f4b6c1391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b6c136dd0 0x7f4b6c1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 msgr2=0x7f4b6c132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.67:3300/0 conn(0x7f4b6c0a58b0 0x7f4b6c132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.17:6789/0 conn(0x7f4b6c0980a0 legacy=0x7f4b6c0984a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:01.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient: _add_conns ranks=[1,3] 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b58003680 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 0x7f4b58009790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4b580072e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 -- 172.21.15.17:0/1102796628 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c098bb0 con 0x7f4b58003680 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 1 -- 172.21.15.17:0/1102796628 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a1580 con 0x7f4b580072e0 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b69ffb640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 msgr2=0x7f4b58009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:01.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.011+0000 7f4b6b7fe640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 0x7f4b58009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:01.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.013+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.67:6789/0 conn(0x7f4b58003680 legacy=0x7f4b58005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:01.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.212+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 msgr2=0x7f4b58009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:01.213 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.212+0000 7f4b6b7fe640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 0x7f4b58009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:09:01.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.613+0000 7f4b6b7fe640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 msgr2=0x7f4b58009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:01.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:01.613+0000 7f4b6b7fe640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 0x7f4b58009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:09:02.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.389+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342893997 0 0) 0x7f4b6c098bb0 con 0x7f4b58003680 2024-09-15T02:09:02.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.389+0000 7f4b6affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.389+0000 7f4b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.389+0000 7f4b6affd640 10 monclient(hunting): my global_id is 204247 2024-09-15T02:09:02.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.389+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b540034e0 con 0x7f4b58003680 2024-09-15T02:09:02.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503604748 0 0) 0x7f4b540034e0 con 0x7f4b58003680 2024-09-15T02:09:02.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 msgr2=0x7f4b58009790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 --2- 172.21.15.17:0/1102796628 >> v2:172.21.15.17:3300/0 conn(0x7f4b580072e0 0x7f4b58009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0a01e0 con 0x7f4b58003680 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4b5c00d480 con 0x7f4b58003680 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b72617640 5 monclient: authenticate success, global_id 204247 2024-09-15T02:09:02.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b72617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: dump: 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.395 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 >> v1:172.21.15.67:6789/0 conn(0x7f4b58003680 legacy=0x7f4b58005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 reap_dead start 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 0x7f4b54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): picked mon.a con 0x7f4b540039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b54004780 0x7f4b54004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): picked mon.c con 0x7f4b54004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): picked mon.b con 0x7f4b54005230 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:02.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6bfff640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 msgr2=0x7f4b54003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0a1580 con 0x7f4b540039d0 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 0x7f4b54003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5400c1f0 con 0x7f4b54005230 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5400c3b0 con 0x7f4b54004780 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b6affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b72617640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b54005230 auth_method 0 2024-09-15T02:09:02.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.390+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_done global_id 204247 payload 293 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b54004780 msgr2=0x7f4b54004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b54004780 0x7f4b54004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 msgr2=0x7f4b54003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 0x7f4b54003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 monclient: found mon.b 2024-09-15T02:09:02.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b540034e0 con 0x7f4b54005230 2024-09-15T02:09:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.391+0000 7f4b70bb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b70bb4640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 msgr2=0x7f4b5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b5400b9d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c06a060 tx=0x7f4b64009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c5211b28ef1d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64030480 con 0x7f4b54005230 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: dump: 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.405 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6402d030 con 0x7f4b54005230 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b72617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b72617640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 1 -- 172.21.15.17:0/1102796628 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64030830 con 0x7f4b54005230 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b72617640 4 set_mon_vals no callback set 2024-09-15T02:09:02.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.394+0000 7f4b6affd640 10 monclient: dump: 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 10 monclient: shutdown 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 -- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 msgr2=0x7f4b5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b5400b9d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c06a060 tx=0x7f4b64009040 comp rx=0 tx=0).stop 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 -- 172.21.15.17:0/1102796628 shutdown_connections 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b5400b9d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b54004780 0x7f4b54004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 --2- 172.21.15.17:0/1102796628 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b540039d0 0x7f4b54003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 -- 172.21.15.17:0/1102796628 >> 172.21.15.17:0/1102796628 conn(0x7f4b6c070ed0 msgr2=0x7f4b6c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 -- 172.21.15.17:0/1102796628 shutdown_connections 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.395+0000 7f4b72617640 1 -- 172.21.15.17:0/1102796628 wait complete. 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.396+0000 7f4b72617640 10 monclient: build_initial_monmap 2024-09-15T02:09:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.396+0000 7f4b72617640 10 monclient: monmap: 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.396+0000 7f4b72617640 1 Processor -- start 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 -- start start 2024-09-15T02:09:02.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient: init 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 0x7f4b6c1343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-c con 0x7f4b540039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 0x7f4b6c134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-a con 0x7f4b54004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): picked mon.noname-b con 0x7f4b54005230 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c11ca50 con 0x7f4b54004780 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c1317c0 con 0x7f4b54005230 2024-09-15T02:09:02.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c097880 con 0x7f4b540039d0 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b72617640 10 monclient(hunting): authenticate will time out at 1802.393560s 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6b7fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 msgr2=0x7f4b6c134920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6b7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 0x7f4b6c134920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6bfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 msgr2=0x7f4b6c1343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6bfff640 1 -- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 msgr2=0x7f4b6c1343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 0x7f4b6c1343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b6bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 0x7f4b6c1343e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59560/0 (socket says 172.21.15.17:59560) 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 1 -- 172.21.15.17:0/3945059955 learned_addr learned my addr 172.21.15.17:0/3945059955 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:02.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request con 0x7f4b54005230 auth_method 0 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.397+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 10 monclient(hunting): handle_auth_done global_id 204250 payload 824 2024-09-15T02:09:02.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 -- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 msgr2=0x7f4b6c1343e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 0x7f4b6c1343e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 -- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 msgr2=0x7f4b6c134920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 0x7f4b6c134920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 monclient: found mon.noname-b 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.398+0000 7f4b70bb4640 1 -- 172.21.15.17:0/3945059955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c11c770 con 0x7f4b54005230 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b70bb4640 1 -- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 msgr2=0x7f4b6c134e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b70bb4640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b6402aa80 tx=0x7f4b64038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6403e440 con 0x7f4b54005230 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: dump: 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.414 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b72617640 5 monclient: authenticate success, global_id 204250 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6400f040 con 0x7f4b54005230 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6403e7b0 con 0x7f4b54005230 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b53fff640 10 monclient: dump: 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.415 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b72617640 10 monclient: _renew_subs 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b72617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c135ff0 con 0x7f4b54005230 2024-09-15T02:09:02.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.400+0000 7f4b6a7fc640 4 set_mon_vals no callback set 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.401+0000 7f4b72617640 10 monclient: _renew_subs 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.401+0000 7f4b72617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.401+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c1353a0 con 0x7f4b54005230 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4b6403a040 con 0x7f4b54005230 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b72617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b72617640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b72617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c0724a0 con 0x7f4b54005230 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b53fff640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 0x7f4b5803b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4b6406c1d0 con 0x7f4b54005230 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.402+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 0x7f4b5803b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.403+0000 7f4b6bfff640 10 monclient: get_auth_request con 0x7f4b58038e00 auth_method 0 2024-09-15T02:09:02.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.405+0000 7f4b6bfff640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 0x7f4b5803b2b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006e70 tx=0x7f4b5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.407+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4b6403b5c0 con 0x7f4b54005230 2024-09-15T02:09:02.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.407+0000 7f4b53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.407+0000 7f4b53fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:02.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b72617640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:02.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b72617640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:02.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b72617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b6c09fc90 con 0x7f4b54005230 2024-09-15T02:09:02.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b53fff640 1 -- 172.21.15.17:0/3945059955 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f4b64043610 con 0x7f4b54005230 2024-09-15T02:09:02.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:02.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.641+0000 7f4b53fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:02.643 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":176,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:02.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 msgr2=0x7f4b5803b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 0x7f4b5803b2b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006e70 tx=0x7f4b5c008040 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 10 monclient: shutdown 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 msgr2=0x7f4b6c134e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b6402aa80 tx=0x7f4b64038000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 shutdown_connections 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4b58038e00 0x7f4b5803b2b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b54004780 0x7f4b6c134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b54005230 0x7f4b6c134e60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 --2- 172.21.15.17:0/3945059955 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4b540039d0 0x7f4b6c1343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 >> 172.21.15.17:0/3945059955 conn(0x7f4b6c070ed0 msgr2=0x7f4b6c068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 shutdown_connections 2024-09-15T02:09:02.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.643+0000 7f4b72617640 1 -- 172.21.15.17:0/3945059955 wait complete. 2024-09-15T02:09:02.656 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:09:02.656 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:09:02.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.753+0000 7f309c59d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.753+0000 7f309c59d640 10 monclient: build_initial_monmap 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.753+0000 7f309c59d640 10 monclient: monmap: 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.755 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:02.754734+0000 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:02.754734+0000 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:02.756 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.754+0000 7f309c59d640 1 Processor -- start 2024-09-15T02:09:02.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.754+0000 7f309c59d640 1 -- start start 2024-09-15T02:09:02.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.754+0000 7f309c59d640 10 monclient: init 2024-09-15T02:09:02.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 0x7f309409adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-a con 0x7f309409a9f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f309412e420 0x7f309409b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f309412e420 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f309412e810 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3094098fb0 con 0x7f309412e810 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309412e130 con 0x7f309409a9f0 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309409b300 con 0x7f309412e420 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309c59d640 10 monclient(hunting): authenticate will time out at 1802.751236s 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309a339640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 msgr2=0x7f309409adc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:02.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309a339640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 0x7f309409adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f309412e420 0x7f309409b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309ab3a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f309412e810 0x7f309409bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:51208/0 (socket says 172.21.15.17:51208) 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f309ab3a640 1 -- 172.21.15.17:0/401071454 learned_addr learned my addr 172.21.15.17:0/401071454 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f309412e420 0x7f309409b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:59574/0 (socket says 172.21.15.17:59574) 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): get_auth_request con 0x7f309412e420 auth_method 0 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient(hunting): handle_auth_done global_id 204253 payload 274 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 -- 172.21.15.17:0/401071454 >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 msgr2=0x7f309409adc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 --2- 172.21.15.17:0/401071454 >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 0x7f309409adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 -- 172.21.15.17:0/401071454 >> v1:172.21.15.67:6789/0 conn(0x7f309412e810 legacy=0x7f309409bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 monclient: found mon.noname-c 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:09:02.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.755+0000 7f3099b38640 1 -- 172.21.15.17:0/401071454 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094098cd0 con 0x7f309412e420 2024-09-15T02:09:02.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.758+0000 7f3099b38640 1 -- 172.21.15.17:0/401071454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e420 msgr2=0x7f309409b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.758+0000 7f3099b38640 1 --2- 172.21.15.17:0/401071454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e420 0x7f309409b850 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30940769e0 tx=0x7f308400b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f56c4baadc0a2e4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 1 -- 172.21.15.17:0/401071454 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840317a0 con 0x7f309412e420 2024-09-15T02:09:02.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: dump: 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 1 -- 172.21.15.17:0/401071454 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084031900 con 0x7f309412e420 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 1 -- 172.21.15.17:0/401071454 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084031ca0 con 0x7f309412e420 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f3099337640 10 monclient: dump: 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 5 monclient: authenticate success, global_id 204253 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 4 set_mon_vals no callback set 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 10 monclient: shutdown 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 -- 172.21.15.17:0/401071454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e420 msgr2=0x7f309409b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 --2- 172.21.15.17:0/401071454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e420 0x7f309409b850 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30940769e0 tx=0x7f308400b3b0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 -- 172.21.15.17:0/401071454 shutdown_connections 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 --2- 172.21.15.17:0/401071454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e420 0x7f309409b850 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 --2- 172.21.15.17:0/401071454 >> v2:172.21.15.17:3300/0 conn(0x7f309409a9f0 0x7f309409adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 -- 172.21.15.17:0/401071454 >> 172.21.15.17:0/401071454 conn(0x7f30940a1000 msgr2=0x7f30940a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.759+0000 7f309c59d640 1 -- 172.21.15.17:0/401071454 shutdown_connections 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 1 -- 172.21.15.17:0/401071454 wait complete. 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 10 monclient: build_initial_monmap 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 10 monclient: monmap: 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:02.771 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 1 Processor -- start 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 1 -- start start 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.760+0000 7f309c59d640 10 monclient: init 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f309409a9f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f309412e420 0x7f309413d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-a con 0x7f309412e420 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e810 0x7f309413d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): picked mon.noname-b con 0x7f309412e810 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:02.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940983b0 con 0x7f309412e420 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309409f360 con 0x7f309412e810 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309406b3a0 con 0x7f309409a9f0 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309c59d640 10 monclient(hunting): authenticate will time out at 1802.757230s 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f3099b38640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f309412e420 msgr2=0x7f309413d190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f3099b38640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f309412e420 0x7f309413d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55510/0 (socket says 172.21.15.17:55510) 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 -- 172.21.15.17:0/1383894171 learned_addr learned my addr 172.21.15.17:0/1383894171 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): get_auth_request con 0x7f309409a9f0 auth_method 0 2024-09-15T02:09:02.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309ab3a640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e810 0x7f309413d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309ab3a640 10 monclient(hunting): get_auth_request con 0x7f309412e810 auth_method 0 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient(hunting): handle_auth_done global_id 214154 payload 824 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e810 msgr2=0x7f309413d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f309412e810 0x7f309413d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f309412e420 msgr2=0x7f309413d190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f309412e420 0x7f309413d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 monclient: found mon.noname-c 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.761+0000 7f309a339640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308400b630 con 0x7f309409a9f0 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.763+0000 7f309a339640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 msgr2=0x7f309406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.763+0000 7f309a339640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f309000b600 tx=0x7f30900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309a339640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f309000b600 tx=0x7f30900091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f3090015070 mon_map magic: 0 v1 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090015070 con 0x7f309409a9f0 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 10 monclient: dump: 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 5 monclient: authenticate success, global_id 214154 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 monclient: _renew_subs 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090003d90 con 0x7f309409a9f0 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309406ce80 con 0x7f309409a9f0 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f308a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f3098b36640 4 set_mon_vals no callback set 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 monclient: _renew_subs 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309406f330 con 0x7f309409a9f0 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 0x7f30940698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3094063c20 con 0x7f309406d0c0 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.764+0000 7f3099b38640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 0x7f30940698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient: get_auth_request con 0x7f309406d0c0 auth_method 0 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090006380 con 0x7f309409a9f0 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 10 monclient: dump: 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3090010440 con 0x7f309409a9f0 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 10 monclient(hunting): handle_auth_done global_id 214157 payload 824 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f3099b38640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30940724a0 con 0x7f309406d0c0 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f308a7fc640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3078038740 0x7f307803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f309ab3a640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3078038740 0x7f307803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.765+0000 7f309ab3a640 10 monclient: get_auth_request con 0x7f3078038740 auth_method 0 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f3099b38640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 msgr2=0x7f30940698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f3099b38640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 0x7f30940698d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3084036000 tx=0x7f3084037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084031c80 con 0x7f309406d0c0 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 10 monclient: dump: 2024-09-15T02:09:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f308403f3f0 con 0x7f309406d0c0 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 msgr2=0x7f30940698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 0x7f30940698d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3084036000 tx=0x7f3084037000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 msgr2=0x7f30940698d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.767+0000 7f308a7fc640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309406d0c0 0x7f30940698d0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3084036000 tx=0x7f3084037000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.768+0000 7f309ab3a640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3078038740 0x7f307803abf0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f309406ae90 tx=0x7f308c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f309c59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f309c59d640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 0x7f309406a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f309c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309406caf0 con 0x7f3094069e10 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 0x7f309406a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient: get_auth_request con 0x7f3094069e10 auth_method 0 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.773+0000 7f3099b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.774+0000 7f3099b38640 10 monclient(hunting): handle_auth_done global_id 214160 payload 824 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.774+0000 7f3099b38640 1 -- 172.21.15.17:0/1383894171 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f309409d6c0 con 0x7f3094069e10 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f3099b38640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 msgr2=0x7f309406a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f3099b38640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 0x7f309406a1e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30940639c0 tx=0x7f3084040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084048c80 con 0x7f3094069e10 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 10 monclient: dump: 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.785 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f30840333a0 con 0x7f3094069e10 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 msgr2=0x7f309406a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 0x7f309406a1e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30940639c0 tx=0x7f3084040000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 msgr2=0x7f309406a1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:02.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.776+0000 7f308a7fc640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f3094069e10 0x7f309406a1e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30940639c0 tx=0x7f3084040000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 176, 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.787 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.788 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.789 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:02.790 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:02.796 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:02.797 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:02.799 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:02.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3078038740 msgr2=0x7f307803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f3078038740 0x7f307803abf0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f309406ae90 tx=0x7f308c009040 comp rx=0 tx=0).stop 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 10 monclient: shutdown 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 msgr2=0x7f309406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f309000b600 tx=0x7f30900091a0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309ab3a640 1 -- 172.21.15.17:0/1383894171 reap_dead start 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f308a7fc640 1 -- 172.21.15.17:0/1383894171 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f309004c160 con 0x7f309409a9f0 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 shutdown_connections 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 --2- 172.21.15.17:0/1383894171 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f309409a9f0 0x7f309406ec50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 >> 172.21.15.17:0/1383894171 conn(0x7f30940a1000 msgr2=0x7f3094130cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 shutdown_connections 2024-09-15T02:09:02.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.777+0000 7f309c59d640 1 -- 172.21.15.17:0/1383894171 wait complete. 2024-09-15T02:09:02.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:09:02.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:09:02.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.898+0000 7f2b8e31d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.898+0000 7f2b8e31d640 10 monclient: build_initial_monmap 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.898+0000 7f2b8e31d640 10 monclient: monmap: 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:02.899610+0000 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:02.899610+0000 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:02.901 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.899+0000 7f2b8e31d640 1 Processor -- start 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.899+0000 7f2b8e31d640 1 -- start start 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.899+0000 7f2b8e31d640 10 monclient: init 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b88098210 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:02.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 0x7f2b880985e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b8809cd90 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b88098b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8809d6d0 con 0x7f2b88098b20 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8809e170 con 0x7f2b8809cd90 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88132430 con 0x7f2b88098210 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b8e31d640 10 monclient(hunting): authenticate will time out at 1802.896203s 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:51234/0 (socket says 172.21.15.17:51234) 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 -- 172.21.15.17:0/485926874 learned_addr learned my addr 172.21.15.17:0/485926874 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:02.905 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): get_auth_request con 0x7f2b88098210 auth_method 0 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 1 --2- 172.21.15.17:0/485926874 >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 0x7f2b880985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 10 monclient(hunting): get_auth_request con 0x7f2b8809cd90 auth_method 0 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient(hunting): handle_auth_done global_id 214163 payload 274 2024-09-15T02:09:02.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 -- 172.21.15.17:0/485926874 >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 msgr2=0x7f2b880985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 --2- 172.21.15.17:0/485926874 >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 0x7f2b880985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 -- 172.21.15.17:0/485926874 >> v1:172.21.15.67:6789/0 conn(0x7f2b88098b20 legacy=0x7f2b88067090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 monclient: found mon.noname-b 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b87fff640 1 -- 172.21.15.17:0/485926874 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b881325a0 con 0x7f2b88098210 2024-09-15T02:09:02.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.900+0000 7f2b877fe640 1 --2- 172.21.15.17:0/485926874 >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 0x7f2b880985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b87fff640 1 -- 172.21.15.17:0/485926874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 msgr2=0x7f2b88131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b87fff640 1 --2- 172.21.15.17:0/485926874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009940 tx=0x7f2b780091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa3fa4b267b0c5ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 1 -- 172.21.15.17:0/485926874 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78032440 con 0x7f2b88098210 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: dump: 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.913 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 1 -- 172.21.15.17:0/485926874 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b780325a0 con 0x7f2b88098210 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 1 -- 172.21.15.17:0/485926874 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78032940 con 0x7f2b88098210 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b8e31d640 5 monclient: authenticate success, global_id 214163 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b8e31d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b8e31d640 4 set_mon_vals no callback set 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.903+0000 7f2b86ffd640 10 monclient: dump: 2024-09-15T02:09:02.915 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 10 monclient: shutdown 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 -- 172.21.15.17:0/485926874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 msgr2=0x7f2b88131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/485926874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009940 tx=0x7f2b780091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 -- 172.21.15.17:0/485926874 shutdown_connections 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/485926874 >> v2:172.21.15.67:3300/0 conn(0x7f2b8809cd90 0x7f2b880985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/485926874 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b88098210 0x7f2b88131ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 -- 172.21.15.17:0/485926874 >> 172.21.15.17:0/485926874 conn(0x7f2b880a1ee0 msgr2=0x7f2b880a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 -- 172.21.15.17:0/485926874 shutdown_connections 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 -- 172.21.15.17:0/485926874 wait complete. 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 10 monclient: build_initial_monmap 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 10 monclient: monmap: 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:02.917 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.904+0000 7f2b8e31d640 1 Processor -- start 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 -- start start 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient: init 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b88098210 0x7f2b88141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b88098210 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b88098b20 0x7f2b88130060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b88098b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b8809cd90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88130f50 con 0x7f2b88098210 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8811b6c0 con 0x7f2b88098b20 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8811d180 con 0x7f2b8809cd90 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8e31d640 10 monclient(hunting): authenticate will time out at 1802.901565s 2024-09-15T02:09:02.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b87fff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b88098210 msgr2=0x7f2b88141200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b88098210 0x7f2b88141200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:51250/0 (socket says 172.21.15.17:51250) 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 learned_addr learned my addr 172.21.15.17:0/3465009551 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): get_auth_request con 0x7f2b8809cd90 auth_method 0 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b877fe640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b88098b20 0x7f2b88130060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient(hunting): handle_auth_done global_id 214166 payload 824 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b88098b20 msgr2=0x7f2b88130060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b88098b20 0x7f2b88130060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b88098210 msgr2=0x7f2b88141200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2b88098210 0x7f2b88141200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient: _finish_hunting 0 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 monclient: found mon.noname-c 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.905+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78009450 con 0x7f2b8809cd90 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 msgr2=0x7f2b88119b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8c8ba640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c810 tx=0x7f2b80002620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000ce00 con 0x7f2b8809cd90 2024-09-15T02:09:02.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: dump: 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: _finish_auth 0 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8000aa60 con 0x7f2b8809cd90 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000ae20 con 0x7f2b8809cd90 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 5 monclient: authenticate success, global_id 214166 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 10 monclient: _renew_subs 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8811d460 con 0x7f2b8809cd90 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b867fc640 4 set_mon_vals no callback set 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b6ffff640 10 monclient: dump: 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 10 monclient: _renew_subs 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.908+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881418e0 con 0x7f2b8809cd90 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b8e31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 0x7f2b8811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88130970 con 0x7f2b881305a0 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2b80012630 con 0x7f2b8809cd90 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b6ffff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b74038740 0x7f2b7403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 0x7f2b8811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f2b8004cad0 con 0x7f2b8809cd90 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient: get_auth_request con 0x7f2b881305a0 auth_method 0 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.925 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b877fe640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b74038740 0x7f2b7403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.909+0000 7f2b877fe640 10 monclient: get_auth_request con 0x7f2b74038740 auth_method 0 2024-09-15T02:09:02.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.910+0000 7f2b87fff640 10 monclient(hunting): handle_auth_done global_id 204262 payload 824 2024-09-15T02:09:02.926 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.910+0000 7f2b87fff640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b8811d7d0 con 0x7f2b881305a0 2024-09-15T02:09:02.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b877fe640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b74038740 0x7f2b7403abf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006e70 tx=0x7f2b7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b87fff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 msgr2=0x7f2b8811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b87fff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 0x7f2b8811a050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2b78034000 tx=0x7f2b78032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7803c9e0 con 0x7f2b881305a0 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 10 monclient: dump: 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2b7803cb40 con 0x7f2b881305a0 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 msgr2=0x7f2b8811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 0x7f2b8811a050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2b78034000 tx=0x7f2b78032000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 msgr2=0x7f2b8811a050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.912+0000 7f2b6ffff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b881305a0 0x7f2b8811a050 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.918+0000 7f2b8e31d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.918+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 0x7f2b880992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.918+0000 7f2b8e31d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.918+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8811b030 con 0x7f2b8811a590 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 0x7f2b880992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient: get_auth_request con 0x7f2b8811a590 auth_method 0 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 10 monclient(hunting): handle_auth_done global_id 204265 payload 824 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.919+0000 7f2b87fff640 1 -- 172.21.15.17:0/3465009551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2b8811a990 con 0x7f2b8811a590 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.921+0000 7f2b87fff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 msgr2=0x7f2b880992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.921+0000 7f2b87fff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 0x7f2b880992c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009940 tx=0x7f2b7803c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78030ae0 con 0x7f2b8811a590 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 10 monclient: dump: 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:02.932 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f2b7803c7d0 con 0x7f2b8811a590 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 msgr2=0x7f2b880992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 0x7f2b880992c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009940 tx=0x7f2b7803c000 comp rx=0 tx=0).stop 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 msgr2=0x7f2b880992c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:02.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b6ffff640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811a590 0x7f2b880992c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 176, 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:02.947 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b74038740 msgr2=0x7f2b7403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2b74038740 0x7f2b7403abf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006e70 tx=0x7f2b7c008040 comp rx=0 tx=0).stop 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 10 monclient: shutdown 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 msgr2=0x7f2b88119b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8c8ba640 1 -- 172.21.15.17:0/3465009551 reap_dead start 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c810 tx=0x7f2b80002620 comp rx=0 tx=0).stop 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 shutdown_connections 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 --2- 172.21.15.17:0/3465009551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2b8809cd90 0x7f2b88119b10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 >> 172.21.15.17:0/3465009551 conn(0x7f2b880a1ee0 msgr2=0x7f2b88134a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:02.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 shutdown_connections 2024-09-15T02:09:02.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:02.922+0000 7f2b8e31d640 1 -- 172.21.15.17:0/3465009551 wait complete. 2024-09-15T02:09:02.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:09:02.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:09:22.951 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:09:22.951 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T02:09:22.952 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T02:09:22.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T02:09:22.955 INFO:tasks.ceph.mon.a:Started 2024-09-15T02:09:22.955 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:09:22.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.062+0000 7fecb349f640 10 monclient: get_monmap_and_config 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.062+0000 7fecb349f640 10 monclient: build_initial_monmap 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.062+0000 7fecb349f640 10 monclient: monmap: 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:23.063220+0000 2024-09-15T02:09:23.064 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:23.063220+0000 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.065 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.062+0000 7fecb349f640 1 Processor -- start 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 -- start start 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient: init 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecac131b20 0x7fecac09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-c con 0x7fecac131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:23.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-a con 0x7fecac0a0280 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecac09f1e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac0a0040 con 0x7fecac09f1e0 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac09bff0 con 0x7fecac0a0280 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0a14a0 con 0x7fecac131b20 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb349f640 10 monclient(hunting): authenticate will time out at 1823.059735s 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb0a3a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb1a3c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fecac09f1e0 0x7fecac068e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38920/0 (socket says 172.21.15.17:38920) 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb1a3c640 1 -- 172.21.15.17:0/3221063365 learned_addr learned my addr 172.21.15.17:0/3221063365 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb0a3a640 10 monclient(hunting): get_auth_request con 0x7fecac0a0280 auth_method 0 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb0a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb0a3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.063+0000 7fecb0a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb0a3a640 1 -- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 msgr2=0x7fecac131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb0a3a640 1 -- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 msgr2=0x7fecac131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb0a3a640 1 --2- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:23.070 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb0a3a640 1 --2- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 --2- 172.21.15.17:0/3221063365 >> v2:172.21.15.67:3300/0 conn(0x7fecac131b20 0x7fecac09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): get_auth_request con 0x7fecac131b20 auth_method 0 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient(hunting): handle_auth_done global_id 204268 payload 274 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 -- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 msgr2=0x7fecac131ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 --2- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 -- 172.21.15.17:0/3221063365 >> v1:172.21.15.17:6789/0 conn(0x7fecac09f1e0 legacy=0x7fecac068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient: _finish_hunting 0 2024-09-15T02:09:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 monclient: found mon.noname-c 2024-09-15T02:09:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:09:23.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.064+0000 7fecb123b640 1 -- 172.21.15.17:0/3221063365 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac132430 con 0x7fecac131b20 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb123b640 1 -- 172.21.15.17:0/3221063365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac131b20 msgr2=0x7fecac09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb123b640 1 --2- 172.21.15.17:0/3221063365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac131b20 0x7fecac09eca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fecac09f710 tx=0x7feca80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a23433b1198d37bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 1 -- 172.21.15.17:0/3221063365 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca802ae00 con 0x7fecac131b20 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: dump: 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:23.079 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: _finish_auth 0 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb349f640 5 monclient: authenticate success, global_id 204268 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 1 -- 172.21.15.17:0/3221063365 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8032440 con 0x7fecac131b20 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb349f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb349f640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7fecb349f640 4 set_mon_vals no callback set 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 1 -- 172.21.15.17:0/3221063365 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80327e0 con 0x7fecac131b20 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.067+0000 7feca3fff640 10 monclient: dump: 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 10 monclient: shutdown 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 -- 172.21.15.17:0/3221063365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac131b20 msgr2=0x7fecac09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 --2- 172.21.15.17:0/3221063365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac131b20 0x7fecac09eca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fecac09f710 tx=0x7feca80090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 -- 172.21.15.17:0/3221063365 shutdown_connections 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 --2- 172.21.15.17:0/3221063365 >> v2:172.21.15.17:3300/0 conn(0x7fecac0a0280 0x7fecac131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.083 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 --2- 172.21.15.17:0/3221063365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac131b20 0x7fecac09eca0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 -- 172.21.15.17:0/3221063365 >> 172.21.15.17:0/3221063365 conn(0x7fecac071270 msgr2=0x7fecac0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 -- 172.21.15.17:0/3221063365 shutdown_connections 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 1 -- 172.21.15.17:0/3221063365 wait complete. 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 10 monclient: build_initial_monmap 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.068+0000 7fecb349f640 10 monclient: monmap: 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:23.084 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 Processor -- start 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 -- start start 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient: init 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:23.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-a con 0x7fecac09f1e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 0x7fecac12ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-b con 0x7fecac0a0280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 0x7fecac098c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): picked mon.noname-c con 0x7fecac131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac11bea0 con 0x7fecac09f1e0 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac131830 con 0x7fecac0a0280 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac141f10 con 0x7fecac131b20 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:23.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.069+0000 7fecb349f640 10 monclient(hunting): authenticate will time out at 1823.065946s 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57406/0 (socket says 172.21.15.17:57406) 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 0x7fecac098c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 -- 172.21.15.17:0/3027488881 learned_addr learned my addr 172.21.15.17:0/3027488881 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 10 monclient(hunting): get_auth_request con 0x7fecac09f1e0 auth_method 0 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:23.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): get_auth_request con 0x7fecac131b20 auth_method 0 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 0x7fecac12ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 msgr2=0x7fecac141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 msgr2=0x7fecac141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): get_auth_request con 0x7fecac0a0280 auth_method 0 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb123b640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient(hunting): handle_auth_done global_id 214169 payload 824 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 msgr2=0x7fecac12ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 0x7fecac12ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 msgr2=0x7fecac141030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:23.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient: _finish_hunting 0 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 monclient: found mon.noname-c 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb1a3c640 1 -- 172.21.15.17:0/3027488881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8009390 con 0x7fecac131b20 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.070+0000 7fecb0a3a640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 0x7fecac12ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7fecb1a3c640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 msgr2=0x7fecac098c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7fecb1a3c640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 0x7fecac098c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feca400cb90 tx=0x7feca40052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca400a850 con 0x7fecac131b20 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 10 monclient: dump: 2024-09-15T02:09:23.090 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 10 monclient: _finish_auth 0 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca400a9d0 con 0x7fecac131b20 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:23.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca400ad90 con 0x7fecac131b20 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.072+0000 7fecb349f640 5 monclient: authenticate success, global_id 214169 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 10 monclient: _renew_subs 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac09ac30 con 0x7fecac131b20 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7feca17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7feca17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7feca17fa640 10 monclient: dump: 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:23.092 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7feca37fe640 4 set_mon_vals no callback set 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 10 monclient: _renew_subs 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.073+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac09b2c0 con 0x7fecac131b20 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb349f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb349f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb349f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7feca4007300 con 0x7fecac131b20 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecac0724a0 con 0x7fecac131b20 2024-09-15T02:09:23.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7feca17fa640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 0x7fec9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7feca404c710 con 0x7fecac131b20 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb123b640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 0x7fec9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.074+0000 7fecb123b640 10 monclient: get_auth_request con 0x7fec90038740 auth_method 0 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.077+0000 7fecb123b640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 0x7fec9003abf0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7feca8035040 tx=0x7feca8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.079+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feca401a360 con 0x7fecac131b20 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.079+0000 7feca17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.079+0000 7feca17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:23.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7fecb349f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7fecb349f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7fecb349f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fecac0008d0 con 0x7fecac131b20 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7feca17fa640 1 -- 172.21.15.17:0/3027488881 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7feca4010af0 con 0x7fecac131b20 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7feca17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.311+0000 7feca17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:23.313 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":176,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 msgr2=0x7fec9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 0x7fec9003abf0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7feca8035040 tx=0x7feca8032000 comp rx=0 tx=0).stop 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 10 monclient: shutdown 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 msgr2=0x7fecac098c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 0x7fecac098c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feca400cb90 tx=0x7feca40052b0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 shutdown_connections 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fec90038740 0x7fec9003abf0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fecac131b20 0x7fecac098c00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac0a0280 0x7fecac12ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 --2- 172.21.15.17:0/3027488881 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fecac09f1e0 0x7fecac141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 >> 172.21.15.17:0/3027488881 conn(0x7fecac071270 msgr2=0x7fecac0a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 shutdown_connections 2024-09-15T02:09:23.316 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:23.313+0000 7fecb349f640 1 -- 172.21.15.17:0/3027488881 wait complete. 2024-09-15T02:09:26.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:26.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.434+0000 7f8d04cc0640 10 monclient: get_monmap_and_config 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.434+0000 7f8d04cc0640 10 monclient: build_initial_monmap 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.434+0000 7f8d04cc0640 10 monclient: monmap: 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:26.435196+0000 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:26.435196+0000 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:26.437 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:26.437 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:26.437 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.434+0000 7f8d04cc0640 1 Processor -- start 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 -- start start 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient: init 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d00132360 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 0x7f8d0006e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f8d0006e140 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 0x7f8d00139020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f8d00136c40 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0009cbb0 con 0x7f8d00132360 2024-09-15T02:09:26.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d00132170 con 0x7f8d00136c40 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00132730 con 0x7f8d0006e140 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8d04cc0640 10 monclient(hunting): authenticate will time out at 1826.431738s 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 0x7f8d0006e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfe59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8d00132360 0x7f8d00131a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:38932/0 (socket says 172.21.15.17:38932) 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfe59c640 1 -- 172.21.15.17:0/3139381317 learned_addr learned my addr 172.21.15.17:0/3139381317 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfdd9b640 10 monclient(hunting): get_auth_request con 0x7f8d0006e140 auth_method 0 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.435+0000 7f8cfdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:26.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfed9d640 1 -- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 msgr2=0x7f8d00139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfed9d640 1 -- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 msgr2=0x7f8d00139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfed9d640 1 --2- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 0x7f8d00139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfed9d640 1 --2- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 0x7f8d00139020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_done global_id 214172 payload 274 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 msgr2=0x7f8d00139020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 1 --2- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 0x7f8d00139020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3139381317 >> v1:172.21.15.17:6789/0 conn(0x7f8d00132360 legacy=0x7f8d00131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 1 monclient: found mon.noname-b 2024-09-15T02:09:26.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:26.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.436+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3139381317 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0009d3f0 con 0x7f8d0006e140 2024-09-15T02:09:26.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3139381317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 msgr2=0x7f8d0006e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:26.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfdd9b640 1 --2- 172.21.15.17:0/3139381317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 0x7f8d0006e540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c1f0 tx=0x7f8ce80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e90c268a954b0d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:26.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 1 -- 172.21.15.17:0/3139381317 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce802ae90 con 0x7f8d0006e140 2024-09-15T02:09:26.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:26.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: dump: 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: _finish_auth 0 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:26.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 1 -- 172.21.15.17:0/3139381317 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8032440 con 0x7f8d0006e140 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 5 monclient: authenticate success, global_id 214172 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 1 -- 172.21.15.17:0/3139381317 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80327e0 con 0x7f8d0006e140 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 4 set_mon_vals no callback set 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8cfd59a640 10 monclient: dump: 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.449 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 10 monclient: shutdown 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3139381317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 msgr2=0x7f8d0006e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3139381317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 0x7f8d0006e540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c1f0 tx=0x7f8ce80090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3139381317 shutdown_connections 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3139381317 >> v2:172.21.15.67:3300/0 conn(0x7f8d00136c40 0x7f8d00139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3139381317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d0006e140 0x7f8d0006e540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3139381317 >> 172.21.15.17:0/3139381317 conn(0x7f8d00067b10 msgr2=0x7f8d000a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3139381317 shutdown_connections 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.439+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3139381317 wait complete. 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.440+0000 7f8d04cc0640 10 monclient: build_initial_monmap 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.440+0000 7f8d04cc0640 10 monclient: monmap: 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:26.451 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.440+0000 7f8d04cc0640 1 Processor -- start 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.440+0000 7f8d04cc0640 1 -- start start 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.440+0000 7f8d04cc0640 10 monclient: init 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f8d0006e140 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f8d00132360 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d00136c40 0x7f8d0006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f8d00136c40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:26.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00132c50 con 0x7f8d0006e140 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00133bb0 con 0x7f8d00136c40 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00133e90 con 0x7f8d00132360 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8d04cc0640 10 monclient(hunting): authenticate will time out at 1826.437264s 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:57422/0 (socket says 172.21.15.17:57422) 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:46240/0 (socket says 172.21.15.17:46240) 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 -- 172.21.15.17:0/3178297640 learned_addr learned my addr 172.21.15.17:0/3178297640 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 10 monclient(hunting): get_auth_request con 0x7f8d0006e140 auth_method 0 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): get_auth_request con 0x7f8d00132360 auth_method 0 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 msgr2=0x7f8d0006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 msgr2=0x7f8d0006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfe59c640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:26.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfed9d640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d00136c40 0x7f8d0006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfed9d640 10 monclient(hunting): get_auth_request con 0x7f8d00136c40 auth_method 0 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient(hunting): handle_auth_done global_id 214175 payload 824 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d00136c40 msgr2=0x7f8d0006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d00136c40 0x7f8d0006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 msgr2=0x7f8d0006bd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient: _finish_hunting 0 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 monclient: found mon.noname-c 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.441+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3178297640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8009390 con 0x7f8d00132360 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cfdd9b640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 msgr2=0x7f8d0006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cfdd9b640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8031000 tx=0x7f8ce8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80318c0 con 0x7f8d00132360 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: dump: 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:26.456 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 5 monclient: authenticate success, global_id 214175 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 10 monclient: _renew_subs 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce802ae90 con 0x7f8d00132360 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0006d890 con 0x7f8d00132360 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8031c70 con 0x7f8d00132360 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:26.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf67fc640 10 monclient: dump: 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8cf7fff640 4 set_mon_vals no callback set 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 10 monclient: _renew_subs 2024-09-15T02:09:26.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.444+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d00133780 con 0x7f8d00132360 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8d04cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8d04cc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8d04cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d000724a0 con 0x7f8d00132360 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8ce8039040 con 0x7f8d00132360 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8cf67fc640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 0x7f8ce003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.445+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8ce806c460 con 0x7f8d00132360 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.446+0000 7f8cfe59c640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 0x7f8ce003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:26.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.446+0000 7f8cfe59c640 10 monclient: get_auth_request con 0x7f8ce0038740 auth_method 0 2024-09-15T02:09:26.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.448+0000 7f8cfe59c640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 0x7f8ce003abf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0005200 tx=0x7f8cf0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:26.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.450+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8ce803bca0 con 0x7f8d00132360 2024-09-15T02:09:26.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.450+0000 7f8cf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:26.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.450+0000 7f8cf67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8d04cc0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8d04cc0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8d04cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d00134250 con 0x7f8d00132360 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8cf67fc640 1 -- 172.21.15.17:0/3178297640 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f8ce8037020 con 0x7f8d00132360 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8cf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.678+0000 7f8cf67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:26.681 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":176,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 msgr2=0x7f8ce003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 0x7f8ce003abf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0005200 tx=0x7f8cf0009040 comp rx=0 tx=0).stop 2024-09-15T02:09:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 10 monclient: shutdown 2024-09-15T02:09:26.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 msgr2=0x7f8d0006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8031000 tx=0x7f8ce8032000 comp rx=0 tx=0).stop 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 shutdown_connections 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8ce0038740 0x7f8ce003abf0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d00136c40 0x7f8d0006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8d00132360 0x7f8d0006c2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 --2- 172.21.15.17:0/3178297640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8d0006e140 0x7f8d0006bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 >> 172.21.15.17:0/3178297640 conn(0x7f8d00067b10 msgr2=0x7f8d0012c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 shutdown_connections 2024-09-15T02:09:26.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:26.680+0000 7f8d04cc0640 1 -- 172.21.15.17:0/3178297640 wait complete. 2024-09-15T02:09:29.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:29.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.800+0000 7f4feb99d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.800+0000 7f4feb99d640 10 monclient: build_initial_monmap 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.800+0000 7f4feb99d640 10 monclient: monmap: 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:29.801321+0000 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:29.801321+0000 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:29.802 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:29.803 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:29.803 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:29.803 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 Processor -- start 2024-09-15T02:09:29.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 -- start start 2024-09-15T02:09:29.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient: init 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe409c0d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fe409c4a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fe409c870 0x7f4fe409e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe409c870 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40699c0 con 0x7f4fe409c4a0 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe406a5e0 con 0x7f4fe409c870 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe406a290 con 0x7f4fe409c0d0 2024-09-15T02:09:29.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.801+0000 7f4feb99d640 10 monclient(hunting): authenticate will time out at 1829.797886s 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:46246/0 (socket says 172.21.15.17:46246) 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 -- 172.21.15.17:0/3248884437 learned_addr learned my addr 172.21.15.17:0/3248884437 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): get_auth_request con 0x7f4fe409c0d0 auth_method 0 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 1 --2- 172.21.15.17:0/3248884437 >> v2:172.21.15.67:3300/0 conn(0x7f4fe409c870 0x7f4fe409e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): get_auth_request con 0x7f4fe409c870 auth_method 0 2024-09-15T02:09:29.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient(hunting): handle_auth_done global_id 214178 payload 274 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 -- 172.21.15.17:0/3248884437 >> v2:172.21.15.67:3300/0 conn(0x7f4fe409c870 msgr2=0x7f4fe409e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 --2- 172.21.15.17:0/3248884437 >> v2:172.21.15.67:3300/0 conn(0x7f4fe409c870 0x7f4fe409e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 -- 172.21.15.17:0/3248884437 >> v1:172.21.15.67:6789/0 conn(0x7f4fe409c4a0 legacy=0x7f4fe406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient: _finish_hunting 0 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 monclient: found mon.noname-b 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:29.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:29.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.802+0000 7f4fe9739640 1 -- 172.21.15.17:0/3248884437 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe409f6e0 con 0x7f4fe409c0d0 2024-09-15T02:09:29.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe9739640 1 -- 172.21.15.17:0/3248884437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 msgr2=0x7f4fe409a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:29.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe9739640 1 --2- 172.21.15.17:0/3248884437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009f20 tx=0x7f4fd4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8f3f2464b89e08 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:29.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 1 -- 172.21.15.17:0/3248884437 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4007610 con 0x7f4fe409c0d0 2024-09-15T02:09:29.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:29.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:29.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: dump: 2024-09-15T02:09:29.819 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:29.820 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:29.820 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.820 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.820 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:29.820 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: _finish_auth 0 2024-09-15T02:09:29.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:29.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 1 -- 172.21.15.17:0/3248884437 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd402d050 con 0x7f4fe409c0d0 2024-09-15T02:09:29.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 5 monclient: authenticate success, global_id 214178 2024-09-15T02:09:29.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:29.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:29.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 1 -- 172.21.15.17:0/3248884437 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40079a0 con 0x7f4fe409c0d0 2024-09-15T02:09:29.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:29.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 4 set_mon_vals no callback set 2024-09-15T02:09:29.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:29.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:29.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4fe3fff640 10 monclient: dump: 2024-09-15T02:09:29.824 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:29.824 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:29.824 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.824 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.824 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:29.825 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.825 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:29.825 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:29.825 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:29.825 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 10 monclient: shutdown 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 1 -- 172.21.15.17:0/3248884437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 msgr2=0x7f4fe409a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.805+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3248884437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009f20 tx=0x7f4fd4009840 comp rx=0 tx=0).stop 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 -- 172.21.15.17:0/3248884437 shutdown_connections 2024-09-15T02:09:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3248884437 >> v2:172.21.15.67:3300/0 conn(0x7f4fe409c870 0x7f4fe409e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:29.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3248884437 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c0d0 0x7f4fe409a920 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:29.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 -- 172.21.15.17:0/3248884437 >> 172.21.15.17:0/3248884437 conn(0x7f4fe40712f0 msgr2=0x7f4fe40a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:29.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 -- 172.21.15.17:0/3248884437 shutdown_connections 2024-09-15T02:09:29.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 -- 172.21.15.17:0/3248884437 wait complete. 2024-09-15T02:09:29.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 10 monclient: build_initial_monmap 2024-09-15T02:09:29.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 10 monclient: monmap: 2024-09-15T02:09:29.828 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:29.828 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:29.828 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:29.828 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:29.829 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.806+0000 7f4feb99d640 1 Processor -- start 2024-09-15T02:09:29.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 -- start start 2024-09-15T02:09:29.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient: init 2024-09-15T02:09:29.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:29.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:09:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe409c0d0 0x7f4fe4131ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe409c0d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe409c4a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe409c870 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe409cd80 con 0x7f4fe409c4a0 2024-09-15T02:09:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40a2a20 con 0x7f4fe409c0d0 2024-09-15T02:09:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4141fd0 con 0x7f4fe409c870 2024-09-15T02:09:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4feb99d640 10 monclient(hunting): authenticate will time out at 1829.803756s 2024-09-15T02:09:29.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:46252/0 (socket says 172.21.15.17:46252) 2024-09-15T02:09:29.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 1 -- 172.21.15.17:0/3182904887 learned_addr learned my addr 172.21.15.17:0/3182904887 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:29.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): get_auth_request con 0x7f4fe409c870 auth_method 0 2024-09-15T02:09:29.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:29.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:29.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:29.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe8f38640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe8f38640 10 monclient(hunting): get_auth_request con 0x7f4fe409c4a0 auth_method 0 2024-09-15T02:09:29.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe8f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:29.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe8f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:29.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe8f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:29.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9739640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe409c0d0 0x7f4fe4131ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:29.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.807+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:29.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:29.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): get_auth_request con 0x7f4fe409c0d0 auth_method 0 2024-09-15T02:09:29.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:29.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:29.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:29.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe8f38640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 msgr2=0x7f4fe406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:09:29.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe8f38640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 msgr2=0x7f4fe406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:09:29.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe8f38640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:29.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe8f38640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:29.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:29.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:29.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:29.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 10 monclient(hunting): handle_auth_done global_id 214181 payload 824 2024-09-15T02:09:29.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe409c0d0 msgr2=0x7f4fe4131ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:29.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe409c0d0 0x7f4fe4131ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:29.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 msgr2=0x7f4fe406ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:29.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:29.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 10 monclient: _finish_hunting 0 2024-09-15T02:09:29.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 monclient: found mon.noname-c 2024-09-15T02:09:29.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:29.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.808+0000 7f4fe9f3a640 1 -- 172.21.15.17:0/3182904887 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4009ae0 con 0x7f4fe409c870 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe9f3a640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 msgr2=0x7f4fe406cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe9f3a640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00ea50 tx=0x7f4fdc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00cb50 con 0x7f4fe409c870 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: dump: 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.841 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: _finish_auth 0 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00ccb0 con 0x7f4fe409c870 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc013470 con 0x7f4fe409c870 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 5 monclient: authenticate success, global_id 214181 2024-09-15T02:09:29.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 10 monclient: _renew_subs 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe411ee00 con 0x7f4fe409c870 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe37fe640 4 set_mon_vals no callback set 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4fe17fa640 10 monclient: dump: 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:29.843 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 10 monclient: _renew_subs 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.810+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe411df10 con 0x7f4fe409c870 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4fdc013880 con 0x7f4fe409c870 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4feb99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4feb99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4feb99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4fe17fa640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 0x7f4fd003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe411f280 con 0x7f4fe409c870 2024-09-15T02:09:29.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.811+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4fdc04c940 con 0x7f4fe409c870 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.812+0000 7f4fe9739640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 0x7f4fd003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.812+0000 7f4fe9739640 10 monclient: get_auth_request con 0x7f4fd0038770 auth_method 0 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.814+0000 7f4fe9739640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 0x7f4fd003ac20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40a2eb0 tx=0x7f4fd4036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.816+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4fdc023ad0 con 0x7f4fe409c870 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.816+0000 7f4fe17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:29.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:29.816+0000 7f4fe17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4feb99d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4feb99d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4feb99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4fe40724a0 con 0x7f4fe409c870 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4fe17fa640 1 -- 172.21.15.17:0/3182904887 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f4fdc0232f0 con 0x7f4fe409c870 2024-09-15T02:09:30.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4fe17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:30.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.043+0000 7f4fe17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:30.046 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":176,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 msgr2=0x7f4fd003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 0x7f4fd003ac20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40a2eb0 tx=0x7f4fd4036000 comp rx=0 tx=0).stop 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 10 monclient: shutdown 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 msgr2=0x7f4fe406cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00ea50 tx=0x7f4fdc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 shutdown_connections 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4fd0038770 0x7f4fd003ac20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fe409c4a0 0x7f4fe406ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4fe409c870 0x7f4fe406cf40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 --2- 172.21.15.17:0/3182904887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe409c0d0 0x7f4fe4131ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 >> 172.21.15.17:0/3182904887 conn(0x7f4fe40712f0 msgr2=0x7f4fe40a0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 shutdown_connections 2024-09-15T02:09:30.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:30.045+0000 7f4feb99d640 1 -- 172.21.15.17:0/3182904887 wait complete. 2024-09-15T02:09:33.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:33.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 10 monclient: get_monmap_and_config 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 10 monclient: build_initial_monmap 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 10 monclient: monmap: 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:33.166902+0000 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:33.166902+0000 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 1 Processor -- start 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 1 -- start start 2024-09-15T02:09:33.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.166+0000 7fe428ac9640 10 monclient: init 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 0x7fe42409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fe424131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4240a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fe42409f1e0 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240a0040 con 0x7fe4240a0280 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42409bff0 con 0x7fe42409f1e0 2024-09-15T02:09:33.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240a14a0 con 0x7fe424131b20 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe428ac9640 10 monclient(hunting): authenticate will time out at 1833.163476s 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 0x7fe42409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe421d9b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe4240a0280 0x7fe424131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:32998/0 (socket says 172.21.15.17:32998) 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe421d9b640 1 -- 172.21.15.17:0/3460623108 learned_addr learned my addr 172.21.15.17:0/3460623108 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 10 monclient(hunting): get_auth_request con 0x7fe42409f1e0 auth_method 0 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): get_auth_request con 0x7fe424131b20 auth_method 0 2024-09-15T02:09:33.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 1 -- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 msgr2=0x7fe424138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 1 -- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 msgr2=0x7fe424138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 1 --2- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe422d9d640 1 --2- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient(hunting): handle_auth_done global_id 214184 payload 274 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 -- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 msgr2=0x7fe424138e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 --2- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 -- 172.21.15.17:0/3460623108 >> v1:172.21.15.17:6789/0 conn(0x7fe4240a0280 legacy=0x7fe424131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient: _finish_hunting 0 2024-09-15T02:09:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 monclient: found mon.noname-b 2024-09-15T02:09:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.167+0000 7fe42259c640 1 -- 172.21.15.17:0/3460623108 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe424132600 con 0x7fe424131b20 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.170+0000 7fe42259c640 1 -- 172.21.15.17:0/3460623108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 msgr2=0x7fe42409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.170+0000 7fe42259c640 1 --2- 172.21.15.17:0/3460623108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 0x7fe42409eca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe414009850 tx=0x7fe4140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee3d2b2eaacb0d73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 1 -- 172.21.15.17:0/3460623108 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41402adf0 con 0x7fe424131b20 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: dump: 2024-09-15T02:09:33.179 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: _finish_auth 0 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 5 monclient: authenticate success, global_id 214184 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 1 -- 172.21.15.17:0/3460623108 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe414032440 con 0x7fe424131b20 2024-09-15T02:09:33.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 1 -- 172.21.15.17:0/3460623108 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4140327e0 con 0x7fe424131b20 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 4 set_mon_vals no callback set 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe42159a640 10 monclient: dump: 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:33.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 10 monclient: shutdown 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 -- 172.21.15.17:0/3460623108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 msgr2=0x7fe42409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3460623108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 0x7fe42409eca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe414009850 tx=0x7fe4140090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 -- 172.21.15.17:0/3460623108 shutdown_connections 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3460623108 >> v2:172.21.15.17:3300/0 conn(0x7fe42409f1e0 0x7fe424138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3460623108 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe424131b20 0x7fe42409eca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 -- 172.21.15.17:0/3460623108 >> 172.21.15.17:0/3460623108 conn(0x7fe424071270 msgr2=0x7fe4240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 -- 172.21.15.17:0/3460623108 shutdown_connections 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.171+0000 7fe428ac9640 1 -- 172.21.15.17:0/3460623108 wait complete. 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.172+0000 7fe428ac9640 10 monclient: build_initial_monmap 2024-09-15T02:09:33.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.172+0000 7fe428ac9640 10 monclient: monmap: 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.172+0000 7fe428ac9640 1 Processor -- start 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.172+0000 7fe428ac9640 1 -- start start 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.172+0000 7fe428ac9640 10 monclient: init 2024-09-15T02:09:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fe42409f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fe4240a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 0x7fe424132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fe424131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240a09b0 con 0x7fe4240a0280 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe424131440 con 0x7fe424131b20 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe424135c50 con 0x7fe42409f1e0 2024-09-15T02:09:33.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe428ac9640 10 monclient(hunting): authenticate will time out at 1833.169231s 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57700/0 (socket says 172.21.15.17:57700) 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 -- 172.21.15.17:0/3207258205 learned_addr learned my addr 172.21.15.17:0/3207258205 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): get_auth_request con 0x7fe42409f1e0 auth_method 0 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 10 monclient(hunting): get_auth_request con 0x7fe4240a0280 auth_method 0 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:33.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 msgr2=0x7fe424132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 msgr2=0x7fe424132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 0x7fe424132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe421d9b640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 10 monclient(hunting): get_auth_request con 0x7fe424131b20 auth_method 0 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:33.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient(hunting): handle_auth_done global_id 214187 payload 824 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 msgr2=0x7fe424132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 0x7fe424132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 msgr2=0x7fe424132970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient: _finish_hunting 0 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 monclient: found mon.noname-c 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe42259c640 1 -- 172.21.15.17:0/3207258205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414009390 con 0x7fe42409f1e0 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.173+0000 7fe422d9d640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 0x7fe424132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.175+0000 7fe42259c640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 msgr2=0x7fe42409b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.175+0000 7fe42259c640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe414009850 tx=0x7fe414032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414031b10 con 0x7fe42409f1e0 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: dump: 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:33.188 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe414031c70 con 0x7fe42409f1e0 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 5 monclient: authenticate success, global_id 214187 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 10 monclient: _renew_subs 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe424133db0 con 0x7fe42409f1e0 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41403c400 con 0x7fe42409f1e0 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:33.189 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 10 monclient: dump: 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe420d99640 4 set_mon_vals no callback set 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 10 monclient: _renew_subs 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4241353c0 con 0x7fe42409f1e0 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.176+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe41403c830 con 0x7fe42409f1e0 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe4127fc640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 0x7fe40403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe428ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:33.191 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe428ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe428ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe41406cfc0 con 0x7fe42409f1e0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4241359f0 con 0x7fe42409f1e0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe421d9b640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 0x7fe40403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.177+0000 7fe421d9b640 10 monclient: get_auth_request con 0x7fe404038790 auth_method 0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.180+0000 7fe421d9b640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 0x7fe40403ac40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe424069e60 tx=0x7fe418009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:33.192 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.181+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe414037020 con 0x7fe42409f1e0 2024-09-15T02:09:33.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.181+0000 7fe4127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:33.193 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.181+0000 7fe4127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:33.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe428ac9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe428ac9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe428ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe4240724a0 con 0x7fe42409f1e0 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe4127fc640 1 -- 172.21.15.17:0/3207258205 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7fe4140428e0 con 0x7fe42409f1e0 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe4127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.411+0000 7fe4127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:33.413 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":176,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 msgr2=0x7fe40403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 0x7fe40403ac40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe424069e60 tx=0x7fe418009040 comp rx=0 tx=0).stop 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 10 monclient: shutdown 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 msgr2=0x7fe42409b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe414009850 tx=0x7fe414032000 comp rx=0 tx=0).stop 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 shutdown_connections 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe404038790 0x7fe40403ac40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4240a0280 0x7fe424132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424131b20 0x7fe424132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 --2- 172.21.15.17:0/3207258205 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe42409f1e0 0x7fe42409b4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 >> 172.21.15.17:0/3207258205 conn(0x7fe424071270 msgr2=0x7fe42409d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 shutdown_connections 2024-09-15T02:09:33.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:33.413+0000 7fe428ac9640 1 -- 172.21.15.17:0/3207258205 wait complete. 2024-09-15T02:09:36.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:36.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.533+0000 7f389179d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:36.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.533+0000 7f389179d640 10 monclient: build_initial_monmap 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.533+0000 7f389179d640 10 monclient: monmap: 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:36.534094+0000 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:36.534094+0000 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:36.535 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 Processor -- start 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 -- start start 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient: init 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f388c131b20 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f388c0a0280 0x7f388c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): picked mon.noname-c con 0x7f388c0a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): picked mon.noname-a con 0x7f388c132430 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0a0040 con 0x7f388c131b20 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c09bff0 con 0x7f388c132430 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c0a14a0 con 0x7f388c0a0280 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f389179d640 10 monclient(hunting): authenticate will time out at 1836.530612s 2024-09-15T02:09:36.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42358/0 (socket says 172.21.15.17:42358) 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 learned_addr learned my addr 172.21.15.17:0/700417129 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): get_auth_request con 0x7f388c132430 auth_method 0 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.534+0000 7f3883fff640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.67:3300/0 conn(0x7f388c0a0280 0x7f388c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f3883fff640 10 monclient(hunting): get_auth_request con 0x7f388c0a0280 auth_method 0 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f3883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f3883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f3883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 10 monclient(hunting): handle_auth_done global_id 224097 payload 274 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 >> v2:172.21.15.67:3300/0 conn(0x7f388c0a0280 msgr2=0x7f388c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.67:3300/0 conn(0x7f388c0a0280 0x7f388c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 >> v1:172.21.15.67:6789/0 conn(0x7f388c131b20 legacy=0x7f388c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 monclient: found mon.noname-a 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c09f3b0 con 0x7f388c132430 2024-09-15T02:09:36.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f3883fff640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.67:3300/0 conn(0x7f388c0a0280 0x7f388c131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 msgr2=0x7f388c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:09:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 msgr2=0x7f388c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f388c0a5010 tx=0x7f3884009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:36.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.535+0000 7f388b7fe640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388b7fe640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388b7fe640 10 monclient: get_auth_request con 0x7f388c132430 auth_method 0 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388b7fe640 0 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388b7fe640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient: ms_handle_reset current mon v2:172.21.15.17:3300/0 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 msgr2=0x7f388c138e60 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 1 --2- 172.21.15.17:0/700417129 >> v2:172.21.15.17:3300/0 conn(0x7f388c132430 0x7f388c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient: _add_conns ranks=[2,0] 2024-09-15T02:09:36.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 1 --2- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 0x7f3878005900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f38780034c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3878006270 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387800a210 con 0x7f3878006270 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387800b410 con 0x7f38780034c0 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388a7fc640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.17:3300/0 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 1 --2- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 0x7f3878005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): get_auth_request con 0x7f38780034c0 auth_method 0 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:36.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient(hunting): handle_auth_done global_id 224097 payload 293 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 1 -- 172.21.15.17:0/700417129 >> v1:172.21.15.17:6789/0 conn(0x7f3878006270 legacy=0x7f3878008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient: _finish_hunting 0 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 1 monclient: found mon.noname-b 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:36.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.736+0000 7f388affd640 1 -- 172.21.15.17:0/700417129 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387800b690 con 0x7f38780034c0 2024-09-15T02:09:36.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388affd640 1 -- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 msgr2=0x7f3878005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388affd640 1 --2- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 0x7f3878005900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f387c0026c0 tx=0x7f387c007ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ba6e15ba397aa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c00c8a0 con 0x7f38780034c0 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: dump: 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.745 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c00ca00 con 0x7f38780034c0 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f389179d640 5 monclient: authenticate success, global_id 224097 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f389179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 1 -- 172.21.15.17:0/700417129 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c00cdc0 con 0x7f38780034c0 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f388a7fc640 10 monclient: dump: 2024-09-15T02:09:36.746 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f389179d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.739+0000 7f389179d640 4 set_mon_vals no callback set 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 10 monclient: shutdown 2024-09-15T02:09:36.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 -- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 msgr2=0x7f3878005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 --2- 172.21.15.17:0/700417129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f38780034c0 0x7f3878005900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f387c0026c0 tx=0x7f387c007ca0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f388b7fe640 1 -- 172.21.15.17:0/700417129 reap_dead start 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 -- 172.21.15.17:0/700417129 shutdown_connections 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 -- 172.21.15.17:0/700417129 >> 172.21.15.17:0/700417129 conn(0x7f388c071270 msgr2=0x7f388c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 -- 172.21.15.17:0/700417129 shutdown_connections 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 1 -- 172.21.15.17:0/700417129 wait complete. 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 10 monclient: build_initial_monmap 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.740+0000 7f389179d640 10 monclient: monmap: 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:36.748 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 Processor -- start 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 -- start start 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient: init 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): picked mon.noname-a con 0x7f38780034c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:36.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f388c12d1b0 0x7f388c12d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): picked mon.noname-b con 0x7f388c12d1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f388c12dac0 0x7f388c1476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): picked mon.noname-c con 0x7f388c12dac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c097e30 con 0x7f38780034c0 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c138f60 con 0x7f388c12d1b0 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0981c0 con 0x7f388c12dac0 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f389179d640 10 monclient(hunting): authenticate will time out at 1836.737606s 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42376/0 (socket says 172.21.15.17:42376) 2024-09-15T02:09:36.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 1 -- 172.21.15.17:0/1692127954 learned_addr learned my addr 172.21.15.17:0/1692127954 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): get_auth_request con 0x7f38780034c0 auth_method 0 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f388c12dac0 0x7f388c1476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): get_auth_request con 0x7f388c12dac0 auth_method 0 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:36.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f3883fff640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f388c12d1b0 0x7f388c12d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f3883fff640 10 monclient(hunting): get_auth_request con 0x7f388c12d1b0 auth_method 0 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f3883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f3883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f3883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.741+0000 7f388affd640 10 monclient(hunting): handle_auth_done global_id 224100 payload 824 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 -- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f388c12dac0 msgr2=0x7f388c1476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f388c12dac0 0x7f388c1476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 -- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f388c12d1b0 msgr2=0x7f388c12d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f388c12d1b0 0x7f388c12d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 10 monclient: _finish_hunting 0 2024-09-15T02:09:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 monclient: found mon.noname-a 2024-09-15T02:09:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.742+0000 7f388affd640 1 -- 172.21.15.17:0/1692127954 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0a09b0 con 0x7f38780034c0 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f388affd640 1 -- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 msgr2=0x7f388c09a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f388affd640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f387c007560 tx=0x7f387c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c007790 con 0x7f38780034c0 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: dump: 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.755 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: _finish_auth 0 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 5 monclient: authenticate success, global_id 224100 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c006bb0 con 0x7f38780034c0 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c01e450 con 0x7f38780034c0 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 10 monclient: _renew_subs 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c097500 con 0x7f38780034c0 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3889ffb640 4 set_mon_vals no callback set 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f3882ffd640 10 monclient: dump: 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 10 monclient: _renew_subs 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.744+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c12df10 con 0x7f38780034c0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f387c01e8c0 con 0x7f38780034c0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f3882ffd640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 0x7f386c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f389179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f387c04e680 con 0x7f38780034c0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f389179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f389179d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.745+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c0724a0 con 0x7f38780034c0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.746+0000 7f3883fff640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 0x7f386c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.746+0000 7f3883fff640 10 monclient: get_auth_request con 0x7f386c038740 auth_method 0 2024-09-15T02:09:36.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.748+0000 7f3883fff640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 0x7f386c03abf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f388c12ea30 tx=0x7f3874008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:36.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.750+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f387c016080 con 0x7f38780034c0 2024-09-15T02:09:36.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.750+0000 7f3882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:36.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.750+0000 7f3882ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:36.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f389179d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f389179d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f389179d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f388c06efc0 con 0x7f38780034c0 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f3882ffd640 1 -- 172.21.15.17:0/1692127954 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f387c034610 con 0x7f38780034c0 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f3882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.977+0000 7f3882ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:36.979 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 msgr2=0x7f386c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 0x7f386c03abf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f388c12ea30 tx=0x7f3874008040 comp rx=0 tx=0).stop 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 10 monclient: shutdown 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 msgr2=0x7f388c09a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f387c007560 tx=0x7f387c015000 comp rx=0 tx=0).stop 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 shutdown_connections 2024-09-15T02:09:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f386c038740 0x7f386c03abf0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f388c12dac0 0x7f388c1476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f388c12d1b0 0x7f388c12d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 --2- 172.21.15.17:0/1692127954 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f38780034c0 0x7f388c09a690 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 >> 172.21.15.17:0/1692127954 conn(0x7f388c071270 msgr2=0x7f388c130390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 shutdown_connections 2024-09-15T02:09:36.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:36.979+0000 7f389179d640 1 -- 172.21.15.17:0/1692127954 wait complete. 2024-09-15T02:09:36.991 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:09:36.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:09:36.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.087+0000 7f9e22f9d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.087+0000 7f9e22f9d640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.087+0000 7f9e22f9d640 10 monclient: monmap: 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:37.088463+0000 2024-09-15T02:09:37.089 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:37.088463+0000 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:37.090 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 Processor -- start 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 -- start start 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient: init 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e1c06eb30 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:37.092 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9e1c09e970 0x7f9e1c09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1c09e970 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e1c09c4d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c09f6a0 con 0x7f9e1c06eb30 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c069f40 con 0x7f9e1c09c4d0 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c09ed40 con 0x7f9e1c09e970 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e22f9d640 10 monclient(hunting): authenticate will time out at 1837.084899s 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.088+0000 7f9e1bfff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9e1c09e970 0x7f9e1c09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e20d39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9e1c06eb30 0x7f9e1c06ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33014/0 (socket says 172.21.15.17:33014) 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request con 0x7f9e1c09e970 auth_method 0 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e20d39640 1 -- 172.21.15.17:0/1337298675 learned_addr learned my addr 172.21.15.17:0/1337298675 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:37.093 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_done global_id 224103 payload 274 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 1 -- 172.21.15.17:0/1337298675 >> v1:172.21.15.67:6789/0 conn(0x7f9e1c09c4d0 legacy=0x7f9e1c139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 1 -- 172.21.15.17:0/1337298675 >> v1:172.21.15.17:6789/0 conn(0x7f9e1c06eb30 legacy=0x7f9e1c06ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 1 monclient: found mon.noname-a 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.094 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.089+0000 7f9e1bfff640 1 -- 172.21.15.17:0/1337298675 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c069950 con 0x7f9e1c09e970 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.092+0000 7f9e1bfff640 1 -- 172.21.15.17:0/1337298675 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c09e970 msgr2=0x7f9e1c09bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.092+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/1337298675 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c09e970 0x7f9e1c09bf90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009850 tx=0x7f9e080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e863ebbace8653eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 1 -- 172.21.15.17:0/1337298675 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08032440 con 0x7f9e1c09e970 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: dump: 2024-09-15T02:09:37.098 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 1 -- 172.21.15.17:0/1337298675 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e080325a0 con 0x7f9e1c09e970 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 1 -- 172.21.15.17:0/1337298675 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08032940 con 0x7f9e1c09e970 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 5 monclient: authenticate success, global_id 224103 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 4 set_mon_vals no callback set 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e1b7fe640 10 monclient: dump: 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 10 monclient: shutdown 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 1 -- 172.21.15.17:0/1337298675 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c09e970 msgr2=0x7f9e1c09bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.093+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/1337298675 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c09e970 0x7f9e1c09bf90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009850 tx=0x7f9e080090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 -- 172.21.15.17:0/1337298675 shutdown_connections 2024-09-15T02:09:37.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/1337298675 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c09e970 0x7f9e1c09bf90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 -- 172.21.15.17:0/1337298675 >> 172.21.15.17:0/1337298675 conn(0x7f9e1c0712f0 msgr2=0x7f9e1c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 -- 172.21.15.17:0/1337298675 shutdown_connections 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 -- 172.21.15.17:0/1337298675 wait complete. 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 10 monclient: monmap: 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:37.102 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.094+0000 7f9e22f9d640 1 Processor -- start 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 -- start start 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient: init 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 0x7f9e1c141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1c06eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e1c09c4d0 0x7f9e1c134060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1c09c4d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e1c09e970 0x7f9e1c11ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e1c09e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c120740 con 0x7f9e1c06eb30 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c142080 con 0x7f9e1c09c4d0 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c122190 con 0x7f9e1c09e970 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e22f9d640 10 monclient(hunting): authenticate will time out at 1837.091500s 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 0x7f9e1c141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e1c09e970 0x7f9e1c11ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 0x7f9e1c141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42396/0 (socket says 172.21.15.17:42396) 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 -- 172.21.15.17:0/3994934181 learned_addr learned my addr 172.21.15.17:0/3994934181 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): get_auth_request con 0x7f9e1c09e970 auth_method 0 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): get_auth_request con 0x7f9e1c06eb30 auth_method 0 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e1c09c4d0 0x7f9e1c134060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request con 0x7f9e1c09c4d0 auth_method 0 2024-09-15T02:09:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient(hunting): handle_auth_done global_id 224106 payload 824 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e1c09e970 msgr2=0x7f9e1c11ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e1c09e970 0x7f9e1c11ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e1c09c4d0 msgr2=0x7f9e1c134060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e1c09c4d0 0x7f9e1c134060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 monclient: found mon.noname-a 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e20d39640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08009390 con 0x7f9e1c06eb30 2024-09-15T02:09:37.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e1c09c4d0 0x7f9e1c134060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:37.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.095+0000 7f9e2153a640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9e1c09e970 0x7f9e1c11ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e20d39640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 msgr2=0x7f9e1c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e20d39640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 0x7f9e1c141220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000b510 tx=0x7f9e100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10011840 con 0x7f9e1c06eb30 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: dump: 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.113 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e100119c0 con 0x7f9e1c06eb30 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 5 monclient: authenticate success, global_id 224106 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 10 monclient: _renew_subs 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c122470 con 0x7f9e1c06eb30 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10011d80 con 0x7f9e1c06eb30 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e18ff9640 10 monclient: dump: 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 10 monclient: _renew_subs 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e1affd640 4 set_mon_vals no callback set 2024-09-15T02:09:37.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.098+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c1212a0 con 0x7f9e1c06eb30 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e22f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9e10010590 con 0x7f9e1c06eb30 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 0x7f9e1c11f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c063b90 con 0x7f9e1c141760 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 0x7f9e1c11f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient: get_auth_request con 0x7f9e1c141760 auth_method 0 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e18ff9640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e00038740 0x7f9e0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9e1004c660 con 0x7f9e1c06eb30 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_done global_id 224109 payload 824 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e1c09cda0 con 0x7f9e1c141760 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e2153a640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e00038740 0x7f9e0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.099+0000 7f9e2153a640 10 monclient: get_auth_request con 0x7f9e00038740 auth_method 0 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 msgr2=0x7f9e1c11f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 0x7f9e1c11f010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0a3e30 tx=0x7f9e08032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080318c0 con 0x7f9e1c141760 2024-09-15T02:09:37.118 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.101+0000 7f9e18ff9640 10 monclient: dump: 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9e08031a20 con 0x7f9e1c141760 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 msgr2=0x7f9e1c11f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 0x7f9e1c11f010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0a3e30 tx=0x7f9e08032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 msgr2=0x7f9e1c11f010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e18ff9640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c141760 0x7f9e1c11f010 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0a3e30 tx=0x7f9e08032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.102+0000 7f9e2153a640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e00038740 0x7f9e0003abf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9e14007fd0 tx=0x7f9e140025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.107+0000 7f9e22f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.107+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 0x7f9e1c11f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.107+0000 7f9e22f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.107+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c122b80 con 0x7f9e1c1345a0 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 0x7f9e1c11f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient: get_auth_request con 0x7f9e1c1345a0 auth_method 0 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 10 monclient(hunting): handle_auth_done global_id 224112 payload 824 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.108+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3994934181 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e1c0724a0 con 0x7f9e1c1345a0 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e1bfff640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 msgr2=0x7f9e1c11f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e1bfff640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 0x7f9e1c11f550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009850 tx=0x7f9e0803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0804a050 con 0x7f9e1c1345a0 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 10 monclient: dump: 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.122 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7f9e0803e400 con 0x7f9e1c1345a0 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 msgr2=0x7f9e1c11f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 0x7f9e1c11f550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009850 tx=0x7f9e0803f000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 msgr2=0x7f9e1c11f550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.110+0000 7f9e18ff9640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c1345a0 0x7f9e1c11f550 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009850 tx=0x7f9e0803f000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:37.124 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.125 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.126 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:37.127 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.128 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.130 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.132 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.135 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:37.136 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:37.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e00038740 msgr2=0x7f9e0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9e00038740 0x7f9e0003abf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9e14007fd0 tx=0x7f9e140025b0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 10 monclient: shutdown 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 msgr2=0x7f9e1c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 --2- 172.21.15.17:0/3994934181 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9e1c06eb30 0x7f9e1c141220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000b510 tx=0x7f9e100090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e2153a640 1 -- 172.21.15.17:0/3994934181 reap_dead start 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e2153a640 1 -- 172.21.15.17:0/3994934181 reap_dead start 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 shutdown_connections 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 >> 172.21.15.17:0/3994934181 conn(0x7f9e1c0712f0 msgr2=0x7f9e1c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 shutdown_connections 2024-09-15T02:09:37.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.111+0000 7f9e22f9d640 1 -- 172.21.15.17:0/3994934181 wait complete. 2024-09-15T02:09:37.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:37.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:09:37.139 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 10 monclient: get_monmap_and_config 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 10 monclient: monmap: 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:37.235928+0000 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:37.235928+0000 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.237 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 1 Processor -- start 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 1 -- start start 2024-09-15T02:09:37.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.235+0000 7f4e90057640 10 monclient: init 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 0x7f4e8809ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8809fad0 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e8809f0f0 0x7f4e8809fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8809f0f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:37.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e880a03e0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8809c7c0 con 0x7f4e880a03e0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8809f4f0 con 0x7f4e8809f0f0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8809c9e0 con 0x7f4e8809fad0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e90057640 10 monclient(hunting): authenticate will time out at 1837.232363s 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e8809f0f0 0x7f4e8809fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4e8809f0f0 0x7f4e8809fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42426/0 (socket says 172.21.15.17:42426) 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/3420531795 learned_addr learned my addr 172.21.15.17:0/3420531795 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): get_auth_request con 0x7f4e8809f0f0 auth_method 0 2024-09-15T02:09:37.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/3420531795 >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 0x7f4e8809ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_done global_id 224115 payload 274 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/3420531795 >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 msgr2=0x7f4e8809ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 --2- 172.21.15.17:0/3420531795 >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 0x7f4e8809ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/3420531795 >> v1:172.21.15.67:6789/0 conn(0x7f4e880a03e0 legacy=0x7f4e88136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 monclient: found mon.noname-a 2024-09-15T02:09:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/3420531795 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8809dbc0 con 0x7f4e8809f0f0 2024-09-15T02:09:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8ddf3640 10 monclient: get_auth_request con 0x7f4e8809fad0 auth_method 0 2024-09-15T02:09:37.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.236+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/3420531795 >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 0x7f4e8809ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/3420531795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809f0f0 msgr2=0x7f4e8809fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8d5f2640 1 --2- 172.21.15.17:0/3420531795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809f0f0 0x7f4e8809fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809c750 tx=0x7f4e780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48554e0d0f9a878d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 1 -- 172.21.15.17:0/3420531795 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78032440 con 0x7f4e8809f0f0 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: dump: 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.249 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 1 -- 172.21.15.17:0/3420531795 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e780325a0 con 0x7f4e8809f0f0 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e90057640 5 monclient: authenticate success, global_id 224115 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e90057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 1 -- 172.21.15.17:0/3420531795 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78032980 con 0x7f4e8809f0f0 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e8cdf1640 10 monclient: dump: 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e90057640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.239+0000 7f4e90057640 4 set_mon_vals no callback set 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 10 monclient: shutdown 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 -- 172.21.15.17:0/3420531795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809f0f0 msgr2=0x7f4e8809fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 --2- 172.21.15.17:0/3420531795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809f0f0 0x7f4e8809fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809c750 tx=0x7f4e780091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 -- 172.21.15.17:0/3420531795 shutdown_connections 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 --2- 172.21.15.17:0/3420531795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809f0f0 0x7f4e8809fea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 --2- 172.21.15.17:0/3420531795 >> v2:172.21.15.67:3300/0 conn(0x7f4e8809fad0 0x7f4e8809ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 -- 172.21.15.17:0/3420531795 >> 172.21.15.17:0/3420531795 conn(0x7f4e880a50e0 msgr2=0x7f4e880a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 -- 172.21.15.17:0/3420531795 shutdown_connections 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.240+0000 7f4e90057640 1 -- 172.21.15.17:0/3420531795 wait complete. 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient: monmap: 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 Processor -- start 2024-09-15T02:09:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 -- start start 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient: init 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8809f0f0 0x7f4e8806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8809f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 0x7f4e8806dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8809fad0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e880a03e0 0x7f4e8806e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): picked mon.noname-c con 0x7f4e880a03e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8809bd20 con 0x7f4e8809fad0 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e88096a40 con 0x7f4e8809f0f0 2024-09-15T02:09:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e88125f60 con 0x7f4e880a03e0 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.241+0000 7f4e90057640 10 monclient(hunting): authenticate will time out at 1837.237910s 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 0x7f4e8806dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e880a03e0 0x7f4e8806e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 0x7f4e8806dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42438/0 (socket says 172.21.15.17:42438) 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/4186831178 learned_addr learned my addr 172.21.15.17:0/4186831178 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): get_auth_request con 0x7f4e8809fad0 auth_method 0 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): get_auth_request con 0x7f4e880a03e0 auth_method 0 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8809f0f0 0x7f4e8806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8ddf3640 10 monclient(hunting): get_auth_request con 0x7f4e8809f0f0 auth_method 0 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8ddf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient(hunting): handle_auth_done global_id 224118 payload 824 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e880a03e0 msgr2=0x7f4e8806e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e880a03e0 0x7f4e8806e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8809f0f0 msgr2=0x7f4e8806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8809f0f0 0x7f4e8806d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 monclient: found mon.noname-a 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78009450 con 0x7f4e8809fad0 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.242+0000 7f4e8e5f4640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e880a03e0 0x7f4e8806e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e8d5f2640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 msgr2=0x7f4e8806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e8d5f2640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 0x7f4e8806dbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009910 tx=0x7f4e78032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78031ca0 con 0x7f4e8809fad0 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: dump: 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.258 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e90057640 5 monclient: authenticate success, global_id 224118 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e78031e00 con 0x7f4e8809fad0 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e90057640 10 monclient: _renew_subs 2024-09-15T02:09:37.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e90057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e88124200 con 0x7f4e8809fad0 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7803c650 con 0x7f4e8809fad0 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7dffb640 10 monclient: dump: 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 10 monclient: _renew_subs 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:37.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.244+0000 7f4e7ffff640 4 set_mon_vals no callback set 2024-09-15T02:09:37.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e88135d10 con 0x7f4e8809fad0 2024-09-15T02:09:37.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4e7803ca80 con 0x7f4e8809fad0 2024-09-15T02:09:37.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 0x7f4e88069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.245+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e88125e20 con 0x7f4e8806e670 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 0x7f4e88069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient: get_auth_request con 0x7f4e8806e670 auth_method 0 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e7dffb640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e6c038790 0x7f4e6c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4e7806cde0 con 0x7f4e8809fad0 2024-09-15T02:09:37.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_done global_id 214199 payload 824 2024-09-15T02:09:37.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8ddf3640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e881261a0 con 0x7f4e8806e670 2024-09-15T02:09:37.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8e5f4640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e6c038790 0x7f4e6c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.246+0000 7f4e8e5f4640 10 monclient: get_auth_request con 0x7f4e6c038790 auth_method 0 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e8ddf3640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 msgr2=0x7f4e88069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 0x7f4e88069e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400f560 tx=0x7f4e84008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84014980 con 0x7f4e8806e670 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 10 monclient: dump: 2024-09-15T02:09:37.266 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4e8400c030 con 0x7f4e8806e670 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:37.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 msgr2=0x7f4e88069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 0x7f4e88069e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400f560 tx=0x7f4e84008040 comp rx=0 tx=0).stop 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 msgr2=0x7f4e88069e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e7dffb640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806e670 0x7f4e88069e90 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400f560 tx=0x7f4e84008040 comp rx=0 tx=0).stop 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.248+0000 7f4e8e5f4640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e6c038790 0x7f4e6c03ac40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e80007fd0 tx=0x7f4e800025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e90057640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e90057640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 0x7f4e8806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e90057640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e88063a40 con 0x7f4e8806a3d0 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 0x7f4e8806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient: get_auth_request con 0x7f4e8806a3d0 auth_method 0 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 10 monclient(hunting): handle_auth_done global_id 214202 payload 824 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.254+0000 7f4e8ddf3640 1 -- 172.21.15.17:0/4186831178 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e880724a0 con 0x7f4e8806a3d0 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e8ddf3640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 msgr2=0x7f4e8806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e8ddf3640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 0x7f4e8806a7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e8401a000 tx=0x7f4e8401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84022070 con 0x7f4e8806a3d0 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 10 monclient: dump: 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f4e8400d840 con 0x7f4e8806a3d0 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 msgr2=0x7f4e8806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 0x7f4e8806a7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e8401a000 tx=0x7f4e8401b000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 msgr2=0x7f4e8806a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.257+0000 7f4e7dffb640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4e8806a3d0 0x7f4e8806a7d0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e8401a000 tx=0x7f4e8401b000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:37.271 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.272 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.273 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.274 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.275 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:37.276 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.277 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.281 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.282 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.284 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:37.284 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:37.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e6c038790 msgr2=0x7f4e6c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4e6c038790 0x7f4e6c03ac40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e80007fd0 tx=0x7f4e800025b0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 10 monclient: shutdown 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 msgr2=0x7f4e8806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 --2- 172.21.15.17:0/4186831178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4e8809fad0 0x7f4e8806dbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009910 tx=0x7f4e78032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e8e5f4640 1 -- 172.21.15.17:0/4186831178 reap_dead start 2024-09-15T02:09:37.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e8e5f4640 1 -- 172.21.15.17:0/4186831178 reap_dead start 2024-09-15T02:09:37.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 shutdown_connections 2024-09-15T02:09:37.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 >> 172.21.15.17:0/4186831178 conn(0x7f4e880a50e0 msgr2=0x7f4e880a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 shutdown_connections 2024-09-15T02:09:37.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.258+0000 7f4e90057640 1 -- 172.21.15.17:0/4186831178 wait complete. 2024-09-15T02:09:37.287 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:37.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:09:37.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:09:37.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.383+0000 7f107d457640 10 monclient: get_monmap_and_config 2024-09-15T02:09:37.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.383+0000 7f107d457640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.383+0000 7f107d457640 10 monclient: monmap: 2024-09-15T02:09:37.385 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:37.384417+0000 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:37.384417+0000 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 Processor -- start 2024-09-15T02:09:37.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 -- start start 2024-09-15T02:09:37.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient: init 2024-09-15T02:09:37.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T02:09:37.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10780a5780 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 0x7f107809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): picked mon.noname-b con 0x7f107809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1078069c90 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780a5b50 con 0x7f1078069c90 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107809c5f0 con 0x7f10780a5780 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10780a18b0 con 0x7f107809ccf0 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f107d457640 10 monclient(hunting): authenticate will time out at 1837.380757s 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10767fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 0x7f107809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10777fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1078069c90 0x7f1078139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33028/0 (socket says 172.21.15.17:33028) 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10777fe640 1 -- 172.21.15.17:0/4083693546 learned_addr learned my addr 172.21.15.17:0/4083693546 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10767fc640 10 monclient(hunting): get_auth_request con 0x7f107809ccf0 auth_method 0 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.384+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient(hunting): handle_auth_done global_id 214205 payload 274 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 1 -- 172.21.15.17:0/4083693546 >> v1:172.21.15.67:6789/0 conn(0x7f10780a5780 legacy=0x7f1078069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 1 -- 172.21.15.17:0/4083693546 >> v1:172.21.15.17:6789/0 conn(0x7f1078069c90 legacy=0x7f1078139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.390 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 1 monclient: found mon.noname-b 2024-09-15T02:09:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.385+0000 7f10767fc640 1 -- 172.21.15.17:0/4083693546 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107809d210 con 0x7f107809ccf0 2024-09-15T02:09:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.387+0000 7f10767fc640 1 -- 172.21.15.17:0/4083693546 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 msgr2=0x7f107809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.387+0000 7f10767fc640 1 --2- 172.21.15.17:0/4083693546 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 0x7f107809bc90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f106c009880 tx=0x7f106c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=192be9fb3f6db0ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 1 -- 172.21.15.17:0/4083693546 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c032440 con 0x7f107809ccf0 2024-09-15T02:09:37.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: dump: 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.397 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 1 -- 172.21.15.17:0/4083693546 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c0325a0 con 0x7f107809ccf0 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 5 monclient: authenticate success, global_id 214205 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 4 set_mon_vals no callback set 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 1 -- 172.21.15.17:0/4083693546 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c032940 con 0x7f107809ccf0 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f1075ffb640 10 monclient: dump: 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 10 monclient: shutdown 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 -- 172.21.15.17:0/4083693546 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 msgr2=0x7f107809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 --2- 172.21.15.17:0/4083693546 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 0x7f107809bc90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f106c009880 tx=0x7f106c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 -- 172.21.15.17:0/4083693546 shutdown_connections 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 --2- 172.21.15.17:0/4083693546 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f107809ccf0 0x7f107809bc90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 -- 172.21.15.17:0/4083693546 >> 172.21.15.17:0/4083693546 conn(0x7f1078070e10 msgr2=0x7f10780a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 -- 172.21.15.17:0/4083693546 shutdown_connections 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.388+0000 7f107d457640 1 -- 172.21.15.17:0/4083693546 wait complete. 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.389+0000 7f107d457640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.389+0000 7f107d457640 10 monclient: monmap: 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:37.400 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.389+0000 7f107d457640 1 Processor -- start 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.389+0000 7f107d457640 1 -- start start 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.389+0000 7f107d457640 10 monclient: init 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 0x7f107813df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): picked mon.noname-c con 0x7f1078069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f107809ccf0 0x7f107813d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): picked mon.noname-a con 0x7f107809ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780a5780 0x7f10781414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): picked mon.noname-b con 0x7f10780a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078122f40 con 0x7f107809ccf0 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078122980 con 0x7f10780a5780 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1078120410 con 0x7f1078069c90 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f107d457640 10 monclient(hunting): authenticate will time out at 1837.386348s 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 0x7f107813df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 0x7f107813df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57786/0 (socket says 172.21.15.17:57786) 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f107809ccf0 0x7f107813d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 -- 172.21.15.17:0/557162921 learned_addr learned my addr 172.21.15.17:0/557162921 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): get_auth_request con 0x7f1078069c90 auth_method 0 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): get_auth_request con 0x7f107809ccf0 auth_method 0 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780a5780 0x7f10781414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): get_auth_request con 0x7f10780a5780 auth_method 0 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f10777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient(hunting): handle_auth_done global_id 214208 payload 824 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780a5780 msgr2=0x7f10781414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780a5780 0x7f10781414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f107809ccf0 msgr2=0x7f107813d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f107809ccf0 0x7f107813d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 monclient: found mon.noname-c 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.390+0000 7f1076ffd640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c009390 con 0x7f1078069c90 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f1076ffd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 msgr2=0x7f107813df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f1076ffd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 0x7f107813df50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f106800c610 tx=0x7f1068007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1068012840 con 0x7f1078069c90 2024-09-15T02:09:37.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: dump: 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10680129e0 con 0x7f1078069c90 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1068012da0 con 0x7f1078069c90 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f105effd640 10 monclient: dump: 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.407 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 5 monclient: authenticate success, global_id 214208 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 10 monclient: _renew_subs 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10781226a0 con 0x7f1078069c90 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f10757fa640 4 set_mon_vals no callback set 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 10 monclient: _renew_subs 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.393+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107813dce0 con 0x7f1078069c90 2024-09-15T02:09:37.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f107d457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f107d457640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 0x7f1078141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078063b20 con 0x7f10781217d0 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1068011590 con 0x7f1078069c90 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f105effd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1064038740 0x7f106403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f106804c120 con 0x7f1078069c90 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 0x7f1078141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient: get_auth_request con 0x7f10781217d0 auth_method 0 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10777fe640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1064038740 0x7f106403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10777fe640 10 monclient: get_auth_request con 0x7f1064038740 auth_method 0 2024-09-15T02:09:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 10 monclient(hunting): handle_auth_done global_id 204301 payload 824 2024-09-15T02:09:37.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.394+0000 7f10767fc640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10781230b0 con 0x7f10781217d0 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.396+0000 7f10767fc640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 msgr2=0x7f1078141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.396+0000 7f10767fc640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 0x7f1078141a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f10780a0c50 tx=0x7f106c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f10777fe640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1064038740 0x7f106403abf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f107813e4d0 tx=0x7f107000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c031bd0 con 0x7f10781217d0 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 10 monclient: dump: 2024-09-15T02:09:37.412 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f106c031d30 con 0x7f10781217d0 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:37.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 msgr2=0x7f1078141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 0x7f1078141a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f10780a0c50 tx=0x7f106c032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 msgr2=0x7f1078141a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.397+0000 7f105effd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10781217d0 0x7f1078141a00 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f10780a0c50 tx=0x7f106c032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f107d457640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f107d457640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 0x7f1078141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f107d457640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078122bc0 con 0x7f107813e8d0 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 0x7f1078141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient: get_auth_request con 0x7f107813e8d0 auth_method 0 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 10 monclient(hunting): handle_auth_done global_id 204304 payload 824 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.403+0000 7f10767fc640 1 -- 172.21.15.17:0/557162921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1078122d80 con 0x7f107813e8d0 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f10767fc640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 msgr2=0x7f1078141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f10767fc640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 0x7f1078141f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f106c009880 tx=0x7f106c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c048960 con 0x7f107813e8d0 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 10 monclient: dump: 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.415 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f106c0443d0 con 0x7f107813e8d0 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 msgr2=0x7f1078141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 0x7f1078141f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f106c009880 tx=0x7f106c03d000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 msgr2=0x7f1078141f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.406+0000 7f105effd640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107813e8d0 0x7f1078141f40 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f106c009880 tx=0x7f106c03d000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:37.431 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:37.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1064038740 msgr2=0x7f106403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1064038740 0x7f106403abf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f107813e4d0 tx=0x7f107000d040 comp rx=0 tx=0).stop 2024-09-15T02:09:37.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 10 monclient: shutdown 2024-09-15T02:09:37.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 msgr2=0x7f107813df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 --2- 172.21.15.17:0/557162921 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1078069c90 0x7f107813df50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f106800c610 tx=0x7f1068007ed0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f10777fe640 1 -- 172.21.15.17:0/557162921 reap_dead start 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f10777fe640 1 -- 172.21.15.17:0/557162921 reap_dead start 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 shutdown_connections 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 >> 172.21.15.17:0/557162921 conn(0x7f1078070e10 msgr2=0x7f1078098e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 shutdown_connections 2024-09-15T02:09:37.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.407+0000 7f107d457640 1 -- 172.21.15.17:0/557162921 wait complete. 2024-09-15T02:09:37.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:37.433 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:09:37.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:09:37.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.528+0000 7f209579d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.528+0000 7f209579d640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.528+0000 7f209579d640 10 monclient: monmap: 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:37.529271+0000 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:37.529271+0000 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:37.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 Processor -- start 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 -- start start 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient: init 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T02:09:37.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20900708a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): picked mon.noname-b con 0x7f2090097870 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2090135110 0x7f20901374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): picked mon.noname-c con 0x7f2090135110 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209006d7d0 con 0x7f20900708a0 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2090096b40 con 0x7f2090135110 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2090097c40 con 0x7f2090097870 2024-09-15T02:09:37.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f209579d640 10 monclient(hunting): authenticate will time out at 1837.525589s 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57800/0 (socket says 172.21.15.17:57800) 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 1 -- 172.21.15.17:0/371345684 learned_addr learned my addr 172.21.15.17:0/371345684 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): get_auth_request con 0x7f2090097870 auth_method 0 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 1 --2- 172.21.15.17:0/371345684 >> v2:172.21.15.67:3300/0 conn(0x7f2090135110 0x7f20901374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): get_auth_request con 0x7f2090135110 auth_method 0 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.529+0000 7f208e7fc640 10 monclient(hunting): handle_auth_done global_id 214211 payload 274 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 1 -- 172.21.15.17:0/371345684 >> v2:172.21.15.67:3300/0 conn(0x7f2090135110 msgr2=0x7f20901374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 1 --2- 172.21.15.17:0/371345684 >> v2:172.21.15.67:3300/0 conn(0x7f2090135110 0x7f20901374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 1 -- 172.21.15.17:0/371345684 >> v1:172.21.15.67:6789/0 conn(0x7f20900708a0 legacy=0x7f209006a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 1 monclient: found mon.noname-b 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.530+0000 7f208e7fc640 1 -- 172.21.15.17:0/371345684 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209006e5a0 con 0x7f2090097870 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.532+0000 7f208e7fc640 1 -- 172.21.15.17:0/371345684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 msgr2=0x7f209006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.532+0000 7f208e7fc640 1 --2- 172.21.15.17:0/371345684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2084009910 tx=0x7f20840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13d5f7c439d648c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 1 -- 172.21.15.17:0/371345684 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208402aea0 con 0x7f2090097870 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: dump: 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.540 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 1 -- 172.21.15.17:0/371345684 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084032440 con 0x7f2090097870 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 5 monclient: authenticate success, global_id 214211 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:37.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 1 -- 172.21.15.17:0/371345684 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084032820 con 0x7f2090097870 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 4 set_mon_vals no callback set 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f208dffb640 10 monclient: dump: 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.542 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 10 monclient: shutdown 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 -- 172.21.15.17:0/371345684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 msgr2=0x7f209006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 --2- 172.21.15.17:0/371345684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2084009910 tx=0x7f20840091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 -- 172.21.15.17:0/371345684 shutdown_connections 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 --2- 172.21.15.17:0/371345684 >> v2:172.21.15.67:3300/0 conn(0x7f2090135110 0x7f20901374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 --2- 172.21.15.17:0/371345684 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209006a8d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 -- 172.21.15.17:0/371345684 >> 172.21.15.17:0/371345684 conn(0x7f2090067ae0 msgr2=0x7f20900a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 -- 172.21.15.17:0/371345684 shutdown_connections 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.533+0000 7f209579d640 1 -- 172.21.15.17:0/371345684 wait complete. 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.534+0000 7f209579d640 10 monclient: build_initial_monmap 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.534+0000 7f209579d640 10 monclient: monmap: 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:37.545 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:37.545 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.534+0000 7f209579d640 1 Processor -- start 2024-09-15T02:09:37.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.534+0000 7f209579d640 1 -- start start 2024-09-15T02:09:37.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.534+0000 7f209579d640 10 monclient: init 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20900708a0 0x7f209011c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): picked mon.noname-b con 0x7f20900708a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): picked mon.noname-c con 0x7f2090097870 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): picked mon.noname-a con 0x7f2090135110 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209006e8d0 con 0x7f2090135110 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209009cba0 con 0x7f20900708a0 2024-09-15T02:09:37.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209011ff70 con 0x7f2090097870 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f209579d640 10 monclient(hunting): authenticate will time out at 1837.531269s 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57806/0 (socket says 172.21.15.17:57806) 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42458/0 (socket says 172.21.15.17:42458) 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 -- 172.21.15.17:0/1488945665 learned_addr learned my addr 172.21.15.17:0/1488945665 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): get_auth_request con 0x7f2090097870 auth_method 0 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): get_auth_request con 0x7f2090135110 auth_method 0 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20900708a0 0x7f209011c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): get_auth_request con 0x7f20900708a0 auth_method 0 2024-09-15T02:09:37.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient(hunting): handle_auth_done global_id 214214 payload 824 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 -- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20900708a0 msgr2=0x7f209011c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20900708a0 0x7f209011c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 -- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 msgr2=0x7f209011d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 monclient: found mon.noname-c 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208e7fc640 1 -- 172.21.15.17:0/1488945665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009450 con 0x7f2090097870 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.535+0000 7f208f7fe640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.537+0000 7f208e7fc640 1 -- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 msgr2=0x7f209011cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.537+0000 7f208e7fc640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2084009210 tx=0x7f2084032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208402aea0 con 0x7f2090097870 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: dump: 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: _finish_auth 0 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:37.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084000d10 con 0x7f2090097870 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084031ea0 con 0x7f2090097870 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 5 monclient: authenticate success, global_id 214214 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 10 monclient: _renew_subs 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209011ed30 con 0x7f2090097870 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f2076ffd640 10 monclient: dump: 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:37.556 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f208d7fa640 4 set_mon_vals no callback set 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 10 monclient: _renew_subs 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.538+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209011fd40 con 0x7f2090097870 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f208403c610 con 0x7f2090097870 2024-09-15T02:09:37.557 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f209579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f209579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f209579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20900724a0 con 0x7f2090097870 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f2076ffd640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 0x7f207c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f208406cef0 con 0x7f2090097870 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f208effd640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 0x7f207c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.539+0000 7f208effd640 10 monclient: get_auth_request con 0x7f207c038790 auth_method 0 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.542+0000 7f208effd640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 0x7f207c03ac40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.543+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f208404a870 con 0x7f2090097870 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.543+0000 7f2076ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:37.558 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.543+0000 7f2076ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.771+0000 7f209579d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:09:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.771+0000 7f209579d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:09:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.771+0000 7f209579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.771+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f20900711e0 con 0x7f2090097870 2024-09-15T02:09:37.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.772+0000 7f2076ffd640 1 -- 172.21.15.17:0/1488945665 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f2084037080 con 0x7f2090097870 2024-09-15T02:09:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.772+0000 7f2076ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:09:37.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.772+0000 7f2076ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:37.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.774 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=3311438681,config=3140514009,health=923807535,kv=1529054990,logm=1012124719}) 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=1597444917,config=3140514009,health=2697108806,kv=1529054990,logm=3646282607}) 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=3311438681,config=3140514009,health=923807535,kv=1529054990,logm=1012124719}) 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.773+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=3427683383,config=3140514009,health=923807535,kv=1529054990,logm=993213445}) 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.775 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704282387}) 2024-09-15T02:09:37.776 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829183012}) 2024-09-15T02:09:37.776 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.776 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704282387}) 2024-09-15T02:09:37.776 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537484485}) 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 msgr2=0x7f207c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 0x7f207c03ac40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007e10 comp rx=0 tx=0).stop 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f209579d640 10 monclient: shutdown 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f209579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.774+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 msgr2=0x7f209011cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2084009210 tx=0x7f2084032000 comp rx=0 tx=0).stop 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 shutdown_connections 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f207c038790 0x7f207c03ac40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2090135110 0x7f209011d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f2090097870 0x7f209011cb60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 --2- 172.21.15.17:0/1488945665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20900708a0 0x7f209011c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:37.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 >> 172.21.15.17:0/1488945665 conn(0x7f2090067ae0 msgr2=0x7f209009b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:37.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 shutdown_connections 2024-09-15T02:09:37.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f209579d640 1 -- 172.21.15.17:0/1488945665 wait complete. 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350798289}) 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743451432}) 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350798289}) 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.775+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3049269948}) 2024-09-15T02:09:37.779 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746716783}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101534961}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746716783}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.776+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139432922}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702352331}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2361054846}) 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.780 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702352331}) 2024-09-15T02:09:37.781 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.777+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793067113}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139347619}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143019592}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139347619}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.778+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276740814}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438078404}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925875362}) 2024-09-15T02:09:37.783 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438078404}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.779+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366096819}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844077774}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2799872211}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844077774}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192542020}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148226111}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535326615}) 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148226111}) 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.780+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=924466988}) 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606119718}) 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032796593}) 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606119718}) 2024-09-15T02:09:37.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.781+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944647744}) 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852747026}) 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984580303}) 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852747026}) 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.782+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095348320}) 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939728516}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772972230}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939728516}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37682542}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76165012}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550549694}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76165012}) 2024-09-15T02:09:37.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.783+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456169739}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194959383}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954160457}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194959383}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.784+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790482898}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916825248}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1076369663}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916825248}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1444750503}) 2024-09-15T02:09:37.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092735105}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2475570334}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092735105}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.785+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686120513}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616865231}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996617646}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616865231}) 2024-09-15T02:09:37.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.786+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164118354}) 2024-09-15T02:09:37.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074779255}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123452428}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074779255}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844322176}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412829719}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.787+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706569803}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412829719}) 2024-09-15T02:09:37.798 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006113598}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235547395}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=946459737}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235547395}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.788+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139055963}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226734080}) 2024-09-15T02:09:37.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258259641}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226734080}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061168999}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923053475}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533667297}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923053475}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.789+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861673388}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567699337}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228476466}) 2024-09-15T02:09:37.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567699337}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.790+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931627842}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407421874}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127507613}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407421874}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2029846354}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771332036}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577476796}) 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771332036}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.791+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814022915}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369487414}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085691673}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369487414}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.792+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3307173438}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777717191}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917195598}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777717191}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1642084600}) 2024-09-15T02:09:37.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548831446}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794031909}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548831446}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.793+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400937694}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983385668}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360460969}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983385668}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.794+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125067319}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151643332}) 2024-09-15T02:09:37.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521763148}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151643332}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=828633647}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814681983}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829669159}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814681983}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.795+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4010459166}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946831737}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749746046}) 2024-09-15T02:09:37.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946831737}) 2024-09-15T02:09:37.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.796+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140772560}) 2024-09-15T02:09:37.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113486626}) 2024-09-15T02:09:37.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3785826248}) 2024-09-15T02:09:37.812 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113486626}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.797+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864215977}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847215287}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866977512}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847215287}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641705523}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645546348}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705363957}) 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.813 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645546348}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.798+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185821515}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419590999}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842686889}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419590999}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.799+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860991987}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035936671}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912669034}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035936671}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309667843}) 2024-09-15T02:09:37.814 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123870077}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468618225}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123870077}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.800+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983963161}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620259072}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102350904}) 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.815 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620259072}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.801+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3980776945}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565843564}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553474728}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565843564}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085643479}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661663744}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031557395}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661663744}) 2024-09-15T02:09:37.816 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.802+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271607184}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307154989}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797013316}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307154989}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.803+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347230490}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641722122}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497424539}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641722122}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550336806}) 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.817 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917057799}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3771041224}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917057799}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.804+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875387509}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251712915}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874106246}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251712915}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.805+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2102976779}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245692393}) 2024-09-15T02:09:37.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806008441}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245692393}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487713439}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199781783}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1600815672}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199781783}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.806+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269122143}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595565966}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013709140}) 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595565966}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.807+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216033134}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280683428}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753732033}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280683428}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218222297}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330323662}) 2024-09-15T02:09:37.820 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146420904}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330323662}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.808+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207978991}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826765041}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008032983}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826765041}) 2024-09-15T02:09:37.821 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.810+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180925263}) 2024-09-15T02:09:37.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380148757}) 2024-09-15T02:09:37.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447569294}) 2024-09-15T02:09:37.822 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380148757}) 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.813+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237362909}) 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989738784}) 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466892668}) 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989738784}) 2024-09-15T02:09:37.823 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.814+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208797405}) 2024-09-15T02:09:37.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344611614}) 2024-09-15T02:09:37.824 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826981285}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344611614}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.821+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448679118}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596786094}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158048050}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596786094}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.822+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=990690420}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359458854}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834208245}) 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.825 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359458854}) 2024-09-15T02:09:37.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1075025536}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221403336}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98237668}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221403336}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.823+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414183036}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921488687}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148585942}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921488687}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110063133}) 2024-09-15T02:09:37.828 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449154105}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409346375}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449154105}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984356389}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316019475}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904638447}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316019475}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.825+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507415809}) 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.829 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277868824}) 2024-09-15T02:09:37.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786573948}) 2024-09-15T02:09:37.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277868824}) 2024-09-15T02:09:37.830 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.826+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697164484}) 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174919175}) 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033416853}) 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174919175}) 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.827+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357499564}) 2024-09-15T02:09:37.832 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265686979}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1710315266}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265686979}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900340860}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291041555}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904389463}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291041555}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.828+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760935628}) 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.833 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147827086}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890361076}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147827086}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.829+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3292326781}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393836658}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936220797}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393836658}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181054577}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599834535}) 2024-09-15T02:09:37.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3310302088}) 2024-09-15T02:09:37.835 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:37.835 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599834535}) 2024-09-15T02:09:37.835 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:37.830+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173887390}) 2024-09-15T02:09:38.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183970156}) 2024-09-15T02:09:38.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654618830}) 2024-09-15T02:09:38.052 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183970156}) 2024-09-15T02:09:38.053 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.050+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479603618}) 2024-09-15T02:09:38.055 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.055 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507631189}) 2024-09-15T02:09:38.055 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860015130}) 2024-09-15T02:09:38.055 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507631189}) 2024-09-15T02:09:38.056 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.054+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=633852872}) 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396794334}) 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247203270}) 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396794334}) 2024-09-15T02:09:38.059 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.057+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80742081}) 2024-09-15T02:09:38.539 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589324809}) 2024-09-15T02:09:38.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3571715714}) 2024-09-15T02:09:38.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589324809}) 2024-09-15T02:09:38.540 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.538+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665427020}) 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738064390}) 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168552638}) 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738064390}) 2024-09-15T02:09:38.688 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.686+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070702705}) 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827832412}) 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089172848}) 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827832412}) 2024-09-15T02:09:38.689 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.687+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505594530}) 2024-09-15T02:09:38.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561471728}) 2024-09-15T02:09:38.747 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=95450438}) 2024-09-15T02:09:38.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561471728}) 2024-09-15T02:09:38.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.746+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185109510}) 2024-09-15T02:09:38.748 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853563555}) 2024-09-15T02:09:38.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455045552}) 2024-09-15T02:09:38.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853563555}) 2024-09-15T02:09:38.749 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.747+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971265103}) 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454956080}) 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1653404092}) 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454956080}) 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853624644}) 2024-09-15T02:09:38.750 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.751 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201265047}) 2024-09-15T02:09:38.751 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=17699509}) 2024-09-15T02:09:38.751 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.751 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201265047}) 2024-09-15T02:09:38.751 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.748+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564325819}) 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957097835}) 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2378530694}) 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957097835}) 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.749+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=255202910}) 2024-09-15T02:09:38.753 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773058625}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714851239}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773058625}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.750+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460407897}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139118974}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933992404}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139118974}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471983159}) 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.754 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365482600}) 2024-09-15T02:09:38.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=784812791}) 2024-09-15T02:09:38.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365482600}) 2024-09-15T02:09:38.755 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.751+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1752479796}) 2024-09-15T02:09:38.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800806476}) 2024-09-15T02:09:38.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219667673}) 2024-09-15T02:09:38.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.758 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800806476}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.752+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84109649}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075084947}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089609856}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075084947}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672090528}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128851140}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352707871}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128851140}) 2024-09-15T02:09:38.759 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.753+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289748197}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033038934}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1703531949}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033038934}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.754+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458258768}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770756876}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526109619}) 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.760 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770756876}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.755+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3292414361}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328033118}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511809811}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328033118}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227233597}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233635927}) 2024-09-15T02:09:38.761 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908102595}) 2024-09-15T02:09:38.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233635927}) 2024-09-15T02:09:38.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.756+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=364701080}) 2024-09-15T02:09:38.762 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.763 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596086098}) 2024-09-15T02:09:38.763 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050294488}) 2024-09-15T02:09:38.763 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.763 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596086098}) 2024-09-15T02:09:38.763 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.757+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331544741}) 2024-09-15T02:09:38.793 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:09:38.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:38.818 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273682350}) 2024-09-15T02:09:38.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744318241}) 2024-09-15T02:09:38.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273682350}) 2024-09-15T02:09:38.819 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.817+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466486264}) 2024-09-15T02:09:38.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005176919}) 2024-09-15T02:09:38.826 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179791429}) 2024-09-15T02:09:38.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:38.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005176919}) 2024-09-15T02:09:38.827 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:38.824+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089247687}) 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.895+0000 7f99dbc57640 10 monclient: get_monmap_and_config 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.895+0000 7f99dbc57640 10 monclient: build_initial_monmap 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.895+0000 7f99dbc57640 10 monclient: monmap: 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:38.895998+0000 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:38.895998+0000 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:38.897 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:38.898 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:38.898 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:38.898 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.895+0000 7f99dbc57640 1 Processor -- start 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 1 -- start start 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient: init 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-b con 0x7f99d409fad0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99d409f0f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:38.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99d40a05b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d409c7c0 con 0x7f99d40a05b0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d409c9e0 con 0x7f99d409f0f0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d409dbc0 con 0x7f99d409fad0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99dbc57640 10 monclient(hunting): authenticate will time out at 1838.892561s 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57822/0 (socket says 172.21.15.17:57822) 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99da1f4640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f99d40a05b0 0x7f99d413d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33038/0 (socket says 172.21.15.17:33038) 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 -- 172.21.15.17:0/215436007 learned_addr learned my addr 172.21.15.17:0/215436007 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): get_auth_request con 0x7f99d409fad0 auth_method 0 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:38.901 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient(hunting): handle_auth_done global_id 214217 payload 274 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 -- 172.21.15.17:0/215436007 >> v1:172.21.15.67:6789/0 conn(0x7f99d409f0f0 legacy=0x7f99d409fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 -- 172.21.15.17:0/215436007 >> v1:172.21.15.17:6789/0 conn(0x7f99d40a05b0 legacy=0x7f99d413d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient: _finish_hunting 0 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 monclient: found mon.noname-b 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:38.902 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.896+0000 7f99d99f3640 1 -- 172.21.15.17:0/215436007 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d409f4c0 con 0x7f99d409fad0 2024-09-15T02:09:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.899+0000 7f99d99f3640 1 -- 172.21.15.17:0/215436007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 msgr2=0x7f99d409ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.899+0000 7f99d99f3640 1 --2- 172.21.15.17:0/215436007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009940 tx=0x7f99d00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9086fe9a5b5107f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 1 -- 172.21.15.17:0/215436007 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0032440 con 0x7f99d409fad0 2024-09-15T02:09:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:38.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: dump: 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: _finish_auth 0 2024-09-15T02:09:38.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 5 monclient: authenticate success, global_id 214217 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 1 -- 172.21.15.17:0/215436007 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d00325a0 con 0x7f99d409fad0 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 4 set_mon_vals no callback set 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 1 -- 172.21.15.17:0/215436007 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0032940 con 0x7f99d409fad0 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99d89f1640 10 monclient: dump: 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:38.909 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 10 monclient: shutdown 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 -- 172.21.15.17:0/215436007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 msgr2=0x7f99d409ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 --2- 172.21.15.17:0/215436007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009940 tx=0x7f99d00091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:38.910 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 -- 172.21.15.17:0/215436007 shutdown_connections 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 --2- 172.21.15.17:0/215436007 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409fad0 0x7f99d409ebb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 -- 172.21.15.17:0/215436007 >> 172.21.15.17:0/215436007 conn(0x7f99d40a50e0 msgr2=0x7f99d40a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 -- 172.21.15.17:0/215436007 shutdown_connections 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.900+0000 7f99dbc57640 1 -- 172.21.15.17:0/215436007 wait complete. 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.901+0000 7f99dbc57640 10 monclient: build_initial_monmap 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.901+0000 7f99dbc57640 10 monclient: monmap: 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.911 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.901+0000 7f99dbc57640 1 Processor -- start 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.901+0000 7f99dbc57640 1 -- start start 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.901+0000 7f99dbc57640 10 monclient: init 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 0x7f99d406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-c con 0x7f99d409f0f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 0x7f99d413e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-b con 0x7f99d409fad0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:38.912 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 0x7f99d4123f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): picked mon.noname-a con 0x7f99d40a05b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d406e880 con 0x7f99d40a05b0 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4126380 con 0x7f99d409fad0 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4126660 con 0x7f99d409f0f0 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99dbc57640 10 monclient(hunting): authenticate will time out at 1838.898133s 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 0x7f99d406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 0x7f99d406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57836/0 (socket says 172.21.15.17:57836) 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 0x7f99d4123f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 1 -- 172.21.15.17:0/70594813 learned_addr learned my addr 172.21.15.17:0/70594813 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:38.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): get_auth_request con 0x7f99d409f0f0 auth_method 0 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): get_auth_request con 0x7f99d40a05b0 auth_method 0 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d99f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:38.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d91f2640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 0x7f99d413e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient(hunting): handle_auth_done global_id 224127 payload 824 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 -- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 msgr2=0x7f99d406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 0x7f99d406dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 -- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 msgr2=0x7f99d413e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 0x7f99d413e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient: _finish_hunting 0 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 monclient: found mon.noname-a 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99da1f4640 1 -- 172.21.15.17:0/70594813 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0009450 con 0x7f99d40a05b0 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d91f2640 10 monclient: get_auth_request con 0x7f99d409fad0 auth_method 0 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.902+0000 7f99d91f2640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 0x7f99d413e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:09:38.915 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99da1f4640 1 -- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 msgr2=0x7f99d4123f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99da1f4640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 0x7f99d4123f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00c790 tx=0x7f99cc004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc005d60 con 0x7f99d40a05b0 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 10 monclient: dump: 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:38.916 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00a640 con 0x7f99d40a05b0 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.904+0000 7f99dbc57640 5 monclient: authenticate success, global_id 224127 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00aa00 con 0x7f99d40a05b0 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 10 monclient: _renew_subs 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d4126940 con 0x7f99d40a05b0 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99cbfff640 4 set_mon_vals no callback set 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 10 monclient: dump: 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 10 monclient: _renew_subs 2024-09-15T02:09:38.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d41459b0 con 0x7f99d40a05b0 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.905+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f99cc0070f0 con 0x7f99d40a05b0 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99c9ffb640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 0x7f99b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f99cc04c2b0 con 0x7f99d40a05b0 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99dbc57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99dbc57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99dbc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:38.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d4126b80 con 0x7f99d40a05b0 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99d99f3640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 0x7f99b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.906+0000 7f99d99f3640 10 monclient: get_auth_request con 0x7f99b8038740 auth_method 0 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.909+0000 7f99d99f3640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 0x7f99b803abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009420 tx=0x7f99d0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.910+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f99d4126b80 con 0x7f99d40a05b0 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.910+0000 7f99c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:38.910+0000 7f99c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99dbc57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:39.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99dbc57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99dbc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f99d4126b80 con 0x7f99d40a05b0 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99c9ffb640 1 -- 172.21.15.17:0/70594813 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f99d4126b80 con 0x7f99d40a05b0 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.136+0000 7f99c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:39.138 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 msgr2=0x7f99b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 0x7f99b803abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009420 tx=0x7f99d0032000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 10 monclient: shutdown 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 msgr2=0x7f99d4123f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 0x7f99d4123f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00c790 tx=0x7f99cc004e20 comp rx=0 tx=0).stop 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 shutdown_connections 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f99b8038740 0x7f99b803abf0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f99d40a05b0 0x7f99d4123f50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99d409fad0 0x7f99d413e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 --2- 172.21.15.17:0/70594813 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f99d409f0f0 0x7f99d406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 >> 172.21.15.17:0/70594813 conn(0x7f99d40a50e0 msgr2=0x7f99d409a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 shutdown_connections 2024-09-15T02:09:39.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f99dbc57640 1 -- 172.21.15.17:0/70594813 wait complete. 2024-09-15T02:09:39.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.138+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178145171}) 2024-09-15T02:09:39.142 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922229525}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178145171}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267699636}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172976891}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348657121}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172976891}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.139+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496492921}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272099531}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839771165}) 2024-09-15T02:09:39.143 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272099531}) 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.140+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893958329}) 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047709893}) 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539908005}) 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047709893}) 2024-09-15T02:09:39.144 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2070316800}) 2024-09-15T02:09:39.145 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760251864}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1918179886}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760251864}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.141+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982194342}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874214411}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116614967}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874214411}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=171159619}) 2024-09-15T02:09:39.146 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=858691603}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114075736}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=858691603}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.142+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556098484}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3902550720,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=1157857860}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2675118055,mdsmap=3585761405,mgr=1712270623,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3609777412}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3902550720,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=1157857860}) 2024-09-15T02:09:39.147 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.143+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=4245762531,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3729836231}) 2024-09-15T02:09:39.151 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:09:39.151 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:09:39.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:09:39.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:09:39.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgrstat=2261287748,mon_config_key=1583035879,monmap=2689118867,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3597327134}) 2024-09-15T02:09:39.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgrstat=2261287748,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1079644522,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3316375716}) 2024-09-15T02:09:39.152 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:09:39.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgrstat=2261287748,mon_config_key=1583035879,monmap=2689118867,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3597327134}) 2024-09-15T02:09:39.153 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:09:39.149+0000 7f2f32df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mgrstat=2261287748,mon_config_key=1583035879,monmap=3140514009,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=4099002257}) 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.287+0000 7f2876f1d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.287+0000 7f2876f1d640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.287+0000 7f2876f1d640 10 monclient: monmap: 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:39.288469+0000 2024-09-15T02:09:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:39.288469+0000 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:39.290 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 Processor -- start 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 -- start start 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient: init 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T02:09:39.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f287009c0d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 0x7f287009c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f287009e970 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f287006eb30 0x7f287006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f287006eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28700699c0 con 0x7f287009c0d0 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287006a5e0 con 0x7f287006eb30 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287006a290 con 0x7f287009e970 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.288+0000 7f2876f1d640 10 monclient(hunting): authenticate will time out at 1839.284917s 2024-09-15T02:09:39.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f287006eb30 0x7f287006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f287006eb30 0x7f287006ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42464/0 (socket says 172.21.15.17:42464) 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 -- 172.21.15.17:0/4004375618 learned_addr learned my addr 172.21.15.17:0/4004375618 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): get_auth_request con 0x7f287006eb30 auth_method 0 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f2874cb9640 1 --1- 172.21.15.17:0/4004375618 >> v1:172.21.15.17:6789/0 conn(0x7f287009c0d0 0x7f287009a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:33048/0 (socket says 172.21.15.17:33048) 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f286ffff640 1 --2- 172.21.15.17:0/4004375618 >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 0x7f287009c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient(hunting): handle_auth_done global_id 224130 payload 274 2024-09-15T02:09:39.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 -- 172.21.15.17:0/4004375618 >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 msgr2=0x7f287009c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 --2- 172.21.15.17:0/4004375618 >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 0x7f287009c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 -- 172.21.15.17:0/4004375618 >> v1:172.21.15.17:6789/0 conn(0x7f287009c0d0 legacy=0x7f287009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 monclient: found mon.noname-a 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f28754ba640 1 -- 172.21.15.17:0/4004375618 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287009ed70 con 0x7f287006eb30 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f286ffff640 10 monclient: get_auth_request con 0x7f287009e970 auth_method 0 2024-09-15T02:09:39.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.289+0000 7f286ffff640 1 --2- 172.21.15.17:0/4004375618 >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 0x7f287009c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:09:39.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f28754ba640 1 -- 172.21.15.17:0/4004375618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 msgr2=0x7f287006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f28754ba640 1 --2- 172.21.15.17:0/4004375618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f287006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f287009f5e0 tx=0x7f2868009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11185d1268d643b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 1 -- 172.21.15.17:0/4004375618 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868007610 con 0x7f287006eb30 2024-09-15T02:09:39.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: dump: 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 5 monclient: authenticate success, global_id 224130 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 1 -- 172.21.15.17:0/4004375618 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286802d050 con 0x7f287006eb30 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 4 set_mon_vals no callback set 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 1 -- 172.21.15.17:0/4004375618 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28680079a0 con 0x7f287006eb30 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f286f7fe640 10 monclient: dump: 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.303 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 10 monclient: shutdown 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 -- 172.21.15.17:0/4004375618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 msgr2=0x7f287006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 --2- 172.21.15.17:0/4004375618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f287006ef20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f287009f5e0 tx=0x7f2868009840 comp rx=0 tx=0).stop 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 -- 172.21.15.17:0/4004375618 shutdown_connections 2024-09-15T02:09:39.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 --2- 172.21.15.17:0/4004375618 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f287006ef20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 --2- 172.21.15.17:0/4004375618 >> v2:172.21.15.67:3300/0 conn(0x7f287009e970 0x7f287009c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.292+0000 7f2876f1d640 1 -- 172.21.15.17:0/4004375618 >> 172.21.15.17:0/4004375618 conn(0x7f28700712f0 msgr2=0x7f28700a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 1 -- 172.21.15.17:0/4004375618 shutdown_connections 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 1 -- 172.21.15.17:0/4004375618 wait complete. 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 10 monclient: monmap: 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.305 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 1 Processor -- start 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 1 -- start start 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.293+0000 7f2876f1d640 10 monclient: init 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f287006eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f287009c0d0 0x7f2870140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f287009c0d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f287009e970 0x7f28701413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f287009e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287009cd80 con 0x7f287006eb30 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870122490 con 0x7f287009e970 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287011fce0 con 0x7f287009c0d0 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2876f1d640 10 monclient(hunting): authenticate will time out at 1839.290399s 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f287009c0d0 0x7f2870140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42470/0 (socket says 172.21.15.17:42470) 2024-09-15T02:09:39.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 -- 172.21.15.17:0/1588253228 learned_addr learned my addr 172.21.15.17:0/1588253228 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): get_auth_request con 0x7f287006eb30 auth_method 0 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): get_auth_request con 0x7f287009c0d0 auth_method 0 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f287009e970 0x7f28701413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 10 monclient(hunting): get_auth_request con 0x7f287009e970 auth_method 0 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient(hunting): handle_auth_done global_id 224133 payload 824 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f287009c0d0 msgr2=0x7f2870140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f287009c0d0 0x7f2870140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f287009e970 msgr2=0x7f28701413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f287009e970 0x7f28701413d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 monclient: found mon.noname-a 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f2874cb9640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009ae0 con 0x7f287006eb30 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f28754ba640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f287009e970 0x7f28701413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.294+0000 7f286ffff640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f287009c0d0 0x7f2870140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2874cb9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 msgr2=0x7f2870133d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2874cb9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f286400c5b0 tx=0x7f2864007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864007510 con 0x7f287006eb30 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:39.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: dump: 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28640057d0 con 0x7f287006eb30 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864007690 con 0x7f287006eb30 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 5 monclient: authenticate success, global_id 224133 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 10 monclient: _renew_subs 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2870121e40 con 0x7f287006eb30 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286effd640 4 set_mon_vals no callback set 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f286cff9640 10 monclient: dump: 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.312 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 10 monclient: _renew_subs 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.297+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870121080 con 0x7f287006eb30 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f2876f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f2876f1d640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 0x7f287011e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870063b90 con 0x7f2870134250 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 0x7f287011e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient: get_auth_request con 0x7f2870134250 auth_method 0 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f286401b400 con 0x7f287006eb30 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 10 monclient(hunting): handle_auth_done global_id 224136 payload 824 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286ffff640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28700724a0 con 0x7f2870134250 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286cff9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2854038790 0x7f285403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f286404c0c0 con 0x7f287006eb30 2024-09-15T02:09:39.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f28754ba640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2854038790 0x7f285403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.315 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.298+0000 7f28754ba640 10 monclient: get_auth_request con 0x7f2854038790 auth_method 0 2024-09-15T02:09:39.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286ffff640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 msgr2=0x7f287011e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286ffff640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 0x7f287011e870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28700a3710 tx=0x7f285c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c013650 con 0x7f2870134250 2024-09-15T02:09:39.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.300+0000 7f286cff9640 10 monclient: dump: 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f285c00c040 con 0x7f2870134250 2024-09-15T02:09:39.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 msgr2=0x7f287011e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 0x7f287011e870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28700a3710 tx=0x7f285c008040 comp rx=0 tx=0).stop 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 msgr2=0x7f287011e870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f286cff9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2870134250 0x7f287011e870 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28700a3710 tx=0x7f285c008040 comp rx=0 tx=0).stop 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.301+0000 7f28754ba640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2854038790 0x7f285403ac40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28700a3e50 tx=0x7f2868036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f2876f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f2876f1d640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 0x7f287011f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f2876f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870134620 con 0x7f287011edb0 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f286ffff640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 0x7f287011f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.319 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f286ffff640 10 monclient: get_auth_request con 0x7f287011edb0 auth_method 0 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.306+0000 7f286ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 10 monclient(hunting): handle_auth_done global_id 224139 payload 824 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.307+0000 7f286ffff640 1 -- 172.21.15.17:0/1588253228 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f287011f6f0 con 0x7f287011edb0 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286ffff640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 msgr2=0x7f287011f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286ffff640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 0x7f287011f1b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f285c01a000 tx=0x7f285c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c023040 con 0x7f287011edb0 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.320 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 10 monclient: dump: 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7f285c005ec0 con 0x7f287011edb0 2024-09-15T02:09:39.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 msgr2=0x7f287011f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 0x7f287011f1b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f285c01a000 tx=0x7f285c01d000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 msgr2=0x7f287011f1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.309+0000 7f286cff9640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287011edb0 0x7f287011f1b0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f285c01a000 tx=0x7f285c01d000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:39.322 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:39.323 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:39.324 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:39.325 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:39.326 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.327 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.330 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.332 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.333 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:39.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.335 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:39.335 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:39.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2854038790 msgr2=0x7f285403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f2854038790 0x7f285403ac40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f28700a3e50 tx=0x7f2868036000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 10 monclient: shutdown 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 msgr2=0x7f2870133d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f286400c5b0 tx=0x7f2864007f00 comp rx=0 tx=0).stop 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f28754ba640 1 -- 172.21.15.17:0/1588253228 reap_dead start 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 shutdown_connections 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 --2- 172.21.15.17:0/1588253228 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f287006eb30 0x7f2870133d10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 >> 172.21.15.17:0/1588253228 conn(0x7f28700712f0 msgr2=0x7f28700a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 shutdown_connections 2024-09-15T02:09:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.310+0000 7f2876f1d640 1 -- 172.21.15.17:0/1588253228 wait complete. 2024-09-15T02:09:39.337 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:39.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:09:39.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.433+0000 7fbf2a514640 10 monclient: get_monmap_and_config 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.433+0000 7fbf2a514640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.433+0000 7fbf2a514640 10 monclient: monmap: 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:39.434468+0000 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:39.434468+0000 2024-09-15T02:09:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:39.436 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 Processor -- start 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 -- start start 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient: init 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-a con 0x7fbf240a5780 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 0x7fbf24069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2409ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf24069ac0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240a5b50 con 0x7fbf24069ac0 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240a18b0 con 0x7fbf240a5780 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2409c5f0 con 0x7fbf2409ccf0 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.434+0000 7fbf2a514640 10 monclient(hunting): authenticate will time out at 1839.430938s 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 0x7fbf24069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42494/0 (socket says 172.21.15.17:42494) 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 1 -- 172.21.15.17:0/1422203364 learned_addr learned my addr 172.21.15.17:0/1422203364 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): get_auth_request con 0x7fbf240a5780 auth_method 0 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): get_auth_request con 0x7fbf2409ccf0 auth_method 0 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient(hunting): handle_auth_done global_id 214226 payload 274 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 -- 172.21.15.17:0/1422203364 >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 msgr2=0x7fbf2409bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 --2- 172.21.15.17:0/1422203364 >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 -- 172.21.15.17:0/1422203364 >> v1:172.21.15.67:6789/0 conn(0x7fbf24069ac0 legacy=0x7fbf24138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 monclient: found mon.noname-b 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf237fe640 1 -- 172.21.15.17:0/1422203364 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2409d210 con 0x7fbf2409ccf0 2024-09-15T02:09:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.435+0000 7fbf23fff640 1 --2- 172.21.15.17:0/1422203364 >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:39.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf237fe640 1 -- 172.21.15.17:0/1422203364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 msgr2=0x7fbf24069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf237fe640 1 --2- 172.21.15.17:0/1422203364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 0x7fbf24069580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bcb5233f3d8b9f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 1 -- 172.21.15.17:0/1422203364 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1802adf0 con 0x7fbf2409ccf0 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: dump: 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.449 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 1 -- 172.21.15.17:0/1422203364 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18032440 con 0x7fbf2409ccf0 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 5 monclient: authenticate success, global_id 214226 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 4 set_mon_vals no callback set 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 1 -- 172.21.15.17:0/1422203364 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180327e0 con 0x7fbf2409ccf0 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf22ffd640 10 monclient: dump: 2024-09-15T02:09:39.450 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 10 monclient: shutdown 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 -- 172.21.15.17:0/1422203364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 msgr2=0x7fbf24069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 --2- 172.21.15.17:0/1422203364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 0x7fbf24069580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf180090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 -- 172.21.15.17:0/1422203364 shutdown_connections 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 --2- 172.21.15.17:0/1422203364 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf2409ccf0 0x7fbf24069580 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 --2- 172.21.15.17:0/1422203364 >> v2:172.21.15.17:3300/0 conn(0x7fbf240a5780 0x7fbf2409bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 -- 172.21.15.17:0/1422203364 >> 172.21.15.17:0/1422203364 conn(0x7fbf24070e10 msgr2=0x7fbf240a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.438+0000 7fbf2a514640 1 -- 172.21.15.17:0/1422203364 shutdown_connections 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 1 -- 172.21.15.17:0/1422203364 wait complete. 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 10 monclient: monmap: 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 1 Processor -- start 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 1 -- start start 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.439+0000 7fbf2a514640 10 monclient: init 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 0x7fbf2413deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-a con 0x7fbf24069ac0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf2409ccf0 0x7fbf2413d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2409ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf240a5780 0x7fbf24141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf24120bf0 con 0x7fbf24069ac0 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf241228e0 con 0x7fbf2409ccf0 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2413e9a0 con 0x7fbf240a5780 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf2a514640 10 monclient(hunting): authenticate will time out at 1839.436439s 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 0x7fbf2413deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 0x7fbf2413deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42502/0 (socket says 172.21.15.17:42502) 2024-09-15T02:09:39.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 -- 172.21.15.17:0/2752284245 learned_addr learned my addr 172.21.15.17:0/2752284245 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): get_auth_request con 0x7fbf24069ac0 auth_method 0 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf240a5780 0x7fbf24141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): get_auth_request con 0x7fbf240a5780 auth_method 0 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf2409ccf0 0x7fbf2413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf28ab1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 10 monclient(hunting): get_auth_request con 0x7fbf2409ccf0 auth_method 0 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient(hunting): handle_auth_done global_id 224145 payload 824 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf240a5780 msgr2=0x7fbf24141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf240a5780 0x7fbf24141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf2409ccf0 msgr2=0x7fbf2413d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf2409ccf0 0x7fbf2413d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 monclient: found mon.noname-a 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf23fff640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14009450 con 0x7fbf24069ac0 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.440+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf2409ccf0 0x7fbf2413d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf23fff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 msgr2=0x7fbf2413deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf23fff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 0x7fbf2413deb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400ec80 tx=0x7fbf140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14009730 con 0x7fbf24069ac0 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: dump: 2024-09-15T02:09:39.457 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1400eec0 con 0x7fbf24069ac0 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 5 monclient: authenticate success, global_id 224145 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 10 monclient: _renew_subs 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf18009390 con 0x7fbf24069ac0 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14013470 con 0x7fbf24069ac0 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf0bfff640 10 monclient: dump: 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.459 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 10 monclient: _renew_subs 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2413daa0 con 0x7fbf24069ac0 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.443+0000 7fbf227fc640 4 set_mon_vals no callback set 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf2a514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf2a514640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 0x7fbf24141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf24122d60 con 0x7fbf24121730 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbf140137e0 con 0x7fbf24069ac0 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 0x7fbf24141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient: get_auth_request con 0x7fbf24121730 auth_method 0 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf0bfff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf10038740 0x7fbf1003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbf1404cb40 con 0x7fbf24069ac0 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 10 monclient(hunting): handle_auth_done global_id 214232 payload 824 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf237fe640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf240724a0 con 0x7fbf24121730 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf28ab1640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf10038740 0x7fbf1003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.444+0000 7fbf28ab1640 10 monclient: get_auth_request con 0x7fbf10038740 auth_method 0 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf237fe640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 msgr2=0x7fbf24141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 0x7fbf24141960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf240a0c50 tx=0x7fbf18032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1803d980 con 0x7fbf24121730 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.446+0000 7fbf0bfff640 10 monclient: dump: 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbf1803dae0 con 0x7fbf24121730 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 msgr2=0x7fbf24141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 0x7fbf24141960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf240a0c50 tx=0x7fbf18032000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf28ab1640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf10038740 0x7fbf1003abf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009d80 tx=0x7fbf1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 msgr2=0x7fbf24141960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.447+0000 7fbf0bfff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24121730 0x7fbf24141960 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf240a0c50 tx=0x7fbf18032000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf2a514640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf2a514640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 0x7fbf24141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf2a514640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf24063b20 con 0x7fbf24120140 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 0x7fbf24141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient: get_auth_request con 0x7fbf24120140 auth_method 0 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 10 monclient(hunting): handle_auth_done global_id 214235 payload 824 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.453+0000 7fbf237fe640 1 -- 172.21.15.17:0/2752284245 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf24122600 con 0x7fbf24120140 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf237fe640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 msgr2=0x7fbf24141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf237fe640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 0x7fbf24141ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf1803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1803c9d0 con 0x7fbf24120140 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.455+0000 7fbf0bfff640 10 monclient: dump: 2024-09-15T02:09:39.467 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fbf1803cb30 con 0x7fbf24120140 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:39.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 msgr2=0x7fbf24141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 0x7fbf24141ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf1803d000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 msgr2=0x7fbf24141ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf0bfff640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbf24120140 0x7fbf24141ea0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf1803d000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:39.469 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:39.470 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:39.472 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:39.473 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.474 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:39.475 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.476 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.481 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:39.482 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:39.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf10038740 msgr2=0x7fbf1003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbf10038740 0x7fbf1003abf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009d80 tx=0x7fbf1c008040 comp rx=0 tx=0).stop 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 10 monclient: shutdown 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 msgr2=0x7fbf2413deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 --2- 172.21.15.17:0/2752284245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbf24069ac0 0x7fbf2413deb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400ec80 tx=0x7fbf140091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf28ab1640 1 -- 172.21.15.17:0/2752284245 reap_dead start 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf28ab1640 1 -- 172.21.15.17:0/2752284245 reap_dead start 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 shutdown_connections 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 >> 172.21.15.17:0/2752284245 conn(0x7fbf24070e10 msgr2=0x7fbf24098d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 shutdown_connections 2024-09-15T02:09:39.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.456+0000 7fbf2a514640 1 -- 172.21.15.17:0/2752284245 wait complete. 2024-09-15T02:09:39.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:39.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:09:39.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:09:39.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.580+0000 7f6637694640 10 monclient: get_monmap_and_config 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.580+0000 7f6637694640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.580+0000 7f6637694640 10 monclient: monmap: 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:39.581169+0000 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:39.581169+0000 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:39.582 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:39.583 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:39.583 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:39.583 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.580+0000 7f6637694640 1 Processor -- start 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 -- start start 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient: init 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T02:09:39.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6630067090 0x7f6630132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): picked mon.noname-a con 0x7f6630067090 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): picked mon.noname-b con 0x7f663009ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66300679a0 0x7f663012e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): picked mon.noname-c con 0x7f66300679a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663009e4c0 con 0x7f6630067090 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663009cbf0 con 0x7f66300679a0 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663009e1e0 con 0x7f663009ce00 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6637694640 10 monclient(hunting): authenticate will time out at 1839.577627s 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f6630067090 0x7f6630132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57880/0 (socket says 172.21.15.17:57880) 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 1 -- 172.21.15.17:0/2210461042 learned_addr learned my addr 172.21.15.17:0/2210461042 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): get_auth_request con 0x7f663009ce00 auth_method 0 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): get_auth_request con 0x7f6630067090 auth_method 0 2024-09-15T02:09:39.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635c31640 1 --2- 172.21.15.17:0/2210461042 >> v2:172.21.15.67:3300/0 conn(0x7f66300679a0 0x7f663012e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6634c2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635c31640 10 monclient(hunting): get_auth_request con 0x7f66300679a0 auth_method 0 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635c31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.581+0000 7f6635c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 10 monclient(hunting): handle_auth_done global_id 224148 payload 274 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 -- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 msgr2=0x7f6630067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 -- 172.21.15.17:0/2210461042 >> v2:172.21.15.67:3300/0 conn(0x7f66300679a0 msgr2=0x7f663012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 --2- 172.21.15.17:0/2210461042 >> v2:172.21.15.67:3300/0 conn(0x7f66300679a0 0x7f663012e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 monclient: found mon.noname-a 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:09:39.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6635430640 1 -- 172.21.15.17:0/2210461042 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630067e00 con 0x7f6630067090 2024-09-15T02:09:39.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.582+0000 7f6634c2f640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.584+0000 7f6635430640 1 -- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 msgr2=0x7f6630132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.584+0000 7f6635430640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630132140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f663009c970 tx=0x7f662c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8001b92be26fb6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 1 -- 172.21.15.17:0/2210461042 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c007760 con 0x7f6630067090 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: dump: 2024-09-15T02:09:39.597 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 5 monclient: authenticate success, global_id 224148 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 1 -- 172.21.15.17:0/2210461042 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c02d050 con 0x7f6630067090 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 4 set_mon_vals no callback set 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 1 -- 172.21.15.17:0/2210461042 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c007af0 con 0x7f6630067090 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6627fff640 10 monclient: dump: 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.599 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 10 monclient: shutdown 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 -- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 msgr2=0x7f6630132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630132140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f663009c970 tx=0x7f662c009780 comp rx=0 tx=0).stop 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 -- 172.21.15.17:0/2210461042 shutdown_connections 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 --2- 172.21.15.17:0/2210461042 >> v2:172.21.15.67:3300/0 conn(0x7f66300679a0 0x7f663012e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f6630067460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 --2- 172.21.15.17:0/2210461042 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630132140 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 -- 172.21.15.17:0/2210461042 >> 172.21.15.17:0/2210461042 conn(0x7f66300a1ee0 msgr2=0x7f66300a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 -- 172.21.15.17:0/2210461042 shutdown_connections 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.585+0000 7f6637694640 1 -- 172.21.15.17:0/2210461042 wait complete. 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.586+0000 7f6637694640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.586+0000 7f6637694640 10 monclient: monmap: 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.601 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.586+0000 7f6637694640 1 Processor -- start 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.586+0000 7f6637694640 1 -- start start 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.586+0000 7f6637694640 10 monclient: init 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): picked mon.noname-a con 0x7f6630067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66300679a0 0x7f6630130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): picked mon.noname-b con 0x7f66300679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f663011b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): picked mon.noname-c con 0x7f663009ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663011f170 con 0x7f6630067090 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663009d5c0 con 0x7f66300679a0 2024-09-15T02:09:39.602 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6630141d80 con 0x7f663009ce00 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6637694640 10 monclient(hunting): authenticate will time out at 1839.583279s 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f663011b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42524/0 (socket says 172.21.15.17:42524) 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 1 -- 172.21.15.17:0/2853508060 learned_addr learned my addr 172.21.15.17:0/2853508060 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): get_auth_request con 0x7f6630067090 auth_method 0 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): get_auth_request con 0x7f663009ce00 auth_method 0 2024-09-15T02:09:39.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6634c2f640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66300679a0 0x7f6630130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6634c2f640 10 monclient(hunting): get_auth_request con 0x7f66300679a0 auth_method 0 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6634c2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6634c2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6634c2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient(hunting): handle_auth_done global_id 214241 payload 824 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66300679a0 msgr2=0x7f6630130a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66300679a0 0x7f6630130a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 msgr2=0x7f6630140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6630067090 0x7f6630140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 monclient: found mon.noname-c 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.587+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620009450 con 0x7f663009ce00 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.589+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 msgr2=0x7f663011b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.589+0000 7f6635c31640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f663011b540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f662800e900 tx=0x7f66280090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662800cb50 con 0x7f663009ce00 2024-09-15T02:09:39.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: dump: 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662800ccb0 con 0x7f663009ce00 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 5 monclient: authenticate success, global_id 214241 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 10 monclient: _renew_subs 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c009a20 con 0x7f663009ce00 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66280136c0 con 0x7f663009ce00 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 10 monclient: dump: 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.607 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66277fe640 4 set_mon_vals no callback set 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 10 monclient: _renew_subs 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663011dc50 con 0x7f663009ce00 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.590+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6628013a30 con 0x7f663009ce00 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6637694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6637694640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 0x7f66300298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663011f030 con 0x7f6630141460 2024-09-15T02:09:39.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f66257fa640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6614038740 0x7f661403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f662804c6d0 con 0x7f663009ce00 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 0x7f66300298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient: get_auth_request con 0x7f6630141460 auth_method 0 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6634c2f640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6614038740 0x7f661403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6634c2f640 10 monclient: get_auth_request con 0x7f6614038740 auth_method 0 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 10 monclient(hunting): handle_auth_done global_id 204325 payload 824 2024-09-15T02:09:39.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.591+0000 7f6635430640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f663011ecf0 con 0x7f6630141460 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f6634c2f640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6614038740 0x7f661403abf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6620009420 tx=0x7f6620008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f6635430640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 msgr2=0x7f66300298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f6635430640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 0x7f66300298c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f662c0099f0 tx=0x7f662c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c041070 con 0x7f6630141460 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 10 monclient: dump: 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.610 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f662c03c9f0 con 0x7f6630141460 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 msgr2=0x7f66300298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 0x7f66300298c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f662c0099f0 tx=0x7f662c037000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 msgr2=0x7f66300298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.594+0000 7f66257fa640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630141460 0x7f66300298c0 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f662c0099f0 tx=0x7f662c037000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6637694640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6637694640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 0x7f6630029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6637694640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6630063870 con 0x7f6630130fc0 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 0x7f6630029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient: get_auth_request con 0x7f6630130fc0 auth_method 0 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 10 monclient(hunting): handle_auth_done global_id 204328 payload 824 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.600+0000 7f6635430640 1 -- 172.21.15.17:0/2853508060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f66300724a0 con 0x7f6630130fc0 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f6635430640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 msgr2=0x7f6630029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f6635430640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 0x7f6630029e00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f662c045000 tx=0x7f662c041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c031820 con 0x7f6630130fc0 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 10 monclient: dump: 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.614 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f662c031980 con 0x7f6630130fc0 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 msgr2=0x7f6630029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 0x7f6630029e00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f662c045000 tx=0x7f662c041000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.615 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 msgr2=0x7f6630029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f66257fa640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630130fc0 0x7f6630029e00 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f662c045000 tx=0x7f662c041000 comp rx=0 tx=0).stop 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 180, 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:09:39.616 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.617 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.618 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:39.619 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:39.620 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.622 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.623 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:39.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.626 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.627 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:39.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:39.629 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:39.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.603+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6614038740 msgr2=0x7f661403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f6614038740 0x7f661403abf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6620009420 tx=0x7f6620008040 comp rx=0 tx=0).stop 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 10 monclient: shutdown 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 msgr2=0x7f663011b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 --2- 172.21.15.17:0/2853508060 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f663009ce00 0x7f663011b540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f662800e900 tx=0x7f66280090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 reap_dead start 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6635c31640 1 -- 172.21.15.17:0/2853508060 reap_dead start 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 shutdown_connections 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 >> 172.21.15.17:0/2853508060 conn(0x7f66300a1ee0 msgr2=0x7f66300992c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 shutdown_connections 2024-09-15T02:09:39.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.604+0000 7f6637694640 1 -- 172.21.15.17:0/2853508060 wait complete. 2024-09-15T02:09:39.632 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:09:39.632 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T02:09:39.632 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:09:39.632 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T02:09:39.632 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:09:39.632 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:09:39.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:09:39.643 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:09:39.643 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:09:39.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.777+0000 7fc13142b640 10 monclient: get_monmap_and_config 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.777+0000 7fc13142b640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.777+0000 7fc13142b640 10 monclient: monmap: 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:39.778562+0000 2024-09-15T02:09:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:39.778562+0000 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:39.780 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 Processor -- start 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 -- start start 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient: init 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc12c0a5780 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-b con 0x7fc12c09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 0x7fc12c139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-c con 0x7fc12c069c90 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0a5b50 con 0x7fc12c0a5780 2024-09-15T02:09:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0a18b0 con 0x7fc12c069c90 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c09c5f0 con 0x7fc12c09ccf0 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.778+0000 7fc13142b640 10 monclient(hunting): authenticate will time out at 1839.774980s 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57888/0 (socket says 172.21.15.17:57888) 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 -- 172.21.15.17:0/2959168398 learned_addr learned my addr 172.21.15.17:0/2959168398 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): get_auth_request con 0x7fc12c09ccf0 auth_method 0 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 1 --2- 172.21.15.17:0/2959168398 >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 0x7fc12c139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 10 monclient(hunting): get_auth_request con 0x7fc12c069c90 auth_method 0 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient(hunting): handle_auth_done global_id 214244 payload 274 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 -- 172.21.15.17:0/2959168398 >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 msgr2=0x7fc12c139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 --2- 172.21.15.17:0/2959168398 >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 0x7fc12c139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 -- 172.21.15.17:0/2959168398 >> v1:172.21.15.67:6789/0 conn(0x7fc12c0a5780 legacy=0x7fc12c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:39.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 monclient: found mon.noname-b 2024-09-15T02:09:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12a7fc640 1 -- 172.21.15.17:0/2959168398 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c09d210 con 0x7fc12c09ccf0 2024-09-15T02:09:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.779+0000 7fc12b7fe640 1 --2- 172.21.15.17:0/2959168398 >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 0x7fc12c139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:39.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc12a7fc640 1 -- 172.21.15.17:0/2959168398 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 msgr2=0x7fc12c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc12a7fc640 1 --2- 172.21.15.17:0/2959168398 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc120009910 tx=0x7fc1200091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4011e5522d868805 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 1 -- 172.21.15.17:0/2959168398 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12002aea0 con 0x7fc12c09ccf0 2024-09-15T02:09:39.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: dump: 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 1 -- 172.21.15.17:0/2959168398 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc120032440 con 0x7fc12c09ccf0 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 1 -- 172.21.15.17:0/2959168398 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1200327e0 con 0x7fc12c09ccf0 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc13142b640 5 monclient: authenticate success, global_id 214244 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc13142b640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc13142b640 4 set_mon_vals no callback set 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.782+0000 7fc129ffb640 10 monclient: dump: 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.793 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 10 monclient: shutdown 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 -- 172.21.15.17:0/2959168398 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 msgr2=0x7fc12c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 --2- 172.21.15.17:0/2959168398 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc120009910 tx=0x7fc1200091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 -- 172.21.15.17:0/2959168398 shutdown_connections 2024-09-15T02:09:39.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 --2- 172.21.15.17:0/2959168398 >> v2:172.21.15.67:3300/0 conn(0x7fc12c069c90 0x7fc12c139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 --2- 172.21.15.17:0/2959168398 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c09ccf0 0x7fc12c09bc90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 -- 172.21.15.17:0/2959168398 >> 172.21.15.17:0/2959168398 conn(0x7fc12c070e10 msgr2=0x7fc12c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 -- 172.21.15.17:0/2959168398 shutdown_connections 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 1 -- 172.21.15.17:0/2959168398 wait complete. 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 10 monclient: build_initial_monmap 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.783+0000 7fc13142b640 10 monclient: monmap: 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:39.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 Processor -- start 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 -- start start 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient: init 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 0x7fc12c13dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-a con 0x7fc12c069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 0x7fc12c13d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-b con 0x7fc12c09ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): picked mon.noname-c con 0x7fc12c0a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1225a0 con 0x7fc12c069c90 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c120ab0 con 0x7fc12c09ccf0 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1227a0 con 0x7fc12c0a5780 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc12affd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 msgr2=0x7fc12c13dd70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc13142b640 10 monclient(hunting): authenticate will time out at 1839.780900s 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc12affd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 0x7fc12c13dd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.784+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57896/0 (socket says 172.21.15.17:57896) 2024-09-15T02:09:39.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 -- 172.21.15.17:0/1573861673 learned_addr learned my addr 172.21.15.17:0/1573861673 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): get_auth_request con 0x7fc12c0a5780 auth_method 0 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 0x7fc12c13d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 10 monclient(hunting): get_auth_request con 0x7fc12c09ccf0 auth_method 0 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient(hunting): handle_auth_done global_id 214247 payload 824 2024-09-15T02:09:39.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 -- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 msgr2=0x7fc12c13d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 0x7fc12c13d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 -- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 msgr2=0x7fc12c13dd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 0x7fc12c13dd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 monclient: found mon.noname-c 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12b7fe640 1 -- 172.21.15.17:0/1573861673 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009450 con 0x7fc12c0a5780 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.785+0000 7fc12a7fc640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 0x7fc12c13d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc12b7fe640 1 -- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 msgr2=0x7fc12c1412e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc12b7fe640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc12400d100 tx=0x7fc1240099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12400dee0 con 0x7fc12c0a5780 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: dump: 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: _finish_auth 0 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12400f040 con 0x7fc12c0a5780 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1240127b0 con 0x7fc12c0a5780 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 5 monclient: authenticate success, global_id 214247 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 10 monclient: _renew_subs 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c13e890 con 0x7fc12c0a5780 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc1297fa640 4 set_mon_vals no callback set 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc112ffd640 10 monclient: dump: 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:39.801 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 10 monclient: _renew_subs 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.787+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c121770 con 0x7fc12c0a5780 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc124012be0 con 0x7fc12c0a5780 2024-09-15T02:09:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc13142b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc13142b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc13142b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c122c70 con 0x7fc12c0a5780 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc112ffd640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 0x7fc11803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.788+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc12404cc60 con 0x7fc12c0a5780 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.789+0000 7fc12affd640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 0x7fc11803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:39.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.789+0000 7fc12affd640 10 monclient: get_auth_request con 0x7fc1180386f0 auth_method 0 2024-09-15T02:09:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.791+0000 7fc12affd640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 0x7fc11803aba0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc11c002060 tx=0x7fc11c0079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.793+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc1240143a0 con 0x7fc12c0a5780 2024-09-15T02:09:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.793+0000 7fc112ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:39.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:39.793+0000 7fc112ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:40.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc13142b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:40.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc13142b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:40.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc13142b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:40.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc12c0724a0 con 0x7fc12c0a5780 2024-09-15T02:09:40.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc112ffd640 1 -- 172.21.15.17:0/1573861673 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fc12401a020 con 0x7fc12c0a5780 2024-09-15T02:09:40.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc112ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:40.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.017+0000 7fc112ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:40.019 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:40.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 msgr2=0x7fc11803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:40.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 0x7fc11803aba0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc11c002060 tx=0x7fc11c0079e0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 10 monclient: shutdown 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 msgr2=0x7fc12c1412e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc12400d100 tx=0x7fc1240099b0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 shutdown_connections 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc1180386f0 0x7fc11803aba0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc12c0a5780 0x7fc12c1412e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc12c09ccf0 0x7fc12c13d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 --2- 172.21.15.17:0/1573861673 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc12c069c90 0x7fc12c13dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 >> 172.21.15.17:0/1573861673 conn(0x7fc12c070e10 msgr2=0x7fc12c09afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 shutdown_connections 2024-09-15T02:09:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:40.019+0000 7fc13142b640 1 -- 172.21.15.17:0/1573861673 wait complete. 2024-09-15T02:09:43.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.136+0000 7f531dcc9640 10 monclient: get_monmap_and_config 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.136+0000 7f531dcc9640 10 monclient: build_initial_monmap 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.136+0000 7f531dcc9640 10 monclient: monmap: 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:43.137218+0000 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:43.137218+0000 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:43.139 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 Processor -- start 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 -- start start 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient: init 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f531806e920 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f531806ecf0 0x7f5318134a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-a con 0x7f531806ecf0 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f531806f110 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531809c860 con 0x7f531806f110 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531809d4b0 con 0x7f531806e920 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531809d090 con 0x7f531806ecf0 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f5316ffd640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f531806ecf0 msgr2=0x7f5318134a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f531dcc9640 10 monclient(hunting): authenticate will time out at 1843.133886s 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f5317fff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f531806f110 legacy=0x7f5318136fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.137+0000 7f5316ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f531806ecf0 0x7f5318134a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.138+0000 7f53177fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f531806e920 0x7f531812fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:48812/0 (socket says 172.21.15.17:48812) 2024-09-15T02:09:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.138+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 learned_addr learned my addr 172.21.15.17:0/3900252950 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 >> v1:172.21.15.67:6789/0 conn(0x7f531806e920 legacy=0x7f531812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422909296 0 0) 0x7f531809d4b0 con 0x7f531806e920 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53167fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53167fc640 10 monclient(hunting): my global_id is 204337 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.140+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53040034e0 con 0x7f531806e920 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486400095 0 0) 0x7f53040034e0 con 0x7f531806e920 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 >> v2:172.21.15.17:3300/0 conn(0x7f531806ecf0 msgr2=0x7f5318134a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 --2- 172.21.15.17:0/3900252950 >> v2:172.21.15.17:3300/0 conn(0x7f531806ecf0 0x7f5318134a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 >> v1:172.21.15.17:6789/0 conn(0x7f531806f110 legacy=0x7f5318136fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:09:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531809cad0 con 0x7f531806e920 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f53080017e0 con 0x7f531806e920 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: dump: 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 >> v1:172.21.15.67:6789/0 conn(0x7f531806e920 legacy=0x7f531812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f5304003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): picked mon.c con 0x7f53040039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5304004780 0x7f5304004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): picked mon.a con 0x7f5304004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304005250 0x7f530400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): picked mon.b con 0x7f5304005250 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531809d090 con 0x7f5304004780 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530400c1a0 con 0x7f5304005250 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530400c360 con 0x7f53040039d0 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53167fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5316ffd640 1 -- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5304004780 msgr2=0x7f5304004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5316ffd640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5304004780 0x7f5304004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f5304003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:43.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): get_auth_request con 0x7f53040039d0 auth_method 0 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304005250 0x7f530400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): get_auth_request con 0x7f5304005250 auth_method 0 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:43.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient(hunting): handle_auth_done global_id 204337 payload 293 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304005250 msgr2=0x7f530400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304005250 0x7f530400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5304004780 msgr2=0x7f5304004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5304004780 0x7f5304004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 monclient: found mon.c 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040034e0 con 0x7f53040039d0 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f53177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f531dcc9640 5 monclient: authenticate success, global_id 204337 2024-09-15T02:09:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f531dcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:43.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.141+0000 7f5317fff640 1 -- 172.21.15.17:0/3900252950 reap_dead start 2024-09-15T02:09:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53177fe640 1 -- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 msgr2=0x7f5304003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53177fe640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f5304003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f530800c5c0 tx=0x7f5308007b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3269b0c6a82aa02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308005890 con 0x7f53040039d0 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: dump: 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:43.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f531dcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308005a10 con 0x7f53040039d0 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f531dcc9640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f531dcc9640 4 set_mon_vals no callback set 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 1 -- 172.21.15.17:0/3900252950 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308005dd0 con 0x7f53040039d0 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.144+0000 7f53167fc640 10 monclient: dump: 2024-09-15T02:09:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 10 monclient: shutdown 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 -- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 msgr2=0x7f5304003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:43.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f5304003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f530800c5c0 tx=0x7f5308007b30 comp rx=0 tx=0).stop 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 -- 172.21.15.17:0/3900252950 shutdown_connections 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3900252950 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f5304003dc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 -- 172.21.15.17:0/3900252950 >> 172.21.15.17:0/3900252950 conn(0x7f53180a1000 msgr2=0x7f53180a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 -- 172.21.15.17:0/3900252950 shutdown_connections 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 1 -- 172.21.15.17:0/3900252950 wait complete. 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 10 monclient: build_initial_monmap 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.145+0000 7f531dcc9640 10 monclient: monmap: 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 Processor -- start 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 -- start start 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient: init 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-c con 0x7f53040039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 0x7f531812edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-a con 0x7f531813dd10 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f531811e020 0x7f53181410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): picked mon.noname-b con 0x7f531811e020 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531811f1f0 con 0x7f531813dd10 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531811ef10 con 0x7f531811e020 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531811f4d0 con 0x7f53040039d0 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f531dcc9640 10 monclient(hunting): authenticate will time out at 1843.142693s 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5316ffd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 msgr2=0x7f531812edc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5316ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 0x7f531812edc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:43.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60894/0 (socket says 172.21.15.17:60894) 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 1 -- 172.21.15.17:0/3516808490 learned_addr learned my addr 172.21.15.17:0/3516808490 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): get_auth_request con 0x7f53040039d0 auth_method 0 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f53177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5317fff640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f531811e020 0x7f53181410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5317fff640 10 monclient(hunting): get_auth_request con 0x7f531811e020 auth_method 0 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5317fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:43.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.146+0000 7f5317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f5317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 10 monclient(hunting): handle_auth_done global_id 214250 payload 824 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 -- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f531811e020 msgr2=0x7f53181410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f531811e020 0x7f53181410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 -- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 msgr2=0x7f531812edc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 0x7f531812edc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 monclient: found mon.noname-c 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.147+0000 7f53177fe640 1 -- 172.21.15.17:0/3516808490 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531811f7b0 con 0x7f53040039d0 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f53177fe640 1 -- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 msgr2=0x7f53180993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f53177fe640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f530800c620 tx=0x7f5308034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530803e070 con 0x7f53040039d0 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: dump: 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:43.165 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308007300 con 0x7f53040039d0 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 5 monclient: authenticate success, global_id 214250 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 10 monclient: _renew_subs 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53080076a0 con 0x7f53040039d0 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531812f300 con 0x7f53040039d0 2024-09-15T02:09:43.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f530f7fe640 10 monclient: dump: 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f5315ffb640 4 set_mon_vals no callback set 2024-09-15T02:09:43.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 10 monclient: _renew_subs 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.149+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531813e790 con 0x7f53040039d0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f530802c040 con 0x7f53040039d0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f531dcc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f531dcc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f531dcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531812f520 con 0x7f53040039d0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f530f7fe640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 0x7f52f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.150+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f530806bfe0 con 0x7f53040039d0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.151+0000 7f5316ffd640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 0x7f52f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.151+0000 7f5316ffd640 10 monclient: get_auth_request con 0x7f52f80386f0 auth_method 0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.153+0000 7f5316ffd640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 0x7f52f803aba0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5300006fd0 tx=0x7f5300006dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:43.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.155+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5308036b60 con 0x7f53040039d0 2024-09-15T02:09:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.155+0000 7f530f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:43.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.155+0000 7f530f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:43.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.382+0000 7f531dcc9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:43.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.382+0000 7f531dcc9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:43.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.382+0000 7f531dcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:43.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.382+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f53180724a0 con 0x7f53040039d0 2024-09-15T02:09:43.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.383+0000 7f530f7fe640 1 -- 172.21.15.17:0/3516808490 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f530802f070 con 0x7f53040039d0 2024-09-15T02:09:43.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.383+0000 7f530f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:43.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.383+0000 7f530f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:43.385 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 msgr2=0x7f52f803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 0x7f52f803aba0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5300006fd0 tx=0x7f5300006dd0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 10 monclient: shutdown 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 msgr2=0x7f53180993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f530800c620 tx=0x7f5308034000 comp rx=0 tx=0).stop 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 shutdown_connections 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f52f80386f0 0x7f52f803aba0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531813dd10 0x7f531812edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f531811e020 0x7f53181410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 --2- 172.21.15.17:0/3516808490 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f53040039d0 0x7f53180993c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:43.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 >> 172.21.15.17:0/3516808490 conn(0x7f53180a1000 msgr2=0x7f53180a0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:43.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 shutdown_connections 2024-09-15T02:09:43.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:43.385+0000 7f531dcc9640 1 -- 172.21.15.17:0/3516808490 wait complete. 2024-09-15T02:09:46.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.505+0000 7fe5ded9d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.505+0000 7fe5ded9d640 10 monclient: build_initial_monmap 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.505+0000 7fe5ded9d640 10 monclient: monmap: 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:46.507 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:46.506480+0000 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:46.506480+0000 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:46.508 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 1 Processor -- start 2024-09-15T02:09:46.510 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 1 -- start start 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 10 monclient: init 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.506+0000 7fe5ded9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d8067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5d809ce00 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 0x7fe5d8139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d8067b70 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d809e1e0 con 0x7fe5d809ce00 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d809e4c0 con 0x7fe5d8067b70 2024-09-15T02:09:46.511 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d809cbf0 con 0x7fe5d8067090 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5ded9d640 10 monclient(hunting): authenticate will time out at 1846.503059s 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dd33a640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 msgr2=0x7fe5d8139010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dd33a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 0x7fe5d8139010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60906/0 (socket says 172.21.15.17:60906) 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 -- 172.21.15.17:0/2449441811 learned_addr learned my addr 172.21.15.17:0/2449441811 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): get_auth_request con 0x7fe5d8067090 auth_method 0 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:46.512 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:46.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:46.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient(hunting): handle_auth_done global_id 214253 payload 274 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 -- 172.21.15.17:0/2449441811 >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 msgr2=0x7fe5d8139010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 --2- 172.21.15.17:0/2449441811 >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 0x7fe5d8139010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 -- 172.21.15.17:0/2449441811 >> v1:172.21.15.67:6789/0 conn(0x7fe5d809ce00 legacy=0x7fe5d8067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient: _finish_hunting 0 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 monclient: found mon.noname-b 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.507+0000 7fe5dcb39640 1 -- 172.21.15.17:0/2449441811 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d806efb0 con 0x7fe5d8067090 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5dcb39640 1 -- 172.21.15.17:0/2449441811 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 msgr2=0x7fe5d8132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5dcb39640 1 --2- 172.21.15.17:0/2449441811 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009910 tx=0x7fe5cc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=723a38c5cfe378a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 1 -- 172.21.15.17:0/2449441811 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc032440 con 0x7fe5d8067090 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: dump: 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:46.520 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:46.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5ded9d640 5 monclient: authenticate success, global_id 214253 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 1 -- 172.21.15.17:0/2449441811 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc0325a0 con 0x7fe5d8067090 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5ded9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5ded9d640 4 set_mon_vals no callback set 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 1 -- 172.21.15.17:0/2449441811 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc032940 con 0x7fe5d8067090 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.510+0000 7fe5d77fe640 10 monclient: dump: 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:46.522 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:46.524 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 10 monclient: shutdown 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2449441811 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 msgr2=0x7fe5d8132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2449441811 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009910 tx=0x7fe5cc0091d0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2449441811 shutdown_connections 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2449441811 >> v2:172.21.15.17:3300/0 conn(0x7fe5d8067b70 0x7fe5d8139010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2449441811 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067090 0x7fe5d8132140 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2449441811 >> 172.21.15.17:0/2449441811 conn(0x7fe5d80a1ee0 msgr2=0x7fe5d80a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2449441811 shutdown_connections 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2449441811 wait complete. 2024-09-15T02:09:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 10 monclient: build_initial_monmap 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 10 monclient: monmap: 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.511+0000 7fe5ded9d640 1 Processor -- start 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 -- start start 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient: init 2024-09-15T02:09:46.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 0x7fe5d81358d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d8067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d8067b70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 0x7fe5d8136350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d809ce00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d812ec20 con 0x7fe5d8067090 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8118650 con 0x7fe5d809ce00 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8138b30 con 0x7fe5d8067b70 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5ded9d640 10 monclient(hunting): authenticate will time out at 1846.508716s 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5dcb39640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 msgr2=0x7fe5d81358d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5dcb39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 0x7fe5d81358d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60910/0 (socket says 172.21.15.17:60910) 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 1 -- 172.21.15.17:0/2764734969 learned_addr learned my addr 172.21.15.17:0/2764734969 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): get_auth_request con 0x7fe5d8067b70 auth_method 0 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:46.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.512+0000 7fe5d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5dd33a640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 0x7fe5d8136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 10 monclient(hunting): handle_auth_done global_id 214256 payload 824 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 -- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 msgr2=0x7fe5d8136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 0x7fe5d8136350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 -- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 msgr2=0x7fe5d81358d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 0x7fe5d81358d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 monclient: found mon.noname-c 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5d7fff640 1 -- 172.21.15.17:0/2764734969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc009450 con 0x7fe5d8067b70 2024-09-15T02:09:46.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5dd33a640 10 monclient: get_auth_request con 0x7fe5d809ce00 auth_method 0 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.513+0000 7fe5dd33a640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 0x7fe5d8136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d7fff640 1 -- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 msgr2=0x7fe5d8135e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d7fff640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c5e0 tx=0x7fe5c4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4012920 con 0x7fe5d8067b70 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: dump: 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.530 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: _finish_auth 0 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c4012a80 con 0x7fe5d8067b70 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 5 monclient: authenticate success, global_id 214256 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 10 monclient: _renew_subs 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d809d620 con 0x7fe5d8067b70 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4012e40 con 0x7fe5d8067b70 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d4ff9640 10 monclient: dump: 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 10 monclient: _renew_subs 2024-09-15T02:09:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d8137270 con 0x7fe5d8067b70 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.515+0000 7fe5d6ffd640 4 set_mon_vals no callback set 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe5c4011590 con 0x7fe5d8067b70 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5ded9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5ded9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d80724a0 con 0x7fe5d8067b70 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5d4ff9640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 0x7fe5bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.516+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe5c404c820 con 0x7fe5d8067b70 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.517+0000 7fe5dcb39640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 0x7fe5bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:46.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.517+0000 7fe5dcb39640 10 monclient: get_auth_request con 0x7fe5bc038740 auth_method 0 2024-09-15T02:09:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.519+0000 7fe5dcb39640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 0x7fe5bc03abf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8068100 tx=0x7fe5cc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.521+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe5c4010ac0 con 0x7fe5d8067b70 2024-09-15T02:09:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.521+0000 7fe5d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:46.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.521+0000 7fe5d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:46.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.745+0000 7fe5ded9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:46.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.745+0000 7fe5ded9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.745+0000 7fe5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.745+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe5d80689a0 con 0x7fe5d8067b70 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.746+0000 7fe5d4ff9640 1 -- 172.21.15.17:0/2764734969 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fe5c4016020 con 0x7fe5d8067b70 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.746+0000 7fe5d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.746+0000 7fe5d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:46.748 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:46.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 msgr2=0x7fe5bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 0x7fe5bc03abf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8068100 tx=0x7fe5cc032000 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 10 monclient: shutdown 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 msgr2=0x7fe5d8135e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c5e0 tx=0x7fe5c4007ed0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 shutdown_connections 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe5bc038740 0x7fe5bc03abf0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5d809ce00 0x7fe5d8136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe5d8067b70 0x7fe5d8135e10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 --2- 172.21.15.17:0/2764734969 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe5d8067090 0x7fe5d81358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:46.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 >> 172.21.15.17:0/2764734969 conn(0x7fe5d80a1ee0 msgr2=0x7fe5d8097e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:46.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 shutdown_connections 2024-09-15T02:09:46.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:46.748+0000 7fe5ded9d640 1 -- 172.21.15.17:0/2764734969 wait complete. 2024-09-15T02:09:49.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:09:49.869 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.867+0000 7f7ab3500640 10 monclient: get_monmap_and_config 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.867+0000 7f7ab3500640 10 monclient: build_initial_monmap 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.867+0000 7f7ab3500640 10 monclient: monmap: 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:49.869050+0000 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:49.869050+0000 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:49.870 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:49.871 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:49.871 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:49.871 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:49.871 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:49.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.867+0000 7f7ab3500640 1 Processor -- start 2024-09-15T02:09:49.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 1 -- start start 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient: init 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-b con 0x7f7aac0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7aac0980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7aac136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0a01e0 con 0x7f7aac0980a0 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0a1580 con 0x7f7aac136dd0 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:49.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0a5c80 con 0x7f7aac0a58b0 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab0a9b640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f7aac0980a0 legacy=0x7f7aac098470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab3500640 10 monclient(hunting): authenticate will time out at 1849.865691s 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60918/0 (socket says 172.21.15.17:60918) 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 learned_addr learned my addr 172.21.15.17:0/2132331077 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 10 monclient(hunting): get_auth_request con 0x7f7aac0a58b0 auth_method 0 2024-09-15T02:09:49.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:49.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:09:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.868+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.869+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:49.877 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:49.871+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.67:6789/0 conn(0x7f7aac136dd0 legacy=0x7f7aac1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:50.071 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.069+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.17:6789/0 conn(0x7f7aac0980a0 legacy=0x7f7aac098470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:50.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:50.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 10 monclient(hunting): get_auth_request con 0x7f7aac0a58b0 auth_method 0 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:50.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.070+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:09:50.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.470+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.17:6789/0 conn(0x7f7aac0980a0 legacy=0x7f7aac098470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:50.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 10 monclient(hunting): get_auth_request con 0x7f7aac0a58b0 auth_method 0 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:50.471+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:09:51.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.271+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.17:6789/0 conn(0x7f7aac0980a0 legacy=0x7f7aac098470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 10 monclient(hunting): get_auth_request con 0x7f7aac0a58b0 auth_method 0 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:51.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:51.272+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:09:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): tick 2024-09-15T02:09:52.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 monclient(hunting): continuing hunt 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 msgr2=0x7f7aac132040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac0a58b0 0x7f7aac132040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.67:6789/0 conn(0x7f7aac136dd0 legacy=0x7f7aac1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 -- 172.21.15.17:0/2132331077 >> v1:172.21.15.17:6789/0 conn(0x7f7aac0980a0 legacy=0x7f7aac098470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f7a90003710 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f7a900061d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:52.873 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 -- 172.21.15.17:0/2132331077 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0a5c80 con 0x7f7a90003710 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 1 -- 172.21.15.17:0/2132331077 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a90008d40 con 0x7f7a900061d0 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.868+0000 7f7aa2ffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 msgr2=0x7f7a90005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request con 0x7f7a900061d0 auth_method 0 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:52.875 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:52.869+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:53.072 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 msgr2=0x7f7a90005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request con 0x7f7a900061d0 auth_method 0 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:53.073 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.070+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:09:53.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 msgr2=0x7f7a90005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request con 0x7f7a900061d0 auth_method 0 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 msgr2=0x7f7a90008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:09:53.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:53.471+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:09:54.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab129c640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 msgr2=0x7f7a90005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.67:3300/0 conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request con 0x7f7a900061d0 auth_method 0 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.272+0000 7f7ab0a9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 10 monclient(hunting): handle_auth_done global_id 204343 payload 274 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 msgr2=0x7f7a90005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> v2:172.21.15.17:3300/0 conn(0x7f7a90003710 0x7f7a90005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 1 monclient: found mon.noname-c 2024-09-15T02:09:54.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:09:54.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.273+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac1326c0 con 0x7f7a900061d0 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7ab0a9b640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 msgr2=0x7f7a90008680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7ab0a9b640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7a90008680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7a900061a0 tx=0x7f7a9c009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b927aad6d1fa32e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 1 -- 172.21.15.17:0/2132331077 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c02be80 con 0x7f7a900061d0 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: dump: 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.279 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 1 -- 172.21.15.17:0/2132331077 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c02f070 con 0x7f7a900061d0 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 1 -- 172.21.15.17:0/2132331077 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c035630 con 0x7f7a900061d0 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:54.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7aa3fff640 10 monclient: dump: 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7ab3500640 5 monclient: authenticate success, global_id 204343 2024-09-15T02:09:54.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7ab3500640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:54.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.276+0000 7f7ab3500640 4 set_mon_vals no callback set 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 10 monclient: shutdown 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 -- 172.21.15.17:0/2132331077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 msgr2=0x7f7a90008680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7a90008680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7a900061a0 tx=0x7f7a9c009820 comp rx=0 tx=0).stop 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2132331077 reap_dead start 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 -- 172.21.15.17:0/2132331077 shutdown_connections 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2132331077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7a90008680 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 -- 172.21.15.17:0/2132331077 >> 172.21.15.17:0/2132331077 conn(0x7f7aac070ed0 msgr2=0x7f7aac0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 -- 172.21.15.17:0/2132331077 shutdown_connections 2024-09-15T02:09:54.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.277+0000 7f7ab3500640 1 -- 172.21.15.17:0/2132331077 wait complete. 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.278+0000 7f7ab3500640 10 monclient: build_initial_monmap 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.278+0000 7f7ab3500640 10 monclient: monmap: 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:54.287 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.278+0000 7f7ab3500640 1 Processor -- start 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 -- start start 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient: init 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7aac136600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-b con 0x7f7a900061d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 0x7f7aac09e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-a con 0x7f7aac09dc90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): picked mon.noname-c con 0x7f7aac09e600 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac098e30 con 0x7f7aac09dc90 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac135540 con 0x7f7a900061d0 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac134880 con 0x7f7aac09e600 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab3500640 10 monclient(hunting): authenticate will time out at 1854.276892s 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab0a9b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 msgr2=0x7f7aac09e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.279+0000 7f7ab0a9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 0x7f7aac09e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:50180/0 (socket says 172.21.15.17:50180) 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2520780637 learned_addr learned my addr 172.21.15.17:0/2520780637 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): get_auth_request con 0x7f7aac09e600 auth_method 0 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7aac136600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): get_auth_request con 0x7f7a900061d0 auth_method 0 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab129c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient(hunting): handle_auth_done global_id 214259 payload 824 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 msgr2=0x7f7aac136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7aac136600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 msgr2=0x7f7aac09e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 0x7f7aac09e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 monclient: found mon.noname-c 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.280+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2520780637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c009ac0 con 0x7f7aac09e600 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.282+0000 7f7ab1a9d640 1 -- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 msgr2=0x7f7aac1479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.282+0000 7f7ab1a9d640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006140 tx=0x7f7aa4007ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa400ca40 con 0x7f7aac09e600 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: dump: 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa400cbc0 con 0x7f7aac09e600 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 5 monclient: authenticate success, global_id 214259 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 10 monclient: _renew_subs 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac13a270 con 0x7f7aac09e600 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40070d0 con 0x7f7aac09e600 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa17fa640 10 monclient: dump: 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7aa37fe640 4 set_mon_vals no callback set 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 10 monclient: _renew_subs 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.283+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac09ea70 con 0x7f7aac09e600 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7aa4007540 con 0x7f7aac09e600 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab3500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab3500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab3500640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac134bc0 con 0x7f7aac09e600 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7aa17fa640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 0x7f7a9003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f7aa404b3e0 con 0x7f7aac09e600 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 0x7f7a9003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.284+0000 7f7ab129c640 10 monclient: get_auth_request con 0x7f7a90039460 auth_method 0 2024-09-15T02:09:54.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.287+0000 7f7ab129c640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 0x7f7a9003b910 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006fd0 tx=0x7f7aa8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.289+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7aac134bc0 con 0x7f7aac09e600 2024-09-15T02:09:54.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.289+0000 7f7aa17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.289+0000 7f7aa17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7ab3500640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7ab3500640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7ab3500640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7aac134bc0 con 0x7f7aac09e600 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7aa17fa640 1 -- 172.21.15.17:0/2520780637 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f7aac134bc0 con 0x7f7aac09e600 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7aa17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:09:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.517+0000 7f7aa17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:54.521 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":184,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 msgr2=0x7f7a9003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 0x7f7a9003b910 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006fd0 tx=0x7f7aa8008040 comp rx=0 tx=0).stop 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 10 monclient: shutdown 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 msgr2=0x7f7aac1479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006140 tx=0x7f7aa4007ee0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.522 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 shutdown_connections 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a90039460 0x7f7a9003b910 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7aac09dc90 0x7f7aac09e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7aac09e600 0x7f7aac1479f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 --2- 172.21.15.17:0/2520780637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a900061d0 0x7f7aac136600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 >> 172.21.15.17:0/2520780637 conn(0x7f7aac070ed0 msgr2=0x7f7aac0a3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 shutdown_connections 2024-09-15T02:09:54.523 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.519+0000 7f7ab3500640 1 -- 172.21.15.17:0/2520780637 wait complete. 2024-09-15T02:09:54.532 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:09:54.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:09:54.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.627+0000 7f852c99d640 10 monclient: get_monmap_and_config 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.627+0000 7f852c99d640 10 monclient: build_initial_monmap 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.627+0000 7f852c99d640 10 monclient: monmap: 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:54.629329+0000 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:54.629329+0000 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.630 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:54.631 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:54.631 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:54.631 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:54.631 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:54.631 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 Processor -- start 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 -- start start 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient: init 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f852809c300 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f852809e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f852806eb30 0x7f852806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-c con 0x7f852806eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:09:54.633 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8528099f30 con 0x7f852809e970 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852809ed70 con 0x7f852809c300 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852806a300 con 0x7f852806eb30 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f852c99d640 10 monclient(hunting): authenticate will time out at 1854.625821s 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.628+0000 7f8525d9b640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f852809e970 legacy=0x7f852809a920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f852806eb30 0x7f852806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f852806eb30 0x7f852806efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:42606/0 (socket says 172.21.15.17:42606) 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f852659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f852809c300 0x7f852809c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:60140/0 (socket says 172.21.15.17:60140) 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 -- 172.21.15.17:0/1374898382 learned_addr learned my addr 172.21.15.17:0/1374898382 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): get_auth_request con 0x7f852806eb30 auth_method 0 2024-09-15T02:09:54.634 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient(hunting): handle_auth_done global_id 204349 payload 274 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 -- 172.21.15.17:0/1374898382 >> v1:172.21.15.67:6789/0 conn(0x7f852809c300 legacy=0x7f852809c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 -- 172.21.15.17:0/1374898382 >> v1:172.21.15.17:6789/0 conn(0x7f852809e970 legacy=0x7f852809a920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 monclient: found mon.noname-c 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:09:54.635 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.629+0000 7f8526d9d640 1 -- 172.21.15.17:0/1374898382 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852806a5c0 con 0x7f852806eb30 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f8526d9d640 1 -- 172.21.15.17:0/1374898382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852806eb30 msgr2=0x7f852806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f8526d9d640 1 --2- 172.21.15.17:0/1374898382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852806eb30 0x7f852806efd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8520009f20 tx=0x7f8520009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecfe3cfa9ab5953 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 1 -- 172.21.15.17:0/1374898382 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852002aea0 con 0x7f852806eb30 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: dump: 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.642 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 1 -- 172.21.15.17:0/1374898382 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852002d050 con 0x7f852806eb30 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.643 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 1 -- 172.21.15.17:0/1374898382 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520007840 con 0x7f852806eb30 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852c99d640 5 monclient: authenticate success, global_id 204349 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852c99d640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852c99d640 4 set_mon_vals no callback set 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.632+0000 7f852559a640 10 monclient: dump: 2024-09-15T02:09:54.644 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 10 monclient: shutdown 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 -- 172.21.15.17:0/1374898382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852806eb30 msgr2=0x7f852806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 --2- 172.21.15.17:0/1374898382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852806eb30 0x7f852806efd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8520009f20 tx=0x7f8520009840 comp rx=0 tx=0).stop 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 -- 172.21.15.17:0/1374898382 shutdown_connections 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 --2- 172.21.15.17:0/1374898382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852806eb30 0x7f852806efd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 -- 172.21.15.17:0/1374898382 >> 172.21.15.17:0/1374898382 conn(0x7f85280a2f70 msgr2=0x7f85280a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 -- 172.21.15.17:0/1374898382 shutdown_connections 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 1 -- 172.21.15.17:0/1374898382 wait complete. 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 10 monclient: build_initial_monmap 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.633+0000 7f852c99d640 10 monclient: monmap: 2024-09-15T02:09:54.646 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 Processor -- start 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 -- start start 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient: init 2024-09-15T02:09:54.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 0x7f8528133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-c con 0x7f852806eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852809c300 0x7f852811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-b con 0x7f852809c300 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f852809e970 0x7f852811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): picked mon.noname-a con 0x7f852809e970 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852811fe30 con 0x7f852809e970 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8528141e30 con 0x7f852809c300 2024-09-15T02:09:54.648 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8528120cb0 con 0x7f852806eb30 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852c99d640 10 monclient(hunting): authenticate will time out at 1854.631816s 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f8526d9d640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f852809e970 msgr2=0x7f852811df20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f8526d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f852809e970 0x7f852811df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:54.649 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 0x7f8528133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 0x7f8528133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:50186/0 (socket says 172.21.15.17:50186) 2024-09-15T02:09:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 1 -- 172.21.15.17:0/2873473498 learned_addr learned my addr 172.21.15.17:0/2873473498 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 10 monclient(hunting): get_auth_request con 0x7f852806eb30 auth_method 0 2024-09-15T02:09:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.634+0000 7f852659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852809c300 0x7f852811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): get_auth_request con 0x7f852809c300 auth_method 0 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient(hunting): handle_auth_done global_id 214262 payload 824 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852809c300 msgr2=0x7f852811d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f852809c300 0x7f852811d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f852809e970 msgr2=0x7f852811df20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:54.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f852809e970 0x7f852811df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 monclient: found mon.noname-c 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.635+0000 7f852659c640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8520009ae0 con 0x7f852806eb30 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852659c640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 msgr2=0x7f8528133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852659c640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 0x7f8528133b90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f851800c5b0 tx=0x7f8518007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8518007510 con 0x7f852806eb30 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: dump: 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.654 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 5 monclient: authenticate success, global_id 214262 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85180057d0 con 0x7f852806eb30 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 10 monclient: _renew_subs 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8528120f90 con 0x7f852806eb30 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8518007690 con 0x7f852806eb30 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f85167fc640 10 monclient: dump: 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.656 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f8524d99640 4 set_mon_vals no callback set 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 10 monclient: _renew_subs 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.637+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8528136490 con 0x7f852806eb30 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f852c99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f852c99d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 0x7f852811e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8528063b90 con 0x7f8528134930 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 0x7f852811e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient: get_auth_request con 0x7f8528134930 auth_method 0 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f851801b400 con 0x7f852806eb30 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 10 monclient(hunting): handle_auth_done global_id 214265 payload 824 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f8525d9b640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f852811f2d0 con 0x7f8528134930 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.638+0000 7f85167fc640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8508038740 0x7f850803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.639+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f851804c010 con 0x7f852806eb30 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.639+0000 7f8526d9d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8508038740 0x7f850803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.639+0000 7f8526d9d640 10 monclient: get_auth_request con 0x7f8508038740 auth_method 0 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f8525d9b640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 msgr2=0x7f852811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f8525d9b640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 0x7f852811e460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8528099a00 tx=0x7f851c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c0128f0 con 0x7f8528134930 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 10 monclient: dump: 2024-09-15T02:09:54.661 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f851c012a50 con 0x7f8528134930 2024-09-15T02:09:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 msgr2=0x7f852811e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 0x7f852811e460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8528099a00 tx=0x7f851c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 msgr2=0x7f852811e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f85167fc640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8528134930 0x7f852811e460 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8528099a00 tx=0x7f851c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.641+0000 7f8526d9d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8508038740 0x7f850803abf0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f852809f470 tx=0x7f8520036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f852c99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f852c99d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 0x7f852811e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f852c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85280724a0 con 0x7f85281340d0 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 0x7f852811e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient: get_auth_request con 0x7f85281340d0 auth_method 0 2024-09-15T02:09:54.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 10 monclient(hunting): handle_auth_done global_id 214268 payload 824 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.647+0000 7f8525d9b640 1 -- 172.21.15.17:0/2873473498 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8528067220 con 0x7f85281340d0 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f8525d9b640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 msgr2=0x7f852811e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f8525d9b640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 0x7f852811e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c025070 con 0x7f85281340d0 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 10 monclient: dump: 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.665 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f851c0202c0 con 0x7f85281340d0 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 msgr2=0x7f852811e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 0x7f852811e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c015000 comp rx=0 tx=0).stop 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 msgr2=0x7f852811e9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:54.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f85167fc640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f85281340d0 0x7f852811e9a0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c015000 comp rx=0 tx=0).stop 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 184, 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:54.681 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:54.682 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8508038740 msgr2=0x7f850803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8508038740 0x7f850803abf0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f852809f470 tx=0x7f8520036000 comp rx=0 tx=0).stop 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 10 monclient: shutdown 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 msgr2=0x7f8528133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f852c99d640 1 --2- 172.21.15.17:0/2873473498 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f852806eb30 0x7f8528133b90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f851800c5b0 tx=0x7f8518007f00 comp rx=0 tx=0).stop 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f8526d9d640 1 -- 172.21.15.17:0/2873473498 reap_dead start 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.650+0000 7f8526d9d640 1 -- 172.21.15.17:0/2873473498 reap_dead start 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.651+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 shutdown_connections 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.651+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 >> 172.21.15.17:0/2873473498 conn(0x7f85280a2f70 msgr2=0x7f85280a1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.651+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 shutdown_connections 2024-09-15T02:09:54.683 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.651+0000 7f852c99d640 1 -- 172.21.15.17:0/2873473498 wait complete. 2024-09-15T02:09:54.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:09:54.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:09:54.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.778+0000 7f23a206f640 10 monclient: get_monmap_and_config 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.778+0000 7f23a206f640 10 monclient: build_initial_monmap 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.778+0000 7f23a206f640 10 monclient: monmap: 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:09:54.780415+0000 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:09:54.780415+0000 2024-09-15T02:09:54.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:09:54.782 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 1 Processor -- start 2024-09-15T02:09:54.783 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 1 -- start start 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient: init 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f239c06e920 0x7f239c12fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-a con 0x7f239c06e920 addr v2:172.21.15.17:3300/0 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f239c06ecf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.779+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f239c06f0f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c09aef0 con 0x7f239c06f0f0 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c06f5b0 con 0x7f239c06ecf0 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.784 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c097a70 con 0x7f239c06e920 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f23a206f640 10 monclient(hunting): authenticate will time out at 1854.777051s 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f239b7fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f239c06e920 msgr2=0x7f239c12fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f239bfff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f239c06f0f0 legacy=0x7f239c097360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f239b7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f239c06e920 0x7f239c12fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f239affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f239c06ecf0 0x7f239c096c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:60154/0 (socket says 172.21.15.17:60154) 2024-09-15T02:09:54.785 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.780+0000 7f239affd640 1 -- 172.21.15.17:0/1648469556 learned_addr learned my addr 172.21.15.17:0/1648469556 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.782+0000 7f239affd640 1 -- 172.21.15.17:0/1648469556 >> v1:172.21.15.67:6789/0 conn(0x7f239c06ecf0 legacy=0x7f239c096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.782+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23819219 0 0) 0x7f239c06f5b0 con 0x7f239c06ecf0 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): my global_id is 204355 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23880034e0 con 0x7f239c06ecf0 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1691032935 0 0) 0x7f23880034e0 con 0x7f239c06ecf0 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 >> v2:172.21.15.17:3300/0 conn(0x7f239c06e920 msgr2=0x7f239c12fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:54.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 --2- 172.21.15.17:0/1648469556 >> v2:172.21.15.17:3300/0 conn(0x7f239c06e920 0x7f239c12fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 >> v1:172.21.15.17:6789/0 conn(0x7f239c06f0f0 legacy=0x7f239c097360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c09d720 con 0x7f239c06ecf0 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f23a206f640 5 monclient: authenticate success, global_id 204355 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f2384001720 con 0x7f239c06ecf0 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f23a206f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:09:54.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: dump: 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 >> v1:172.21.15.67:6789/0 conn(0x7f239c06ecf0 legacy=0x7f239c096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 0x7f2388003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): picked mon.c con 0x7f23880039d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2388004780 0x7f2388004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): picked mon.a con 0x7f2388004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2388005230 0x7f238800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): picked mon.b con 0x7f2388005230 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c097a70 con 0x7f2388004780 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238800c1f0 con 0x7f2388005230 2024-09-15T02:09:54.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238800c3b0 con 0x7f23880039d0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239affd640 1 -- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2388004780 msgr2=0x7f2388004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f23a206f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239affd640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2388004780 0x7f2388004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 0x7f2388003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): get_auth_request con 0x7f23880039d0 auth_method 0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2388005230 0x7f238800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): get_auth_request con 0x7f2388005230 auth_method 0 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient(hunting): handle_auth_done global_id 204355 payload 293 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 -- 172.21.15.17:0/1648469556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2388005230 msgr2=0x7f238800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2388005230 0x7f238800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 -- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2388004780 msgr2=0x7f2388004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2388004780 0x7f2388004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 monclient: found mon.c 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 1 -- 172.21.15.17:0/1648469556 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23880034e0 con 0x7f23880039d0 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.783+0000 7f239bfff640 1 -- 172.21.15.17:0/1648469556 reap_dead start 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239b7fe640 1 -- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 msgr2=0x7f2388003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239b7fe640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 0x7f2388003dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f238c0099c0 tx=0x7f238c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a60674864b80f839 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c003b80 con 0x7f23880039d0 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: dump: 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f23a206f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c003d20 con 0x7f23880039d0 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f23a206f640 10 monclient: get_monmap_and_config success 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f23a206f640 4 set_mon_vals no callback set 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 1 -- 172.21.15.17:0/1648469556 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c02ab80 con 0x7f23880039d0 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.801 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.786+0000 7f239a7fc640 10 monclient: dump: 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 10 monclient: shutdown 2024-09-15T02:09:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 -- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 msgr2=0x7f2388003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 0x7f2388003dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f238c0099c0 tx=0x7f238c009060 comp rx=0 tx=0).stop 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 -- 172.21.15.17:0/1648469556 shutdown_connections 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 --2- 172.21.15.17:0/1648469556 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f23880039d0 0x7f2388003dc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 -- 172.21.15.17:0/1648469556 >> 172.21.15.17:0/1648469556 conn(0x7f239c0a1000 msgr2=0x7f239c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 -- 172.21.15.17:0/1648469556 shutdown_connections 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 1 -- 172.21.15.17:0/1648469556 wait complete. 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 10 monclient: build_initial_monmap 2024-09-15T02:09:54.803 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.787+0000 7f23a206f640 10 monclient: monmap: 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 Processor -- start 2024-09-15T02:09:54.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 -- start start 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient: init 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient: _reopen_session rank -1 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23880039d0 0x7f239c13ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-b con 0x7f23880039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-c con 0x7f239c12eca0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f239c1187e0 0x7f239c118bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): picked mon.noname-a con 0x7f239c1187e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c11c0a0 con 0x7f239c1187e0 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.805 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c12f5b0 con 0x7f23880039d0 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c09bbb0 con 0x7f239c12eca0 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): _renew_subs 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f23a206f640 10 monclient(hunting): authenticate will time out at 1854.785785s 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239bfff640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f239c1187e0 msgr2=0x7f239c118bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239bfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f239c1187e0 0x7f239c118bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:50220/0 (socket says 172.21.15.17:50220) 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 1 -- 172.21.15.17:0/2105635236 learned_addr learned my addr 172.21.15.17:0/2105635236 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): get_auth_request con 0x7f239c12eca0 auth_method 0 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.806 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.788+0000 7f239b7fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23880039d0 0x7f239c13ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): get_auth_request con 0x7f23880039d0 auth_method 0 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 10 monclient(hunting): handle_auth_done global_id 214271 payload 824 2024-09-15T02:09:54.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23880039d0 msgr2=0x7f239c13ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23880039d0 0x7f239c13ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f239c1187e0 msgr2=0x7f239c118bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f239c1187e0 0x7f239c118bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 10 monclient: _finish_hunting 0 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 monclient: found mon.noname-c 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.789+0000 7f239affd640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0a02c0 con 0x7f239c12eca0 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f239affd640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 msgr2=0x7f239c12f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f239affd640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f238400c840 tx=0x7f2384001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384007570 con 0x7f239c12eca0 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f239affd640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f238400c840 tx=0x7f2384001150 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f23840076f0 config(0 keys) v1 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 10 monclient: dump: 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.813 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 10 monclient: _finish_auth 0 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23840076f0 con 0x7f239c12eca0 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 5 monclient: authenticate success, global_id 214271 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 10 monclient: _renew_subs 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.814 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c11aec0 con 0x7f239c12eca0 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 10 monclient: _renew_subs 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f2399ffb640 4 set_mon_vals no callback set 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.791+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c13fd40 con 0x7f239c12eca0 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23a206f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23a206f640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 0x7f239c06ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c063c20 con 0x7f239c119130 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840049f0 con 0x7f239c12eca0 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 10 monclient: dump: 2024-09-15T02:09:54.815 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f238400a5e0 con 0x7f239c12eca0 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 0x7f239c06ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient: get_auth_request con 0x7f239c119130 auth_method 0 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.792+0000 7f23937fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f237c0386f0 0x7f237c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.793+0000 7f239b7fe640 10 monclient(hunting): handle_auth_done global_id 204361 payload 824 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.793+0000 7f239b7fe640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f239c11c2c0 con 0x7f239c119130 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.793+0000 7f239bfff640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f237c0386f0 0x7f237c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.793+0000 7f239bfff640 10 monclient: get_auth_request con 0x7f237c0386f0 auth_method 0 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f239b7fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 msgr2=0x7f239c06ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f239b7fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 0x7f239c06ad50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f238c032000 tx=0x7f238c030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c03a040 con 0x7f239c119130 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 10 monclient: dump: 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.818 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f239bfff640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f237c0386f0 0x7f237c03aba0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f239c06f0c0 tx=0x7f23940095d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f238c005490 con 0x7f239c119130 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 msgr2=0x7f239c06ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 0x7f239c06ad50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f238c032000 tx=0x7f238c030000 comp rx=0 tx=0).stop 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 msgr2=0x7f239c06ad50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:54.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.795+0000 7f23937fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c119130 0x7f239c06ad50 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f238c032000 tx=0x7f238c030000 comp rx=0 tx=0).stop 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.801+0000 7f23a206f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.801+0000 7f23a206f640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 0x7f239c06b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.801+0000 7f23a206f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.801+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0724a0 con 0x7f239c06b290 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 0x7f239c06b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:09:54.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient: get_auth_request con 0x7f239c06b290 auth_method 0 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 10 monclient(hunting): handle_auth_done global_id 204364 payload 824 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.802+0000 7f239b7fe640 1 -- 172.21.15.17:0/2105635236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f239c067290 con 0x7f239c06b290 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.804+0000 7f239b7fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 msgr2=0x7f239c06b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:09:54.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.804+0000 7f239b7fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 0x7f239c06b690 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f239c09bd90 tx=0x7f238c037070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.804+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c03e810 con 0x7f239c06b290 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 10 monclient: dump: 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:09:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f238c03e970 con 0x7f239c06b290 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 msgr2=0x7f239c06b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 0x7f239c06b690 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f239c09bd90 tx=0x7f238c037070 comp rx=0 tx=0).stop 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 msgr2=0x7f239c06b690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:09:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239c06b290 0x7f239c06b690 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f239c09bd90 tx=0x7f238c037070 comp rx=0 tx=0).stop 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 184, 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:09:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:09:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:09:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:09:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:09:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:09:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:09:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:09:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:09:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:09:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f237c0386f0 msgr2=0x7f237c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f237c0386f0 0x7f237c03aba0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f239c06f0c0 tx=0x7f23940095d0 comp rx=0 tx=0).stop 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 10 monclient: shutdown 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 msgr2=0x7f239c12f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 --2- 172.21.15.17:0/2105635236 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f239c12eca0 0x7f239c12f070 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f238400c840 tx=0x7f2384001150 comp rx=0 tx=0).stop 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f239bfff640 1 -- 172.21.15.17:0/2105635236 reap_dead start 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23937fe640 1 -- 172.21.15.17:0/2105635236 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f238404bf80 con 0x7f239c12eca0 2024-09-15T02:09:54.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f239bfff640 1 -- 172.21.15.17:0/2105635236 reap_dead start 2024-09-15T02:09:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 shutdown_connections 2024-09-15T02:09:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 >> 172.21.15.17:0/2105635236 conn(0x7f239c0a1000 msgr2=0x7f239c1328b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:09:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 shutdown_connections 2024-09-15T02:09:54.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:09:54.805+0000 7f23a206f640 1 -- 172.21.15.17:0/2105635236 wait complete. 2024-09-15T02:09:54.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:09:54.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:10:14.841 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:10:14.841 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T02:10:14.841 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T02:10:14.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T02:10:14.844 INFO:tasks.ceph.mon.a:Started 2024-09-15T02:10:14.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:10:14.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 10 monclient: get_monmap_and_config 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 10 monclient: build_initial_monmap 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 10 monclient: monmap: 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:14.964 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:14.963029+0000 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:14.963029+0000 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 1 Processor -- start 2024-09-15T02:10:14.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 1 -- start start 2024-09-15T02:10:14.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.962+0000 7fdc44494640 10 monclient: init 2024-09-15T02:10:14.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:14.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T02:10:14.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c131b20 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc3c0a0280 0x7fdc3c131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c0a0280 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c132430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3c0a0040 con 0x7fdc3c131b20 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3c09bff0 con 0x7fdc3c0a0280 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c0a14a0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc44494640 10 monclient(hunting): authenticate will time out at 1874.959578s 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46392/0 (socket says 172.21.15.17:46392) 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 -- 172.21.15.17:0/3486316201 learned_addr learned my addr 172.21.15.17:0/3486316201 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 10 monclient(hunting): get_auth_request con 0x7fdc3c131b20 auth_method 0 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): get_auth_request con 0x7fdc3c09f1e0 auth_method 0 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc41a2f640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.67:3300/0 conn(0x7fdc3c0a0280 0x7fdc3c131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 -- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 msgr2=0x7fdc3c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 -- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 msgr2=0x7fdc3c09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:14.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc41a2f640 10 monclient(hunting): get_auth_request con 0x7fdc3c0a0280 auth_method 0 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc41a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc41a2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc41a2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.963+0000 7fdc42230640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc41a2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc41a2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc41a2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 10 monclient(hunting): handle_auth_done global_id 214274 payload 274 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 -- 172.21.15.17:0/3486316201 >> v2:172.21.15.67:3300/0 conn(0x7fdc3c0a0280 msgr2=0x7fdc3c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.67:3300/0 conn(0x7fdc3c0a0280 0x7fdc3c131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 -- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 msgr2=0x7fdc3c09eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 10 monclient: _finish_hunting 0 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 monclient: found mon.noname-b 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.964+0000 7fdc42a31640 1 -- 172.21.15.17:0/3486316201 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c068e80 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.966+0000 7fdc42a31640 1 -- 172.21.15.17:0/3486316201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 msgr2=0x7fdc3c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.966+0000 7fdc42a31640 1 --2- 172.21.15.17:0/3486316201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c132430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0a5010 tx=0x7fdc34009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a96cb7679b49b565 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 1 -- 172.21.15.17:0/3486316201 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34007760 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: dump: 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:14.978 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: _finish_auth 0 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 1 -- 172.21.15.17:0/3486316201 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3402d050 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 5 monclient: authenticate success, global_id 214274 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 1 -- 172.21.15.17:0/3486316201 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34007af0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc4122e640 10 monclient: dump: 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:14.980 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 4 set_mon_vals no callback set 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 10 monclient: shutdown 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 -- 172.21.15.17:0/3486316201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 msgr2=0x7fdc3c132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 --2- 172.21.15.17:0/3486316201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c132430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0a5010 tx=0x7fdc34009780 comp rx=0 tx=0).stop 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 -- 172.21.15.17:0/3486316201 shutdown_connections 2024-09-15T02:10:14.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.67:3300/0 conn(0x7fdc3c0a0280 0x7fdc3c131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 --2- 172.21.15.17:0/3486316201 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c132430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 --2- 172.21.15.17:0/3486316201 >> v2:172.21.15.17:3300/0 conn(0x7fdc3c131b20 0x7fdc3c09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 -- 172.21.15.17:0/3486316201 >> 172.21.15.17:0/3486316201 conn(0x7fdc3c071270 msgr2=0x7fdc3c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 -- 172.21.15.17:0/3486316201 shutdown_connections 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.967+0000 7fdc44494640 1 -- 172.21.15.17:0/3486316201 wait complete. 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.968+0000 7fdc44494640 10 monclient: build_initial_monmap 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.968+0000 7fdc44494640 10 monclient: monmap: 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:14.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.968+0000 7fdc44494640 1 Processor -- start 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.968+0000 7fdc44494640 1 -- start start 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.968+0000 7fdc44494640 10 monclient: init 2024-09-15T02:10:14.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 0x7fdc3c141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c09d330 con 0x7fdc3c0a0280 2024-09-15T02:10:14.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c0a09b0 con 0x7fdc3c131b20 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c11ea40 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc44494640 10 monclient(hunting): authenticate will time out at 1874.965162s 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60330/0 (socket says 172.21.15.17:60330) 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46394/0 (socket says 172.21.15.17:46394) 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 -- 172.21.15.17:0/613163809 learned_addr learned my addr 172.21.15.17:0/613163809 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): get_auth_request con 0x7fdc3c09f1e0 auth_method 0 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 10 monclient(hunting): get_auth_request con 0x7fdc3c0a0280 auth_method 0 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 0x7fdc3c141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 msgr2=0x7fdc3c141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 msgr2=0x7fdc3c141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 10 monclient(hunting): get_auth_request con 0x7fdc3c131b20 auth_method 0 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc41a2f640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient(hunting): handle_auth_done global_id 214277 payload 824 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 msgr2=0x7fdc3c141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 0x7fdc3c141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 msgr2=0x7fdc3c141520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient: _finish_hunting 0 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 monclient: found mon.noname-c 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:14.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42230640 1 -- 172.21.15.17:0/613163809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc34009a20 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.969+0000 7fdc42a31640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 0x7fdc3c141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:10:14.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc42230640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 msgr2=0x7fdc3c140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc42230640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800e870 tx=0x7fdc38008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800c6d0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 10 monclient: dump: 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 10 monclient: _finish_auth 0 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3800c830 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc44494640 5 monclient: authenticate success, global_id 214277 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc44494640 10 monclient: _renew_subs 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800cbf0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc44494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c11e7c0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc327fc640 10 monclient: dump: 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.971+0000 7fdc40a2d640 4 set_mon_vals no callback set 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 10 monclient: _renew_subs 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c11de70 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdc38013410 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.972+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c0724a0 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.973+0000 7fdc327fc640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 0x7fdc2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:14.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.973+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fdc3804c550 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.973+0000 7fdc41a2f640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 0x7fdc2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.973+0000 7fdc41a2f640 10 monclient: get_auth_request con 0x7fdc20038740 auth_method 0 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.976+0000 7fdc41a2f640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 0x7fdc2003abf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c11da70 tx=0x7fdc2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.977+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdc38023300 con 0x7fdc3c09f1e0 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.977+0000 7fdc327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:14.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:14.977+0000 7fdc327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:15.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc44494640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:15.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc44494640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:15.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc44494640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:15.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc3c06efc0 con 0x7fdc3c09f1e0 2024-09-15T02:10:15.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc327fc640 1 -- 172.21.15.17:0/613163809 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7fdc38023300 con 0x7fdc3c09f1e0 2024-09-15T02:10:15.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:15.208 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.204+0000 7fdc327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:15.208 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":184,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:15.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 msgr2=0x7fdc2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:15.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 0x7fdc2003abf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c11da70 tx=0x7fdc2c008040 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 10 monclient: shutdown 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 msgr2=0x7fdc3c140fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800e870 tx=0x7fdc38008040 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 shutdown_connections 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fdc20038740 0x7fdc2003abf0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fdc3c0a0280 0x7fdc3c141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3c131b20 0x7fdc3c141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 --2- 172.21.15.17:0/613163809 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fdc3c09f1e0 0x7fdc3c140fe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 >> 172.21.15.17:0/613163809 conn(0x7fdc3c071270 msgr2=0x7fdc3c130920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 shutdown_connections 2024-09-15T02:10:15.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:15.206+0000 7fdc44494640 1 -- 172.21.15.17:0/613163809 wait complete. 2024-09-15T02:10:18.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:18.327 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.326+0000 7f09a90e8640 10 monclient: get_monmap_and_config 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.326+0000 7f09a90e8640 10 monclient: build_initial_monmap 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.326+0000 7f09a90e8640 10 monclient: monmap: 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:18.327304+0000 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:18.327304+0000 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:18.328 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:18.329 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:18.329 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:18.329 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:18.329 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 Processor -- start 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 -- start start 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient: init 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:10:18.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-b con 0x7f09a409c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09a409c580 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-a con 0x7f09a409c990 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a409f790 con 0x7f09a409c580 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a4069d40 con 0x7f09a409c990 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4068b60 con 0x7f09a409c1b0 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a90e8640 10 monclient(hunting): authenticate will time out at 1878.323713s 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46400/0 (socket says 172.21.15.17:46400) 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60380/0 (socket says 172.21.15.17:60380) 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 1 -- 172.21.15.17:0/1099898602 learned_addr learned my addr 172.21.15.17:0/1099898602 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 10 monclient(hunting): get_auth_request con 0x7f09a409c990 auth_method 0 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 10 monclient(hunting): get_auth_request con 0x7f09a409c1b0 auth_method 0 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.327+0000 7f09a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a359e640 1 -- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 msgr2=0x7f09a409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a359e640 1 -- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 msgr2=0x7f09a409e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a359e640 1 --2- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a359e640 1 --2- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_done global_id 214280 payload 274 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 1 -- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 msgr2=0x7f09a409e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 1 --2- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 1 -- 172.21.15.17:0/1099898602 >> v1:172.21.15.67:6789/0 conn(0x7f09a409c580 legacy=0x7f09a406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 1 monclient: found mon.noname-b 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:18.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.328+0000 7f09a2d9d640 1 -- 172.21.15.17:0/1099898602 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4069aa0 con 0x7f09a409c1b0 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a2d9d640 1 -- 172.21.15.17:0/1099898602 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 msgr2=0x7f09a409a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a2d9d640 1 --2- 172.21.15.17:0/1099898602 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0994009850 tx=0x7f09940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd35151b5b49521 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 1 -- 172.21.15.17:0/1099898602 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099402adf0 con 0x7f09a409c1b0 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: dump: 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: _finish_auth 0 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 1 -- 172.21.15.17:0/1099898602 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0994032440 con 0x7f09a409c1b0 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 1 -- 172.21.15.17:0/1099898602 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0994032820 con 0x7f09a409c1b0 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 5 monclient: authenticate success, global_id 214280 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 4 set_mon_vals no callback set 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:18.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a1d9b640 10 monclient: dump: 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 10 monclient: shutdown 2024-09-15T02:10:18.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 -- 172.21.15.17:0/1099898602 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 msgr2=0x7f09a409a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 --2- 172.21.15.17:0/1099898602 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0994009850 tx=0x7f09940090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 -- 172.21.15.17:0/1099898602 shutdown_connections 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 --2- 172.21.15.17:0/1099898602 >> v2:172.21.15.17:3300/0 conn(0x7f09a409c990 0x7f09a409e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 --2- 172.21.15.17:0/1099898602 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a409a990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 -- 172.21.15.17:0/1099898602 >> 172.21.15.17:0/1099898602 conn(0x7f09a4071300 msgr2=0x7f09a40a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 -- 172.21.15.17:0/1099898602 shutdown_connections 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.331+0000 7f09a90e8640 1 -- 172.21.15.17:0/1099898602 wait complete. 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.332+0000 7f09a90e8640 10 monclient: build_initial_monmap 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.332+0000 7f09a90e8640 10 monclient: monmap: 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:18.344 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.332+0000 7f09a90e8640 1 Processor -- start 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 -- start start 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient: init 2024-09-15T02:10:18.345 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-c con 0x7f09a409c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-a con 0x7f09a409c580 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 0x7f09a4097cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): picked mon.noname-b con 0x7f09a409c990 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a40a1260 con 0x7f09a409c580 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a409cdb0 con 0x7f09a409c990 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a411db70 con 0x7f09a409c1b0 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a90e8640 10 monclient(hunting): authenticate will time out at 1878.329595s 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60396/0 (socket says 172.21.15.17:60396) 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 -- 172.21.15.17:0/2126059055 learned_addr learned my addr 172.21.15.17:0/2126059055 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): get_auth_request con 0x7f09a409c1b0 auth_method 0 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 10 monclient(hunting): get_auth_request con 0x7f09a409c580 auth_method 0 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:18.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 msgr2=0x7f09a4132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 msgr2=0x7f09a4132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a259c640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a359e640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 0x7f09a4097cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient(hunting): handle_auth_done global_id 214283 payload 824 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 msgr2=0x7f09a4097cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:18.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 0x7f09a4097cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 msgr2=0x7f09a4132070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 monclient: found mon.noname-c 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.333+0000 7f09a2d9d640 1 -- 172.21.15.17:0/2126059055 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994009390 con 0x7f09a409c1b0 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.334+0000 7f09a359e640 10 monclient: get_auth_request con 0x7f09a409c990 auth_method 0 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.334+0000 7f09a359e640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 0x7f09a4097cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:10:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a2d9d640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 msgr2=0x7f09a41413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a2d9d640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0994009850 tx=0x7f0994032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0994031b10 con 0x7f09a409c1b0 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: dump: 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.350 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: _finish_auth 0 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0994031c70 con 0x7f09a409c1b0 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 5 monclient: authenticate success, global_id 214283 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 10 monclient: _renew_subs 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a411e290 con 0x7f09a409c1b0 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099403d440 con 0x7f09a409c1b0 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f099affd640 10 monclient: dump: 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:18.352 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 10 monclient: _renew_subs 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a159a640 4 set_mon_vals no callback set 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.336+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a4142170 con 0x7f09a409c1b0 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0994039040 con 0x7f09a409c1b0 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f09a90e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f09a90e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f09a90e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a411f600 con 0x7f09a409c1b0 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f099affd640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 0x7f098403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.337+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f099406c120 con 0x7f09a409c1b0 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.338+0000 7f09a259c640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 0x7f098403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.338+0000 7f09a259c640 10 monclient: get_auth_request con 0x7f0984038740 auth_method 0 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.340+0000 7f09a259c640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 0x7f098403abf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f09a409c180 tx=0x7f098c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:18.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.342+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0994044d80 con 0x7f09a409c1b0 2024-09-15T02:10:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.342+0000 7f099affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:18.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.342+0000 7f099affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:18.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.570+0000 7f09a90e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:18.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f09a90e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f09a90e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f09a40724a0 con 0x7f09a409c1b0 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f099affd640 1 -- 172.21.15.17:0/2126059055 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f0994037080 con 0x7f09a409c1b0 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f099affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.571+0000 7f099affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:18.573 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":184,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 msgr2=0x7f098403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 0x7f098403abf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f09a409c180 tx=0x7f098c009040 comp rx=0 tx=0).stop 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 10 monclient: shutdown 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 msgr2=0x7f09a41413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0994009850 tx=0x7f0994032000 comp rx=0 tx=0).stop 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 shutdown_connections 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0984038740 0x7f098403abf0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f09a409c580 0x7f09a4132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a409c990 0x7f09a4097cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 --2- 172.21.15.17:0/2126059055 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f09a409c1b0 0x7f09a41413e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:18.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 >> 172.21.15.17:0/2126059055 conn(0x7f09a4071300 msgr2=0x7f09a409e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:18.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 shutdown_connections 2024-09-15T02:10:18.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:18.573+0000 7f09a90e8640 1 -- 172.21.15.17:0/2126059055 wait complete. 2024-09-15T02:10:21.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.693+0000 7f427c11d640 10 monclient: get_monmap_and_config 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.693+0000 7f427c11d640 10 monclient: build_initial_monmap 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.693+0000 7f427c11d640 10 monclient: monmap: 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:21.694445+0000 2024-09-15T02:10:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:21.694445+0000 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:21.696 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.694+0000 7f427c11d640 1 Processor -- start 2024-09-15T02:10:21.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.694+0000 7f427c11d640 1 -- start start 2024-09-15T02:10:21.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.694+0000 7f427c11d640 10 monclient: init 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 0x7f427406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-a con 0x7f427409c500 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f427409c8d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f427406f280 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427406f680 con 0x7f427406f280 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427409ba20 con 0x7f427409c8d0 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427409e780 con 0x7f427409c500 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:21.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427c11d640 10 monclient(hunting): authenticate will time out at 1881.691171s 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 0x7f427406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427a6ba640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f427406f280 0x7f427409e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:50838/0 (socket says 172.21.15.17:50838) 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 10 monclient(hunting): get_auth_request con 0x7f427409c500 auth_method 0 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f427a6ba640 1 -- 172.21.15.17:0/4279104692 learned_addr learned my addr 172.21.15.17:0/4279104692 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 msgr2=0x7f427406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 msgr2=0x7f427406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 0x7f427406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.695+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 0x7f427406ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:21.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f427a6ba640 1 -- 172.21.15.17:0/4279104692 >> v1:172.21.15.17:6789/0 conn(0x7f427406f280 legacy=0x7f427409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f42796b8640 1 -- 172.21.15.17:0/4279104692 >> v1:172.21.15.67:6789/0 conn(0x7f427409c8d0 legacy=0x7f427409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527726482 0 0) 0x7f427409ba20 con 0x7f427409c8d0 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f4278eb7640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f4278eb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f4278eb7640 10 monclient(hunting): my global_id is 204373 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.697+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42600034e0 con 0x7f427409c8d0 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125869002 0 0) 0x7f42600034e0 con 0x7f427409c8d0 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 msgr2=0x7f427406ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 --2- 172.21.15.17:0/4279104692 >> v2:172.21.15.17:3300/0 conn(0x7f427409c500 0x7f427406ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 >> v1:172.21.15.17:6789/0 conn(0x7f427406f280 legacy=0x7f427409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: _finish_hunting 0 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427409f6c0 con 0x7f427409c8d0 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: _finish_auth 0 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427c11d640 5 monclient: authenticate success, global_id 204373 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f4264001830 con 0x7f427409c8d0 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427c11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: dump: 2024-09-15T02:10:21.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:10:21.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 >> v1:172.21.15.67:6789/0 conn(0x7f427409c8d0 legacy=0x7f427409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 0x7f4260003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): picked mon.c con 0x7f4260003a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260004780 0x7f4260004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): picked mon.b con 0x7f4260004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4260005250 0x7f426000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): picked mon.a con 0x7f4260005250 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427409e780 con 0x7f4260005250 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426000c210 con 0x7f4260004780 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426000c3d0 con 0x7f4260003a50 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4278eb7640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427c11d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 0x7f4260003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): get_auth_request con 0x7f4260003a50 auth_method 0 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4260005250 0x7f426000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): get_auth_request con 0x7f4260005250 auth_method 0 2024-09-15T02:10:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260004780 0x7f4260004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): get_auth_request con 0x7f4260004780 auth_method 0 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:21.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient(hunting): handle_auth_done global_id 204373 payload 293 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260004780 msgr2=0x7f4260004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260004780 0x7f4260004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4260005250 msgr2=0x7f426000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4260005250 0x7f426000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient: _finish_hunting 0 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 monclient: found mon.c 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:21.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600034e0 con 0x7f4260003a50 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 10 monclient: _finish_auth 0 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f4279eb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4260005250 0x7f426000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.698+0000 7f427a6ba640 1 -- 172.21.15.17:0/4279104692 reap_dead start 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4279eb9640 1 -- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 msgr2=0x7f4260003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4279eb9640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 0x7f4260003e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4270006fd0 tx=0x7f4270008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f661cf2ac54cb24b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427002add0 con 0x7f4260003a50 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: dump: 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f427c11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427002e070 con 0x7f4260003a50 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f427c11d640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f427c11d640 4 set_mon_vals no callback set 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 1 -- 172.21.15.17:0/4279104692 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270034660 con 0x7f4260003a50 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.701+0000 7f4278eb7640 10 monclient: dump: 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 10 monclient: shutdown 2024-09-15T02:10:21.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 -- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 msgr2=0x7f4260003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 0x7f4260003e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4270006fd0 tx=0x7f4270008040 comp rx=0 tx=0).stop 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 -- 172.21.15.17:0/4279104692 shutdown_connections 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 --2- 172.21.15.17:0/4279104692 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4260003a50 0x7f4260003e60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 -- 172.21.15.17:0/4279104692 >> 172.21.15.17:0/4279104692 conn(0x7f42740712f0 msgr2=0x7f42740a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 -- 172.21.15.17:0/4279104692 shutdown_connections 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 1 -- 172.21.15.17:0/4279104692 wait complete. 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 10 monclient: build_initial_monmap 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.702+0000 7f427c11d640 10 monclient: monmap: 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:21.718 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 Processor -- start 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 -- start start 2024-09-15T02:10:21.719 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient: init 2024-09-15T02:10:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:10:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 0x7f4274141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-b con 0x7f4260003a50 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-c con 0x7f4274141550 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): picked mon.noname-a con 0x7f42741322b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274069eb0 con 0x7f42741322b0 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274121190 con 0x7f4260003a50 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427409cf00 con 0x7f4274141550 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427c11d640 10 monclient(hunting): authenticate will time out at 1881.699796s 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427a6ba640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:60428/0 (socket says 172.21.15.17:60428) 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 1 -- 172.21.15.17:0/415065385 learned_addr learned my addr 172.21.15.17:0/415065385 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 10 monclient(hunting): get_auth_request con 0x7f4274141550 auth_method 0 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f42796b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427a6ba640 10 monclient(hunting): get_auth_request con 0x7f42741322b0 auth_method 0 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427a6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427a6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.703+0000 7f427a6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:21.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f427a6ba640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 msgr2=0x7f42740689a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f427a6ba640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 msgr2=0x7f42740689a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f427a6ba640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f427a6ba640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f4279eb9640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 0x7f4274141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient(hunting): handle_auth_done global_id 214286 payload 824 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 msgr2=0x7f4274141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 0x7f4274141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 msgr2=0x7f42740689a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient: _finish_hunting 0 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 monclient: found mon.noname-c 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f42796b8640 1 -- 172.21.15.17:0/415065385 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274098ba0 con 0x7f4274141550 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f4279eb9640 10 monclient: get_auth_request con 0x7f4260003a50 auth_method 0 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.704+0000 7f4279eb9640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 0x7f4274141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f42796b8640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 msgr2=0x7f4274131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f42796b8640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f426400ffd0 tx=0x7f4264009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264001210 con 0x7f4274141550 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:21.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: dump: 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: _finish_auth 0 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640070d0 con 0x7f4274141550 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 5 monclient: authenticate success, global_id 214286 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 10 monclient: _renew_subs 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4274120eb0 con 0x7f4274141550 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264007490 con 0x7f4274141550 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f4269ffb640 10 monclient: dump: 2024-09-15T02:10:21.729 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 10 monclient: _renew_subs 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.730 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f426bfff640 4 set_mon_vals no callback set 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.706+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4274068ee0 con 0x7f4274141550 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4264005990 con 0x7f4274141550 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f427c11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f427c11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f427c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f4269ffb640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 0x7f425803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.707+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42741326a0 con 0x7f4274141550 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.708+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f426404c570 con 0x7f4274141550 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.708+0000 7f4279eb9640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 0x7f425803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.708+0000 7f4279eb9640 10 monclient: get_auth_request con 0x7f42580387c0 auth_method 0 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.710+0000 7f4279eb9640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 0x7f425803ac70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4270006a70 tx=0x7f4270035000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.712+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f42741326a0 con 0x7f4274141550 2024-09-15T02:10:21.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.712+0000 7f4269ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:21.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.712+0000 7f4269ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:21.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.938+0000 7f427c11d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:21.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.938+0000 7f427c11d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:21.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.938+0000 7f427c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:21.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.938+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f42741326a0 con 0x7f4274141550 2024-09-15T02:10:21.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.939+0000 7f4269ffb640 1 -- 172.21.15.17:0/415065385 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f42741326a0 con 0x7f4274141550 2024-09-15T02:10:21.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.939+0000 7f4269ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:21.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.939+0000 7f4269ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:21.941 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":184,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 msgr2=0x7f425803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 0x7f425803ac70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4270006a70 tx=0x7f4270035000 comp rx=0 tx=0).stop 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 10 monclient: shutdown 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 msgr2=0x7f4274131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f426400ffd0 tx=0x7f4264009040 comp rx=0 tx=0).stop 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 shutdown_connections 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f42580387c0 0x7f425803ac70 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f42741322b0 0x7f42740689a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4274141550 0x7f4274131d70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 --2- 172.21.15.17:0/415065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4260003a50 0x7f4274141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:21.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 >> 172.21.15.17:0/415065385 conn(0x7f42740712f0 msgr2=0x7f4274132840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:21.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 shutdown_connections 2024-09-15T02:10:21.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:21.941+0000 7f427c11d640 1 -- 172.21.15.17:0/415065385 wait complete. 2024-09-15T02:10:24.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:25.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.060+0000 7fe765d9d640 10 monclient: get_monmap_and_config 2024-09-15T02:10:25.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.060+0000 7fe765d9d640 10 monclient: build_initial_monmap 2024-09-15T02:10:25.062 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.060+0000 7fe765d9d640 10 monclient: monmap: 2024-09-15T02:10:25.062 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:25.061665+0000 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:25.061665+0000 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:25.063 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.061+0000 7fe765d9d640 1 Processor -- start 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.061+0000 7fe765d9d640 1 -- start start 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.061+0000 7fe765d9d640 10 monclient: init 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe760068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7600692f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe760136fe0 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:25.066 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600a1950 con 0x7fe7600692f0 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe760069dd0 con 0x7fe760136fe0 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600a1670 con 0x7fe760068f20 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe765d9d640 10 monclient(hunting): authenticate will time out at 1885.058332s 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42724/0 (socket says 172.21.15.17:42724) 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 learned_addr learned my addr 172.21.15.17:0/479365230 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:25.067 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75effd640 1 --1- 172.21.15.17:0/479365230 >> v1:172.21.15.17:6789/0 conn(0x7fe7600692f0 0x7fe7600696c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54654/0 (socket says 172.21.15.17:54654) 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe760136fe0 auth_method 0 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:25.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.062+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:25.069 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.064+0000 7fe75effd640 1 -- 172.21.15.17:0/479365230 >> v1:172.21.15.17:6789/0 conn(0x7fe7600692f0 legacy=0x7fe7600696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe760136fe0 auth_method 0 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:25.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:10:25.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.263+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe760136fe0 auth_method 0 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:10:25.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:25.664+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe760136fe0 auth_method 0 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:26.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:10:26.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.465+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704319900 0 0) 0x7fe7600a1950 con 0x7fe7600692f0 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): my global_id is 234099 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7480034e0 con 0x7fe7600692f0 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3630673120 0 0) 0x7fe7480034e0 con 0x7fe7600692f0 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601321c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:26.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 msgr2=0x7fe7601393c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 --2- 172.21.15.17:0/479365230 >> v2:172.21.15.17:3300/0 conn(0x7fe760136fe0 0x7fe7601393c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600a0020 con 0x7fe7600692f0 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7fe754001720 con 0x7fe7600692f0 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:10:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: dump: 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 >> v1:172.21.15.17:6789/0 conn(0x7fe7600692f0 legacy=0x7fe7600696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe748003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): picked mon.c con 0x7fe760068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe748004150 0x7fe748004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): picked mon.a con 0x7fe748004150 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe748004d30 0x7fe7480051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): picked mon.b con 0x7fe748004d30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760069dd0 con 0x7fe748004150 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600a1670 con 0x7fe748004d30 2024-09-15T02:10:26.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74800bcb0 con 0x7fe760068f20 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.988+0000 7fe75e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe748004150 0x7fe748004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe748003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): get_auth_request con 0x7fe748004150 auth_method 0 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe748004d30 0x7fe7480051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe748004d30 auth_method 0 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:26.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient(hunting): handle_auth_done global_id 234099 payload 293 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe748004d30 msgr2=0x7fe7480051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe748004d30 0x7fe7480051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe748004150 msgr2=0x7fe748004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe748004150 0x7fe748004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 monclient: found mon.c 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480034e0 con 0x7fe760068f20 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:26.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:26.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe765d9d640 5 monclient: authenticate success, global_id 234099 2024-09-15T02:10:26.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe765d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:26.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.989+0000 7fe75effd640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe748004150 0x7fe748004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:27.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75f7fe640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe748003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:27.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe748003ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe750006ea0 tx=0x7fe750008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6950fe07e3693a96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:27.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750034730 con 0x7fe760068f20 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: dump: 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7500348b0 con 0x7fe760068f20 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe765d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 1 -- 172.21.15.17:0/479365230 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750034c70 con 0x7fe760068f20 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe765d9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe765d9d640 4 set_mon_vals no callback set 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.992+0000 7fe75e7fc640 10 monclient: dump: 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 10 monclient: shutdown 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 -- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe748003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 --2- 172.21.15.17:0/479365230 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe748003ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe750006ea0 tx=0x7fe750008040 comp rx=0 tx=0).stop 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe75ffff640 1 -- 172.21.15.17:0/479365230 reap_dead start 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 -- 172.21.15.17:0/479365230 shutdown_connections 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 -- 172.21.15.17:0/479365230 >> 172.21.15.17:0/479365230 conn(0x7fe760070fb0 msgr2=0x7fe7600a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 -- 172.21.15.17:0/479365230 shutdown_connections 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.993+0000 7fe765d9d640 1 -- 172.21.15.17:0/479365230 wait complete. 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.994+0000 7fe765d9d640 10 monclient: build_initial_monmap 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.994+0000 7fe765d9d640 10 monclient: monmap: 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:27.008 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 Processor -- start 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 -- start start 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient: init 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:27.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe760068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7601358e0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 0x7fe760143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe760136250 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600a0750 con 0x7fe7601358e0 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76009c550 con 0x7fe760136250 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe760099120 con 0x7fe760068f20 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:27.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.995+0000 7fe765d9d640 10 monclient(hunting): authenticate will time out at 1886.991915s 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56354/0 (socket says 172.21.15.17:56354) 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 -- 172.21.15.17:0/2663689372 learned_addr learned my addr 172.21.15.17:0/2663689372 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request con 0x7fe760068f20 auth_method 0 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): get_auth_request con 0x7fe7601358e0 auth_method 0 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:27.011 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 0x7fe760143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 msgr2=0x7fe760135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 msgr2=0x7fe760135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): get_auth_request con 0x7fe760136250 auth_method 0 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75effd640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient(hunting): handle_auth_done global_id 204376 payload 824 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 msgr2=0x7fe7601353a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 msgr2=0x7fe760135d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 monclient: found mon.noname-b 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.996+0000 7fe75ffff640 1 -- 172.21.15.17:0/2663689372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600972f0 con 0x7fe760136250 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe75ffff640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 msgr2=0x7fe760143440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe75ffff640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 0x7fe760143440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe760096ed0 tx=0x7fe75800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758018870 con 0x7fe760136250 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: dump: 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.014 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75800f040 con 0x7fe760136250 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 5 monclient: authenticate success, global_id 204376 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 10 monclient: _renew_subs 2024-09-15T02:10:27.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760098db0 con 0x7fe760136250 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758018c40 con 0x7fe760136250 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe7477fe640 10 monclient: dump: 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:27.016 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 10 monclient: _renew_subs 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe75dffb640 4 set_mon_vals no callback set 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:26.999+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760097ec0 con 0x7fe760136250 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe7580173b0 con 0x7fe760136250 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe7477fe640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 0x7fe74c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe754007e40 con 0x7fe760136250 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe765d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe765d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe765d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.000+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7600724a0 con 0x7fe760136250 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.001+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 0x7fe74c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.001+0000 7fe75f7fe640 10 monclient: get_auth_request con 0x7fe74c0387c0 auth_method 0 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.003+0000 7fe75f7fe640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 0x7fe74c03ac70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe750003000 tx=0x7fe75003e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:27.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.006+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe75801c070 con 0x7fe760136250 2024-09-15T02:10:27.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.006+0000 7fe7477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:27.019 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.006+0000 7fe7477fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:27.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.235+0000 7fe765d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:27.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.235+0000 7fe765d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:27.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.235+0000 7fe765d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:10:27.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.235+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe76006e220 con 0x7fe760136250 2024-09-15T02:10:27.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.236+0000 7fe7477fe640 1 -- 172.21.15.17:0/2663689372 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7fe758017e60 con 0x7fe760136250 2024-09-15T02:10:27.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.236+0000 7fe7477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:27.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.236+0000 7fe7477fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:27.238 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":188,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 msgr2=0x7fe74c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 0x7fe74c03ac70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe750003000 tx=0x7fe75003e000 comp rx=0 tx=0).stop 2024-09-15T02:10:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 10 monclient: shutdown 2024-09-15T02:10:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:27.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 msgr2=0x7fe760143440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 0x7fe760143440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe760096ed0 tx=0x7fe75800b040 comp rx=0 tx=0).stop 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 shutdown_connections 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.237+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe74c0387c0 0x7fe74c03ac70 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe7601358e0 0x7fe760135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe760136250 0x7fe760143440 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 --2- 172.21.15.17:0/2663689372 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe760068f20 0x7fe7601353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 >> 172.21.15.17:0/2663689372 conn(0x7fe760070fb0 msgr2=0x7fe76009e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 shutdown_connections 2024-09-15T02:10:27.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:27.238+0000 7fe765d9d640 1 -- 172.21.15.17:0/2663689372 wait complete. 2024-09-15T02:10:30.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:30.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.358+0000 7f176a046640 10 monclient: get_monmap_and_config 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.358+0000 7f176a046640 10 monclient: build_initial_monmap 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.358+0000 7f176a046640 10 monclient: monmap: 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:30.359309+0000 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:30.359309+0000 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:30.361 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:30.361 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:30.361 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 Processor -- start 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 -- start start 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient: init 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1764068f20 0x7f17641321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): picked mon.noname-c con 0x7f1764068f20 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17640692f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:30.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1764136fe0 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640a1950 con 0x7f1764136fe0 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640a1670 con 0x7f17640692f0 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17640a0020 con 0x7f1764068f20 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.359+0000 7f176a046640 10 monclient(hunting): authenticate will time out at 1890.355944s 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f1763fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1764136fe0 0x7f17641393c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:54656/0 (socket says 172.21.15.17:54656) 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f1763fff640 1 -- 172.21.15.17:0/1133952492 learned_addr learned my addr 172.21.15.17:0/1133952492 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 1 --2- 172.21.15.17:0/1133952492 >> v2:172.21.15.67:3300/0 conn(0x7f1764068f20 0x7f17641321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): get_auth_request con 0x7f1764068f20 auth_method 0 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:30.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient(hunting): handle_auth_done global_id 204379 payload 274 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 1 -- 172.21.15.17:0/1133952492 >> v1:172.21.15.67:6789/0 conn(0x7f17640692f0 legacy=0x7f17640696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 1 -- 172.21.15.17:0/1133952492 >> v1:172.21.15.17:6789/0 conn(0x7f1764136fe0 legacy=0x7f17641393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient: _finish_hunting 0 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 1 monclient: found mon.noname-c 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:10:30.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.360+0000 7f17637fe640 1 -- 172.21.15.17:0/1133952492 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640a1390 con 0x7f1764068f20 2024-09-15T02:10:30.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.363+0000 7f17637fe640 1 -- 172.21.15.17:0/1133952492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764068f20 msgr2=0x7f17641321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:30.369 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.363+0000 7f17637fe640 1 --2- 172.21.15.17:0/1133952492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764068f20 0x7f17641321c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f17540098b0 tx=0x7f17540090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4cb828666a15505 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 1 -- 172.21.15.17:0/1133952492 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175402ae00 con 0x7f1764068f20 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: dump: 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: _finish_auth 0 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 1 -- 172.21.15.17:0/1133952492 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175402f4b0 con 0x7f1764068f20 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 1 -- 172.21.15.17:0/1133952492 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175402f850 con 0x7f1764068f20 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 5 monclient: authenticate success, global_id 204379 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 4 set_mon_vals no callback set 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f17627fc640 10 monclient: dump: 2024-09-15T02:10:30.371 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 10 monclient: shutdown 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 -- 172.21.15.17:0/1133952492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764068f20 msgr2=0x7f17641321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:30.372 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 --2- 172.21.15.17:0/1133952492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764068f20 0x7f17641321c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f17540098b0 tx=0x7f17540090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 -- 172.21.15.17:0/1133952492 shutdown_connections 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 --2- 172.21.15.17:0/1133952492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764068f20 0x7f17641321c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 -- 172.21.15.17:0/1133952492 >> 172.21.15.17:0/1133952492 conn(0x7f1764070fb0 msgr2=0x7f17640a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 -- 172.21.15.17:0/1133952492 shutdown_connections 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.364+0000 7f176a046640 1 -- 172.21.15.17:0/1133952492 wait complete. 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.365+0000 7f176a046640 10 monclient: build_initial_monmap 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.365+0000 7f176a046640 10 monclient: monmap: 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.373 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.365+0000 7f176a046640 1 Processor -- start 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.365+0000 7f176a046640 1 -- start start 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.365+0000 7f176a046640 10 monclient: init 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): picked mon.noname-c con 0x7f1764068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 0x7f1764099df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): picked mon.noname-a con 0x7f17640692f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764136fe0 0x7f176409a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:30.374 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): picked mon.noname-b con 0x7f1764136fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640a0720 con 0x7f17640692f0 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1764137880 con 0x7f1764136fe0 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176411c7f0 con 0x7f1764068f20 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f176a046640 10 monclient(hunting): authenticate will time out at 1890.362441s 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 0x7f1764099df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:56372/0 (socket says 172.21.15.17:56372) 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 -- 172.21.15.17:0/3731126162 learned_addr learned my addr 172.21.15.17:0/3731126162 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:30.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 10 monclient(hunting): get_auth_request con 0x7f1764068f20 auth_method 0 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): get_auth_request con 0x7f17640692f0 auth_method 0 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 msgr2=0x7f17641309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 msgr2=0x7f17641309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:30.376 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f17637fe640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1763fff640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764136fe0 0x7f176409a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1763fff640 10 monclient(hunting): get_auth_request con 0x7f1764136fe0 auth_method 0 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1763fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1763fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient(hunting): handle_auth_done global_id 234105 payload 824 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 msgr2=0x7f17641309d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764136fe0 msgr2=0x7f176409a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764136fe0 0x7f176409a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 monclient: found mon.noname-a 2024-09-15T02:10:30.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:30.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:30.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.366+0000 7f1762ffd640 1 -- 172.21.15.17:0/3731126162 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1754009390 con 0x7f17640692f0 2024-09-15T02:10:30.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f1762ffd640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 msgr2=0x7f1764099df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:30.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f1762ffd640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 0x7f1764099df0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f175800c4e0 tx=0x7f1758007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:30.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17580117a0 con 0x7f17640692f0 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: dump: 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:30.380 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1758011920 con 0x7f17640692f0 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758011d20 con 0x7f17640692f0 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 5 monclient: authenticate success, global_id 234105 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 10 monclient: _renew_subs 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176411bb80 con 0x7f17640692f0 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f1761ffb640 4 set_mon_vals no callback set 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:30.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f174b7fe640 10 monclient: dump: 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 10 monclient: _renew_subs 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.369+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176411cb90 con 0x7f17640692f0 2024-09-15T02:10:30.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1758012630 con 0x7f17640692f0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f176a046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f176a046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f176a046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17640724a0 con 0x7f17640692f0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f174b7fe640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 0x7f175003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f175804c280 con 0x7f17640692f0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.370+0000 7f17637fe640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 0x7f175003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.371+0000 7f17637fe640 10 monclient: get_auth_request con 0x7f1750038790 auth_method 0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.373+0000 7f17637fe640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 0x7f175003ac40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1754009130 tx=0x7f1754032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.375+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f17580128e0 con 0x7f17640692f0 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.375+0000 7f174b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:30.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.375+0000 7f174b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f176a046640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f176a046640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f176a046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f176409b8f0 con 0x7f17640692f0 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f174b7fe640 1 -- 172.21.15.17:0/3731126162 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f1758006790 con 0x7f17640692f0 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f174b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:30.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.606+0000 7f174b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:30.609 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":188,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:30.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 msgr2=0x7f175003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:30.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 0x7f175003ac40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1754009130 tx=0x7f1754032000 comp rx=0 tx=0).stop 2024-09-15T02:10:30.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 10 monclient: shutdown 2024-09-15T02:10:30.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:30.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 msgr2=0x7f1764099df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 0x7f1764099df0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f175800c4e0 tx=0x7f1758007e10 comp rx=0 tx=0).stop 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 shutdown_connections 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1750038790 0x7f175003ac40 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f17640692f0 0x7f1764099df0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1764136fe0 0x7f176409a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 --2- 172.21.15.17:0/3731126162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1764068f20 0x7f17641309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 >> 172.21.15.17:0/3731126162 conn(0x7f1764070fb0 msgr2=0x7f176409e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 shutdown_connections 2024-09-15T02:10:30.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:30.608+0000 7f176a046640 1 -- 172.21.15.17:0/3731126162 wait complete. 2024-09-15T02:10:33.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:33.727 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.726+0000 7f9d4939d640 10 monclient: get_monmap_and_config 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.726+0000 7f9d4939d640 10 monclient: build_initial_monmap 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.726+0000 7f9d4939d640 10 monclient: monmap: 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:33.727240+0000 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:33.727240+0000 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:33.728 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:33.729 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:33.729 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.726+0000 7f9d4939d640 1 Processor -- start 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 -- start start 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient: init 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d44067090 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d4409ce00 0x7f9d44067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4409ce00 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:33.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9d440679a0 0x7f9d44139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d440679a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d44132850 con 0x7f9d44067090 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4409e4c0 con 0x7f9d440679a0 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4409cbf0 con 0x7f9d4409ce00 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d4939d640 10 monclient(hunting): authenticate will time out at 1893.723599s 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f9d440679a0 0x7f9d44139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d42ffd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f9d44067090 0x7f9d44132140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34538/0 (socket says 172.21.15.17:34538) 2024-09-15T02:10:33.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d42ffd640 1 -- 172.21.15.17:0/577875116 learned_addr learned my addr 172.21.15.17:0/577875116 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): get_auth_request con 0x7f9d440679a0 auth_method 0 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 10 monclient(hunting): handle_auth_done global_id 234108 payload 274 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d427fc640 1 --2- 172.21.15.17:0/577875116 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409ce00 0x7f9d44067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 1 -- 172.21.15.17:0/577875116 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409ce00 msgr2=0x7f9d44067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.727+0000 7f9d437fe640 1 --2- 172.21.15.17:0/577875116 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409ce00 0x7f9d44067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 1 -- 172.21.15.17:0/577875116 >> v1:172.21.15.17:6789/0 conn(0x7f9d44067090 legacy=0x7f9d44132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:33.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 10 monclient: _finish_hunting 0 2024-09-15T02:10:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 1 monclient: found mon.noname-a 2024-09-15T02:10:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.728+0000 7f9d437fe640 1 -- 172.21.15.17:0/577875116 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4409e1e0 con 0x7f9d440679a0 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.730+0000 7f9d437fe640 1 -- 172.21.15.17:0/577875116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d440679a0 msgr2=0x7f9d44139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d437fe640 1 --2- 172.21.15.17:0/577875116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d440679a0 0x7f9d44139010 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9d4409faa0 tx=0x7f9d3c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48e8913c15bf43e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 1 -- 172.21.15.17:0/577875116 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c007840 con 0x7f9d440679a0 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: dump: 2024-09-15T02:10:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: _finish_auth 0 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 1 -- 172.21.15.17:0/577875116 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c02d050 con 0x7f9d440679a0 2024-09-15T02:10:33.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 1 -- 172.21.15.17:0/577875116 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c007c10 con 0x7f9d440679a0 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 5 monclient: authenticate success, global_id 234108 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 4 set_mon_vals no callback set 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d41ffb640 10 monclient: dump: 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.742 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 10 monclient: shutdown 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 -- 172.21.15.17:0/577875116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d440679a0 msgr2=0x7f9d44139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 --2- 172.21.15.17:0/577875116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d440679a0 0x7f9d44139010 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9d4409faa0 tx=0x7f9d3c009780 comp rx=0 tx=0).stop 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 -- 172.21.15.17:0/577875116 shutdown_connections 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 --2- 172.21.15.17:0/577875116 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d440679a0 0x7f9d44139010 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 --2- 172.21.15.17:0/577875116 >> v2:172.21.15.67:3300/0 conn(0x7f9d4409ce00 0x7f9d44067460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 -- 172.21.15.17:0/577875116 >> 172.21.15.17:0/577875116 conn(0x7f9d440a1ee0 msgr2=0x7f9d440a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 -- 172.21.15.17:0/577875116 shutdown_connections 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.731+0000 7f9d4939d640 1 -- 172.21.15.17:0/577875116 wait complete. 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.732+0000 7f9d4939d640 10 monclient: build_initial_monmap 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.732+0000 7f9d4939d640 10 monclient: monmap: 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:33.744 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.732+0000 7f9d4939d640 1 Processor -- start 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.732+0000 7f9d4939d640 1 -- start start 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.732+0000 7f9d4939d640 10 monclient: init 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d44067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d440679a0 0x7f9d441351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d440679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d4409ce00 0x7f9d44135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.745 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4409ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440a0220 con 0x7f9d44067090 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44118e50 con 0x7f9d440679a0 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44137280 con 0x7f9d4409ce00 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d4939d640 10 monclient(hunting): authenticate will time out at 1893.729568s 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48948/0 (socket says 172.21.15.17:48948) 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 1 -- 172.21.15.17:0/4132692579 learned_addr learned my addr 172.21.15.17:0/4132692579 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): get_auth_request con 0x7f9d44067090 auth_method 0 2024-09-15T02:10:33.746 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d4409ce00 0x7f9d44135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): get_auth_request con 0x7f9d4409ce00 auth_method 0 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d440679a0 0x7f9d441351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): get_auth_request con 0x7f9d440679a0 auth_method 0 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:33.747 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_done global_id 234111 payload 824 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 1 -- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d4409ce00 msgr2=0x7f9d44135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.733+0000 7f9d42ffd640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d4409ce00 0x7f9d44135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 1 -- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d440679a0 msgr2=0x7f9d441351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:33.748 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d440679a0 0x7f9d441351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 1 monclient: found mon.noname-a 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.734+0000 7f9d42ffd640 1 -- 172.21.15.17:0/4132692579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009a20 con 0x7f9d44067090 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d42ffd640 1 -- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 msgr2=0x7f9d4412f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d42ffd640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c5e0 tx=0x7f9d34004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005cb0 con 0x7f9d44067090 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: dump: 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:33.749 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: _finish_auth 0 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 5 monclient: authenticate success, global_id 234111 2024-09-15T02:10:33.750 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34005e30 con 0x7f9d44067090 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 10 monclient: _renew_subs 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4412e780 con 0x7f9d44067090 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34007340 con 0x7f9d44067090 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d2affd640 10 monclient: dump: 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:33.751 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 10 monclient: _renew_subs 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d441374a0 con 0x7f9d44067090 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.736+0000 7f9d417fa640 4 set_mon_vals no callback set 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d3401b400 con 0x7f9d44067090 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d4939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d4939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d4939d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.752 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d2affd640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 0x7f9d3003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d440724a0 con 0x7f9d44067090 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f9d3404c210 con 0x7f9d44067090 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d427fc640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 0x7f9d3003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.737+0000 7f9d427fc640 10 monclient: get_auth_request con 0x7f9d30038740 auth_method 0 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.740+0000 7f9d427fc640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 0x7f9d3003abf0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800ac20 tx=0x7f9d38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.742+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d3401b6b0 con 0x7f9d44067090 2024-09-15T02:10:33.753 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.742+0000 7f9d2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:33.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.742+0000 7f9d2affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d4939d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:33.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d4939d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d4939d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d440689a0 con 0x7f9d44067090 2024-09-15T02:10:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d2affd640 1 -- 172.21.15.17:0/4132692579 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f9d3401a360 con 0x7f9d44067090 2024-09-15T02:10:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:33.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.970+0000 7f9d2affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:33.973 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":192,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 msgr2=0x7f9d3003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 0x7f9d3003abf0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800ac20 tx=0x7f9d38008040 comp rx=0 tx=0).stop 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 10 monclient: shutdown 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 msgr2=0x7f9d4412f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c5e0 tx=0x7f9d34004960 comp rx=0 tx=0).stop 2024-09-15T02:10:33.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 shutdown_connections 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f9d30038740 0x7f9d3003abf0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f9d4409ce00 0x7f9d44135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d440679a0 0x7f9d441351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 --2- 172.21.15.17:0/4132692579 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9d44067090 0x7f9d4412f290 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 >> 172.21.15.17:0/4132692579 conn(0x7f9d440a1ee0 msgr2=0x7f9d44138db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 shutdown_connections 2024-09-15T02:10:33.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:33.972+0000 7f9d4939d640 1 -- 172.21.15.17:0/4132692579 wait complete. 2024-09-15T02:10:33.984 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:10:33.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:10:33.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.080+0000 7f1033ac9640 10 monclient: get_monmap_and_config 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.080+0000 7f1033ac9640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.080+0000 7f1033ac9640 10 monclient: monmap: 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:34.081455+0000 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:34.081455+0000 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.082 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:34.083 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 Processor -- start 2024-09-15T02:10:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 -- start start 2024-09-15T02:10:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient: init 2024-09-15T02:10:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f102c09c1b0 0x7f102c09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f102c09c1b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f102c09c580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f102c09c990 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c069d40 con 0x7f102c09c990 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c09f790 con 0x7f102c09c1b0 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c068b60 con 0x7f102c09c580 2024-09-15T02:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.081+0000 7f1033ac9640 10 monclient(hunting): authenticate will time out at 1894.077894s 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48956/0 (socket says 172.21.15.17:48956) 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47788/0 (socket says 172.21.15.17:47788) 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 1 -- 172.21.15.17:0/2782526199 learned_addr learned my addr 172.21.15.17:0/2782526199 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): get_auth_request con 0x7f102c09c990 auth_method 0 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): get_auth_request con 0x7f102c09c580 auth_method 0 2024-09-15T02:10:34.087 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031865640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.67:3300/0 conn(0x7f102c09c1b0 0x7f102c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031865640 10 monclient(hunting): get_auth_request con 0x7f102c09c1b0 auth_method 0 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031865640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.088 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient(hunting): handle_auth_done global_id 214292 payload 274 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 -- 172.21.15.17:0/2782526199 >> v2:172.21.15.67:3300/0 conn(0x7f102c09c1b0 msgr2=0x7f102c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.67:3300/0 conn(0x7f102c09c1b0 0x7f102c09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 -- 172.21.15.17:0/2782526199 >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 msgr2=0x7f102c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 monclient: found mon.noname-b 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1031064640 1 -- 172.21.15.17:0/2782526199 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c069aa0 con 0x7f102c09c580 2024-09-15T02:10:34.089 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.082+0000 7f1032066640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1031064640 1 -- 172.21.15.17:0/2782526199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 msgr2=0x7f102c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1031064640 1 --2- 172.21.15.17:0/2782526199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f102c068b30 tx=0x7f101c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94829dff449dce3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 1 -- 172.21.15.17:0/2782526199 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c02adf0 con 0x7f102c09c580 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: dump: 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.096 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 1 -- 172.21.15.17:0/2782526199 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c032440 con 0x7f102c09c580 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 5 monclient: authenticate success, global_id 214292 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 1 -- 172.21.15.17:0/2782526199 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c0327e0 con 0x7f102c09c580 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1030863640 10 monclient: dump: 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.098 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 4 set_mon_vals no callback set 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 10 monclient: shutdown 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 -- 172.21.15.17:0/2782526199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 msgr2=0x7f102c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 --2- 172.21.15.17:0/2782526199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f102c068b30 tx=0x7f101c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 -- 172.21.15.17:0/2782526199 shutdown_connections 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.17:3300/0 conn(0x7f102c09c990 0x7f102c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 --2- 172.21.15.17:0/2782526199 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c06eb30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 --2- 172.21.15.17:0/2782526199 >> v2:172.21.15.67:3300/0 conn(0x7f102c09c1b0 0x7f102c09a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.085+0000 7f1033ac9640 1 -- 172.21.15.17:0/2782526199 >> 172.21.15.17:0/2782526199 conn(0x7f102c071300 msgr2=0x7f102c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 1 -- 172.21.15.17:0/2782526199 shutdown_connections 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 1 -- 172.21.15.17:0/2782526199 wait complete. 2024-09-15T02:10:34.099 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 10 monclient: monmap: 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.100 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 1 Processor -- start 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 1 -- start start 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.086+0000 7f1033ac9640 10 monclient: init 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c09c1b0 0x7f102c141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f102c09c1b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f102c09c580 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c09c990 0x7f102c11e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f102c09c990 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c06f090 con 0x7f102c09c1b0 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.101 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c121f10 con 0x7f102c09c990 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c1221f0 con 0x7f102c09c580 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1033ac9640 10 monclient(hunting): authenticate will time out at 1894.083381s 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c09c1b0 0x7f102c141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47790/0 (socket says 172.21.15.17:47790) 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 -- 172.21.15.17:0/3388480551 learned_addr learned my addr 172.21.15.17:0/3388480551 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): get_auth_request con 0x7f102c09c580 auth_method 0 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.102 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): get_auth_request con 0x7f102c09c1b0 auth_method 0 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c09c990 0x7f102c11e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 10 monclient(hunting): get_auth_request con 0x7f102c09c990 auth_method 0 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.103 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient(hunting): handle_auth_done global_id 214295 payload 824 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c09c990 msgr2=0x7f102c11e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c09c990 0x7f102c11e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c09c1b0 msgr2=0x7f102c141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c09c1b0 0x7f102c141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 monclient: found mon.noname-c 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1031064640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024009ae0 con 0x7f102c09c580 2024-09-15T02:10:34.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.087+0000 7f1032066640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c09c990 0x7f102c11e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1031064640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 msgr2=0x7f102c134df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1031064640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f101c02a5e0 tx=0x7f101c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c031940 con 0x7f102c09c580 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: dump: 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.105 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c031aa0 con 0x7f102c09c580 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 5 monclient: authenticate success, global_id 214295 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c031e60 con 0x7f102c09c580 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1021ffb640 10 monclient: dump: 2024-09-15T02:10:34.106 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 10 monclient: _renew_subs 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1023fff640 4 set_mon_vals no callback set 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c009390 con 0x7f102c09c580 2024-09-15T02:10:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 10 monclient: _renew_subs 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.090+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c120450 con 0x7f102c09c580 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1033ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1033ac9640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 0x7f102c11ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c063bb0 con 0x7f102c121020 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031064640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f101c02a5e0 tx=0x7f101c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f101c06ceb0 osd_map(257..257 src has 229..257) v4 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f101c03c610 con 0x7f102c09c580 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 0x7f102c11ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient: get_auth_request con 0x7f102c121020 auth_method 0 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1021ffb640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1010038740 0x7f101003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 10 monclient(hunting): handle_auth_done global_id 234120 payload 824 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1031865640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102c122080 con 0x7f102c121020 2024-09-15T02:10:34.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1032066640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1010038740 0x7f101003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.091+0000 7f1032066640 10 monclient: get_auth_request con 0x7f1010038740 auth_method 0 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1031865640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 msgr2=0x7f102c11ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1031865640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 0x7f102c11ee00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1028010800 tx=0x7f1028008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028010ec0 con 0x7f102c121020 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1021ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.093+0000 7f1021ffb640 10 monclient: dump: 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.110 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f102800ed50 con 0x7f102c121020 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 msgr2=0x7f102c11ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 0x7f102c11ee00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1028010800 tx=0x7f1028008040 comp rx=0 tx=0).stop 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 msgr2=0x7f102c11ee00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1021ffb640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c121020 0x7f102c11ee00 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1028010800 tx=0x7f1028008040 comp rx=0 tx=0).stop 2024-09-15T02:10:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.094+0000 7f1032066640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1010038740 0x7f101003abf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1024009fd0 tx=0x7f102400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1033ac9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1033ac9640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 0x7f102c11f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1033ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0724a0 con 0x7f102c135330 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 0x7f102c11f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient: get_auth_request con 0x7f102c135330 auth_method 0 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 10 monclient(hunting): handle_auth_done global_id 234123 payload 824 2024-09-15T02:10:34.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.100+0000 7f1031865640 1 -- 172.21.15.17:0/3388480551 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f102c141e60 con 0x7f102c135330 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1031865640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 msgr2=0x7f102c11f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1031865640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 0x7f102c11f340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1028019000 tx=0x7f102801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102801cc10 con 0x7f102c135330 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1021ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.102+0000 7f1021ffb640 10 monclient: dump: 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7f10280133d0 con 0x7f102c135330 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 msgr2=0x7f102c11f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 0x7f102c11f340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1028019000 tx=0x7f102801c000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 msgr2=0x7f102c11f340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f102c135330 0x7f102c11f340 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1028019000 tx=0x7f102801c000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:34.117 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.118 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.119 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:34.120 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.121 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.123 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.125 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.127 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.128 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:34.129 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1010038740 msgr2=0x7f101003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1010038740 0x7f101003abf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1024009fd0 tx=0x7f102400b040 comp rx=0 tx=0).stop 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 10 monclient: shutdown 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 msgr2=0x7f102c134df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 --2- 172.21.15.17:0/3388480551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f102c09c580 0x7f102c134df0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f101c02a5e0 tx=0x7f101c032000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1032066640 1 -- 172.21.15.17:0/3388480551 reap_dead start 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1021ffb640 1 -- 172.21.15.17:0/3388480551 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7f101c06ceb0 con 0x7f102c09c580 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1032066640 1 -- 172.21.15.17:0/3388480551 reap_dead start 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 shutdown_connections 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 >> 172.21.15.17:0/3388480551 conn(0x7f102c071300 msgr2=0x7f102c09de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 shutdown_connections 2024-09-15T02:10:34.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.103+0000 7f1033ac9640 1 -- 172.21.15.17:0/3388480551 wait complete. 2024-09-15T02:10:34.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:34.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:10:34.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.228+0000 7f1014154640 10 monclient: get_monmap_and_config 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.228+0000 7f1014154640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.228+0000 7f1014154640 10 monclient: monmap: 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:34.229414+0000 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:34.229414+0000 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.230 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:34.231 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 Processor -- start 2024-09-15T02:10:34.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 -- start start 2024-09-15T02:10:34.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient: init 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f100c09c1b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f100c09e9e0 0x7f100c09c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): picked mon.noname-c con 0x7f100c09e9e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f100c06eb30 0x7f100c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): picked mon.noname-a con 0x7f100c06eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c09f790 con 0x7f100c09c1b0 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c069d40 con 0x7f100c06eb30 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c09ee20 con 0x7f100c09e9e0 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.229+0000 7f1014154640 10 monclient(hunting): authenticate will time out at 1894.225876s 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f100c06eb30 0x7f100c06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f100c06eb30 0x7f100c06ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:48988/0 (socket says 172.21.15.17:48988) 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 -- 172.21.15.17:0/2335924102 learned_addr learned my addr 172.21.15.17:0/2335924102 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): get_auth_request con 0x7f100c06eb30 auth_method 0 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10116ef640 1 --2- 172.21.15.17:0/2335924102 >> v2:172.21.15.67:3300/0 conn(0x7f100c09e9e0 0x7f100c09c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient(hunting): handle_auth_done global_id 234126 payload 274 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 -- 172.21.15.17:0/2335924102 >> v2:172.21.15.67:3300/0 conn(0x7f100c09e9e0 msgr2=0x7f100c09c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 --2- 172.21.15.17:0/2335924102 >> v2:172.21.15.67:3300/0 conn(0x7f100c09e9e0 0x7f100c09c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 -- 172.21.15.17:0/2335924102 >> v1:172.21.15.67:6789/0 conn(0x7f100c09c1b0 legacy=0x7f100c09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 monclient: found mon.noname-a 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.230+0000 7f10126f1640 1 -- 172.21.15.17:0/2335924102 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c068b60 con 0x7f100c06eb30 2024-09-15T02:10:34.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f10126f1640 1 -- 172.21.15.17:0/2335924102 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 msgr2=0x7f100c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f10126f1640 1 --2- 172.21.15.17:0/2335924102 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c06ef90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f100c060cf0 tx=0x7f1004009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46565c8286509f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 1 -- 172.21.15.17:0/2335924102 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004007610 con 0x7f100c06eb30 2024-09-15T02:10:34.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: dump: 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 5 monclient: authenticate success, global_id 234126 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 1 -- 172.21.15.17:0/2335924102 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100402d050 con 0x7f100c06eb30 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 4 set_mon_vals no callback set 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 1 -- 172.21.15.17:0/2335924102 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10040079a0 con 0x7f100c06eb30 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1010eee640 10 monclient: dump: 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.248 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 10 monclient: shutdown 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 -- 172.21.15.17:0/2335924102 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 msgr2=0x7f100c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 --2- 172.21.15.17:0/2335924102 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c06ef90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f100c060cf0 tx=0x7f1004009840 comp rx=0 tx=0).stop 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 -- 172.21.15.17:0/2335924102 shutdown_connections 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 --2- 172.21.15.17:0/2335924102 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c06ef90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 --2- 172.21.15.17:0/2335924102 >> v2:172.21.15.67:3300/0 conn(0x7f100c09e9e0 0x7f100c09c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 -- 172.21.15.17:0/2335924102 >> 172.21.15.17:0/2335924102 conn(0x7f100c071300 msgr2=0x7f100c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.233+0000 7f1014154640 1 -- 172.21.15.17:0/2335924102 shutdown_connections 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 1 -- 172.21.15.17:0/2335924102 wait complete. 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 10 monclient: monmap: 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.250 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 1 Processor -- start 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 1 -- start start 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.234+0000 7f1014154640 10 monclient: init 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:10:34.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): picked mon.noname-a con 0x7f100c06eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c09c1b0 0x7f100c11da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): picked mon.noname-b con 0x7f100c09c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c09e9e0 0x7f100c11df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): picked mon.noname-c con 0x7f100c09e9e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c121080 con 0x7f100c06eb30 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c11fe40 con 0x7f100c09c1b0 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c120da0 con 0x7f100c09e9e0 2024-09-15T02:10:34.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1014154640 10 monclient(hunting): authenticate will time out at 1894.231213s 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49004/0 (socket says 172.21.15.17:49004) 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 -- 172.21.15.17:0/2209203369 learned_addr learned my addr 172.21.15.17:0/2209203369 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): get_auth_request con 0x7f100c06eb30 auth_method 0 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c09e9e0 0x7f100c11df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): get_auth_request con 0x7f100c09e9e0 auth_method 0 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c09c1b0 0x7f100c11da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): get_auth_request con 0x7f100c09c1b0 auth_method 0 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10116ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient(hunting): handle_auth_done global_id 234129 payload 824 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c09e9e0 msgr2=0x7f100c11df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c09e9e0 0x7f100c11df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c09c1b0 msgr2=0x7f100c11da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c09c1b0 0x7f100c11da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 monclient: found mon.noname-a 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f1011ef0640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004009ae0 con 0x7f100c06eb30 2024-09-15T02:10:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.235+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c09e9e0 0x7f100c11df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.237+0000 7f1011ef0640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 msgr2=0x7f100c133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.237+0000 7f1011ef0640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f100800c5e0 tx=0x7f1008007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008012840 con 0x7f100c06eb30 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: dump: 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.256 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 5 monclient: authenticate success, global_id 234129 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 10 monclient: _renew_subs 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10080129c0 con 0x7f100c06eb30 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c121360 con 0x7f100c06eb30 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008012d80 con 0x7f100c06eb30 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 10 monclient: dump: 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.258 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f10037fe640 4 set_mon_vals no callback set 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 10 monclient: _renew_subs 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c141fc0 con 0x7f100c06eb30 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.238+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1008011590 con 0x7f100c06eb30 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1001ffb640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ff0038740 0x7f0ff003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1014154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1014154640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 0x7f100c11e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c063bb0 con 0x7f100c1341c0 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f100804c730 con 0x7f100c06eb30 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 0x7f100c11e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient: get_auth_request con 0x7f100c1341c0 auth_method 0 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10116ef640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ff0038740 0x7f0ff003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10116ef640 10 monclient: get_auth_request con 0x7f0ff0038740 auth_method 0 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 10 monclient(hunting): handle_auth_done global_id 214301 payload 824 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.239+0000 7f10126f1640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f100c1211c0 con 0x7f100c1341c0 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.241+0000 7f10126f1640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 msgr2=0x7f100c11e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.241+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 0x7f100c11e4d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f100c068b30 tx=0x7f1004036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.241+0000 7f10116ef640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ff0038740 0x7f0ff003abf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00ac20 tx=0x7f0ffc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.241+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004044410 con 0x7f100c1341c0 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.241+0000 7f1001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 10 monclient: dump: 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.261 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1004040070 con 0x7f100c1341c0 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 msgr2=0x7f100c11e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 0x7f100c11e4d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f100c068b30 tx=0x7f1004036000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 msgr2=0x7f100c11e4d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.242+0000 7f1001ffb640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c1341c0 0x7f100c11e4d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f1014154640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f1014154640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 0x7f100c0a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f1014154640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c134590 con 0x7f100c11ea10 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 0x7f100c0a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient: get_auth_request con 0x7f100c11ea10 auth_method 0 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 10 monclient(hunting): handle_auth_done global_id 214304 payload 824 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.248+0000 7f10126f1640 1 -- 172.21.15.17:0/2209203369 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f100c11ee10 con 0x7f100c11ea10 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f10126f1640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 msgr2=0x7f100c0a1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f10126f1640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 0x7f100c0a1c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10040073d0 tx=0x7f100403e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004049070 con 0x7f100c11ea10 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f1001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f1001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f1001ffb640 10 monclient: dump: 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.250+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f10040079d0 con 0x7f100c11ea10 2024-09-15T02:10:34.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 msgr2=0x7f100c0a1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 0x7f100c0a1c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10040073d0 tx=0x7f100403e070 comp rx=0 tx=0).stop 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 msgr2=0x7f100c0a1c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1001ffb640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f100c11ea10 0x7f100c0a1c80 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10040073d0 tx=0x7f100403e070 comp rx=0 tx=0).stop 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:34.267 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.268 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.269 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.270 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:34.271 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.272 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.274 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:34.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.278 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:10:34.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:34.280 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ff0038740 msgr2=0x7f0ff003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0ff0038740 0x7f0ff003abf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00ac20 tx=0x7f0ffc008040 comp rx=0 tx=0).stop 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 10 monclient: shutdown 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 msgr2=0x7f100c133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f10126f1640 1 -- 172.21.15.17:0/2209203369 reap_dead start 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f100800c5e0 tx=0x7f1008007ed0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 shutdown_connections 2024-09-15T02:10:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 --2- 172.21.15.17:0/2209203369 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f100c06eb30 0x7f100c133c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 >> 172.21.15.17:0/2209203369 conn(0x7f100c071300 msgr2=0x7f100c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 shutdown_connections 2024-09-15T02:10:34.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.251+0000 7f1014154640 1 -- 172.21.15.17:0/2209203369 wait complete. 2024-09-15T02:10:34.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:34.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:10:34.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.379+0000 7f471a500640 10 monclient: get_monmap_and_config 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.379+0000 7f471a500640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.379+0000 7f471a500640 10 monclient: monmap: 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.383 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:34.380259+0000 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:34.380259+0000 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 Processor -- start 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 -- start start 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient: init 2024-09-15T02:10:34.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): picked mon.noname-b con 0x7f471406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f471406f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f471409e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4714069c60 con 0x7f471409e970 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471409f690 con 0x7f471406f440 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471409edd0 con 0x7f471406eb30 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f471a500640 10 monclient(hunting): authenticate will time out at 1894.376734s 2024-09-15T02:10:34.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4718a9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f471409e970 0x7f471409c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34554/0 (socket says 172.21.15.17:34554) 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47824/0 (socket says 172.21.15.17:47824) 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4718a9d640 1 -- 172.21.15.17:0/3751143321 learned_addr learned my addr 172.21.15.17:0/3751143321 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): get_auth_request con 0x7f471406eb30 auth_method 0 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.380+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 10 monclient(hunting): handle_auth_done global_id 214307 payload 274 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 1 -- 172.21.15.17:0/3751143321 >> v1:172.21.15.67:6789/0 conn(0x7f471406f440 legacy=0x7f471409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 1 -- 172.21.15.17:0/3751143321 >> v1:172.21.15.17:6789/0 conn(0x7f471409e970 legacy=0x7f471409c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 1 monclient: found mon.noname-b 2024-09-15T02:10:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.381+0000 7f4713fff640 1 -- 172.21.15.17:0/3751143321 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471409ad40 con 0x7f471406eb30 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4713fff640 1 -- 172.21.15.17:0/3751143321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 msgr2=0x7f471406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4713fff640 1 --2- 172.21.15.17:0/3751143321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4704009880 tx=0x7f47040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef0c526e67fc9ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 1 -- 172.21.15.17:0/3751143321 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704032440 con 0x7f471406eb30 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: dump: 2024-09-15T02:10:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 1 -- 172.21.15.17:0/3751143321 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47040325a0 con 0x7f471406eb30 2024-09-15T02:10:34.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 5 monclient: authenticate success, global_id 214307 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 1 -- 172.21.15.17:0/3751143321 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704032940 con 0x7f471406eb30 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 4 set_mon_vals no callback set 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f4712ffd640 10 monclient: dump: 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.395 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 10 monclient: shutdown 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 1 -- 172.21.15.17:0/3751143321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 msgr2=0x7f471406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.384+0000 7f471a500640 1 --2- 172.21.15.17:0/3751143321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4704009880 tx=0x7f47040090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 -- 172.21.15.17:0/3751143321 shutdown_connections 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 --2- 172.21.15.17:0/3751143321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406eb30 0x7f471406ef00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.396 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 -- 172.21.15.17:0/3751143321 >> 172.21.15.17:0/3751143321 conn(0x7f47140712f0 msgr2=0x7f47140a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 -- 172.21.15.17:0/3751143321 shutdown_connections 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 -- 172.21.15.17:0/3751143321 wait complete. 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 10 monclient: monmap: 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:34.397 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.385+0000 7f471a500640 1 Processor -- start 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 -- start start 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient: init 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): picked mon.noname-a con 0x7f471406eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406f440 0x7f4714133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): picked mon.noname-c con 0x7f471406f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471409e970 0x7f471411da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): picked mon.noname-b con 0x7f471409e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471411fb60 con 0x7f471406eb30 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714121380 con 0x7f471409e970 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4714134cb0 con 0x7f471406f440 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f471a500640 10 monclient(hunting): authenticate will time out at 1894.382546s 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406f440 0x7f4714133f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49006/0 (socket says 172.21.15.17:49006) 2024-09-15T02:10:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 -- 172.21.15.17:0/3428861555 learned_addr learned my addr 172.21.15.17:0/3428861555 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): get_auth_request con 0x7f471406eb30 auth_method 0 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): get_auth_request con 0x7f471406f440 auth_method 0 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471409e970 0x7f471411da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): get_auth_request con 0x7f471409e970 auth_method 0 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4718a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient(hunting): handle_auth_done global_id 234132 payload 824 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406f440 msgr2=0x7f4714133f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406f440 0x7f4714133f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471409e970 msgr2=0x7f471411da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471409e970 0x7f471411da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.386+0000 7f4713fff640 1 monclient: found mon.noname-a 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.387+0000 7f4713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.387+0000 7f4713fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.387+0000 7f4713fff640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4704009390 con 0x7f471406eb30 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.387+0000 7f47137fe640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f471406f440 0x7f4714133f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f4713fff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 msgr2=0x7f4714141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f4713fff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4704009880 tx=0x7f4704032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704031910 con 0x7f471406eb30 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: dump: 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.403 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4704031a90 con 0x7f471406eb30 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704031e50 con 0x7f471406eb30 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f46fbfff640 10 monclient: dump: 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.404 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 5 monclient: authenticate success, global_id 234132 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 10 monclient: _renew_subs 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4714141cd0 con 0x7f471406eb30 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f47127fc640 4 set_mon_vals no callback set 2024-09-15T02:10:34.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 10 monclient: _renew_subs 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.389+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471411f5b0 con 0x7f471406eb30 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f471a500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f471a500640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 0x7f471411df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714063b90 con 0x7f4714134490 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4704039040 con 0x7f471406eb30 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f46fbfff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4700038740 0x7f470003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 0x7f471411df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient: get_auth_request con 0x7f4714134490 auth_method 0 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4704041030 con 0x7f471406eb30 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f4718a9d640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4700038740 0x7f470003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f4718a9d640 10 monclient: get_auth_request con 0x7f4700038740 auth_method 0 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 10 monclient(hunting): handle_auth_done global_id 204400 payload 824 2024-09-15T02:10:34.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.390+0000 7f47137fe640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4714121550 con 0x7f4714134490 2024-09-15T02:10:34.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f47137fe640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 msgr2=0x7f471411df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f47137fe640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 0x7f471411df60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f470800fb00 tx=0x7f4708007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f4718a9d640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4700038740 0x7f470003abf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f470c006ce0 tx=0x7f470c0052b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47080026c0 con 0x7f4714134490 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 10 monclient: dump: 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.410 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f470800dda0 con 0x7f4714134490 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 msgr2=0x7f471411df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 0x7f471411df60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f470800fb00 tx=0x7f4708007e10 comp rx=0 tx=0).stop 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 msgr2=0x7f471411df60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.393+0000 7f46fbfff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4714134490 0x7f471411df60 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f470800fb00 tx=0x7f4708007e10 comp rx=0 tx=0).stop 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f471a500640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f471a500640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 0x7f47140a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f471a500640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714134860 con 0x7f471411e4a0 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 0x7f47140a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient: get_auth_request con 0x7f471411e4a0 auth_method 0 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.399+0000 7f47137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.400+0000 7f47137fe640 10 monclient(hunting): handle_auth_done global_id 204403 payload 824 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.400+0000 7f47137fe640 1 -- 172.21.15.17:0/3428861555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f471411e8a0 con 0x7f471411e4a0 2024-09-15T02:10:34.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f47137fe640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 msgr2=0x7f47140a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f47137fe640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 0x7f47140a1530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4708019000 tx=0x7f470801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470801ec50 con 0x7f471411e4a0 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 10 monclient: dump: 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.413 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f470801edb0 con 0x7f471411e4a0 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 msgr2=0x7f47140a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 0x7f47140a1530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4708019000 tx=0x7f470801a000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 msgr2=0x7f47140a1530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.402+0000 7f46fbfff640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471411e4a0 0x7f47140a1530 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4708019000 tx=0x7f470801a000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:10:34.415 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.416 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:34.418 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:34.419 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.420 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.421 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:34.422 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.423 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:34.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.426 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:10:34.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:34.428 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:34.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4700038740 msgr2=0x7f470003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4700038740 0x7f470003abf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f470c006ce0 tx=0x7f470c0052b0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 10 monclient: shutdown 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 msgr2=0x7f4714141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4704009880 tx=0x7f4704032000 comp rx=0 tx=0).stop 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f4718a9d640 1 -- 172.21.15.17:0/3428861555 reap_dead start 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 shutdown_connections 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 --2- 172.21.15.17:0/3428861555 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f471406eb30 0x7f4714141370 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 >> 172.21.15.17:0/3428861555 conn(0x7f47140712f0 msgr2=0x7f4714134ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 shutdown_connections 2024-09-15T02:10:34.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.403+0000 7f471a500640 1 -- 172.21.15.17:0/3428861555 wait complete. 2024-09-15T02:10:34.431 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:34.431 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:10:34.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:10:34.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 10 monclient: get_monmap_and_config 2024-09-15T02:10:34.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 10 monclient: monmap: 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:34.526930+0000 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:34.526930+0000 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:34.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 1 Processor -- start 2024-09-15T02:10:34.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 1 -- start start 2024-09-15T02:10:34.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.526+0000 7fa3b30af640 10 monclient: init 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3ac131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3ac0a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa3ac137e60 0x7fa3ac13a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac137e60 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0a0040 con 0x7fa3ac131b20 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac09bff0 con 0x7fa3ac0a0280 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0a14a0 con 0x7fa3ac137e60 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b30af640 10 monclient(hunting): authenticate will time out at 1894.523398s 2024-09-15T02:10:34.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa3ac137e60 0x7fa3ac13a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b0e4b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fa3ac131b20 0x7fa3ac09eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34556/0 (socket says 172.21.15.17:34556) 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b0e4b640 1 -- 172.21.15.17:0/1739425382 learned_addr learned my addr 172.21.15.17:0/1739425382 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): get_auth_request con 0x7fa3ac137e60 auth_method 0 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient(hunting): handle_auth_done global_id 234135 payload 274 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 1 -- 172.21.15.17:0/1739425382 >> v1:172.21.15.67:6789/0 conn(0x7fa3ac0a0280 legacy=0x7fa3ac131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 1 -- 172.21.15.17:0/1739425382 >> v1:172.21.15.17:6789/0 conn(0x7fa3ac131b20 legacy=0x7fa3ac09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:34.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 1 monclient: found mon.noname-a 2024-09-15T02:10:34.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.527+0000 7fa3b164c640 1 -- 172.21.15.17:0/1739425382 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac09f3b0 con 0x7fa3ac137e60 2024-09-15T02:10:34.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b164c640 1 -- 172.21.15.17:0/1739425382 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac137e60 msgr2=0x7fa3ac13a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b164c640 1 --2- 172.21.15.17:0/1739425382 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac137e60 0x7fa3ac13a240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009850 tx=0x7fa3a40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86595d8890e888a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 1 -- 172.21.15.17:0/1739425382 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4032440 con 0x7fa3ac137e60 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: dump: 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.539 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b30af640 5 monclient: authenticate success, global_id 234135 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b30af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 1 -- 172.21.15.17:0/1739425382 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a40325a0 con 0x7fa3ac137e60 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b30af640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3b30af640 4 set_mon_vals no callback set 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 1 -- 172.21.15.17:0/1739425382 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4032940 con 0x7fa3ac137e60 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.530+0000 7fa3ab7fe640 10 monclient: dump: 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 10 monclient: shutdown 2024-09-15T02:10:34.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 -- 172.21.15.17:0/1739425382 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac137e60 msgr2=0x7fa3ac13a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 --2- 172.21.15.17:0/1739425382 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac137e60 0x7fa3ac13a240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009850 tx=0x7fa3a40090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 -- 172.21.15.17:0/1739425382 shutdown_connections 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 --2- 172.21.15.17:0/1739425382 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac137e60 0x7fa3ac13a240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 -- 172.21.15.17:0/1739425382 >> 172.21.15.17:0/1739425382 conn(0x7fa3ac071270 msgr2=0x7fa3ac0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 -- 172.21.15.17:0/1739425382 shutdown_connections 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 1 -- 172.21.15.17:0/1739425382 wait complete. 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 10 monclient: build_initial_monmap 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.531+0000 7fa3b30af640 10 monclient: monmap: 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:34.542 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 Processor -- start 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 -- start start 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient: init 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 0x7fa3ac09b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac0a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:34.543 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3ac137e60 0x7fa3ac1341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac09eed0 con 0x7fa3ac131b20 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac135f30 con 0x7fa3ac137e60 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac09b7c0 con 0x7fa3ac0a0280 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b30af640 10 monclient(hunting): authenticate will time out at 1894.528938s 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.544 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.532+0000 7fa3b0e4b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 0x7fa3ac09b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49032/0 (socket says 172.21.15.17:49032) 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 0x7fa3ac09b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47848/0 (socket says 172.21.15.17:47848) 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 -- 172.21.15.17:0/3700922815 learned_addr learned my addr 172.21.15.17:0/3700922815 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): get_auth_request con 0x7fa3ac131b20 auth_method 0 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): get_auth_request con 0x7fa3ac0a0280 auth_method 0 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3ac137e60 0x7fa3ac1341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): get_auth_request con 0x7fa3ac137e60 auth_method 0 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b0e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3b164c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient(hunting): handle_auth_done global_id 234138 payload 824 2024-09-15T02:10:34.546 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 -- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 msgr2=0x7fa3ac09b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 0x7fa3ac09b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 -- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3ac137e60 msgr2=0x7fa3ac1341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3ac137e60 0x7fa3ac1341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 monclient: found mon.noname-a 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.533+0000 7fa3abfff640 1 -- 172.21.15.17:0/3700922815 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4009390 con 0x7fa3ac131b20 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3abfff640 1 -- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 msgr2=0x7fa3ac133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3abfff640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa39400c610 tx=0x7fa394007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa394012840 con 0x7fa3ac131b20 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.547 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: dump: 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.548 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: _finish_auth 0 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3940129e0 con 0x7fa3ac131b20 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 5 monclient: authenticate success, global_id 234138 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 10 monclient: _renew_subs 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac0a09b0 con 0x7fa3ac131b20 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa394012da0 con 0x7fa3ac131b20 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3a8ff9640 10 monclient: dump: 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:34.549 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3aaffd640 4 set_mon_vals no callback set 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 10 monclient: _renew_subs 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.550 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.535+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac135c80 con 0x7fa3ac131b20 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa394011590 con 0x7fa3ac131b20 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3a8ff9640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 0x7fa38c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa39404be30 con 0x7fa3ac131b20 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3b30af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3b30af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3b30af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.536+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac1361c0 con 0x7fa3ac131b20 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.537+0000 7fa3b0e4b640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 0x7fa38c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:34.551 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.537+0000 7fa3b0e4b640 10 monclient: get_auth_request con 0x7fa38c038790 auth_method 0 2024-09-15T02:10:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.539+0000 7fa3b0e4b640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 0x7fa38c03ac40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa39c006df0 tx=0x7fa39c005330 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.541+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa39400e890 con 0x7fa3ac131b20 2024-09-15T02:10:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.541+0000 7fa3a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:34.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.541+0000 7fa3a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:34.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.766+0000 7fa3b30af640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:10:34.767 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.766+0000 7fa3b30af640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:10:34.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.766+0000 7fa3b30af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:34.768 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.766+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa3ac0724a0 con 0x7fa3ac131b20 2024-09-15T02:10:34.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.767+0000 7fa3a8ff9640 1 -- 172.21.15.17:0/3700922815 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa39400e290 con 0x7fa3ac131b20 2024-09-15T02:10:34.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.767+0000 7fa3a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:10:34.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.767+0000 7fa3a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:34.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3014515987,config=3140514009,health=536113986,kv=1529054990,logm=2545992832}) 2024-09-15T02:10:34.771 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=1869466384,config=3140514009,health=536113986,kv=1529054990,logm=1639490912}) 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3014515987,config=3140514009,health=536113986,kv=1529054990,logm=2545992832}) 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.767+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=8,logm=54} crc {auth=1089286002,config=3140514009,health=536113986,kv=1529054990,logm=910394125}) 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753751257}) 2024-09-15T02:10:34.772 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741919692}) 2024-09-15T02:10:34.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753751257}) 2024-09-15T02:10:34.773 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.768+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830669771}) 2024-09-15T02:10:34.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 msgr2=0x7fa38c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 0x7fa38c03ac40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa39c006df0 tx=0x7fa39c005330 comp rx=0 tx=0).stop 2024-09-15T02:10:34.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 10 monclient: shutdown 2024-09-15T02:10:34.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:34.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 msgr2=0x7fa3ac133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:34.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa39400c610 tx=0x7fa394007ed0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 shutdown_connections 2024-09-15T02:10:34.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa38c038790 0x7fa38c03ac40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa3ac131b20 0x7fa3ac133c90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3ac137e60 0x7fa3ac1341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 --2- 172.21.15.17:0/3700922815 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa3ac0a0280 0x7fa3ac09b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:34.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 >> 172.21.15.17:0/3700922815 conn(0x7fa3ac071270 msgr2=0x7fa3ac0a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:34.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 shutdown_connections 2024-09-15T02:10:34.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619999316}) 2024-09-15T02:10:34.777 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3185836000}) 2024-09-15T02:10:34.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:34.769+0000 7fa3b30af640 1 -- 172.21.15.17:0/3700922815 wait complete. 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619999316}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.769+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259575995}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613088062}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257269216}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613088062}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.770+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688625605}) 2024-09-15T02:10:34.784 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919566015}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390165363}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919566015}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.771+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3672092651}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260364421}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2006095614}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260364421}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.772+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646272663}) 2024-09-15T02:10:34.785 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766525344}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153851024}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766525344}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777425988}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.773+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430374}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=572382629}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430374}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629351950}) 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.786 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946827847}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020133815}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946827847}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.774+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869506326}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599014987}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941686833}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599014987}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.775+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847907475}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871898414}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279374126}) 2024-09-15T02:10:34.787 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871898414}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924738694}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034453772}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=641539110}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034453772}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.776+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868748611}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613704806}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915776483}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613704806}) 2024-09-15T02:10:34.788 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.777+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2255162641}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160995319}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000080707}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160995319}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500558066}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944652429}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075705506}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944652429}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.778+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167609773}) 2024-09-15T02:10:34.789 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148123544}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895163535}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148123544}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.779+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923476249}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=420749684}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630120271}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=420749684}) 2024-09-15T02:10:34.790 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.780+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762965306}) 2024-09-15T02:10:34.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.791 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768076464}) 2024-09-15T02:10:34.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=66347628}) 2024-09-15T02:10:34.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768076464}) 2024-09-15T02:10:34.792 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.781+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967676208}) 2024-09-15T02:10:34.793 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:10:34.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187804890}) 2024-09-15T02:10:34.799 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686332991}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187804890}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540053271}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740718814}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113159517}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740718814}) 2024-09-15T02:10:34.800 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.782+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007406173}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708050528}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655884759}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708050528}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.783+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789883189}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559920685}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461982096}) 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.801 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559920685}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135030040}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655852391}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119382733}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655852391}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.784+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942867843}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015733253}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795648359}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015733253}) 2024-09-15T02:10:34.802 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.785+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732329178}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013974319}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562313581}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013974319}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281981164}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628621771}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786922520}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628621771}) 2024-09-15T02:10:34.803 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.786+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452535347}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988649299}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325806786}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988649299}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.787+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076112584}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166083773}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52766232}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166083773}) 2024-09-15T02:10:34.804 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.788+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013232858}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792178257}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2486100247}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792178257}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.789+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2910544023}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047865887}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277573904}) 2024-09-15T02:10:34.805 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047865887}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579345655}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662628444}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808892885}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662628444}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.790+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169974977}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384082334}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=145418141}) 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.806 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384082334}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.791+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922184094}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143580604}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060587766}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143580604}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614209377}) 2024-09-15T02:10:34.807 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286464364}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878003118}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286464364}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.792+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1570347710}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122571495}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225575002}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122571495}) 2024-09-15T02:10:34.808 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.793+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188202683}) 2024-09-15T02:10:34.809 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745532472}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3857768872}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745532472}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.801+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594990602}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314694514}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=295078891}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314694514}) 2024-09-15T02:10:34.810 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.802+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096217311}) 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567385676}) 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902713347}) 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567385676}) 2024-09-15T02:10:34.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.942+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543308050}) 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.944+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.944+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891085458}) 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.944+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1759221280}) 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.944+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.944+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891085458}) 2024-09-15T02:10:34.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:34.945+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649945027}) 2024-09-15T02:10:35.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001147526}) 2024-09-15T02:10:35.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454070725}) 2024-09-15T02:10:35.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.182 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001147526}) 2024-09-15T02:10:35.183 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.180+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219722243}) 2024-09-15T02:10:35.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.190 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2744327063}) 2024-09-15T02:10:35.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449799660}) 2024-09-15T02:10:35.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2744327063}) 2024-09-15T02:10:35.191 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.188+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057367150}) 2024-09-15T02:10:35.554 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.554 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889039405}) 2024-09-15T02:10:35.555 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689860571}) 2024-09-15T02:10:35.555 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.555 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889039405}) 2024-09-15T02:10:35.555 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.552+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571359912}) 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670312159}) 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536433171}) 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670312159}) 2024-09-15T02:10:35.632 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.630+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830396116}) 2024-09-15T02:10:35.794 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:10:35.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 10 monclient: get_monmap_and_config 2024-09-15T02:10:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 10 monclient: build_initial_monmap 2024-09-15T02:10:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 10 monclient: monmap: 2024-09-15T02:10:35.903 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:35.902992+0000 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:35.902992+0000 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:35.904 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 1 Processor -- start 2024-09-15T02:10:35.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 1 -- start start 2024-09-15T02:10:35.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.902+0000 7fcd2ad14640 10 monclient: init 2024-09-15T02:10:35.906 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd24131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd240a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 0x7fcd2413a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-b con 0x7fcd24137e60 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd240a0040 con 0x7fcd24131b20 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2409bff0 con 0x7fcd240a0280 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240a14a0 con 0x7fcd24137e60 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd2ad14640 10 monclient(hunting): authenticate will time out at 1895.899503s 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd28ab0640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fcd24131b20 0x7fcd2409eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34560/0 (socket says 172.21.15.17:34560) 2024-09-15T02:10:35.907 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd28ab0640 1 -- 172.21.15.17:0/3969747495 learned_addr learned my addr 172.21.15.17:0/3969747495 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 1 --2- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 0x7fcd2413a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): get_auth_request con 0x7fcd24137e60 auth_method 0 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient(hunting): handle_auth_done global_id 214316 payload 274 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 1 -- 172.21.15.17:0/3969747495 >> v1:172.21.15.67:6789/0 conn(0x7fcd240a0280 legacy=0x7fcd24131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 1 -- 172.21.15.17:0/3969747495 >> v1:172.21.15.17:6789/0 conn(0x7fcd24131b20 legacy=0x7fcd2409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient: _finish_hunting 0 2024-09-15T02:10:35.908 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 1 monclient: found mon.noname-b 2024-09-15T02:10:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:35.909 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.903+0000 7fcd292b1640 1 -- 172.21.15.17:0/3969747495 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2409f3b0 con 0x7fcd24137e60 2024-09-15T02:10:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.906+0000 7fcd292b1640 1 -- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 msgr2=0x7fcd2413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.906+0000 7fcd292b1640 1 --2- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 0x7fcd2413a240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009e90 tx=0x7fcd1c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2defea630c4b1c09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 1 -- 172.21.15.17:0/3969747495 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c007840 con 0x7fcd24137e60 2024-09-15T02:10:35.913 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: dump: 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.914 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:35.915 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 1 -- 172.21.15.17:0/3969747495 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c02d050 con 0x7fcd24137e60 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 5 monclient: authenticate success, global_id 214316 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:35.916 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 4 set_mon_vals no callback set 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 1 -- 172.21.15.17:0/3969747495 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c007bd0 con 0x7fcd24137e60 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd237fe640 10 monclient: dump: 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:35.917 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 10 monclient: shutdown 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 msgr2=0x7fcd2413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 0x7fcd2413a240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009e90 tx=0x7fcd1c009780 comp rx=0 tx=0).stop 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3969747495 shutdown_connections 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3969747495 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd24137e60 0x7fcd2413a240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3969747495 >> 172.21.15.17:0/3969747495 conn(0x7fcd24071270 msgr2=0x7fcd240a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3969747495 shutdown_connections 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.907+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3969747495 wait complete. 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.908+0000 7fcd2ad14640 10 monclient: build_initial_monmap 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.908+0000 7fcd2ad14640 10 monclient: monmap: 2024-09-15T02:10:35.918 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.908+0000 7fcd2ad14640 1 Processor -- start 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.908+0000 7fcd2ad14640 1 -- start start 2024-09-15T02:10:35.919 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.908+0000 7fcd2ad14640 10 monclient: init 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-c con 0x7fcd240a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 0x7fcd2412c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-a con 0x7fcd24131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24137e60 0x7fcd2412c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:35.920 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): picked mon.noname-b con 0x7fcd24137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24096b40 con 0x7fcd24131b20 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240a09b0 con 0x7fcd24137e60 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:35.921 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2409eed0 con 0x7fcd240a0280 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd2ad14640 10 monclient(hunting): authenticate will time out at 1895.905316s 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 0x7fcd2412c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47870/0 (socket says 172.21.15.17:47870) 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 1 -- 172.21.15.17:0/3018628403 learned_addr learned my addr 172.21.15.17:0/3018628403 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): get_auth_request con 0x7fcd240a0280 auth_method 0 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): get_auth_request con 0x7fcd24131b20 auth_method 0 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:35.922 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24137e60 0x7fcd2412c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): get_auth_request con 0x7fcd24137e60 auth_method 0 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:35.923 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd292b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient(hunting): handle_auth_done global_id 234141 payload 824 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 -- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 msgr2=0x7fcd2409b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 -- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24137e60 msgr2=0x7fcd2412c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24137e60 0x7fcd2412c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 monclient: found mon.noname-a 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd23fff640 1 -- 172.21.15.17:0/3018628403 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c009a20 con 0x7fcd24131b20 2024-09-15T02:10:35.924 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.909+0000 7fcd28ab0640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556333991}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1520511107}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556333991}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.910+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782495803}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772694192}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778096351}) 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.926 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772694192}) 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935789695}) 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386128132}) 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204649503}) 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386128132}) 2024-09-15T02:10:35.927 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.911+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774520013}) 2024-09-15T02:10:35.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd23fff640 1 -- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 msgr2=0x7fcd2412c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:35.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd23fff640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 0x7fcd2412c020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c510 tx=0x7fcd0c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:35.928 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0117a0 con 0x7fcd24131b20 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: dump: 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:35.929 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: _finish_auth 0 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c011920 con 0x7fcd24131b20 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 5 monclient: authenticate success, global_id 234141 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 10 monclient: _renew_subs 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2412da40 con 0x7fcd24131b20 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c011ce0 con 0x7fcd24131b20 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd20ff9640 10 monclient: dump: 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:35.930 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:35.931 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:35.931 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd22ffd640 4 set_mon_vals no callback set 2024-09-15T02:10:35.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 10 monclient: _renew_subs 2024-09-15T02:10:35.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:35.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.912+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd240971b0 con 0x7fcd24131b20 2024-09-15T02:10:35.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191784989}) 2024-09-15T02:10:35.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351940668}) 2024-09-15T02:10:35.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191784989}) 2024-09-15T02:10:35.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.912+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143728992}) 2024-09-15T02:10:35.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.913+0000 7fcd2ad14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:35.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.913+0000 7fcd2ad14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:35.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.913+0000 7fcd2ad14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:35.934 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.913+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2412f8a0 con 0x7fcd24131b20 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124769635}) 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222660096}) 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124769635}) 2024-09-15T02:10:35.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.913+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906637353}) 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399796765}) 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2217398562}) 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399796765}) 2024-09-15T02:10:35.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.914+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798739535}) 2024-09-15T02:10:35.936 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.915+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcd0c012630 con 0x7fcd24131b20 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660263461}) 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886581844}) 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660263461}) 2024-09-15T02:10:35.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.915+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647526274}) 2024-09-15T02:10:35.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.915+0000 7fcd20ff9640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 0x7fcd0403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:35.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.915+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fcd0c04ccc0 con 0x7fcd24131b20 2024-09-15T02:10:35.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.915+0000 7fcd28ab0640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 0x7fcd0403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:35.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.915+0000 7fcd28ab0640 10 monclient: get_auth_request con 0x7fcd04038740 auth_method 0 2024-09-15T02:10:35.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.918+0000 7fcd28ab0640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 0x7fcd0403abf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006fd0 tx=0x7fcd14004960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730804164}) 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932540399}) 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730804164}) 2024-09-15T02:10:35.939 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.920+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2709680302}) 2024-09-15T02:10:35.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.920+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcd0c01c460 con 0x7fcd24131b20 2024-09-15T02:10:35.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.920+0000 7fcd20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:35.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:35.920+0000 7fcd20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:35.943 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430079493}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1258833347}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430079493}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2746835541}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1896440947}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223650203}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1896440947}) 2024-09-15T02:10:35.944 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.921+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309286698}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628242265}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393614529}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628242265}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.922+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204519394}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200148149}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417220767}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200148149}) 2024-09-15T02:10:35.945 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986376281}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315356367}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047857206}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.923+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315356367}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239277572}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215187684}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211791118}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215187684}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.924+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59459081}) 2024-09-15T02:10:35.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092667532}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099364871}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092667532}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.925+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639250855}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650421958}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3567159838}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650421958}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.926+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149822818}) 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278354162}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3920065164}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278354162}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.927+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589262332}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181008404}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570974502}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181008404}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069040041}) 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583838700}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4056250337}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583838700}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.928+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573115013}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724134114}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813789970}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724134114}) 2024-09-15T02:10:35.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:35.929+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3586965475}) 2024-09-15T02:10:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd2ad14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd2ad14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd2ad14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd240724a0 con 0x7fcd24131b20 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd20ff9640 1 -- 172.21.15.17:0/3018628403 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fcd0c01bbb0 con 0x7fcd24131b20 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:10:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.149+0000 7fcd20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:36.152 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":192,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 msgr2=0x7fcd0403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 0x7fcd0403abf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006fd0 tx=0x7fcd14004960 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 10 monclient: shutdown 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 msgr2=0x7fcd2412c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 0x7fcd2412c020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c510 tx=0x7fcd0c007e10 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 shutdown_connections 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcd04038740 0x7fcd0403abf0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcd24131b20 0x7fcd2412c020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24137e60 0x7fcd2412c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 --2- 172.21.15.17:0/3018628403 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcd240a0280 0x7fcd2409b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 >> 172.21.15.17:0/3018628403 conn(0x7fcd24071270 msgr2=0x7fcd24129920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 shutdown_connections 2024-09-15T02:10:36.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.151+0000 7fcd2ad14640 1 -- 172.21.15.17:0/3018628403 wait complete. 2024-09-15T02:10:36.163 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:10:36.163 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:10:36.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:10:36.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516740260}) 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186581713}) 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516740260}) 2024-09-15T02:10:36.174 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.172+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470739760}) 2024-09-15T02:10:36.183 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.184 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162856651}) 2024-09-15T02:10:36.184 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292394943}) 2024-09-15T02:10:36.184 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.184 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162856651}) 2024-09-15T02:10:36.184 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.182+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525196652}) 2024-09-15T02:10:36.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.187 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138039253}) 2024-09-15T02:10:36.188 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757019439}) 2024-09-15T02:10:36.188 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.188 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138039253}) 2024-09-15T02:10:36.188 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.185+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3130980711}) 2024-09-15T02:10:36.194 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036462681}) 2024-09-15T02:10:36.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542943191}) 2024-09-15T02:10:36.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036462681}) 2024-09-15T02:10:36.195 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.193+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950609096}) 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.260+0000 7fe756080640 10 monclient: get_monmap_and_config 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.260+0000 7fe756080640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.260+0000 7fe756080640 10 monclient: monmap: 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:36.261690+0000 2024-09-15T02:10:36.262 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:36.261690+0000 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:36.263 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.261+0000 7fe756080640 1 Processor -- start 2024-09-15T02:10:36.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.261+0000 7fe756080640 1 -- start start 2024-09-15T02:10:36.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.261+0000 7fe756080640 10 monclient: init 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe75006e920 0x7fe75012fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): picked mon.noname-a con 0x7fe75006e920 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe75006ecf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe75006f0f0 0x7fe750097360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): picked mon.noname-c con 0x7fe75006f0f0 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75009aef0 con 0x7fe75006ecf0 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75006f5b0 con 0x7fe75006e920 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75009d720 con 0x7fe75006f0f0 2024-09-15T02:10:36.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe756080640 10 monclient(hunting): authenticate will time out at 1896.258123s 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe75006e920 0x7fe75012fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74effd640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe75006ecf0 0x7fe750096c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34576/0 (socket says 172.21.15.17:34576) 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74effd640 1 -- 172.21.15.17:0/912075133 learned_addr learned my addr 172.21.15.17:0/912075133 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request con 0x7fe75006e920 auth_method 0 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_done global_id 234144 payload 274 2024-09-15T02:10:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74ffff640 1 --2- 172.21.15.17:0/912075133 >> v2:172.21.15.67:3300/0 conn(0x7fe75006f0f0 0x7fe750097360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 -- 172.21.15.17:0/912075133 >> v2:172.21.15.67:3300/0 conn(0x7fe75006f0f0 msgr2=0x7fe750097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/912075133 >> v2:172.21.15.67:3300/0 conn(0x7fe75006f0f0 0x7fe750097360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 -- 172.21.15.17:0/912075133 >> v1:172.21.15.17:6789/0 conn(0x7fe75006ecf0 legacy=0x7fe750096c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 monclient: found mon.noname-a 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.262+0000 7fe74f7fe640 1 -- 172.21.15.17:0/912075133 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75009b0c0 con 0x7fe75006e920 2024-09-15T02:10:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74f7fe640 1 -- 172.21.15.17:0/912075133 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006e920 msgr2=0x7fe75012fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/912075133 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006e920 0x7fe75012fbf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7500689e0 tx=0x7fe7400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ea2ded9732253b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 1 -- 172.21.15.17:0/912075133 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740032440 con 0x7fe75006e920 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: dump: 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 1 -- 172.21.15.17:0/912075133 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400325a0 con 0x7fe75006e920 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe756080640 5 monclient: authenticate success, global_id 234144 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe756080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 1 -- 172.21.15.17:0/912075133 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740032940 con 0x7fe75006e920 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe74e7fc640 10 monclient: dump: 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe756080640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.265+0000 7fe756080640 4 set_mon_vals no callback set 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 10 monclient: shutdown 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 -- 172.21.15.17:0/912075133 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006e920 msgr2=0x7fe75012fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 --2- 172.21.15.17:0/912075133 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006e920 0x7fe75012fbf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7500689e0 tx=0x7fe7400090f0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 -- 172.21.15.17:0/912075133 shutdown_connections 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 --2- 172.21.15.17:0/912075133 >> v2:172.21.15.67:3300/0 conn(0x7fe75006f0f0 0x7fe750097360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 --2- 172.21.15.17:0/912075133 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006e920 0x7fe75012fbf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 -- 172.21.15.17:0/912075133 >> 172.21.15.17:0/912075133 conn(0x7fe7500a1000 msgr2=0x7fe7500a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 -- 172.21.15.17:0/912075133 shutdown_connections 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 1 -- 172.21.15.17:0/912075133 wait complete. 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.266+0000 7fe756080640 10 monclient: monmap: 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 Processor -- start 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 -- start start 2024-09-15T02:10:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient: init 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 0x7fe75012f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): picked mon.noname-c con 0x7fe75006e920 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 0x7fe750119b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): picked mon.noname-a con 0x7fe75006ecf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe75006f0f0 0x7fe75011a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): picked mon.noname-b con 0x7fe75006f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75011d2f0 con 0x7fe75006ecf0 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75011b9f0 con 0x7fe75006f0f0 2024-09-15T02:10:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75011b4e0 con 0x7fe75006e920 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe756080640 10 monclient(hunting): authenticate will time out at 1896.263786s 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 0x7fe75012f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 0x7fe750119b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 0x7fe75012f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47884/0 (socket says 172.21.15.17:47884) 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 0x7fe750119b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49060/0 (socket says 172.21.15.17:49060) 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 1 -- 172.21.15.17:0/4205881454 learned_addr learned my addr 172.21.15.17:0/4205881454 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request con 0x7fe75006e920 auth_method 0 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 10 monclient(hunting): get_auth_request con 0x7fe75006ecf0 auth_method 0 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.267+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74ffff640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe75006f0f0 0x7fe75011a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74ffff640 10 monclient(hunting): get_auth_request con 0x7fe75006f0f0 auth_method 0 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient(hunting): handle_auth_done global_id 234147 payload 824 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 msgr2=0x7fe75012f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 0x7fe75012f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe75006f0f0 msgr2=0x7fe75011a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe75006f0f0 0x7fe75011a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 monclient: found mon.noname-a 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74effd640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740009390 con 0x7fe75006ecf0 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.268+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe75006e920 0x7fe75012f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe74effd640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 msgr2=0x7fe750119b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe74effd640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 0x7fe750119b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe74400c490 tx=0x7fe744007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744005cb0 con 0x7fe75006ecf0 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: dump: 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe744005e30 con 0x7fe75006ecf0 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744007320 con 0x7fe75006ecf0 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe7377fe640 10 monclient: dump: 2024-09-15T02:10:36.284 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 5 monclient: authenticate success, global_id 234147 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 10 monclient: _renew_subs 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75011d010 con 0x7fe75006ecf0 2024-09-15T02:10:36.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe74dffb640 4 set_mon_vals no callback set 2024-09-15T02:10:36.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 10 monclient: _renew_subs 2024-09-15T02:10:36.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.270+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75011ded0 con 0x7fe75006ecf0 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe756080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe756080640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 0x7fe75011a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750063c20 con 0x7fe75002aa40 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe74401b3b0 con 0x7fe75006ecf0 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 0x7fe75011a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient: get_auth_request con 0x7fe75002aa40 auth_method 0 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.271+0000 7fe7377fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe73c038790 0x7fe73c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.272+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe74404bb10 con 0x7fe75006ecf0 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.272+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_done global_id 234150 payload 824 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.272+0000 7fe74f7fe640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe75002ae10 con 0x7fe75002aa40 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.272+0000 7fe74ffff640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe73c038790 0x7fe73c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.272+0000 7fe74ffff640 10 monclient: get_auth_request con 0x7fe73c038790 auth_method 0 2024-09-15T02:10:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe74f7fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 msgr2=0x7fe75011a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 0x7fe75011a5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe740009130 tx=0x7fe740032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740031e90 con 0x7fe75002aa40 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 10 monclient: dump: 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe74003c650 con 0x7fe75002aa40 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 msgr2=0x7fe75011a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 0x7fe75011a5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe740009130 tx=0x7fe740032000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 msgr2=0x7fe75011a5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe7377fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75002aa40 0x7fe75011a5a0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe740009130 tx=0x7fe740032000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.274+0000 7fe74ffff640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe73c038790 0x7fe73c03ac40 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe74800e4c0 tx=0x7fe7480076d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe756080640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe756080640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 0x7fe75005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe756080640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75011d7f0 con 0x7fe75005e310 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 0x7fe75005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient: get_auth_request con 0x7fe75005e310 auth_method 0 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_done global_id 234153 payload 824 2024-09-15T02:10:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.280+0000 7fe74f7fe640 1 -- 172.21.15.17:0/4205881454 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe7500724a0 con 0x7fe75005e310 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe74f7fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 msgr2=0x7fe75005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe74f7fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 0x7fe75005e710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe75009bc90 tx=0x7fe74003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740046a40 con 0x7fe75005e310 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 10 monclient: dump: 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7fe74003b3d0 con 0x7fe75005e310 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 msgr2=0x7fe75005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 0x7fe75005e710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe75009bc90 tx=0x7fe74003d000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 msgr2=0x7fe75005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe7377fe640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75005e310 0x7fe75005e710 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe75009bc90 tx=0x7fe74003d000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:36.296 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.297 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.298 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:36.299 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.300 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:36.308 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:36.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe73c038790 msgr2=0x7fe73c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe73c038790 0x7fe73c03ac40 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe74800e4c0 tx=0x7fe7480076d0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 10 monclient: shutdown 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 msgr2=0x7fe750119b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 --2- 172.21.15.17:0/4205881454 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe75006ecf0 0x7fe750119b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe74400c490 tx=0x7fe744007f00 comp rx=0 tx=0).stop 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe74ffff640 1 -- 172.21.15.17:0/4205881454 reap_dead start 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe74ffff640 1 -- 172.21.15.17:0/4205881454 reap_dead start 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 shutdown_connections 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 >> 172.21.15.17:0/4205881454 conn(0x7fe7500a1000 msgr2=0x7fe750069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 shutdown_connections 2024-09-15T02:10:36.310 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.283+0000 7fe756080640 1 -- 172.21.15.17:0/4205881454 wait complete. 2024-09-15T02:10:36.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:36.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:10:36.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270949817}) 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795334740}) 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270949817}) 2024-09-15T02:10:36.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.419+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689160515}) 2024-09-15T02:10:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 10 monclient: get_monmap_and_config 2024-09-15T02:10:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 10 monclient: monmap: 2024-09-15T02:10:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:36.447941+0000 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:36.447941+0000 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:36.449 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 1 Processor -- start 2024-09-15T02:10:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 1 -- start start 2024-09-15T02:10:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.447+0000 7f40b3100640 10 monclient: init 2024-09-15T02:10:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f40ac0a58b0 0x7f40ac132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-a con 0x7f40ac0a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40ac0980a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 0x7f40ac1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-b con 0x7f40ac136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac0a01e0 con 0x7f40ac0980a0 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac0a1580 con 0x7f40ac0a58b0 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac0a5c80 con 0x7f40ac136dd0 2024-09-15T02:10:36.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b3100640 10 monclient(hunting): authenticate will time out at 1896.444450s 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f40ac0a58b0 0x7f40ac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f40ac0a58b0 0x7f40ac132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49098/0 (socket says 172.21.15.17:49098) 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2295369942 learned_addr learned my addr 172.21.15.17:0/2295369942 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): get_auth_request con 0x7f40ac0a58b0 auth_method 0 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 0x7f40ac1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): get_auth_request con 0x7f40ac136dd0 auth_method 0 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_done global_id 234156 payload 274 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 msgr2=0x7f40ac1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 0x7f40ac1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2295369942 >> v1:172.21.15.67:6789/0 conn(0x7f40ac0980a0 legacy=0x7f40ac0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 monclient: found mon.noname-a 2024-09-15T02:10:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:10:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2295369942 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac098bb0 con 0x7f40ac0a58b0 2024-09-15T02:10:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.448+0000 7f40b169d640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 0x7f40ac1391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:36.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac0a58b0 msgr2=0x7f40ac132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40b0e9c640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac0a58b0 0x7f40ac132040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f409c009910 tx=0x7f409c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67f020cdec65935 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 1 -- 172.21.15.17:0/2295369942 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c032440 con 0x7f40ac0a58b0 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: dump: 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.461 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 1 -- 172.21.15.17:0/2295369942 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c0325a0 con 0x7f40ac0a58b0 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 1 -- 172.21.15.17:0/2295369942 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c032940 con 0x7f40ac0a58b0 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40b3100640 5 monclient: authenticate success, global_id 234156 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40b3100640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40b3100640 4 set_mon_vals no callback set 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.451+0000 7f40ab7fe640 10 monclient: dump: 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.462 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 10 monclient: shutdown 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 -- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac0a58b0 msgr2=0x7f40ac132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac0a58b0 0x7f40ac132040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f409c009910 tx=0x7f409c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 -- 172.21.15.17:0/2295369942 shutdown_connections 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac136dd0 0x7f40ac1391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 --2- 172.21.15.17:0/2295369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac0a58b0 0x7f40ac132040 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 -- 172.21.15.17:0/2295369942 >> 172.21.15.17:0/2295369942 conn(0x7f40ac070ed0 msgr2=0x7f40ac0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 -- 172.21.15.17:0/2295369942 shutdown_connections 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 1 -- 172.21.15.17:0/2295369942 wait complete. 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.452+0000 7f40b3100640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 10 monclient: monmap: 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 1 Processor -- start 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 1 -- start start 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 10 monclient: init 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.453+0000 7f40b3100640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-c con 0x7f40ac0980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:36.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0a58b0 0x7f40ac133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-b con 0x7f40ac0a58b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136dd0 0x7f40ac1335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): picked mon.noname-a con 0x7f40ac136dd0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac0a32a0 con 0x7f40ac136dd0 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac134520 con 0x7f40ac0a58b0 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac135ad0 con 0x7f40ac0980a0 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b3100640 10 monclient(hunting): authenticate will time out at 1896.450072s 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47908/0 (socket says 172.21.15.17:47908) 2024-09-15T02:10:36.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2775552779 learned_addr learned my addr 172.21.15.17:0/2775552779 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136dd0 0x7f40ac1335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): get_auth_request con 0x7f40ac0980a0 auth_method 0 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): get_auth_request con 0x7f40ac136dd0 auth_method 0 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b169d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40abfff640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0a58b0 0x7f40ac133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient(hunting): handle_auth_done global_id 214328 payload 824 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0a58b0 msgr2=0x7f40ac133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0a58b0 0x7f40ac133060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136dd0 msgr2=0x7f40ac1335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f40ac136dd0 0x7f40ac1335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 monclient: found mon.noname-c 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.454+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a4009a20 con 0x7f40ac0980a0 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b0e9c640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 msgr2=0x7f40ac132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b0e9c640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f409c035040 tx=0x7f409c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0319c0 con 0x7f40ac0980a0 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: dump: 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.469 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c031b60 con 0x7f40ac0980a0 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 5 monclient: authenticate success, global_id 214328 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 10 monclient: _renew_subs 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c009450 con 0x7f40ac0980a0 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c03d420 con 0x7f40ac0980a0 2024-09-15T02:10:36.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40a8ff9640 10 monclient: dump: 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 10 monclient: _renew_subs 2024-09-15T02:10:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac136790 con 0x7f40ac0980a0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.456+0000 7f40aaffd640 4 set_mon_vals no callback set 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40b3100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40b3100640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 0x7f40ac1371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac063b20 con 0x7f40ac133ae0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f409c039040 con 0x7f40ac0980a0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 0x7f40ac1371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient: get_auth_request con 0x7f40ac133ae0 auth_method 0 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40a8ff9640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4098038740 0x7f409803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.457+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f409c06c370 con 0x7f40ac0980a0 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.458+0000 7f40abfff640 10 monclient(hunting): handle_auth_done global_id 214331 payload 824 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.458+0000 7f40abfff640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40ac135e30 con 0x7f40ac133ae0 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.458+0000 7f40b169d640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4098038740 0x7f409803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.458+0000 7f40b169d640 10 monclient: get_auth_request con 0x7f4098038740 auth_method 0 2024-09-15T02:10:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40abfff640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 msgr2=0x7f40ac1371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40abfff640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 0x7f40ac1371a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0010790 con 0x7f40ac133ae0 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 10 monclient: dump: 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.476 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f40a00108f0 con 0x7f40ac133ae0 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 msgr2=0x7f40ac1371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 0x7f40ac1371a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0008040 comp rx=0 tx=0).stop 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 msgr2=0x7f40ac1371a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40a8ff9640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac133ae0 0x7f40ac1371a0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0008040 comp rx=0 tx=0).stop 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.460+0000 7f40b169d640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4098038740 0x7f409803abf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f40a400f700 tx=0x7f40a400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40b3100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40b3100640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 0x7f40ac137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40b3100640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac098e80 con 0x7f40ac1376e0 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 0x7f40ac137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient: get_auth_request con 0x7f40ac1376e0 auth_method 0 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 10 monclient(hunting): handle_auth_done global_id 214334 payload 824 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.466+0000 7f40abfff640 1 -- 172.21.15.17:0/2775552779 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f40ac0724a0 con 0x7f40ac1376e0 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40abfff640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 msgr2=0x7f40ac137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40abfff640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 0x7f40ac137ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a001a400 con 0x7f40ac1376e0 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 10 monclient: dump: 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.479 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f40a001a560 con 0x7f40ac1376e0 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 msgr2=0x7f40ac137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 0x7f40ac137ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0018000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 msgr2=0x7f40ac137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40a8ff9640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac1376e0 0x7f40ac137ae0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c400 tx=0x7f40a0018000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:36.480 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.481 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.482 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:36.483 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:36.484 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.485 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.486 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.489 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.490 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.491 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.492 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.493 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:10:36.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.495 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.495 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:36.495 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4098038740 msgr2=0x7f409803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4098038740 0x7f409803abf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f40a400f700 tx=0x7f40a400b040 comp rx=0 tx=0).stop 2024-09-15T02:10:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 10 monclient: shutdown 2024-09-15T02:10:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 msgr2=0x7f40ac132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f409c035040 tx=0x7f409c032000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b169d640 1 -- 172.21.15.17:0/2775552779 reap_dead start 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 shutdown_connections 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 --2- 172.21.15.17:0/2775552779 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f40ac0980a0 0x7f40ac132b20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 >> 172.21.15.17:0/2775552779 conn(0x7f40ac070ed0 msgr2=0x7f40ac068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 shutdown_connections 2024-09-15T02:10:36.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.469+0000 7f40b3100640 1 -- 172.21.15.17:0/2775552779 wait complete. 2024-09-15T02:10:36.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:36.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:10:36.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.593+0000 7fcb8a2dd640 10 monclient: get_monmap_and_config 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.593+0000 7fcb8a2dd640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.593+0000 7fcb8a2dd640 10 monclient: monmap: 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:36.594568+0000 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:36.594568+0000 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:36.596 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:36.596 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:36.596 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:36.596 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:36.596 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 1 Processor -- start 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 1 -- start start 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 10 monclient: init 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.594+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb8409c2c0 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb8409e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb8406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-b con 0x7fcb8406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8409ed80 con 0x7fcb8409e970 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8406a5c0 con 0x7fcb8409c2c0 2024-09-15T02:10:36.598 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84099f50 con 0x7fcb8406eb30 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8a2dd640 10 monclient(hunting): authenticate will time out at 1896.591051s 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb837fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fcb8409e970 0x7fcb8409c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:34584/0 (socket says 172.21.15.17:34584) 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb8406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb837fe640 1 -- 172.21.15.17:0/1040696989 learned_addr learned my addr 172.21.15.17:0/1040696989 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): get_auth_request con 0x7fcb8406eb30 auth_method 0 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.599 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient(hunting): handle_auth_done global_id 214337 payload 274 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 1 -- 172.21.15.17:0/1040696989 >> v1:172.21.15.67:6789/0 conn(0x7fcb8409c2c0 legacy=0x7fcb8409a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 1 -- 172.21.15.17:0/1040696989 >> v1:172.21.15.17:6789/0 conn(0x7fcb8409e970 legacy=0x7fcb8409c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 1 monclient: found mon.noname-b 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.600 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.595+0000 7fcb8887a640 1 -- 172.21.15.17:0/1040696989 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84068ac0 con 0x7fcb8406eb30 2024-09-15T02:10:36.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8887a640 1 -- 172.21.15.17:0/1040696989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 msgr2=0x7fcb8406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8887a640 1 --2- 172.21.15.17:0/1040696989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb8406ef70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009e90 tx=0x7fcb7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25add03f55cf1928 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 1 -- 172.21.15.17:0/1040696989 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007840 con 0x7fcb8406eb30 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: dump: 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.604 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 1 -- 172.21.15.17:0/1040696989 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c02d050 con 0x7fcb8406eb30 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 5 monclient: authenticate success, global_id 214337 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 4 set_mon_vals no callback set 2024-09-15T02:10:36.605 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 1 -- 172.21.15.17:0/1040696989 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007bd0 con 0x7fcb8406eb30 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb82ffd640 10 monclient: dump: 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.606 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 10 monclient: shutdown 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1040696989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 msgr2=0x7fcb8406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1040696989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb8406ef70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009e90 tx=0x7fcb7c009780 comp rx=0 tx=0).stop 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1040696989 shutdown_connections 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1040696989 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb8406ef70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.598+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1040696989 >> 172.21.15.17:0/1040696989 conn(0x7fcb840a2f70 msgr2=0x7fcb840a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.599+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1040696989 shutdown_connections 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.599+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1040696989 wait complete. 2024-09-15T02:10:36.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.599+0000 7fcb8a2dd640 10 monclient: build_initial_monmap 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.599+0000 7fcb8a2dd640 10 monclient: monmap: 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.599+0000 7fcb8a2dd640 1 Processor -- start 2024-09-15T02:10:36.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 -- start start 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient: init 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb84133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8406eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 0x7fcb8411d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8409c2c0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb8409e970 0x7fcb8411df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): picked mon.noname-b con 0x7fcb8409e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8409ce60 con 0x7fcb8409c2c0 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84120fa0 con 0x7fcb8409e970 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8411fe60 con 0x7fcb8406eb30 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8a2dd640 10 monclient(hunting): authenticate will time out at 1896.596536s 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 0x7fcb8411d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb84133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 0x7fcb8411d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:49118/0 (socket says 172.21.15.17:49118) 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb84133ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:47954/0 (socket says 172.21.15.17:47954) 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 1 -- 172.21.15.17:0/1842276332 learned_addr learned my addr 172.21.15.17:0/1842276332 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): get_auth_request con 0x7fcb8409c2c0 auth_method 0 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): get_auth_request con 0x7fcb8406eb30 auth_method 0 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb8409e970 0x7fcb8411df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): get_auth_request con 0x7fcb8409e970 auth_method 0 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.611 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb8887a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient(hunting): handle_auth_done global_id 214340 payload 824 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb8409e970 msgr2=0x7fcb8411df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb8409e970 0x7fcb8411df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 msgr2=0x7fcb8411d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 0x7fcb8411d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient: _finish_hunting 0 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 monclient: found mon.noname-c 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.612 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb83fff640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c009a20 con 0x7fcb8406eb30 2024-09-15T02:10:36.613 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.600+0000 7fcb837fe640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcb8409c2c0 0x7fcb8411d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb83fff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 msgr2=0x7fcb84133ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb83fff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb84133ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c510 tx=0x7fcb74007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740117a0 con 0x7fcb8406eb30 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: dump: 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.616 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: _finish_auth 0 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 5 monclient: authenticate success, global_id 214340 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74011920 con 0x7fcb8406eb30 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74011ce0 con 0x7fcb8406eb30 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 10 monclient: _renew_subs 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb84141e20 con 0x7fcb8406eb30 2024-09-15T02:10:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb6bfff640 10 monclient: dump: 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb827fc640 4 set_mon_vals no callback set 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 10 monclient: _renew_subs 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.603+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb84136450 con 0x7fcb8406eb30 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8a2dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb74012630 con 0x7fcb8406eb30 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 0x7fcb8411e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84120e60 con 0x7fcb84134940 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb6bfff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb70038740 0x7fcb7003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 0x7fcb8411e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient: get_auth_request con 0x7fcb84134940 auth_method 0 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fcb7404c220 con 0x7fcb8406eb30 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8887a640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb70038740 0x7fcb7003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb8887a640 10 monclient: get_auth_request con 0x7fcb70038740 auth_method 0 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 10 monclient(hunting): handle_auth_done global_id 204418 payload 824 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.604+0000 7fcb837fe640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb840724a0 con 0x7fcb84134940 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.606+0000 7fcb837fe640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 msgr2=0x7fcb8411e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.606+0000 7fcb837fe640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 0x7fcb8411e460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb780072f0 tx=0x7fcb78004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb8887a640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb70038740 0x7fcb7003abf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcb8406ab80 tx=0x7fcb7c037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7800fcf0 con 0x7fcb84134940 2024-09-15T02:10:36.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 10 monclient: dump: 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcb7800fe50 con 0x7fcb84134940 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 msgr2=0x7fcb8411e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 0x7fcb8411e460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb780072f0 tx=0x7fcb78004960 comp rx=0 tx=0).stop 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 msgr2=0x7fcb8411e460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.607+0000 7fcb6bfff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb84134940 0x7fcb8411e460 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb780072f0 tx=0x7fcb78004960 comp rx=0 tx=0).stop 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb8a2dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 0x7fcb8411e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb8a2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84063b90 con 0x7fcb841340e0 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 0x7fcb8411e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient: get_auth_request con 0x7fcb841340e0 auth_method 0 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 10 monclient(hunting): handle_auth_done global_id 204421 payload 824 2024-09-15T02:10:36.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.613+0000 7fcb837fe640 1 -- 172.21.15.17:0/1842276332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcb84067220 con 0x7fcb841340e0 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb837fe640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 msgr2=0x7fcb8411e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb837fe640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 0x7fcb8411e9a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb78018000 tx=0x7fcb78019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7801dca0 con 0x7fcb841340e0 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 10 monclient: dump: 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.624 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fcb7801de00 con 0x7fcb841340e0 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 msgr2=0x7fcb8411e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 0x7fcb8411e9a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb78018000 tx=0x7fcb78019000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 msgr2=0x7fcb8411e9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:10:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.616+0000 7fcb6bfff640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb841340e0 0x7fcb8411e9a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 192, 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:10:36.626 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 3, 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.627 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:10:36.629 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:10:36.630 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:10:36.632 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:10:36.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:10:36.636 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:10:36.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:10:36.638 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:10:36.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:10:36.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:10:36.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:10:36.639 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:10:36.639 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231661087}) 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748992694}) 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231661087}) 2024-09-15T02:10:36.640 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.616+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723867641}) 2024-09-15T02:10:36.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb70038740 msgr2=0x7fcb7003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.641 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fcb70038740 0x7fcb7003abf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcb8406ab80 tx=0x7fcb7c037000 comp rx=0 tx=0).stop 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 10 monclient: shutdown 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 msgr2=0x7fcb84133ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 --2- 172.21.15.17:0/1842276332 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fcb8406eb30 0x7fcb84133ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c510 tx=0x7fcb74007e10 comp rx=0 tx=0).stop 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8887a640 1 -- 172.21.15.17:0/1842276332 reap_dead start 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8887a640 1 -- 172.21.15.17:0/1842276332 reap_dead start 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 shutdown_connections 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 >> 172.21.15.17:0/1842276332 conn(0x7fcb840a2f70 msgr2=0x7fcb84134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 shutdown_connections 2024-09-15T02:10:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:36.617+0000 7fcb8a2dd640 1 -- 172.21.15.17:0/1842276332 wait complete. 2024-09-15T02:10:36.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:10:36.643 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T02:10:36.643 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-15T02:10:36.643 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T02:10:36.643 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:10:36.643 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:10:36.644 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710394577}) 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226494103}) 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710394577}) 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.637+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1657192474}) 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.647 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202966816}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2734879051}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202966816}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.638+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178279200}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549634866}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=849272728}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549634866}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.639+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772453659}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292829759}) 2024-09-15T02:10:36.648 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519750217}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292829759}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.640+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162634161}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740383786}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861353625}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740383786}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570209222}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864955509}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548220943}) 2024-09-15T02:10:36.649 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.641+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864955509}) 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737449037}) 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613045094}) 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358498687}) 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613045094}) 2024-09-15T02:10:36.650 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:10:36.642+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359627232}) 2024-09-15T02:10:36.662 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:10:36.662 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:10:36.662 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:10:36.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:10:56.663 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:10:56.664 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:10:56.664 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:10:56.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:10:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.771+0000 7f11ae034640 10 monclient: get_monmap_and_config 2024-09-15T02:10:56.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.771+0000 7f11ae034640 10 monclient: build_initial_monmap 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.771+0000 7f11ae034640 10 monclient: monmap: 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:10:56.772110+0000 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:10:56.772110+0000 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:10:56.773 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 Processor -- start 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 -- start start 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient: init 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient: _reopen_session rank -1 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-c con 0x7f11a806eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11a806f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11a809e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:10:56.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a809f6a0 con 0x7f11a809e970 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a809ed70 con 0x7f11a806f440 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a8069f40 con 0x7f11a806eb30 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11ae034640 10 monclient(hunting): authenticate will time out at 1916.768667s 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11a77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 msgr2=0x7f11a806ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11a6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f11a809bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11a7fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f11a809e970 0x7f11a8139b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:51340/0 (socket says 172.21.15.17:51340) 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 learned_addr learned my addr 172.21.15.17:0/3810780867 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:10:56.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.772+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:56.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.775+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.17:6789/0 conn(0x7f11a809e970 legacy=0x7f11a8139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:10:56.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.973+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 msgr2=0x7f11a806ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:10:56.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.973+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f11a809bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:10:56.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:56.973+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:10:57.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:57.374+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f11a809bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:10:57.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:57.374+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 msgr2=0x7f11a806ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:10:57.375 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:57.374+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:10:58.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:58.175+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f11a809bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:10:58.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:58.175+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 msgr2=0x7f11a806ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:10:58.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:58.175+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): tick 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 monclient(hunting): continuing hunt 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 msgr2=0x7f11a806ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.67:3300/0 conn(0x7f11a806eb30 0x7f11a806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:10:59.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f11a809bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.17:6789/0 conn(0x7f11a809e970 legacy=0x7f11a8139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient: _add_conns ranks=[2,3] 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f1194003700 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f1194006270 addr v2:172.21.15.17:3300/0 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a8069f40 con 0x7f1194006270 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1194008de0 con 0x7f1194003700 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a57fa640 10 monclient(hunting): _renew_subs 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:59.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 10 monclient(hunting): get_auth_request con 0x7f1194003700 auth_method 0 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:10:59.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.773+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 10 monclient(hunting): get_auth_request con 0x7f1194003700 auth_method 0 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:10:59.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:10:59.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:10:59.974+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 10 monclient(hunting): get_auth_request con 0x7f1194003700 auth_method 0 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:00.377 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:11:00.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:00.375+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:01.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 10 monclient(hunting): get_auth_request con 0x7f1194003700 auth_method 0 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:11:01.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:01.176+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): tick 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 monclient(hunting): continuing hunt 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 msgr2=0x7f1194005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1194003700 0x7f1194005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 msgr2=0x7f1194008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f1194008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T02:11:02.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11a806f440 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 reap_dead start 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11a809e970 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f11940100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f1194006270 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a8069950 con 0x7f11a809e970 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a809ed70 con 0x7f11a806f440 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 1 -- 172.21.15.17:0/3810780867 --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a8069f40 con 0x7f1194006270 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a57fa640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a77fe640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f119400b210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 1 --2- 172.21.15.17:0/3810780867 >> v2:172.21.15.17:3300/0 conn(0x7f1194006270 0x7f11940100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:11:02.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.773+0000 7f11a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a6ffd640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.17:6789/0 conn(0x7f11a809e970 legacy=0x7f119400e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:02.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient(hunting): handle_auth_done global_id 234165 payload 274 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.67:6789/0 conn(0x7f11a806f440 legacy=0x7f119400b210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 >> v1:172.21.15.17:6789/0 conn(0x7f11a809e970 legacy=0x7f119400e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 1 monclient: found mon.noname-a 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:02.778 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.775+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194008780 con 0x7f1194006270 2024-09-15T02:11:02.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a7fff640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 msgr2=0x7f11940100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:02.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a7fff640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11940100f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11a8067140 tx=0x7f11a0001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9454d8bb4b70fc65 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:02.780 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 1 -- 172.21.15.17:0/3810780867 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a002abf0 con 0x7f1194006270 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: dump: 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:02.781 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: _finish_auth 0 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 1 -- 172.21.15.17:0/3810780867 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a002ad50 con 0x7f1194006270 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 1 -- 172.21.15.17:0/3810780867 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00336a0 con 0x7f1194006270 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11ae034640 5 monclient: authenticate success, global_id 234165 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11ae034640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11ae034640 4 set_mon_vals no callback set 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.779+0000 7f11a67fc640 10 monclient: dump: 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:02.782 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:02.783 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 10 monclient: shutdown 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 -- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 msgr2=0x7f11940100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11940100f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11a8067140 tx=0x7f11a0001d00 comp rx=0 tx=0).stop 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 -- 172.21.15.17:0/3810780867 shutdown_connections 2024-09-15T02:11:02.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 --2- 172.21.15.17:0/3810780867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11940100f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 -- 172.21.15.17:0/3810780867 >> 172.21.15.17:0/3810780867 conn(0x7f11a80712f0 msgr2=0x7f11a80a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 -- 172.21.15.17:0/3810780867 shutdown_connections 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 1 -- 172.21.15.17:0/3810780867 wait complete. 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 10 monclient: build_initial_monmap 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.780+0000 7f11ae034640 10 monclient: monmap: 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:02.788 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.781+0000 7f11ae034640 1 Processor -- start 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.781+0000 7f11ae034640 1 -- start start 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.781+0000 7f11ae034640 10 monclient: init 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-a con 0x7f1194006270 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 0x7f11a811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-c con 0x7f11a806f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 0x7f11a811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:02.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): picked mon.noname-b con 0x7f11a809e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a8120ff0 con 0x7f1194006270 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a8120d10 con 0x7f11a809e970 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a81212d0 con 0x7f11a806f440 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11ae034640 10 monclient(hunting): authenticate will time out at 1922.778194s 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 0x7f11a811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:56742/0 (socket says 172.21.15.17:56742) 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 -- 172.21.15.17:0/3248164634 learned_addr learned my addr 172.21.15.17:0/3248164634 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:02.790 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a7fff640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 msgr2=0x7f11a811df20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): get_auth_request con 0x7f1194006270 auth_method 0 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a7fff640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 0x7f11a811df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request con 0x7f11a806f440 auth_method 0 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:02.791 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient(hunting): handle_auth_done global_id 234171 payload 824 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 -- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 msgr2=0x7f11a811d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 0x7f11a811d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 -- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 msgr2=0x7f11a811df20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 0x7f11a811df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 monclient: found mon.noname-a 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:02.792 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a77fe640 1 -- 172.21.15.17:0/3248164634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a809f6a0 con 0x7f1194006270 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.782+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 0x7f11a811d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11a77fe640 1 -- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 msgr2=0x7f11a8133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11a77fe640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1198007090 tx=0x7f1198005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198012ea0 con 0x7f1194006270 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: dump: 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:02.793 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198016440 con 0x7f1194006270 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11980167e0 con 0x7f1194006270 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f118f7fe640 10 monclient: dump: 2024-09-15T02:11:02.794 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 5 monclient: authenticate success, global_id 234171 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 10 monclient: _renew_subs 2024-09-15T02:11:02.795 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11a5ffb640 4 set_mon_vals no callback set 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a8136430 con 0x7f1194006270 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 10 monclient: _renew_subs 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.785+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a811fdb0 con 0x7f1194006270 2024-09-15T02:11:02.796 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1198016b50 con 0x7f1194006270 2024-09-15T02:11:02.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f118f7fe640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 0x7f119403b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:02.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f119804d250 con 0x7f1194006270 2024-09-15T02:11:02.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11ae034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:02.797 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11ae034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11ae034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a80724a0 con 0x7f1194006270 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 0x7f119403b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.786+0000 7f11a6ffd640 10 monclient: get_auth_request con 0x7f1194039540 auth_method 0 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.789+0000 7f11a6ffd640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 0x7f119403b9f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f119c003870 tx=0x7f119c003f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.791+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f11980153c0 con 0x7f1194006270 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.791+0000 7f118f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:02.798 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:02.791+0000 7f118f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f11ae034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f11ae034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f11ae034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f11a8067220 con 0x7f1194006270 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f118f7fe640 1 -- 172.21.15.17:0/3248164634 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f1198014320 con 0x7f1194006270 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f118f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:03.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.022+0000 7f118f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:03.025 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":196,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:03.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 msgr2=0x7f119403b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 0x7f119403b9f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f119c003870 tx=0x7f119c003f90 comp rx=0 tx=0).stop 2024-09-15T02:11:03.026 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 10 monclient: shutdown 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 msgr2=0x7f11a8133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1198007090 tx=0x7f1198005040 comp rx=0 tx=0).stop 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 shutdown_connections 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1194039540 0x7f119403b9f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11a809e970 0x7f11a811df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f11a806f440 0x7f11a811d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 --2- 172.21.15.17:0/3248164634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1194006270 0x7f11a8133bf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 >> 172.21.15.17:0/3248164634 conn(0x7f11a80712f0 msgr2=0x7f11a8134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:03.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 shutdown_connections 2024-09-15T02:11:03.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.024+0000 7f11ae034640 1 -- 172.21.15.17:0/3248164634 wait complete. 2024-09-15T02:11:03.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:11:03.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T02:11:03.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:11:03.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:03.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 10 monclient: build_initial_monmap 2024-09-15T02:11:03.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 10 monclient: monmap: 2024-09-15T02:11:03.134 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:03.133969+0000 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:03.133969+0000 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:03.135 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 1 Processor -- start 2024-09-15T02:11:03.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 1 -- start start 2024-09-15T02:11:03.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.133+0000 7f416819d640 10 monclient: init 2024-09-15T02:11:03.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:03.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T02:11:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 0x7f4160132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): picked mon.noname-c con 0x7f4160067090 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): picked mon.noname-b con 0x7f416009ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41600679a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416009e1e0 con 0x7f41600679a0 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416009e4c0 con 0x7f4160067090 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416009cbf0 con 0x7f416009ce00 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416819d640 10 monclient(hunting): authenticate will time out at 1923.130466s 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55718/0 (socket says 172.21.15.17:55718) 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f416673a640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f41600679a0 legacy=0x7f416012e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:11:03.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 -- 172.21.15.17:0/3849811807 learned_addr learned my addr 172.21.15.17:0/3849811807 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): get_auth_request con 0x7f416009ce00 auth_method 0 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165f39640 1 -- 172.21.15.17:0/3849811807 >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 msgr2=0x7f4160132140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4164f37640 1 -- 172.21.15.17:0/3849811807 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165f39640 1 --2- 172.21.15.17:0/3849811807 >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 0x7f4160132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient(hunting): handle_auth_done global_id 214346 payload 274 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 -- 172.21.15.17:0/3849811807 >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 msgr2=0x7f4160132140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 --2- 172.21.15.17:0/3849811807 >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 0x7f4160132140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 -- 172.21.15.17:0/3849811807 >> v1:172.21.15.67:6789/0 conn(0x7f41600679a0 legacy=0x7f416012e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient: _finish_hunting 0 2024-09-15T02:11:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 monclient: found mon.noname-b 2024-09-15T02:11:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:03.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.134+0000 7f4165738640 1 -- 172.21.15.17:0/3849811807 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4160067e00 con 0x7f416009ce00 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4165738640 1 -- 172.21.15.17:0/3849811807 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 msgr2=0x7f4160067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4165738640 1 --2- 172.21.15.17:0/3849811807 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4150009910 tx=0x7f41500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27650bdd9848723c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 1 -- 172.21.15.17:0/3849811807 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150032440 con 0x7f416009ce00 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:03.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: dump: 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: _finish_auth 0 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:03.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 1 -- 172.21.15.17:0/3849811807 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41500325a0 con 0x7f416009ce00 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f416819d640 5 monclient: authenticate success, global_id 214346 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f416819d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f416819d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f416819d640 4 set_mon_vals no callback set 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 1 -- 172.21.15.17:0/3849811807 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150032940 con 0x7f416009ce00 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.137+0000 7f4164f37640 10 monclient: dump: 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.150 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 10 monclient: shutdown 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 -- 172.21.15.17:0/3849811807 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 msgr2=0x7f4160067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 --2- 172.21.15.17:0/3849811807 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4150009910 tx=0x7f41500091d0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 -- 172.21.15.17:0/3849811807 shutdown_connections 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 --2- 172.21.15.17:0/3849811807 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f416009ce00 0x7f4160067460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 --2- 172.21.15.17:0/3849811807 >> v2:172.21.15.67:3300/0 conn(0x7f4160067090 0x7f4160132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 -- 172.21.15.17:0/3849811807 >> 172.21.15.17:0/3849811807 conn(0x7f41600a1ee0 msgr2=0x7f41600a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 -- 172.21.15.17:0/3849811807 shutdown_connections 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.138+0000 7f416819d640 1 -- 172.21.15.17:0/3849811807 wait complete. 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient: build_initial_monmap 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient: monmap: 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:03.152 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 Processor -- start 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 -- start start 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient: init 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 0x7f416012dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): picked mon.noname-c con 0x7f4160067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41600679a0 0x7f416011a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): picked mon.noname-b con 0x7f41600679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009ce00 0x7f416011ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): picked mon.noname-a con 0x7f416009ce00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416009d5c0 con 0x7f416009ce00 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600989f0 con 0x7f41600679a0 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416011e580 con 0x7f4160067090 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f416819d640 10 monclient(hunting): authenticate will time out at 1923.135882s 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.139+0000 7f4165f39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 0x7f416012dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009ce00 0x7f416011ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 0x7f416012dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55728/0 (socket says 172.21.15.17:55728) 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165738640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41600679a0 msgr2=0x7f416011a950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 -- 172.21.15.17:0/643574456 learned_addr learned my addr 172.21.15.17:0/643574456 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:03.154 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): get_auth_request con 0x7f4160067090 auth_method 0 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165738640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41600679a0 0x7f416011a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): get_auth_request con 0x7f416009ce00 auth_method 0 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f416673a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient(hunting): handle_auth_done global_id 214349 payload 824 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41600679a0 msgr2=0x7f416011a950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41600679a0 0x7f416011a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009ce00 msgr2=0x7f416011ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009ce00 0x7f416011ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient: _finish_hunting 0 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 monclient: found mon.noname-c 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.140+0000 7f4165f39640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150009450 con 0x7f4160067090 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4165f39640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 msgr2=0x7f416012dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4165f39640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 0x7f416012dfe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b600 tx=0x7f415c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c015070 con 0x7f4160067090 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: dump: 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.157 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c003d90 con 0x7f4160067090 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 5 monclient: authenticate success, global_id 214349 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 10 monclient: _renew_subs 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416012d580 con 0x7f4160067090 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c006380 con 0x7f4160067090 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4155ffb640 10 monclient: dump: 2024-09-15T02:11:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f4157fff640 4 set_mon_vals no callback set 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 10 monclient: _renew_subs 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.142+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416011cd90 con 0x7f4160067090 2024-09-15T02:11:03.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f415c010440 con 0x7f4160067090 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f416819d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f416819d640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 0x7f416009a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4160063870 con 0x7f416011b3d0 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f4155ffb640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f41440386f0 0x7f414403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.143+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f415c04c210 con 0x7f4160067090 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 0x7f416009a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient: get_auth_request con 0x7f416011b3d0 auth_method 0 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 10 monclient(hunting): handle_auth_done global_id 234177 payload 824 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f4165738640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f416011b7a0 con 0x7f416011b3d0 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f416673a640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f41440386f0 0x7f414403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.144+0000 7f416673a640 10 monclient: get_auth_request con 0x7f41440386f0 auth_method 0 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4165738640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 msgr2=0x7f416009a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4165738640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 0x7f416009a2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416009c970 tx=0x7f4150032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150032440 con 0x7f416011b3d0 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 10 monclient: dump: 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.161 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f416673a640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f41440386f0 0x7f414403aba0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4158009fd0 tx=0x7f415800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4150031e50 con 0x7f416011b3d0 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 msgr2=0x7f416009a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 0x7f416009a2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416009c970 tx=0x7f4150032000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 msgr2=0x7f416009a2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:03.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.146+0000 7f4155ffb640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416011b3d0 0x7f416009a2d0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416009c970 tx=0x7f4150032000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f416819d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:03.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f416819d640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 0x7f416009ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f416819d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600724a0 con 0x7f416009a810 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 0x7f416009ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient: get_auth_request con 0x7f416009a810 auth_method 0 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.152+0000 7f4165738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.153+0000 7f4165738640 10 monclient(hunting): handle_auth_done global_id 234180 payload 824 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.153+0000 7f4165738640 1 -- 172.21.15.17:0/643574456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f416011e310 con 0x7f416009a810 2024-09-15T02:11:03.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4165738640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 msgr2=0x7f416009ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4165738640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 0x7f416009ac10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4150009910 tx=0x7f415003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500498a0 con 0x7f416009a810 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 10 monclient: dump: 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.166 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7f4150049a00 con 0x7f416009a810 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 msgr2=0x7f416009ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 0x7f416009ac10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4150009910 tx=0x7f415003d000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 msgr2=0x7f416009ac10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f4155ffb640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f416009a810 0x7f416009ac10 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4150009910 tx=0x7f415003d000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:03.167 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 196, 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:03.168 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.169 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:03.170 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:03.171 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.172 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:03.173 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.174 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.176 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.179 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.180 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:03.181 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:03.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f41440386f0 msgr2=0x7f414403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.155+0000 7f416819d640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f41440386f0 0x7f414403aba0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4158009fd0 tx=0x7f415800d040 comp rx=0 tx=0).stop 2024-09-15T02:11:03.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 10 monclient: shutdown 2024-09-15T02:11:03.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 msgr2=0x7f416012dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 --2- 172.21.15.17:0/643574456 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4160067090 0x7f416012dfe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b600 tx=0x7f415c0091a0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416673a640 1 -- 172.21.15.17:0/643574456 reap_dead start 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416673a640 1 -- 172.21.15.17:0/643574456 reap_dead start 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 shutdown_connections 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 >> 172.21.15.17:0/643574456 conn(0x7f41600a1ee0 msgr2=0x7f4160097020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 shutdown_connections 2024-09-15T02:11:03.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.156+0000 7f416819d640 1 -- 172.21.15.17:0/643574456 wait complete. 2024-09-15T02:11:03.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:11:03.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:11:03.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.279+0000 7fba48b9d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.279+0000 7fba48b9d640 10 monclient: build_initial_monmap 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.279+0000 7fba48b9d640 10 monclient: monmap: 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:03.280639+0000 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:03.280639+0000 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:03.282 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.280+0000 7fba48b9d640 1 Processor -- start 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.280+0000 7fba48b9d640 1 -- start start 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.280+0000 7fba48b9d640 10 monclient: init 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba4409c1b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 0x7fba4409c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fba4409e9e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba4406eb30 0x7fba4406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fba4406eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4409f790 con 0x7fba4409c1b0 2024-09-15T02:11:03.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba44069d40 con 0x7fba4406eb30 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4409ee20 con 0x7fba4409e9e0 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba48b9d640 10 monclient(hunting): authenticate will time out at 1923.277185s 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba41d9b640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 msgr2=0x7fba4409c580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:03.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba4259c640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7fba4409c1b0 legacy=0x7fba4409a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba41d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 0x7fba4409c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba4406eb30 0x7fba4406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fba4406eb30 0x7fba4406ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:56790/0 (socket says 172.21.15.17:56790) 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 -- 172.21.15.17:0/3701809606 learned_addr learned my addr 172.21.15.17:0/3701809606 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): get_auth_request con 0x7fba4406eb30 auth_method 0 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient(hunting): handle_auth_done global_id 234183 payload 274 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 -- 172.21.15.17:0/3701809606 >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 msgr2=0x7fba4409c580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 --2- 172.21.15.17:0/3701809606 >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 0x7fba4409c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 -- 172.21.15.17:0/3701809606 >> v1:172.21.15.67:6789/0 conn(0x7fba4409c1b0 legacy=0x7fba4409a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 monclient: found mon.noname-a 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:03.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.281+0000 7fba42d9d640 1 -- 172.21.15.17:0/3701809606 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44068b60 con 0x7fba4406eb30 2024-09-15T02:11:03.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba42d9d640 1 -- 172.21.15.17:0/3701809606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 msgr2=0x7fba4406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba42d9d640 1 --2- 172.21.15.17:0/3701809606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba4406ef90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba44060cf0 tx=0x7fba3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9359fa0d22f95172 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 1 -- 172.21.15.17:0/3701809606 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c02aea0 con 0x7fba4406eb30 2024-09-15T02:11:03.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: dump: 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: _finish_auth 0 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 1 -- 172.21.15.17:0/3701809606 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c02d050 con 0x7fba4406eb30 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 1 -- 172.21.15.17:0/3701809606 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c007820 con 0x7fba4406eb30 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba48b9d640 5 monclient: authenticate success, global_id 234183 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba48b9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba48b9d640 4 set_mon_vals no callback set 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.284+0000 7fba4159a640 10 monclient: dump: 2024-09-15T02:11:03.296 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.297 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 10 monclient: shutdown 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 -- 172.21.15.17:0/3701809606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 msgr2=0x7fba4406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3701809606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba4406ef90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba44060cf0 tx=0x7fba3c009840 comp rx=0 tx=0).stop 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 -- 172.21.15.17:0/3701809606 shutdown_connections 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3701809606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba4406ef90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3701809606 >> v2:172.21.15.67:3300/0 conn(0x7fba4409e9e0 0x7fba4409c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 -- 172.21.15.17:0/3701809606 >> 172.21.15.17:0/3701809606 conn(0x7fba44071300 msgr2=0x7fba440a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 -- 172.21.15.17:0/3701809606 shutdown_connections 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 -- 172.21.15.17:0/3701809606 wait complete. 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 10 monclient: build_initial_monmap 2024-09-15T02:11:03.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 10 monclient: monmap: 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.285+0000 7fba48b9d640 1 Processor -- start 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 -- start start 2024-09-15T02:11:03.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient: init 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fba4406eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4409c1b0 0x7fba44133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fba4409c1b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4409e9e0 0x7fba4411da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fba4409e9e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba441420a0 con 0x7fba4406eb30 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4411f450 con 0x7fba4409e9e0 2024-09-15T02:11:03.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba44120f10 con 0x7fba4409c1b0 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba48b9d640 10 monclient(hunting): authenticate will time out at 1923.282549s 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:56798/0 (socket says 172.21.15.17:56798) 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 -- 172.21.15.17:0/3944951448 learned_addr learned my addr 172.21.15.17:0/3944951448 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): get_auth_request con 0x7fba4406eb30 auth_method 0 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4409c1b0 0x7fba44133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): get_auth_request con 0x7fba4409c1b0 auth_method 0 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba42d9d640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4409e9e0 msgr2=0x7fba4411da50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba42d9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4409e9e0 0x7fba4411da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient(hunting): handle_auth_done global_id 234186 payload 824 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4409c1b0 msgr2=0x7fba44133df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4409c1b0 0x7fba44133df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4409e9e0 msgr2=0x7fba4411da50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4409e9e0 0x7fba4411da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient: _finish_hunting 0 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 monclient: found mon.noname-a 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.286+0000 7fba4259c640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c009ae0 con 0x7fba4406eb30 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba4259c640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 msgr2=0x7fba44141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba4259c640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b5c0 tx=0x7fba34009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba4259c640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b5c0 tx=0x7fba34009120 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fba34003e40 config(0 keys) v1 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34003cc0 con 0x7fba4406eb30 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 10 monclient: dump: 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:11:03.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba34003e40 con 0x7fba4406eb30 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 5 monclient: authenticate success, global_id 234186 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 10 monclient: _renew_subs 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba441211f0 con 0x7fba4406eb30 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba40d99640 4 set_mon_vals no callback set 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 10 monclient: _renew_subs 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.289+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44120020 con 0x7fba4406eb30 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34005340 con 0x7fba4406eb30 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:03.305 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 10 monclient: dump: 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fba3401a460 con 0x7fba4406eb30 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba48b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 0x7fba4411df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.306 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba44063bb0 con 0x7fba44134330 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 0x7fba4411df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba3a7fc640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba24038740 0x7fba2403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient: get_auth_request con 0x7fba44134330 auth_method 0 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 10 monclient(hunting): handle_auth_done global_id 214355 payload 824 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba41d9b640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba441202f0 con 0x7fba44134330 2024-09-15T02:11:03.307 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba42d9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba24038740 0x7fba2403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.290+0000 7fba42d9d640 10 monclient: get_auth_request con 0x7fba24038740 auth_method 0 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba41d9b640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 msgr2=0x7fba4411df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba41d9b640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 0x7fba4411df90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fba4411f0e0 tx=0x7fba2c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00e9f0 con 0x7fba44134330 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.292+0000 7fba3a7fc640 10 monclient: dump: 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.308 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fba2c00eb50 con 0x7fba44134330 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 msgr2=0x7fba4411df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 0x7fba4411df90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fba4411f0e0 tx=0x7fba2c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 msgr2=0x7fba4411df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba3a7fc640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba44134330 0x7fba4411df90 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fba4411f0e0 tx=0x7fba2c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:03.309 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.293+0000 7fba42d9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba24038740 0x7fba2403abf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fba44068b30 tx=0x7fba3c036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.298+0000 7fba48b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:03.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.298+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 0x7fba440a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:03.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba48b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:03.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba44134700 con 0x7fba4411e4d0 2024-09-15T02:11:03.311 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 0x7fba440a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient: get_auth_request con 0x7fba4411e4d0 auth_method 0 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 10 monclient(hunting): handle_auth_done global_id 214358 payload 824 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.299+0000 7fba41d9b640 1 -- 172.21.15.17:0/3944951448 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fba4411e8d0 con 0x7fba4411e4d0 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba41d9b640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 msgr2=0x7fba440a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba41d9b640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 0x7fba440a1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fba2c003fd0 tx=0x7fba2c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:03.312 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01b940 con 0x7fba4411e4d0 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 10 monclient: dump: 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:03.313 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7fba2c01baa0 con 0x7fba4411e4d0 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 msgr2=0x7fba440a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 0x7fba440a1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fba2c003fd0 tx=0x7fba2c015000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 msgr2=0x7fba440a1af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.301+0000 7fba3a7fc640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fba4411e4d0 0x7fba440a1af0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fba2c003fd0 tx=0x7fba2c015000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:03.314 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 196, 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 1, 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.315 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.316 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:03.317 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:03.318 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.319 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:03.321 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.323 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.324 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:11:03.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:03.327 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba24038740 msgr2=0x7fba2403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fba24038740 0x7fba2403abf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fba44068b30 tx=0x7fba3c036000 comp rx=0 tx=0).stop 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 10 monclient: shutdown 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 msgr2=0x7fba44141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 --2- 172.21.15.17:0/3944951448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fba4406eb30 0x7fba44141210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b5c0 tx=0x7fba34009120 comp rx=0 tx=0).stop 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba42d9d640 1 -- 172.21.15.17:0/3944951448 reap_dead start 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba3a7fc640 1 -- 172.21.15.17:0/3944951448 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (unknown 0 0 0) 0x7fba3404c1e0 con 0x7fba4406eb30 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba42d9d640 1 -- 172.21.15.17:0/3944951448 reap_dead start 2024-09-15T02:11:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 shutdown_connections 2024-09-15T02:11:03.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 >> 172.21.15.17:0/3944951448 conn(0x7fba44071300 msgr2=0x7fba44134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:03.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 shutdown_connections 2024-09-15T02:11:03.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:03.302+0000 7fba48b9d640 1 -- 172.21.15.17:0/3944951448 wait complete. 2024-09-15T02:11:03.330 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:11:03.330 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:11:13.936 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=110 2024-09-15T02:11:13.936 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 110 -ne 110 ]] 2024-09-15T02:11:13.936 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:11:13.936 INFO:tasks.workunit.client.0.smithi067.stderr:+ export CEPH_ARGS= 2024-09-15T02:11:13.937 INFO:tasks.workunit.client.0.smithi067.stderr:+ CEPH_ARGS= 2024-09-15T02:11:13.937 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph auth del client.bar 2024-09-15T02:11:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 10 monclient: build_initial_monmap 2024-09-15T02:11:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 10 monclient: monmap: 2024-09-15T02:11:14.018 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:11:14.017212+0000 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:11:14.017212+0000 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 1 Processor -- start 2024-09-15T02:11:14.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 1 -- start start 2024-09-15T02:11:14.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.016+0000 7f8beed1d640 10 monclient: init 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8be809cb60 0x7f8be809ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8be809cb60 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8be809d6f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8be809b1b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be80997d0 con 0x7f8be809d6f0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8becab9640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f8be809cb60 msgr2=0x7f8be809ac70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be8099180 con 0x7f8be809b1b0 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be8098780 con 0x7f8be809cb60 2024-09-15T02:11:14.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8becab9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8be809cb60 0x7f8be809ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8bed2ba640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f8be809b1b0 legacy=0x7f8be80970e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2024-09-15T02:11:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8beed1d640 10 monclient(hunting): authenticate will time out at 1906.486616s 2024-09-15T02:11:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8be7fff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f8be809d6f0 0x7f8be80969d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.67:40196/0 (socket says 172.21.15.67:40196) 2024-09-15T02:11:14.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.017+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 learned_addr learned my addr 172.21.15.67:0/4026398072 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 >> v1:172.21.15.17:6789/0 conn(0x7f8be809d6f0 legacy=0x7f8be80969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224038769 0 0) 0x7f8be80997d0 con 0x7f8be809d6f0 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): my global_id is 234189 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd40034e0 con 0x7f8be809d6f0 2024-09-15T02:11:14.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 680404384 0 0) 0x7f8bd40034e0 con 0x7f8be809d6f0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 >> v2:172.21.15.67:3300/0 conn(0x7f8be809cb60 msgr2=0x7f8be809ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 --2- 172.21.15.67:0/4026398072 >> v2:172.21.15.67:3300/0 conn(0x7f8be809cb60 0x7f8be809ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 >> v1:172.21.15.67:6789/0 conn(0x7f8be809b1b0 legacy=0x7f8be80970e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be809cf30 con 0x7f8be809d6f0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: _finish_auth 0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f8bd00017e0 con 0x7f8be809d6f0 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:14.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: dump: 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 >> v1:172.21.15.17:6789/0 conn(0x7f8be809d6f0 legacy=0x7f8be80969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd40039d0 0x7f8bd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): picked mon.a con 0x7f8bd40039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 0x7f8bd4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): picked mon.c con 0x7f8bd4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bd4005280 0x7f8bd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): picked mon.b con 0x7f8bd4005280 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8098780 con 0x7f8bd40039d0 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd400c240 con 0x7f8bd4005280 2024-09-15T02:11:14.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd400c400 con 0x7f8bd4004780 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8bed2ba640 1 -- 172.21.15.67:0/4026398072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bd4005280 msgr2=0x7f8bd400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8bed2ba640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bd4005280 0x7f8bd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8becab9640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd40039d0 0x7f8bd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 0x7f8bd4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8becab9640 10 monclient(hunting): get_auth_request con 0x7f8bd40039d0 auth_method 0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8becab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8becab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8becab9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 10 monclient(hunting): get_auth_request con 0x7f8bd4004780 auth_method 0 2024-09-15T02:11:14.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.020+0000 7f8be7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient(hunting): handle_auth_done global_id 234189 payload 293 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bd4005280 msgr2=0x7f8bd400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bd4005280 0x7f8bd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd40039d0 msgr2=0x7f8bd4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd40039d0 0x7f8bd4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 monclient: found mon.c 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8bed2ba640 1 -- 172.21.15.67:0/4026398072 reap_dead start 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40034e0 con 0x7f8bd4004780 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 10 monclient: _finish_auth 0 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8beed1d640 5 monclient: authenticate success, global_id 234189 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8becab9640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd40039d0 0x7f8bd4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.021+0000 7f8beed1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be7fff640 1 -- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 msgr2=0x7f8bd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:14.032 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be7fff640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 0x7f8bd4004bf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000c5c0 tx=0x7f8bd0007b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3a8f727e5676e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0005890 con 0x7f8bd4004780 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: dump: 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:14.033 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0005a10 con 0x7f8bd4004780 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8beed1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8beed1d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 1 -- 172.21.15.67:0/4026398072 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0005dd0 con 0x7f8bd4004780 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8beed1d640 4 set_mon_vals no callback set 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.024+0000 7f8be77fe640 10 monclient: dump: 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.034 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:14.035 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.035 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:14.035 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:14.035 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:14.035 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 10 monclient: shutdown 2024-09-15T02:11:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 -- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 msgr2=0x7f8bd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:14.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 0x7f8bd4004bf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000c5c0 tx=0x7f8bd0007b30 comp rx=0 tx=0).stop 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 -- 172.21.15.67:0/4026398072 shutdown_connections 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 --2- 172.21.15.67:0/4026398072 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8bd4004780 0x7f8bd4004bf0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 -- 172.21.15.67:0/4026398072 >> 172.21.15.67:0/4026398072 conn(0x7f8be80716e0 msgr2=0x7f8be80a1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 -- 172.21.15.67:0/4026398072 shutdown_connections 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 1 -- 172.21.15.67:0/4026398072 wait complete. 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 10 monclient: build_initial_monmap 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.025+0000 7f8beed1d640 10 monclient: monmap: 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:11:14.041 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 1 Processor -- start 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 1 -- start start 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 10 monclient: init 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd4004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.026+0000 7f8beed1d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8be8145e10 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 0x7f8be8119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): picked mon.noname-b con 0x7f8be8118d00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be812ebf0 con 0x7f8bd4004780 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80a09d0 con 0x7f8be8118d00 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be811a860 con 0x7f8be8145e10 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8beed1d640 10 monclient(hunting): authenticate will time out at 1906.496074s 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8bed2ba640 1 -- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 msgr2=0x7f8be8119150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:3300/0 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8bed2ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 0x7f8be8119150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:14.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be4ff9640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.67:55904/0 (socket says 172.21.15.67:55904) 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 1 -- 172.21.15.67:0/359407271 learned_addr learned my addr 172.21.15.67:0/359407271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): get_auth_request con 0x7f8bd4004780 auth_method 0 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.67:37956/0 (socket says 172.21.15.67:37956) 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): get_auth_request con 0x7f8be8145e10 auth_method 0 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:14.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient(hunting): handle_auth_done global_id 214361 payload 824 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 -- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 msgr2=0x7f8be8119150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 0x7f8be8119150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 -- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 msgr2=0x7f8be81458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 monclient: found mon.noname-c 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8be7fff640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8146720 con 0x7f8be8145e10 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.027+0000 7f8becab9640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be7fff640 1 -- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 msgr2=0x7f8be81461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be7fff640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400c590 tx=0x7f8bd0034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd003e070 con 0x7f8be8145e10 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: dump: 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:14.046 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: _finish_auth 0 2024-09-15T02:11:14.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:14.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0007510 con 0x7f8be8145e10 2024-09-15T02:11:14.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 5 monclient: authenticate success, global_id 214361 2024-09-15T02:11:14.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 10 monclient: _renew_subs 2024-09-15T02:11:14.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.053 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be811c380 con 0x7f8be8145e10 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd00047a0 con 0x7f8be8145e10 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be4ff9640 10 monclient: dump: 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:14.054 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8be6ffd640 4 set_mon_vals no callback set 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 10 monclient: _renew_subs 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.030+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be8125e50 con 0x7f8be8145e10 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8bd0004bd0 con 0x7f8be8145e10 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8beed1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8beed1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8beed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be80a1770 con 0x7f8be8145e10 2024-09-15T02:11:14.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8be4ff9640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 0x7f8bc803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8bd006c960 con 0x7f8be8145e10 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8becab9640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 0x7f8bc803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.031+0000 7f8becab9640 10 monclient: get_auth_request con 0x7f8bc8038720 auth_method 0 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.034+0000 7f8becab9640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 0x7f8bc803abd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009a30 tx=0x7f8bd800e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.036+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8bd002dba0 con 0x7f8be8145e10 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.036+0000 7f8be4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:14.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.036+0000 7f8be4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:14.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.262+0000 7f8beed1d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.262+0000 7f8beed1d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.262+0000 7f8beed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.262+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f8be809ef60 con 0x7f8be8145e10 2024-09-15T02:11:14.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.267+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8bd002d3c0 con 0x7f8be8145e10 2024-09-15T02:11:14.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.267+0000 7f8be4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.268 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.267+0000 7f8be4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:14.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.422+0000 7f8beed1d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.422+0000 7f8beed1d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.422+0000 7f8beed1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:14.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.422+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f8be804bc80 con 0x7f8be8145e10 2024-09-15T02:11:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.424+0000 7f8be4ff9640 1 -- 172.21.15.67:0/359407271 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8bd002d5a0 con 0x7f8be8145e10 2024-09-15T02:11:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.424+0000 7f8be4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T02:11:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.424+0000 7f8be4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:11:14.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 msgr2=0x7f8bc803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:14.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 0x7f8bc803abd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009a30 tx=0x7f8bd800e040 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 10 monclient: shutdown 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 msgr2=0x7f8be81461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.425+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400c590 tx=0x7f8bd0034000 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 shutdown_connections 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8bc8038720 0x7f8bc803abd0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8be8118d00 0x7f8be8119150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8be8145e10 0x7f8be81461e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 --2- 172.21.15.67:0/359407271 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bd4004780 0x7f8be81458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 >> 172.21.15.67:0/359407271 conn(0x7f8be80716e0 msgr2=0x7f8be80a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 shutdown_connections 2024-09-15T02:11:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:14.426+0000 7f8beed1d640 1 -- 172.21.15.67:0/359407271 wait complete. 2024-09-15T02:11:14.436 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-15T02:11:14.436 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-15T02:11:14.437 INFO:tasks.workunit.client.0.smithi067.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:11:14.437 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=13 2024-09-15T02:11:14.437 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:11:14.437 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T02:11:23.330 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T02:11:23.331 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T02:11:23.331 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:11:23.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:11:23.334 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:11:23.334 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T02:11:23.335 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T02:11:23.335 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-15T02:11:43.336 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T02:11:43.337 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T02:11:43.337 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:11:43.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:43.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.445+0000 7fe4e1657640 10 monclient: get_monmap_and_config 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.446+0000 7fe4e1657640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.446+0000 7fe4e1657640 10 monclient: monmap: 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:43.446838+0000 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:43.446838+0000 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:43.448 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.446+0000 7fe4e1657640 1 Processor -- start 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.446+0000 7fe4e1657640 1 -- start start 2024-09-15T02:11:43.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.446+0000 7fe4e1657640 10 monclient: init 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe4dc097230 0x7fe4dc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-a con 0x7fe4dc097230 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4dc098c40 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4dc06f270 0x7fe4dc09d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-c con 0x7fe4dc06f270 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:43.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc098260 con 0x7fe4dc098c40 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc06f670 con 0x7fe4dc097230 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc097600 con 0x7fe4dc06f270 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4e1657640 10 monclient(hunting): authenticate will time out at 1963.443238s 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe4dc097230 0x7fe4dc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe4dc097230 0x7fe4dc06ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59884/0 (socket says 172.21.15.17:59884) 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 -- 172.21.15.17:0/2229323537 learned_addr learned my addr 172.21.15.17:0/2229323537 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): get_auth_request con 0x7fe4dc097230 auth_method 0 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4db7fe640 1 --2- 172.21.15.17:0/2229323537 >> v2:172.21.15.67:3300/0 conn(0x7fe4dc06f270 0x7fe4dc09d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient(hunting): handle_auth_done global_id 234204 payload 274 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 -- 172.21.15.17:0/2229323537 >> v2:172.21.15.67:3300/0 conn(0x7fe4dc06f270 msgr2=0x7fe4dc09d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 --2- 172.21.15.17:0/2229323537 >> v2:172.21.15.67:3300/0 conn(0x7fe4dc06f270 0x7fe4dc09d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 -- 172.21.15.17:0/2229323537 >> v1:172.21.15.67:6789/0 conn(0x7fe4dc098c40 legacy=0x7fe4dc09cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 monclient: found mon.noname-a 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.447+0000 7fe4daffd640 1 -- 172.21.15.17:0/2229323537 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc097850 con 0x7fe4dc097230 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4daffd640 1 -- 172.21.15.17:0/2229323537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 msgr2=0x7fe4dc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4daffd640 1 --2- 172.21.15.17:0/2229323537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc06ed30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00d1a0 tx=0x7fe4cc00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f8c1de1b1f6afe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 1 -- 172.21.15.17:0/2229323537 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc009610 con 0x7fe4dc097230 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: dump: 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.459 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 1 -- 172.21.15.17:0/2229323537 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc009770 con 0x7fe4dc097230 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 1 -- 172.21.15.17:0/2229323537 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc009b10 con 0x7fe4dc097230 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4e1657640 5 monclient: authenticate success, global_id 234204 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4e1657640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:43.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4e1657640 4 set_mon_vals no callback set 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.450+0000 7fe4d9ffb640 10 monclient: dump: 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.461 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 10 monclient: shutdown 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 -- 172.21.15.17:0/2229323537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 msgr2=0x7fe4dc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 --2- 172.21.15.17:0/2229323537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc06ed30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00d1a0 tx=0x7fe4cc00ba50 comp rx=0 tx=0).stop 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 -- 172.21.15.17:0/2229323537 shutdown_connections 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 --2- 172.21.15.17:0/2229323537 >> v2:172.21.15.67:3300/0 conn(0x7fe4dc06f270 0x7fe4dc09d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 --2- 172.21.15.17:0/2229323537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc06ed30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 -- 172.21.15.17:0/2229323537 >> 172.21.15.17:0/2229323537 conn(0x7fe4dc0a2100 msgr2=0x7fe4dc0a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 -- 172.21.15.17:0/2229323537 shutdown_connections 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.451+0000 7fe4e1657640 1 -- 172.21.15.17:0/2229323537 wait complete. 2024-09-15T02:11:43.462 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient: monmap: 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 Processor -- start 2024-09-15T02:11:43.463 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 -- start start 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient: init 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc06f270 0x7fe4dc13cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-b con 0x7fe4dc06f270 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-a con 0x7fe4dc097230 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 0x7fe4dc119580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): picked mon.noname-c con 0x7fe4dc098c40 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc11cb60 con 0x7fe4dc097230 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc097a60 con 0x7fe4dc06f270 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc09a610 con 0x7fe4dc098c40 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.452+0000 7fe4e1657640 10 monclient(hunting): authenticate will time out at 1963.448912s 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 0x7fe4dc119580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59888/0 (socket says 172.21.15.17:59888) 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 -- 172.21.15.17:0/1670191324 learned_addr learned my addr 172.21.15.17:0/1670191324 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): get_auth_request con 0x7fe4dc097230 auth_method 0 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): get_auth_request con 0x7fe4dc098c40 auth_method 0 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc06f270 0x7fe4dc13cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): get_auth_request con 0x7fe4dc06f270 auth_method 0 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient(hunting): handle_auth_done global_id 234207 payload 824 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 -- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 msgr2=0x7fe4dc119580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 0x7fe4dc119580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 -- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc06f270 msgr2=0x7fe4dc13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc06f270 0x7fe4dc13cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 monclient: found mon.noname-a 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4da7fc640 1 -- 172.21.15.17:0/1670191324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc00bcf0 con 0x7fe4dc097230 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.453+0000 7fe4db7fe640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 0x7fe4dc119580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4da7fc640 1 -- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 msgr2=0x7fe4dc12fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4da7fc640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400c640 tx=0x7fe4c4007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4012840 con 0x7fe4dc097230 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: dump: 2024-09-15T02:11:43.468 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c40129a0 con 0x7fe4dc097230 2024-09-15T02:11:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4012d60 con 0x7fe4dc097230 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 5 monclient: authenticate success, global_id 234207 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 10 monclient: _renew_subs 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc13dbe0 con 0x7fe4dc097230 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d8ff9640 4 set_mon_vals no callback set 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4d2ffd640 10 monclient: dump: 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.470 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 10 monclient: _renew_subs 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.455+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc11b4f0 con 0x7fe4dc097230 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.456+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe4c4011590 con 0x7fe4dc097230 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4e1657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4e1657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4e1657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc0724a0 con 0x7fe4dc097230 2024-09-15T02:11:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4d2ffd640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe4c404c730 con 0x7fe4dc097230 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4daffd640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.457+0000 7fe4daffd640 10 monclient: get_auth_request con 0x7fe4bc038790 auth_method 0 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.460+0000 7fe4daffd640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc09bab0 tx=0x7fe4cc039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.461+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe4c4010ac0 con 0x7fe4dc097230 2024-09-15T02:11:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.461+0000 7fe4d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.461+0000 7fe4d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4e1657640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4e1657640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4e1657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe4dc0008d0 con 0x7fe4dc097230 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4d2ffd640 1 -- 172.21.15.17:0/1670191324 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fe4c4016080 con 0x7fe4dc097230 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:43.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.692+0000 7fe4d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:43.696 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 msgr2=0x7fe4bc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc09bab0 tx=0x7fe4cc039000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 10 monclient: shutdown 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 msgr2=0x7fe4dc12fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400c640 tx=0x7fe4c4007ed0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 shutdown_connections 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe4bc038790 0x7fe4bc03ac40 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4dc097230 0x7fe4dc12fa40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe4dc098c40 0x7fe4dc119580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 --2- 172.21.15.17:0/1670191324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc06f270 0x7fe4dc13cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 >> 172.21.15.17:0/1670191324 conn(0x7fe4dc0a2100 msgr2=0x7fe4dc099340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 shutdown_connections 2024-09-15T02:11:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.694+0000 7fe4e1657640 1 -- 172.21.15.17:0/1670191324 wait complete. 2024-09-15T02:11:43.707 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:11:43.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:11:43.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.802+0000 7ff0a0634640 10 monclient: get_monmap_and_config 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.802+0000 7ff0a0634640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.802+0000 7ff0a0634640 10 monclient: monmap: 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:43.803531+0000 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:43.803531+0000 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.804 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:43.805 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.803+0000 7ff0a0634640 1 Processor -- start 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.803+0000 7ff0a0634640 1 -- start start 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.803+0000 7ff0a0634640 10 monclient: init 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.803+0000 7ff0a0634640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.803+0000 7ff0a0634640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 0x7ff098132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-a con 0x7ff098067090 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff09809ce00 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-b con 0x7ff098067b70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09809e1e0 con 0x7ff09809ce00 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09809e4c0 con 0x7ff098067090 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09809cbf0 con 0x7ff098067b70 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff0a0634640 10 monclient(hunting): authenticate will time out at 1963.800112s 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 0x7ff098132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09dbcf640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff09809ce00 0x7ff098067460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:45834/0 (socket says 172.21.15.17:45834) 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1883714845 learned_addr learned my addr 172.21.15.17:0/1883714845 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): get_auth_request con 0x7ff098067b70 auth_method 0 2024-09-15T02:11:43.808 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request con 0x7ff098067090 auth_method 0 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_done global_id 214379 payload 274 2024-09-15T02:11:43.809 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1883714845 >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 msgr2=0x7ff098132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 --2- 172.21.15.17:0/1883714845 >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 0x7ff098132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1883714845 >> v1:172.21.15.17:6789/0 conn(0x7ff09809ce00 legacy=0x7ff098067460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 monclient: found mon.noname-b 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1883714845 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09806efb0 con 0x7ff098067b70 2024-09-15T02:11:43.810 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.804+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1883714845 >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 0x7ff098132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1883714845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 msgr2=0x7ff098139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09ebd1640 1 --2- 172.21.15.17:0/1883714845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098139010 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff090009e60 tx=0x7ff090009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=785e9bca35136bc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 1 -- 172.21.15.17:0/1883714845 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090007760 con 0x7ff098067b70 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:43.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: dump: 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 1 -- 172.21.15.17:0/1883714845 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09002d050 con 0x7ff098067b70 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 5 monclient: authenticate success, global_id 214379 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 1 -- 172.21.15.17:0/1883714845 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090007af0 con 0x7ff098067b70 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 4 set_mon_vals no callback set 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff09d3ce640 10 monclient: dump: 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.817 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 10 monclient: shutdown 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 1 -- 172.21.15.17:0/1883714845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 msgr2=0x7ff098139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.807+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1883714845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098139010 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff090009e60 tx=0x7ff090009780 comp rx=0 tx=0).stop 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 -- 172.21.15.17:0/1883714845 shutdown_connections 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1883714845 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098139010 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1883714845 >> v2:172.21.15.17:3300/0 conn(0x7ff098067090 0x7ff098132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 -- 172.21.15.17:0/1883714845 >> 172.21.15.17:0/1883714845 conn(0x7ff0980a1ee0 msgr2=0x7ff0980a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 -- 172.21.15.17:0/1883714845 shutdown_connections 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 1 -- 172.21.15.17:0/1883714845 wait complete. 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.808+0000 7ff0a0634640 10 monclient: monmap: 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:43.819 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 Processor -- start 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 -- start start 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient: init 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff098067090 0x7ff09812f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-b con 0x7ff098067090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098135ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-c con 0x7ff098067b70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff09809ce00 0x7ff0981363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): picked mon.noname-a con 0x7ff09809ce00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff098138710 con 0x7ff09809ce00 2024-09-15T02:11:43.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0980a03e0 con 0x7ff098067090 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff098138300 con 0x7ff098067b70 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff0a0634640 10 monclient(hunting): authenticate will time out at 1963.805664s 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098135ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff09809ce00 0x7ff0981363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098135ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54900/0 (socket says 172.21.15.17:54900) 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1244132154 learned_addr learned my addr 172.21.15.17:0/1244132154 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff09809ce00 0x7ff0981363e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59914/0 (socket says 172.21.15.17:59914) 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): get_auth_request con 0x7ff098067b70 auth_method 0 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): get_auth_request con 0x7ff09809ce00 auth_method 0 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09dbcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09ebd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.809+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff098067090 0x7ff09812f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request con 0x7ff098067090 auth_method 0 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09e3d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09e3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 10 monclient(hunting): handle_auth_done global_id 214382 payload 824 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff098067090 msgr2=0x7ff09812f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff098067090 0x7ff09812f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff09809ce00 msgr2=0x7ff0981363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff09809ce00 0x7ff0981363e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 monclient: found mon.noname-c 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:43.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.810+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009390 con 0x7ff098067b70 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff09dbcf640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 msgr2=0x7ff098135ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff09dbcf640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098135ea0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff09400c5e0 tx=0x7ff094004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094005cb0 con 0x7ff098067b70 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: dump: 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.824 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff094005e30 con 0x7ff098067b70 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 5 monclient: authenticate success, global_id 214382 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 10 monclient: _renew_subs 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff090009a20 con 0x7ff098067b70 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094007340 con 0x7ff098067b70 2024-09-15T02:11:43.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff08a7fc640 10 monclient: dump: 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff09cbcd640 4 set_mon_vals no callback set 2024-09-15T02:11:43.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 10 monclient: _renew_subs 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.812+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff098137e10 con 0x7ff098067b70 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff0a0634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 0x7ff09812fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff098063870 con 0x7ff098136920 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff09401b400 con 0x7ff098067b70 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 0x7ff09812fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient: get_auth_request con 0x7ff098136920 auth_method 0 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff08a7fc640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff07c038740 0x7ff07c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_done global_id 234216 payload 824 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09e3d0640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff098118620 con 0x7ff098136920 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff09404c180 con 0x7ff098067b70 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09ebd1640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff07c038740 0x7ff07c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.813+0000 7ff09ebd1640 10 monclient: get_auth_request con 0x7ff07c038740 auth_method 0 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.815+0000 7ff09e3d0640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 msgr2=0x7ff09812fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.815+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 0x7ff09812fd40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff08c011840 tx=0x7ff08c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00fd60 con 0x7ff098136920 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 10 monclient: dump: 2024-09-15T02:11:43.830 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff08c00fec0 con 0x7ff098136920 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:43.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 msgr2=0x7ff09812fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 0x7ff09812fd40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff08c011840 tx=0x7ff08c008040 comp rx=0 tx=0).stop 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 msgr2=0x7ff09812fd40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff08a7fc640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098136920 0x7ff09812fd40 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff08c011840 tx=0x7ff08c008040 comp rx=0 tx=0).stop 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.816+0000 7ff09ebd1640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff07c038740 0x7ff07c03abf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff09809faa0 tx=0x7ff090037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff0a0634640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 0x7ff098130680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff0a0634640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff098118900 con 0x7ff098130280 2024-09-15T02:11:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 0x7ff098130680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient: get_auth_request con 0x7ff098130280 auth_method 0 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 10 monclient(hunting): handle_auth_done global_id 234219 payload 824 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.822+0000 7ff09e3d0640 1 -- 172.21.15.17:0/1244132154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff098136cf0 con 0x7ff098130280 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff09e3d0640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 msgr2=0x7ff098130680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff09e3d0640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 0x7ff098130680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff08c01b000 tx=0x7ff08c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c022070 con 0x7ff098130280 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 10 monclient: dump: 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7ff08c005e70 con 0x7ff098130280 2024-09-15T02:11:43.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 msgr2=0x7ff098130680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.824+0000 7ff08a7fc640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 0x7ff098130680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff08c01b000 tx=0x7ff08c01c000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff08a7fc640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 msgr2=0x7ff098130680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff08a7fc640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff098130280 0x7ff098130680 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff08c01b000 tx=0x7ff08c01c000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:11:43.835 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.836 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:43.838 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.839 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:43.840 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.843 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.846 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:43.847 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:43.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff07c038740 msgr2=0x7ff07c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff07c038740 0x7ff07c03abf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff09809faa0 tx=0x7ff090037000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 10 monclient: shutdown 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 msgr2=0x7ff098135ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 --2- 172.21.15.17:0/1244132154 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff098067b70 0x7ff098135ea0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff09400c5e0 tx=0x7ff094004960 comp rx=0 tx=0).stop 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1244132154 reap_dead start 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff09ebd1640 1 -- 172.21.15.17:0/1244132154 reap_dead start 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 shutdown_connections 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 >> 172.21.15.17:0/1244132154 conn(0x7ff0980a1ee0 msgr2=0x7ff098097ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 shutdown_connections 2024-09-15T02:11:43.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.825+0000 7ff0a0634640 1 -- 172.21.15.17:0/1244132154 wait complete. 2024-09-15T02:11:43.850 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:43.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:11:43.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.945+0000 7f033986f640 10 monclient: get_monmap_and_config 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.945+0000 7f033986f640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.945+0000 7f033986f640 10 monclient: monmap: 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:43.946440+0000 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:43.946440+0000 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.947 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:43.948 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:43.948 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:43.948 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:43.948 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:43.948 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 Processor -- start 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 -- start start 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient: init 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): picked mon.noname-b con 0x7f0334067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:43.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0334067460 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0334136f90 0x7f0334139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): picked mon.noname-a con 0x7f0334136f90 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033409cbc0 con 0x7f0334067460 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0334067f40 con 0x7f0334136f90 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033409e550 con 0x7f0334067090 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f033986f640 10 monclient(hunting): authenticate will time out at 1963.942805s 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0334136f90 0x7f0334139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f0334136f90 0x7f0334139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59954/0 (socket says 172.21.15.17:59954) 2024-09-15T02:11:43.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54912/0 (socket says 172.21.15.17:54912) 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 1 -- 172.21.15.17:0/2712419722 learned_addr learned my addr 172.21.15.17:0/2712419722 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 10 monclient(hunting): get_auth_request con 0x7f0334136f90 auth_method 0 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f03337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 10 monclient(hunting): get_auth_request con 0x7f0334067090 auth_method 0 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.946+0000 7f0332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient(hunting): handle_auth_done global_id 234222 payload 274 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 1 -- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 msgr2=0x7f03341321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 1 -- 172.21.15.17:0/2712419722 >> v1:172.21.15.67:6789/0 conn(0x7f0334067460 legacy=0x7f0334067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 1 monclient: found mon.noname-a 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f03337fe640 1 -- 172.21.15.17:0/2712419722 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033409e1e0 con 0x7f0334136f90 2024-09-15T02:11:43.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.947+0000 7f0332ffd640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f03337fe640 1 -- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334136f90 msgr2=0x7f0334139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f03337fe640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334136f90 0x7f0334139370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f032c009f20 tx=0x7f032c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437d3ba297b8b04c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 1 -- 172.21.15.17:0/2712419722 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c007610 con 0x7f0334136f90 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: dump: 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.959 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 1 -- 172.21.15.17:0/2712419722 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c02d050 con 0x7f0334136f90 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 5 monclient: authenticate success, global_id 234222 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 1 -- 172.21.15.17:0/2712419722 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c0079e0 con 0x7f0334136f90 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 4 set_mon_vals no callback set 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f0331ffb640 10 monclient: dump: 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 10 monclient: shutdown 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 -- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334136f90 msgr2=0x7f0334139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334136f90 0x7f0334139370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f032c009f20 tx=0x7f032c009840 comp rx=0 tx=0).stop 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 -- 172.21.15.17:0/2712419722 shutdown_connections 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334136f90 0x7f0334139370 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 --2- 172.21.15.17:0/2712419722 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334067090 0x7f03341321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 -- 172.21.15.17:0/2712419722 >> 172.21.15.17:0/2712419722 conn(0x7f03340a1ee0 msgr2=0x7f03340a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 -- 172.21.15.17:0/2712419722 shutdown_connections 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.950+0000 7f033986f640 1 -- 172.21.15.17:0/2712419722 wait complete. 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.951+0000 7f033986f640 10 monclient: build_initial_monmap 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.951+0000 7f033986f640 10 monclient: monmap: 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.951+0000 7f033986f640 1 Processor -- start 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.951+0000 7f033986f640 1 -- start start 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.951+0000 7f033986f640 10 monclient: init 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:43.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 0x7f0334118540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): picked mon.noname-a con 0x7f0334067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0334067460 0x7f0334118a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): picked mon.noname-b con 0x7f0334067460 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334136f90 0x7f0334118fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): picked mon.noname-c con 0x7f0334136f90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033409d530 con 0x7f0334067090 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033409fe80 con 0x7f0334067460 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033411ba00 con 0x7f0334136f90 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:43.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f033986f640 10 monclient(hunting): authenticate will time out at 1963.948270s 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 0x7f0334118540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334136f90 0x7f0334118fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 0x7f0334118540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59960/0 (socket says 172.21.15.17:59960) 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 -- 172.21.15.17:0/937034143 learned_addr learned my addr 172.21.15.17:0/937034143 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): get_auth_request con 0x7f0334067090 auth_method 0 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): get_auth_request con 0x7f0334136f90 auth_method 0 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03327fc640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0334067460 0x7f0334118a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03327fc640 10 monclient(hunting): get_auth_request con 0x7f0334067460 auth_method 0 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f03327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient(hunting): handle_auth_done global_id 234225 payload 824 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334136f90 msgr2=0x7f0334118fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334136f90 0x7f0334118fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0334067460 msgr2=0x7f0334118a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0334067460 0x7f0334118a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 monclient: found mon.noname-a 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.952+0000 7f0332ffd640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324009390 con 0x7f0334067090 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.954+0000 7f0332ffd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 msgr2=0x7f0334118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.954+0000 7f0332ffd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 0x7f0334118540 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f032400ec80 tx=0x7f03240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032400eec0 con 0x7f0334067090 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:43.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: dump: 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: _finish_auth 0 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032400cdd0 con 0x7f0334067090 2024-09-15T02:11:43.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 5 monclient: authenticate success, global_id 234225 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324013680 con 0x7f0334067090 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 10 monclient: _renew_subs 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c009ae0 con 0x7f0334067090 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f031affd640 10 monclient: dump: 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.969 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f03317fa640 4 set_mon_vals no callback set 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 10 monclient: _renew_subs 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.955+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033411c350 con 0x7f0334067090 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f033986f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f033986f640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 0x7f0334119500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0334063870 con 0x7f033411a7a0 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f03240139f0 con 0x7f0334067090 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 0x7f0334119500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient: get_auth_request con 0x7f033411a7a0 auth_method 0 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 10 monclient(hunting): handle_auth_done global_id 214391 payload 824 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03327fc640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03340724a0 con 0x7f033411a7a0 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f031affd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0320038740 0x7f032003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f032404ca50 con 0x7f0334067090 2024-09-15T02:11:43.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03337fe640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0320038740 0x7f032003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.956+0000 7f03337fe640 10 monclient: get_auth_request con 0x7f0320038740 auth_method 0 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f03327fc640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 msgr2=0x7f0334119500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f03327fc640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 0x7f0334119500 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f0328009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328007450 con 0x7f033411a7a0 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f031affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.958+0000 7f031affd640 10 monclient: dump: 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.974 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f03280075b0 con 0x7f033411a7a0 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 msgr2=0x7f0334119500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 0x7f0334119500 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f0328009040 comp rx=0 tx=0).stop 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 msgr2=0x7f0334119500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f031affd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f033411a7a0 0x7f0334119500 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f0328009040 comp rx=0 tx=0).stop 2024-09-15T02:11:43.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.959+0000 7f03337fe640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0320038740 0x7f032003abf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f032c033000 tx=0x7f032c036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f033986f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f033986f640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 0x7f0334132e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f033986f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033411bb40 con 0x7f0334132a80 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f03327fc640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 0x7f0334132e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f03327fc640 10 monclient: get_auth_request con 0x7f0334132a80 auth_method 0 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f03327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f03327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.964+0000 7f03327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.965+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.965+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.965+0000 7f03327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:43.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.965+0000 7f03327fc640 10 monclient(hunting): handle_auth_done global_id 214394 payload 824 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.965+0000 7f03327fc640 1 -- 172.21.15.17:0/937034143 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f03340689a0 con 0x7f0334132a80 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f03327fc640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 msgr2=0x7f0334132e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f03327fc640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 0x7f0334132e80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f032801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328013c40 con 0x7f0334132a80 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 10 monclient: dump: 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:43.977 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f0328013da0 con 0x7f0334132a80 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 msgr2=0x7f0334132e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 0x7f0334132e80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f032801b000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 msgr2=0x7f0334132e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.967+0000 7f031affd640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0334132a80 0x7f0334132e80 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f032800e510 tx=0x7f032801b000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.978 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:43.979 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:43.980 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:43.981 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:43.982 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:43.983 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:43.986 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:43.987 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.988 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:43.991 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:43.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0320038740 msgr2=0x7f032003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0320038740 0x7f032003abf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f032c033000 tx=0x7f032c036000 comp rx=0 tx=0).stop 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 10 monclient: shutdown 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 msgr2=0x7f0334118540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 --2- 172.21.15.17:0/937034143 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0334067090 0x7f0334118540 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f032400ec80 tx=0x7f03240090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f03337fe640 1 -- 172.21.15.17:0/937034143 reap_dead start 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f03337fe640 1 -- 172.21.15.17:0/937034143 reap_dead start 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 shutdown_connections 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 >> 172.21.15.17:0/937034143 conn(0x7f03340a1ee0 msgr2=0x7f03341310d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 shutdown_connections 2024-09-15T02:11:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:43.968+0000 7f033986f640 1 -- 172.21.15.17:0/937034143 wait complete. 2024-09-15T02:11:43.993 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:43.994 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:11:43.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:11:44.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.089+0000 7ff4df95d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:44.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.089+0000 7ff4df95d640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.091 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.089+0000 7ff4df95d640 10 monclient: monmap: 2024-09-15T02:11:44.091 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:44.091 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:44.090788+0000 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:44.090788+0000 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:44.092 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.090+0000 7ff4df95d640 1 Processor -- start 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.090+0000 7ff4df95d640 1 -- start start 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.090+0000 7ff4df95d640 10 monclient: init 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4d809a9f0 0x7ff4d809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d809a9f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4d812e420 0x7ff4d809b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d812e420 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4d812e810 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:44.095 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d8098fb0 con 0x7ff4d812e810 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d812e130 con 0x7ff4d809a9f0 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d809b300 con 0x7ff4d812e420 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4df95d640 10 monclient(hunting): authenticate will time out at 1964.087304s 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4d809a9f0 0x7ff4d809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff4d809a9f0 0x7ff4d809adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59964/0 (socket says 172.21.15.17:59964) 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/2656544928 learned_addr learned my addr 172.21.15.17:0/2656544928 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): get_auth_request con 0x7ff4d809a9f0 auth_method 0 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.096 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/2656544928 >> v2:172.21.15.67:3300/0 conn(0x7ff4d812e420 0x7ff4d809b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request con 0x7ff4d812e420 auth_method 0 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dcef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dcef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_done global_id 234228 payload 274 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/2656544928 >> v2:172.21.15.67:3300/0 conn(0x7ff4d812e420 msgr2=0x7ff4d809b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 --2- 172.21.15.17:0/2656544928 >> v2:172.21.15.67:3300/0 conn(0x7ff4d812e420 0x7ff4d809b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.097 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/2656544928 >> v1:172.21.15.67:6789/0 conn(0x7ff4d812e810 legacy=0x7ff4d809bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 monclient: found mon.noname-a 2024-09-15T02:11:44.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.098 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.091+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/2656544928 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8098cd0 con 0x7ff4d809a9f0 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/2656544928 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 msgr2=0x7ff4d809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4dd6f9640 1 --2- 172.21.15.17:0/2656544928 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d809adc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80769e0 tx=0x7ff4d400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d812dcba1c700e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 1 -- 172.21.15.17:0/2656544928 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4032530 con 0x7ff4d809a9f0 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: dump: 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 1 -- 172.21.15.17:0/2656544928 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4032690 con 0x7ff4d809a9f0 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4df95d640 5 monclient: authenticate success, global_id 234228 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4df95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:44.105 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4df95d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 1 -- 172.21.15.17:0/2656544928 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4032a30 con 0x7ff4d809a9f0 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4df95d640 4 set_mon_vals no callback set 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.094+0000 7ff4cffff640 10 monclient: dump: 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.106 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 10 monclient: shutdown 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 -- 172.21.15.17:0/2656544928 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 msgr2=0x7ff4d809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 --2- 172.21.15.17:0/2656544928 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d809adc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80769e0 tx=0x7ff4d400b460 comp rx=0 tx=0).stop 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 -- 172.21.15.17:0/2656544928 shutdown_connections 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 --2- 172.21.15.17:0/2656544928 >> v2:172.21.15.67:3300/0 conn(0x7ff4d812e420 0x7ff4d809b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 --2- 172.21.15.17:0/2656544928 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d809adc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 -- 172.21.15.17:0/2656544928 >> 172.21.15.17:0/2656544928 conn(0x7ff4d80a1000 msgr2=0x7ff4d80a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 -- 172.21.15.17:0/2656544928 shutdown_connections 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 1 -- 172.21.15.17:0/2656544928 wait complete. 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.095+0000 7ff4df95d640 10 monclient: monmap: 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 Processor -- start 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 -- start start 2024-09-15T02:11:44.108 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient: init 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d809a9f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d812e420 0x7ff4d8127a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d812e420 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d812e810 0x7ff4d806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d812e810 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8117bb0 con 0x7ff4d809a9f0 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d813df70 con 0x7ff4d812e420 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d809f2e0 con 0x7ff4d812e810 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4df95d640 10 monclient(hunting): authenticate will time out at 1964.092730s 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59970/0 (socket says 172.21.15.17:59970) 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/4108010625 learned_addr learned my addr 172.21.15.17:0/4108010625 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): get_auth_request con 0x7ff4d809a9f0 auth_method 0 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4ddefa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d812e810 0x7ff4d806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4ddefa640 10 monclient(hunting): get_auth_request con 0x7ff4d812e810 auth_method 0 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4ddefa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4ddefa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4ddefa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d812e420 0x7ff4d8127a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request con 0x7ff4d812e420 auth_method 0 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dcef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.096+0000 7ff4dcef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4ddefa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4ddefa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4ddefa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 10 monclient(hunting): handle_auth_done global_id 234231 payload 824 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d812e810 msgr2=0x7ff4d806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d812e810 0x7ff4d806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d812e420 msgr2=0x7ff4d8127a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d812e420 0x7ff4d8127a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 monclient: found mon.noname-a 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.097+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d400b700 con 0x7ff4d809a9f0 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4dd6f9640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 msgr2=0x7ff4d813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4dd6f9640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400b6d0 tx=0x7ff4d403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4034ac0 con 0x7ff4d809a9f0 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.113 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4042070 con 0x7ff4d809a9f0 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 5 monclient: authenticate success, global_id 234231 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 10 monclient: _renew_subs 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d80983b0 con 0x7ff4d809a9f0 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4034e90 con 0x7ff4d809a9f0 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:44.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4cf7fe640 4 set_mon_vals no callback set 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 10 monclient: _renew_subs 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.099+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d806f0d0 con 0x7ff4d809a9f0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff4d4032070 con 0x7ff4d809a9f0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4df95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4df95d640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 0x7ff4d806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8063c20 con 0x7ff4d8127fa0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4cd7fa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4bc038740 0x7ff4bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff4d4044070 con 0x7ff4d809a9f0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 0x7ff4d806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient: get_auth_request con 0x7ff4d8127fa0 auth_method 0 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.116 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4ddefa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4bc038740 0x7ff4bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4ddefa640 10 monclient: get_auth_request con 0x7ff4bc038740 auth_method 0 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.100+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.101+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.101+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_done global_id 244114 payload 824 2024-09-15T02:11:44.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.101+0000 7ff4dcef8640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d80724a0 con 0x7ff4d8127fa0 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4ddefa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4bc038740 0x7ff4bc03abf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8117ae0 tx=0x7ff4d00025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4dcef8640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 msgr2=0x7ff4d806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 0x7ff4d806cf90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4d812bb10 tx=0x7ff4c8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8010eb0 con 0x7ff4d8127fa0 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:44.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.103+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff4c800ed20 con 0x7ff4d8127fa0 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 msgr2=0x7ff4d806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 0x7ff4d806cf90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4d812bb10 tx=0x7ff4c8008040 comp rx=0 tx=0).stop 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 msgr2=0x7ff4d806cf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.104+0000 7ff4cd7fa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8127fa0 0x7ff4d806cf90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.109+0000 7ff4df95d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.109+0000 7ff4df95d640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 0x7ff4d812bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.109+0000 7ff4df95d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.109+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8128370 con 0x7ff4d806d4d0 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 0x7ff4d812bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient: get_auth_request con 0x7ff4d806d4d0 auth_method 0 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 10 monclient(hunting): handle_auth_done global_id 244117 payload 824 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.110+0000 7ff4dcef8640 1 -- 172.21.15.17:0/4108010625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff4d806d8d0 con 0x7ff4d806d4d0 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.112+0000 7ff4dcef8640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 msgr2=0x7ff4d812bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.112+0000 7ff4dcef8640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 0x7ff4d812bbb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4d812bb10 tx=0x7ff4c801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8024070 con 0x7ff4d806d4d0 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.122 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7ff4c800eb10 con 0x7ff4d806d4d0 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 msgr2=0x7ff4d812bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 0x7ff4d812bbb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4d812bb10 tx=0x7ff4c801b000 comp rx=0 tx=0).stop 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 msgr2=0x7ff4d812bbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:44.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4cd7fa640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d806d4d0 0x7ff4d812bbb0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4d812bb10 tx=0x7ff4c801b000 comp rx=0 tx=0).stop 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:44.124 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:44.125 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:44.126 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:44.127 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:44.128 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:44.136 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:44.137 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:44.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4bc038740 msgr2=0x7ff4bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4bc038740 0x7ff4bc03abf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8117ae0 tx=0x7ff4d00025b0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 10 monclient: shutdown 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 msgr2=0x7ff4d813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400b6d0 tx=0x7ff4d403a000 comp rx=0 tx=0).stop 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4ddefa640 1 -- 172.21.15.17:0/4108010625 reap_dead start 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 shutdown_connections 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 --2- 172.21.15.17:0/4108010625 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d809a9f0 0x7ff4d813d090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 >> 172.21.15.17:0/4108010625 conn(0x7ff4d80a1000 msgr2=0x7ff4d80997b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.113+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 shutdown_connections 2024-09-15T02:11:44.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.114+0000 7ff4df95d640 1 -- 172.21.15.17:0/4108010625 wait complete. 2024-09-15T02:11:44.140 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:44.140 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:11:44.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.234+0000 7f1b48234640 10 monclient: get_monmap_and_config 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.234+0000 7f1b48234640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.234+0000 7f1b48234640 10 monclient: monmap: 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:44.235555+0000 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:44.235555+0000 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:44.237 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.235+0000 7f1b48234640 1 Processor -- start 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.235+0000 7f1b48234640 1 -- start start 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.235+0000 7f1b48234640 10 monclient: init 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.235+0000 7f1b48234640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.235+0000 7f1b48234640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 0x7f1b40131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-b con 0x7f1b401319a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b401322b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b4009f1d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400a1530 con 0x7f1b4009f1d0 2024-09-15T02:11:44.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b40132680 con 0x7f1b401322b0 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4009fca0 con 0x7f1b401319a0 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b48234640 10 monclient(hunting): authenticate will time out at 1964.232075s 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b467d1640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1b4009f1d0 0x7f1b40138ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:45850/0 (socket says 172.21.15.17:45850) 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b467d1640 1 -- 172.21.15.17:0/2964434568 learned_addr learned my addr 172.21.15.17:0/2964434568 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 1 --2- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 0x7f1b40131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): get_auth_request con 0x7f1b401319a0 auth_method 0 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_done global_id 214400 payload 274 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 1 -- 172.21.15.17:0/2964434568 >> v1:172.21.15.67:6789/0 conn(0x7f1b401322b0 legacy=0x7f1b4009eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 1 -- 172.21.15.17:0/2964434568 >> v1:172.21.15.17:6789/0 conn(0x7f1b4009f1d0 legacy=0x7f1b40138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 1 monclient: found mon.noname-b 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.242 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.236+0000 7f1b45fd0640 1 -- 172.21.15.17:0/2964434568 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400a00e0 con 0x7f1b401319a0 2024-09-15T02:11:44.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b45fd0640 1 -- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 msgr2=0x7f1b40131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b45fd0640 1 --2- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 0x7f1b40131d70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009880 tx=0x7f1b3c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6348addae42d853 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 1 -- 172.21.15.17:0/2964434568 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c032440 con 0x7f1b401319a0 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: dump: 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.248 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 1 -- 172.21.15.17:0/2964434568 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c0325a0 con 0x7f1b401319a0 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b48234640 5 monclient: authenticate success, global_id 214400 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b48234640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b48234640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 1 -- 172.21.15.17:0/2964434568 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c032940 con 0x7f1b401319a0 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b48234640 4 set_mon_vals no callback set 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.239+0000 7f1b44fce640 10 monclient: dump: 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.250 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 10 monclient: shutdown 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 -- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 msgr2=0x7f1b40131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 --2- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 0x7f1b40131d70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009880 tx=0x7f1b3c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 -- 172.21.15.17:0/2964434568 shutdown_connections 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 --2- 172.21.15.17:0/2964434568 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b401319a0 0x7f1b40131d70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.251 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 -- 172.21.15.17:0/2964434568 >> 172.21.15.17:0/2964434568 conn(0x7f1b400a5000 msgr2=0x7f1b400a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 -- 172.21.15.17:0/2964434568 shutdown_connections 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 1 -- 172.21.15.17:0/2964434568 wait complete. 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.240+0000 7f1b48234640 10 monclient: monmap: 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.252 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 Processor -- start 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 -- start start 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient: init 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4009f1d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:44.253 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b401319a0 0x7f1b40134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-b con 0x7f1b401319a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 0x7f1b40134870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): picked mon.noname-a con 0x7f1b401322b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b401378f0 con 0x7f1b401322b0 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400a4c80 con 0x7f1b401319a0 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b40137bd0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b48234640 10 monclient(hunting): authenticate will time out at 1964.237706s 2024-09-15T02:11:44.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54954/0 (socket says 172.21.15.17:54954) 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 0x7f1b40134870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 1 -- 172.21.15.17:0/3415566058 learned_addr learned my addr 172.21.15.17:0/3415566058 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): get_auth_request con 0x7f1b4009f1d0 auth_method 0 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): get_auth_request con 0x7f1b401322b0 auth_method 0 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b467d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b457cf640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b401319a0 0x7f1b40134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b457cf640 10 monclient(hunting): get_auth_request con 0x7f1b401319a0 auth_method 0 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b457cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b457cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.241+0000 7f1b457cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 10 monclient(hunting): handle_auth_done global_id 214403 payload 824 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 -- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b401319a0 msgr2=0x7f1b40134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b401319a0 0x7f1b40134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 -- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 msgr2=0x7f1b40134870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 0x7f1b40134870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 monclient: found mon.noname-c 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b45fd0640 1 -- 172.21.15.17:0/3415566058 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c009390 con 0x7f1b4009f1d0 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.242+0000 7f1b467d1640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 0x7f1b40134870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b45fd0640 1 -- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 msgr2=0x7f1b401385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b45fd0640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009880 tx=0x7f1b3c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c031990 con 0x7f1b4009f1d0 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: dump: 2024-09-15T02:11:44.258 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c031af0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c031e90 con 0x7f1b4009f1d0 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b35ffb640 10 monclient: dump: 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 5 monclient: authenticate success, global_id 214403 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 10 monclient: _renew_subs 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b37fff640 4 set_mon_vals no callback set 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b40139720 con 0x7f1b4009f1d0 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 10 monclient: _renew_subs 2024-09-15T02:11:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.244+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b40138c90 con 0x7f1b4009f1d0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.245+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1b3c039040 con 0x7f1b4009f1d0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.245+0000 7f1b48234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.245+0000 7f1b48234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.245+0000 7f1b48234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.245+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4009bfc0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.246+0000 7f1b35ffb640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 0x7f1b2403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.246+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1b3c06cc90 con 0x7f1b4009f1d0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.246+0000 7f1b457cf640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 0x7f1b2403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.246+0000 7f1b457cf640 10 monclient: get_auth_request con 0x7f1b24038790 auth_method 0 2024-09-15T02:11:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.248+0000 7f1b457cf640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 0x7f1b2403ac40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1b40135ac0 tx=0x7f1b30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.250+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1b4009bfc0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.250+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.250+0000 7f1b35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:44.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.482+0000 7f1b48234640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:11:44.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.482+0000 7f1b48234640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:11:44.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.482+0000 7f1b48234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:44.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.482+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1b4009bfc0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.485 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.485 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=11,kv=7,logm=54} crc {auth=417221073,config=1228700967,health=1194250342,kv=3442201590,logm=577380807}) 2024-09-15T02:11:44.486 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=3450614150,config=3140514009,health=1975858333,kv=1529054990,logm=99259161}) 2024-09-15T02:11:44.486 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.486 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=11,kv=7,logm=54} crc {auth=417221073,config=1228700967,health=1194250342,kv=3442201590,logm=577380807}) 2024-09-15T02:11:44.486 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=7,logm=52} crc {auth=4222339273,config=3140514009,health=247690561,kv=947847570,logm=4285755613}) 2024-09-15T02:11:44.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f1b35ffb640 1 -- 172.21.15.17:0/3415566058 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1b4009bfc0 con 0x7f1b4009f1d0 2024-09-15T02:11:44.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:11:44.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.484+0000 7f1b35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077473562}) 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1015168839}) 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077473562}) 2024-09-15T02:11:44.487 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.485+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786233840}) 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 msgr2=0x7f1b2403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 0x7f1b2403ac40 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1b40135ac0 tx=0x7f1b30009040 comp rx=0 tx=0).stop 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 10 monclient: shutdown 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 msgr2=0x7f1b401385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009880 tx=0x7f1b3c032000 comp rx=0 tx=0).stop 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 shutdown_connections 2024-09-15T02:11:44.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1b24038790 0x7f1b2403ac40 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1b401322b0 0x7f1b40134870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b401319a0 0x7f1b40134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 --2- 172.21.15.17:0/3415566058 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1b4009f1d0 0x7f1b401385b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 >> 172.21.15.17:0/3415566058 conn(0x7f1b400a5000 msgr2=0x7f1b4009e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 shutdown_connections 2024-09-15T02:11:44.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:44.486+0000 7f1b48234640 1 -- 172.21.15.17:0/3415566058 wait complete. 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167107476}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545164421}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167107476}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.487+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827322053}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624430679}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214198093}) 2024-09-15T02:11:44.490 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624430679}) 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.489+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=400259163}) 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170721324}) 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652931196}) 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.491 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170721324}) 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.490+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870075234}) 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272227120}) 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372963790}) 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.492 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272227120}) 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132249107}) 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567249485}) 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3791650057}) 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.493 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567249485}) 2024-09-15T02:11:44.494 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.491+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1436848796}) 2024-09-15T02:11:44.494 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063967549}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2849929245}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063967549}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.492+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382396291}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491478620}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663077332}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491478620}) 2024-09-15T02:11:44.495 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.493+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751848160}) 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147649461}) 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360592288}) 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147649461}) 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143838246}) 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.497 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304830382}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806317411}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304830382}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.494+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018305360}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561569009}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4241694823}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561569009}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.495+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265976290}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425815080}) 2024-09-15T02:11:44.498 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981019238}) 2024-09-15T02:11:44.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425815080}) 2024-09-15T02:11:44.499 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315324963}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187373058}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337053939}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187373058}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.496+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3361410606}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310879600}) 2024-09-15T02:11:44.500 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808889294}) 2024-09-15T02:11:44.501 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.501 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310879600}) 2024-09-15T02:11:44.501 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.497+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770384460}) 2024-09-15T02:11:44.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:11:44.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986124065}) 2024-09-15T02:11:44.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30889371}) 2024-09-15T02:11:44.502 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986124065}) 2024-09-15T02:11:44.503 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.498+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230940491}) 2024-09-15T02:11:44.514 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=13 2024-09-15T02:11:44.514 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T02:11:44.514 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:11:44.514 INFO:tasks.workunit.client.0.smithi067.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T02:11:44.515 INFO:tasks.workunit.client.0.smithi067.stderr:+ cat 2024-09-15T02:11:44.515 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph auth import -i - 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.592+0000 7fb1390a8640 10 monclient: get_monmap_and_config 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.592+0000 7fb1390a8640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.592+0000 7fb1390a8640 10 monclient: monmap: 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.597 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:11:44.593664+0000 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:11:44.593664+0000 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 Processor -- start 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 -- start start 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient: init 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.598 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1340696c0 0x7fb13409c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb1340696c0 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb13409d4d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb13406ed00 0x7fb13409e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb13406ed00 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13406a5e0 con 0x7fb13409d4d0 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13409f7f0 con 0x7fb13406ed00 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13409fa30 con 0x7fb1340696c0 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb1390a8640 10 monclient(hunting): authenticate will time out at 1937.062695s 2024-09-15T02:11:44.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1340696c0 0x7fb13409c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb13259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb13409d4d0 0x7fb13409dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41048/0 (socket says 172.21.15.67:41048) 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb13259c640 1 -- 172.21.15.67:0/3923053600 learned_addr learned my addr 172.21.15.67:0/3923053600 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): get_auth_request con 0x7fb1340696c0 auth_method 0 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.593+0000 7fb13359e640 1 --2- 172.21.15.67:0/3923053600 >> v2:172.21.15.17:3300/0 conn(0x7fb13406ed00 0x7fb13409e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): get_auth_request con 0x7fb13406ed00 auth_method 0 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 10 monclient(hunting): handle_auth_done global_id 244126 payload 274 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 1 -- 172.21.15.67:0/3923053600 >> v2:172.21.15.17:3300/0 conn(0x7fb13406ed00 msgr2=0x7fb13409e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 1 --2- 172.21.15.67:0/3923053600 >> v2:172.21.15.17:3300/0 conn(0x7fb13406ed00 0x7fb13409e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 1 -- 172.21.15.67:0/3923053600 >> v1:172.21.15.67:6789/0 conn(0x7fb13409d4d0 legacy=0x7fb13409dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 1 monclient: found mon.noname-c 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:11:44.601 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.594+0000 7fb132d9d640 1 -- 172.21.15.67:0/3923053600 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13406a270 con 0x7fb1340696c0 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.596+0000 7fb132d9d640 1 -- 172.21.15.67:0/3923053600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 msgr2=0x7fb13409c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.596+0000 7fb132d9d640 1 --2- 172.21.15.67:0/3923053600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb13409c790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb13406b2f0 tx=0x7fb1240090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d6854554372cd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 1 -- 172.21.15.67:0/3923053600 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12402ae00 con 0x7fb1340696c0 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: dump: 2024-09-15T02:11:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 1 -- 172.21.15.67:0/3923053600 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb124032440 con 0x7fb1340696c0 2024-09-15T02:11:44.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 5 monclient: authenticate success, global_id 244126 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 1 -- 172.21.15.67:0/3923053600 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1240327e0 con 0x7fb1340696c0 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 4 set_mon_vals no callback set 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb131d9b640 10 monclient: dump: 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 10 monclient: shutdown 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 -- 172.21.15.67:0/3923053600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 msgr2=0x7fb13409c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3923053600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb13409c790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb13406b2f0 tx=0x7fb1240090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 -- 172.21.15.67:0/3923053600 shutdown_connections 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3923053600 >> v2:172.21.15.17:3300/0 conn(0x7fb13406ed00 0x7fb13409e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3923053600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb13409c790 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 -- 172.21.15.67:0/3923053600 >> 172.21.15.67:0/3923053600 conn(0x7fb1340714c0 msgr2=0x7fb1340a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:44.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 -- 172.21.15.67:0/3923053600 shutdown_connections 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.597+0000 7fb1390a8640 1 -- 172.21.15.67:0/3923053600 wait complete. 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.598+0000 7fb1390a8640 10 monclient: build_initial_monmap 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.598+0000 7fb1390a8640 10 monclient: monmap: 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.598+0000 7fb1390a8640 1 Processor -- start 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.598+0000 7fb1390a8640 1 -- start start 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.598+0000 7fb1390a8640 10 monclient: init 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb1340696c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb13406ed00 0x7fb134134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb13406ed00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 0x7fb134134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb13409d4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134069f20 con 0x7fb13409d4d0 2024-09-15T02:11:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134134f60 con 0x7fb1340696c0 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb134121740 con 0x7fb13406ed00 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb1390a8640 10 monclient(hunting): authenticate will time out at 1937.068340s 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49954/0 (socket says 172.21.15.67:49954) 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 1 -- 172.21.15.67:0/3351784321 learned_addr learned my addr 172.21.15.67:0/3351784321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): get_auth_request con 0x7fb1340696c0 auth_method 0 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.613 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 0x7fb134134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): get_auth_request con 0x7fb13409d4d0 auth_method 0 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb13406ed00 0x7fb134134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): get_auth_request con 0x7fb13406ed00 auth_method 0 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient(hunting): handle_auth_done global_id 234246 payload 824 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 -- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb13406ed00 msgr2=0x7fb134134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb13406ed00 0x7fb134134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 -- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 msgr2=0x7fb1341417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient: _finish_hunting 0 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 monclient: found mon.noname-a 2024-09-15T02:11:44.615 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb13359e640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124009390 con 0x7fb13409d4d0 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.599+0000 7fb132d9d640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb13359e640 1 -- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 msgr2=0x7fb134134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb13359e640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 0x7fb134134880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009e80 tx=0x7fb12c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c014480 con 0x7fb13409d4d0 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: dump: 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.616 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: _finish_auth 0 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c00f040 con 0x7fb13409d4d0 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 5 monclient: authenticate success, global_id 234246 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 10 monclient: _renew_subs 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.617 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb134142680 con 0x7fb13409d4d0 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c014830 con 0x7fb13409d4d0 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb11affd640 10 monclient: dump: 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:44.618 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 10 monclient: _renew_subs 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb13159a640 4 set_mon_vals no callback set 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.602+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb134120500 con 0x7fb13409d4d0 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb12c014ba0 con 0x7fb13409d4d0 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb1390a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb1390a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb1390a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1341218b0 con 0x7fb13409d4d0 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb11affd640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 0x7fb12003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:44.619 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb12c04d4b0 con 0x7fb13409d4d0 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb132d9d640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 0x7fb12003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.603+0000 7fb132d9d640 10 monclient: get_auth_request con 0x7fb120038740 auth_method 0 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.606+0000 7fb132d9d640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 0x7fb12003abf0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb124009130 tx=0x7fb124032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.608+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb12c024300 con 0x7fb13409d4d0 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.608+0000 7fb11affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:44.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.608+0000 7fb11affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:44.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.832+0000 7fb1390a8640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-15T02:11:44.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.832+0000 7fb1390a8640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-15T02:11:44.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.832+0000 7fb1390a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.832+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fb13404bc80 con 0x7fb13409d4d0 2024-09-15T02:11:44.834 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.833+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1586230190}) 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265981018}) 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1586230190}) 2024-09-15T02:11:44.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.834+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827741812}) 2024-09-15T02:11:44.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.836+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "auth import"}]=0 v37) v1 ==== 59+0+0 (secure 0 0 0) 0x7fb12c018070 con 0x7fb13409d4d0 2024-09-15T02:11:44.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.836+0000 7fb11affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-15T02:11:44.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.836+0000 7fb11affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332081959}) 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170839315}) 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332081959}) 2024-09-15T02:11:44.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.837+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717552041}) 2024-09-15T02:11:44.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.996+0000 7fb1390a8640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-15T02:11:44.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.996+0000 7fb1390a8640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-15T02:11:44.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.996+0000 7fb1390a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:44.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:44.996+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fb13404b720 con 0x7fb13409d4d0 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297366794}) 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487269096}) 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297366794}) 2024-09-15T02:11:45.000 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:44.998+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=55009153}) 2024-09-15T02:11:45.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.001+0000 7fb11affd640 1 -- 172.21.15.67:0/3351784321 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "auth import"}]=0 v38) v1 ==== 59+0+0 (secure 0 0 0) 0x7fb12c040070 con 0x7fb13409d4d0 2024-09-15T02:11:45.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.001+0000 7fb11affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-15T02:11:45.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.001+0000 7fb11affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:11:45.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499115940}) 2024-09-15T02:11:45.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2914627178}) 2024-09-15T02:11:45.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.003 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499115940}) 2024-09-15T02:11:45.004 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.001+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130230169}) 2024-09-15T02:11:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 msgr2=0x7fb12003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 0x7fb12003abf0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb124009130 tx=0x7fb124032000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 10 monclient: shutdown 2024-09-15T02:11:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 msgr2=0x7fb134134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 0x7fb134134880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009e80 tx=0x7fb12c00b040 comp rx=0 tx=0).stop 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 shutdown_connections 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb120038740 0x7fb12003abf0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb13409d4d0 0x7fb134134880 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb13406ed00 0x7fb134134340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 --2- 172.21.15.67:0/3351784321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1340696c0 0x7fb1341417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 >> 172.21.15.67:0/3351784321 conn(0x7fb1340714c0 msgr2=0x7fb134135170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 shutdown_connections 2024-09-15T02:11:45.006 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.003+0000 7fb1390a8640 1 -- 172.21.15.67:0/3351784321 wait complete. 2024-09-15T02:11:45.013 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph health 2024-09-15T02:11:45.014 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep AUTH_BAD_CAP 2024-09-15T02:11:45.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:45.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 10 monclient: monmap: 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:11:45.082205+0000 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:11:45.082205+0000 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:45.083 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 1 Processor -- start 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 1 -- start start 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.081+0000 7f150c59d640 10 monclient: init 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1504098540 0x7f1504098910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f1504098540 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1504097890 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1504097c90 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:45.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040a1c20 con 0x7f1504097c90 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040a0470 con 0x7f1504097890 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15040a06b0 con 0x7f1504098540 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150c59d640 10 monclient(hunting): authenticate will time out at 1937.551439s 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1504098540 0x7f1504098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f1509b38640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1504097890 0x7f15040699a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41062/0 (socket says 172.21.15.67:41062) 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1504098540 0x7f1504098910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49958/0 (socket says 172.21.15.67:49958) 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f1509b38640 1 -- 172.21.15.67:0/3758174535 learned_addr learned my addr 172.21.15.67:0/3758174535 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): get_auth_request con 0x7f1504098540 auth_method 0 2024-09-15T02:11:45.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient(hunting): handle_auth_done global_id 244132 payload 274 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 -- 172.21.15.67:0/3758174535 >> v1:172.21.15.67:6789/0 conn(0x7f1504097890 legacy=0x7f15040699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 -- 172.21.15.67:0/3758174535 >> v1:172.21.15.17:6789/0 conn(0x7f1504097c90 legacy=0x7f150406a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 monclient: found mon.noname-c 2024-09-15T02:11:45.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.082+0000 7f150a339640 1 -- 172.21.15.67:0/3758174535 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15040a1940 con 0x7f1504098540 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150a339640 1 -- 172.21.15.67:0/3758174535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 msgr2=0x7f1504098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150a339640 1 --2- 172.21.15.67:0/3758174535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f1504098910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009e90 tx=0x7f14f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ada9662de25e7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 1 -- 172.21.15.67:0/3758174535 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4007840 con 0x7f1504098540 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: dump: 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.095 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150c59d640 5 monclient: authenticate success, global_id 244132 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150c59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 1 -- 172.21.15.67:0/3758174535 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f402d050 con 0x7f1504098540 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150c59d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f150c59d640 4 set_mon_vals no callback set 2024-09-15T02:11:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 1 -- 172.21.15.67:0/3758174535 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4007bd0 con 0x7f1504098540 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.085+0000 7f1509337640 10 monclient: dump: 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.098 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 10 monclient: shutdown 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 -- 172.21.15.67:0/3758174535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 msgr2=0x7f1504098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 --2- 172.21.15.67:0/3758174535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f1504098910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009e90 tx=0x7f14f4009780 comp rx=0 tx=0).stop 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 -- 172.21.15.67:0/3758174535 shutdown_connections 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 --2- 172.21.15.67:0/3758174535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f1504098910 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 -- 172.21.15.67:0/3758174535 >> 172.21.15.67:0/3758174535 conn(0x7f15040a5510 msgr2=0x7f15040a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 -- 172.21.15.67:0/3758174535 shutdown_connections 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 1 -- 172.21.15.67:0/3758174535 wait complete. 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.086+0000 7f150c59d640 10 monclient: monmap: 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 Processor -- start 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 -- start start 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient: init 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1504097890 0x7f1504141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-a con 0x7f1504097890 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1504097c90 0x7f1504132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f1504097c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): picked mon.noname-b con 0x7f1504098540 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150411f160 con 0x7f1504097890 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150411dc20 con 0x7f1504098540 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150411d6a0 con 0x7f1504097c90 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150c59d640 10 monclient(hunting): authenticate will time out at 1937.556820s 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49964/0 (socket says 172.21.15.67:49964) 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 1 -- 172.21.15.67:0/3973300551 learned_addr learned my addr 172.21.15.67:0/3973300551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 10 monclient(hunting): get_auth_request con 0x7f1504098540 auth_method 0 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.087+0000 7f150a339640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1504097890 0x7f1504141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1504097c90 0x7f1504132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): get_auth_request con 0x7f1504097890 auth_method 0 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): get_auth_request con 0x7f1504097c90 auth_method 0 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150a339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f1509b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient(hunting): handle_auth_done global_id 244135 payload 824 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 -- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1504097c90 msgr2=0x7f1504132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1504097c90 0x7f1504132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 -- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1504097890 msgr2=0x7f1504141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1504097890 0x7f1504141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 monclient: found mon.noname-b 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.088+0000 7f150ab3a640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f4009a20 con 0x7f1504098540 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150ab3a640 1 -- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 msgr2=0x7f150411bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150ab3a640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b4f0 tx=0x7f14fc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.105 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00f720 con 0x7f1504098540 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: dump: 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.106 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc00f8c0 con 0x7f1504098540 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 5 monclient: authenticate success, global_id 244135 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 10 monclient: _renew_subs 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1504097370 con 0x7f1504098540 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00fc80 con 0x7f1504098540 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:45.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f15027fc640 10 monclient: dump: 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 10 monclient: _renew_subs 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f1508b36640 4 set_mon_vals no callback set 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.090+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1504141ad0 con 0x7f1504098540 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f14fc011610 con 0x7f1504098540 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f150c59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f150c59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f150c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f15027fc640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 0x7f14f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f14fc04bf90 con 0x7f1504098540 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.091+0000 7f150a339640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 0x7f14f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.092+0000 7f150a339640 10 monclient: get_auth_request con 0x7f14f0038740 auth_method 0 2024-09-15T02:11:45.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.094+0000 7f150a339640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 0x7f14f003abf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009e60 tx=0x7f14f4037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.096+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.096+0000 7f15027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.096+0000 7f15027fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f150c59d640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f150c59d640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f150c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (secure 0 0 0) 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f15027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-15T02:11:45.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.293+0000 7f15027fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562549023}) 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339149435}) 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562549023}) 2024-09-15T02:11:45.296 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.294+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641596290}) 2024-09-15T02:11:45.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562730775}) 2024-09-15T02:11:45.317 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594212739}) 2024-09-15T02:11:45.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562730775}) 2024-09-15T02:11:45.318 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146082004}) 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.316+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796338800}) 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2028490020}) 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796338800}) 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2741700335}) 2024-09-15T02:11:45.319 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343607748}) 2024-09-15T02:11:45.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857172550}) 2024-09-15T02:11:45.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343607748}) 2024-09-15T02:11:45.320 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.317+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110496169}) 2024-09-15T02:11:45.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418015326}) 2024-09-15T02:11:45.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596333887}) 2024-09-15T02:11:45.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.321 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418015326}) 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.318+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125830057}) 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118447209}) 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78644350}) 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118447209}) 2024-09-15T02:11:45.322 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.319+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703082939}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511185460}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022048325}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511185460}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.320+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232002190}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3116082491}) 2024-09-15T02:11:45.324 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037496853}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3116082491}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.321+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076044461}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178231105}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170528809}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178231105}) 2024-09-15T02:11:45.325 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.322+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=542506429}) 2024-09-15T02:11:45.330 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.330 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586312828}) 2024-09-15T02:11:45.330 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579847577}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586312828}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.323+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702351933}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410229759}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282342498}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410229759}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.324+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1301609383}) 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.331 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516592878}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823401798}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516592878}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.325+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2725732168}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405235308}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434820281}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405235308}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.326+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=176327871}) 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.332 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593943827}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4122590869}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593943827}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.327+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2427331346}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946231609}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125211468}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946231609}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.328+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969103379}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001211505}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218251245}) 2024-09-15T02:11:45.333 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.334 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001211505}) 2024-09-15T02:11:45.334 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.329+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80651210}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289366662}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016864724}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289366662}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.330+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108021669}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106497776}) 2024-09-15T02:11:45.338 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667596933}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106497776}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.331+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037918360}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372087281}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869769826}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372087281}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3928724403}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.332+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778526695}) 2024-09-15T02:11:45.339 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004636024}) 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778526695}) 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094014366}) 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891681225}) 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3105519856}) 2024-09-15T02:11:45.340 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891681225}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.333+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=378982558}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74478968}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726346027}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74478968}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.334+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667873228}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187762684}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757950471}) 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.341 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187762684}) 2024-09-15T02:11:45.342 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.335+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346024952}) 2024-09-15T02:11:45.344 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931401893}) 2024-09-15T02:11:45.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682461553}) 2024-09-15T02:11:45.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931401893}) 2024-09-15T02:11:45.345 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.343+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571392349}) 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725964300}) 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689165302}) 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725964300}) 2024-09-15T02:11:45.346 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.344+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524397478}) 2024-09-15T02:11:45.347 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779672619}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908160205}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779672619}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.345+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546199153}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728464334}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873463705}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728464334}) 2024-09-15T02:11:45.348 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.346+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882382786}) 2024-09-15T02:11:45.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911000483}) 2024-09-15T02:11:45.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=241470957}) 2024-09-15T02:11:45.350 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911000483}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.347+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076168342}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201351641}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960411214}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201351641}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1596522986}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124077298}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.348+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342037484}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124077298}) 2024-09-15T02:11:45.351 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808570458}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396969374}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548538199}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396969374}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.349+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021332932}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333597788}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572156427}) 2024-09-15T02:11:45.354 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333597788}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.350+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949743800}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313416395}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678270739}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313416395}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.351+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464854606}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714274147}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956568316}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714274147}) 2024-09-15T02:11:45.355 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.352+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4072361848}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900676725}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412157231}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900676725}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.353+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661052332}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425385305}) 2024-09-15T02:11:45.358 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222346446}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425385305}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.354+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451578832}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4215597573}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838684793}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4215597573}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.355+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859315269}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801734037}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507312542}) 2024-09-15T02:11:45.359 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801734037}) 2024-09-15T02:11:45.360 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.356+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1459287840}) 2024-09-15T02:11:45.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.362 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567629096}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2877309809}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567629096}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.357+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450546579}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725818306}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931034230}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725818306}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.358+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067307624}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013219974}) 2024-09-15T02:11:45.363 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617630025}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013219974}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.359+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631310554}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461397029}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300537970}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461397029}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.360+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243189328}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319134310}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424153216}) 2024-09-15T02:11:45.364 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.365 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319134310}) 2024-09-15T02:11:45.365 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.361+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878135951}) 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706168530}) 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=885348665}) 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706168530}) 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.362+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21441816}) 2024-09-15T02:11:45.367 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29097604}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1380449982}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29097604}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867368933}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577736491}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528289858}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577736491}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.363+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1054066252}) 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.368 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287105780}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161062959}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287105780}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.364+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006594850}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657214592}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912979157}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657214592}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.365+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110821089}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155637908}) 2024-09-15T02:11:45.369 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561656690}) 2024-09-15T02:11:45.370 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.370 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155637908}) 2024-09-15T02:11:45.370 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094356453}) 2024-09-15T02:11:45.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975968152}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180200596}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975968152}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.366+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740358594}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946502009}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013483298}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946502009}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.367+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3911540657}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117052518}) 2024-09-15T02:11:45.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417370910}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117052518}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.368+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558928773}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968169026}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460233044}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968169026}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714106726}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679486466}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306588384}) 2024-09-15T02:11:45.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679486466}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.369+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886110001}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206178369}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448641883}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206178369}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.370+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976647282}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312571823}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2234954802}) 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312571823}) 2024-09-15T02:11:45.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=57006232}) 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=480596113}) 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3683442860}) 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.371+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=480596113}) 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229854213}) 2024-09-15T02:11:45.380 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557133138}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487878107}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557133138}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.372+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936975758}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431459344}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013640211}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431459344}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.373+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1197085005}) 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.381 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923165070}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3836458857}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923165070}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929436287}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167508169}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515263117}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167508169}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.374+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525269428}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325545559}) 2024-09-15T02:11:45.382 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804181287}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325545559}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.375+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3429652387}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723737579}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=687330093}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723737579}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882215130}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885837497}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4016851079}) 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885837497}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.376+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146754936}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319569209}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204996005}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319569209}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.377+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2316181205}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178069421}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155542849}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178069421}) 2024-09-15T02:11:45.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350112312}) 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952380654}) 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229976382}) 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952380654}) 2024-09-15T02:11:45.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.378+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625864333}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883447587}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884680583}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883447587}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.379+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985049626}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604256498}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075291622}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604256498}) 2024-09-15T02:11:45.393 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.380+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2102326462}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590167261}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307824234}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590167261}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2286906048}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041083632}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226933886}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041083632}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.381+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3949077868}) 2024-09-15T02:11:45.394 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920889198}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221493301}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920889198}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.382+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229894978}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239016467}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695260853}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239016467}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290927872}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236475484}) 2024-09-15T02:11:45.395 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510024933}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236475484}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.383+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176265809}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924891749}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938565044}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924891749}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.384+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879585700}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994523440}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769727182}) 2024-09-15T02:11:45.396 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994523440}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2196369321}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730695138}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077954312}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730695138}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.385+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705888503}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647916097}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092432187}) 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.397 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647916097}) 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.386+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4101922231}) 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307632526}) 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928116420}) 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.398 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307632526}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984957296}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611255115}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996617574}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611255115}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.387+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374322381}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069207366}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509938782}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069207366}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.388+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014630376}) 2024-09-15T02:11:45.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236394668}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558046603}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236394668}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484286813}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650353414}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.389+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918363960}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650353414}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837456524}) 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491515698}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=237759920}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491515698}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.390+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837156122}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=2278672191,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2073680172}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=3698361451,mdsmap=1531151339,mgr=1206855712,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3373088050}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=2278672191,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2073680172}) 2024-09-15T02:11:45.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.391+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3} crc {logm=2939243965,mdsmap=1531151339,mgr=2913004691,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=894760682,mon_config_key=2516852880}) 2024-09-15T02:11:45.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgrstat=1639422153,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=4174495578}) 2024-09-15T02:11:45.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=53} crc {mgrstat=2242671514,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1149731095,osd_pg_creating=1863806571,osd_snap=3718248,osdmap=270829888}) 2024-09-15T02:11:45.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:11:45.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgrstat=1639422153,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=4174495578}) 2024-09-15T02:11:45.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:11:45.397+0000 7f409c9f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=55} crc {mgrstat=1639422153,mon_config_key=666446000,monmap=3178194975,osd_metadata=3375641773,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=922745256}) 2024-09-15T02:11:45.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.425+0000 7f150c59d640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.425+0000 7f150c59d640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.425+0000 7f150c59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.425+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.426+0000 7f15027fc640 1 -- 172.21.15.67:0/3973300551 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (secure 0 0 0) 0x7f150411f4a0 con 0x7f1504098540 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.426+0000 7f15027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-15T02:11:45.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.426+0000 7f15027fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 msgr2=0x7f14f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 0x7f14f003abf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009e60 tx=0x7f14f4037000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 10 monclient: shutdown 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 msgr2=0x7f150411bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.430 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b4f0 tx=0x7f14fc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 shutdown_connections 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f14f0038740 0x7f14f003abf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504098540 0x7f150411bb70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1504097c90 0x7f1504132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 --2- 172.21.15.67:0/3973300551 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1504097890 0x7f1504141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 >> 172.21.15.67:0/3973300551 conn(0x7f15040a5510 msgr2=0x7f15040a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 shutdown_connections 2024-09-15T02:11:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.428+0000 7f150c59d640 1 -- 172.21.15.67:0/3973300551 wait complete. 2024-09-15T02:11:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph health detail 2024-09-15T02:11:45.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ grep client.bad 2024-09-15T02:11:45.503 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:11:45.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:45.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 10 monclient: get_monmap_and_config 2024-09-15T02:11:45.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 10 monclient: monmap: 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:11:45.509231+0000 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:11:45.509231+0000 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:45.510 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 1 Processor -- start 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 1 -- start start 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.508+0000 7fd05a754640 10 monclient: init 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T02:11:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd054098540 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd05406a9b0 0x7fd054097890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-a con 0x7fd05406a9b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 0x7fd0541393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-b con 0x7fd054137000 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05406adb0 con 0x7fd054098540 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0540a1c20 con 0x7fd05406a9b0 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0540a0470 con 0x7fd054137000 2024-09-15T02:11:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd05a754640 10 monclient(hunting): authenticate will time out at 1937.978368s 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd053fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd054098540 0x7fd054098910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41066/0 (socket says 172.21.15.67:41066) 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd053fff640 1 -- 172.21.15.67:0/1448570695 learned_addr learned my addr 172.21.15.67:0/1448570695 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 --2- 172.21.15.67:0/1448570695 >> v2:172.21.15.17:3300/0 conn(0x7fd05406a9b0 0x7fd054097890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 0x7fd0541393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): get_auth_request con 0x7fd05406a9b0 auth_method 0 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): get_auth_request con 0x7fd054137000 auth_method 0 2024-09-15T02:11:45.515 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient(hunting): handle_auth_done global_id 234252 payload 274 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 -- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 msgr2=0x7fd0541393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 0x7fd0541393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 -- 172.21.15.67:0/1448570695 >> v1:172.21.15.67:6789/0 conn(0x7fd054098540 legacy=0x7fd054098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 monclient: found mon.noname-a 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd0537fe640 1 -- 172.21.15.67:0/1448570695 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0540a06b0 con 0x7fd05406a9b0 2024-09-15T02:11:45.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.509+0000 7fd058cf1640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 0x7fd0541393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.512+0000 7fd0537fe640 1 -- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd05406a9b0 msgr2=0x7fd054097890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.512+0000 7fd0537fe640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd05406a9b0 0x7fd054097890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd0480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e78bb89f719613 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 1 -- 172.21.15.67:0/1448570695 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04802adf0 con 0x7fd05406a9b0 2024-09-15T02:11:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: dump: 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 5 monclient: authenticate success, global_id 234252 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 1 -- 172.21.15.67:0/1448570695 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048032440 con 0x7fd05406a9b0 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 4 set_mon_vals no callback set 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 1 -- 172.21.15.67:0/1448570695 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480327e0 con 0x7fd05406a9b0 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:45.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd052ffd640 10 monclient: dump: 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 10 monclient: shutdown 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 -- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd05406a9b0 msgr2=0x7fd054097890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd05406a9b0 0x7fd054097890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd0480090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 -- 172.21.15.67:0/1448570695 shutdown_connections 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054137000 0x7fd0541393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 --2- 172.21.15.67:0/1448570695 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd05406a9b0 0x7fd054097890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 -- 172.21.15.67:0/1448570695 >> 172.21.15.67:0/1448570695 conn(0x7fd0540a5510 msgr2=0x7fd0540a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 -- 172.21.15.67:0/1448570695 shutdown_connections 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.513+0000 7fd05a754640 1 -- 172.21.15.67:0/1448570695 wait complete. 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.514+0000 7fd05a754640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.514+0000 7fd05a754640 10 monclient: monmap: 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:45.526 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.514+0000 7fd05a754640 1 Processor -- start 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.514+0000 7fd05a754640 1 -- start start 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.514+0000 7fd05a754640 10 monclient: init 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-b con 0x7fd05406a9b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054098540 0x7fd0541350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-c con 0x7fd054098540 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd054137000 0x7fd05411dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): picked mon.noname-a con 0x7fd054137000 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05411fc40 con 0x7fd054137000 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd054097370 con 0x7fd05406a9b0 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0540a4a20 con 0x7fd054098540 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd05a754640 10 monclient(hunting): authenticate will time out at 1937.984123s 2024-09-15T02:11:45.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49970/0 (socket says 172.21.15.67:49970) 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 -- 172.21.15.67:0/2542891820 learned_addr learned my addr 172.21.15.67:0/2542891820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): get_auth_request con 0x7fd05406a9b0 auth_method 0 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054098540 0x7fd0541350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd054137000 0x7fd05411dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): get_auth_request con 0x7fd054098540 auth_method 0 2024-09-15T02:11:45.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): get_auth_request con 0x7fd054137000 auth_method 0 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd0537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd058cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient(hunting): handle_auth_done global_id 244138 payload 824 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 -- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054098540 msgr2=0x7fd0541350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054098540 0x7fd0541350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 -- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd054137000 msgr2=0x7fd05411dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd054137000 0x7fd05411dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 monclient: found mon.noname-b 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.515+0000 7fd053fff640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c009a20 con 0x7fd05406a9b0 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd053fff640 1 -- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 msgr2=0x7fd0541416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd053fff640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd04400c5e0 tx=0x7fd044004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044005cb0 con 0x7fd05406a9b0 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: dump: 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd05a754640 5 monclient: authenticate success, global_id 244138 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044005e30 con 0x7fd05406a9b0 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd05a754640 10 monclient: _renew_subs 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd05a754640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd048009390 con 0x7fd05406a9b0 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044007320 con 0x7fd05406a9b0 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd03bfff640 10 monclient: dump: 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:45.533 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.534 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.534 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.534 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.534 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:45.535 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.535 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 10 monclient: _renew_subs 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.517+0000 7fd0527fc640 4 set_mon_vals no callback set 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05411f6c0 con 0x7fd05406a9b0 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd04401b3b0 con 0x7fd05406a9b0 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.518+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd054071750 con 0x7fd05406a9b0 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.519+0000 7fd03bfff640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 0x7fd04003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.519+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd04404bb10 con 0x7fd05406a9b0 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.519+0000 7fd0537fe640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 0x7fd04003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.519+0000 7fd0537fe640 10 monclient: get_auth_request con 0x7fd040038740 auth_method 0 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.521+0000 7fd0537fe640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 0x7fd04003abf0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd054068a00 tx=0x7fd048032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.523+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd04401a360 con 0x7fd05406a9b0 2024-09-15T02:11:45.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.523+0000 7fd03bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.523+0000 7fd03bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.602+0000 7f81d699d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:45.603 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.602+0000 7f81d699d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.602+0000 7f81d699d640 10 monclient: monmap: 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:45.603112+0000 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:45.603112+0000 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:45.604 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.602+0000 7f81d699d640 1 Processor -- start 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 -- start start 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient: init 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f81d0070a40 0x7f81d009ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-a con 0x7f81d0070a40 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:45.607 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 0x7f81d009b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-c con 0x7f81d006a550 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81d009b8a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d006eb80 con 0x7f81d009b8a0 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d009dd00 con 0x7f81d0070a40 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d0070f30 con 0x7f81d006a550 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d699d640 10 monclient(hunting): authenticate will time out at 1965.599587s 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f81d0070a40 0x7f81d009ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d4f3a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f81d009b8a0 0x7f81d0131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:45864/0 (socket says 172.21.15.17:45864) 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81d4f3a640 1 -- 172.21.15.17:0/4206163971 learned_addr learned my addr 172.21.15.17:0/4206163971 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d0070a40 auth_method 0 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cf7fe640 1 --2- 172.21.15.17:0/4206163971 >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 0x7f81d009b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d006a550 auth_method 0 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.609 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient(hunting): handle_auth_done global_id 234258 payload 274 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 1 -- 172.21.15.17:0/4206163971 >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 msgr2=0x7f81d009b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 1 --2- 172.21.15.17:0/4206163971 >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 0x7f81d009b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 1 -- 172.21.15.17:0/4206163971 >> v1:172.21.15.17:6789/0 conn(0x7f81d009b8a0 legacy=0x7f81d0131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.603+0000 7f81cffff640 1 monclient: found mon.noname-a 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.604+0000 7f81cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.604+0000 7f81cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.604+0000 7f81cffff640 1 -- 172.21.15.17:0/4206163971 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d009c470 con 0x7f81d0070a40 2024-09-15T02:11:45.610 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.604+0000 7f81cf7fe640 1 --2- 172.21.15.17:0/4206163971 >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 0x7f81d009b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.606+0000 7f81cffff640 1 -- 172.21.15.17:0/4206163971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d0070a40 msgr2=0x7f81d009ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.606+0000 7f81cffff640 1 --2- 172.21.15.17:0/4206163971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d0070a40 0x7f81d009ae20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81d006a010 tx=0x7f81c00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ac7025ff3d9099 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 1 -- 172.21.15.17:0/4206163971 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c002ae00 con 0x7f81d0070a40 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: dump: 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 5 monclient: authenticate success, global_id 234258 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:45.617 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 1 -- 172.21.15.17:0/4206163971 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c0032440 con 0x7f81d0070a40 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 4 set_mon_vals no callback set 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 1 -- 172.21.15.17:0/4206163971 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c00327e0 con 0x7f81d0070a40 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81ceffd640 10 monclient: dump: 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 10 monclient: shutdown 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 -- 172.21.15.17:0/4206163971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d0070a40 msgr2=0x7f81d009ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 --2- 172.21.15.17:0/4206163971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d0070a40 0x7f81d009ae20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81d006a010 tx=0x7f81c00090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 -- 172.21.15.17:0/4206163971 shutdown_connections 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 --2- 172.21.15.17:0/4206163971 >> v2:172.21.15.67:3300/0 conn(0x7f81d006a550 0x7f81d009b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 --2- 172.21.15.17:0/4206163971 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d0070a40 0x7f81d009ae20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 -- 172.21.15.17:0/4206163971 >> 172.21.15.17:0/4206163971 conn(0x7f81d0067ae0 msgr2=0x7f81d00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.619 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 -- 172.21.15.17:0/4206163971 shutdown_connections 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.607+0000 7f81d699d640 1 -- 172.21.15.17:0/4206163971 wait complete. 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.608+0000 7f81d699d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.608+0000 7f81d699d640 10 monclient: monmap: 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:45.620 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.608+0000 7f81d699d640 1 Processor -- start 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.608+0000 7f81d699d640 1 -- start start 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.608+0000 7f81d699d640 10 monclient: init 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-a con 0x7f81d006a550 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d0070a40 0x7f81d013d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-b con 0x7f81d0070a40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81d009b8a0 0x7f81d0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): picked mon.noname-c con 0x7f81d009b8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d01229a0 con 0x7f81d006a550 2024-09-15T02:11:45.621 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d01223e0 con 0x7f81d0070a40 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d011ff50 con 0x7f81d009b8a0 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d699d640 10 monclient(hunting): authenticate will time out at 1965.605131s 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:59986/0 (socket says 172.21.15.17:59986) 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 -- 172.21.15.17:0/3200124531 learned_addr learned my addr 172.21.15.17:0/3200124531 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d006a550 auth_method 0 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81d009b8a0 0x7f81d0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): get_auth_request con 0x7f81d009b8a0 auth_method 0 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d0070a40 0x7f81d013d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d0070a40 auth_method 0 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.623 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81d4f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient(hunting): handle_auth_done global_id 234261 payload 824 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 -- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81d009b8a0 msgr2=0x7f81d0140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81d009b8a0 0x7f81d0140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 -- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d0070a40 msgr2=0x7f81d013d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d0070a40 0x7f81d013d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 monclient: found mon.noname-a 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.609+0000 7f81cffff640 1 -- 172.21.15.17:0/3200124531 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c0009390 con 0x7f81d006a550 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81cffff640 1 -- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 msgr2=0x7f81d013d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81cffff640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009880 tx=0x7f81c0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0031c70 con 0x7f81d006a550 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81d699d640 5 monclient: authenticate success, global_id 234261 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c0031dd0 con 0x7f81d006a550 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c003d670 con 0x7f81d006a550 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 10 monclient: _renew_subs 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d0122100 con 0x7f81d006a550 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.626 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.611+0000 7f81ce7fc640 4 set_mon_vals no callback set 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 10 monclient: _renew_subs 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.612+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d013d740 con 0x7f81d006a550 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81c0039040 con 0x7f81d006a550 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81d699d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81d699d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81d699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d00724a0 con 0x7f81d006a550 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81b7fff640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 0x7f81bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f81c006c6d0 con 0x7f81d006a550 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81cf7fe640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 0x7f81bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.613+0000 7f81cf7fe640 10 monclient: get_auth_request con 0x7f81bc038740 auth_method 0 2024-09-15T02:11:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.616+0000 7f81cf7fe640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 0x7f81bc03abf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f81d009d350 tx=0x7f81c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.617+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f81c002f2a0 con 0x7f81d006a550 2024-09-15T02:11:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.617+0000 7f81b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.629 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.617+0000 7f81b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:45.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.748+0000 7fd05a754640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.750 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.748+0000 7fd05a754640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.748+0000 7fd05a754640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.748+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd05404bc80 con 0x7fd05406a9b0 2024-09-15T02:11:45.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.749+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (secure 0 0 0) 0x7fd044017090 con 0x7fd05406a9b0 2024-09-15T02:11:45.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.749+0000 7fd03bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.751 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.749+0000 7fd03bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:45.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81d699d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:45.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81d699d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:45.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81d699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:45.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81d00711e0 con 0x7f81d006a550 2024-09-15T02:11:45.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81b7fff640 1 -- 172.21.15.17:0/3200124531 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f81c0042610 con 0x7f81d006a550 2024-09-15T02:11:45.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:45.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.843+0000 7f81b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:45.845 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:45.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 msgr2=0x7f81bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 0x7f81bc03abf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f81d009d350 tx=0x7f81c4008040 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 10 monclient: shutdown 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 msgr2=0x7f81d013d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009880 tx=0x7f81c0032000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 shutdown_connections 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f81bc038740 0x7f81bc03abf0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f81d009b8a0 0x7f81d0140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d0070a40 0x7f81d013d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 --2- 172.21.15.17:0/3200124531 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f81d006a550 0x7f81d013d9b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 >> 172.21.15.17:0/3200124531 conn(0x7f81d0067ae0 msgr2=0x7f81d009fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 shutdown_connections 2024-09-15T02:11:45.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.845+0000 7f81d699d640 1 -- 172.21.15.17:0/3200124531 wait complete. 2024-09-15T02:11:45.857 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:11:45.857 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T02:11:45.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:11:45.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:11:45.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.906+0000 7fd05a754640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.906+0000 7fd05a754640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.906+0000 7fd05a754640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.906+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd05404b720 con 0x7fd05406a9b0 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.907+0000 7fd03bfff640 1 -- 172.21.15.67:0/2542891820 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (secure 0 0 0) 0x7fd04400e290 con 0x7fd05406a9b0 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.907+0000 7fd03bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-15T02:11:45.909 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.907+0000 7fd03bfff640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 msgr2=0x7fd04003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 0x7fd04003abf0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd054068a00 tx=0x7fd048032000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 10 monclient: shutdown 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 msgr2=0x7fd0541416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd04400c5e0 tx=0x7fd044004960 comp rx=0 tx=0).stop 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 shutdown_connections 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd040038740 0x7fd04003abf0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.911 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd054137000 0x7fd05411dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd054098540 0x7fd0541350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 --2- 172.21.15.67:0/2542891820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05406a9b0 0x7fd0541416f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 >> 172.21.15.67:0/2542891820 conn(0x7fd0540a5510 msgr2=0x7fd054135f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 shutdown_connections 2024-09-15T02:11:45.912 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.909+0000 7fd05a754640 1 -- 172.21.15.67:0/2542891820 wait complete. 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-15T02:11:45.919 INFO:tasks.workunit.client.0.smithi067.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-15T02:11:45.920 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph auth rm client.bad 2024-09-15T02:11:45.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.953+0000 7f0a9535d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.953+0000 7f0a9535d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.953+0000 7f0a9535d640 10 monclient: monmap: 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:45.954276+0000 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:45.954276+0000 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:45.955 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:45.956 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:45.956 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 Processor -- start 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 -- start start 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient: init 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 0x7f0a90132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a90067090 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a9009ce00 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a900679a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9009e1e0 con 0x7f0a900679a0 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9009e4c0 con 0x7f0a90067090 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9009cbf0 con 0x7f0a9009ce00 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a9535d640 10 monclient(hunting): authenticate will time out at 1965.950586s 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54966/0 (socket says 172.21.15.17:54966) 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/416873700 learned_addr learned my addr 172.21.15.17:0/416873700 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f0a9009ce00 auth_method 0 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 1 --2- 172.21.15.17:0/416873700 >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 0x7f0a90132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 10 monclient(hunting): get_auth_request con 0x7f0a90067090 auth_method 0 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_done global_id 214421 payload 274 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/416873700 >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 msgr2=0x7f0a90132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/416873700 >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 0x7f0a90132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/416873700 >> v1:172.21.15.67:6789/0 conn(0x7f0a900679a0 legacy=0x7f0a9012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:45.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 monclient: found mon.noname-b 2024-09-15T02:11:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/416873700 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90067e00 con 0x7f0a9009ce00 2024-09-15T02:11:45.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.954+0000 7f0a8effd640 1 --2- 172.21.15.17:0/416873700 >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 0x7f0a90132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.957+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/416873700 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 msgr2=0x7f0a90067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.957+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/416873700 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009910 tx=0x7f0a840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6382596277a4d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 1 -- 172.21.15.17:0/416873700 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8402aea0 con 0x7f0a9009ce00 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:45.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: dump: 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.967 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 1 -- 172.21.15.17:0/416873700 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84032440 con 0x7f0a9009ce00 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 5 monclient: authenticate success, global_id 214421 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 4 set_mon_vals no callback set 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 1 -- 172.21.15.17:0/416873700 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840327e0 con 0x7f0a9009ce00 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a8dffb640 10 monclient: dump: 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.968 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 10 monclient: shutdown 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 -- 172.21.15.17:0/416873700 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 msgr2=0x7f0a90067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 --2- 172.21.15.17:0/416873700 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009910 tx=0x7f0a840091d0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 -- 172.21.15.17:0/416873700 shutdown_connections 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 --2- 172.21.15.17:0/416873700 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a9009ce00 0x7f0a90067460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 --2- 172.21.15.17:0/416873700 >> v2:172.21.15.67:3300/0 conn(0x7f0a90067090 0x7f0a90132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 -- 172.21.15.17:0/416873700 >> 172.21.15.17:0/416873700 conn(0x7f0a900a1ee0 msgr2=0x7f0a900a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 -- 172.21.15.17:0/416873700 shutdown_connections 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.958+0000 7f0a9535d640 1 -- 172.21.15.17:0/416873700 wait complete. 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.959+0000 7f0a9535d640 10 monclient: build_initial_monmap 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.959+0000 7f0a9535d640 10 monclient: monmap: 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:45.970 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.959+0000 7f0a9535d640 1 Processor -- start 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.959+0000 7f0a9535d640 1 -- start start 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.959+0000 7f0a9535d640 10 monclient: init 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 0x7f0a9009b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a90067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a900679a0 0x7f0a90118a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a900679a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9009ce00 0x7f0a9012da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a9009ce00 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9009bfc0 con 0x7f0a9009ce00 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90119820 con 0x7f0a900679a0 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90118f50 con 0x7f0a90067090 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a9535d640 10 monclient(hunting): authenticate will time out at 1965.956241s 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 0x7f0a9009b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 0x7f0a9009b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54972/0 (socket says 172.21.15.17:54972) 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 -- 172.21.15.17:0/4062877964 learned_addr learned my addr 172.21.15.17:0/4062877964 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): get_auth_request con 0x7f0a90067090 auth_method 0 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9009ce00 0x7f0a9012da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a900679a0 0x7f0a90118a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): get_auth_request con 0x7f0a9009ce00 auth_method 0 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f0a900679a0 auth_method 0 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient(hunting): handle_auth_done global_id 214424 payload 824 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a900679a0 msgr2=0x7f0a90118a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a900679a0 0x7f0a90118a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9009ce00 msgr2=0x7f0a9012da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9009ce00 0x7f0a9012da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 monclient: found mon.noname-c 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.960+0000 7f0a8effd640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84009450 con 0x7f0a90067090 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.962+0000 7f0a8effd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 msgr2=0x7f0a9009b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.962+0000 7f0a8effd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 0x7f0a9009b520 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000d890 tx=0x7f0a80008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000de80 con 0x7f0a90067090 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: dump: 2024-09-15T02:11:45.975 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000bae0 con 0x7f0a90067090 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 5 monclient: authenticate success, global_id 214424 2024-09-15T02:11:45.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 monclient: _renew_subs 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9011b110 con 0x7f0a90067090 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000bea0 con 0x7f0a90067090 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a76ffd640 10 monclient: dump: 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.977 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a8d7fa640 4 set_mon_vals no callback set 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 monclient: _renew_subs 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9011ae30 con 0x7f0a90067090 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 0x7f0a90141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.963+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90063870 con 0x7f0a9012dfd0 2024-09-15T02:11:45.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 0x7f0a90141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a80014640 con 0x7f0a90067090 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient: get_auth_request con 0x7f0a9012dfd0 auth_method 0 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_done global_id 234267 payload 824 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a76ffd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a7c038790 0x7f0a7c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a9011ac80 con 0x7f0a9012dfd0 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f0a8004cc90 con 0x7f0a90067090 2024-09-15T02:11:45.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8f7fe640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a7c038790 0x7f0a7c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.964+0000 7f0a8f7fe640 10 monclient: get_auth_request con 0x7f0a7c038790 auth_method 0 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 msgr2=0x7f0a90141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 0x7f0a90141090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a84034000 tx=0x7f0a84032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8402aea0 con 0x7f0a9012dfd0 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.966+0000 7f0a76ffd640 10 monclient: dump: 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a8f7fe640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a7c038790 0x7f0a7c03ac40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007fd0 tx=0x7f0a880025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0a84031d30 con 0x7f0a9012dfd0 2024-09-15T02:11:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 msgr2=0x7f0a90141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 0x7f0a90141090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a84034000 tx=0x7f0a84032000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 msgr2=0x7f0a90141090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.967+0000 7f0a76ffd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a9012dfd0 0x7f0a90141090 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a84034000 tx=0x7f0a84032000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a9535d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a9535d640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 0x7f0a901419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a9535d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90141f10 con 0x7f0a901415d0 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 0x7f0a901419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient: get_auth_request con 0x7f0a901415d0 auth_method 0 2024-09-15T02:11:45.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 10 monclient(hunting): handle_auth_done global_id 234270 payload 824 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.973+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/4062877964 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a900724a0 con 0x7f0a901415d0 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a8e7fc640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 msgr2=0x7f0a901419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a8e7fc640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 0x7f0a901419d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0a90064560 tx=0x7f0a8403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8403cc00 con 0x7f0a901415d0 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.975+0000 7f0a76ffd640 10 monclient: dump: 2024-09-15T02:11:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f0a8403cd60 con 0x7f0a901415d0 2024-09-15T02:11:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 msgr2=0x7f0a901419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 0x7f0a901419d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0a90064560 tx=0x7f0a8403d000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 msgr2=0x7f0a901419d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a76ffd640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0a901415d0 0x7f0a901419d0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0a90064560 tx=0x7f0a8403d000 comp rx=0 tx=0).stop 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a7c038790 msgr2=0x7f0a7c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f0a7c038790 0x7f0a7c03ac40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007fd0 tx=0x7f0a880025b0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 10 monclient: shutdown 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 msgr2=0x7f0a9009b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 --2- 172.21.15.17:0/4062877964 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f0a90067090 0x7f0a9009b520 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000d890 tx=0x7f0a80008040 comp rx=0 tx=0).stop 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a8f7fe640 1 -- 172.21.15.17:0/4062877964 reap_dead start 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a8f7fe640 1 -- 172.21.15.17:0/4062877964 reap_dead start 2024-09-15T02:11:46.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 shutdown_connections 2024-09-15T02:11:46.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 >> 172.21.15.17:0/4062877964 conn(0x7f0a900a1ee0 msgr2=0x7f0a90097060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 shutdown_connections 2024-09-15T02:11:46.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:45.976+0000 7f0a9535d640 1 -- 172.21.15.17:0/4062877964 wait complete. 2024-09-15T02:11:46.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:46.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:11:46.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:11:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.990+0000 7fa74579d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:46.010 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.990+0000 7fa74579d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.990+0000 7fa74579d640 10 monclient: monmap: 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T02:11:45.991498+0000 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T02:11:45.991498+0000 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:46.011 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.990+0000 7fa74579d640 1 Processor -- start 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 -- start start 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient: init 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa740097260 0x7fa74009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-b con 0x7fa740097260 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa74009bf90 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fa74009c360 0x7fa74009a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-a con 0x7fa74009c360 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa740098950 con 0x7fa74009bf90 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa740097630 con 0x7fa74009c360 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa740097fb0 con 0x7fa740097260 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa74579d640 10 monclient(hunting): authenticate will time out at 1938.460580s 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73e7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa74009bf90 0x7fa740099900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41078/0 (socket says 172.21.15.67:41078) 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73e7fc640 1 -- 172.21.15.67:0/3951454215 learned_addr learned my addr 172.21.15.67:0/3951454215 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73f7fe640 1 --2- 172.21.15.67:0/3951454215 >> v2:172.21.15.17:3300/0 conn(0x7fa74009c360 0x7fa74009a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73effd640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa740097260 0x7fa74009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73effd640 10 monclient(hunting): get_auth_request con 0x7fa740097260 auth_method 0 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73f7fe640 10 monclient(hunting): get_auth_request con 0x7fa74009c360 auth_method 0 2024-09-15T02:11:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.991+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient(hunting): handle_auth_done global_id 234273 payload 274 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 1 -- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa740097260 msgr2=0x7fa74009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa740097260 0x7fa74009ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 1 -- 172.21.15.67:0/3951454215 >> v1:172.21.15.67:6789/0 conn(0x7fa74009bf90 legacy=0x7fa740099900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 1 monclient: found mon.noname-a 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.992+0000 7fa73f7fe640 1 -- 172.21.15.67:0/3951454215 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740098670 con 0x7fa74009c360 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73f7fe640 1 -- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c360 msgr2=0x7fa74009a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73f7fe640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c360 0x7fa74009a010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa740067520 tx=0x7fa738009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd3cdb2217f518a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 1 -- 172.21.15.67:0/3951454215 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738007760 con 0x7fa74009c360 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: dump: 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.015 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 1 -- 172.21.15.67:0/3951454215 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73802e050 con 0x7fa74009c360 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 1 -- 172.21.15.67:0/3951454215 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738007af0 con 0x7fa74009c360 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa74579d640 5 monclient: authenticate success, global_id 234273 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa74579d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa74579d640 4 set_mon_vals no callback set 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.995+0000 7fa73dffb640 10 monclient: dump: 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.017 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 10 monclient: shutdown 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 -- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c360 msgr2=0x7fa74009a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c360 0x7fa74009a010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa740067520 tx=0x7fa738009780 comp rx=0 tx=0).stop 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 -- 172.21.15.67:0/3951454215 shutdown_connections 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa74009c360 0x7fa74009a010 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 --2- 172.21.15.67:0/3951454215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa740097260 0x7fa74009ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 -- 172.21.15.67:0/3951454215 >> 172.21.15.67:0/3951454215 conn(0x7fa7400716e0 msgr2=0x7fa7400a1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 -- 172.21.15.67:0/3951454215 shutdown_connections 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 1 -- 172.21.15.67:0/3951454215 wait complete. 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.996+0000 7fa74579d640 10 monclient: monmap: 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 0 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 0.000000 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:created 0.000000 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 Processor -- start 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 -- start start 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient: init 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 0x7fa7401413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-a con 0x7fa740097260 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-b con 0x7fa74009bf90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 0x7fa74009ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): picked mon.noname-c con 0x7fa74009c360 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011f100 con 0x7fa740097260 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011ee20 con 0x7fa74009bf90 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011f3e0 con 0x7fa74009c360 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa74579d640 10 monclient(hunting): authenticate will time out at 1938.466609s 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49974/0 (socket says 172.21.15.67:49974) 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 1 -- 172.21.15.67:0/704630840 learned_addr learned my addr 172.21.15.67:0/704630840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): get_auth_request con 0x7fa74009bf90 auth_method 0 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73f7fe640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 0x7fa74009ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73effd640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 0x7fa7401413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73f7fe640 10 monclient(hunting): get_auth_request con 0x7fa74009c360 auth_method 0 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73effd640 10 monclient(hunting): get_auth_request con 0x7fa740097260 auth_method 0 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.997+0000 7fa73e7fc640 10 monclient(hunting): handle_auth_done global_id 244153 payload 824 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 -- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 msgr2=0x7fa74009ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 0x7fa74009ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 -- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 msgr2=0x7fa7401413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 0x7fa7401413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 monclient: found mon.noname-b 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73e7fc640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009a20 con 0x7fa74009bf90 2024-09-15T02:11:46.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73f7fe640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 0x7fa74009ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:45.998+0000 7fa73effd640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 0x7fa7401413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa73e7fc640 1 -- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 msgr2=0x7fa74012fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa73e7fc640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa73400c5e0 tx=0x7fa734004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734005cb0 con 0x7fa74009bf90 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:46.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: dump: 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734005e30 con 0x7fa74009bf90 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 5 monclient: authenticate success, global_id 244153 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 10 monclient: _renew_subs 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740142210 con 0x7fa74009bf90 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734007340 con 0x7fa74009bf90 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa726ffd640 10 monclient: dump: 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:epoch 1 2024-09-15T02:11:46.026 INFO:tasks.workunit.client.0.smithi067.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:election_strategy: 1 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 10 monclient: _renew_subs 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74011df30 con 0x7fa74009bf90 2024-09-15T02:11:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.000+0000 7fa73d7fa640 4 set_mon_vals no callback set 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa73401b400 con 0x7fa74009bf90 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa74579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa74579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa74579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400a1770 con 0x7fa74009bf90 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa726ffd640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 0x7fa72c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa73404c250 con 0x7fa74009bf90 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa73effd640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 0x7fa72c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.001+0000 7fa73effd640 10 monclient: get_auth_request con 0x7fa72c038740 auth_method 0 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.004+0000 7fa73effd640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 0x7fa72c03abf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa730009cf0 tx=0x7fa730008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.006+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa73401b6b0 con 0x7fa74009bf90 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.006+0000 7fa726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.006+0000 7fa726ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.105+0000 7f8db815d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.105+0000 7f8db815d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.105+0000 7f8db815d640 10 monclient: monmap: 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:46.106727+0000 2024-09-15T02:11:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:46.106727+0000 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:46.108 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.110 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.106+0000 7f8db815d640 1 Processor -- start 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.106+0000 7f8db815d640 1 -- start start 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.106+0000 7f8db815d640 10 monclient: init 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-b con 0x7f8db00a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8db00980a0 0x7f8db0098470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-a con 0x7f8db00980a0 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8db00989b0 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db00a01e0 con 0x7f8db00989b0 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db00a1580 con 0x7f8db00980a0 2024-09-15T02:11:46.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db00a5c80 con 0x7f8db00a58b0 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db815d640 10 monclient(hunting): authenticate will time out at 1966.103209s 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8db00980a0 0x7f8db0098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f8db00980a0 0x7f8db0098470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60016/0 (socket says 172.21.15.17:60016) 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54982/0 (socket says 172.21.15.17:54982) 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 -- 172.21.15.17:0/3961350247 learned_addr learned my addr 172.21.15.17:0/3961350247 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): get_auth_request con 0x7f8db00980a0 auth_method 0 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.112 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request con 0x7f8db00a58b0 auth_method 0 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient(hunting): handle_auth_done global_id 234279 payload 274 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 -- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 msgr2=0x7f8db0132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 -- 172.21.15.17:0/3961350247 >> v1:172.21.15.67:6789/0 conn(0x7f8db00989b0 legacy=0x7f8db0068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.113 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 monclient: found mon.noname-a 2024-09-15T02:11:46.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:46.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db56f8640 1 -- 172.21.15.17:0/3961350247 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db01326c0 con 0x7f8db00980a0 2024-09-15T02:11:46.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.107+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:46.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db56f8640 1 -- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00980a0 msgr2=0x7f8db0098470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db56f8640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00980a0 0x7f8db0098470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009910 tx=0x7f8da40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7985ecaaabcc07 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 1 -- 172.21.15.17:0/3961350247 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4032440 con 0x7f8db00980a0 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: dump: 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.121 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db815d640 5 monclient: authenticate success, global_id 234279 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db815d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 1 -- 172.21.15.17:0/3961350247 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da40325a0 con 0x7f8db00980a0 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db815d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 1 -- 172.21.15.17:0/3961350247 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4032940 con 0x7f8db00980a0 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db815d640 4 set_mon_vals no callback set 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.110+0000 7f8db4ef7640 10 monclient: dump: 2024-09-15T02:11:46.122 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 10 monclient: shutdown 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 -- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00980a0 msgr2=0x7f8db0098470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00980a0 0x7f8db0098470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009910 tx=0x7f8da40091d0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 -- 172.21.15.17:0/3961350247 shutdown_connections 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00980a0 0x7f8db0098470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 --2- 172.21.15.17:0/3961350247 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00a58b0 0x7f8db0132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 -- 172.21.15.17:0/3961350247 >> 172.21.15.17:0/3961350247 conn(0x7f8db0070ed0 msgr2=0x7f8db00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 -- 172.21.15.17:0/3961350247 shutdown_connections 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 1 -- 172.21.15.17:0/3961350247 wait complete. 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.111+0000 7f8db815d640 10 monclient: monmap: 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.124 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 Processor -- start 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 -- start start 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient: init 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:11:46.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8db00980a0 0x7f8db009afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-b con 0x7f8db00980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-c con 0x7f8db00989b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00a58b0 0x7f8db01412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): picked mon.noname-a con 0x7f8db00a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db0098dc0 con 0x7f8db00a58b0 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db009eaa0 con 0x7f8db00980a0 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db011dfc0 con 0x7f8db00989b0 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db815d640 10 monclient(hunting): authenticate will time out at 1966.108540s 2024-09-15T02:11:46.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00a58b0 0x7f8db01412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:54996/0 (socket says 172.21.15.17:54996) 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 -- 172.21.15.17:0/3627933479 learned_addr learned my addr 172.21.15.17:0/3627933479 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): get_auth_request con 0x7f8db00989b0 auth_method 0 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): get_auth_request con 0x7f8db00a58b0 auth_method 0 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8db00980a0 0x7f8db009afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request con 0x7f8db00980a0 auth_method 0 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient(hunting): handle_auth_done global_id 214436 payload 824 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8db00980a0 msgr2=0x7f8db009afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8db00980a0 0x7f8db009afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00a58b0 msgr2=0x7f8db01412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00a58b0 0x7f8db01412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 monclient: found mon.noname-c 2024-09-15T02:11:46.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db56f8640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0009390 con 0x7f8db00989b0 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.112+0000 7f8db66fa640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8db00a58b0 0x7f8db01412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db56f8640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 msgr2=0x7f8db0130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db56f8640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009970 tx=0x7f8da4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4031a10 con 0x7f8db00989b0 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: dump: 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da4031b90 con 0x7f8db00989b0 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 5 monclient: authenticate success, global_id 214436 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 10 monclient: _renew_subs 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da4009450 con 0x7f8db00989b0 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da403d440 con 0x7f8db00989b0 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8dadffb640 10 monclient: dump: 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.132 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 10 monclient: _renew_subs 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db011c7f0 con 0x7f8db00989b0 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.115+0000 7f8daffff640 4 set_mon_vals no callback set 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db815d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db815d640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 0x7f8db0141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db0063b20 con 0x7f8db0131100 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8da4039040 con 0x7f8db00989b0 2024-09-15T02:11:46.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 0x7f8db0141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient: get_auth_request con 0x7f8db0131100 auth_method 0 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8dadffb640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d9c038790 0x7f8d9c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8da406c3d0 con 0x7f8db00989b0 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_done global_id 214439 payload 824 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.116+0000 7f8db5ef9640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8db009bb90 con 0x7f8db0131100 2024-09-15T02:11:46.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.117+0000 7f8db66fa640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d9c038790 0x7f8d9c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.117+0000 7f8db66fa640 10 monclient: get_auth_request con 0x7f8d9c038790 auth_method 0 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8db5ef9640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 msgr2=0x7f8db0141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 0x7f8db0141830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da000fe90 con 0x7f8db0131100 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 10 monclient: dump: 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.135 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8da0014650 con 0x7f8db0131100 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 msgr2=0x7f8db0141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 0x7f8db0141830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da00090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8db66fa640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d9c038790 0x7f8d9c03ac40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8da8007fd0 tx=0x7f8da80025b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 msgr2=0x7f8db0141830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:46.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.119+0000 7f8dadffb640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0131100 0x7f8db0141830 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da00090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db815d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:46.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db815d640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 0x7f8db009c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db815d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db01314d0 con 0x7f8db0141d70 2024-09-15T02:11:46.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 0x7f8db009c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient: get_auth_request con 0x7f8db0141d70 auth_method 0 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 10 monclient(hunting): handle_auth_done global_id 214442 payload 824 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.125+0000 7f8db5ef9640 1 -- 172.21.15.17:0/3627933479 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8db0142170 con 0x7f8db0141d70 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db5ef9640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 msgr2=0x7f8db009c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db5ef9640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 0x7f8db009c550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0025040 con 0x7f8db0141d70 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 10 monclient: dump: 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.140 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f8da001c3e0 con 0x7f8db0141d70 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 msgr2=0x7f8db009c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 0x7f8db009c550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da001d000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 msgr2=0x7f8db009c550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:46.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8dadffb640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db0141d70 0x7f8db009c550 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8da0011bf0 tx=0x7f8da001d000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:46.142 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:46.143 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:46.144 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:46.145 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.146 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:46.148 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:46.151 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.152 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.153 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:46.154 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:46.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d9c038790 msgr2=0x7f8d9c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.155 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f8d9c038790 0x7f8d9c03ac40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8da8007fd0 tx=0x7f8da80025b0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 10 monclient: shutdown 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 msgr2=0x7f8db0130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009970 tx=0x7f8da4032000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db66fa640 1 -- 172.21.15.17:0/3627933479 reap_dead start 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 shutdown_connections 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 --2- 172.21.15.17:0/3627933479 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f8db00989b0 0x7f8db0130bc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 >> 172.21.15.17:0/3627933479 conn(0x7f8db0070ed0 msgr2=0x7f8db00a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 shutdown_connections 2024-09-15T02:11:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.128+0000 7f8db815d640 1 -- 172.21.15.17:0/3627933479 wait complete. 2024-09-15T02:11:46.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:46.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:11:46.157 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:11:46.233 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.232+0000 7fa74579d640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:11:46.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.232+0000 7fa74579d640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:11:46.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.232+0000 7fa74579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:11:46.234 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:46.232+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fa74011f5a0 con 0x7fa74009bf90 2024-09-15T02:11:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.253+0000 7ff29fb00640 10 monclient: get_monmap_and_config 2024-09-15T02:11:46.254 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.253+0000 7ff29fb00640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.253+0000 7ff29fb00640 10 monclient: monmap: 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:46.254111+0000 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:46.254111+0000 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:46.255 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.253+0000 7ff29fb00640 1 Processor -- start 2024-09-15T02:11:46.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 -- start start 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient: init 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff298131b20 0x7ff29809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-c con 0x7ff298131b20 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2980a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-b con 0x7ff29809f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980a0040 con 0x7ff2980a0280 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29809bff0 con 0x7ff298131b20 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.258 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980a14a0 con 0x7ff29809f1e0 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29fb00640 10 monclient(hunting): authenticate will time out at 1966.250568s 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55014/0 (socket says 172.21.15.17:55014) 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 -- 172.21.15.17:0/2744877537 learned_addr learned my addr 172.21.15.17:0/2744877537 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): get_auth_request con 0x7ff29809f1e0 auth_method 0 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 1 --2- 172.21.15.17:0/2744877537 >> v2:172.21.15.67:3300/0 conn(0x7ff298131b20 0x7ff29809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): get_auth_request con 0x7ff298131b20 auth_method 0 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient(hunting): handle_auth_done global_id 214445 payload 274 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 -- 172.21.15.17:0/2744877537 >> v2:172.21.15.67:3300/0 conn(0x7ff298131b20 msgr2=0x7ff29809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 --2- 172.21.15.17:0/2744877537 >> v2:172.21.15.67:3300/0 conn(0x7ff298131b20 0x7ff29809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 -- 172.21.15.17:0/2744877537 >> v1:172.21.15.67:6789/0 conn(0x7ff2980a0280 legacy=0x7ff298131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 monclient: found mon.noname-b 2024-09-15T02:11:46.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.254+0000 7ff29e09d640 1 -- 172.21.15.17:0/2744877537 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298132600 con 0x7ff29809f1e0 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29e09d640 1 -- 172.21.15.17:0/2744877537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 msgr2=0x7ff298138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29e09d640 1 --2- 172.21.15.17:0/2744877537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff290009910 tx=0x7ff2900091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba926eb8dd22dc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 1 -- 172.21.15.17:0/2744877537 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29002aea0 con 0x7ff29809f1e0 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: dump: 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.267 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 1 -- 172.21.15.17:0/2744877537 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff290032440 con 0x7ff29809f1e0 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 1 -- 172.21.15.17:0/2744877537 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2900327e0 con 0x7ff29809f1e0 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29fb00640 5 monclient: authenticate success, global_id 214445 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29fb00640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29fb00640 4 set_mon_vals no callback set 2024-09-15T02:11:46.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.257+0000 7ff29c89a640 10 monclient: dump: 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 10 monclient: shutdown 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 -- 172.21.15.17:0/2744877537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 msgr2=0x7ff298138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 --2- 172.21.15.17:0/2744877537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff290009910 tx=0x7ff2900091d0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 -- 172.21.15.17:0/2744877537 shutdown_connections 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 --2- 172.21.15.17:0/2744877537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff29809f1e0 0x7ff298138e60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 --2- 172.21.15.17:0/2744877537 >> v2:172.21.15.67:3300/0 conn(0x7ff298131b20 0x7ff29809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 -- 172.21.15.17:0/2744877537 >> 172.21.15.17:0/2744877537 conn(0x7ff298071270 msgr2=0x7ff2980a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 -- 172.21.15.17:0/2744877537 shutdown_connections 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.258+0000 7ff29fb00640 1 -- 172.21.15.17:0/2744877537 wait complete. 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient: monmap: 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:46.270 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 1 Processor -- start 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 1 -- start start 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient: init 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29809f1e0 0x7ff29812f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-b con 0x7ff29809f1e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 0x7ff298096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-c con 0x7ff2980a0280 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.259+0000 7ff29fb00640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 0x7ff298096f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): picked mon.noname-a con 0x7ff298131b20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298099850 con 0x7ff298131b20 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298099570 con 0x7ff29809f1e0 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298099b30 con 0x7ff2980a0280 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29fb00640 10 monclient(hunting): authenticate will time out at 1966.256030s 2024-09-15T02:11:46.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 0x7ff298096f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 0x7ff298096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 0x7ff298096f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:60032/0 (socket says 172.21.15.17:60032) 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 learned_addr learned my addr 172.21.15.17:0/306650140 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): get_auth_request con 0x7ff298131b20 auth_method 0 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 0x7ff298096a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55030/0 (socket says 172.21.15.17:55030) 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): get_auth_request con 0x7ff2980a0280 auth_method 0 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29809f1e0 0x7ff29812f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): get_auth_request con 0x7ff29809f1e0 auth_method 0 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient(hunting): handle_auth_done global_id 234285 payload 824 2024-09-15T02:11:46.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 msgr2=0x7ff298096a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 0x7ff298096a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29809f1e0 msgr2=0x7ff29812f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29809f1e0 0x7ff29812f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 monclient: found mon.noname-a 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290009450 con 0x7ff298131b20 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.260+0000 7ff29d09b640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff2980a0280 0x7ff298096a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 msgr2=0x7ff298096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29e09d640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 0x7ff298096f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff290035040 tx=0x7ff290032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29002aea0 con 0x7ff298131b20 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: dump: 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.277 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff290031d40 con 0x7ff298131b20 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29fb00640 5 monclient: authenticate success, global_id 234285 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29003d6b0 con 0x7ff298131b20 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29fb00640 10 monclient: _renew_subs 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29fb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29809a230 con 0x7ff298131b20 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff295ffb640 10 monclient: dump: 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.262+0000 7ff297fff640 4 set_mon_vals no callback set 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff29fb00640 10 monclient: _renew_subs 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff29fb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2980987b0 con 0x7ff298131b20 2024-09-15T02:11:46.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff290039040 con 0x7ff298131b20 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff29fb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.263+0000 7ff29fb00640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 0x7ff29812bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29812f610 con 0x7ff298097490 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff295ffb640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff284038740 0x7ff28403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff29006c2a0 con 0x7ff298131b20 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 0x7ff29812bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient: get_auth_request con 0x7ff298097490 auth_method 0 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d09b640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff284038740 0x7ff28403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d09b640 10 monclient: get_auth_request con 0x7ff284038740 auth_method 0 2024-09-15T02:11:46.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 10 monclient(hunting): handle_auth_done global_id 244165 payload 824 2024-09-15T02:11:46.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.264+0000 7ff29d89c640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff29809b920 con 0x7ff298097490 2024-09-15T02:11:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.266+0000 7ff29d89c640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 msgr2=0x7ff29812bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.266+0000 7ff29d89c640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 0x7ff29812bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff28800f200 tx=0x7ff28800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.266+0000 7ff29d09b640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff284038740 0x7ff28403abf0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff28c007fd0 tx=0x7ff28c007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.266+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28800e040 con 0x7ff298097490 2024-09-15T02:11:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 10 monclient: dump: 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.284 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff28800f040 con 0x7ff298097490 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 msgr2=0x7ff29812bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 0x7ff29812bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff28800f200 tx=0x7ff28800b040 comp rx=0 tx=0).stop 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 msgr2=0x7ff29812bc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.267+0000 7ff295ffb640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff298097490 0x7ff29812bc40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29fb00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29fb00640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 0x7ff29812c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29fb00640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298063a40 con 0x7ff29812c180 2024-09-15T02:11:46.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 0x7ff29812c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient: get_auth_request con 0x7ff29812c180 auth_method 0 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 10 monclient(hunting): handle_auth_done global_id 244168 payload 824 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.273+0000 7ff29d89c640 1 -- 172.21.15.17:0/306650140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff298097860 con 0x7ff29812c180 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff29d89c640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 msgr2=0x7ff29812c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff29d89c640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 0x7ff29812c580 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff28801b000 tx=0x7ff28801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288024070 con 0x7ff29812c180 2024-09-15T02:11:46.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 10 monclient: dump: 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7ff2880148b0 con 0x7ff29812c180 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 msgr2=0x7ff29812c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 0x7ff29812c580 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff28801b000 tx=0x7ff28801c000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 msgr2=0x7ff29812c580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff295ffb640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29812c180 0x7ff29812c580 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff28801b000 tx=0x7ff28801c000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:46.288 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 204, 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.289 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.290 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:11:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:11:46.292 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:11:46.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:11:46.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:11:46.299 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.300 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:11:46.301 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:11:46.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.276+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff284038740 msgr2=0x7ff28403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff284038740 0x7ff28403abf0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff28c007fd0 tx=0x7ff28c007e10 comp rx=0 tx=0).stop 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 10 monclient: shutdown 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 msgr2=0x7ff298096f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 --2- 172.21.15.17:0/306650140 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff298131b20 0x7ff298096f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff290035040 tx=0x7ff290032000 comp rx=0 tx=0).stop 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 reap_dead start 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29e09d640 1 -- 172.21.15.17:0/306650140 reap_dead start 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 shutdown_connections 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 >> 172.21.15.17:0/306650140 conn(0x7ff298071270 msgr2=0x7ff29812a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 shutdown_connections 2024-09-15T02:11:46.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.277+0000 7ff29fb00640 1 -- 172.21.15.17:0/306650140 wait complete. 2024-09-15T02:11:46.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:11:46.304 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T02:11:46.304 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T02:11:46.304 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T02:11:46.304 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:11:46.304 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:11:46.305 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:11:46.316 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:11:46.316 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T02:11:46.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.450+0000 7f58c151d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.450+0000 7f58c151d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.450+0000 7f58c151d640 10 monclient: monmap: 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:46.451582+0000 2024-09-15T02:11:46.452 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:46.451582+0000 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:46.453 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 Processor -- start 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 -- start start 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient: init 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58bc09a590 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 0x7f58bc12edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc09a960 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0985d0 con 0x7f58bc09a590 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc09d200 con 0x7f58bc09a960 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc097c30 con 0x7f58bc097320 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.457 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58c151d640 10 monclient(hunting): authenticate will time out at 1966.447990s 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58ba7fc640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f58bc09a590 legacy=0x7f58bc12e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.451+0000 7f58bb7fe640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 msgr2=0x7f58bc12edb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58bb7fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 0x7f58bc12edb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55034/0 (socket says 172.21.15.17:55034) 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 -- 172.21.15.17:0/4136361481 learned_addr learned my addr 172.21.15.17:0/4136361481 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): get_auth_request con 0x7f58bc097320 auth_method 0 2024-09-15T02:11:46.458 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient(hunting): handle_auth_done global_id 214451 payload 274 2024-09-15T02:11:46.459 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 -- 172.21.15.17:0/4136361481 >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 msgr2=0x7f58bc12edb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 --2- 172.21.15.17:0/4136361481 >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 0x7f58bc12edb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 -- 172.21.15.17:0/4136361481 >> v1:172.21.15.17:6789/0 conn(0x7f58bc09a590 legacy=0x7f58bc12e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 monclient: found mon.noname-b 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.460 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.452+0000 7f58baffd640 1 -- 172.21.15.17:0/4136361481 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0992f0 con 0x7f58bc097320 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58baffd640 1 -- 172.21.15.17:0/4136361481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 msgr2=0x7f58bc0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58baffd640 1 --2- 172.21.15.17:0/4136361481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009910 tx=0x7f58ac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8789b94a27a2c3f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 1 -- 172.21.15.17:0/4136361481 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac032440 con 0x7f58bc097320 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: dump: 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.468 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 1 -- 172.21.15.17:0/4136361481 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac0325a0 con 0x7f58bc097320 2024-09-15T02:11:46.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 1 -- 172.21.15.17:0/4136361481 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac032940 con 0x7f58bc097320 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58b9ffb640 10 monclient: dump: 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.470 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 5 monclient: authenticate success, global_id 214451 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 4 set_mon_vals no callback set 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 10 monclient: shutdown 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:46.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 -- 172.21.15.17:0/4136361481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 msgr2=0x7f58bc0976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 --2- 172.21.15.17:0/4136361481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009910 tx=0x7f58ac0091d0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 -- 172.21.15.17:0/4136361481 shutdown_connections 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 --2- 172.21.15.17:0/4136361481 >> v2:172.21.15.17:3300/0 conn(0x7f58bc09a960 0x7f58bc12edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 --2- 172.21.15.17:0/4136361481 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc0976f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.455+0000 7f58c151d640 1 -- 172.21.15.17:0/4136361481 >> 172.21.15.17:0/4136361481 conn(0x7f58bc0a1000 msgr2=0x7f58bc0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 1 -- 172.21.15.17:0/4136361481 shutdown_connections 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 1 -- 172.21.15.17:0/4136361481 wait complete. 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 10 monclient: build_initial_monmap 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 10 monclient: monmap: 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 1 Processor -- start 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 1 -- start start 2024-09-15T02:11:46.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.456+0000 7f58c151d640 10 monclient: init 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc097320 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 0x7f58bc12e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc09a590 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 0x7f58bc1174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc09a960 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc09ad50 con 0x7f58bc09a590 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc118c80 con 0x7f58bc09a960 2024-09-15T02:11:46.474 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc119170 con 0x7f58bc097320 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58c151d640 10 monclient(hunting): authenticate will time out at 1966.453432s 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58ba7fc640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 msgr2=0x7f58bc12e610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58ba7fc640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 0x7f58bc12e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55048/0 (socket says 172.21.15.17:55048) 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 -- 172.21.15.17:0/642771595 learned_addr learned my addr 172.21.15.17:0/642771595 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): get_auth_request con 0x7f58bc097320 auth_method 0 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58bb7fe640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 0x7f58bc1174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient(hunting): handle_auth_done global_id 214454 payload 824 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 -- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 msgr2=0x7f58bc1174c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 0x7f58bc1174c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 -- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 msgr2=0x7f58bc12e610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 0x7f58bc12e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 monclient: found mon.noname-c 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:46.476 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58baffd640 1 -- 172.21.15.17:0/642771595 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac009450 con 0x7f58bc097320 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58bb7fe640 10 monclient: get_auth_request con 0x7f58bc09a960 auth_method 0 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.457+0000 7f58bb7fe640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 0x7f58bc1174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58baffd640 1 -- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 msgr2=0x7f58bc141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58baffd640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009970 tx=0x7f58ac032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac031af0 con 0x7f58bc097320 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: dump: 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac031c50 con 0x7f58bc097320 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 5 monclient: authenticate success, global_id 214454 2024-09-15T02:11:46.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 10 monclient: _renew_subs 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc06eba0 con 0x7f58bc097320 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac03c410 con 0x7f58bc097320 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58a2ffd640 10 monclient: dump: 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:46.479 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 10 monclient: _renew_subs 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc142360 con 0x7f58bc097320 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.460+0000 7f58b97fa640 4 set_mon_vals no callback set 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58c151d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58c151d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58c151d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0724a0 con 0x7f58bc097320 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f58ac039070 con 0x7f58bc097320 2024-09-15T02:11:46.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58a2ffd640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 0x7f58a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f58ac06cb40 con 0x7f58bc097320 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58ba7fc640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 0x7f58a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.461+0000 7f58ba7fc640 10 monclient: get_auth_request con 0x7f58a80386f0 auth_method 0 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.464+0000 7f58ba7fc640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 0x7f58a803aba0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f58bc119ab0 tx=0x7f58b0006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.465+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f58ac03a900 con 0x7f58bc097320 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.465+0000 7f58a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:46.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.465+0000 7f58a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:46.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.691+0000 7f58c151d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:46.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.691+0000 7f58c151d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:46.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.691+0000 7f58c151d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:46.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.691+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f58bc11a390 con 0x7f58bc097320 2024-09-15T02:11:46.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:46.691+0000 7f58baffd640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009970 tx=0x7f58ac032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f58ac037020 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 2024-09-15T02:11:47.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.519+0000 7f58a2ffd640 1 -- 172.21.15.17:0/642771595 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7f58ac037020 con 0x7f58bc097320 2024-09-15T02:11:47.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.519+0000 7f58a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:47.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.519+0000 7f58a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:47.521 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 msgr2=0x7f58a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 0x7f58a803aba0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f58bc119ab0 tx=0x7f58b0006d80 comp rx=0 tx=0).stop 2024-09-15T02:11:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 10 monclient: shutdown 2024-09-15T02:11:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 msgr2=0x7f58bc141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009970 tx=0x7f58ac032000 comp rx=0 tx=0).stop 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 shutdown_connections 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f58a80386f0 0x7f58a803aba0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f58bc09a590 0x7f58bc12e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc09a960 0x7f58bc1174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 --2- 172.21.15.17:0/642771595 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f58bc097320 0x7f58bc141e20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 >> 172.21.15.17:0/642771595 conn(0x7f58bc0a1000 msgr2=0x7f58bc06e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 shutdown_connections 2024-09-15T02:11:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:47.522+0000 7f58c151d640 1 -- 172.21.15.17:0/642771595 wait complete. 2024-09-15T02:11:48.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:48.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:48.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:48.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:49.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:48.998+0000 7fa725ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:49.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:49.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:49.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:49.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:50.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:50.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.645+0000 7fa5f98a8640 10 monclient: get_monmap_and_config 2024-09-15T02:11:50.646 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.645+0000 7fa5f98a8640 10 monclient: build_initial_monmap 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.645+0000 7fa5f98a8640 10 monclient: monmap: 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:50.646183+0000 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:50.646183+0000 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:50.648 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:50.648 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:50.648 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.645+0000 7fa5f98a8640 1 Processor -- start 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 1 -- start start 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient: init 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T02:11:50.650 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5f40a1c20 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:50.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5f409b240 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:50.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f412ea70 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f409b660 con 0x7fa5f40a1c20 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f406ea80 con 0x7fa5f409b240 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f406e8f0 con 0x7fa5f412ea70 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f98a8640 10 monclient(hunting): authenticate will time out at 1970.642517s 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f2ffd640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fa5f40a1c20 legacy=0x7fa5f409ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55058/0 (socket says 172.21.15.17:55058) 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 -- 172.21.15.17:0/1017815166 learned_addr learned my addr 172.21.15.17:0/1017815166 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): get_auth_request con 0x7fa5f412ea70 auth_method 0 2024-09-15T02:11:50.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient(hunting): handle_auth_done global_id 214457 payload 274 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 -- 172.21.15.17:0/1017815166 >> v1:172.21.15.67:6789/0 conn(0x7fa5f409b240 legacy=0x7fa5f4136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 -- 172.21.15.17:0/1017815166 >> v1:172.21.15.17:6789/0 conn(0x7fa5f40a1c20 legacy=0x7fa5f409ab30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 monclient: found mon.noname-b 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.646+0000 7fa5f37fe640 1 -- 172.21.15.17:0/1017815166 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f412eed0 con 0x7fa5f412ea70 2024-09-15T02:11:50.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f37fe640 1 -- 172.21.15.17:0/1017815166 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 msgr2=0x7fa5f413a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f37fe640 1 --2- 172.21.15.17:0/1017815166 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009f50 tx=0x7fa5ec009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecf2c4d8b923dff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f1ffb640 1 -- 172.21.15.17:0/1017815166 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec007610 con 0x7fa5f412ea70 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.649+0000 7fa5f1ffb640 10 monclient: dump: 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.659 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 5 monclient: authenticate success, global_id 214457 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 1 -- 172.21.15.17:0/1017815166 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec02d050 con 0x7fa5f412ea70 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 4 set_mon_vals no callback set 2024-09-15T02:11:50.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 1 -- 172.21.15.17:0/1017815166 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0079a0 con 0x7fa5f412ea70 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f1ffb640 10 monclient: dump: 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 10 monclient: shutdown 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1017815166 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 msgr2=0x7fa5f413a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1017815166 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009f50 tx=0x7fa5ec009840 comp rx=0 tx=0).stop 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1017815166 shutdown_connections 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1017815166 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f412ea70 0x7fa5f413a3d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1017815166 >> 172.21.15.17:0/1017815166 conn(0x7fa5f4067920 msgr2=0x7fa5f40a11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1017815166 shutdown_connections 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.650+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1017815166 wait complete. 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.651+0000 7fa5f98a8640 10 monclient: build_initial_monmap 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.651+0000 7fa5f98a8640 10 monclient: monmap: 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:50.662 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.651+0000 7fa5f98a8640 1 Processor -- start 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.651+0000 7fa5f98a8640 1 -- start start 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.651+0000 7fa5f98a8640 10 monclient: init 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:11:50.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 0x7fa5f4131740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f409b240 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f40a1c20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 0x7fa5f412b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f412ea70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4132130 con 0x7fa5f409b240 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4127a90 con 0x7fa5f412ea70 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f41287b0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:50.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f2ffd640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 msgr2=0x7fa5f4131740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f98a8640 10 monclient(hunting): authenticate will time out at 1970.648177s 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f2ffd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 0x7fa5f4131740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55066/0 (socket says 172.21.15.17:55066) 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 -- 172.21.15.17:0/1995464544 learned_addr learned my addr 172.21.15.17:0/1995464544 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): get_auth_request con 0x7fa5f40a1c20 auth_method 0 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f37fe640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 0x7fa5f412b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:50.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient(hunting): handle_auth_done global_id 214460 payload 824 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 -- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 msgr2=0x7fa5f412b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 0x7fa5f412b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 -- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 msgr2=0x7fa5f4131740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 0x7fa5f4131740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient: _finish_hunting 0 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 monclient: found mon.noname-c 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f27fc640 1 -- 172.21.15.17:0/1995464544 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009ae0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f37fe640 10 monclient: get_auth_request con 0x7fa5f412ea70 auth_method 0 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.652+0000 7fa5f37fe640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 0x7fa5f412b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f27fc640 1 -- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 msgr2=0x7fa5f4126730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:50.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f27fc640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800b4f0 tx=0x7fa5e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800f720 con 0x7fa5f40a1c20 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: dump: 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:50.667 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: _finish_auth 0 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e800f880 con 0x7fa5f40a1c20 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f98a8640 5 monclient: authenticate success, global_id 214460 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f98a8640 10 monclient: _renew_subs 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f98a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f4128a80 con 0x7fa5f40a1c20 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800fc20 con 0x7fa5f40a1c20 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5daffd640 10 monclient: dump: 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:50.668 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.654+0000 7fa5f0ff9640 4 set_mon_vals no callback set 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.655+0000 7fa5f98a8640 10 monclient: _renew_subs 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.655+0000 7fa5f98a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.655+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f412ba60 con 0x7fa5f40a1c20 2024-09-15T02:11:50.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa5e8011610 con 0x7fa5f40a1c20 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f98a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f98a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f98a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f41288f0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5daffd640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 0x7fa5e003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa5e804c1c0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f2ffd640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 0x7fa5e003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.656+0000 7fa5f2ffd640 10 monclient: get_auth_request con 0x7fa5e00386f0 auth_method 0 2024-09-15T02:11:50.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.659+0000 7fa5f2ffd640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 0x7fa5e003aba0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006fd0 tx=0x7fa5e4006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:50.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.660+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa5e800fe00 con 0x7fa5f40a1c20 2024-09-15T02:11:50.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.660+0000 7fa5daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:50.671 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.660+0000 7fa5daffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:50.894 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5f98a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5f98a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5f98a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa5f40724a0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5daffd640 1 -- 172.21.15.17:0/1995464544 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fa5e80172f0 con 0x7fa5f40a1c20 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.893+0000 7fa5daffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:50.895 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:50.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 msgr2=0x7fa5e003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:50.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 0x7fa5e003aba0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006fd0 tx=0x7fa5e4006d30 comp rx=0 tx=0).stop 2024-09-15T02:11:50.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 10 monclient: shutdown 2024-09-15T02:11:50.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:50.897 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 msgr2=0x7fa5f4126730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800b4f0 tx=0x7fa5e80090f0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 shutdown_connections 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa5e00386f0 0x7fa5e003aba0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5f412ea70 0x7fa5f412b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa5f40a1c20 0x7fa5f4126730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 --2- 172.21.15.17:0/1995464544 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa5f409b240 0x7fa5f4131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 >> 172.21.15.17:0/1995464544 conn(0x7fa5f4067920 msgr2=0x7fa5f409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 shutdown_connections 2024-09-15T02:11:50.898 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:50.895+0000 7fa5f98a8640 1 -- 172.21.15.17:0/1995464544 wait complete. 2024-09-15T02:11:50.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:50.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:51.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:50.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:51.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:51.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:52.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:51.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:53.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:52.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:53.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:52.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:53.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:54.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:53.998+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:54.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:53.998+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:54.017 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.016+0000 7fb389a34640 10 monclient: get_monmap_and_config 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.016+0000 7fb389a34640 10 monclient: build_initial_monmap 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.016+0000 7fb389a34640 10 monclient: monmap: 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:54.017217+0000 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:54.017217+0000 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:54.018 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:54.019 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:54.019 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 Processor -- start 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 -- start start 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient: init 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb384068f20 addr v1:172.21.15.67:6789/0 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 0x7fb3840696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-a con 0x7fb3840692f0 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb384136fe0 0x7fb3841393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.021 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-c con 0x7fb384136fe0 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3840a1670 con 0x7fb384068f20 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb3827fc640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 msgr2=0x7fb3840696f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3841328d0 con 0x7fb3840692f0 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3840a0020 con 0x7fb384136fe0 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb389a34640 10 monclient(hunting): authenticate will time out at 1974.013805s 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.017+0000 7fb3827fc640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 0x7fb3840696f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb384136fe0 0x7fb3841393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb382ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb384068f20 0x7fb3841321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.17:39616/0 (socket says 172.21.15.17:39616) 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb382ffd640 1 -- 172.21.15.17:0/4238506547 learned_addr learned my addr 172.21.15.17:0/4238506547 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:11:54.022 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): get_auth_request con 0x7fb384136fe0 auth_method 0 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient(hunting): handle_auth_done global_id 244171 payload 274 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 -- 172.21.15.17:0/4238506547 >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 msgr2=0x7fb3840696f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 --2- 172.21.15.17:0/4238506547 >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 0x7fb3840696f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 -- 172.21.15.17:0/4238506547 >> v1:172.21.15.67:6789/0 conn(0x7fb384068f20 legacy=0x7fb3841321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient: _finish_hunting 0 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 monclient: found mon.noname-c 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:54.023 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:11:54.024 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.018+0000 7fb3837fe640 1 -- 172.21.15.17:0/4238506547 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3840a1390 con 0x7fb384136fe0 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb3837fe640 1 -- 172.21.15.17:0/4238506547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb384136fe0 msgr2=0x7fb3841393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb3837fe640 1 --2- 172.21.15.17:0/4238506547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb384136fe0 0x7fb3841393c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009e60 tx=0x7fb37c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea489733de5f137c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 1 -- 172.21.15.17:0/4238506547 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c02aea0 con 0x7fb384136fe0 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: dump: 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: _finish_auth 0 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 1 -- 172.21.15.17:0/4238506547 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c02d050 con 0x7fb384136fe0 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 1 -- 172.21.15.17:0/4238506547 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c007a70 con 0x7fb384136fe0 2024-09-15T02:11:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb381ffb640 10 monclient: dump: 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb389a34640 5 monclient: authenticate success, global_id 244171 2024-09-15T02:11:54.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb389a34640 10 monclient: get_monmap_and_config success 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.021+0000 7fb389a34640 4 set_mon_vals no callback set 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 10 monclient: shutdown 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 -- 172.21.15.17:0/4238506547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb384136fe0 msgr2=0x7fb3841393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 --2- 172.21.15.17:0/4238506547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb384136fe0 0x7fb3841393c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009e60 tx=0x7fb37c009780 comp rx=0 tx=0).stop 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 -- 172.21.15.17:0/4238506547 shutdown_connections 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 --2- 172.21.15.17:0/4238506547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb384136fe0 0x7fb3841393c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 --2- 172.21.15.17:0/4238506547 >> v2:172.21.15.17:3300/0 conn(0x7fb3840692f0 0x7fb3840696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 -- 172.21.15.17:0/4238506547 >> 172.21.15.17:0/4238506547 conn(0x7fb384070fb0 msgr2=0x7fb3840a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 -- 172.21.15.17:0/4238506547 shutdown_connections 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.022+0000 7fb389a34640 1 -- 172.21.15.17:0/4238506547 wait complete. 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 10 monclient: build_initial_monmap 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 10 monclient: monmap: 2024-09-15T02:11:54.030 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 1 Processor -- start 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 1 -- start start 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 10 monclient: init 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.023+0000 7fb389a34640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:54.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-c con 0x7fb384068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 0x7fb384099e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-b con 0x7fb3840692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 0x7fb38409a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): picked mon.noname-a con 0x7fb384136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840a07c0 con 0x7fb384136fe0 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38409c550 con 0x7fb3840692f0 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38411c480 con 0x7fb384068f20 2024-09-15T02:11:54.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb389a34640 10 monclient(hunting): authenticate will time out at 1974.020091s 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb3837fe640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 msgr2=0x7fb38409a3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb3837fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 0x7fb38409a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57276/0 (socket says 172.21.15.17:57276) 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 -- 172.21.15.17:0/1632161030 learned_addr learned my addr 172.21.15.17:0/1632161030 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): get_auth_request con 0x7fb384068f20 auth_method 0 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:11:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb3827fc640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 0x7fb384099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient(hunting): handle_auth_done global_id 214463 payload 824 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 -- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 msgr2=0x7fb384099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 0x7fb384099e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 -- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 msgr2=0x7fb38409a3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 0x7fb38409a3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 monclient: found mon.noname-c 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb382ffd640 1 -- 172.21.15.17:0/1632161030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c009a20 con 0x7fb384068f20 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb3827fc640 10 monclient: get_auth_request con 0x7fb3840692f0 auth_method 0 2024-09-15T02:11:54.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.024+0000 7fb3827fc640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 0x7fb384099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb382ffd640 1 -- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 msgr2=0x7fb384099950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb382ffd640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c5e0 tx=0x7fb374004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374005cb0 con 0x7fb384068f20 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: dump: 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.037 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: _finish_auth 0 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb389a34640 5 monclient: authenticate success, global_id 214463 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb389a34640 10 monclient: _renew_subs 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb374005e30 con 0x7fb384068f20 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb389a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38409b480 con 0x7fb384068f20 2024-09-15T02:11:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374007340 con 0x7fb384068f20 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb36affd640 10 monclient: dump: 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:11:54.039 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.027+0000 7fb389a34640 10 monclient: _renew_subs 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.027+0000 7fb389a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.027+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38411cb00 con 0x7fb384068f20 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.026+0000 7fb3817fa640 4 set_mon_vals no callback set 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.027+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb37401b400 con 0x7fb384068f20 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.027+0000 7fb389a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb389a34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb389a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3840724a0 con 0x7fb384068f20 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb36affd640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 0x7fb37003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb37404c310 con 0x7fb384068f20 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb3827fc640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 0x7fb37003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.028+0000 7fb3827fc640 10 monclient: get_auth_request con 0x7fb3700386f0 auth_method 0 2024-09-15T02:11:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.031+0000 7fb3827fc640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 0x7fb37003aba0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb384136ab0 tx=0x7fb378008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:11:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.032+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb37401b6b0 con 0x7fb384068f20 2024-09-15T02:11:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.032+0000 7fb36affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:11:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.032+0000 7fb36affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:11:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb389a34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:11:54.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb389a34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb389a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb38411cdf0 con 0x7fb384068f20 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb36affd640 1 -- 172.21.15.17:0/1632161030 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fb37401a360 con 0x7fb384068f20 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb36affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.259+0000 7fb36affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:11:54.262 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:11:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 msgr2=0x7fb37003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 0x7fb37003aba0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb384136ab0 tx=0x7fb378008040 comp rx=0 tx=0).stop 2024-09-15T02:11:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 10 monclient: shutdown 2024-09-15T02:11:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:11:54.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 msgr2=0x7fb384099950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c5e0 tx=0x7fb374004960 comp rx=0 tx=0).stop 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 shutdown_connections 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3700386f0 0x7fb37003aba0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb384136fe0 0x7fb38409a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3840692f0 0x7fb384099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 --2- 172.21.15.17:0/1632161030 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb384068f20 0x7fb384099950 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 >> 172.21.15.17:0/1632161030 conn(0x7fb384070fb0 msgr2=0x7fb38409e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 shutdown_connections 2024-09-15T02:11:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:54.261+0000 7fb389a34640 1 -- 172.21.15.17:0/1632161030 wait complete. 2024-09-15T02:11:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:54.999+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:54.999+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:56.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:55.999+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:56.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:55.999+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:56.999+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:57.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:56.999+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:57.274 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:11:57.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 10 monclient: get_monmap_and_config 2024-09-15T02:11:57.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 10 monclient: build_initial_monmap 2024-09-15T02:11:57.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 10 monclient: monmap: 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:11:57.382098+0000 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:11:57.382098+0000 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:11:57.383 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:11:57.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 1 Processor -- start 2024-09-15T02:11:57.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 1 -- start start 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.381+0000 7f1dc099d640 10 monclient: init 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient: _reopen_session rank -1 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1dbc0a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-a con 0x7f1dbc132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-c con 0x7f1dbc136dd0 addr v2:172.21.15.67:3300/0 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc0a01e0 con 0x7f1dbc0a58b0 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dbc0a1580 con 0x7f1dbc132040 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:11:57.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dbc0a5c80 con 0x7f1dbc136dd0 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): _renew_subs 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dc099d640 10 monclient(hunting): authenticate will time out at 1977.378654s 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1dba59c640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f1dbc0a58b0 legacy=0x7f1dbc0980a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1db9d9b640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 msgr2=0x7f1dbc0987b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.382+0000 7f1db9d9b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:35394/0 (socket says 172.21.15.17:35394) 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 learned_addr learned my addr 172.21.15.17:0/1010231867 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request con 0x7f1dbc136dd0 auth_method 0 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:11:57.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:11:57.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:57.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:57.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.383+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:11:57.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.583+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.17:6789/0 conn(0x7f1dbc0a58b0 legacy=0x7f1dbc0980a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:57.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.583+0000 7f1db9d9b640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 msgr2=0x7f1dbc0987b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:57.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.583+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:11:57.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:57.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request con 0x7f1dbc136dd0 auth_method 0 2024-09-15T02:11:57.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:57.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.584+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:11:57.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.984+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.17:6789/0 conn(0x7f1dbc0a58b0 legacy=0x7f1dbc0980a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:57.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.984+0000 7f1db9d9b640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 msgr2=0x7f1dbc0987b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:57.985 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.984+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request con 0x7f1dbc136dd0 auth_method 0 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:57.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:57.985+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:11:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:57.999+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:57.999+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:58.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.785+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.17:6789/0 conn(0x7f1dbc0a58b0 legacy=0x7f1dbc0980a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:11:58.786 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.785+0000 7f1db9d9b640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 msgr2=0x7f1dbc0987b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:11:58.787 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.785+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:11:58.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:11:58.788 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request con 0x7f1dbc136dd0 auth_method 0 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:11:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:11:58.787+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:11:59.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:58.999+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:11:59.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:58.999+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:11:59.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:11:58.999+0000 7fa725ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.000+0000 7fa725ffb640 10 monclient: tick 2024-09-15T02:12:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.000+0000 7fa725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 10 monclient(hunting): tick 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 monclient(hunting): continuing hunt 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 msgr2=0x7f1dbc1391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.67:3300/0 conn(0x7f1dbc136dd0 0x7f1dbc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 -- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 msgr2=0x7f1dbc0987b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 --2- 172.21.15.17:0/1010231867 >> v2:172.21.15.17:3300/0 conn(0x7f1dbc132040 0x7f1dbc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.17:6789/0 conn(0x7f1dbc0a58b0 legacy=0x7f1dbc0980a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d9c003650 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.382+0000 7f1dabfff640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1d9c0097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1dabfff640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9c007310 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1dabfff640 1 -- 172.21.15.17:0/1010231867 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc132440 con 0x7f1d9c003650 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1dabfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1dabfff640 1 -- 172.21.15.17:0/1010231867 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc0a1580 con 0x7f1d9c007310 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1dabfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1d9c0097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 10 monclient(hunting): get_auth_request con 0x7f1d9c007310 auth_method 0 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1d9c0097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1d9c0097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1d9c0097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.383+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1d9c0097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.385+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.67:6789/0 conn(0x7f1d9c003650 legacy=0x7f1d9c005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.459+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325420569 0 0) 0x7f1dbc132440 con 0x7f1d9c003650 2024-09-15T02:12:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.459+0000 7f1db959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.459+0000 7f1db959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.459+0000 7f1db959a640 10 monclient(hunting): my global_id is 244174 2024-09-15T02:12:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.459+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da40034e0 con 0x7f1d9c003650 2024-09-15T02:12:00.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.459+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v39) v1 ==== 79+0+0 (secure 0 0 0) 0x7fa73401a360 con 0x7fa74009bf90 2024-09-15T02:12:00.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.459+0000 7fa726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:12:00.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.459+0000 7fa726ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:00.464 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079041967 0 0) 0x7f1da40034e0 con 0x7f1d9c003650 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1d9c0097c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1d9c0097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc0a01e0 con 0x7f1d9c003650 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.460+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f1dac004300 con 0x7f1d9c003650 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.465 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient: dump: 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.466 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:12:00.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 >> v1:172.21.15.67:6789/0 conn(0x7f1d9c003650 legacy=0x7f1d9c005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.467 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 reap_dead start 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1da40039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): picked mon.c con 0x7f1d9c007310 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4003fe0 0x7f1da4004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): picked mon.b con 0x7f1da4003fe0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 0x7f1da4005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): picked mon.a con 0x7f1da4004cc0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc0a1580 con 0x7f1da4004cc0 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da400bb90 con 0x7f1da4003fe0 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.468 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da400bd50 con 0x7f1d9c007310 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dbad9d640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 msgr2=0x7f1da4005130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1da40039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 0x7f1da4005130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): get_auth_request con 0x7f1d9c007310 auth_method 0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4003fe0 0x7f1da4004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): get_auth_request con 0x7f1da4003fe0 auth_method 0 2024-09-15T02:12:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1db9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient(hunting): handle_auth_done global_id 244174 payload 293 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4003fe0 msgr2=0x7f1da4004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4003fe0 0x7f1da4004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 msgr2=0x7f1da4005130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 0x7f1da4005130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.470 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 monclient: found mon.c 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40034e0 con 0x7f1d9c007310 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dc099d640 5 monclient: authenticate success, global_id 244174 2024-09-15T02:12:00.471 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.461+0000 7f1dc099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.464+0000 7f1dba59c640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1da40039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.464+0000 7f1dba59c640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1da40039d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1dac003940 tx=0x7f1dac007bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ead3742c163415 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac007150 con 0x7f1d9c007310 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: dump: 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.477 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1dc099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dac0072d0 con 0x7f1d9c007310 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1dc099d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 1 -- 172.21.15.17:0/1010231867 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac007690 con 0x7f1d9c007310 2024-09-15T02:12:00.478 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1dc099d640 4 set_mon_vals no callback set 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.466+0000 7f1db959a640 10 monclient: dump: 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.479 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 10 monclient: shutdown 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 -- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1da40039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1da40039d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1dac003940 tx=0x7f1dac007bd0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 -- 172.21.15.17:0/1010231867 shutdown_connections 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4004cc0 0x7f1da4005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4003fe0 0x7f1da4004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 --2- 172.21.15.17:0/1010231867 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1da40039d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 -- 172.21.15.17:0/1010231867 >> 172.21.15.17:0/1010231867 conn(0x7f1dbc070ed0 msgr2=0x7f1dbc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 -- 172.21.15.17:0/1010231867 shutdown_connections 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.467+0000 7f1dc099d640 1 -- 172.21.15.17:0/1010231867 wait complete. 2024-09-15T02:12:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.468+0000 7f1dc099d640 10 monclient: build_initial_monmap 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.468+0000 7f1dc099d640 10 monclient: monmap: 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.468+0000 7f1dc099d640 1 Processor -- start 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.468+0000 7f1dc099d640 1 -- start start 2024-09-15T02:12:00.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.468+0000 7f1dc099d640 10 monclient: init 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d9c007310 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 0x7f1dbc1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-a con 0x7f1da4003fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 0x7f1dbc134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): picked mon.noname-b con 0x7f1da4004cc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc11bb80 con 0x7f1da4003fe0 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc131930 con 0x7f1da4004cc0 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc098dc0 con 0x7f1d9c007310 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dc099d640 10 monclient(hunting): authenticate will time out at 1980.465145s 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1db9d9b640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 msgr2=0x7f1dbc1419b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1db9d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 0x7f1dbc1419b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57320/0 (socket says 172.21.15.17:57320) 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 1 -- 172.21.15.17:0/2291338639 learned_addr learned my addr 172.21.15.17:0/2291338639 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): get_auth_request con 0x7f1d9c007310 auth_method 0 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 0x7f1dbc134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.483 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request con 0x7f1da4004cc0 auth_method 0 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.469+0000 7f1dbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 10 monclient(hunting): handle_auth_done global_id 214466 payload 824 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 -- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 msgr2=0x7f1dbc134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 0x7f1dbc134370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.484 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 -- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 msgr2=0x7f1dbc1419b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 0x7f1dbc1419b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 monclient: found mon.noname-c 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dba59c640 1 -- 172.21.15.17:0/2291338639 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc097470 con 0x7f1d9c007310 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.470+0000 7f1dbad9d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 0x7f1dbc134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dba59c640 1 -- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1dbc141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dba59c640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00c630 tx=0x7f1dac034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac03e070 con 0x7f1d9c007310 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:00.485 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: dump: 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dac004c40 con 0x7f1d9c007310 2024-09-15T02:12:00.486 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dc099d640 5 monclient: authenticate success, global_id 214466 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dc099d640 10 monclient: _renew_subs 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dc099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc136670 con 0x7f1d9c007310 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac02c480 con 0x7f1d9c007310 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1daa7fc640 10 monclient: dump: 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.487 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.472+0000 7f1db8d99640 4 set_mon_vals no callback set 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.473+0000 7f1dc099d640 10 monclient: _renew_subs 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.473+0000 7f1dc099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.473+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc135f00 con 0x7f1d9c007310 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1dac02c7f0 con 0x7f1d9c007310 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1daa7fc640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 0x7f1d9c03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1dac06c1e0 con 0x7f1d9c007310 2024-09-15T02:12:00.488 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1dc099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1dc099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1dc099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc0724a0 con 0x7f1d9c007310 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 0x7f1d9c03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.474+0000 7f1db9d9b640 10 monclient: get_auth_request con 0x7f1d9c038ea0 auth_method 0 2024-09-15T02:12:00.489 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.477+0000 7f1db9d9b640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 0x7f1d9c03b350 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1db0003940 tx=0x7f1db0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.479+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1dac031080 con 0x7f1d9c007310 2024-09-15T02:12:00.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.479+0000 7f1daa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:00.490 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.479+0000 7f1daa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa74579d640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa74579d640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa74579d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fa74004bc80 con 0x7fa74009bf90 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa726ffd640 1 -- 172.21.15.67:0/704630840 <== mon.1 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v39) v1 ==== 79+0+0 (secure 0 0 0) 0x7fa73401a540 con 0x7fa74009bf90 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa726ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T02:12:00.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.623+0000 7fa726ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T02:12:00.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 msgr2=0x7fa72c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 0x7fa72c03abf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa730009cf0 tx=0x7fa730008040 comp rx=0 tx=0).stop 2024-09-15T02:12:00.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 10 monclient: shutdown 2024-09-15T02:12:00.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 msgr2=0x7fa74012fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa73400c5e0 tx=0x7fa734004960 comp rx=0 tx=0).stop 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 shutdown_connections 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fa72c038740 0x7fa72c03abf0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fa74009c360 0x7fa74009ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74009bf90 0x7fa74012fce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 --2- 172.21.15.67:0/704630840 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa740097260 0x7fa7401413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 >> 172.21.15.67:0/704630840 conn(0x7fa7400716e0 msgr2=0x7fa740099890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 shutdown_connections 2024-09-15T02:12:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-09-15T02:12:00.625+0000 7fa74579d640 1 -- 172.21.15.67:0/704630840 wait complete. 2024-09-15T02:12:00.635 INFO:tasks.workunit.client.0.smithi067.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-15T02:12:00.635 INFO:tasks.workunit.client.0.smithi067.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-15T02:12:00.636 INFO:tasks.workunit.client.0.smithi067.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-15T02:12:00.636 INFO:tasks.workunit.client.0.smithi067.stderr:+ expected_ret=1 2024-09-15T02:12:00.636 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-15T02:12:00.636 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1dc099d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1dc099d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1dc099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1dbc11b960 con 0x7f1d9c007310 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1daa7fc640 1 -- 172.21.15.17:0/2291338639 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f1dac02d450 con 0x7f1d9c007310 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1daa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.707+0000 7f1daa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:00.709 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 msgr2=0x7f1d9c03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 0x7f1d9c03b350 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1db0003940 tx=0x7f1db0008040 comp rx=0 tx=0).stop 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 10 monclient: shutdown 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 msgr2=0x7f1dbc141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00c630 tx=0x7f1dac034000 comp rx=0 tx=0).stop 2024-09-15T02:12:00.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 shutdown_connections 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1d9c038ea0 0x7f1d9c03b350 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1da4003fe0 0x7f1dbc1419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da4004cc0 0x7f1dbc134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 --2- 172.21.15.17:0/2291338639 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1d9c007310 0x7f1dbc141470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 >> 172.21.15.17:0/2291338639 conn(0x7f1dbc070ed0 msgr2=0x7f1dbc09e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 shutdown_connections 2024-09-15T02:12:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.709+0000 7f1dc099d640 1 -- 172.21.15.17:0/2291338639 wait complete. 2024-09-15T02:12:00.721 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T02:12:00.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T02:12:00.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:12:00.818 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.817+0000 7f50b4e78640 10 monclient: get_monmap_and_config 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.817+0000 7f50b4e78640 10 monclient: build_initial_monmap 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.817+0000 7f50b4e78640 10 monclient: monmap: 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:00.818194+0000 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:00.818194+0000 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:00.820 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:00.820 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:00.820 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:00.820 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:00.820 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 Processor -- start 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 -- start start 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient: init 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 0x7f50b009a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-a con 0x7f50b009c0d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:00.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50b009c4a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-b con 0x7f50b009c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00699c0 con 0x7f50b009c4a0 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b006a5e0 con 0x7f50b009c0d0 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b006a290 con 0x7f50b009c8a0 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50b4e78640 10 monclient(hunting): authenticate will time out at 1980.814678s 2024-09-15T02:12:00.823 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50a7fff640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7f50b009c4a0 legacy=0x7f50b006eb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50ae59c640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 msgr2=0x7f50b009a920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50ae59c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 0x7f50b009a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57326/0 (socket says 172.21.15.17:57326) 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 1 -- 172.21.15.17:0/636280971 learned_addr learned my addr 172.21.15.17:0/636280971 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): get_auth_request con 0x7f50b009c8a0 auth_method 0 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.818+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 10 monclient(hunting): handle_auth_done global_id 214475 payload 274 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 1 -- 172.21.15.17:0/636280971 >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 msgr2=0x7f50b009a920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 1 --2- 172.21.15.17:0/636280971 >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 0x7f50b009a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 1 -- 172.21.15.17:0/636280971 >> v1:172.21.15.17:6789/0 conn(0x7f50b009c4a0 legacy=0x7f50b006eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 1 monclient: found mon.noname-b 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.819+0000 7f50aed9d640 1 -- 172.21.15.17:0/636280971 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b009f6e0 con 0x7f50b009c8a0 2024-09-15T02:12:00.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50aed9d640 1 -- 172.21.15.17:0/636280971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 msgr2=0x7f50b009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50aed9d640 1 --2- 172.21.15.17:0/636280971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f50b00a3e50 tx=0x7f50a8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c09813197dd63df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 1 -- 172.21.15.17:0/636280971 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8007610 con 0x7f50b009c8a0 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: dump: 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.833 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 1 -- 172.21.15.17:0/636280971 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a802d050 con 0x7f50b009c8a0 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 5 monclient: authenticate success, global_id 214475 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 1 -- 172.21.15.17:0/636280971 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a80079a0 con 0x7f50b009c8a0 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 4 set_mon_vals no callback set 2024-09-15T02:12:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50add9b640 10 monclient: dump: 2024-09-15T02:12:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 10 monclient: shutdown 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 -- 172.21.15.17:0/636280971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 msgr2=0x7f50b009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 --2- 172.21.15.17:0/636280971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f50b00a3e50 tx=0x7f50a8009840 comp rx=0 tx=0).stop 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 -- 172.21.15.17:0/636280971 shutdown_connections 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 --2- 172.21.15.17:0/636280971 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b009e970 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 --2- 172.21.15.17:0/636280971 >> v2:172.21.15.17:3300/0 conn(0x7f50b009c0d0 0x7f50b009a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 -- 172.21.15.17:0/636280971 >> 172.21.15.17:0/636280971 conn(0x7f50b00712f0 msgr2=0x7f50b00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 -- 172.21.15.17:0/636280971 shutdown_connections 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.822+0000 7f50b4e78640 1 -- 172.21.15.17:0/636280971 wait complete. 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient: build_initial_monmap 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient: monmap: 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:00.837 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 1 Processor -- start 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 1 -- start start 2024-09-15T02:12:00.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient: init 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50b009c0d0 0x7f50b0141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-a con 0x7f50b009c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b009c4a0 0x7f50b0134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.823+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-b con 0x7f50b009c4a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b011e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): picked mon.noname-c con 0x7f50b009c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0122000 con 0x7f50b009c0d0 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50ae59c640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50b009c0d0 msgr2=0x7f50b0141000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0122650 con 0x7f50b009c4a0 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b009cd80 con 0x7f50b009c8a0 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50b4e78640 10 monclient(hunting): authenticate will time out at 1980.820040s 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50ae59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50b009c0d0 0x7f50b0141000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b011e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b011e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57330/0 (socket says 172.21.15.17:57330) 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 learned_addr learned my addr 172.21.15.17:0/3886486569 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): get_auth_request con 0x7f50b009c8a0 auth_method 0 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50a7fff640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b009c4a0 0x7f50b0134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient(hunting): handle_auth_done global_id 214478 payload 824 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b009c4a0 msgr2=0x7f50b0134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b009c4a0 0x7f50b0134ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50b009c0d0 msgr2=0x7f50b0141000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f50b009c0d0 0x7f50b0141000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 monclient: found mon.noname-c 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8009ae0 con 0x7f50b009c8a0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50a7fff640 10 monclient: get_auth_request con 0x7f50b009c4a0 auth_method 0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.824+0000 7f50a7fff640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b009c4a0 0x7f50b0134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 msgr2=0x7f50b011e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50aed9d640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b011e990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009ab0 tx=0x7f50a8036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8031cd0 con 0x7f50b009c8a0 2024-09-15T02:12:00.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: dump: 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a8031e30 con 0x7f50b009c8a0 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 5 monclient: authenticate success, global_id 214478 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 10 monclient: _renew_subs 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b009ed00 con 0x7f50b009c8a0 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a803c630 con 0x7f50b009c8a0 2024-09-15T02:12:00.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50a67fc640 10 monclient: dump: 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.845 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50ad59a640 4 set_mon_vals no callback set 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 10 monclient: _renew_subs 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.826+0000 7f50b4e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b0141db0 con 0x7f50b009c8a0 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50b4e78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50b4e78640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 0x7f50b011eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0063b90 con 0x7f50b0135420 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f50a803ca20 con 0x7f50b009c8a0 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50ae59c640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 0x7f50b011eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50ae59c640 10 monclient: get_auth_request con 0x7f50b0135420 auth_method 0 2024-09-15T02:12:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.827+0000 7f50ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50a67fc640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50900386f0 0x7f509003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50ae59c640 10 monclient(hunting): handle_auth_done global_id 214481 payload 824 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f50a806d7c0 con 0x7f50b009c8a0 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50ae59c640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50b00724a0 con 0x7f50b0135420 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50a7fff640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50900386f0 0x7f509003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.828+0000 7f50a7fff640 10 monclient: get_auth_request con 0x7f50900386f0 auth_method 0 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50ae59c640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 msgr2=0x7f50b011eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50ae59c640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 0x7f50b011eed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b440 tx=0x7f50a0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0003df0 con 0x7f50b0135420 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 10 monclient: dump: 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f50a00050f0 con 0x7f50b0135420 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 msgr2=0x7f50b011eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 0x7f50b011eed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b440 tx=0x7f50a0009120 comp rx=0 tx=0).stop 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 msgr2=0x7f50b011eed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a67fc640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b0135420 0x7f50b011eed0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.830+0000 7f50a7fff640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50900386f0 0x7f509003aba0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5098003940 tx=0x7f5098008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50b4e78640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:12:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50b4e78640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 0x7f50b006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50b4e78640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b01357f0 con 0x7f50b011f410 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 0x7f50b006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient: get_auth_request con 0x7f50b011f410 auth_method 0 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 10 monclient(hunting): handle_auth_done global_id 214484 payload 824 2024-09-15T02:12:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.836+0000 7f50ae59c640 1 -- 172.21.15.17:0/3886486569 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f50b011f810 con 0x7f50b011f410 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50ae59c640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 msgr2=0x7f50b006bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50ae59c640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 0x7f50b006bb00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b440 tx=0x7f50a0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0005850 con 0x7f50b011f410 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 10 monclient: dump: 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.853 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4555 (secure 0 0 0) 0x7f50a0003be0 con 0x7f50b011f410 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 msgr2=0x7f50b006bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 0x7f50b006bb00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b440 tx=0x7f50a0015000 comp rx=0 tx=0).stop 2024-09-15T02:12:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 msgr2=0x7f50b006bb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50a67fc640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b011f410 0x7f50b006bb00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 206, 2024-09-15T02:12:00.855 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:00.856 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:00.857 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:12:00.858 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:00.859 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:12:00.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:00.863 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:00.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.865 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:00.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:12:00.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:00.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:00.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:00.870 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:12:00.870 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:00.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50900386f0 msgr2=0x7f509003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f50900386f0 0x7f509003aba0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5098003940 tx=0x7f5098008040 comp rx=0 tx=0).stop 2024-09-15T02:12:00.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 10 monclient: shutdown 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 msgr2=0x7f50b011e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 --2- 172.21.15.17:0/3886486569 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f50b009c8a0 0x7f50b011e990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009ab0 tx=0x7f50a8036000 comp rx=0 tx=0).stop 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 reap_dead start 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50aed9d640 1 -- 172.21.15.17:0/3886486569 reap_dead start 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 shutdown_connections 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.839+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 >> 172.21.15.17:0/3886486569 conn(0x7f50b00712f0 msgr2=0x7f50b0098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.840+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 shutdown_connections 2024-09-15T02:12:00.872 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.840+0000 7f50b4e78640 1 -- 172.21.15.17:0/3886486569 wait complete. 2024-09-15T02:12:00.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:12:00.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T02:12:00.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:12:00.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.968+0000 7fb1a008b640 10 monclient: get_monmap_and_config 2024-09-15T02:12:00.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.968+0000 7fb1a008b640 10 monclient: build_initial_monmap 2024-09-15T02:12:00.970 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.968+0000 7fb1a008b640 10 monclient: monmap: 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:00.969761+0000 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:00.969761+0000 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:00.971 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:00.972 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.969+0000 7fb1a008b640 1 Processor -- start 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.969+0000 7fb1a008b640 1 -- start start 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.969+0000 7fb1a008b640 10 monclient: init 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T02:12:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 0x7fb19809a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-a con 0x7fb19809c0d0 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb19809c4a0 0x7fb19806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-c con 0x7fb19809c4a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb19809c8a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980699c0 con 0x7fb19809c8a0 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19806a5e0 con 0x7fb19809c0d0 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19806a290 con 0x7fb19809c4a0 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb1a008b640 10 monclient(hunting): authenticate will time out at 1980.966265s 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19e628640 1 -- >> v1:172.21.15.17:6789/0 conn(0x7fb19809c8a0 legacy=0x7fb19806f070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.17:6789/0 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19de27640 1 -- >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 msgr2=0x7fb19809a920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19de27640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 0x7fb19809a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb19809c4a0 0x7fb19806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb19809c4a0 0x7fb19806eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.17:35450/0 (socket says 172.21.15.17:35450) 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 -- 172.21.15.17:0/1859725940 learned_addr learned my addr 172.21.15.17:0/1859725940 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:00.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): get_auth_request con 0x7fb19809c4a0 auth_method 0 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient(hunting): handle_auth_done global_id 244183 payload 274 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 -- 172.21.15.17:0/1859725940 >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 msgr2=0x7fb19809a920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 --2- 172.21.15.17:0/1859725940 >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 0x7fb19809a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 -- 172.21.15.17:0/1859725940 >> v1:172.21.15.17:6789/0 conn(0x7fb19809c8a0 legacy=0x7fb19806f070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:00.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 monclient: found mon.noname-c 2024-09-15T02:12:00.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:12:00.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.970+0000 7fb19d626640 1 -- 172.21.15.17:0/1859725940 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19809f6e0 con 0x7fb19809c4a0 2024-09-15T02:12:00.986 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.973+0000 7fb19d626640 1 -- 172.21.15.17:0/1859725940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 msgr2=0x7fb19806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.973+0000 7fb19d626640 1 --2- 172.21.15.17:0/1859725940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19806eb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb19809f5e0 tx=0x7fb1880091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fd14e010d96241 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 1 -- 172.21.15.17:0/1859725940 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188032440 con 0x7fb19809c4a0 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: dump: 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.987 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 1 -- 172.21.15.17:0/1859725940 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1880325a0 con 0x7fb19809c4a0 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 1 -- 172.21.15.17:0/1859725940 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188032940 con 0x7fb19809c4a0 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 5 monclient: authenticate success, global_id 244183 2024-09-15T02:12:00.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 4 set_mon_vals no callback set 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb19ce25640 10 monclient: dump: 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 10 monclient: shutdown 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 -- 172.21.15.17:0/1859725940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 msgr2=0x7fb19806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 --2- 172.21.15.17:0/1859725940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19806eb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb19809f5e0 tx=0x7fb1880091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 -- 172.21.15.17:0/1859725940 shutdown_connections 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 --2- 172.21.15.17:0/1859725940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19806eb30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 --2- 172.21.15.17:0/1859725940 >> v2:172.21.15.17:3300/0 conn(0x7fb19809c0d0 0x7fb19809a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 -- 172.21.15.17:0/1859725940 >> 172.21.15.17:0/1859725940 conn(0x7fb1980712f0 msgr2=0x7fb1980a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 -- 172.21.15.17:0/1859725940 shutdown_connections 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.974+0000 7fb1a008b640 1 -- 172.21.15.17:0/1859725940 wait complete. 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.975+0000 7fb1a008b640 10 monclient: build_initial_monmap 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.975+0000 7fb1a008b640 10 monclient: monmap: 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.991 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.975+0000 7fb1a008b640 1 Processor -- start 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.975+0000 7fb1a008b640 1 -- start start 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.975+0000 7fb1a008b640 10 monclient: init 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19809c0d0 0x7fb19813db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-a con 0x7fb19809c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:00.992 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19813d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-b con 0x7fb19809c4a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19de27640 1 -- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19809c0d0 msgr2=0x7fb19813db90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:3300/0 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): picked mon.noname-c con 0x7fb19809c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19809cd80 con 0x7fb19809c0d0 2024-09-15T02:12:00.993 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19de27640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19809c0d0 0x7fb19813db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1980a1140 con 0x7fb19809c4a0 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb198122650 con 0x7fb19809c8a0 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb1a008b640 10 monclient(hunting): authenticate will time out at 1980.972232s 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:57362/0 (socket says 172.21.15.17:57362) 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 learned_addr learned my addr 172.21.15.17:0/3018836326 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): get_auth_request con 0x7fb19809c8a0 auth_method 0 2024-09-15T02:12:00.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19813d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): get_auth_request con 0x7fb19809c4a0 auth_method 0 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19d626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient(hunting): handle_auth_done global_id 214487 payload 824 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 msgr2=0x7fb19813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19809c4a0 0x7fb19813d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19809c0d0 msgr2=0x7fb19813db90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb19809c0d0 0x7fb19813db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:00.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient: _finish_hunting 0 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 monclient: found mon.noname-c 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.976+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb188009450 con 0x7fb19809c8a0 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.978+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 msgr2=0x7fb1981410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.978+0000 7fb19e628640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb190009fc0 tx=0x7fb190009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb190016430 con 0x7fb19809c8a0 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: dump: 2024-09-15T02:12:00.997 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1900165d0 con 0x7fb19809c8a0 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 5 monclient: authenticate success, global_id 214487 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 10 monclient: _renew_subs 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1981207f0 con 0x7fb19809c8a0 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb190016780 con 0x7fb19809c8a0 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18dffb640 10 monclient: dump: 2024-09-15T02:12:00.999 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb18ffff640 4 set_mon_vals no callback set 2024-09-15T02:12:01.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 10 monclient: _renew_subs 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.979+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb198122390 con 0x7fb19809c8a0 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb190012020 con 0x7fb19809c8a0 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb1a008b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb1a008b640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 0x7fb198141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:01.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb198063b90 con 0x7fb19811fef0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb18dffb640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb17c038740 0x7fb17c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb19004cbc0 con 0x7fb19809c8a0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 0x7fb198141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient: get_auth_request con 0x7fb19811fef0 auth_method 0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19d626640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb17c038740 0x7fb17c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19d626640 10 monclient: get_auth_request con 0x7fb17c038740 auth_method 0 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:01.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 10 monclient(hunting): handle_auth_done global_id 244189 payload 824 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.980+0000 7fb19de27640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb198122790 con 0x7fb19811fef0 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.983+0000 7fb19de27640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 msgr2=0x7fb198141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.983+0000 7fb19de27640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 0x7fb198141600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb19400b440 tx=0x7fb194009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.983+0000 7fb19d626640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb17c038740 0x7fb17c03abf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb188031000 tx=0x7fb188032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb194003e70 con 0x7fb19811fef0 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 10 monclient: dump: 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:01.003 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb1940050f0 con 0x7fb19811fef0 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:01.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 msgr2=0x7fb198141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 0x7fb198141600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb19400b440 tx=0x7fb194009120 comp rx=0 tx=0).stop 2024-09-15T02:12:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 msgr2=0x7fb198141600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:01.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.985+0000 7fb18dffb640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19811fef0 0x7fb198141600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb1a008b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb1a008b640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 0x7fb19806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb1a008b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb198141f40 con 0x7fb198141b40 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 0x7fb19806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient: get_auth_request con 0x7fb198141b40 auth_method 0 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:01.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 10 monclient(hunting): handle_auth_done global_id 244192 payload 824 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.991+0000 7fb19de27640 1 -- 172.21.15.17:0/3018836326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1981215a0 con 0x7fb198141b40 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb19de27640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 msgr2=0x7fb19806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb19de27640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 0x7fb19806bb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb194014000 tx=0x7fb194015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb194005990 con 0x7fb198141b40 2024-09-15T02:12:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 10 monclient: dump: 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:01.009 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4727 (secure 0 0 0) 0x7fb194003c60 con 0x7fb198141b40 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 msgr2=0x7fb19806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 0x7fb19806bb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb194014000 tx=0x7fb194015000 comp rx=0 tx=0).stop 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 msgr2=0x7fb19806bb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.994+0000 7fb18dffb640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198141b40 0x7fb19806bb00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:01.010 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 206, 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 0, 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:12:01.011 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:12:01.012 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.013 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:12:01.014 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:01.015 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.016 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:01.017 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:01.018 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:01.019 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.020 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:01.021 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.023 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:01.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 3 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:12:01.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:12:01.026 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb17c038740 msgr2=0x7fb17c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb17c038740 0x7fb17c03abf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb188031000 tx=0x7fb188032000 comp rx=0 tx=0).stop 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 10 monclient: shutdown 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 msgr2=0x7fb1981410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb19e628640 1 -- 172.21.15.17:0/3018836326 reap_dead start 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb190009fc0 tx=0x7fb190009730 comp rx=0 tx=0).stop 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 shutdown_connections 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 --2- 172.21.15.17:0/3018836326 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb19809c8a0 0x7fb1981410c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 >> 172.21.15.17:0/3018836326 conn(0x7fb1980712f0 msgr2=0x7fb198098a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 shutdown_connections 2024-09-15T02:12:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:00.995+0000 7fb1a008b640 1 -- 172.21.15.17:0/3018836326 wait complete. 2024-09-15T02:12:01.030 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T02:12:01.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T02:12:01.785 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=1 2024-09-15T02:12:01.785 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ 1 -ne 1 ]] 2024-09-15T02:12:01.785 INFO:tasks.workunit.client.0.smithi067.stderr:+ return 0 2024-09-15T02:12:01.786 INFO:tasks.workunit.client.0.smithi067.stdout:OK 2024-09-15T02:12:01.786 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo OK 2024-09-15T02:12:01.787 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:12:01.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:12:01.812 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-15T02:12:01.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T02:12:02.129 DEBUG:teuthology.parallel:result is None 2024-09-15T02:12:02.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:12:02.155 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:12:02.155 DEBUG:teuthology.orchestra.run.smithi067:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:12:02.215 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T02:12:02.215 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-15T02:12:02.225 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-15T02:12:21.030 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T02:12:21.031 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T02:12:21.031 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T02:12:21.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T02:12:21.035 INFO:tasks.ceph.mon.a:Started 2024-09-15T02:12:21.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:12:21.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:12:21.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:21.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:21.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 10 monclient: monmap: 2024-09-15T02:12:21.144 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:21.144 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:21.143854+0000 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:21.143854+0000 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:21.145 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 1 Processor -- start 2024-09-15T02:12:21.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 1 -- start start 2024-09-15T02:12:21.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.143+0000 7f27c1b9d640 10 monclient: init 2024-09-15T02:12:21.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:21.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27bc0a58b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc132040 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27bc136dd0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0a01e0 con 0x7f27bc0a58b0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0a1580 con 0x7f27bc136dd0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0a5c80 con 0x7f27bc132040 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27c1b9d640 10 monclient(hunting): authenticate will time out at 2001.140511s 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27bb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f27bc0a58b0 0x7f27bc0980a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:49618/0 (socket says 172.21.15.17:49618) 2024-09-15T02:12:21.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27bb7fe640 1 -- 172.21.15.17:0/1043135177 learned_addr learned my addr 172.21.15.17:0/1043135177 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51068/0 (socket says 172.21.15.17:51068) 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request con 0x7f27bc132040 auth_method 0 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 msgr2=0x7f27bc0987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 msgr2=0x7f27bc0987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.144+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:21.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.146+0000 7f27bb7fe640 1 -- 172.21.15.17:0/1043135177 >> v1:172.21.15.17:6789/0 conn(0x7f27bc0a58b0 legacy=0x7f27bc0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:21.158 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.146+0000 7f27bbfff640 1 -- 172.21.15.17:0/1043135177 >> v1:172.21.15.67:6789/0 conn(0x7f27bc136dd0 legacy=0x7f27bc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480550686 0 0) 0x7f27bc0a1580 con 0x7f27bc136dd0 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): my global_id is 244195 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a80034e0 con 0x7f27bc136dd0 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492599927 0 0) 0x7f27a80034e0 con 0x7f27bc136dd0 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 msgr2=0x7f27bc0987b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 --2- 172.21.15.17:0/1043135177 >> v2:172.21.15.17:3300/0 conn(0x7f27bc132040 0x7f27bc0987b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 >> v1:172.21.15.17:6789/0 conn(0x7f27bc0a58b0 legacy=0x7f27bc0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: _finish_hunting 0 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:12:21.159 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc132440 con 0x7f27bc136dd0 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f27b4001bb0 con 0x7f27bc136dd0 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: dump: 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 >> v1:172.21.15.67:6789/0 conn(0x7f27bc136dd0 legacy=0x7f27bc1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.161 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27a8003a50 0x7f27a8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): picked mon.c con 0x7f27a8003a50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27a8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): picked mon.a con 0x7f27a8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a8005330 0x7f27a800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): picked mon.b con 0x7f27a8005330 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0a5c80 con 0x7f27a8004780 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.162 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a800c2d0 con 0x7f27a8005330 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a800c490 con 0x7f27a8003a50 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27baffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27a8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27a8003a50 0x7f27a8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request con 0x7f27a8004780 auth_method 0 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request con 0x7f27a8003a50 auth_method 0 2024-09-15T02:12:21.163 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a8005330 0x7f27a800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): get_auth_request con 0x7f27a8005330 auth_method 0 2024-09-15T02:12:21.164 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.165 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:21.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:21.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:21.168 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient(hunting): handle_auth_done global_id 244195 payload 293 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27a8003a50 msgr2=0x7f27a8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27a8003a50 0x7f27a8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a8005330 msgr2=0x7f27a800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a8005330 0x7f27a800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 monclient: found mon.a 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a80034e0 con 0x7f27a8004780 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:21.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bbfff640 1 -- 172.21.15.17:0/1043135177 reap_dead start 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27c1b9d640 5 monclient: authenticate success, global_id 244195 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27c1b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.147+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27a8003a50 0x7f27a8003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27b2ffd640 1 -- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 msgr2=0x7f27a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27a8004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9d83a91feba0c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a402add0 con 0x7f27a8004780 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: dump: 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.170 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27c1b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a402e070 con 0x7f27a8004780 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 1 -- 172.21.15.17:0/1043135177 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4034660 con 0x7f27a8004780 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27c1b9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:21.171 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27c1b9d640 4 set_mon_vals no callback set 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.150+0000 7f27baffd640 10 monclient: dump: 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.172 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 10 monclient: shutdown 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 -- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 msgr2=0x7f27a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27a8004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006fd0 tx=0x7f27a4008040 comp rx=0 tx=0).stop 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 -- 172.21.15.17:0/1043135177 shutdown_connections 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/1043135177 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27a8004bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.173 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 -- 172.21.15.17:0/1043135177 >> 172.21.15.17:0/1043135177 conn(0x7f27bc070ed0 msgr2=0x7f27bc0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 -- 172.21.15.17:0/1043135177 shutdown_connections 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 1 -- 172.21.15.17:0/1043135177 wait complete. 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.151+0000 7f27c1b9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient: monmap: 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.175 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 1 Processor -- start 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 1 -- start start 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient: init 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f27a8004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc12fbf0 0x7f27bc133a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.152+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc12fbf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc130060 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc11bd00 con 0x7f27a8004780 2024-09-15T02:12:21.176 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0a3770 con 0x7f27bc12fbf0 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc1359c0 con 0x7f27bc130060 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27c1b9d640 10 monclient(hunting): authenticate will time out at 2001.149046s 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:51076/0 (socket says 172.21.15.17:51076) 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:44254/0 (socket says 172.21.15.17:44254) 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 -- 172.21.15.17:0/2000560346 learned_addr learned my addr 172.21.15.17:0/2000560346 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request con 0x7f27a8004780 auth_method 0 2024-09-15T02:12:21.177 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): get_auth_request con 0x7f27bc130060 auth_method 0 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 msgr2=0x7f27bc141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 msgr2=0x7f27bc141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:21.178 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27b2ffd640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc12fbf0 0x7f27bc133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request con 0x7f27bc12fbf0 auth_method 0 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient(hunting): handle_auth_done global_id 214490 payload 824 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc12fbf0 msgr2=0x7f27bc133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc12fbf0 0x7f27bc133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 msgr2=0x7f27bc141320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 monclient: found mon.noname-c 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.153+0000 7f27bbfff640 1 -- 172.21.15.17:0/2000560346 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc098dc0 con 0x7f27bc130060 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27bbfff640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 msgr2=0x7f27bc133f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27bbfff640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27bc096b10 tx=0x7f27bc096df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4013600 con 0x7f27bc130060 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 10 monclient: dump: 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 10 monclient: _finish_auth 0 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27c1b9d640 5 monclient: authenticate success, global_id 214490 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4013780 con 0x7f27bc130060 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4013b40 con 0x7f27bc130060 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27c1b9d640 10 monclient: _renew_subs 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27c1b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:21.181 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc1360d0 con 0x7f27bc130060 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.155+0000 7f27ba7fc640 4 set_mon_vals no callback set 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27b3fff640 10 monclient: dump: 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 10 monclient: _renew_subs 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc142100 con 0x7f27bc130060 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f27b401b3b0 con 0x7f27bc130060 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.156+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc071340 con 0x7f27bc130060 2024-09-15T02:12:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.157+0000 7f27b3fff640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 0x7f279c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.157+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f27b404bef0 con 0x7f27bc130060 2024-09-15T02:12:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.157+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 0x7f279c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.157+0000 7f27bb7fe640 10 monclient: get_auth_request con 0x7f279c038740 auth_method 0 2024-09-15T02:12:21.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.159+0000 7f27bb7fe640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 0x7f279c03abf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f27ac001670 tx=0x7f27ac007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.161+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f27bc071340 con 0x7f27bc130060 2024-09-15T02:12:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.161+0000 7f27b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:21.185 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.161+0000 7f27b3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:21.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27c1b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:21.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27c1b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:21.391 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27c1b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:21.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f27bc071340 con 0x7f27bc130060 2024-09-15T02:12:21.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27b3fff640 1 -- 172.21.15.17:0/2000560346 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7f27bc071340 con 0x7f27bc130060 2024-09-15T02:12:21.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:21.392 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.390+0000 7f27b3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:21.392 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 msgr2=0x7f279c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 0x7f279c03abf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f27ac001670 tx=0x7f27ac007d50 comp rx=0 tx=0).stop 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 10 monclient: shutdown 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 msgr2=0x7f27bc133f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27bc096b10 tx=0x7f27bc096df0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 shutdown_connections 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f279c038740 0x7f279c03abf0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f27bc130060 0x7f27bc133f50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc12fbf0 0x7f27bc133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 --2- 172.21.15.17:0/2000560346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27a8004780 0x7f27bc141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:21.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 >> 172.21.15.17:0/2000560346 conn(0x7f27bc070ed0 msgr2=0x7f27bc09b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:21.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 shutdown_connections 2024-09-15T02:12:21.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:21.392+0000 7f27c1b9d640 1 -- 172.21.15.17:0/2000560346 wait complete. 2024-09-15T02:12:24.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.512+0000 7ff29319d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.512+0000 7ff29319d640 10 monclient: build_initial_monmap 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.512+0000 7ff29319d640 10 monclient: monmap: 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:24.513673+0000 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:24.513673+0000 2024-09-15T02:12:24.514 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:24.515 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.513+0000 7ff29319d640 1 Processor -- start 2024-09-15T02:12:24.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.513+0000 7ff29319d640 1 -- start start 2024-09-15T02:12:24.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.513+0000 7ff29319d640 10 monclient: init 2024-09-15T02:12:24.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff28c06e920 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 0x7ff28c06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-b con 0x7ff28c06ecf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff28c134b50 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c09aef0 con 0x7ff28c134b50 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c09d720 con 0x7ff28c06e920 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c06f600 con 0x7ff28c06ecf0 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29319d640 10 monclient(hunting): authenticate will time out at 2004.510245s 2024-09-15T02:12:24.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 0x7ff28c06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29173a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff28c134b50 0x7ff28c136f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:36116/0 (socket says 172.21.15.17:36116) 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff29173a640 1 -- 172.21.15.17:0/2357679360 learned_addr learned my addr 172.21.15.17:0/2357679360 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): get_auth_request con 0x7ff28c06ecf0 auth_method 0 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient(hunting): handle_auth_done global_id 214493 payload 274 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 1 -- 172.21.15.17:0/2357679360 >> v1:172.21.15.67:6789/0 conn(0x7ff28c06e920 legacy=0x7ff28c12fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 1 -- 172.21.15.17:0/2357679360 >> v1:172.21.15.17:6789/0 conn(0x7ff28c134b50 legacy=0x7ff28c136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:24.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:24.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 1 monclient: found mon.noname-b 2024-09-15T02:12:24.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:24.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.514+0000 7ff28bfff640 1 -- 172.21.15.17:0/2357679360 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c09b0c0 con 0x7ff28c06ecf0 2024-09-15T02:12:24.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28bfff640 1 -- 172.21.15.17:0/2357679360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 msgr2=0x7ff28c06f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:24.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28bfff640 1 --2- 172.21.15.17:0/2357679360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 0x7ff28c06f0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff280009880 tx=0x7ff2800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3c73c34eb7e532 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:24.524 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 1 -- 172.21.15.17:0/2357679360 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280032440 con 0x7ff28c06ecf0 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: dump: 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:24.525 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: _finish_auth 0 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 1 -- 172.21.15.17:0/2357679360 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2800325a0 con 0x7ff28c06ecf0 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 1 -- 172.21.15.17:0/2357679360 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280032940 con 0x7ff28c06ecf0 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff29319d640 5 monclient: authenticate success, global_id 214493 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff29319d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff29319d640 4 set_mon_vals no callback set 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.517+0000 7ff28b7fe640 10 monclient: dump: 2024-09-15T02:12:24.526 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 10 monclient: shutdown 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 -- 172.21.15.17:0/2357679360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 msgr2=0x7ff28c06f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:24.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 --2- 172.21.15.17:0/2357679360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 0x7ff28c06f0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff280009880 tx=0x7ff2800090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 -- 172.21.15.17:0/2357679360 shutdown_connections 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 --2- 172.21.15.17:0/2357679360 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c06ecf0 0x7ff28c06f0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 -- 172.21.15.17:0/2357679360 >> 172.21.15.17:0/2357679360 conn(0x7ff28c0a1000 msgr2=0x7ff28c0a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 -- 172.21.15.17:0/2357679360 shutdown_connections 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 1 -- 172.21.15.17:0/2357679360 wait complete. 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.518+0000 7ff29319d640 10 monclient: build_initial_monmap 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.519+0000 7ff29319d640 10 monclient: monmap: 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:24.528 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.519+0000 7ff29319d640 1 Processor -- start 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.519+0000 7ff29319d640 1 -- start start 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.519+0000 7ff29319d640 10 monclient: init 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-a con 0x7ff28c06e920 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ecf0 0x7ff28c13feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:24.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-b con 0x7ff28c06ecf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 0x7ff28c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): picked mon.noname-c con 0x7ff28c134b50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c06d490 con 0x7ff28c06e920 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c09fa80 con 0x7ff28c06ecf0 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c06cf10 con 0x7ff28c134b50 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29319d640 10 monclient(hunting): authenticate will time out at 2004.516151s 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 0x7ff28c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39952/0 (socket says 172.21.15.17:39952) 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 -- 172.21.15.17:0/2482797075 learned_addr learned my addr 172.21.15.17:0/2482797075 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 10 monclient(hunting): get_auth_request con 0x7ff28c06e920 auth_method 0 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): get_auth_request con 0x7ff28c134b50 auth_method 0 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ecf0 0x7ff28c13feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 msgr2=0x7ff28c13f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 msgr2=0x7ff28c13f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:24.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): get_auth_request con 0x7ff28c06ecf0 auth_method 0 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff290f39640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient(hunting): handle_auth_done global_id 214496 payload 824 2024-09-15T02:12:24.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ecf0 msgr2=0x7ff28c13feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ecf0 0x7ff28c13feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 msgr2=0x7ff28c13f970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 monclient: found mon.noname-c 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.520+0000 7ff29173a640 1 -- 172.21.15.17:0/2482797075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280009390 con 0x7ff28c134b50 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.522+0000 7ff29173a640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 msgr2=0x7ff28c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.522+0000 7ff29173a640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 0x7ff28c06b410 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff28400cb60 tx=0x7ff2840052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400a850 con 0x7ff28c134b50 2024-09-15T02:12:24.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: dump: 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: _finish_auth 0 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28400a9d0 con 0x7ff28c134b50 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 5 monclient: authenticate success, global_id 214496 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 10 monclient: _renew_subs 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c09bc10 con 0x7ff28c134b50 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400ad90 con 0x7ff28c134b50 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 10 monclient: dump: 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff28affd640 4 set_mon_vals no callback set 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 10 monclient: _renew_subs 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c117e90 con 0x7ff28c134b50 2024-09-15T02:12:24.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.523+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff284007300 con 0x7ff28c134b50 2024-09-15T02:12:24.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff29319d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:24.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff29319d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff29319d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c0724a0 con 0x7ff28c134b50 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff288ff9640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 0x7ff27803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff28404c710 con 0x7ff28c134b50 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff290f39640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 0x7ff27803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.524+0000 7ff290f39640 10 monclient: get_auth_request con 0x7ff278038790 auth_method 0 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.527+0000 7ff290f39640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 0x7ff27803ac40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff27c007ce0 tx=0x7ff27c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.528+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff2840075b0 con 0x7ff28c134b50 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.528+0000 7ff288ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:24.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.528+0000 7ff288ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:24.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.752+0000 7ff29319d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:24.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff29319d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:24.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff29319d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:24.754 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff28c067290 con 0x7ff28c134b50 2024-09-15T02:12:24.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff288ff9640 1 -- 172.21.15.17:0/2482797075 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1523 (secure 0 0 0) 0x7ff284010300 con 0x7ff28c134b50 2024-09-15T02:12:24.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff288ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:24.755 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.753+0000 7ff288ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:24.755 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:24.756 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 msgr2=0x7ff27803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 0x7ff27803ac40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff27c007ce0 tx=0x7ff27c009040 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 10 monclient: shutdown 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 msgr2=0x7ff28c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 0x7ff28c06b410 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff28400cb60 tx=0x7ff2840052b0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 shutdown_connections 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff278038790 0x7ff27803ac40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff28c134b50 0x7ff28c06b410 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ecf0 0x7ff28c13feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 --2- 172.21.15.17:0/2482797075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff28c06e920 0x7ff28c13f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:24.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 >> 172.21.15.17:0/2482797075 conn(0x7ff28c0a1000 msgr2=0x7ff28c0a0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:24.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 shutdown_connections 2024-09-15T02:12:24.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:24.755+0000 7ff29319d640 1 -- 172.21.15.17:0/2482797075 wait complete. 2024-09-15T02:12:27.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:12:27.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.874+0000 7f15fe79d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:27.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.874+0000 7f15fe79d640 10 monclient: build_initial_monmap 2024-09-15T02:12:27.876 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.874+0000 7f15fe79d640 10 monclient: monmap: 2024-09-15T02:12:27.876 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:27.876 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:27.875560+0000 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:27.875560+0000 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:27.877 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:27.878 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:27.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.875+0000 7f15fe79d640 1 Processor -- start 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.875+0000 7f15fe79d640 1 -- start start 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.875+0000 7f15fe79d640 10 monclient: init 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-a con 0x7f15f8067090 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:27.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-b con 0x7f15f8067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-c con 0x7f15f812e6a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f809cbc0 con 0x7f15f8067090 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f809e550 con 0x7f15f812e6a0 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f809e1e0 con 0x7f15f8067460 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fe79d640 10 monclient(hunting): authenticate will time out at 2007.872123s 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:27.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:52918/0 (socket says 172.21.15.17:52918) 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:39958/0 (socket says 172.21.15.17:39958) 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 learned_addr learned my addr 172.21.15.17:0/763684652 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15f8067460 auth_method 0 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:27.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15f8067090 auth_method 0 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:27.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15f812e6a0 auth_method 0 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:27.885 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:12:27.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:27.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:27.886 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:27.876+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:28.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.079 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15f8067090 auth_method 0 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15f8067460 auth_method 0 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:28.080 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15f812e6a0 auth_method 0 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:12:28.081 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.082 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.077+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T02:12:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15f8067460 auth_method 0 2024-09-15T02:12:28.480 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15f8067090 auth_method 0 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.481 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15f812e6a0 auth_method 0 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:28.482 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:28.478+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T02:12:29.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15f8067090 auth_method 0 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15f8067460 auth_method 0 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T02:12:29.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15f812e6a0 auth_method 0 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T02:12:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:29.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:29.279+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T02:12:30.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient(hunting): tick 2024-09-15T02:12:30.879 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 monclient(hunting): continuing hunt 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 msgr2=0x7f15f8067830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15f8067460 0x7f15f8067830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 msgr2=0x7f15f812eb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.67:3300/0 conn(0x7f15f812e6a0 0x7f15f812eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:30.880 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 -- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 msgr2=0x7f15f81321a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 --2- 172.21.15.17:0/763684652 >> v2:172.21.15.17:3300/0 conn(0x7f15f8067090 0x7f15f81321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient: _add_conns ranks=[0,1] 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15e0003650 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15e0007350 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 -- 172.21.15.17:0/763684652 --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f809cfa0 con 0x7f15e0003650 2024-09-15T02:12:30.881 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 1 -- 172.21.15.17:0/763684652 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e000c0c0 con 0x7f15e0007350 2024-09-15T02:12:30.882 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.876+0000 7f15f5ffb640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:30.883 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.878+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 legacy=0x7f15e0005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:30.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.879+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.67:6789/0 conn(0x7f15e0007350 legacy=0x7f15e0009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:30.884 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:30.879+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 DispatchQueue::entry inject delay of 0.002000 2024-09-15T02:12:31.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:31.130+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 legacy=0x7f15e0005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T02:12:31.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:31.130+0000 7f15f7fff640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 legacy=0x7f15e0005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:31.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:31.130+0000 7f15f7fff640 1 --1- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 0x7f15e0005a80 :-1 s=OPENED pgs=27 cs=1 l=1).handle_message read tag failed 2024-09-15T02:12:31.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:31.130+0000 7f15f7fff640 1 --1- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 0x7f15e0005a80 :-1 s=OPENED pgs=27 cs=1 l=1).fault on lossy channel, failing 2024-09-15T02:12:31.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:31.130+0000 7f15f6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.17:6789/0 2024-09-15T02:12:32.378 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.372+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639355209 0 0) 0x7f15e000c0c0 con 0x7f15e0007350 2024-09-15T02:12:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.372+0000 7f15f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.372+0000 7f15f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.372+0000 7f15f6ffd640 10 monclient(hunting): my global_id is 244204 2024-09-15T02:12:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.372+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e40034e0 con 0x7f15e0007350 2024-09-15T02:12:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744938381 0 0) 0x7f15e40034e0 con 0x7f15e0007350 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.17:6789/0 conn(0x7f15e0003650 legacy=0x7f15e0005a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.67:6789/0 2024-09-15T02:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e000d2c0 con 0x7f15e0007350 2024-09-15T02:12:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f15ec00c680 con 0x7f15e0007350 2024-09-15T02:12:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:32.381 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: dump: 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.382 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2024-09-15T02:12:32.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:32.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 >> v1:172.21.15.67:6789/0 conn(0x7f15e0007350 legacy=0x7f15e0009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:32.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:32.384 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 1 -- 172.21.15.17:0/763684652 reap_dead start 2024-09-15T02:12:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e40039d0 0x7f15e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): picked mon.a con 0x7f15e40039d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 0x7f15e4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): picked mon.b con 0x7f15e4004780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:32.385 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15e400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): picked mon.c con 0x7f15e4005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e400c2a0 con 0x7f15e40039d0 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e400c410 con 0x7f15e4004780 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e400c5d0 con 0x7f15e4005330 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15e400b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 0x7f15e4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.386 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15e4005330 auth_method 0 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15e4004780 auth_method 0 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15fcd3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:32.387 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f7fff640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e40039d0 0x7f15e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15e40039d0 auth_method 0 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 10 monclient(hunting): handle_auth_done global_id 244204 payload 293 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 msgr2=0x7f15e400b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15e400b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e40039d0 msgr2=0x7f15e4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.373+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e40039d0 0x7f15e4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 10 monclient: _finish_hunting 0 2024-09-15T02:12:32.388 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 1 monclient: found mon.b 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40034e0 con 0x7f15e4004780 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 10 monclient: _finish_auth 0 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15fe79d640 5 monclient: authenticate success, global_id 244204 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.374+0000 7f15fe79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.377+0000 7f15f77fe640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 msgr2=0x7f15e4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:32.389 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.377+0000 7f15f77fe640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 0x7f15e4004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006f60 tx=0x7f15ec008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f553afdcbd3bb46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec034410 con 0x7f15e4004780 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: dump: 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:32.395 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:32.396 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ec00c040 con 0x7f15e4004780 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 1 -- 172.21.15.17:0/763684652 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0345d0 con 0x7f15e4004780 2024-09-15T02:12:32.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15fe79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15fe79d640 4 set_mon_vals no callback set 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.378+0000 7f15f6ffd640 10 monclient: dump: 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:32.398 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:32.399 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:32.400 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:32.400 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 10 monclient: shutdown 2024-09-15T02:12:32.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:32.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 -- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 msgr2=0x7f15e4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 0x7f15e4004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006f60 tx=0x7f15ec008040 comp rx=0 tx=0).stop 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 -- 172.21.15.17:0/763684652 shutdown_connections 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15e400b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e4004780 0x7f15e4004bf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 --2- 172.21.15.17:0/763684652 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e40039d0 0x7f15e4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 -- 172.21.15.17:0/763684652 >> 172.21.15.17:0/763684652 conn(0x7f15f80a1ee0 msgr2=0x7f15f80a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 -- 172.21.15.17:0/763684652 shutdown_connections 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.379+0000 7f15fe79d640 1 -- 172.21.15.17:0/763684652 wait complete. 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.380+0000 7f15fe79d640 10 monclient: build_initial_monmap 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.380+0000 7f15fe79d640 10 monclient: monmap: 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:32.402 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:32.403 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:32.404 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:32.404 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.380+0000 7f15fe79d640 1 Processor -- start 2024-09-15T02:12:32.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 -- start start 2024-09-15T02:12:32.404 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient: init 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 0x7f15f811a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-b con 0x7f15e40039d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-a con 0x7f15e4004780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): picked mon.noname-c con 0x7f15e4005330 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f811d740 con 0x7f15e4004780 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f811e610 con 0x7f15e40039d0 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f811da20 con 0x7f15e4005330 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.381+0000 7f15fe79d640 10 monclient(hunting): authenticate will time out at 2012.377947s 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 0x7f15f811a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:40010/0 (socket says 172.21.15.17:40010) 2024-09-15T02:12:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 -- 172.21.15.17:0/1566404485 learned_addr learned my addr 172.21.15.17:0/1566404485 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15e4004780 auth_method 0 2024-09-15T02:12:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request con 0x7f15e4005330 auth_method 0 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:32.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15e40039d0 auth_method 0 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 msgr2=0x7f15f811ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 msgr2=0x7f15f811ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:32.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15fcd3a640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 msgr2=0x7f15f811a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 msgr2=0x7f15f811a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f77fe640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient(hunting): handle_auth_done global_id 244207 payload 824 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 msgr2=0x7f15f811ac90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 msgr2=0x7f15f811a750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 monclient: found mon.noname-b 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.382+0000 7f15f7fff640 1 -- 172.21.15.17:0/1566404485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f811b9b0 con 0x7f15e40039d0 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15f7fff640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 msgr2=0x7f15f811a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15f7fff640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 0x7f15f811a210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006ea0 tx=0x7f15e8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e800de60 con 0x7f15e40039d0 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:32.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: dump: 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:32.414 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: _finish_auth 0 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:32.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e8013ce0 con 0x7f15e40039d0 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8017650 con 0x7f15e40039d0 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15dffff640 10 monclient: dump: 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:32.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:32.417 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 5 monclient: authenticate success, global_id 244207 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 10 monclient: _renew_subs 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f811d460 con 0x7f15e40039d0 2024-09-15T02:12:32.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15f67fc640 4 set_mon_vals no callback set 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 10 monclient: _renew_subs 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.385+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f811d0e0 con 0x7f15e40039d0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f15e8017ac0 con 0x7f15e40039d0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15dffff640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 0x7f15e003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15fe79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15fe79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15fe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f811bdd0 con 0x7f15e40039d0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f15e804d590 con 0x7f15e40039d0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15f77fe640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 0x7f15e003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.386+0000 7f15f77fe640 10 monclient: get_auth_request con 0x7f15e0038fe0 auth_method 0 2024-09-15T02:12:32.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.389+0000 7f15f77fe640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 0x7f15e003b490 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f15ec008080 tx=0x7f15ec039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.391+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f15e8029550 con 0x7f15e40039d0 2024-09-15T02:12:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.391+0000 7f15dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:32.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.391+0000 7f15dffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.625+0000 7f15fe79d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.625+0000 7f15fe79d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.625+0000 7f15fe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.67:3300/0 2024-09-15T02:12:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.625+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f15f80724a0 con 0x7f15e40039d0 2024-09-15T02:12:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.627+0000 7f15dffff640 1 -- 172.21.15.17:0/1566404485 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1522 (secure 0 0 0) 0x7f15e80203b0 con 0x7f15e40039d0 2024-09-15T02:12:32.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.627+0000 7f15dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:32.628 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.627+0000 7f15dffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:32.628 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":210,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 msgr2=0x7f15e003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 0x7f15e003b490 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f15ec008080 tx=0x7f15ec039000 comp rx=0 tx=0).stop 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 10 monclient: shutdown 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 msgr2=0x7f15f811a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:32.630 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 0x7f15f811a210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006ea0 tx=0x7f15e8008040 comp rx=0 tx=0).stop 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 shutdown_connections 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f15e0038fe0 0x7f15e003b490 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f15e4004780 0x7f15f811a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f15e4005330 0x7f15f811ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 --2- 172.21.15.17:0/1566404485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e40039d0 0x7f15f811a210 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.628+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 >> 172.21.15.17:0/1566404485 conn(0x7f15f80a1ee0 msgr2=0x7f15f80980a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.629+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 shutdown_connections 2024-09-15T02:12:32.631 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:32.629+0000 7f15fe79d640 1 -- 172.21.15.17:0/1566404485 wait complete. 2024-09-15T02:12:35.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:12:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.749+0000 7fd593cd4640 10 monclient: get_monmap_and_config 2024-09-15T02:12:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.749+0000 7fd593cd4640 10 monclient: build_initial_monmap 2024-09-15T02:12:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.749+0000 7fd593cd4640 10 monclient: monmap: 2024-09-15T02:12:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:35.750616+0000 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:35.750616+0000 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:35.752 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.750+0000 7fd593cd4640 1 Processor -- start 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.750+0000 7fd593cd4640 1 -- start start 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.750+0000 7fd593cd4640 10 monclient: init 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd58c09c1b0 0x7fd58c09a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.757 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c09c1b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd58c09c580 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 0x7fd58c09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c09c9c0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58c09f790 con 0x7fd58c09c580 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c069d40 con 0x7fd58c09c1b0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c068b60 con 0x7fd58c09c9c0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd593cd4640 10 monclient(hunting): authenticate will time out at 2015.747222s 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd58c09c1b0 0x7fd58c09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd58c09c1b0 0x7fd58c09a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46852/0 (socket says 172.21.15.17:46852) 2024-09-15T02:12:35.758 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd59126f640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fd58c09c580 0x7fd58c06eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59420/0 (socket says 172.21.15.17:59420) 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 -- 172.21.15.17:0/1899494416 learned_addr learned my addr 172.21.15.17:0/1899494416 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): get_auth_request con 0x7fd58c09c1b0 auth_method 0 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd592271640 1 --2- 172.21.15.17:0/1899494416 >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 0x7fd58c09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient(hunting): handle_auth_done global_id 254100 payload 274 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 -- 172.21.15.17:0/1899494416 >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 msgr2=0x7fd58c09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 --2- 172.21.15.17:0/1899494416 >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 0x7fd58c09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.759 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 -- 172.21.15.17:0/1899494416 >> v1:172.21.15.17:6789/0 conn(0x7fd58c09c580 legacy=0x7fd58c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient: _finish_hunting 0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 monclient: found mon.noname-a 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd591a70640 1 -- 172.21.15.17:0/1899494416 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c069aa0 con 0x7fd58c09c1b0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd592271640 10 monclient: get_auth_request con 0x7fd58c09c9c0 auth_method 0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.751+0000 7fd592271640 1 --2- 172.21.15.17:0/1899494416 >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 0x7fd58c09e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd591a70640 1 -- 172.21.15.17:0/1899494416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c1b0 msgr2=0x7fd58c09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd591a70640 1 --2- 172.21.15.17:0/1899494416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c1b0 0x7fd58c09a990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06a750 tx=0x7fd5880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a0e87f437d2b9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 1 -- 172.21.15.17:0/1899494416 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588032440 con 0x7fd58c09c1b0 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:35.760 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: dump: 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: _finish_auth 0 2024-09-15T02:12:35.761 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 1 -- 172.21.15.17:0/1899494416 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5880325a0 con 0x7fd58c09c1b0 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd593cd4640 5 monclient: authenticate success, global_id 254100 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd593cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd593cd4640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 1 -- 172.21.15.17:0/1899494416 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588032940 con 0x7fd58c09c1b0 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd593cd4640 4 set_mon_vals no callback set 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.754+0000 7fd590a6e640 10 monclient: dump: 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:35.762 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 10 monclient: shutdown 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 -- 172.21.15.17:0/1899494416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c1b0 msgr2=0x7fd58c09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 --2- 172.21.15.17:0/1899494416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c1b0 0x7fd58c09a990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06a750 tx=0x7fd5880091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 -- 172.21.15.17:0/1899494416 shutdown_connections 2024-09-15T02:12:35.763 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 --2- 172.21.15.17:0/1899494416 >> v2:172.21.15.67:3300/0 conn(0x7fd58c09c9c0 0x7fd58c09e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 --2- 172.21.15.17:0/1899494416 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c1b0 0x7fd58c09a990 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 -- 172.21.15.17:0/1899494416 >> 172.21.15.17:0/1899494416 conn(0x7fd58c071300 msgr2=0x7fd58c0a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 -- 172.21.15.17:0/1899494416 shutdown_connections 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 1 -- 172.21.15.17:0/1899494416 wait complete. 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 10 monclient: build_initial_monmap 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.755+0000 7fd593cd4640 10 monclient: monmap: 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.764 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 Processor -- start 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 -- start start 2024-09-15T02:12:35.765 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient: init 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58c09c1b0 0x7fd58c141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c09c1b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c09c580 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 0x7fd58c11dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c09c9c0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:35.769 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c09cdb0 con 0x7fd58c09c580 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c0a31f0 con 0x7fd58c09c1b0 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c11faf0 con 0x7fd58c09c9c0 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd593cd4640 10 monclient(hunting): authenticate will time out at 2015.752785s 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 0x7fd58c11dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46866/0 (socket says 172.21.15.17:46866) 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 1 -- 172.21.15.17:0/2024068871 learned_addr learned my addr 172.21.15.17:0/2024068871 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 10 monclient(hunting): get_auth_request con 0x7fd58c09c580 auth_method 0 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 0x7fd58c11dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38690/0 (socket says 172.21.15.17:38690) 2024-09-15T02:12:35.770 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd59126f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 10 monclient(hunting): get_auth_request con 0x7fd58c09c9c0 auth_method 0 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd592271640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.756+0000 7fd591a70640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58c09c1b0 0x7fd58c141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): get_auth_request con 0x7fd58c09c1b0 auth_method 0 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:35.771 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd592271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd592271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd592271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd591a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient(hunting): handle_auth_done global_id 254103 payload 824 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 -- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 msgr2=0x7fd58c11dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 0x7fd58c11dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 -- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58c09c1b0 msgr2=0x7fd58c141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58c09c1b0 0x7fd58c141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient: _finish_hunting 0 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 monclient: found mon.noname-a 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.757+0000 7fd59126f640 1 -- 172.21.15.17:0/2024068871 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588009450 con 0x7fd58c09c580 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd59126f640 1 -- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 msgr2=0x7fd58c133ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd59126f640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c5b0 tx=0x7fd57c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c007510 con 0x7fd58c09c580 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: dump: 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:35.773 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c0057d0 con 0x7fd58c09c580 2024-09-15T02:12:35.774 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 5 monclient: authenticate success, global_id 254103 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 10 monclient: _renew_subs 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c120210 con 0x7fd58c09c580 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c007690 con 0x7fd58c09c580 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd581ffb640 10 monclient: dump: 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:35.775 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 10 monclient: _renew_subs 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd5837fe640 4 set_mon_vals no callback set 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.759+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c11f5e0 con 0x7fd58c09c580 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.760+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd57c01b400 con 0x7fd58c09c580 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.760+0000 7fd593cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:35.776 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.760+0000 7fd593cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd593cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c0724a0 con 0x7fd58c09c580 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd581ffb640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd57c04c0d0 con 0x7fd58c09c580 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd591a70640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.761+0000 7fd591a70640 10 monclient: get_auth_request con 0x7fd570038790 auth_method 0 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.764+0000 7fd591a70640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd58c060cf0 tx=0x7fd588032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.765+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd57c01a940 con 0x7fd58c09c580 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.765+0000 7fd581ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:35.777 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.765+0000 7fd581ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd593cd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd593cd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd593cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd58c067220 con 0x7fd58c09c580 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd581ffb640 1 -- 172.21.15.17:0/2024068871 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7fd57c01a360 con 0x7fd58c09c580 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd581ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:35.994 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.992+0000 7fd581ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:35.995 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:35.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 msgr2=0x7fd57003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:35.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd58c060cf0 tx=0x7fd588032000 comp rx=0 tx=0).stop 2024-09-15T02:12:35.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 10 monclient: shutdown 2024-09-15T02:12:35.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 msgr2=0x7fd58c133ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c5b0 tx=0x7fd57c007f00 comp rx=0 tx=0).stop 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 shutdown_connections 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd570038790 0x7fd57003ac40 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd58c09c580 0x7fd58c133ff0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.994+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd58c09c9c0 0x7fd58c11dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.995+0000 7fd593cd4640 1 --2- 172.21.15.17:0/2024068871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58c09c1b0 0x7fd58c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.995+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 >> 172.21.15.17:0/2024068871 conn(0x7fd58c071300 msgr2=0x7fd58c0a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.995+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 shutdown_connections 2024-09-15T02:12:35.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:35.995+0000 7fd593cd4640 1 -- 172.21.15.17:0/2024068871 wait complete. 2024-09-15T02:12:36.007 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:12:36.007 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T02:12:36.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T02:12:36.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.102+0000 7fe3c26c9640 10 monclient: get_monmap_and_config 2024-09-15T02:12:36.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.103+0000 7fe3c26c9640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.117 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.103+0000 7fe3c26c9640 10 monclient: monmap: 2024-09-15T02:12:36.117 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.117 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:36.103845+0000 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:36.103845+0000 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:36.118 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.103+0000 7fe3c26c9640 1 Processor -- start 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.103+0000 7fe3c26c9640 1 -- start start 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.103+0000 7fe3c26c9640 10 monclient: init 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3bc0a5780 0x7fe3bc09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe3bc0a5780 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3bc09ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe3bc069c90 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.119 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc0a5b50 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0a18b0 con 0x7fe3bc0a5780 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc09c5f0 con 0x7fe3bc069c90 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c26c9640 10 monclient(hunting): authenticate will time out at 2016.100320s 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38702/0 (socket says 172.21.15.17:38702) 2024-09-15T02:12:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bb7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe3bc09ccf0 0x7fe3bc069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59424/0 (socket says 172.21.15.17:59424) 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 -- 172.21.15.17:0/2564556190 learned_addr learned my addr 172.21.15.17:0/2564556190 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): get_auth_request con 0x7fe3bc069c90 auth_method 0 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/2564556190 >> v2:172.21.15.67:3300/0 conn(0x7fe3bc0a5780 0x7fe3bc09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request con 0x7fe3bc0a5780 auth_method 0 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.122 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_done global_id 214502 payload 274 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 -- 172.21.15.17:0/2564556190 >> v2:172.21.15.67:3300/0 conn(0x7fe3bc0a5780 msgr2=0x7fe3bc09bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 --2- 172.21.15.17:0/2564556190 >> v2:172.21.15.67:3300/0 conn(0x7fe3bc0a5780 0x7fe3bc09bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 -- 172.21.15.17:0/2564556190 >> v1:172.21.15.17:6789/0 conn(0x7fe3bc09ccf0 legacy=0x7fe3bc069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 monclient: found mon.noname-b 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.104+0000 7fe3c0c66640 1 -- 172.21.15.17:0/2564556190 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc09d210 con 0x7fe3bc069c90 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c0c66640 1 -- 172.21.15.17:0/2564556190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 msgr2=0x7fe3bc139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c0c66640 1 --2- 172.21.15.17:0/2564556190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0a19f0 tx=0x7fe3b4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=804e4cf92807fa21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 1 -- 172.21.15.17:0/2564556190 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4007610 con 0x7fe3bc069c90 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: dump: 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.124 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 1 -- 172.21.15.17:0/2564556190 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b402d050 con 0x7fe3bc069c90 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c26c9640 5 monclient: authenticate success, global_id 214502 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c26c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c26c9640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 1 -- 172.21.15.17:0/2564556190 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40079e0 con 0x7fe3bc069c90 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3c26c9640 4 set_mon_vals no callback set 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.107+0000 7fe3baffd640 10 monclient: dump: 2024-09-15T02:12:36.126 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 10 monclient: shutdown 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 -- 172.21.15.17:0/2564556190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 msgr2=0x7fe3bc139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/2564556190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0a19f0 tx=0x7fe3b4009840 comp rx=0 tx=0).stop 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 -- 172.21.15.17:0/2564556190 shutdown_connections 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/2564556190 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc069c90 0x7fe3bc139880 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/2564556190 >> v2:172.21.15.67:3300/0 conn(0x7fe3bc0a5780 0x7fe3bc09bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 -- 172.21.15.17:0/2564556190 >> 172.21.15.17:0/2564556190 conn(0x7fe3bc070e10 msgr2=0x7fe3bc0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 -- 172.21.15.17:0/2564556190 shutdown_connections 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 1 -- 172.21.15.17:0/2564556190 wait complete. 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.108+0000 7fe3c26c9640 10 monclient: monmap: 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.129 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 Processor -- start 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 -- start start 2024-09-15T02:12:36.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient: init 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3bc069c90 0x7fe3bc141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe3bc069c90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 0x7fe3bc131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe3bc09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc0a5780 0x7fe3bc11b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe3bc0a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1420b0 con 0x7fe3bc0a5780 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc11ec30 con 0x7fe3bc069c90 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0a0920 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c26c9640 10 monclient(hunting): authenticate will time out at 2016.105846s 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 0x7fe3bc131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc0a5780 0x7fe3bc11b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 0x7fe3bc131b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38716/0 (socket says 172.21.15.17:38716) 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc0a5780 0x7fe3bc11b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46874/0 (socket says 172.21.15.17:46874) 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 1 -- 172.21.15.17:0/3695832687 learned_addr learned my addr 172.21.15.17:0/3695832687 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 10 monclient(hunting): get_auth_request con 0x7fe3bc09ccf0 auth_method 0 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 10 monclient(hunting): get_auth_request con 0x7fe3bc0a5780 auth_method 0 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.109+0000 7fe3c0c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3c0c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3bc069c90 0x7fe3bc141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient(hunting): handle_auth_done global_id 214505 payload 824 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3bc069c90 msgr2=0x7fe3bc141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3bc069c90 0x7fe3bc141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc0a5780 msgr2=0x7fe3bc11b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc0a5780 0x7fe3bc11b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 monclient: found mon.noname-c 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bb7fe640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b4009ae0 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bbfff640 10 monclient: get_auth_request con 0x7fe3bc069c90 auth_method 0 2024-09-15T02:12:36.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.110+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3bc069c90 0x7fe3bc141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:36.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3bb7fe640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 msgr2=0x7fe3bc131b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3bb7fe640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 0x7fe3bc131b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c5e0 tx=0x7fe3b0007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0007510 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: dump: 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.136 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.137 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.138 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00057d0 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0007690 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 5 monclient: authenticate success, global_id 214505 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 10 monclient: _renew_subs 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc11d610 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3ba7fc640 4 set_mon_vals no callback set 2024-09-15T02:12:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3a3fff640 10 monclient: dump: 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 10 monclient: _renew_subs 2024-09-15T02:12:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.112+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc11d100 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3c26c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 0x7fe3bc11bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc063b20 con 0x7fe3bc132050 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe3b001b400 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 0x7fe3bc11bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient: get_auth_request con 0x7fe3bc132050 auth_method 0 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3a3fff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe3a8038790 0x7fe3a803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_done global_id 254109 payload 824 2024-09-15T02:12:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3bbfff640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc11dd90 con 0x7fe3bc132050 2024-09-15T02:12:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.113+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe3b004c0d0 con 0x7fe3bc09ccf0 2024-09-15T02:12:36.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.114+0000 7fe3c0c66640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe3a8038790 0x7fe3a803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.114+0000 7fe3c0c66640 10 monclient: get_auth_request con 0x7fe3a8038790 auth_method 0 2024-09-15T02:12:36.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3bbfff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 msgr2=0x7fe3bc11bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 0x7fe3bc11bc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac010920 tx=0x7fe3ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac00ecf0 con 0x7fe3bc132050 2024-09-15T02:12:36.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 10 monclient: dump: 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.147 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe3ac00ee50 con 0x7fe3bc132050 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 msgr2=0x7fe3bc11bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 0x7fe3bc11bc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac010920 tx=0x7fe3ac008040 comp rx=0 tx=0).stop 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 msgr2=0x7fe3bc11bc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3a3fff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc132050 0x7fe3bc11bc10 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac010920 tx=0x7fe3ac008040 comp rx=0 tx=0).stop 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.116+0000 7fe3c0c66640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe3a8038790 0x7fe3a803ac40 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4033000 tx=0x7fe3b4036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3c26c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:12:36.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 0x7fe3bc096aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3c26c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc132420 con 0x7fe3bc11c150 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 0x7fe3bc096aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient: get_auth_request con 0x7fe3bc11c150 auth_method 0 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_done global_id 254112 payload 824 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.122+0000 7fe3bbfff640 1 -- 172.21.15.17:0/3695832687 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3bc11c550 con 0x7fe3bc11c150 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3bbfff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 msgr2=0x7fe3bc096aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3bbfff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 0x7fe3bc096aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac01a000 tx=0x7fe3ac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac021070 con 0x7fe3bc11c150 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.124+0000 7fe3a3fff640 10 monclient: dump: 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.150 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 <== mon.0 v2:172.21.15.17:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4398 (secure 0 0 0) 0x7fe3ac014a70 con 0x7fe3bc11c150 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 msgr2=0x7fe3bc096aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.151 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 0x7fe3bc096aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac01a000 tx=0x7fe3ac01b000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 msgr2=0x7fe3bc096aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3a3fff640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe3bc11c150 0x7fe3bc096aa0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac01a000 tx=0x7fe3ac01b000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "leader", 2024-09-15T02:12:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 214, 2024-09-15T02:12:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:12:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:12:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:12:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:12:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:12:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:12:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:12:36.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:12:36.167 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe3a8038790 msgr2=0x7fe3a803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe3a8038790 0x7fe3a803ac40 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4033000 tx=0x7fe3b4036000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 10 monclient: shutdown 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 msgr2=0x7fe3bc131b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.169 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 --2- 172.21.15.17:0/3695832687 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe3bc09ccf0 0x7fe3bc131b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c5e0 tx=0x7fe3b0007f00 comp rx=0 tx=0).stop 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c0c66640 1 -- 172.21.15.17:0/3695832687 reap_dead start 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c0c66640 1 -- 172.21.15.17:0/3695832687 reap_dead start 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 shutdown_connections 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 >> 172.21.15.17:0/3695832687 conn(0x7fe3bc070e10 msgr2=0x7fe3bc09d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 shutdown_connections 2024-09-15T02:12:36.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.125+0000 7fe3c26c9640 1 -- 172.21.15.17:0/3695832687 wait complete. 2024-09-15T02:12:36.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:12:36.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T02:12:36.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T02:12:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 10 monclient: get_monmap_and_config 2024-09-15T02:12:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 10 monclient: monmap: 2024-09-15T02:12:36.267 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:36.266934+0000 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:36.266934+0000 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:36.268 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 1 Processor -- start 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 1 -- start start 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.266+0000 7ff2510dd640 10 monclient: init 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T02:12:36.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff24c0a58b0 0x7ff24c132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c0a58b0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff24c0980a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 0x7ff24c1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c136dd0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0a01e0 con 0x7ff24c0980a0 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c0a1580 con 0x7ff24c0a58b0 2024-09-15T02:12:36.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0a5c80 con 0x7ff24c136dd0 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff2510dd640 10 monclient(hunting): authenticate will time out at 2016.263224s 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 0x7ff24c1391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24a59c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff24c0980a0 0x7ff24c0984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59426/0 (socket says 172.21.15.17:59426) 2024-09-15T02:12:36.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24a59c640 1 -- 172.21.15.17:0/2409992196 learned_addr learned my addr 172.21.15.17:0/2409992196 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): get_auth_request con 0x7ff24c136dd0 auth_method 0 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24ad9d640 1 --2- 172.21.15.17:0/2409992196 >> v2:172.21.15.67:3300/0 conn(0x7ff24c0a58b0 0x7ff24c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request con 0x7ff24c0a58b0 auth_method 0 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient(hunting): handle_auth_done global_id 214508 payload 274 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 -- 172.21.15.17:0/2409992196 >> v2:172.21.15.67:3300/0 conn(0x7ff24c0a58b0 msgr2=0x7ff24c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 --2- 172.21.15.17:0/2409992196 >> v2:172.21.15.67:3300/0 conn(0x7ff24c0a58b0 0x7ff24c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 -- 172.21.15.17:0/2409992196 >> v1:172.21.15.17:6789/0 conn(0x7ff24c0980a0 legacy=0x7ff24c0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 monclient: found mon.noname-b 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.267+0000 7ff24b59e640 1 -- 172.21.15.17:0/2409992196 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c098bb0 con 0x7ff24c136dd0 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff24b59e640 1 -- 172.21.15.17:0/2409992196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 msgr2=0x7ff24c1391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff24b59e640 1 --2- 172.21.15.17:0/2409992196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 0x7ff24c1391b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff244009850 tx=0x7ff2440090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abdd47bfe9c9594 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 1 -- 172.21.15.17:0/2409992196 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24402ae80 con 0x7ff24c136dd0 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: dump: 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.278 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 1 -- 172.21.15.17:0/2409992196 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244032440 con 0x7ff24c136dd0 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 1 -- 172.21.15.17:0/2409992196 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440327c0 con 0x7ff24c136dd0 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff2510dd640 5 monclient: authenticate success, global_id 214508 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff2510dd640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff2510dd640 4 set_mon_vals no callback set 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.270+0000 7ff249d9b640 10 monclient: dump: 2024-09-15T02:12:36.279 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 10 monclient: shutdown 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 -- 172.21.15.17:0/2409992196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 msgr2=0x7ff24c1391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2409992196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 0x7ff24c1391b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff244009850 tx=0x7ff2440090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 -- 172.21.15.17:0/2409992196 shutdown_connections 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2409992196 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c136dd0 0x7ff24c1391b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2409992196 >> v2:172.21.15.67:3300/0 conn(0x7ff24c0a58b0 0x7ff24c132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 -- 172.21.15.17:0/2409992196 >> 172.21.15.17:0/2409992196 conn(0x7ff24c070ed0 msgr2=0x7ff24c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 -- 172.21.15.17:0/2409992196 shutdown_connections 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 1 -- 172.21.15.17:0/2409992196 wait complete. 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.271+0000 7ff2510dd640 10 monclient: monmap: 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 Processor -- start 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 -- start start 2024-09-15T02:12:36.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient: init 2024-09-15T02:12:36.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:12:36.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c0980a0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:36.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c0a58b0 0x7ff24c133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c0a58b0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c136dd0 0x7ff24c1335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c136dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c098dc0 con 0x7ff24c0980a0 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c139360 con 0x7ff24c136dd0 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c136390 con 0x7ff24c0a58b0 2024-09-15T02:12:36.290 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff2510dd640 10 monclient(hunting): authenticate will time out at 2016.268699s 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c0a58b0 0x7ff24c133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46906/0 (socket says 172.21.15.17:46906) 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): get_auth_request con 0x7ff24c0a58b0 auth_method 0 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 1 -- 172.21.15.17:0/2603746057 learned_addr learned my addr 172.21.15.17:0/2603746057 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request con 0x7ff24c0980a0 auth_method 0 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24b59e640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c136dd0 0x7ff24c1335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24b59e640 10 monclient(hunting): get_auth_request con 0x7ff24c136dd0 auth_method 0 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.272+0000 7ff24b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_done global_id 254115 payload 824 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c0a58b0 msgr2=0x7ff24c133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c0a58b0 0x7ff24c133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c136dd0 msgr2=0x7ff24c1335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c136dd0 0x7ff24c1335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 monclient: found mon.noname-a 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24ad9d640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244009390 con 0x7ff24c0980a0 2024-09-15T02:12:36.293 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.273+0000 7ff24a59c640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c0a58b0 0x7ff24c133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff24ad9d640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 msgr2=0x7ff24c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff24ad9d640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e190 tx=0x7ff23c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c013590 con 0x7ff24c0980a0 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: dump: 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.294 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c0136f0 con 0x7ff24c0980a0 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 5 monclient: authenticate success, global_id 254115 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 10 monclient: _renew_subs 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c134270 con 0x7ff24c0980a0 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c013a90 con 0x7ff24c0980a0 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.295 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff232ffd640 10 monclient: dump: 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 10 monclient: _renew_subs 2024-09-15T02:12:36.296 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff24959a640 4 set_mon_vals no callback set 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.275+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c131870 con 0x7ff24c0980a0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff2510dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 0x7ff24c1371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c136c00 con 0x7ff24c133ae0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff23c01e460 con 0x7ff24c0980a0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 0x7ff24c1371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient: get_auth_request con 0x7ff24c133ae0 auth_method 0 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.297 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff232ffd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff238038740 0x7ff23803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 10 monclient(hunting): handle_auth_done global_id 214514 payload 824 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff23c04ce00 con 0x7ff24c0980a0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24a59c640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c063ae0 con 0x7ff24c133ae0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24b59e640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff238038740 0x7ff23803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.276+0000 7ff24b59e640 10 monclient: get_auth_request con 0x7ff238038740 auth_method 0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.278+0000 7ff24a59c640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 msgr2=0x7ff24c1371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.278+0000 7ff24a59c640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 0x7ff24c1371a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff240006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24000ce80 con 0x7ff24c133ae0 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.298 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 10 monclient: dump: 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff240013470 con 0x7ff24c133ae0 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.299 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 msgr2=0x7ff24c1371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 0x7ff24c1371a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff240006f50 comp rx=0 tx=0).stop 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 msgr2=0x7ff24c1371a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff232ffd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c133ae0 0x7ff24c1371a0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff240006f50 comp rx=0 tx=0).stop 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.279+0000 7ff24b59e640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff238038740 0x7ff23803abf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff24c09f780 tx=0x7ff244032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff2510dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 0x7ff24c137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff2510dd640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c135fe0 con 0x7ff24c1376e0 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 0x7ff24c137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient: get_auth_request con 0x7ff24c1376e0 auth_method 0 2024-09-15T02:12:36.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 10 monclient(hunting): handle_auth_done global_id 214517 payload 824 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.285+0000 7ff24a59c640 1 -- 172.21.15.17:0/2603746057 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff24c0724a0 con 0x7ff24c1376e0 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff24a59c640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 msgr2=0x7ff24c137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff24a59c640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 0x7ff24c137ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff24001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240023070 con 0x7ff24c1376e0 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.301 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.287+0000 7ff232ffd640 10 monclient: dump: 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 <== mon.2 v2:172.21.15.17:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7ff240005e10 con 0x7ff24c1376e0 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:12:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 msgr2=0x7ff24c137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 0x7ff24c137ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff24001a000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 msgr2=0x7ff24c137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff232ffd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff24c1376e0 0x7ff24c137ae0 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb30 tx=0x7ff24001a000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:12:36.303 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 214, 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.304 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:12:36.306 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:12:36.307 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:12:36.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.310 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.314 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 5 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.315 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 4 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:12:36.316 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff238038740 msgr2=0x7ff23803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff238038740 0x7ff23803abf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff24c09f780 tx=0x7ff244032000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 10 monclient: shutdown 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 msgr2=0x7ff24c132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.317 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e190 tx=0x7ff23c009920 comp rx=0 tx=0).stop 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff24b59e640 1 -- 172.21.15.17:0/2603746057 reap_dead start 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 shutdown_connections 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 --2- 172.21.15.17:0/2603746057 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff24c0980a0 0x7ff24c132b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 >> 172.21.15.17:0/2603746057 conn(0x7ff24c070ed0 msgr2=0x7ff24c068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 shutdown_connections 2024-09-15T02:12:36.318 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.288+0000 7ff2510dd640 1 -- 172.21.15.17:0/2603746057 wait complete. 2024-09-15T02:12:36.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:12:36.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T02:12:36.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T02:12:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 10 monclient: get_monmap_and_config 2024-09-15T02:12:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 10 monclient: monmap: 2024-09-15T02:12:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:36.417008+0000 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:36.417008+0000 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:36.418 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 1 Processor -- start 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 1 -- start start 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.416+0000 7f7a80b09640 10 monclient: init 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a7c06eb30 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a7c09e970 0x7f7a7c09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7c09e970 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-b con 0x7f7a7c09c4d0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.421 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c09f6a0 con 0x7f7a7c06eb30 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c069f40 con 0x7f7a7c09e970 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c09ed40 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a80b09640 10 monclient(hunting): authenticate will time out at 2016.413493s 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38766/0 (socket says 172.21.15.17:38766) 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a7c09e970 0x7f7a7c09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/1010551949 learned_addr learned my addr 172.21.15.17:0/1010551949 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): get_auth_request con 0x7f7a7c09c4d0 auth_method 0 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): get_auth_request con 0x7f7a7c09e970 auth_method 0 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient(hunting): handle_auth_done global_id 214520 payload 274 2024-09-15T02:12:36.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/1010551949 >> v2:172.21.15.67:3300/0 conn(0x7f7a7c09e970 msgr2=0x7f7a7c09bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/1010551949 >> v2:172.21.15.67:3300/0 conn(0x7f7a7c09e970 0x7f7a7c09bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/1010551949 >> v1:172.21.15.67:6789/0 conn(0x7f7a7c06eb30 legacy=0x7f7a7c06ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 monclient: found mon.noname-b 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.417+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/1010551949 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c069950 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/1010551949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 msgr2=0x7f7a7c139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/1010551949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009f20 tx=0x7f7a74009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f863125681df9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 1 -- 172.21.15.17:0/1010551949 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74007610 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:36.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 10 monclient: dump: 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.433 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 1 -- 172.21.15.17:0/1010551949 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7402d050 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.420+0000 7f7a7959a640 1 -- 172.21.15.17:0/1010551949 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740079a0 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 5 monclient: authenticate success, global_id 214520 2024-09-15T02:12:36.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 4 set_mon_vals no callback set 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a7959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a7959a640 10 monclient: dump: 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 10 monclient: shutdown 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 -- 172.21.15.17:0/1010551949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 msgr2=0x7f7a7c139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 --2- 172.21.15.17:0/1010551949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009f20 tx=0x7f7a74009840 comp rx=0 tx=0).stop 2024-09-15T02:12:36.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 -- 172.21.15.17:0/1010551949 shutdown_connections 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 --2- 172.21.15.17:0/1010551949 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c09c4d0 0x7f7a7c139b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 --2- 172.21.15.17:0/1010551949 >> v2:172.21.15.67:3300/0 conn(0x7f7a7c09e970 0x7f7a7c09bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 -- 172.21.15.17:0/1010551949 >> 172.21.15.17:0/1010551949 conn(0x7f7a7c0712f0 msgr2=0x7f7a7c0a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 -- 172.21.15.17:0/1010551949 shutdown_connections 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.421+0000 7f7a80b09640 1 -- 172.21.15.17:0/1010551949 wait complete. 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient: monmap: 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.438 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 Processor -- start 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 -- start start 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient: init 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 0x7f7a7c134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7c06eb30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 0x7f7a7c133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-a con 0x7f7a7c09c4d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:36.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c09e970 0x7f7a7c11e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): picked mon.noname-b con 0x7f7a7c09e970 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c09cd80 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c122080 con 0x7f7a7c09e970 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c135410 con 0x7f7a7c06eb30 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a80b09640 10 monclient(hunting): authenticate will time out at 2016.418905s 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.422+0000 7f7a7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 0x7f7a7c134ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 0x7f7a7c133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 0x7f7a7c134ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38772/0 (socket says 172.21.15.17:38772) 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 1 -- 172.21.15.17:0/2586084070 learned_addr learned my addr 172.21.15.17:0/2586084070 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): get_auth_request con 0x7f7a7c06eb30 auth_method 0 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 0x7f7a7c133c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46908/0 (socket says 172.21.15.17:46908) 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): get_auth_request con 0x7f7a7c09c4d0 auth_method 0 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.441 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c09e970 0x7f7a7c11e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 10 monclient(hunting): get_auth_request con 0x7f7a7c09e970 auth_method 0 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient(hunting): handle_auth_done global_id 254118 payload 824 2024-09-15T02:12:36.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 msgr2=0x7f7a7c134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 0x7f7a7c134ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c09e970 msgr2=0x7f7a7c11e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c09e970 0x7f7a7c11e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 monclient: found mon.noname-a 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a79d9b640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74009ae0 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c09e970 0x7f7a7c11e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.423+0000 7f7a7a59c640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f7a7c06eb30 0x7f7a7c134ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a79d9b640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 msgr2=0x7f7a7c133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a79d9b640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 0x7f7a7c133c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000e960 tx=0x7f7a700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70009540 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: dump: 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.444 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 5 monclient: authenticate success, global_id 254118 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7000cb50 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 10 monclient: _renew_subs 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c141c20 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70013470 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:36.445 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a6a7fc640 10 monclient: dump: 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 10 monclient: _renew_subs 2024-09-15T02:12:36.446 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a78d99640 4 set_mon_vals no callback set 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.425+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c1341b0 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a70013860 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a80b09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a80b09640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 0x7f7a7c11f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.447 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c121f40 con 0x7f7a7c11ec70 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a6a7fc640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.426+0000 7f7a7a59c640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 0x7f7a7c11f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient: get_auth_request con 0x7f7a7c11ec70 auth_method 0 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f7a7004c900 con 0x7f7a7c09c4d0 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7ad9d640 10 monclient: get_auth_request con 0x7f7a5c038740 auth_method 0 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_done global_id 244228 payload 824 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.427+0000 7f7a7a59c640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a7c120300 con 0x7f7a7c11ec70 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a7a59c640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 msgr2=0x7f7a7c11f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a7a59c640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 0x7f7a7c11f040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a7ad9d640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a3e30 tx=0x7f7a74036000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c00cee0 con 0x7f7a7c11ec70 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.429+0000 7f7a6a7fc640 10 monclient: dump: 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.449 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7a6c0123f0 con 0x7f7a7c11ec70 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:36.450 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 msgr2=0x7f7a7c11f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 0x7f7a7c11f040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 msgr2=0x7f7a7c11f040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.430+0000 7f7a6a7fc640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11ec70 0x7f7a7c11f040 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a80b09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a80b09640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 0x7f7a7c096a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a80b09640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c063b90 con 0x7f7a7c11f580 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 0x7f7a7c096a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient: get_auth_request con 0x7f7a7c11f580 auth_method 0 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.436+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.437+0000 7f7a7a59c640 10 monclient(hunting): handle_auth_done global_id 244231 payload 824 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.437+0000 7f7a7a59c640 1 -- 172.21.15.17:0/2586084070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a7c122480 con 0x7f7a7c11f580 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a7a59c640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 msgr2=0x7f7a7c096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a7a59c640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 0x7f7a7c096a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c01cd10 con 0x7f7a7c11f580 2024-09-15T02:12:36.454 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.439+0000 7f7a6a7fc640 10 monclient: dump: 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.455 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 <== mon.1 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f7a6c020430 con 0x7f7a7c11f580 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 msgr2=0x7f7a7c096a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 0x7f7a7c096a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c015000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 msgr2=0x7f7a7c096a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:36.456 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a6a7fc640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a7c11f580 0x7f7a7c096a10 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00eb30 tx=0x7f7a6c015000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.457 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "state": "peon", 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "election_epoch": 214, 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum": [ 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_age": 2, 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.458 INFO:teuthology.orchestra.run.smithi017.stdout: "required_con": "2449958755906961412", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "required_mon": [ 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_con": "4540138320759226367", 2024-09-15T02:12:36.459 INFO:teuthology.orchestra.run.smithi017.stdout: "quorum_mon": [ 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.460 INFO:teuthology.orchestra.run.smithi017.stdout: "outside_quorum": [], 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "extra_probe_peers": [], 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_provider": [], 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": { 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "e3089478-c52a-4174-94a2-258b1876359d", 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-15T01:50:53.865793Z", 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release": 17, 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "min_mon_release_name": "quincy", 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "election_strategy": 1, 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "disallowed_leaders: ": "", 2024-09-15T02:12:36.461 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false, 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "tiebreaker_mon": "", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_ranks: ": "", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "features": { 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "persistent": [ 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "kraken", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "luminous", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "mimic", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "osdmap-prune", 2024-09-15T02:12:36.462 INFO:teuthology.orchestra.run.smithi017.stdout: "nautilus", 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "octopus", 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "pacific", 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "elector-pinging", 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "quincy" 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "optional": [] 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mons": [ 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 0, 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "a", 2024-09-15T02:12:36.463 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3300", 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789", 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6789/0", 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 1, 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "b", 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.465 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:3300", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.67:6789/0", 2024-09-15T02:12:36.466 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "rank": 2, 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "c", 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:3301", 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790", 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 0 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6790/0", 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "priority": 0, 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "{}" 2024-09-15T02:12:36.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "feature_map": { 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": [ 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": [ 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.469 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "client": [ 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 2 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": [ 2024-09-15T02:12:36.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "release": "luminous", 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "num": 1 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": false 2024-09-15T02:12:36.471 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 msgr2=0x7f7a5c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f7a5c038740 0x7f7a5c03abf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0a3e30 tx=0x7f7a74036000 comp rx=0 tx=0).stop 2024-09-15T02:12:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 10 monclient: shutdown 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 msgr2=0x7f7a7c133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 --2- 172.21.15.17:0/2586084070 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a7c09c4d0 0x7f7a7c133c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000e960 tx=0x7f7a700090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/2586084070 reap_dead start 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a7ad9d640 1 -- 172.21.15.17:0/2586084070 reap_dead start 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 shutdown_connections 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 >> 172.21.15.17:0/2586084070 conn(0x7f7a7c0712f0 msgr2=0x7f7a7c0a1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 shutdown_connections 2024-09-15T02:12:36.473 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.440+0000 7f7a80b09640 1 -- 172.21.15.17:0/2586084070 wait complete. 2024-09-15T02:12:36.473 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T02:12:36.474 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T02:12:36.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.568+0000 7fde75ca8640 10 monclient: get_monmap_and_config 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.568+0000 7fde75ca8640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.568+0000 7fde75ca8640 10 monclient: monmap: 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:36.569484+0000 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:36.569484+0000 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.570 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:36.571 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:36.571 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:36.571 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:36.571 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:36.571 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 1 Processor -- start 2024-09-15T02:12:36.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 1 -- start start 2024-09-15T02:12:36.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 10 monclient: init 2024-09-15T02:12:36.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.569+0000 7fde75ca8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde7006e920 0x7fde7012fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-c con 0x7fde7006e920 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-b con 0x7fde7006ecf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde70096c50 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7009aef0 con 0x7fde70096c50 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7006f600 con 0x7fde7006e920 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7009d720 con 0x7fde7006ecf0 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde75ca8640 10 monclient(hunting): authenticate will time out at 2016.566088s 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38788/0 (socket says 172.21.15.17:38788) 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 -- 172.21.15.17:0/3162617864 learned_addr learned my addr 172.21.15.17:0/3162617864 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): get_auth_request con 0x7fde7006ecf0 auth_method 0 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6f7fe640 1 --2- 172.21.15.17:0/3162617864 >> v2:172.21.15.67:3300/0 conn(0x7fde7006e920 0x7fde7012fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6f7fe640 10 monclient(hunting): get_auth_request con 0x7fde7006e920 auth_method 0 2024-09-15T02:12:36.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient(hunting): handle_auth_done global_id 214526 payload 274 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 -- 172.21.15.17:0/3162617864 >> v2:172.21.15.67:3300/0 conn(0x7fde7006e920 msgr2=0x7fde7012fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 --2- 172.21.15.17:0/3162617864 >> v2:172.21.15.67:3300/0 conn(0x7fde7006e920 0x7fde7012fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 -- 172.21.15.17:0/3162617864 >> v1:172.21.15.67:6789/0 conn(0x7fde70096c50 legacy=0x7fde700970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 monclient: found mon.noname-b 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.570+0000 7fde6effd640 1 -- 172.21.15.17:0/3162617864 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7009b0c0 con 0x7fde7006ecf0 2024-09-15T02:12:36.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6effd640 1 -- 172.21.15.17:0/3162617864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 msgr2=0x7fde7006f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6effd640 1 --2- 172.21.15.17:0/3162617864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde64009910 tx=0x7fde640091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9070d3367cb450f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 1 -- 172.21.15.17:0/3162617864 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64032440 con 0x7fde7006ecf0 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: dump: 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.582 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 1 -- 172.21.15.17:0/3162617864 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde640325a0 con 0x7fde7006ecf0 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 1 -- 172.21.15.17:0/3162617864 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64032980 con 0x7fde7006ecf0 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde75ca8640 5 monclient: authenticate success, global_id 214526 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde75ca8640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde75ca8640 4 set_mon_vals no callback set 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.573+0000 7fde6e7fc640 10 monclient: dump: 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 10 monclient: shutdown 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 -- 172.21.15.17:0/3162617864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 msgr2=0x7fde7006f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 --2- 172.21.15.17:0/3162617864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde64009910 tx=0x7fde640091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 -- 172.21.15.17:0/3162617864 shutdown_connections 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 --2- 172.21.15.17:0/3162617864 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde7006ecf0 0x7fde7006f0c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 --2- 172.21.15.17:0/3162617864 >> v2:172.21.15.67:3300/0 conn(0x7fde7006e920 0x7fde7012fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 -- 172.21.15.17:0/3162617864 >> 172.21.15.17:0/3162617864 conn(0x7fde700a1000 msgr2=0x7fde700a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 -- 172.21.15.17:0/3162617864 shutdown_connections 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 1 -- 172.21.15.17:0/3162617864 wait complete. 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 10 monclient: build_initial_monmap 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.574+0000 7fde75ca8640 10 monclient: monmap: 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 Processor -- start 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 -- start start 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient: init 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-a con 0x7fde7006e920 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7006ecf0 0x7fde7012ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.586 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-b con 0x7fde7006ecf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 0x7fde7013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): picked mon.noname-c con 0x7fde70096c50 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7009bbb0 con 0x7fde7006e920 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7011c480 con 0x7fde7006ecf0 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7011c110 con 0x7fde70096c50 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde75ca8640 10 monclient(hunting): authenticate will time out at 2016.571651s 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 0x7fde7013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46924/0 (socket says 172.21.15.17:46924) 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 1 -- 172.21.15.17:0/1145072428 learned_addr learned my addr 172.21.15.17:0/1145072428 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): get_auth_request con 0x7fde7006e920 auth_method 0 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): get_auth_request con 0x7fde70096c50 auth_method 0 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6effd640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7006ecf0 0x7fde7012ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6effd640 10 monclient(hunting): get_auth_request con 0x7fde7006ecf0 auth_method 0 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.575+0000 7fde6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 10 monclient(hunting): handle_auth_done global_id 214529 payload 824 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 -- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7006ecf0 msgr2=0x7fde7012ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7006ecf0 0x7fde7012ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.589 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 -- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 msgr2=0x7fde7013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 monclient: found mon.noname-c 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6ffff640 1 -- 172.21.15.17:0/1145072428 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009450 con 0x7fde70096c50 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.576+0000 7fde6f7fe640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde6ffff640 1 -- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 msgr2=0x7fde7013f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde6ffff640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 0x7fde7013f180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fde6800c7e0 tx=0x7fde680025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800a850 con 0x7fde70096c50 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:36.590 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: dump: 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: _finish_auth 0 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:36.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 5 monclient: authenticate success, global_id 214529 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6800a9d0 con 0x7fde70096c50 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 10 monclient: _renew_subs 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800ad90 con 0x7fde70096c50 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7011a370 con 0x7fde70096c50 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde577fe640 10 monclient: dump: 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:36.592 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde6dffb640 4 set_mon_vals no callback set 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 10 monclient: _renew_subs 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.578+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7013c6e0 con 0x7fde70096c50 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde68012630 con 0x7fde70096c50 2024-09-15T02:12:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde75ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde75ca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde75ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde700724a0 con 0x7fde70096c50 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde577fe640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 0x7fde5c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.579+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fde6804c940 con 0x7fde70096c50 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.580+0000 7fde6f7fe640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 0x7fde5c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:36.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.580+0000 7fde6f7fe640 10 monclient: get_auth_request con 0x7fde5c038790 auth_method 0 2024-09-15T02:12:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.582+0000 7fde6f7fe640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 0x7fde5c03ac40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fde60010370 tx=0x7fde60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.584+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fde68022d70 con 0x7fde70096c50 2024-09-15T02:12:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.584+0000 7fde577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:36.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.584+0000 7fde577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.810+0000 7fde75ca8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T02:12:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.810+0000 7fde75ca8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T02:12:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.810+0000 7fde75ca8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:36.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.810+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fde70067290 con 0x7fde70096c50 2024-09-15T02:12:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.812+0000 7fde577fe640 1 -- 172.21.15.17:0/1145072428 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fde68016020 con 0x7fde70096c50 2024-09-15T02:12:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.812+0000 7fde577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T02:12:36.813 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.812+0000 7fde577fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:36.815 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.813+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 msgr2=0x7fde5c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 0x7fde5c03ac40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fde60010370 tx=0x7fde60008040 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 10 monclient: shutdown 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 msgr2=0x7fde7013f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 0x7fde7013f180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fde6800c7e0 tx=0x7fde680025b0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 shutdown_connections 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fde5c038790 0x7fde5c03ac40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fde70096c50 0x7fde7013f180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7006ecf0 0x7fde7012ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 --2- 172.21.15.17:0/1145072428 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde7006e920 0x7fde7013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 >> 172.21.15.17:0/1145072428 conn(0x7fde700a1000 msgr2=0x7fde701328b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:36.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 shutdown_connections 2024-09-15T02:12:36.817 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:36.814+0000 7fde75ca8640 1 -- 172.21.15.17:0/1145072428 wait complete. 2024-09-15T02:12:36.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=8,logm=54} crc {auth=2952431323,config=1228700967,health=1492728775,kv=1529054990,logm=522352671}) 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3844768077,config=3140514009,health=3513420105,kv=1529054990,logm=1708457122}) 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=8,logm=54} crc {auth=2952431323,config=1228700967,health=1492728775,kv=1529054990,logm=522352671}) 2024-09-15T02:12:36.836 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.835+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=1,health=12,kv=8,logm=52} crc {auth=3844768077,config=1228700967,health=3513420105,kv=1529054990,logm=630379946}) 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419781314}) 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368107461}) 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419781314}) 2024-09-15T02:12:36.843 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.841+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506761535}) 2024-09-15T02:12:36.932 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253289813}) 2024-09-15T02:12:36.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2234802387}) 2024-09-15T02:12:36.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253289813}) 2024-09-15T02:12:36.933 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.931+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500202450}) 2024-09-15T02:12:36.934 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.934 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240631182}) 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993725036}) 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240631182}) 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.932+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766295240}) 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445087751}) 2024-09-15T02:12:36.935 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714981201}) 2024-09-15T02:12:36.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445087751}) 2024-09-15T02:12:36.936 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714596949}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005613342}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238416679}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005613342}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.934+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710779446}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905289980}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140213325}) 2024-09-15T02:12:36.937 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905289980}) 2024-09-15T02:12:36.938 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.935+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306438429}) 2024-09-15T02:12:36.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379494564}) 2024-09-15T02:12:36.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927798734}) 2024-09-15T02:12:36.946 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379494564}) 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.936+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097879826}) 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388178918}) 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013558864}) 2024-09-15T02:12:36.947 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388178918}) 2024-09-15T02:12:36.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291201476}) 2024-09-15T02:12:36.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489253994}) 2024-09-15T02:12:36.948 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149162931}) 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489253994}) 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.937+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2070490669}) 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423911674}) 2024-09-15T02:12:36.949 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=449592413}) 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423911674}) 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.938+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522015604}) 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750207005}) 2024-09-15T02:12:36.950 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075122495}) 2024-09-15T02:12:36.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750207005}) 2024-09-15T02:12:36.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224188334}) 2024-09-15T02:12:36.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.951 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031938181}) 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4003078303}) 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031938181}) 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.939+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=353888559}) 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.952 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387633132}) 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121521556}) 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387633132}) 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.940+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1028305363}) 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.953 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265527151}) 2024-09-15T02:12:36.954 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4000563804}) 2024-09-15T02:12:36.954 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.954 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265527151}) 2024-09-15T02:12:36.954 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526919832}) 2024-09-15T02:12:36.954 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817014384}) 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933678228}) 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817014384}) 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.941+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2363231787}) 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.955 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687896184}) 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964595977}) 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687896184}) 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.942+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423471448}) 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.956 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650502850}) 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1783671707}) 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650502850}) 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463344092}) 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.957 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210192658}) 2024-09-15T02:12:36.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037275616}) 2024-09-15T02:12:36.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210192658}) 2024-09-15T02:12:36.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.943+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483553811}) 2024-09-15T02:12:36.958 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552555056}) 2024-09-15T02:12:36.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2064319750}) 2024-09-15T02:12:36.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552555056}) 2024-09-15T02:12:36.959 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.944+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792556296}) 2024-09-15T02:12:36.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881090048}) 2024-09-15T02:12:36.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964924010}) 2024-09-15T02:12:36.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.971 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881090048}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.945+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670012550}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167276835}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219795621}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167276835}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.946+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1124860367}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724966666}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563230946}) 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.972 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724966666}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.947+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=5688130}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295818294}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58456938}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295818294}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.954+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498859639}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=587243191}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497285229}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=587243191}) 2024-09-15T02:12:36.973 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118136611}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483395016}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365749536}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.955+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483395016}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868593570}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367190845}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2265622017}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367190845}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.956+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421800091}) 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.974 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29347713}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2751225490}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29347713}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.957+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3654543827}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945583621}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838774152}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945583621}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3249822338}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936060246}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313178020}) 2024-09-15T02:12:36.975 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936060246}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.958+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294713164}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401574665}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609706941}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401574665}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.959+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2620311916}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871842120}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165730858}) 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.976 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871842120}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1532798014}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214303033}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401964100}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214303033}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.960+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241126061}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460794744}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111888537}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460794744}) 2024-09-15T02:12:36.977 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.961+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178081736}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974599711}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2145344901}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974599711}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=119622379}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092590517}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264785765}) 2024-09-15T02:12:36.978 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092590517}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.962+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298824029}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680187109}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360445808}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680187109}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.963+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610840267}) 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.979 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289375610}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385170391}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289375610}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212445212}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027532103}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413328361}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027532103}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.964+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029989521}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648206565}) 2024-09-15T02:12:36.980 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560765595}) 2024-09-15T02:12:36.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:36.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648206565}) 2024-09-15T02:12:36.981 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:36.965+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053582411}) 2024-09-15T02:12:37.365 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.365 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167535398}) 2024-09-15T02:12:37.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546480062}) 2024-09-15T02:12:37.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167535398}) 2024-09-15T02:12:37.366 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034486483}) 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657890398}) 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210989624}) 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657890398}) 2024-09-15T02:12:37.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.579+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185897274}) 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921801680}) 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148941777}) 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921801680}) 2024-09-15T02:12:37.668 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.666+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696336066}) 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002323168}) 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=117714607}) 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002323168}) 2024-09-15T02:12:37.767 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.765+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2745220740}) 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150077403}) 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229185733}) 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150077403}) 2024-09-15T02:12:37.768 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.766+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547313238}) 2024-09-15T02:12:37.826 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T02:12:37.826 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T02:12:37.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=543954013}) 2024-09-15T02:12:37.839 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948721102}) 2024-09-15T02:12:37.840 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.840 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=543954013}) 2024-09-15T02:12:37.840 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.838+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=948842645}) 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.842+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.842+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039631086}) 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.842+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276776295}) 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.842+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.843+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039631086}) 2024-09-15T02:12:37.844 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.843+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842702769}) 2024-09-15T02:12:37.856 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.856 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026164637}) 2024-09-15T02:12:37.856 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994344419}) 2024-09-15T02:12:37.856 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:37.856 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026164637}) 2024-09-15T02:12:37.857 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:37.855+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359186858}) 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.936+0000 7ff9ce4a8640 10 monclient: get_monmap_and_config 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.936+0000 7ff9ce4a8640 10 monclient: build_initial_monmap 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.936+0000 7ff9ce4a8640 10 monclient: monmap: 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:37.937730+0000 2024-09-15T02:12:37.938 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:37.937730+0000 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:37.939 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.937+0000 7ff9ce4a8640 1 Processor -- start 2024-09-15T02:12:37.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.937+0000 7ff9ce4a8640 1 -- start start 2024-09-15T02:12:37.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.937+0000 7ff9ce4a8640 10 monclient: init 2024-09-15T02:12:37.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9c80a5780 0x7ff9c809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c80a5780 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9c809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9c8069c90 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80a5b50 con 0x7ff9c8069c90 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c809c5f0 con 0x7ff9c809ccf0 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c80a18b0 con 0x7ff9c80a5780 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9ce4a8640 10 monclient(hunting): authenticate will time out at 2017.934313s 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9c80a5780 0x7ff9c809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:37.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9cca45640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff9c8069c90 0x7ff9c8139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59428/0 (socket says 172.21.15.17:59428) 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9cca45640 1 -- 172.21.15.17:0/2210092927 learned_addr learned my addr 172.21.15.17:0/2210092927 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): get_auth_request con 0x7ff9c80a5780 auth_method 0 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_done global_id 254124 payload 274 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 1 -- 172.21.15.17:0/2210092927 >> v1:172.21.15.67:6789/0 conn(0x7ff9c809ccf0 legacy=0x7ff9c8069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:37.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 1 -- 172.21.15.17:0/2210092927 >> v1:172.21.15.17:6789/0 conn(0x7ff9c8069c90 legacy=0x7ff9c8139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:37.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:37.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 1 monclient: found mon.noname-a 2024-09-15T02:12:37.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:37.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:37.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.938+0000 7ff9c7fff640 1 -- 172.21.15.17:0/2210092927 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c809d210 con 0x7ff9c80a5780 2024-09-15T02:12:37.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c7fff640 1 -- 172.21.15.17:0/2210092927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c80a5780 msgr2=0x7ff9c809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:37.949 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c7fff640 1 --2- 172.21.15.17:0/2210092927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c80a5780 0x7ff9c809bc90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009850 tx=0x7ff9b80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95f600de3bbb962 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 1 -- 172.21.15.17:0/2210092927 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8032440 con 0x7ff9c80a5780 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: dump: 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:37.950 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 1 -- 172.21.15.17:0/2210092927 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b80325a0 con 0x7ff9c80a5780 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9ce4a8640 5 monclient: authenticate success, global_id 254124 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9ce4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9ce4a8640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:37.951 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9ce4a8640 4 set_mon_vals no callback set 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 1 -- 172.21.15.17:0/2210092927 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8032940 con 0x7ff9c80a5780 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.941+0000 7ff9c6ffd640 10 monclient: dump: 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.952 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 10 monclient: shutdown 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/2210092927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c80a5780 msgr2=0x7ff9c809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/2210092927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c80a5780 0x7ff9c809bc90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009850 tx=0x7ff9b80090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/2210092927 shutdown_connections 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/2210092927 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c80a5780 0x7ff9c809bc90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/2210092927 >> 172.21.15.17:0/2210092927 conn(0x7ff9c8070e10 msgr2=0x7ff9c80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/2210092927 shutdown_connections 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/2210092927 wait complete. 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 10 monclient: build_initial_monmap 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.942+0000 7ff9ce4a8640 10 monclient: monmap: 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:37.954 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 Processor -- start 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 -- start start 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient: init 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c8069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 0x7ff9c813d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c809ccf0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 0x7ff9c8141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c80a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:37.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8120f20 con 0x7ff9c8069c90 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c813ed10 con 0x7ff9c809ccf0 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8122c10 con 0x7ff9c80a5780 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.943+0000 7ff9ce4a8640 10 monclient(hunting): authenticate will time out at 2017.939984s 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:46938/0 (socket says 172.21.15.17:46938) 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 -- 172.21.15.17:0/1467211627 learned_addr learned my addr 172.21.15.17:0/1467211627 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): get_auth_request con 0x7ff9c8069c90 auth_method 0 2024-09-15T02:12:37.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 0x7ff9c8141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): get_auth_request con 0x7ff9c80a5780 auth_method 0 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c77fe640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 0x7ff9c813d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient(hunting): handle_auth_done global_id 254127 payload 824 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 -- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 msgr2=0x7ff9c8141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 0x7ff9c8141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 -- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 msgr2=0x7ff9c813d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 0x7ff9c813d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 monclient: found mon.noname-a 2024-09-15T02:12:37.958 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c7fff640 1 -- 172.21.15.17:0/1467211627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8009390 con 0x7ff9c8069c90 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c77fe640 10 monclient: get_auth_request con 0x7ff9c809ccf0 auth_method 0 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9cca45640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 0x7ff9c8141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.944+0000 7ff9c77fe640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 0x7ff9c813d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9c7fff640 1 -- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 msgr2=0x7ff9c813e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9c7fff640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009360 tx=0x7ff9b8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80318c0 con 0x7ff9c8069c90 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9affff640 10 monclient: dump: 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.946+0000 7ff9affff640 10 monclient: _finish_auth 0 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:37.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b8031a20 con 0x7ff9c8069c90 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8031de0 con 0x7ff9c8069c90 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 5 monclient: authenticate success, global_id 254127 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 10 monclient: _renew_subs 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8122930 con 0x7ff9c8069c90 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 10 monclient: dump: 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:37.961 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9c67fc640 4 set_mon_vals no callback set 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 10 monclient: _renew_subs 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c813ddd0 con 0x7ff9c8069c90 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.947+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff9b8039040 con 0x7ff9c8069c90 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9ce4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9ce4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9ce4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9affff640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 0x7ff9b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c813dfc0 con 0x7ff9c8069c90 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff9b806c8c0 con 0x7ff9c8069c90 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9c77fe640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 0x7ff9b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.948+0000 7ff9c77fe640 10 monclient: get_auth_request con 0x7ff9b4038740 auth_method 0 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.951+0000 7ff9c77fe640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 0x7ff9b403abf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9c806a690 tx=0x7ff9bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.952+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff9b803f020 con 0x7ff9c8069c90 2024-09-15T02:12:37.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.952+0000 7ff9affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:37.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:37.952+0000 7ff9affff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9ce4a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9ce4a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9ce4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9c813dfc0 con 0x7ff9c8069c90 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9affff640 1 -- 172.21.15.17:0/1467211627 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1528 (secure 0 0 0) 0x7ff9b8037070 con 0x7ff9c8069c90 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T02:12:38.184 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.182+0000 7ff9affff640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:38.185 INFO:teuthology.orchestra.run.smithi017.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","modified":"2024-09-15T01:50:53.865793Z","created":"2024-09-15T01:50:53.865793Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T02:12:38.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 msgr2=0x7ff9b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 0x7ff9b403abf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9c806a690 tx=0x7ff9bc008040 comp rx=0 tx=0).stop 2024-09-15T02:12:38.186 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 10 monclient: shutdown 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 msgr2=0x7ff9c813e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009360 tx=0x7ff9b8032000 comp rx=0 tx=0).stop 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 shutdown_connections 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff9b4038740 0x7ff9b403abf0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9c80a5780 0x7ff9c8141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c809ccf0 0x7ff9c813d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 --2- 172.21.15.17:0/1467211627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9c8069c90 0x7ff9c813e1e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.184+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 >> 172.21.15.17:0/1467211627 conn(0x7ff9c8070e10 msgr2=0x7ff9c8098da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.185+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 shutdown_connections 2024-09-15T02:12:38.187 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.185+0000 7ff9ce4a8640 1 -- 172.21.15.17:0/1467211627 wait complete. 2024-09-15T02:12:38.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T02:12:38.196 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T02:12:38.211 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T02:12:38.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:12:38.346 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.331+0000 7effe7cd4640 10 monclient: get_monmap_and_config 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.331+0000 7effe7cd4640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.331+0000 7effe7cd4640 10 monclient: monmap: 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:38.332096+0000 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:38.332096+0000 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.347 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.331+0000 7effe7cd4640 1 Processor -- start 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 -- start start 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient: init 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effe0097320 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:38.348 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7effe009a590 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effe009a960 0x7effe0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-c con 0x7effe009a960 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe00985d0 con 0x7effe0097320 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe009d200 con 0x7effe009a960 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe00992f0 con 0x7effe009a590 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe7cd4640 10 monclient(hunting): authenticate will time out at 2018.328532s 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38818/0 (socket says 172.21.15.17:38818) 2024-09-15T02:12:38.349 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 1 -- 172.21.15.17:0/3045862215 learned_addr learned my addr 172.21.15.17:0/3045862215 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): get_auth_request con 0x7effe009a590 auth_method 0 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe6271640 1 --2- 172.21.15.17:0/3045862215 >> v2:172.21.15.67:3300/0 conn(0x7effe009a960 0x7effe0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe6271640 10 monclient(hunting): get_auth_request con 0x7effe009a960 auth_method 0 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe6271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe6271640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe6271640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.350 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.332+0000 7effe526f640 10 monclient(hunting): handle_auth_done global_id 214535 payload 274 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 1 -- 172.21.15.17:0/3045862215 >> v2:172.21.15.67:3300/0 conn(0x7effe009a960 msgr2=0x7effe0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 1 --2- 172.21.15.17:0/3045862215 >> v2:172.21.15.67:3300/0 conn(0x7effe009a960 0x7effe0135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 1 -- 172.21.15.17:0/3045862215 >> v1:172.21.15.67:6789/0 conn(0x7effe0097320 legacy=0x7effe00976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 1 monclient: found mon.noname-b 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.333+0000 7effe526f640 1 -- 172.21.15.17:0/3045862215 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe009cb90 con 0x7effe009a590 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.335+0000 7effe526f640 1 -- 172.21.15.17:0/3045862215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 msgr2=0x7effe01339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.335+0000 7effe526f640 1 --2- 172.21.15.17:0/3045862215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effd000bbc0 tx=0x7effd000b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac243fc64f5afd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 1 -- 172.21.15.17:0/3045862215 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0030050 con 0x7effe009a590 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:38.351 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: dump: 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.352 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 1 -- 172.21.15.17:0/3045862215 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0032710 con 0x7effe009a590 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 1 -- 172.21.15.17:0/3045862215 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0032ab0 con 0x7effe009a590 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 5 monclient: authenticate success, global_id 214535 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 4 set_mon_vals no callback set 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.353 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe4a6e640 10 monclient: dump: 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.354 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 10 monclient: shutdown 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 -- 172.21.15.17:0/3045862215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 msgr2=0x7effe01339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 --2- 172.21.15.17:0/3045862215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effd000bbc0 tx=0x7effd000b460 comp rx=0 tx=0).stop 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 -- 172.21.15.17:0/3045862215 shutdown_connections 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 --2- 172.21.15.17:0/3045862215 >> v2:172.21.15.67:3300/0 conn(0x7effe009a960 0x7effe0135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 --2- 172.21.15.17:0/3045862215 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a590 0x7effe01339d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 -- 172.21.15.17:0/3045862215 >> 172.21.15.17:0/3045862215 conn(0x7effe00a1000 msgr2=0x7effe00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 -- 172.21.15.17:0/3045862215 shutdown_connections 2024-09-15T02:12:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.336+0000 7effe7cd4640 1 -- 172.21.15.17:0/3045862215 wait complete. 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient: monmap: 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.356 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 1 Processor -- start 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 1 -- start start 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient: init 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 0x7effe012e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-a con 0x7effe0097320 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 0x7effe011b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.357 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7effe009a590 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.337+0000 7effe7cd4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): picked mon.noname-c con 0x7effe009a960 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe009ad50 con 0x7effe0097320 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe006ef10 con 0x7effe009a590 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe012fbe0 con 0x7effe009a960 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe7cd4640 10 monclient(hunting): authenticate will time out at 2018.334035s 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 0x7effe012e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.358 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38820/0 (socket says 172.21.15.17:38820) 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 -- 172.21.15.17:0/509098404 learned_addr learned my addr 172.21.15.17:0/509098404 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): get_auth_request con 0x7effe009a960 auth_method 0 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): get_auth_request con 0x7effe0097320 auth_method 0 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.359 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 0x7effe011b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 10 monclient(hunting): get_auth_request con 0x7effe009a590 auth_method 0 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient(hunting): handle_auth_done global_id 214538 payload 824 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 -- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 msgr2=0x7effe011b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.360 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 0x7effe011b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 -- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 msgr2=0x7effe012e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 0x7effe012e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 monclient: found mon.noname-c 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe6271640 1 -- 172.21.15.17:0/509098404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd000b700 con 0x7effe009a960 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe526f640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 0x7effe011b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.338+0000 7effe5a70640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 0x7effe012e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe6271640 1 -- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 msgr2=0x7effe011b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe6271640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7effd8009fd0 tx=0x7effd8009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd800ee90 con 0x7effe009a960 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: dump: 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.362 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8013590 con 0x7effe009a960 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe7cd4640 5 monclient: authenticate success, global_id 214538 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe7cd4640 10 monclient: _renew_subs 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe011e630 con 0x7effe009a960 2024-09-15T02:12:38.363 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8013950 con 0x7effe009a960 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd5ffb640 10 monclient: dump: 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.364 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.341+0000 7effe7cd4640 10 monclient: _renew_subs 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.341+0000 7effe7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.365 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.341+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe011e2d0 con 0x7effe009a960 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.340+0000 7effd7fff640 4 set_mon_vals no callback set 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.341+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7effd8013d00 con 0x7effe009a960 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe7cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe7cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe00724a0 con 0x7effe009a960 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effd5ffb640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 0x7effc403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7effd804cde0 con 0x7effe009a960 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe5a70640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 0x7effc403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.342+0000 7effe5a70640 10 monclient: get_auth_request con 0x7effc4038740 auth_method 0 2024-09-15T02:12:38.366 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.345+0000 7effe5a70640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 0x7effc403abf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7effdc007fd0 tx=0x7effdc007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.346+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7effd80128e0 con 0x7effe009a960 2024-09-15T02:12:38.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.346+0000 7effd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.367 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.346+0000 7effd5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278716490}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466418694}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278716490}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.361+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918294898}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934772685}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568318519}) 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.371 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934772685}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.362+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=859334849}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278671665}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827885019}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278671665}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105746432}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679837658}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713810909}) 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.372 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679837658}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.363+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907466869}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15565209}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1334444641}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15565209}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.364+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266818097}) 2024-09-15T02:12:38.373 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324226618}) 2024-09-15T02:12:38.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2844749042}) 2024-09-15T02:12:38.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.374 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324226618}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.365+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2179733314}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933222712}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205894351}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933222712}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835699682}) 2024-09-15T02:12:38.375 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006818115}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820151335}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006818115}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.366+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742481394}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813002436}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300439712}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813002436}) 2024-09-15T02:12:38.376 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.367+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512609700}) 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354156762}) 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144499591}) 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354156762}) 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.368+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813647245}) 2024-09-15T02:12:38.377 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.378 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692959239}) 2024-09-15T02:12:38.378 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711676727}) 2024-09-15T02:12:38.378 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.378 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692959239}) 2024-09-15T02:12:38.378 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.369+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123490915}) 2024-09-15T02:12:38.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131733154}) 2024-09-15T02:12:38.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527566787}) 2024-09-15T02:12:38.383 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131733154}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557351398}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030810542}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1981113197}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030810542}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.370+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4249398856}) 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.384 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219302594}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977582769}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219302594}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.371+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928700112}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246698280}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690938010}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246698280}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.372+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3265200214}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123585837}) 2024-09-15T02:12:38.385 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3092152733}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123585837}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.373+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183027542}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242983548}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752666035}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242983548}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479010660}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697897770}) 2024-09-15T02:12:38.386 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093945677}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697897770}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.374+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367295345}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290319901}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15209214}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290319901}) 2024-09-15T02:12:38.387 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.375+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942625593}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006433020}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3633721015}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006433020}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044458498}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123217722}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383615574}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123217722}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.376+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306799215}) 2024-09-15T02:12:38.388 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535397875}) 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774257665}) 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535397875}) 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.378+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883584909}) 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040892200}) 2024-09-15T02:12:38.389 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521333844}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040892200}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354261461}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377791318}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=891711187}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377791318}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.379+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926165228}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295602408}) 2024-09-15T02:12:38.390 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3191492941}) 2024-09-15T02:12:38.391 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.391 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295602408}) 2024-09-15T02:12:38.391 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.380+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684236570}) 2024-09-15T02:12:38.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740954434}) 2024-09-15T02:12:38.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=868399872}) 2024-09-15T02:12:38.399 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740954434}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.387+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781146719}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751388953}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3843138653}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751388953}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.388+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226859921}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468820837}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=897732282}) 2024-09-15T02:12:38.400 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468820837}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.389+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840609629}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017551397}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481092943}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017551397}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372077068}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915473019}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904941943}) 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.401 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915473019}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.390+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581221118}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836567400}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333182789}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836567400}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.391+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1339809934}) 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.402 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686632759}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941232348}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686632759}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.392+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409993814}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504992993}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4237269328}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504992993}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.393+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2747749562}) 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.403 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444422422}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807183760}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444422422}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293300359}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283385977}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.394+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1257693289}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283385977}) 2024-09-15T02:12:38.404 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1491472281}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2418779263}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345735576}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2418779263}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.395+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635846072}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616470148}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162488201}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616470148}) 2024-09-15T02:12:38.405 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.396+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286472118}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515985175}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282363488}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515985175}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2859940584}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008193724}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.397+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231194268}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008193724}) 2024-09-15T02:12:38.406 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227390052}) 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824969998}) 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196439086}) 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824969998}) 2024-09-15T02:12:38.408 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.398+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431115182}) 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647692467}) 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956699424}) 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647692467}) 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.399+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122034351}) 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.409 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445101906}) 2024-09-15T02:12:38.410 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337669589}) 2024-09-15T02:12:38.410 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.410 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445101906}) 2024-09-15T02:12:38.410 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.400+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48150161}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257430590}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3402546409}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257430590}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.407+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552838403}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862427325}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997286904}) 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.413 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862427325}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.408+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464815618}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362046246}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926698779}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362046246}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.409+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353616009}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996823387}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919051230}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996823387}) 2024-09-15T02:12:38.414 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074648642}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628803140}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2848195423}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628803140}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.410+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271902585}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122671636}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=33576794}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122671636}) 2024-09-15T02:12:38.415 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.411+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711309760}) 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534376944}) 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763448973}) 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534376944}) 2024-09-15T02:12:38.418 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.412+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024062048}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513050936}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124904379}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513050936}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774895592}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608555819}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094022807}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608555819}) 2024-09-15T02:12:38.419 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.413+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206138019}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468719162}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709778440}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468719162}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.414+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=353532157}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264765333}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224528796}) 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.420 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264765333}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849383243}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479287472}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054898459}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479287472}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.415+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=793978458}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67599533}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073416738}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67599533}) 2024-09-15T02:12:38.421 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.416+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157351439}) 2024-09-15T02:12:38.423 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.423 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966693991}) 2024-09-15T02:12:38.423 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170448396}) 2024-09-15T02:12:38.423 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.423 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966693991}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.417+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421013661}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867526680}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4001544047}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867526680}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231668393}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872369211}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.418+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3278825525}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872369211}) 2024-09-15T02:12:38.424 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2412954617}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850474135}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16040406}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850474135}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.419+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1233204397}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806404081}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192315964}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806404081}) 2024-09-15T02:12:38.425 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.420+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162086301}) 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153026599}) 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173009105}) 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153026599}) 2024-09-15T02:12:38.432 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.430+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3502411173}) 2024-09-15T02:12:38.433 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.433 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279345768}) 2024-09-15T02:12:38.433 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1241778269}) 2024-09-15T02:12:38.433 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.433 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279345768}) 2024-09-15T02:12:38.434 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.431+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573900820}) 2024-09-15T02:12:38.437 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.437 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217426342}) 2024-09-15T02:12:38.437 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023487593}) 2024-09-15T02:12:38.437 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.437 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217426342}) 2024-09-15T02:12:38.438 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.432+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967102649}) 2024-09-15T02:12:38.438 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.438 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105921959}) 2024-09-15T02:12:38.438 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=449436701}) 2024-09-15T02:12:38.438 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.439 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105921959}) 2024-09-15T02:12:38.439 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108527053}) 2024-09-15T02:12:38.439 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.439 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456238845}) 2024-09-15T02:12:38.439 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=802149649}) 2024-09-15T02:12:38.440 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.440 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456238845}) 2024-09-15T02:12:38.440 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.433+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247248996}) 2024-09-15T02:12:38.440 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.440 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670229627}) 2024-09-15T02:12:38.441 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2669808081}) 2024-09-15T02:12:38.441 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.441 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670229627}) 2024-09-15T02:12:38.441 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.434+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216204217}) 2024-09-15T02:12:38.443 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.443 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {logm=3498044579,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3120698008,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1136440199,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3961210460}) 2024-09-15T02:12:38.443 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=1343732602,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=2525985551,mon_config_key=1583035879,monmap=3178194974,osd_metadata=3375641772,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=1499582695}) 2024-09-15T02:12:38.443 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.444 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {logm=3498044579,mdsmap=1531151339,mgr=584503845,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=3120698008,mon_config_key=1583035879,monmap=3178194974,osd_metadata=1136440199,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=3961210460}) 2024-09-15T02:12:38.444 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.436+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=47} crc {logm=1343732602,mdsmap=1531151339,mgr=4173831893,mgr_command_descs=3545737492,mgr_metadata=2029235354,mgrstat=328851731,mon_config_key=1583035880,monmap=3178194974,osd_metadata=1802323575,osd_pg_creating=1863806571,osd_snap=1063953654,osdmap=8288244}) 2024-09-15T02:12:38.444 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.440+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T02:12:38.444 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.440+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=1089334901}) 2024-09-15T02:12:38.444 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-15T02:12:38.440+0000 7fc851ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=1133394043}) 2024-09-15T02:12:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.526+0000 7effe7cd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:38.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.526+0000 7effe7cd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.526+0000 7effe7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.526+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effe009cfe0 con 0x7effe009a960 2024-09-15T02:12:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.527+0000 7effd5ffb640 1 -- 172.21.15.17:0/509098404 <== mon.2 v2:172.21.15.17:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+20446 (secure 0 0 0) 0x7effd801b310 con 0x7effe009a960 2024-09-15T02:12:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.527+0000 7effd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:38.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.527+0000 7effd5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:38.529 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:38.529 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":257,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T02:00:51.417741+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:55:54.795952+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":181,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T01:51:28.688141+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T01:51:29.355809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-15T01:56:08.635533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-15T01:56:09.758237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7b19bad9-2bed-4846-b592-7009ac3d12e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.593725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.627101+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.602809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.641423+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.589740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.592626+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.2","mappings":[{"from":0,"to":1}]},{"pgid":"9.5","mappings":[{"from":0,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 msgr2=0x7effc403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 0x7effc403abf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7effdc007fd0 tx=0x7effdc007e10 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 10 monclient: shutdown 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 msgr2=0x7effe011b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7effd8009fd0 tx=0x7effd8009900 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 shutdown_connections 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7effc4038740 0x7effc403abf0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7effe009a960 0x7effe011b550 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effe009a590 0x7effe011b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 --2- 172.21.15.17:0/509098404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7effe0097320 0x7effe012e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 >> 172.21.15.17:0/509098404 conn(0x7effe00a1000 msgr2=0x7effe006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 shutdown_connections 2024-09-15T02:12:38.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.529+0000 7effe7cd4640 1 -- 172.21.15.17:0/509098404 wait complete. 2024-09-15T02:12:38.541 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:12:38.542 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:12:38.637 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.636+0000 7f4846d5d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:38.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.636+0000 7f4846d5d640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.638 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.636+0000 7f4846d5d640 10 monclient: monmap: 2024-09-15T02:12:38.638 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:38.637290+0000 2024-09-15T02:12:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:38.637290+0000 2024-09-15T02:12:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:38.640 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.651 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 Processor -- start 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 -- start start 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient: init 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 0x7f48401321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4840068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48400692f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4840136fe0 0x7f48401393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4840136fe0 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48400a1950 con 0x7f48400692f0 2024-09-15T02:12:38.652 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4840069dd0 con 0x7f4840136fe0 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48400a1670 con 0x7f4840068f20 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4846d5d640 10 monclient(hunting): authenticate will time out at 2018.633906s 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.637+0000 7f4844af9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 0x7f48401321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f483ffff640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f48400692f0 0x7f48400696c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59440/0 (socket says 172.21.15.17:59440) 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f4840136fe0 0x7f48401393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f483ffff640 1 -- 172.21.15.17:0/1593377317 learned_addr learned my addr 172.21.15.17:0/1593377317 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): get_auth_request con 0x7f4840068f20 auth_method 0 2024-09-15T02:12:38.653 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): get_auth_request con 0x7f4840136fe0 auth_method 0 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient(hunting): handle_auth_done global_id 254133 payload 274 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 -- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 msgr2=0x7f48401321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 0x7f48401321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 -- 172.21.15.17:0/1593377317 >> v1:172.21.15.17:6789/0 conn(0x7f48400692f0 legacy=0x7f48400696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 monclient: found mon.noname-a 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f48452fa640 1 -- 172.21.15.17:0/1593377317 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48400a0020 con 0x7f4840136fe0 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.638+0000 7f4844af9640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 0x7f48401321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f48452fa640 1 -- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840136fe0 msgr2=0x7f48401393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f48452fa640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840136fe0 0x7f48401393c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4828009850 tx=0x7f48280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104b77d25421acb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 1 -- 172.21.15.17:0/1593377317 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482802ae80 con 0x7f4840136fe0 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: dump: 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.656 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 5 monclient: authenticate success, global_id 254133 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 1 -- 172.21.15.17:0/1593377317 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4828032440 con 0x7f4840136fe0 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 4 set_mon_vals no callback set 2024-09-15T02:12:38.657 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 1 -- 172.21.15.17:0/1593377317 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48280327c0 con 0x7f4840136fe0 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f483f7fe640 10 monclient: dump: 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.658 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 10 monclient: shutdown 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 -- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840136fe0 msgr2=0x7f48401393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840136fe0 0x7f48401393c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4828009850 tx=0x7f48280090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 -- 172.21.15.17:0/1593377317 shutdown_connections 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840136fe0 0x7f48401393c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 --2- 172.21.15.17:0/1593377317 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840068f20 0x7f48401321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 -- 172.21.15.17:0/1593377317 >> 172.21.15.17:0/1593377317 conn(0x7f4840070fb0 msgr2=0x7f48400a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 -- 172.21.15.17:0/1593377317 shutdown_connections 2024-09-15T02:12:38.659 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.641+0000 7f4846d5d640 1 -- 172.21.15.17:0/1593377317 wait complete. 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.642+0000 7f4846d5d640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.642+0000 7f4846d5d640 10 monclient: monmap: 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:38.660 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.642+0000 7f4846d5d640 1 Processor -- start 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.642+0000 7f4846d5d640 1 -- start start 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.642+0000 7f4846d5d640 10 monclient: init 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4840068f20 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 0x7f4840141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f48400692f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 0x7f4840141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4840136fe0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484011ba10 con 0x7f4840068f20 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484011dc40 con 0x7f48400692f0 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48400a0750 con 0x7f4840136fe0 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4846d5d640 10 monclient(hunting): authenticate will time out at 2018.639393s 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.662 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 0x7f4840141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47006/0 (socket says 172.21.15.17:47006) 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 1 -- 172.21.15.17:0/3700601015 learned_addr learned my addr 172.21.15.17:0/3700601015 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): get_auth_request con 0x7f4840068f20 auth_method 0 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): get_auth_request con 0x7f4840136fe0 auth_method 0 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.663 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 0x7f4840141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 10 monclient(hunting): get_auth_request con 0x7f48400692f0 auth_method 0 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient(hunting): handle_auth_done global_id 214544 payload 824 2024-09-15T02:12:38.664 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 -- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 msgr2=0x7f4840141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 0x7f4840141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 -- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 msgr2=0x7f4840141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 monclient: found mon.noname-c 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f48452fa640 1 -- 172.21.15.17:0/3700601015 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838009390 con 0x7f4840136fe0 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f4844af9640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.643+0000 7f483ffff640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 0x7f4840141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f48452fa640 1 -- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 msgr2=0x7f4840141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f48452fa640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 0x7f4840141ed0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4828031000 tx=0x7f4828032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828031e90 con 0x7f4840136fe0 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: dump: 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.666 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 5 monclient: authenticate success, global_id 214544 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482803e450 con 0x7f4840136fe0 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 10 monclient: _renew_subs 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4828009390 con 0x7f4840136fe0 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482803e7f0 con 0x7f4840136fe0 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:38.667 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483cff9640 10 monclient: dump: 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 10 monclient: _renew_subs 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f483effd640 4 set_mon_vals no callback set 2024-09-15T02:12:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.646+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484009bb70 con 0x7f4840136fe0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4828046400 con 0x7f4840136fe0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f483cff9640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 0x7f482003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f482806c740 con 0x7f4840136fe0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4846d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4846d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4846d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48400724a0 con 0x7f4840136fe0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4844af9640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 0x7f482003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.647+0000 7f4844af9640 10 monclient: get_auth_request con 0x7f4820038740 auth_method 0 2024-09-15T02:12:38.669 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.650+0000 7f4844af9640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 0x7f482003abf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4838011640 tx=0x7f48380090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.652+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4828042020 con 0x7f4840136fe0 2024-09-15T02:12:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.652+0000 7f483cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.670 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.652+0000 7f483cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:38.812 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.810+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48400a4490 con 0x7f4820038740 2024-09-15T02:12:38.816 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.814+0000 7f483cff9640 1 -- 172.21.15.17:0/3700601015 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98323 (secure 0 0 0) 0x7f48400a4490 con 0x7f4820038740 2024-09-15T02:12:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:38.818 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 msgr2=0x7f482003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 0x7f482003abf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4838011640 tx=0x7f48380090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 10 monclient: shutdown 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 msgr2=0x7f4840141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 0x7f4840141ed0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4828031000 tx=0x7f4828032000 comp rx=0 tx=0).stop 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 shutdown_connections 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4820038740 0x7f482003abf0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f4840136fe0 0x7f4840141ed0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.820 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48400692f0 0x7f4840141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 --2- 172.21.15.17:0/3700601015 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4840068f20 0x7f4840141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 >> 172.21.15.17:0/3700601015 conn(0x7f4840070fb0 msgr2=0x7f484009dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 shutdown_connections 2024-09-15T02:12:38.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.817+0000 7f4846d5d640 1 -- 172.21.15.17:0/3700601015 wait complete. 2024-09-15T02:12:38.829 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-15T02:12:37.578853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812796,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418244,"statfs":{"total":579820584960,"available":577964281856,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":172,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421220+0000","last_change":"2024-09-15T01:58:09.930154+0000","last_active":"2024-09-15T02:00:50.421220+0000","last_peered":"2024-09-15T02:00:50.421220+0000","last_clean":"2024-09-15T02:00:50.421220+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:00:50.421220+0000","last_undegraded":"2024-09-15T02:00:50.421220+0000","last_fullsized":"2024-09-15T02:00:50.421220+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:15:30.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066273+0000","last_change":"2024-09-15T01:58:09.929368+0000","last_active":"2024-09-15T02:00:51.066273+0000","last_peered":"2024-09-15T02:00:51.066273+0000","last_clean":"2024-09-15T02:00:51.066273+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:00:51.066273+0000","last_undegraded":"2024-09-15T02:00:51.066273+0000","last_fullsized":"2024-09-15T02:00:51.066273+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:01:09.892426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":149,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.071093+0000","last_change":"2024-09-15T01:58:09.926074+0000","last_active":"2024-09-15T02:00:51.071093+0000","last_peered":"2024-09-15T02:00:51.071093+0000","last_clean":"2024-09-15T02:00:51.071093+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:00:51.071093+0000","last_undegraded":"2024-09-15T02:00:51.071093+0000","last_fullsized":"2024-09-15T02:00:51.071093+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:58:38.255694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066315+0000","last_change":"2024-09-15T01:58:09.925557+0000","last_active":"2024-09-15T02:00:51.066315+0000","last_peered":"2024-09-15T02:00:51.066315+0000","last_clean":"2024-09-15T02:00:51.066315+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:00:51.066315+0000","last_undegraded":"2024-09-15T02:00:51.066315+0000","last_fullsized":"2024-09-15T02:00:51.066315+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:44:24.015162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":123,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066328+0000","last_change":"2024-09-15T01:58:11.289109+0000","last_active":"2024-09-15T02:00:51.066328+0000","last_peered":"2024-09-15T02:00:51.066328+0000","last_clean":"2024-09-15T02:00:51.066328+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:00:51.066328+0000","last_undegraded":"2024-09-15T02:00:51.066328+0000","last_fullsized":"2024-09-15T02:00:51.066328+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:27:20.657827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":104,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421956+0000","last_change":"2024-09-15T01:58:09.929748+0000","last_active":"2024-09-15T02:00:50.421956+0000","last_peered":"2024-09-15T02:00:50.421956+0000","last_clean":"2024-09-15T02:00:50.421956+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:00:50.421956+0000","last_undegraded":"2024-09-15T02:00:50.421956+0000","last_fullsized":"2024-09-15T02:00:50.421956+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:28:34.534341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":79,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962564+0000","last_change":"2024-09-15T01:58:09.931531+0000","last_active":"2024-09-15T02:00:48.962564+0000","last_peered":"2024-09-15T02:00:48.962564+0000","last_clean":"2024-09-15T02:00:48.962564+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:00:48.962564+0000","last_undegraded":"2024-09-15T02:00:48.962564+0000","last_fullsized":"2024-09-15T02:00:48.962564+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:32.099924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421858+0000","last_change":"2024-09-15T01:58:09.937601+0000","last_active":"2024-09-15T02:00:50.421858+0000","last_peered":"2024-09-15T02:00:50.421858+0000","last_clean":"2024-09-15T02:00:50.421858+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:00:50.421858+0000","last_undegraded":"2024-09-15T02:00:50.421858+0000","last_fullsized":"2024-09-15T02:00:50.421858+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:47:48.897436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":106,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066219+0000","last_change":"2024-09-15T01:58:09.926660+0000","last_active":"2024-09-15T02:00:51.066219+0000","last_peered":"2024-09-15T02:00:51.066219+0000","last_clean":"2024-09-15T02:00:51.066219+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:00:51.066219+0000","last_undegraded":"2024-09-15T02:00:51.066219+0000","last_fullsized":"2024-09-15T02:00:51.066219+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:24.643662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962416+0000","last_change":"2024-09-15T01:58:09.925970+0000","last_active":"2024-09-15T02:00:48.962416+0000","last_peered":"2024-09-15T02:00:48.962416+0000","last_clean":"2024-09-15T02:00:48.962416+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:00:48.962416+0000","last_undegraded":"2024-09-15T02:00:48.962416+0000","last_fullsized":"2024-09-15T02:00:48.962416+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:31.344565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":125,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260042+0000","last_change":"2024-09-15T01:58:09.929440+0000","last_active":"2024-09-15T02:00:51.260042+0000","last_peered":"2024-09-15T02:00:51.260042+0000","last_clean":"2024-09-15T02:00:51.260042+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:00:51.260042+0000","last_undegraded":"2024-09-15T02:00:51.260042+0000","last_fullsized":"2024-09-15T02:00:51.260042+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:12.548233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421711+0000","last_change":"2024-09-15T01:58:09.927140+0000","last_active":"2024-09-15T02:00:50.421711+0000","last_peered":"2024-09-15T02:00:50.421711+0000","last_clean":"2024-09-15T02:00:50.421711+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:00:50.421711+0000","last_undegraded":"2024-09-15T02:00:50.421711+0000","last_fullsized":"2024-09-15T02:00:50.421711+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:14:46.532027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":84,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421823+0000","last_change":"2024-09-15T01:58:09.927406+0000","last_active":"2024-09-15T02:00:50.421823+0000","last_peered":"2024-09-15T02:00:50.421823+0000","last_clean":"2024-09-15T02:00:50.421823+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:00:50.421823+0000","last_undegraded":"2024-09-15T02:00:50.421823+0000","last_fullsized":"2024-09-15T02:00:50.421823+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:44:15.914758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":179,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421309+0000","last_change":"2024-09-15T01:58:09.926338+0000","last_active":"2024-09-15T02:00:50.421309+0000","last_peered":"2024-09-15T02:00:50.421309+0000","last_clean":"2024-09-15T02:00:50.421309+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:00:50.421309+0000","last_undegraded":"2024-09-15T02:00:50.421309+0000","last_fullsized":"2024-09-15T02:00:50.421309+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.854672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.070924+0000","last_change":"2024-09-15T01:58:11.002561+0000","last_active":"2024-09-15T02:00:51.070924+0000","last_peered":"2024-09-15T02:00:51.070924+0000","last_clean":"2024-09-15T02:00:51.070924+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:00:51.070924+0000","last_undegraded":"2024-09-15T02:00:51.070924+0000","last_fullsized":"2024-09-15T02:00:51.070924+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:09:57.334699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421716+0000","last_change":"2024-09-15T01:58:09.929239+0000","last_active":"2024-09-15T02:00:50.421716+0000","last_peered":"2024-09-15T02:00:50.421716+0000","last_clean":"2024-09-15T02:00:50.421716+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:00:50.421716+0000","last_undegraded":"2024-09-15T02:00:50.421716+0000","last_fullsized":"2024-09-15T02:00:50.421716+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:42:24.014259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":236,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962476+0000","last_change":"2024-09-15T01:58:09.925695+0000","last_active":"2024-09-15T02:00:48.962476+0000","last_peered":"2024-09-15T02:00:48.962476+0000","last_clean":"2024-09-15T02:00:48.962476+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:00:48.962476+0000","last_undegraded":"2024-09-15T02:00:48.962476+0000","last_fullsized":"2024-09-15T02:00:48.962476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:39.999243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":230,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066193+0000","last_change":"2024-09-15T01:58:10.926968+0000","last_active":"2024-09-15T02:00:51.066193+0000","last_peered":"2024-09-15T02:00:51.066193+0000","last_clean":"2024-09-15T02:00:51.066193+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:00:51.066193+0000","last_undegraded":"2024-09-15T02:00:51.066193+0000","last_fullsized":"2024-09-15T02:00:51.066193+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:11:04.527920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":85,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260044+0000","last_change":"2024-09-15T01:58:09.934342+0000","last_active":"2024-09-15T02:00:51.260044+0000","last_peered":"2024-09-15T02:00:51.260044+0000","last_clean":"2024-09-15T02:00:51.260044+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:00:51.260044+0000","last_undegraded":"2024-09-15T02:00:51.260044+0000","last_fullsized":"2024-09-15T02:00:51.260044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:09:06.563997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":89,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421728+0000","last_change":"2024-09-15T01:58:09.937375+0000","last_active":"2024-09-15T02:00:50.421728+0000","last_peered":"2024-09-15T02:00:50.421728+0000","last_clean":"2024-09-15T02:00:50.421728+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:00:50.421728+0000","last_undegraded":"2024-09-15T02:00:50.421728+0000","last_fullsized":"2024-09-15T02:00:50.421728+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:25:49.703445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":363,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962422+0000","last_change":"2024-09-15T01:58:09.940565+0000","last_active":"2024-09-15T02:00:48.962422+0000","last_peered":"2024-09-15T02:00:48.962422+0000","last_clean":"2024-09-15T02:00:48.962422+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:00:48.962422+0000","last_undegraded":"2024-09-15T02:00:48.962422+0000","last_fullsized":"2024-09-15T02:00:48.962422+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:27:19.210899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421850+0000","last_change":"2024-09-15T01:58:09.935097+0000","last_active":"2024-09-15T02:00:50.421850+0000","last_peered":"2024-09-15T02:00:50.421850+0000","last_clean":"2024-09-15T02:00:50.421850+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:00:50.421850+0000","last_undegraded":"2024-09-15T02:00:50.421850+0000","last_fullsized":"2024-09-15T02:00:50.421850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:26:20.884430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260084+0000","last_change":"2024-09-15T01:58:09.921899+0000","last_active":"2024-09-15T02:00:51.260084+0000","last_peered":"2024-09-15T02:00:51.260084+0000","last_clean":"2024-09-15T02:00:51.260084+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:00:51.260084+0000","last_undegraded":"2024-09-15T02:00:51.260084+0000","last_fullsized":"2024-09-15T02:00:51.260084+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:34:26.752431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962465+0000","last_change":"2024-09-15T01:58:09.923781+0000","last_active":"2024-09-15T02:00:48.962465+0000","last_peered":"2024-09-15T02:00:48.962465+0000","last_clean":"2024-09-15T02:00:48.962465+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:00:48.962465+0000","last_undegraded":"2024-09-15T02:00:48.962465+0000","last_fullsized":"2024-09-15T02:00:48.962465+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:13.093666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":237,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066329+0000","last_change":"2024-09-15T01:58:09.929011+0000","last_active":"2024-09-15T02:00:51.066329+0000","last_peered":"2024-09-15T02:00:51.066329+0000","last_clean":"2024-09-15T02:00:51.066329+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:00:51.066329+0000","last_undegraded":"2024-09-15T02:00:51.066329+0000","last_fullsized":"2024-09-15T02:00:51.066329+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:11:45.098864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.504,"15min":0.496},"min":{"1min":0.375,"5min":0.315,"15min":0.253},"max":{"1min":0.69299999999999995,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.51800000000000002},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56000000000000005},"min":{"1min":0.45300000000000001,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38900000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.501,"5min":0.745,"15min":0.745},"last":0.48599999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41299999999999998,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.44900000000000001,"5min":0.879,"15min":0.879},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.501,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.57399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298500,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073340,"statfs":{"total":96636764160,"available":96331100160,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.62,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.746,"15min":0.80200000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.65700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.29999999999999999},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.56899999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.247},"max":{"1min":0.63,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.69799999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303172,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068668,"statfs":{"total":96636764160,"available":96326316032,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.51900000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68000000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66000000000000003,"5min":1.022,"15min":1.022},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":1.008,"15min":1.008},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68600000000000005,"5min":0.747,"15min":0.747},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.436}]},{"osd":4,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39500000000000002,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.52500000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42299999999999999,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.56000000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.35899999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23499999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":0.64400000000000002,"15min":0.64700000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.36699999999999999,"15min":0.375},"min":{"1min":0.249,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.40799999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59299999999999997,"15min":0.56499999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.82199999999999995,"15min":0.88100000000000001},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.55700000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.315,"15min":0.32800000000000001},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.45800000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.33800000000000002},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.56899999999999995,"15min":0.66700000000000004},"last":0.307}]},{"osd":2,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73099999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.38700000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.52300000000000002,"5min":0.66800000000000004,"15min":0.75800000000000001},"last":0.374}]},{"osd":3,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.78000000000000003,"15min":0.84999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.77800000000000002},"last":0.51400000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56299999999999994}]}]},{"osd":0,"up_from":11,"seq":47244640514,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.316,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44800000000000001,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.19400000000000001},"max":{"1min":0.53200000000000003,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.36399999999999999},"min":{"1min":0.23200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":0.56399999999999995,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.313}]},{"osd":3,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.39700000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.075},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.443}]},{"osd":5,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":1.2729999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:12:38.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:12:38.931 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.929+0000 7fe95939d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.929+0000 7fe95939d640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.930+0000 7fe95939d640 10 monclient: monmap: 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:38.930838+0000 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:38.930838+0000 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:38.932 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.930+0000 7fe95939d640 1 Processor -- start 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.930+0000 7fe95939d640 1 -- start start 2024-09-15T02:12:38.933 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.930+0000 7fe95939d640 10 monclient: init 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-b con 0x7fe954068f20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9540692f0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe954136fe0 0x7fe9541393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-c con 0x7fe954136fe0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:38.937 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9540a1950 con 0x7fe9540692f0 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe954069dd0 con 0x7fe954136fe0 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540a1670 con 0x7fe954068f20 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe95939d640 10 monclient(hunting): authenticate will time out at 2018.927159s 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38874/0 (socket says 172.21.15.17:38874) 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 -- 172.21.15.17:0/2140797987 learned_addr learned my addr 172.21.15.17:0/2140797987 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): get_auth_request con 0x7fe954068f20 auth_method 0 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe9537fe640 1 --2- 172.21.15.17:0/2140797987 >> v2:172.21.15.67:3300/0 conn(0x7fe954136fe0 0x7fe9541393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe9537fe640 10 monclient(hunting): get_auth_request con 0x7fe954136fe0 auth_method 0 2024-09-15T02:12:38.939 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe9537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe9537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe9537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient(hunting): handle_auth_done global_id 214547 payload 274 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 -- 172.21.15.17:0/2140797987 >> v2:172.21.15.67:3300/0 conn(0x7fe954136fe0 msgr2=0x7fe9541393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 --2- 172.21.15.17:0/2140797987 >> v2:172.21.15.67:3300/0 conn(0x7fe954136fe0 0x7fe9541393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 -- 172.21.15.17:0/2140797987 >> v1:172.21.15.67:6789/0 conn(0x7fe9540692f0 legacy=0x7fe9540696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 monclient: found mon.noname-b 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.940 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.931+0000 7fe952ffd640 1 -- 172.21.15.17:0/2140797987 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9540a0020 con 0x7fe954068f20 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe952ffd640 1 -- 172.21.15.17:0/2140797987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 msgr2=0x7fe9541321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe952ffd640 1 --2- 172.21.15.17:0/2140797987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe944009910 tx=0x7fe9440091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f448e4a4ea165368 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 1 -- 172.21.15.17:0/2140797987 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944032440 con 0x7fe954068f20 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: dump: 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.941 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 1 -- 172.21.15.17:0/2140797987 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9440325a0 con 0x7fe954068f20 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe95939d640 5 monclient: authenticate success, global_id 214547 2024-09-15T02:12:38.942 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe95939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 1 -- 172.21.15.17:0/2140797987 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944032980 con 0x7fe954068f20 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe951ffb640 10 monclient: dump: 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.943 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe95939d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.934+0000 7fe95939d640 4 set_mon_vals no callback set 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 10 monclient: shutdown 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 -- 172.21.15.17:0/2140797987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 msgr2=0x7fe9541321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 --2- 172.21.15.17:0/2140797987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe944009910 tx=0x7fe9440091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.944 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 -- 172.21.15.17:0/2140797987 shutdown_connections 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 --2- 172.21.15.17:0/2140797987 >> v2:172.21.15.67:3300/0 conn(0x7fe954136fe0 0x7fe9541393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 --2- 172.21.15.17:0/2140797987 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe954068f20 0x7fe9541321c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 -- 172.21.15.17:0/2140797987 >> 172.21.15.17:0/2140797987 conn(0x7fe954070fb0 msgr2=0x7fe9540a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 -- 172.21.15.17:0/2140797987 shutdown_connections 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 1 -- 172.21.15.17:0/2140797987 wait complete. 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 10 monclient: build_initial_monmap 2024-09-15T02:12:38.945 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.935+0000 7fe95939d640 10 monclient: monmap: 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:38.946 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 Processor -- start 2024-09-15T02:12:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 -- start start 2024-09-15T02:12:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient: init 2024-09-15T02:12:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:38.952 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe954068f20 0x7fe954141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-b con 0x7fe954068f20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-c con 0x7fe9540692f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe954136fe0 0x7fe954141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): picked mon.noname-a con 0x7fe954136fe0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95409ca40 con 0x7fe954136fe0 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95411d000 con 0x7fe954068f20 2024-09-15T02:12:38.953 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95411cc90 con 0x7fe9540692f0 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe95939d640 10 monclient(hunting): authenticate will time out at 2018.932835s 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38888/0 (socket says 172.21.15.17:38888) 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 1 -- 172.21.15.17:0/418586517 learned_addr learned my addr 172.21.15.17:0/418586517 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 10 monclient(hunting): get_auth_request con 0x7fe9540692f0 auth_method 0 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.936+0000 7fe9537fe640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe954136fe0 0x7fe954141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.954 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): get_auth_request con 0x7fe954136fe0 auth_method 0 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe954068f20 0x7fe954141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): get_auth_request con 0x7fe954068f20 auth_method 0 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:38.955 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe952ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient(hunting): handle_auth_done global_id 214550 payload 824 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 -- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe954068f20 msgr2=0x7fe954141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe954068f20 0x7fe954141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 -- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe954136fe0 msgr2=0x7fe954141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe954136fe0 0x7fe954141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient: _finish_hunting 0 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 monclient: found mon.noname-c 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.937+0000 7fe9527fc640 1 -- 172.21.15.17:0/418586517 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944009450 con 0x7fe9540692f0 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe9527fc640 1 -- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 msgr2=0x7fe954141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe9527fc640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00c5e0 tx=0x7fe93c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c012840 con 0x7fe9540692f0 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: dump: 2024-09-15T02:12:38.957 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.958 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: _finish_auth 0 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 5 monclient: authenticate success, global_id 214550 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 10 monclient: _renew_subs 2024-09-15T02:12:38.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95411dc20 con 0x7fe9540692f0 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c0129a0 con 0x7fe9540692f0 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c012da0 con 0x7fe9540692f0 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe94affd640 10 monclient: dump: 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:38.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe9517fa640 4 set_mon_vals no callback set 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 10 monclient: _renew_subs 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.939+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95409bea0 con 0x7fe9540692f0 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe93c011590 con 0x7fe9540692f0 2024-09-15T02:12:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe95939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe95939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe95939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9540724a0 con 0x7fe9540692f0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe94affd640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 0x7fe93403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.940+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe93c04c780 con 0x7fe9540692f0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.941+0000 7fe952ffd640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 0x7fe93403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.941+0000 7fe952ffd640 10 monclient: get_auth_request con 0x7fe934038740 auth_method 0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.943+0000 7fe952ffd640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 0x7fe93403abf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe954071760 tx=0x7fe944032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.945+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe93c010ac0 con 0x7fe9540692f0 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.945+0000 7fe94affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:38.945+0000 7fe94affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:39.107 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.106+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe95411d140 con 0x7fe934038740 2024-09-15T02:12:39.111 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.110+0000 7fe94affd640 1 -- 172.21.15.17:0/418586517 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98323 (secure 0 0 0) 0x7fe95411d140 con 0x7fe934038740 2024-09-15T02:12:39.112 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:39.112 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 msgr2=0x7fe93403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 0x7fe93403abf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe954071760 tx=0x7fe944032000 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 10 monclient: shutdown 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 msgr2=0x7fe954141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.111+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00c5e0 tx=0x7fe93c007ed0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 shutdown_connections 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe934038740 0x7fe93403abf0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe954136fe0 0x7fe954141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe9540692f0 0x7fe954141990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 --2- 172.21.15.17:0/418586517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe954068f20 0x7fe954141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.114 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 >> 172.21.15.17:0/418586517 conn(0x7fe954070fb0 msgr2=0x7fe95409e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:39.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 shutdown_connections 2024-09-15T02:12:39.115 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.112+0000 7fe95939d640 1 -- 172.21.15.17:0/418586517 wait complete. 2024-09-15T02:12:39.123 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-15T02:12:37.578853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812796,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418244,"statfs":{"total":579820584960,"available":577964281856,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":172,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421220+0000","last_change":"2024-09-15T01:58:09.930154+0000","last_active":"2024-09-15T02:00:50.421220+0000","last_peered":"2024-09-15T02:00:50.421220+0000","last_clean":"2024-09-15T02:00:50.421220+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:00:50.421220+0000","last_undegraded":"2024-09-15T02:00:50.421220+0000","last_fullsized":"2024-09-15T02:00:50.421220+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:15:30.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066273+0000","last_change":"2024-09-15T01:58:09.929368+0000","last_active":"2024-09-15T02:00:51.066273+0000","last_peered":"2024-09-15T02:00:51.066273+0000","last_clean":"2024-09-15T02:00:51.066273+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:00:51.066273+0000","last_undegraded":"2024-09-15T02:00:51.066273+0000","last_fullsized":"2024-09-15T02:00:51.066273+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:01:09.892426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":149,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.071093+0000","last_change":"2024-09-15T01:58:09.926074+0000","last_active":"2024-09-15T02:00:51.071093+0000","last_peered":"2024-09-15T02:00:51.071093+0000","last_clean":"2024-09-15T02:00:51.071093+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:00:51.071093+0000","last_undegraded":"2024-09-15T02:00:51.071093+0000","last_fullsized":"2024-09-15T02:00:51.071093+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:58:38.255694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066315+0000","last_change":"2024-09-15T01:58:09.925557+0000","last_active":"2024-09-15T02:00:51.066315+0000","last_peered":"2024-09-15T02:00:51.066315+0000","last_clean":"2024-09-15T02:00:51.066315+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:00:51.066315+0000","last_undegraded":"2024-09-15T02:00:51.066315+0000","last_fullsized":"2024-09-15T02:00:51.066315+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:44:24.015162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":123,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066328+0000","last_change":"2024-09-15T01:58:11.289109+0000","last_active":"2024-09-15T02:00:51.066328+0000","last_peered":"2024-09-15T02:00:51.066328+0000","last_clean":"2024-09-15T02:00:51.066328+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:00:51.066328+0000","last_undegraded":"2024-09-15T02:00:51.066328+0000","last_fullsized":"2024-09-15T02:00:51.066328+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:27:20.657827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":104,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421956+0000","last_change":"2024-09-15T01:58:09.929748+0000","last_active":"2024-09-15T02:00:50.421956+0000","last_peered":"2024-09-15T02:00:50.421956+0000","last_clean":"2024-09-15T02:00:50.421956+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:00:50.421956+0000","last_undegraded":"2024-09-15T02:00:50.421956+0000","last_fullsized":"2024-09-15T02:00:50.421956+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:28:34.534341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":79,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962564+0000","last_change":"2024-09-15T01:58:09.931531+0000","last_active":"2024-09-15T02:00:48.962564+0000","last_peered":"2024-09-15T02:00:48.962564+0000","last_clean":"2024-09-15T02:00:48.962564+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:00:48.962564+0000","last_undegraded":"2024-09-15T02:00:48.962564+0000","last_fullsized":"2024-09-15T02:00:48.962564+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:32.099924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421858+0000","last_change":"2024-09-15T01:58:09.937601+0000","last_active":"2024-09-15T02:00:50.421858+0000","last_peered":"2024-09-15T02:00:50.421858+0000","last_clean":"2024-09-15T02:00:50.421858+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:00:50.421858+0000","last_undegraded":"2024-09-15T02:00:50.421858+0000","last_fullsized":"2024-09-15T02:00:50.421858+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:47:48.897436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":106,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066219+0000","last_change":"2024-09-15T01:58:09.926660+0000","last_active":"2024-09-15T02:00:51.066219+0000","last_peered":"2024-09-15T02:00:51.066219+0000","last_clean":"2024-09-15T02:00:51.066219+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:00:51.066219+0000","last_undegraded":"2024-09-15T02:00:51.066219+0000","last_fullsized":"2024-09-15T02:00:51.066219+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:24.643662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962416+0000","last_change":"2024-09-15T01:58:09.925970+0000","last_active":"2024-09-15T02:00:48.962416+0000","last_peered":"2024-09-15T02:00:48.962416+0000","last_clean":"2024-09-15T02:00:48.962416+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:00:48.962416+0000","last_undegraded":"2024-09-15T02:00:48.962416+0000","last_fullsized":"2024-09-15T02:00:48.962416+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:31.344565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":125,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260042+0000","last_change":"2024-09-15T01:58:09.929440+0000","last_active":"2024-09-15T02:00:51.260042+0000","last_peered":"2024-09-15T02:00:51.260042+0000","last_clean":"2024-09-15T02:00:51.260042+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:00:51.260042+0000","last_undegraded":"2024-09-15T02:00:51.260042+0000","last_fullsized":"2024-09-15T02:00:51.260042+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:12.548233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421711+0000","last_change":"2024-09-15T01:58:09.927140+0000","last_active":"2024-09-15T02:00:50.421711+0000","last_peered":"2024-09-15T02:00:50.421711+0000","last_clean":"2024-09-15T02:00:50.421711+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:00:50.421711+0000","last_undegraded":"2024-09-15T02:00:50.421711+0000","last_fullsized":"2024-09-15T02:00:50.421711+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:14:46.532027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":84,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421823+0000","last_change":"2024-09-15T01:58:09.927406+0000","last_active":"2024-09-15T02:00:50.421823+0000","last_peered":"2024-09-15T02:00:50.421823+0000","last_clean":"2024-09-15T02:00:50.421823+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:00:50.421823+0000","last_undegraded":"2024-09-15T02:00:50.421823+0000","last_fullsized":"2024-09-15T02:00:50.421823+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:44:15.914758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":179,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421309+0000","last_change":"2024-09-15T01:58:09.926338+0000","last_active":"2024-09-15T02:00:50.421309+0000","last_peered":"2024-09-15T02:00:50.421309+0000","last_clean":"2024-09-15T02:00:50.421309+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:00:50.421309+0000","last_undegraded":"2024-09-15T02:00:50.421309+0000","last_fullsized":"2024-09-15T02:00:50.421309+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.854672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.070924+0000","last_change":"2024-09-15T01:58:11.002561+0000","last_active":"2024-09-15T02:00:51.070924+0000","last_peered":"2024-09-15T02:00:51.070924+0000","last_clean":"2024-09-15T02:00:51.070924+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:00:51.070924+0000","last_undegraded":"2024-09-15T02:00:51.070924+0000","last_fullsized":"2024-09-15T02:00:51.070924+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:09:57.334699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421716+0000","last_change":"2024-09-15T01:58:09.929239+0000","last_active":"2024-09-15T02:00:50.421716+0000","last_peered":"2024-09-15T02:00:50.421716+0000","last_clean":"2024-09-15T02:00:50.421716+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:00:50.421716+0000","last_undegraded":"2024-09-15T02:00:50.421716+0000","last_fullsized":"2024-09-15T02:00:50.421716+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:42:24.014259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":236,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962476+0000","last_change":"2024-09-15T01:58:09.925695+0000","last_active":"2024-09-15T02:00:48.962476+0000","last_peered":"2024-09-15T02:00:48.962476+0000","last_clean":"2024-09-15T02:00:48.962476+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:00:48.962476+0000","last_undegraded":"2024-09-15T02:00:48.962476+0000","last_fullsized":"2024-09-15T02:00:48.962476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:39.999243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":230,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066193+0000","last_change":"2024-09-15T01:58:10.926968+0000","last_active":"2024-09-15T02:00:51.066193+0000","last_peered":"2024-09-15T02:00:51.066193+0000","last_clean":"2024-09-15T02:00:51.066193+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:00:51.066193+0000","last_undegraded":"2024-09-15T02:00:51.066193+0000","last_fullsized":"2024-09-15T02:00:51.066193+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:11:04.527920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":85,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260044+0000","last_change":"2024-09-15T01:58:09.934342+0000","last_active":"2024-09-15T02:00:51.260044+0000","last_peered":"2024-09-15T02:00:51.260044+0000","last_clean":"2024-09-15T02:00:51.260044+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:00:51.260044+0000","last_undegraded":"2024-09-15T02:00:51.260044+0000","last_fullsized":"2024-09-15T02:00:51.260044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:09:06.563997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":89,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421728+0000","last_change":"2024-09-15T01:58:09.937375+0000","last_active":"2024-09-15T02:00:50.421728+0000","last_peered":"2024-09-15T02:00:50.421728+0000","last_clean":"2024-09-15T02:00:50.421728+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:00:50.421728+0000","last_undegraded":"2024-09-15T02:00:50.421728+0000","last_fullsized":"2024-09-15T02:00:50.421728+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:25:49.703445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":363,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962422+0000","last_change":"2024-09-15T01:58:09.940565+0000","last_active":"2024-09-15T02:00:48.962422+0000","last_peered":"2024-09-15T02:00:48.962422+0000","last_clean":"2024-09-15T02:00:48.962422+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:00:48.962422+0000","last_undegraded":"2024-09-15T02:00:48.962422+0000","last_fullsized":"2024-09-15T02:00:48.962422+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:27:19.210899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421850+0000","last_change":"2024-09-15T01:58:09.935097+0000","last_active":"2024-09-15T02:00:50.421850+0000","last_peered":"2024-09-15T02:00:50.421850+0000","last_clean":"2024-09-15T02:00:50.421850+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:00:50.421850+0000","last_undegraded":"2024-09-15T02:00:50.421850+0000","last_fullsized":"2024-09-15T02:00:50.421850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:26:20.884430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260084+0000","last_change":"2024-09-15T01:58:09.921899+0000","last_active":"2024-09-15T02:00:51.260084+0000","last_peered":"2024-09-15T02:00:51.260084+0000","last_clean":"2024-09-15T02:00:51.260084+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:00:51.260084+0000","last_undegraded":"2024-09-15T02:00:51.260084+0000","last_fullsized":"2024-09-15T02:00:51.260084+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:34:26.752431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962465+0000","last_change":"2024-09-15T01:58:09.923781+0000","last_active":"2024-09-15T02:00:48.962465+0000","last_peered":"2024-09-15T02:00:48.962465+0000","last_clean":"2024-09-15T02:00:48.962465+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:00:48.962465+0000","last_undegraded":"2024-09-15T02:00:48.962465+0000","last_fullsized":"2024-09-15T02:00:48.962465+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:13.093666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":237,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066329+0000","last_change":"2024-09-15T01:58:09.929011+0000","last_active":"2024-09-15T02:00:51.066329+0000","last_peered":"2024-09-15T02:00:51.066329+0000","last_clean":"2024-09-15T02:00:51.066329+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:00:51.066329+0000","last_undegraded":"2024-09-15T02:00:51.066329+0000","last_fullsized":"2024-09-15T02:00:51.066329+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:11:45.098864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.504,"15min":0.496},"min":{"1min":0.375,"5min":0.315,"15min":0.253},"max":{"1min":0.69299999999999995,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.51800000000000002},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56000000000000005},"min":{"1min":0.45300000000000001,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38900000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.501,"5min":0.745,"15min":0.745},"last":0.48599999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41299999999999998,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.44900000000000001,"5min":0.879,"15min":0.879},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.501,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.57399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298500,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073340,"statfs":{"total":96636764160,"available":96331100160,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.62,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.746,"15min":0.80200000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.65700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.29999999999999999},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.56899999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.247},"max":{"1min":0.63,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.69799999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303172,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068668,"statfs":{"total":96636764160,"available":96326316032,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.51900000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68000000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66000000000000003,"5min":1.022,"15min":1.022},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":1.008,"15min":1.008},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68600000000000005,"5min":0.747,"15min":0.747},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.436}]},{"osd":4,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39500000000000002,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.52500000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42299999999999999,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.56000000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.35899999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23499999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":0.64400000000000002,"15min":0.64700000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.36699999999999999,"15min":0.375},"min":{"1min":0.249,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.40799999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59299999999999997,"15min":0.56499999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.82199999999999995,"15min":0.88100000000000001},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.55700000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.315,"15min":0.32800000000000001},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.45800000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.33800000000000002},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.56899999999999995,"15min":0.66700000000000004},"last":0.307}]},{"osd":2,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73099999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.38700000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.52300000000000002,"5min":0.66800000000000004,"15min":0.75800000000000001},"last":0.374}]},{"osd":3,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.78000000000000003,"15min":0.84999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.77800000000000002},"last":0.51400000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56299999999999994}]}]},{"osd":0,"up_from":11,"seq":47244640514,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.316,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44800000000000001,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.19400000000000001},"max":{"1min":0.53200000000000003,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.36399999999999999},"min":{"1min":0.23200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":0.56399999999999995,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.313}]},{"osd":3,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.39700000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.075},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.443}]},{"osd":5,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":1.2729999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:12:39.125 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:12:39.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.220+0000 7fbe4e79d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.220+0000 7fbe4e79d640 10 monclient: build_initial_monmap 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.220+0000 7fbe4e79d640 10 monclient: monmap: 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:39.221319+0000 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:39.221319+0000 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:39.223 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:39.223 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:39.223 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 Processor -- start 2024-09-15T02:12:39.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 -- start start 2024-09-15T02:12:39.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient: init 2024-09-15T02:12:39.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:39.224 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe480a5780 0x7fbe4809bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe480a5780 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe4809ccf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe48069c90 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe480a5b50 con 0x7fbe48069c90 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4809c5f0 con 0x7fbe4809ccf0 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe480a18b0 con 0x7fbe480a5780 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.221+0000 7fbe4e79d640 10 monclient(hunting): authenticate will time out at 2019.217926s 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe4cd3a640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fbe48069c90 0x7fbe48139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59442/0 (socket says 172.21.15.17:59442) 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2551583832 learned_addr learned my addr 172.21.15.17:0/2551583832 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:39.225 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2551583832 >> v2:172.21.15.67:3300/0 conn(0x7fbe480a5780 0x7fbe4809bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): get_auth_request con 0x7fbe480a5780 auth_method 0 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.226 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.222+0000 7fbe47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2551583832 >> v1:172.21.15.17:6789/0 conn(0x7fbe48069c90 legacy=0x7fbe48139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568720790 0 0) 0x7fbe480a5b50 con 0x7fbe48069c90 2024-09-15T02:12:39.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.227 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 10 monclient(hunting): my global_id is 254142 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe300034e0 con 0x7fbe48069c90 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe477fe640 1 -- 172.21.15.17:0/2551583832 >> v1:172.21.15.67:6789/0 conn(0x7fbe4809ccf0 legacy=0x7fbe48069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 10 monclient(hunting): handle_auth_done global_id 244255 payload 274 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 1 -- 172.21.15.17:0/2551583832 >> v1:172.21.15.67:6789/0 conn(0x7fbe4809ccf0 legacy=0x7fbe48069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 1 -- 172.21.15.17:0/2551583832 >> v1:172.21.15.17:6789/0 conn(0x7fbe48069c90 legacy=0x7fbe48139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 301897491 0 0) 0x7fbe300034e0 con 0x7fbe48069c90 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 1 monclient: found mon.noname-c 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe47fff640 1 -- 172.21.15.17:0/2551583832 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4809d210 con 0x7fbe480a5780 2024-09-15T02:12:39.228 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.224+0000 7fbe46ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:12:39.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe47fff640 1 -- 172.21.15.17:0/2551583832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe480a5780 msgr2=0x7fbe4809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2551583832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe480a5780 0x7fbe4809bc90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009950 tx=0x7fbe380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75edd7aecc20a6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40001aa0 con 0x7fbe480a5780 2024-09-15T02:12:39.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.232 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: dump: 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.233 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3802acd0 con 0x7fbe480a5780 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 1 -- 172.21.15.17:0/2551583832 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38032670 con 0x7fbe480a5780 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe4e79d640 5 monclient: authenticate success, global_id 244255 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe4e79d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe4e79d640 4 set_mon_vals no callback set 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.227+0000 7fbe46ffd640 10 monclient: dump: 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.234 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 10 monclient: shutdown 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2551583832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe480a5780 msgr2=0x7fbe4809bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2551583832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe480a5780 0x7fbe4809bc90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009950 tx=0x7fbe380090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.235 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2551583832 shutdown_connections 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2551583832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe480a5780 0x7fbe4809bc90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2551583832 >> 172.21.15.17:0/2551583832 conn(0x7fbe48070e10 msgr2=0x7fbe480a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2551583832 shutdown_connections 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2551583832 wait complete. 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 10 monclient: build_initial_monmap 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.228+0000 7fbe4e79d640 10 monclient: monmap: 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.236 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 Processor -- start 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 -- start start 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient: init 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 0x7fbe4813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe48069c90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4809ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:39.237 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 0x7fbe48141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe480a5780 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48122a20 con 0x7fbe480a5780 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4813ea70 con 0x7fbe48069c90 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48122740 con 0x7fbe4809ccf0 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe4e79d640 10 monclient(hunting): authenticate will time out at 2019.225893s 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.229+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38896/0 (socket says 172.21.15.17:38896) 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 0x7fbe48141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.238 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 1 -- 172.21.15.17:0/2570023129 learned_addr learned my addr 172.21.15.17:0/2570023129 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): get_auth_request con 0x7fbe4809ccf0 auth_method 0 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): get_auth_request con 0x7fbe480a5780 auth_method 0 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.239 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 0x7fbe4813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 10 monclient(hunting): get_auth_request con 0x7fbe48069c90 auth_method 0 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient(hunting): handle_auth_done global_id 254145 payload 824 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 msgr2=0x7fbe4813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 msgr2=0x7fbe4813dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 0x7fbe4813dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.240 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 monclient: found mon.noname-a 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2570023129 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38009390 con 0x7fbe480a5780 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 0x7fbe4813dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:39.241 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.230+0000 7fbe477fe640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4cd3a640 1 -- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 msgr2=0x7fbe48141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4cd3a640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 0x7fbe48141560 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe400018d0 tx=0x7fbe40001920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4000a6e0 con 0x7fbe480a5780 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: dump: 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: _finish_auth 0 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4000a860 con 0x7fbe480a5780 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 5 monclient: authenticate success, global_id 254145 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 10 monclient: _renew_subs 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.244 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe48122d00 con 0x7fbe480a5780 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4000ac20 con 0x7fbe480a5780 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe2ffff640 10 monclient: dump: 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.245 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe45ffb640 4 set_mon_vals no callback set 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 10 monclient: _renew_subs 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.232+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe48121870 con 0x7fbe480a5780 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe40007360 con 0x7fbe480a5780 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe2ffff640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 0x7fbe3403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbe4004c0f0 con 0x7fbe480a5780 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe4e79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe4e79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe4e79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.233+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe480724a0 con 0x7fbe480a5780 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.234+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 0x7fbe3403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.246 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.234+0000 7fbe47fff640 10 monclient: get_auth_request con 0x7fbe34038740 auth_method 0 2024-09-15T02:12:39.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.236+0000 7fbe47fff640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 0x7fbe3403abf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbe4806f800 tx=0x7fbe38032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.238+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbe4000ae40 con 0x7fbe480a5780 2024-09-15T02:12:39.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.238+0000 7fbe2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.247 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.238+0000 7fbe2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:39.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.397+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbe48122f20 con 0x7fbe34038740 2024-09-15T02:12:39.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.400+0000 7fbe2ffff640 1 -- 172.21.15.17:0/2570023129 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98323 (secure 0 0 0) 0x7fbe48122f20 con 0x7fbe34038740 2024-09-15T02:12:39.402 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:39.403 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 msgr2=0x7fbe3403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 0x7fbe3403abf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbe4806f800 tx=0x7fbe38032000 comp rx=0 tx=0).stop 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 10 monclient: shutdown 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 msgr2=0x7fbe48141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 0x7fbe48141560 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe400018d0 tx=0x7fbe40001920 comp rx=0 tx=0).stop 2024-09-15T02:12:39.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 shutdown_connections 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fbe34038740 0x7fbe3403abf0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbe480a5780 0x7fbe48141560 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fbe4809ccf0 0x7fbe4813d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 --2- 172.21.15.17:0/2570023129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe48069c90 0x7fbe4813dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 >> 172.21.15.17:0/2570023129 conn(0x7fbe48070e10 msgr2=0x7fbe480a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 shutdown_connections 2024-09-15T02:12:39.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.403+0000 7fbe4e79d640 1 -- 172.21.15.17:0/2570023129 wait complete. 2024-09-15T02:12:39.414 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-15T02:12:37.578853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812796,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418244,"statfs":{"total":579820584960,"available":577964281856,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":172,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421220+0000","last_change":"2024-09-15T01:58:09.930154+0000","last_active":"2024-09-15T02:00:50.421220+0000","last_peered":"2024-09-15T02:00:50.421220+0000","last_clean":"2024-09-15T02:00:50.421220+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:00:50.421220+0000","last_undegraded":"2024-09-15T02:00:50.421220+0000","last_fullsized":"2024-09-15T02:00:50.421220+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:15:30.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066273+0000","last_change":"2024-09-15T01:58:09.929368+0000","last_active":"2024-09-15T02:00:51.066273+0000","last_peered":"2024-09-15T02:00:51.066273+0000","last_clean":"2024-09-15T02:00:51.066273+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:00:51.066273+0000","last_undegraded":"2024-09-15T02:00:51.066273+0000","last_fullsized":"2024-09-15T02:00:51.066273+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:01:09.892426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":149,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.071093+0000","last_change":"2024-09-15T01:58:09.926074+0000","last_active":"2024-09-15T02:00:51.071093+0000","last_peered":"2024-09-15T02:00:51.071093+0000","last_clean":"2024-09-15T02:00:51.071093+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:00:51.071093+0000","last_undegraded":"2024-09-15T02:00:51.071093+0000","last_fullsized":"2024-09-15T02:00:51.071093+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:58:38.255694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066315+0000","last_change":"2024-09-15T01:58:09.925557+0000","last_active":"2024-09-15T02:00:51.066315+0000","last_peered":"2024-09-15T02:00:51.066315+0000","last_clean":"2024-09-15T02:00:51.066315+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:00:51.066315+0000","last_undegraded":"2024-09-15T02:00:51.066315+0000","last_fullsized":"2024-09-15T02:00:51.066315+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:44:24.015162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":123,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066328+0000","last_change":"2024-09-15T01:58:11.289109+0000","last_active":"2024-09-15T02:00:51.066328+0000","last_peered":"2024-09-15T02:00:51.066328+0000","last_clean":"2024-09-15T02:00:51.066328+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:00:51.066328+0000","last_undegraded":"2024-09-15T02:00:51.066328+0000","last_fullsized":"2024-09-15T02:00:51.066328+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:27:20.657827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":104,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421956+0000","last_change":"2024-09-15T01:58:09.929748+0000","last_active":"2024-09-15T02:00:50.421956+0000","last_peered":"2024-09-15T02:00:50.421956+0000","last_clean":"2024-09-15T02:00:50.421956+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:00:50.421956+0000","last_undegraded":"2024-09-15T02:00:50.421956+0000","last_fullsized":"2024-09-15T02:00:50.421956+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:28:34.534341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":79,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962564+0000","last_change":"2024-09-15T01:58:09.931531+0000","last_active":"2024-09-15T02:00:48.962564+0000","last_peered":"2024-09-15T02:00:48.962564+0000","last_clean":"2024-09-15T02:00:48.962564+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:00:48.962564+0000","last_undegraded":"2024-09-15T02:00:48.962564+0000","last_fullsized":"2024-09-15T02:00:48.962564+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:32.099924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421858+0000","last_change":"2024-09-15T01:58:09.937601+0000","last_active":"2024-09-15T02:00:50.421858+0000","last_peered":"2024-09-15T02:00:50.421858+0000","last_clean":"2024-09-15T02:00:50.421858+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:00:50.421858+0000","last_undegraded":"2024-09-15T02:00:50.421858+0000","last_fullsized":"2024-09-15T02:00:50.421858+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:47:48.897436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":106,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066219+0000","last_change":"2024-09-15T01:58:09.926660+0000","last_active":"2024-09-15T02:00:51.066219+0000","last_peered":"2024-09-15T02:00:51.066219+0000","last_clean":"2024-09-15T02:00:51.066219+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:00:51.066219+0000","last_undegraded":"2024-09-15T02:00:51.066219+0000","last_fullsized":"2024-09-15T02:00:51.066219+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:24.643662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962416+0000","last_change":"2024-09-15T01:58:09.925970+0000","last_active":"2024-09-15T02:00:48.962416+0000","last_peered":"2024-09-15T02:00:48.962416+0000","last_clean":"2024-09-15T02:00:48.962416+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:00:48.962416+0000","last_undegraded":"2024-09-15T02:00:48.962416+0000","last_fullsized":"2024-09-15T02:00:48.962416+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:31.344565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":125,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260042+0000","last_change":"2024-09-15T01:58:09.929440+0000","last_active":"2024-09-15T02:00:51.260042+0000","last_peered":"2024-09-15T02:00:51.260042+0000","last_clean":"2024-09-15T02:00:51.260042+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:00:51.260042+0000","last_undegraded":"2024-09-15T02:00:51.260042+0000","last_fullsized":"2024-09-15T02:00:51.260042+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:12.548233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421711+0000","last_change":"2024-09-15T01:58:09.927140+0000","last_active":"2024-09-15T02:00:50.421711+0000","last_peered":"2024-09-15T02:00:50.421711+0000","last_clean":"2024-09-15T02:00:50.421711+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:00:50.421711+0000","last_undegraded":"2024-09-15T02:00:50.421711+0000","last_fullsized":"2024-09-15T02:00:50.421711+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:14:46.532027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":84,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421823+0000","last_change":"2024-09-15T01:58:09.927406+0000","last_active":"2024-09-15T02:00:50.421823+0000","last_peered":"2024-09-15T02:00:50.421823+0000","last_clean":"2024-09-15T02:00:50.421823+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:00:50.421823+0000","last_undegraded":"2024-09-15T02:00:50.421823+0000","last_fullsized":"2024-09-15T02:00:50.421823+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:44:15.914758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":179,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421309+0000","last_change":"2024-09-15T01:58:09.926338+0000","last_active":"2024-09-15T02:00:50.421309+0000","last_peered":"2024-09-15T02:00:50.421309+0000","last_clean":"2024-09-15T02:00:50.421309+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:00:50.421309+0000","last_undegraded":"2024-09-15T02:00:50.421309+0000","last_fullsized":"2024-09-15T02:00:50.421309+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.854672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":82,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.070924+0000","last_change":"2024-09-15T01:58:11.002561+0000","last_active":"2024-09-15T02:00:51.070924+0000","last_peered":"2024-09-15T02:00:51.070924+0000","last_clean":"2024-09-15T02:00:51.070924+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:00:51.070924+0000","last_undegraded":"2024-09-15T02:00:51.070924+0000","last_fullsized":"2024-09-15T02:00:51.070924+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:09:57.334699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421716+0000","last_change":"2024-09-15T01:58:09.929239+0000","last_active":"2024-09-15T02:00:50.421716+0000","last_peered":"2024-09-15T02:00:50.421716+0000","last_clean":"2024-09-15T02:00:50.421716+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:00:50.421716+0000","last_undegraded":"2024-09-15T02:00:50.421716+0000","last_fullsized":"2024-09-15T02:00:50.421716+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:42:24.014259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":236,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962476+0000","last_change":"2024-09-15T01:58:09.925695+0000","last_active":"2024-09-15T02:00:48.962476+0000","last_peered":"2024-09-15T02:00:48.962476+0000","last_clean":"2024-09-15T02:00:48.962476+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:00:48.962476+0000","last_undegraded":"2024-09-15T02:00:48.962476+0000","last_fullsized":"2024-09-15T02:00:48.962476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:39.999243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":230,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066193+0000","last_change":"2024-09-15T01:58:10.926968+0000","last_active":"2024-09-15T02:00:51.066193+0000","last_peered":"2024-09-15T02:00:51.066193+0000","last_clean":"2024-09-15T02:00:51.066193+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:00:51.066193+0000","last_undegraded":"2024-09-15T02:00:51.066193+0000","last_fullsized":"2024-09-15T02:00:51.066193+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:11:04.527920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":85,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260044+0000","last_change":"2024-09-15T01:58:09.934342+0000","last_active":"2024-09-15T02:00:51.260044+0000","last_peered":"2024-09-15T02:00:51.260044+0000","last_clean":"2024-09-15T02:00:51.260044+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:00:51.260044+0000","last_undegraded":"2024-09-15T02:00:51.260044+0000","last_fullsized":"2024-09-15T02:00:51.260044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:09:06.563997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":89,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421728+0000","last_change":"2024-09-15T01:58:09.937375+0000","last_active":"2024-09-15T02:00:50.421728+0000","last_peered":"2024-09-15T02:00:50.421728+0000","last_clean":"2024-09-15T02:00:50.421728+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:00:50.421728+0000","last_undegraded":"2024-09-15T02:00:50.421728+0000","last_fullsized":"2024-09-15T02:00:50.421728+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:25:49.703445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":363,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962422+0000","last_change":"2024-09-15T01:58:09.940565+0000","last_active":"2024-09-15T02:00:48.962422+0000","last_peered":"2024-09-15T02:00:48.962422+0000","last_clean":"2024-09-15T02:00:48.962422+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:00:48.962422+0000","last_undegraded":"2024-09-15T02:00:48.962422+0000","last_fullsized":"2024-09-15T02:00:48.962422+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:27:19.210899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421850+0000","last_change":"2024-09-15T01:58:09.935097+0000","last_active":"2024-09-15T02:00:50.421850+0000","last_peered":"2024-09-15T02:00:50.421850+0000","last_clean":"2024-09-15T02:00:50.421850+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:00:50.421850+0000","last_undegraded":"2024-09-15T02:00:50.421850+0000","last_fullsized":"2024-09-15T02:00:50.421850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:26:20.884430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260084+0000","last_change":"2024-09-15T01:58:09.921899+0000","last_active":"2024-09-15T02:00:51.260084+0000","last_peered":"2024-09-15T02:00:51.260084+0000","last_clean":"2024-09-15T02:00:51.260084+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:00:51.260084+0000","last_undegraded":"2024-09-15T02:00:51.260084+0000","last_fullsized":"2024-09-15T02:00:51.260084+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:34:26.752431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-15T02:00:48.962465+0000","last_change":"2024-09-15T01:58:09.923781+0000","last_active":"2024-09-15T02:00:48.962465+0000","last_peered":"2024-09-15T02:00:48.962465+0000","last_clean":"2024-09-15T02:00:48.962465+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:00:48.962465+0000","last_undegraded":"2024-09-15T02:00:48.962465+0000","last_fullsized":"2024-09-15T02:00:48.962465+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:13.093666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":237,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.066329+0000","last_change":"2024-09-15T01:58:09.929011+0000","last_active":"2024-09-15T02:00:51.066329+0000","last_peered":"2024-09-15T02:00:51.066329+0000","last_clean":"2024-09-15T02:00:51.066329+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:00:51.066329+0000","last_undegraded":"2024-09-15T02:00:51.066329+0000","last_fullsized":"2024-09-15T02:00:51.066329+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:11:45.098864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.504,"15min":0.496},"min":{"1min":0.375,"5min":0.315,"15min":0.253},"max":{"1min":0.69299999999999995,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.51800000000000002},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56000000000000005},"min":{"1min":0.45300000000000001,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38900000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.501,"5min":0.745,"15min":0.745},"last":0.48599999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41299999999999998,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.44900000000000001,"5min":0.879,"15min":0.879},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.501,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.57399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298500,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073340,"statfs":{"total":96636764160,"available":96331100160,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.62,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.746,"15min":0.80200000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.65700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.29999999999999999},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.56899999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.247},"max":{"1min":0.63,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.69799999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303172,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068668,"statfs":{"total":96636764160,"available":96326316032,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.51900000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68000000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66000000000000003,"5min":1.022,"15min":1.022},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":1.008,"15min":1.008},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68600000000000005,"5min":0.747,"15min":0.747},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.436}]},{"osd":4,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39500000000000002,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.52500000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42299999999999999,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.56000000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.35899999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23499999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":0.64400000000000002,"15min":0.64700000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.36699999999999999,"15min":0.375},"min":{"1min":0.249,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.40799999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59299999999999997,"15min":0.56499999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.82199999999999995,"15min":0.88100000000000001},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.55700000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.315,"15min":0.32800000000000001},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.45800000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.33800000000000002},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.56899999999999995,"15min":0.66700000000000004},"last":0.307}]},{"osd":2,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73099999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.38700000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.52300000000000002,"5min":0.66800000000000004,"15min":0.75800000000000001},"last":0.374}]},{"osd":3,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.78000000000000003,"15min":0.84999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.77800000000000002},"last":0.51400000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56299999999999994}]}]},{"osd":0,"up_from":11,"seq":47244640514,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.316,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44800000000000001,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.19400000000000001},"max":{"1min":0.53200000000000003,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.36399999999999999},"min":{"1min":0.23200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":0.56399999999999995,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.313}]},{"osd":3,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.39700000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.075},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.443}]},{"osd":5,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":1.2729999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:12:39.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.511+0000 7f4093a80640 10 monclient: get_monmap_and_config 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.511+0000 7f4093a80640 10 monclient: build_initial_monmap 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.511+0000 7f4093a80640 10 monclient: monmap: 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:39.512666+0000 2024-09-15T02:12:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:39.512666+0000 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:39.514 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.512+0000 7f4093a80640 1 Processor -- start 2024-09-15T02:12:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.512+0000 7f4093a80640 1 -- start start 2024-09-15T02:12:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.512+0000 7f4093a80640 10 monclient: init 2024-09-15T02:12:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f408c0a5780 0x7f408c09bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-c con 0x7f408c0a5780 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f408c09ccf0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f408c069c90 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0a5b50 con 0x7f408c09ccf0 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c09c5f0 con 0x7f408c069c90 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c0a18b0 con 0x7f408c0a5780 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:39.518 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f4093a80640 10 monclient(hunting): authenticate will time out at 2019.509127s 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409101b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f408c09ccf0 0x7f408c069580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59448/0 (socket says 172.21.15.17:59448) 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409101b640 1 -- 172.21.15.17:0/2624027647 learned_addr learned my addr 172.21.15.17:0/2624027647 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 1 --2- 172.21.15.17:0/2624027647 >> v2:172.21.15.67:3300/0 conn(0x7f408c0a5780 0x7f408c09bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): get_auth_request con 0x7f408c0a5780 auth_method 0 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.513+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409101b640 1 -- 172.21.15.17:0/2624027647 >> v1:172.21.15.17:6789/0 conn(0x7f408c09ccf0 legacy=0x7f408c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409081a640 1 -- 172.21.15.17:0/2624027647 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62159564 0 0) 0x7f408c0a5b50 con 0x7f408c09ccf0 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409081a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409081a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409081a640 10 monclient(hunting): my global_id is 254148 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409081a640 1 -- 172.21.15.17:0/2624027647 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40740034e0 con 0x7f408c09ccf0 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409201d640 1 -- 172.21.15.17:0/2624027647 >> v1:172.21.15.67:6789/0 conn(0x7f408c069c90 legacy=0x7f408c139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 10 monclient(hunting): handle_auth_done global_id 244264 payload 274 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 1 -- 172.21.15.17:0/2624027647 >> v1:172.21.15.67:6789/0 conn(0x7f408c069c90 legacy=0x7f408c139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 1 -- 172.21.15.17:0/2624027647 >> v1:172.21.15.17:6789/0 conn(0x7f408c09ccf0 legacy=0x7f408c069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 1 monclient: found mon.noname-c 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:12:39.521 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.515+0000 7f409181c640 1 -- 172.21.15.17:0/2624027647 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c09d210 con 0x7f408c0a5780 2024-09-15T02:12:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409181c640 1 -- 172.21.15.17:0/2624027647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 msgr2=0x7f408c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409181c640 1 --2- 172.21.15.17:0/2624027647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c09bc90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f407c009f60 tx=0x7f407c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5966f506872689ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 1 -- 172.21.15.17:0/2624027647 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c007760 con 0x7f408c0a5780 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: dump: 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: _finish_auth 0 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f4093a80640 5 monclient: authenticate success, global_id 244264 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f4093a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 1 -- 172.21.15.17:0/2624027647 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c02d050 con 0x7f408c0a5780 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f4093a80640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f4093a80640 4 set_mon_vals no callback set 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 1 -- 172.21.15.17:0/2624027647 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c007af0 con 0x7f408c0a5780 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.518+0000 7f409081a640 10 monclient: dump: 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.529 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 10 monclient: shutdown 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 -- 172.21.15.17:0/2624027647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 msgr2=0x7f408c09bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 --2- 172.21.15.17:0/2624027647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c09bc90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f407c009f60 tx=0x7f407c009780 comp rx=0 tx=0).stop 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 -- 172.21.15.17:0/2624027647 shutdown_connections 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 --2- 172.21.15.17:0/2624027647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c09bc90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.530 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 -- 172.21.15.17:0/2624027647 >> 172.21.15.17:0/2624027647 conn(0x7f408c070e10 msgr2=0x7f408c0a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 -- 172.21.15.17:0/2624027647 shutdown_connections 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 1 -- 172.21.15.17:0/2624027647 wait complete. 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 10 monclient: build_initial_monmap 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.519+0000 7f4093a80640 10 monclient: monmap: 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:39.531 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 Processor -- start 2024-09-15T02:12:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 -- start start 2024-09-15T02:12:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient: init 2024-09-15T02:12:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:39.532 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-a con 0x7f408c069c90 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f408c09ccf0 0x7f408c13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-c con 0x7f408c09ccf0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): picked mon.noname-b con 0x7f408c0a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c122d90 con 0x7f408c069c90 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.533 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1227d0 con 0x7f408c0a5780 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1202b0 con 0x7f408c09ccf0 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f4093a80640 10 monclient(hunting): authenticate will time out at 2019.516793s 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47038/0 (socket says 172.21.15.17:47038) 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 1 -- 172.21.15.17:0/748293633 learned_addr learned my addr 172.21.15.17:0/748293633 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 10 monclient(hunting): get_auth_request con 0x7f408c069c90 auth_method 0 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409101b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f408c09ccf0 0x7f408c13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409181c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409101b640 10 monclient(hunting): get_auth_request con 0x7f408c09ccf0 auth_method 0 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409101b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409101b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.520+0000 7f409101b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409101b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409101b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409101b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 10 monclient(hunting): get_auth_request con 0x7f408c0a5780 auth_method 0 2024-09-15T02:12:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient(hunting): handle_auth_done global_id 254151 payload 824 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 -- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f408c09ccf0 msgr2=0x7f408c13d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f408c09ccf0 0x7f408c13d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 -- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 msgr2=0x7f408c141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 monclient: found mon.noname-a 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409181c640 1 -- 172.21.15.17:0/748293633 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c009a20 con 0x7f408c069c90 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.521+0000 7f409201d640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f409181c640 1 -- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 msgr2=0x7f408c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f409181c640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f407c02a390 tx=0x7f407c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c000fc0 con 0x7f408c069c90 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: dump: 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c040070 con 0x7f408c069c90 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:39.538 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c031d10 con 0x7f408c069c90 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4089ffb640 10 monclient: dump: 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:39.539 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 5 monclient: authenticate success, global_id 254151 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 10 monclient: _renew_subs 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c1224f0 con 0x7f408c069c90 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f408bfff640 4 set_mon_vals no callback set 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 10 monclient: _renew_subs 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.523+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c13d990 con 0x7f408c069c90 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f407c042030 con 0x7f408c069c90 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4093a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4093a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.540 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4093a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c13db80 con 0x7f408c069c90 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4089ffb640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 0x7f407803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.524+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f407c06dd30 con 0x7f408c069c90 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.525+0000 7f409101b640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 0x7f407803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.525+0000 7f409101b640 10 monclient: get_auth_request con 0x7f4078038740 auth_method 0 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.527+0000 7f409101b640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 0x7f407803abf0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f408c063630 tx=0x7f4080007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:39.541 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.529+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f407c04e440 con 0x7f408c069c90 2024-09-15T02:12:39.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.529+0000 7f4089ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:39.542 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.529+0000 7f4089ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.692+0000 7f4093a80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.692+0000 7f4093a80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.692+0000 7f4093a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.692+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f408c0724a0 con 0x7f408c069c90 2024-09-15T02:12:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.693+0000 7f4089ffb640 1 -- 172.21.15.17:0/748293633 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+20446 (secure 0 0 0) 0x7f407c03a3a0 con 0x7f408c069c90 2024-09-15T02:12:39.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.693+0000 7f4089ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T02:12:39.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.693+0000 7f4089ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:12:39.695 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:39.695 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":257,"fsid":"e3089478-c52a-4174-94a2-258b1876359d","created":"2024-09-15T01:51:17.448714+0000","modified":"2024-09-15T02:00:51.417741+0000","last_up_change":"2024-09-15T01:51:27.483335+0000","last_in_change":"2024-09-15T01:55:54.795952+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":181,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T01:51:28.688141+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T01:51:29.355809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-15T01:56:08.635533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-15T01:56:09.758237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4a708981-55e8-4fe3-ab03-03a86c8340f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":31930},{"type":"v1","addr":"172.21.15.17:6801","nonce":31930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":31930},{"type":"v1","addr":"172.21.15.17:6803","nonce":31930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":31930},{"type":"v1","addr":"172.21.15.17:6807","nonce":31930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":31930},{"type":"v1","addr":"172.21.15.17:6805","nonce":31930}]},"public_addr":"172.21.15.17:6801/31930","cluster_addr":"172.21.15.17:6803/31930","heartbeat_back_addr":"172.21.15.17:6807/31930","heartbeat_front_addr":"172.21.15.17:6805/31930","state":["exists","up"]},{"osd":1,"uuid":"a9c20296-137e-4c23-9966-a98fd2ec1eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":31931},{"type":"v1","addr":"172.21.15.17:6809","nonce":31931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":31931},{"type":"v1","addr":"172.21.15.17:6811","nonce":31931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":31931},{"type":"v1","addr":"172.21.15.17:6815","nonce":31931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":31931},{"type":"v1","addr":"172.21.15.17:6813","nonce":31931}]},"public_addr":"172.21.15.17:6809/31931","cluster_addr":"172.21.15.17:6811/31931","heartbeat_back_addr":"172.21.15.17:6815/31931","heartbeat_front_addr":"172.21.15.17:6813/31931","state":["exists","up"]},{"osd":2,"uuid":"4e77b2f6-442a-4b8e-a4f9-3243120ad5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":31929},{"type":"v1","addr":"172.21.15.17:6817","nonce":31929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":31929},{"type":"v1","addr":"172.21.15.17:6819","nonce":31929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":31929},{"type":"v1","addr":"172.21.15.17:6823","nonce":31929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":31929},{"type":"v1","addr":"172.21.15.17:6821","nonce":31929}]},"public_addr":"172.21.15.17:6817/31929","cluster_addr":"172.21.15.17:6819/31929","heartbeat_back_addr":"172.21.15.17:6823/31929","heartbeat_front_addr":"172.21.15.17:6821/31929","state":["exists","up"]},{"osd":3,"uuid":"83be4a40-aa29-4bbb-817a-d31d9bff815e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":31880},{"type":"v1","addr":"172.21.15.67:6817","nonce":31880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":31880},{"type":"v1","addr":"172.21.15.67:6819","nonce":31880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":31880},{"type":"v1","addr":"172.21.15.67:6823","nonce":31880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":31880},{"type":"v1","addr":"172.21.15.67:6821","nonce":31880}]},"public_addr":"172.21.15.67:6817/31880","cluster_addr":"172.21.15.67:6819/31880","heartbeat_back_addr":"172.21.15.67:6823/31880","heartbeat_front_addr":"172.21.15.67:6821/31880","state":["exists","up"]},{"osd":4,"uuid":"27d9cb97-3b36-4002-8665-cf5170791748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":31881},{"type":"v1","addr":"172.21.15.67:6804","nonce":31881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":31881},{"type":"v1","addr":"172.21.15.67:6808","nonce":31881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":31881},{"type":"v1","addr":"172.21.15.67:6815","nonce":31881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":31881},{"type":"v1","addr":"172.21.15.67:6812","nonce":31881}]},"public_addr":"172.21.15.67:6804/31881","cluster_addr":"172.21.15.67:6808/31881","heartbeat_back_addr":"172.21.15.67:6815/31881","heartbeat_front_addr":"172.21.15.67:6812/31881","state":["exists","up"]},{"osd":5,"uuid":"142778f7-2757-4349-8124-1946d5d29c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":31882},{"type":"v1","addr":"172.21.15.67:6801","nonce":31882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":31882},{"type":"v1","addr":"172.21.15.67:6805","nonce":31882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":31882},{"type":"v1","addr":"172.21.15.67:6813","nonce":31882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":31882},{"type":"v1","addr":"172.21.15.67:6809","nonce":31882}]},"public_addr":"172.21.15.67:6801/31882","cluster_addr":"172.21.15.67:6805/31882","heartbeat_back_addr":"172.21.15.67:6813/31882","heartbeat_front_addr":"172.21.15.67:6809/31882","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7b19bad9-2bed-4846-b592-7009ac3d12e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.593725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.627101+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.602809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.641423+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.589740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T01:51:25.592626+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.2","mappings":[{"from":0,"to":1}]},{"pgid":"9.5","mappings":[{"from":0,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:12:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 msgr2=0x7f407803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 0x7f407803abf0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f408c063630 tx=0x7f4080007b60 comp rx=0 tx=0).stop 2024-09-15T02:12:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 10 monclient: shutdown 2024-09-15T02:12:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 msgr2=0x7f408c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f407c02a390 tx=0x7f407c037000 comp rx=0 tx=0).stop 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 shutdown_connections 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f4078038740 0x7f407803abf0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f408c0a5780 0x7f408c141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f408c09ccf0 0x7f408c13d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 --2- 172.21.15.17:0/748293633 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f408c069c90 0x7f408c13dda0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.695+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 >> 172.21.15.17:0/748293633 conn(0x7f408c070e10 msgr2=0x7f408c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.696+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 shutdown_connections 2024-09-15T02:12:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:39.696+0000 7f4093a80640 1 -- 172.21.15.17:0/748293633 wait complete. 2024-09-15T02:12:40.708 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T02:12:40.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:40.821 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.820+0000 7f64af80f640 10 monclient: get_monmap_and_config 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.820+0000 7f64af80f640 10 monclient: build_initial_monmap 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.820+0000 7f64af80f640 10 monclient: monmap: 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:40.821227+0000 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:40.821227+0000 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:40.822 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:40.823 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 Processor -- start 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 -- start start 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient: init 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64a809cef0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 0x7f64a813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-b con 0x7f64a806ed80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:40.825 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 0x7f64a813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-a con 0x7f64a813b410 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a8097bb0 con 0x7f64a809cef0 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a806f7d0 con 0x7f64a813b410 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.821+0000 7f64af80f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a809a750 con 0x7f64a806ed80 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64af80f640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64af80f640 10 monclient(hunting): authenticate will time out at 2020.818032s 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64ad5ab640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f64a809cef0 0x7f64a809a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59456/0 (socket says 172.21.15.17:59456) 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64ad5ab640 1 -- 172.21.15.17:0/185541475 learned_addr learned my addr 172.21.15.17:0/185541475 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 1 --2- 172.21.15.17:0/185541475 >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 0x7f64a813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 --2- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 0x7f64a813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.826 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): get_auth_request con 0x7f64a813b410 auth_method 0 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): get_auth_request con 0x7f64a806ed80 auth_method 0 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient(hunting): handle_auth_done global_id 214559 payload 274 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 -- 172.21.15.17:0/185541475 >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 msgr2=0x7f64a813d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 --2- 172.21.15.17:0/185541475 >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 0x7f64a813d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 -- 172.21.15.17:0/185541475 >> v1:172.21.15.17:6789/0 conn(0x7f64a809cef0 legacy=0x7f64a809a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient: _finish_hunting 0 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 monclient: found mon.noname-b 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64acdaa640 1 -- 172.21.15.17:0/185541475 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8097db0 con 0x7f64a806ed80 2024-09-15T02:12:40.828 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.822+0000 7f64addac640 1 --2- 172.21.15.17:0/185541475 >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 0x7f64a813d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:40.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64acdaa640 1 -- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 msgr2=0x7f64a813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64acdaa640 1 --2- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 0x7f64a813aed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f64a806f910 tx=0x7f64980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e965dac8968651a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 1 -- 172.21.15.17:0/185541475 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649802add0 con 0x7f64a806ed80 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: dump: 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.834 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: _finish_auth 0 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64af80f640 5 monclient: authenticate success, global_id 214559 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 1 -- 172.21.15.17:0/185541475 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6498032440 con 0x7f64a806ed80 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64af80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64af80f640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f64af80f640 4 set_mon_vals no callback set 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 1 -- 172.21.15.17:0/185541475 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498032820 con 0x7f64a806ed80 2024-09-15T02:12:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.825+0000 7f649ffff640 10 monclient: dump: 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.836 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 10 monclient: shutdown 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 -- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 msgr2=0x7f64a813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 --2- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 0x7f64a813aed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f64a806f910 tx=0x7f64980090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 -- 172.21.15.17:0/185541475 shutdown_connections 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 --2- 172.21.15.17:0/185541475 >> v2:172.21.15.17:3300/0 conn(0x7f64a813b410 0x7f64a813d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 --2- 172.21.15.17:0/185541475 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a806ed80 0x7f64a813aed0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 -- 172.21.15.17:0/185541475 >> 172.21.15.17:0/185541475 conn(0x7f64a80a1270 msgr2=0x7f64a80a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 -- 172.21.15.17:0/185541475 shutdown_connections 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 1 -- 172.21.15.17:0/185541475 wait complete. 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 10 monclient: build_initial_monmap 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.826+0000 7f64af80f640 10 monclient: monmap: 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:40.837 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:40.838 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:40.838 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:40.838 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:40.838 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.839 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:40.839 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:40.839 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 Processor -- start 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 -- start start 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient: init 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 0x7f64a8137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-a con 0x7f64a806ed80 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a809cef0 0x7f64a8120930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-b con 0x7f64a809cef0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 0x7f64a8120e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): picked mon.noname-c con 0x7f64a813b410 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8138ed0 con 0x7f64a806ed80 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a813bca0 con 0x7f64a809cef0 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8124150 con 0x7f64a813b410 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:40.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64af80f640 10 monclient(hunting): authenticate will time out at 2020.823745s 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 0x7f64a8137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 0x7f64a8120e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 0x7f64a8137d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47066/0 (socket says 172.21.15.17:47066) 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 1 -- 172.21.15.17:0/520359728 learned_addr learned my addr 172.21.15.17:0/520359728 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): get_auth_request con 0x7f64a806ed80 auth_method 0 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): get_auth_request con 0x7f64a813b410 auth_method 0 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64ad5ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.827+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a809cef0 0x7f64a8120930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 10 monclient(hunting): get_auth_request con 0x7f64a809cef0 auth_method 0 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 10 monclient(hunting): handle_auth_done global_id 214562 payload 824 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a809cef0 msgr2=0x7f64a8120930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a809cef0 0x7f64a8120930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 msgr2=0x7f64a8137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 0x7f64a8137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 10 monclient: _finish_hunting 0 2024-09-15T02:12:40.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 monclient: found mon.noname-c 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64addac640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0009a20 con 0x7f64a813b410 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64ad5ab640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a806ed80 0x7f64a8137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.828+0000 7f64acdaa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a809cef0 0x7f64a8120930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64addac640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 msgr2=0x7f64a8120e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64addac640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 0x7f64a8120e70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ee0 tx=0x7f64a000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a000ce10 con 0x7f64a813b410 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: dump: 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.846 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: _finish_auth 0 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 5 monclient: authenticate success, global_id 214562 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a000f040 con 0x7f64a813b410 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 10 monclient: _renew_subs 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6498009390 con 0x7f64a813b410 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:40.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00148e0 con 0x7f64a813b410 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649d7fa640 10 monclient: dump: 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.848 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 10 monclient: _renew_subs 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f649f7fe640 4 set_mon_vals no callback set 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.830+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a8122f10 con 0x7f64a813b410 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.831+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f64a0003020 con 0x7f64a813b410 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.831+0000 7f64af80f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.831+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64a80724a0 con 0x7f64a813b410 2024-09-15T02:12:40.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.831+0000 7f649d7fa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f648c038740 0x7f648c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.831+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f64a004d150 con 0x7f64a813b410 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f649d7fa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7f648c0416b0 0x7f648c043b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f648c044210 con 0x7f648c0416b0 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64a004f670 con 0x7f64a813b410 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f649d7fa640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f64ad5ab640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f648c038740 0x7f648c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f64ad5ab640 10 monclient: get_auth_request con 0x7f648c038740 auth_method 0 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f64acdaa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7f648c0416b0 0x7f648c043b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.832+0000 7f64acdaa640 10 monclient: get_auth_request con 0x7f648c0416b0 auth_method 0 2024-09-15T02:12:40.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.834+0000 7f64ad5ab640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f648c038740 0x7f648c03abf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f64a4006fd0 tx=0x7f64a4004960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.836+0000 7f64acdaa640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7f648c0416b0 0x7f648c043b40 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.837+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== osd.0 v2:172.21.15.17:6800/31930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f648c044210 con 0x7f648c0416b0 2024-09-15T02:12:40.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 --> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f64a81464b0 con 0x7f648c0416b0 2024-09-15T02:12:40.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f649d7fa640 1 -- 172.21.15.17:0/520359728 <== osd.0 v2:172.21.15.17:6800/31930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f64a81464b0 con 0x7f648c0416b0 2024-09-15T02:12:40.852 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:40.852 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:40.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7f648c0416b0 msgr2=0x7f648c043b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:6800/31930,v1:172.21.15.17:6801/31930] conn(0x7f648c0416b0 0x7f648c043b40 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f648c038740 msgr2=0x7f648c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f648c038740 0x7f648c03abf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f64a4006fd0 tx=0x7f64a4004960 comp rx=0 tx=0).stop 2024-09-15T02:12:40.853 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 10 monclient: shutdown 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 msgr2=0x7f64a8120e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 --2- 172.21.15.17:0/520359728 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f64a813b410 0x7f64a8120e70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ee0 tx=0x7f64a000b040 comp rx=0 tx=0).stop 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64addac640 1 -- 172.21.15.17:0/520359728 reap_dead start 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 shutdown_connections 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.849+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 >> 172.21.15.17:0/520359728 conn(0x7f64a80a1270 msgr2=0x7f64a809b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.850+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 shutdown_connections 2024-09-15T02:12:40.854 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.850+0000 7f64af80f640 1 -- 172.21.15.17:0/520359728 wait complete. 2024-09-15T02:12:40.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T02:12:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 10 monclient: monmap: 2024-09-15T02:12:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:40.959 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:40.959005+0000 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:40.959005+0000 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:40.960 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.962 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 1 Processor -- start 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 1 -- start start 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.958+0000 7fd17bb9d640 10 monclient: init 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd17409c0d0 0x7fd17409a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd17409c0d0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd17409c4a0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd17409c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740699c0 con 0x7fd17409c4a0 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.963 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17406a5e0 con 0x7fd17409c0d0 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17406a290 con 0x7fd17409c8a0 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17bb9d640 10 monclient(hunting): authenticate will time out at 2020.955488s 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38932/0 (socket says 172.21.15.17:38932) 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 -- 172.21.15.17:0/3822144162 learned_addr learned my addr 172.21.15.17:0/3822144162 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): get_auth_request con 0x7fd17409c8a0 auth_method 0 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.964 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd179939640 1 --2- 172.21.15.17:0/3822144162 >> v2:172.21.15.67:3300/0 conn(0x7fd17409c0d0 0x7fd17409a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd179939640 10 monclient(hunting): get_auth_request con 0x7fd17409c0d0 auth_method 0 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd179939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd179939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd179939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient(hunting): handle_auth_done global_id 214565 payload 274 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 -- 172.21.15.17:0/3822144162 >> v2:172.21.15.67:3300/0 conn(0x7fd17409c0d0 msgr2=0x7fd17409a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 --2- 172.21.15.17:0/3822144162 >> v2:172.21.15.67:3300/0 conn(0x7fd17409c0d0 0x7fd17409a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 -- 172.21.15.17:0/3822144162 >> v1:172.21.15.67:6789/0 conn(0x7fd17409c4a0 legacy=0x7fd17406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 monclient: found mon.noname-b 2024-09-15T02:12:40.965 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:40.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:40.966 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.959+0000 7fd17a13a640 1 -- 172.21.15.17:0/3822144162 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17409f6e0 con 0x7fd17409c8a0 2024-09-15T02:12:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.962+0000 7fd17a13a640 1 -- 172.21.15.17:0/3822144162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 msgr2=0x7fd17409e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.962+0000 7fd17a13a640 1 --2- 172.21.15.17:0/3822144162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1740a3e50 tx=0x7fd16c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7432a42180d6a5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 1 -- 172.21.15.17:0/3822144162 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c02ae80 con 0x7fd17409c8a0 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: dump: 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.972 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: _finish_auth 0 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 5 monclient: authenticate success, global_id 214565 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 1 -- 172.21.15.17:0/3822144162 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c032440 con 0x7fd17409c8a0 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 1 -- 172.21.15.17:0/3822144162 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0327c0 con 0x7fd17409c8a0 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 4 set_mon_vals no callback set 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:40.973 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd178937640 10 monclient: dump: 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 10 monclient: shutdown 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:40.974 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 -- 172.21.15.17:0/3822144162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 msgr2=0x7fd17409e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/3822144162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1740a3e50 tx=0x7fd16c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 -- 172.21.15.17:0/3822144162 shutdown_connections 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/3822144162 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17409e970 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/3822144162 >> v2:172.21.15.67:3300/0 conn(0x7fd17409c0d0 0x7fd17409a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 -- 172.21.15.17:0/3822144162 >> 172.21.15.17:0/3822144162 conn(0x7fd1740712f0 msgr2=0x7fd1740a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 -- 172.21.15.17:0/3822144162 shutdown_connections 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.963+0000 7fd17bb9d640 1 -- 172.21.15.17:0/3822144162 wait complete. 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient: monmap: 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:40.975 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 Processor -- start 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 -- start start 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient: init 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:40.976 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd17409c0d0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 0x7fd174134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd17409c4a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17411eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd17409c8a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17409cd80 con 0x7fd17409c0d0 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17409ecc0 con 0x7fd17409c4a0 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd174122360 con 0x7fd17409c8a0 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.964+0000 7fd17bb9d640 10 monclient(hunting): authenticate will time out at 2020.960980s 2024-09-15T02:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47078/0 (socket says 172.21.15.17:47078) 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 -- 172.21.15.17:0/2107656400 learned_addr learned my addr 172.21.15.17:0/2107656400 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): get_auth_request con 0x7fd17409c0d0 auth_method 0 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17411eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): get_auth_request con 0x7fd17409c8a0 auth_method 0 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.978 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd17a13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient(hunting): handle_auth_done global_id 254160 payload 824 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 -- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 msgr2=0x7fd17411eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17411eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 -- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 msgr2=0x7fd174134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179138640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 0x7fd174134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 0x7fd174134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient: _finish_hunting 0 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 monclient: found mon.noname-a 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:40.979 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179939640 1 -- 172.21.15.17:0/2107656400 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009450 con 0x7fd17409c0d0 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179138640 10 monclient: get_auth_request con 0x7fd17409c4a0 auth_method 0 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.965+0000 7fd179138640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 0x7fd174134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd179939640 1 -- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 msgr2=0x7fd1741410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd179939640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e1c0 tx=0x7fd164009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164013590 con 0x7fd17409c0d0 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: dump: 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.980 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640136f0 con 0x7fd17409c0d0 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd17bb9d640 5 monclient: authenticate success, global_id 254160 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd17bb9d640 10 monclient: _renew_subs 2024-09-15T02:12:40.981 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd17bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd174141f40 con 0x7fd17409c0d0 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164013a90 con 0x7fd17409c0d0 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd171ffb640 10 monclient: dump: 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd17bb9d640 10 monclient: _renew_subs 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.967+0000 7fd173fff640 4 set_mon_vals no callback set 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd17bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174121190 con 0x7fd17409c0d0 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd16401e460 con 0x7fd17409c0d0 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd171ffb640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 0x7fd16003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.968+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd16404cdb0 con 0x7fd17409c0d0 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd17bb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd17bb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:40.983 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd17bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1740724a0 con 0x7fd17409c0d0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd179138640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 0x7fd16003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.969+0000 7fd179138640 10 monclient: get_auth_request con 0x7fd160038740 auth_method 0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.971+0000 7fd179138640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 0x7fd16003abf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd174120250 tx=0x7fd168009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.973+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd164014420 con 0x7fd17409c0d0 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.973+0000 7fd171ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:40.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:40.973+0000 7fd171ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.139+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd174067220 con 0x7fd160038740 2024-09-15T02:12:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.140+0000 7fd171ffb640 1 -- 172.21.15.17:0/2107656400 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd174067220 con 0x7fd160038740 2024-09-15T02:12:41.141 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T02:12:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 msgr2=0x7fd16003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 0x7fd16003abf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd174120250 tx=0x7fd168009040 comp rx=0 tx=0).stop 2024-09-15T02:12:41.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 10 monclient: shutdown 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 msgr2=0x7fd1741410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e1c0 tx=0x7fd164009920 comp rx=0 tx=0).stop 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 shutdown_connections 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd160038740 0x7fd16003abf0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd17409c8a0 0x7fd17411eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17409c4a0 0x7fd174134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 --2- 172.21.15.17:0/2107656400 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd17409c0d0 0x7fd1741410a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 >> 172.21.15.17:0/2107656400 conn(0x7fd1740712f0 msgr2=0x7fd1740a1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 shutdown_connections 2024-09-15T02:12:41.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.142+0000 7fd17bb9d640 1 -- 172.21.15.17:0/2107656400 wait complete. 2024-09-15T02:12:41.154 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T02:12:41.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.254+0000 7f5adfd9d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.254+0000 7f5adfd9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.254+0000 7f5adfd9d640 10 monclient: monmap: 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:41.255403+0000 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:41.255403+0000 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.256 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:41.257 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:41.257 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:41.257 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:41.257 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:41.257 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 Processor -- start 2024-09-15T02:12:41.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 -- start start 2024-09-15T02:12:41.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient: init 2024-09-15T02:12:41.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ad809c0d0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ad809e970 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ad806eb30 0x7f5ad806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad806eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad809ed70 con 0x7f5ad809c0d0 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad809cbb0 con 0x7f5ad809e970 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad806a320 con 0x7f5ad806eb30 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.255+0000 7f5adfd9d640 10 monclient(hunting): authenticate will time out at 2021.251876s 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5addb39640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f5ad809c0d0 0x7f5ad809a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59468/0 (socket says 172.21.15.17:59468) 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5addb39640 1 -- 172.21.15.17:0/2089890047 learned_addr learned my addr 172.21.15.17:0/2089890047 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:41.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 1 --2- 172.21.15.17:0/2089890047 >> v2:172.21.15.67:3300/0 conn(0x7f5ad806eb30 0x7f5ad806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): get_auth_request con 0x7f5ad806eb30 auth_method 0 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.256+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 1 -- 172.21.15.17:0/2089890047 >> v1:172.21.15.17:6789/0 conn(0x7f5ad809c0d0 legacy=0x7f5ad809a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443540194 0 0) 0x7f5ad809ed70 con 0x7f5ad809c0d0 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.263 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): my global_id is 254163 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac40034e0 con 0x7f5ad809c0d0 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 1 -- 172.21.15.17:0/2089890047 >> v1:172.21.15.67:6789/0 conn(0x7f5ad809e970 legacy=0x7f5ad809c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491719507 0 0) 0x7f5ac40034e0 con 0x7f5ad809c0d0 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 >> v2:172.21.15.67:3300/0 conn(0x7f5ad806eb30 msgr2=0x7f5ad806efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 --2- 172.21.15.17:0/2089890047 >> v2:172.21.15.67:3300/0 conn(0x7f5ad806eb30 0x7f5ad806efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 >> v1:172.21.15.67:6789/0 conn(0x7f5ad809e970 legacy=0x7f5ad809c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 monclient: found mon.noname-a-legacy 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.17:6789/0 2024-09-15T02:12:41.264 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 --> v1:172.21.15.17:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad806a5e0 con 0x7f5ad809c0d0 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v1:172.21.15.17:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3582481286 0 0) 0x7f5ad4001720 con 0x7f5ad809c0d0 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: dump: 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.265 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] but i'm connected to v1:172.21.15.17:6789/0, reconnecting 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 >> v1:172.21.15.17:6789/0 conn(0x7f5ad809c0d0 legacy=0x7f5ad809a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac4003b20 0x7f5ac4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): picked mon.b con 0x7f5ac4003b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac4004780 0x7f5ac4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): picked mon.c con 0x7f5ac4004780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ac400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): picked mon.a con 0x7f5ac4005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:41.266 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad806a320 con 0x7f5ac4005330 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac400c2d0 con 0x7f5ac4003b20 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac400c490 con 0x7f5ac4004780 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5adcb37640 10 monclient(hunting): _finish_auth 0 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ac400b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac4004780 0x7f5ac4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): get_auth_request con 0x7f5ac4005330 auth_method 0 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac4003b20 0x7f5ac4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): get_auth_request con 0x7f5ac4004780 auth_method 0 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): get_auth_request con 0x7f5ac4003b20 auth_method 0 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5ade33a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.258+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 10 monclient(hunting): handle_auth_done global_id 254163 payload 293 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 -- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac4004780 msgr2=0x7f5ac4004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ac4004780 0x7f5ac4004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 -- 172.21.15.17:0/2089890047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac4003b20 msgr2=0x7f5ac4003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac4003b20 0x7f5ac4003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 monclient: found mon.a 2024-09-15T02:12:41.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 -- 172.21.15.17:0/2089890047 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40034e0 con 0x7f5ac4005330 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5adfd9d640 5 monclient: authenticate success, global_id 254163 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5adfd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:41.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.259+0000 7f5ade33a640 1 -- 172.21.15.17:0/2089890047 reap_dead start 2024-09-15T02:12:41.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.261+0000 7f5ade33a640 1 -- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 msgr2=0x7f5ac400b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.261+0000 7f5ade33a640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ac400b9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e210 tx=0x7f5ad000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7ac02d9fd58567 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.261+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad002c040 con 0x7f5ac4005330 2024-09-15T02:12:41.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.261+0000 7f5adcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: dump: 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad000f040 con 0x7f5ac4005330 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 4 set_mon_vals no callback set 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 1 -- 172.21.15.17:0/2089890047 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0030860 con 0x7f5ac4005330 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adcb37640 10 monclient: dump: 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.278 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 10 monclient: shutdown 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 -- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 msgr2=0x7f5ac400b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ac400b9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e210 tx=0x7f5ad000b040 comp rx=0 tx=0).stop 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 -- 172.21.15.17:0/2089890047 shutdown_connections 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 --2- 172.21.15.17:0/2089890047 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ac400b9f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 -- 172.21.15.17:0/2089890047 >> 172.21.15.17:0/2089890047 conn(0x7f5ad80711d0 msgr2=0x7f5ad80a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 -- 172.21.15.17:0/2089890047 shutdown_connections 2024-09-15T02:12:41.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.262+0000 7f5adfd9d640 1 -- 172.21.15.17:0/2089890047 wait complete. 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient: monmap: 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 Processor -- start 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 -- start start 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient: init 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ad8141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac4005330 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 0x7f5ad8134100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad8133d30 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad811d9e0 0x7f5ad811ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad811d9e0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad8134640 con 0x7f5ac4005330 2024-09-15T02:12:41.281 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad811f400 con 0x7f5ad811d9e0 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad8142000 con 0x7f5ad8133d30 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5adfd9d640 10 monclient(hunting): authenticate will time out at 2021.259907s 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.263+0000 7f5add338640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 0x7f5ad8134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 0x7f5ad8134100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38958/0 (socket says 172.21.15.17:38958) 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ad8141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 1 -- 172.21.15.17:0/4241148951 learned_addr learned my addr 172.21.15.17:0/4241148951 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): get_auth_request con 0x7f5ad8133d30 auth_method 0 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.282 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): get_auth_request con 0x7f5ac4005330 auth_method 0 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5ade33a640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad811d9e0 0x7f5ad811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient(hunting): handle_auth_done global_id 254166 payload 824 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 msgr2=0x7f5ad8134100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.283 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 0x7f5ad8134100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad811d9e0 msgr2=0x7f5ad811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad811d9e0 0x7f5ad811ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 monclient: found mon.noname-a 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5addb39640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8121160 con 0x7f5ac4005330 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5ade33a640 10 monclient: get_auth_request con 0x7f5ad811d9e0 auth_method 0 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5ade33a640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad811d9e0 0x7f5ad811ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.264+0000 7f5add338640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5ad8133d30 0x7f5ad8134100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5addb39640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 msgr2=0x7f5ad8141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.284 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5addb39640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ad8141110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000c00 tx=0x7f5ad4007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4018030 con 0x7f5ac4005330 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: dump: 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.285 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 5 monclient: authenticate success, global_id 254166 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad4012770 con 0x7f5ac4005330 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 10 monclient: _renew_subs 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad8120eb0 con 0x7f5ac4005330 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4012b10 con 0x7f5ac4005330 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acdffb640 10 monclient: dump: 2024-09-15T02:12:41.286 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5acffff640 4 set_mon_vals no callback set 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 10 monclient: _renew_subs 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.266+0000 7f5adfd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.287 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.267+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad81417f0 con 0x7f5ac4005330 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.267+0000 7f5adfd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.267+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ad80724a0 con 0x7f5ac4005330 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.267+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ad401b070 con 0x7f5ac4005330 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5abc0387c0 0x7f5abc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f5ad4019070 con 0x7f5ac4005330 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7f5abc0417b0 0x7f5abc043c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5abc044310 con 0x7f5abc0417b0 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ad404e4a0 con 0x7f5ac4005330 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5acdffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5add338640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5abc0387c0 0x7f5abc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5add338640 10 monclient: get_auth_request con 0x7f5abc0387c0 auth_method 0 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5ade33a640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7f5abc0417b0 0x7f5abc043c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.288 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.268+0000 7f5ade33a640 10 monclient: get_auth_request con 0x7f5abc0417b0 auth_method 0 2024-09-15T02:12:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.271+0000 7f5add338640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5abc0387c0 0x7f5abc03ac70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8001eb0 tx=0x7f5ac8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.273+0000 7f5ade33a640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7f5abc0417b0 0x7f5abc043c40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.273+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== osd.1 v2:172.21.15.17:6808/31931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5abc044310 con 0x7f5abc0417b0 2024-09-15T02:12:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.284+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 --> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5ad8120200 con 0x7f5abc0417b0 2024-09-15T02:12:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5acdffb640 1 -- 172.21.15.17:0/4241148951 <== osd.1 v2:172.21.15.17:6808/31931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5ad8120200 con 0x7f5abc0417b0 2024-09-15T02:12:41.290 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:41.290 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:41.290 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7f5abc0417b0 msgr2=0x7f5abc043c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:6808/31931,v1:172.21.15.17:6809/31931] conn(0x7f5abc0417b0 0x7f5abc043c40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5abc0387c0 msgr2=0x7f5abc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5abc0387c0 0x7f5abc03ac70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8001eb0 tx=0x7f5ac8009040 comp rx=0 tx=0).stop 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 10 monclient: shutdown 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 msgr2=0x7f5ad8141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 --2- 172.21.15.17:0/4241148951 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5ac4005330 0x7f5ad8141110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000c00 tx=0x7f5ad4007a50 comp rx=0 tx=0).stop 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5ade33a640 1 -- 172.21.15.17:0/4241148951 reap_dead start 2024-09-15T02:12:41.291 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 shutdown_connections 2024-09-15T02:12:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 >> 172.21.15.17:0/4241148951 conn(0x7f5ad80711d0 msgr2=0x7f5ad809b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 shutdown_connections 2024-09-15T02:12:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.285+0000 7f5adfd9d640 1 -- 172.21.15.17:0/4241148951 wait complete. 2024-09-15T02:12:41.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.392+0000 7fc0506c0640 10 monclient: get_monmap_and_config 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.392+0000 7fc0506c0640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.392+0000 7fc0506c0640 10 monclient: monmap: 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:41.393416+0000 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:41.393416+0000 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:41.394 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:41.395 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:41.395 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:41.395 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:41.395 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.397 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 Processor -- start 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 -- start start 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient: init 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc04809a9f0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc04809adc0 0x7fc0481330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-c con 0x7fc04809adc0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc048133630 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc048098fb0 con 0x7fc04809a9f0 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04812eb30 con 0x7fc048133630 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.398 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc048098cd0 con 0x7fc04809adc0 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc0506c0640 10 monclient(hunting): authenticate will time out at 2021.389852s 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc04e45c640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fc04809a9f0 0x7fc04812e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59476/0 (socket says 172.21.15.17:59476) 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.393+0000 7fc04e45c640 1 -- 172.21.15.17:0/2620747503 learned_addr learned my addr 172.21.15.17:0/2620747503 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/2620747503 >> v2:172.21.15.67:3300/0 conn(0x7fc04809adc0 0x7fc0481330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): get_auth_request con 0x7fc04809adc0 auth_method 0 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.394+0000 7fc04dc5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.399 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04e45c640 1 -- 172.21.15.17:0/2620747503 >> v1:172.21.15.17:6789/0 conn(0x7fc04809a9f0 legacy=0x7fc04812e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 <== mon.0 v1:172.21.15.17:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385216422 0 0) 0x7fc048098fb0 con 0x7fc04809a9f0 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 10 monclient(hunting): my global_id is 254169 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 --> v1:172.21.15.17:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0340034e0 con 0x7fc04809a9f0 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04ec5d640 1 -- 172.21.15.17:0/2620747503 >> v1:172.21.15.67:6789/0 conn(0x7fc048133630 legacy=0x7fc048135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 <== mon.0 v1:172.21.15.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2055028384 0 0) 0x7fc0340034e0 con 0x7fc04809a9f0 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 10 monclient(hunting): handle_auth_done global_id 244282 payload 274 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 1 -- 172.21.15.17:0/2620747503 >> v1:172.21.15.67:6789/0 conn(0x7fc048133630 legacy=0x7fc048135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 1 -- 172.21.15.17:0/2620747503 >> v1:172.21.15.17:6789/0 conn(0x7fc04809a9f0 legacy=0x7fc04812e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 1 monclient: found mon.noname-c 2024-09-15T02:12:41.400 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3300/0 2024-09-15T02:12:41.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04dc5b640 1 -- 172.21.15.17:0/2620747503 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480989f0 con 0x7fc04809adc0 2024-09-15T02:12:41.401 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.396+0000 7fc04d45a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04dc5b640 1 -- 172.21.15.17:0/2620747503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 msgr2=0x7fc0481330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/2620747503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc0481330f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc038009950 tx=0x7fc0380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe678a7285cf94b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044001720 con 0x7fc04809adc0 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: dump: 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.405 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.406 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc0506c0640 5 monclient: authenticate success, global_id 244282 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc0506c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03802acd0 con 0x7fc04809adc0 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 1 -- 172.21.15.17:0/2620747503 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038032650 con 0x7fc04809adc0 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc0506c0640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc0506c0640 4 set_mon_vals no callback set 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.399+0000 7fc04d45a640 10 monclient: dump: 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.407 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 10 monclient: shutdown 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 -- 172.21.15.17:0/2620747503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 msgr2=0x7fc0481330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 --2- 172.21.15.17:0/2620747503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc0481330f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc038009950 tx=0x7fc0380090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 -- 172.21.15.17:0/2620747503 shutdown_connections 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 --2- 172.21.15.17:0/2620747503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc0481330f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.408 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 -- 172.21.15.17:0/2620747503 >> 172.21.15.17:0/2620747503 conn(0x7fc0480a1000 msgr2=0x7fc0480a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 -- 172.21.15.17:0/2620747503 shutdown_connections 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.400+0000 7fc0506c0640 1 -- 172.21.15.17:0/2620747503 wait complete. 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient: monmap: 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:41.409 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 Processor -- start 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 -- start start 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient: init 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-a con 0x7fc04809a9f0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc048138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-b con 0x7fc04809adc0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 0x7fc048138540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): picked mon.noname-c con 0x7fc048133630 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc048130fd0 con 0x7fc04809a9f0 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04809bfe0 con 0x7fc04809adc0 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480983b0 con 0x7fc048133630 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc0506c0640 10 monclient(hunting): authenticate will time out at 2021.397819s 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47106/0 (socket says 172.21.15.17:47106) 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 1 -- 172.21.15.17:0/277729434 learned_addr learned my addr 172.21.15.17:0/277729434 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 10 monclient(hunting): get_auth_request con 0x7fc04809a9f0 auth_method 0 2024-09-15T02:12:41.411 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04e45c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04ec5d640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 0x7fc048138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04ec5d640 10 monclient(hunting): get_auth_request con 0x7fc048133630 auth_method 0 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04ec5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04ec5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.401+0000 7fc04ec5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04ec5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04ec5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04ec5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc048138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 10 monclient(hunting): get_auth_request con 0x7fc04809adc0 auth_method 0 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient(hunting): handle_auth_done global_id 254172 payload 824 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 -- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 msgr2=0x7fc048138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 0x7fc048138540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 -- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 msgr2=0x7fc048138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc048138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 monclient: found mon.noname-a 2024-09-15T02:12:41.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04e45c640 1 -- 172.21.15.17:0/277729434 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc038009390 con 0x7fc04809a9f0 2024-09-15T02:12:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04ec5d640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 0x7fc048138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:41.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.402+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc048138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc04e45c640 1 -- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 msgr2=0x7fc048141550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc04e45c640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc04400c810 tx=0x7fc044001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400ce00 con 0x7fc04809a9f0 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: dump: 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.416 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc044007570 con 0x7fc04809a9f0 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 5 monclient: authenticate success, global_id 254172 2024-09-15T02:12:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0440049f0 con 0x7fc04809a9f0 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 10 monclient: _renew_subs 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc048130ac0 con 0x7fc04809a9f0 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc04cc59640 4 set_mon_vals no callback set 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc03e7fc640 10 monclient: dump: 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.418 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 10 monclient: _renew_subs 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.404+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0481422c0 con 0x7fc04809a9f0 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc04400a640 con 0x7fc04809a9f0 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc0506c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc0506c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc0506c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.419 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0480724a0 con 0x7fc04809a9f0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc03e7fc640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 0x7fc02c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.405+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc04404be60 con 0x7fc04809a9f0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.406+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 0x7fc02c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.406+0000 7fc04dc5b640 10 monclient: get_auth_request con 0x7fc02c038740 auth_method 0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.408+0000 7fc04dc5b640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 0x7fc02c03abf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc048068db0 tx=0x7fc038032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.410+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc044004e60 con 0x7fc04809a9f0 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.410+0000 7fc03e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.420 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.410+0000 7fc03e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:41.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.576+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc04809d6c0 con 0x7fc02c038740 2024-09-15T02:12:41.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.577+0000 7fc03e7fc640 1 -- 172.21.15.17:0/277729434 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc04809d6c0 con 0x7fc02c038740 2024-09-15T02:12:41.578 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 msgr2=0x7fc02c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 0x7fc02c03abf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc048068db0 tx=0x7fc038032000 comp rx=0 tx=0).stop 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 10 monclient: shutdown 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 msgr2=0x7fc048141550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc04400c810 tx=0x7fc044001630 comp rx=0 tx=0).stop 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 shutdown_connections 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fc02c038740 0x7fc02c03abf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fc048133630 0x7fc048138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04809adc0 0x7fc048138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 --2- 172.21.15.17:0/277729434 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc04809a9f0 0x7fc048141550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 >> 172.21.15.17:0/277729434 conn(0x7fc0480a1000 msgr2=0x7fc04809f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 shutdown_connections 2024-09-15T02:12:41.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.579+0000 7fc0506c0640 1 -- 172.21.15.17:0/277729434 wait complete. 2024-09-15T02:12:41.591 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T02:12:41.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.689+0000 7f565c4d4640 10 monclient: get_monmap_and_config 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.689+0000 7f565c4d4640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.689+0000 7f565c4d4640 10 monclient: monmap: 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:41.690606+0000 2024-09-15T02:12:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:41.690606+0000 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:41.692 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.690+0000 7f565c4d4640 1 Processor -- start 2024-09-15T02:12:41.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.690+0000 7f565c4d4640 1 -- start start 2024-09-15T02:12:41.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.690+0000 7f565c4d4640 10 monclient: init 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-b con 0x7f5654067090 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5654067460 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5654137000 0x7f56541393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-c con 0x7f5654137000 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565409e5b0 con 0x7f5654067460 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565409ccb0 con 0x7f5654137000 2024-09-15T02:12:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565409cef0 con 0x7f5654067090 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565c4d4640 10 monclient(hunting): authenticate will time out at 2021.687107s 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:38980/0 (socket says 172.21.15.17:38980) 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 -- 172.21.15.17:0/3767032116 learned_addr learned my addr 172.21.15.17:0/3767032116 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): get_auth_request con 0x7f5654067090 auth_method 0 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.696 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565aa71640 1 --2- 172.21.15.17:0/3767032116 >> v2:172.21.15.67:3300/0 conn(0x7f5654137000 0x7f56541393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565aa71640 10 monclient(hunting): get_auth_request con 0x7f5654137000 auth_method 0 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565aa71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565aa71640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565aa71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient(hunting): handle_auth_done global_id 214577 payload 274 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 -- 172.21.15.17:0/3767032116 >> v2:172.21.15.67:3300/0 conn(0x7f5654137000 msgr2=0x7f56541393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 --2- 172.21.15.17:0/3767032116 >> v2:172.21.15.67:3300/0 conn(0x7f5654137000 0x7f56541393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 -- 172.21.15.17:0/3767032116 >> v1:172.21.15.67:6789/0 conn(0x7f5654067460 legacy=0x7f5654067830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 monclient: found mon.noname-b 2024-09-15T02:12:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.698 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.691+0000 7f565a270640 1 -- 172.21.15.17:0/3767032116 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565409e2d0 con 0x7f5654067090 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565a270640 1 -- 172.21.15.17:0/3767032116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 msgr2=0x7f5654132190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565a270640 1 --2- 172.21.15.17:0/3767032116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5654096f20 tx=0x7f56500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=529697daef5046 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 1 -- 172.21.15.17:0/3767032116 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565002ae80 con 0x7f5654067090 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: dump: 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 5 monclient: authenticate success, global_id 214577 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 1 -- 172.21.15.17:0/3767032116 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650032440 con 0x7f5654067090 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 4 set_mon_vals no callback set 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 1 -- 172.21.15.17:0/3767032116 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56500327c0 con 0x7f5654067090 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565926e640 10 monclient: dump: 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.706 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 10 monclient: shutdown 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 1 -- 172.21.15.17:0/3767032116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 msgr2=0x7f5654132190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.694+0000 7f565c4d4640 1 --2- 172.21.15.17:0/3767032116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5654096f20 tx=0x7f56500091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 -- 172.21.15.17:0/3767032116 shutdown_connections 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 --2- 172.21.15.17:0/3767032116 >> v2:172.21.15.67:3300/0 conn(0x7f5654137000 0x7f56541393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 --2- 172.21.15.17:0/3767032116 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067090 0x7f5654132190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 -- 172.21.15.17:0/3767032116 >> 172.21.15.17:0/3767032116 conn(0x7f56540a1f60 msgr2=0x7f56540a2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 -- 172.21.15.17:0/3767032116 shutdown_connections 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 -- 172.21.15.17:0/3767032116 wait complete. 2024-09-15T02:12:41.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 10 monclient: monmap: 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 Processor -- start 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 1 -- start start 2024-09-15T02:12:41.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.695+0000 7f565c4d4640 10 monclient: init 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 0x7f56541308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-a con 0x7f5654067090 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067460 0x7f56541193c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-c con 0x7f5654067460 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5654137000 0x7f5654119900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): picked mon.noname-b con 0x7f5654137000 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565411d030 con 0x7f5654067090 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565411ccc0 con 0x7f5654137000 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565411aeb0 con 0x7f5654067460 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565c4d4640 10 monclient(hunting): authenticate will time out at 2021.692425s 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 0x7f56541308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067460 0x7f56541193c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 0x7f56541308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47116/0 (socket says 172.21.15.17:47116) 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 -- 172.21.15.17:0/1469684571 learned_addr learned my addr 172.21.15.17:0/1469684571 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): get_auth_request con 0x7f5654067090 auth_method 0 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): get_auth_request con 0x7f5654067460 auth_method 0 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5654137000 0x7f5654119900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): get_auth_request con 0x7f5654137000 auth_method 0 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565aa71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient(hunting): handle_auth_done global_id 254175 payload 824 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067460 msgr2=0x7f56541193c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067460 0x7f56541193c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5654137000 msgr2=0x7f5654119900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5654137000 0x7f5654119900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 monclient: found mon.noname-a 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f565a270640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650009450 con 0x7f5654067090 2024-09-15T02:12:41.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.696+0000 7f5659a6f640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f5654067460 0x7f56541193c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565a270640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 msgr2=0x7f56541308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565a270640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 0x7f56541308d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5650035040 tx=0x7f5650032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650031910 con 0x7f5654067090 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: dump: 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.713 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650031ab0 con 0x7f5654067090 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 5 monclient: authenticate success, global_id 254175 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 10 monclient: _renew_subs 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565411c9e0 con 0x7f5654067090 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650031e70 con 0x7f5654067090 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f56467fc640 10 monclient: dump: 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f5658a6d640 4 set_mon_vals no callback set 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 10 monclient: _renew_subs 2024-09-15T02:12:41.715 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.699+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5654137a40 con 0x7f5654067090 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f565c4d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56540724a0 con 0x7f5654067090 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5650039040 con 0x7f5654067090 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5638038740 0x7f563803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f565006cc10 con 0x7f5654067090 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7f56380416b0 0x7f5638043b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5638044210 con 0x7f56380416b0 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56500712a0 con 0x7f5654067090 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f56467fc640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f565aa71640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7f56380416b0 0x7f5638043b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f565aa71640 10 monclient: get_auth_request con 0x7f56380416b0 auth_method 0 2024-09-15T02:12:41.716 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.700+0000 7f5659a6f640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5638038740 0x7f563803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.701+0000 7f5659a6f640 10 monclient: get_auth_request con 0x7f5638038740 auth_method 0 2024-09-15T02:12:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.703+0000 7f5659a6f640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5638038740 0x7f563803abf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5648007fd0 tx=0x7f5648007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.705+0000 7f565aa71640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7f56380416b0 0x7f5638043b40 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.705+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== osd.2 v2:172.21.15.17:6816/31929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5638044210 con 0x7f56380416b0 2024-09-15T02:12:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 --> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f565411bcc0 con 0x7f56380416b0 2024-09-15T02:12:41.718 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f56467fc640 1 -- 172.21.15.17:0/1469684571 <== osd.2 v2:172.21.15.17:6816/31929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f565411bcc0 con 0x7f56380416b0 2024-09-15T02:12:41.719 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:41.719 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:41.719 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7f56380416b0 msgr2=0x7f5638043b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:6816/31929,v1:172.21.15.17:6817/31929] conn(0x7f56380416b0 0x7f5638043b40 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5638038740 msgr2=0x7f563803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5638038740 0x7f563803abf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5648007fd0 tx=0x7f5648007e10 comp rx=0 tx=0).stop 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 10 monclient: shutdown 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 msgr2=0x7f56541308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565c4d4640 1 --2- 172.21.15.17:0/1469684571 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5654067090 0x7f56541308d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5650035040 tx=0x7f5650032000 comp rx=0 tx=0).stop 2024-09-15T02:12:41.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.717+0000 7f565aa71640 1 -- 172.21.15.17:0/1469684571 reap_dead start 2024-09-15T02:12:41.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.718+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 shutdown_connections 2024-09-15T02:12:41.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.718+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 >> 172.21.15.17:0/1469684571 conn(0x7f56540a1f60 msgr2=0x7f565409b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.718+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 shutdown_connections 2024-09-15T02:12:41.721 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.718+0000 7f565c4d4640 1 -- 172.21.15.17:0/1469684571 wait complete. 2024-09-15T02:12:41.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.825+0000 7fb46c55d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.825+0000 7fb46c55d640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.825+0000 7fb46c55d640 10 monclient: monmap: 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:41.826636+0000 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:41.826636+0000 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.827 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:41.828 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.826+0000 7fb46c55d640 1 Processor -- start 2024-09-15T02:12:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.826+0000 7fb46c55d640 1 -- start start 2024-09-15T02:12:41.830 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.826+0000 7fb46c55d640 10 monclient: init 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4640a5780 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb46409ccf0 0x7fb464069580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fb46409ccf0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb464069ac0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4640a5b50 con 0x7fb464069ac0 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4640a18b0 con 0x7fb46409ccf0 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46409c5f0 con 0x7fb4640a5780 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46c55d640 10 monclient(hunting): authenticate will time out at 2021.823227s 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39000/0 (socket says 172.21.15.17:39000) 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 -- 172.21.15.17:0/2246058522 learned_addr learned my addr 172.21.15.17:0/2246058522 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): get_auth_request con 0x7fb4640a5780 auth_method 0 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb469af8640 1 --2- 172.21.15.17:0/2246058522 >> v2:172.21.15.67:3300/0 conn(0x7fb46409ccf0 0x7fb464069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb469af8640 10 monclient(hunting): get_auth_request con 0x7fb46409ccf0 auth_method 0 2024-09-15T02:12:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb469af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb469af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb469af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_done global_id 214583 payload 274 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 -- 172.21.15.17:0/2246058522 >> v2:172.21.15.67:3300/0 conn(0x7fb46409ccf0 msgr2=0x7fb464069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 --2- 172.21.15.17:0/2246058522 >> v2:172.21.15.67:3300/0 conn(0x7fb46409ccf0 0x7fb464069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 -- 172.21.15.17:0/2246058522 >> v1:172.21.15.67:6789/0 conn(0x7fb464069ac0 legacy=0x7fb464138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 monclient: found mon.noname-b 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.833 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.827+0000 7fb46a2f9640 1 -- 172.21.15.17:0/2246058522 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46409d210 con 0x7fb4640a5780 2024-09-15T02:12:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46a2f9640 1 -- 172.21.15.17:0/2246058522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 msgr2=0x7fb46409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46a2f9640 1 --2- 172.21.15.17:0/2246058522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb460009910 tx=0x7fb4600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67fea01b5bb72f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 1 -- 172.21.15.17:0/2246058522 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46002aea0 con 0x7fb4640a5780 2024-09-15T02:12:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: dump: 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.839 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 1 -- 172.21.15.17:0/2246058522 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460032440 con 0x7fb4640a5780 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46c55d640 5 monclient: authenticate success, global_id 214583 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46c55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46c55d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 1 -- 172.21.15.17:0/2246058522 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4600327e0 con 0x7fb4640a5780 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb46c55d640 4 set_mon_vals no callback set 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:41.840 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.830+0000 7fb4692f7640 10 monclient: dump: 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 10 monclient: shutdown 2024-09-15T02:12:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 -- 172.21.15.17:0/2246058522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 msgr2=0x7fb46409bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 --2- 172.21.15.17:0/2246058522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb460009910 tx=0x7fb4600091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 -- 172.21.15.17:0/2246058522 shutdown_connections 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 --2- 172.21.15.17:0/2246058522 >> v2:172.21.15.67:3300/0 conn(0x7fb46409ccf0 0x7fb464069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 --2- 172.21.15.17:0/2246058522 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb4640a5780 0x7fb46409bc90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 -- 172.21.15.17:0/2246058522 >> 172.21.15.17:0/2246058522 conn(0x7fb464070e10 msgr2=0x7fb4640a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 -- 172.21.15.17:0/2246058522 shutdown_connections 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 -- 172.21.15.17:0/2246058522 wait complete. 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 10 monclient: build_initial_monmap 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 10 monclient: monmap: 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:41.842 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.831+0000 7fb46c55d640 1 Processor -- start 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 -- start start 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient: init 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fb464069ac0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb46409ccf0 0x7fb46413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-a con 0x7fb46409ccf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4640a5780 0x7fb464141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4640a5780 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb464121620 con 0x7fb46409ccf0 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4641225d0 con 0x7fb4640a5780 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4641227d0 con 0x7fb464069ac0 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46c55d640 10 monclient(hunting): authenticate will time out at 2021.828499s 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39012/0 (socket says 172.21.15.17:39012) 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 -- 172.21.15.17:0/1301881079 learned_addr learned my addr 172.21.15.17:0/1301881079 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): get_auth_request con 0x7fb464069ac0 auth_method 0 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb46409ccf0 0x7fb46413d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): get_auth_request con 0x7fb46409ccf0 auth_method 0 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4640a5780 0x7fb464141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): get_auth_request con 0x7fb4640a5780 auth_method 0 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb469af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46aafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient(hunting): handle_auth_done global_id 214586 payload 824 2024-09-15T02:12:41.846 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 -- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4640a5780 msgr2=0x7fb464141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4640a5780 0x7fb464141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 -- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb46409ccf0 msgr2=0x7fb46413d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb46409ccf0 0x7fb46413d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient: _finish_hunting 0 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 monclient: found mon.noname-c 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.832+0000 7fb46a2f9640 1 -- 172.21.15.17:0/1301881079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460009450 con 0x7fb464069ac0 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46a2f9640 1 -- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 msgr2=0x7fb46413dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46a2f9640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb460009910 tx=0x7fb460032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46002aea0 con 0x7fb464069ac0 2024-09-15T02:12:41.847 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: dump: 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.848 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: _finish_auth 0 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 5 monclient: authenticate success, global_id 214586 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460031c70 con 0x7fb464069ac0 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 10 monclient: _renew_subs 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb464122d90 con 0x7fb464069ac0 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46003d440 con 0x7fb464069ac0 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb45a7fc640 10 monclient: dump: 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 10 monclient: _renew_subs 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464122b60 con 0x7fb464069ac0 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.835+0000 7fb468af6640 4 set_mon_vals no callback set 2024-09-15T02:12:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb46c55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb46c55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb46c55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4640724a0 con 0x7fb464069ac0 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb460039040 con 0x7fb464069ac0 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb45a7fc640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 0x7fb44803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb46006c4b0 con 0x7fb464069ac0 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb469af8640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 0x7fb44803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.836+0000 7fb469af8640 10 monclient: get_auth_request con 0x7fb448038740 auth_method 0 2024-09-15T02:12:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.839+0000 7fb469af8640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 0x7fb44803abf0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb454007ba0 tx=0x7fb454008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.840+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb46002f2a0 con 0x7fb464069ac0 2024-09-15T02:12:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.840+0000 7fb45a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:41.840+0000 7fb45a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:42.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.005+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb464122990 con 0x7fb448038740 2024-09-15T02:12:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.006+0000 7fb45a7fc640 1 -- 172.21.15.17:0/1301881079 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb454010d50 con 0x7fb448038740 2024-09-15T02:12:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T02:12:42.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 msgr2=0x7fb44803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 0x7fb44803abf0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb454007ba0 tx=0x7fb454008040 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 10 monclient: shutdown 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 msgr2=0x7fb46413dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb460009910 tx=0x7fb460032000 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 shutdown_connections 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb448038740 0x7fb44803abf0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb46409ccf0 0x7fb46413d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4640a5780 0x7fb464141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 --2- 172.21.15.17:0/1301881079 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb464069ac0 0x7fb46413dda0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 >> 172.21.15.17:0/1301881079 conn(0x7fb464070e10 msgr2=0x7fb46409af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 shutdown_connections 2024-09-15T02:12:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.008+0000 7fb46c55d640 1 -- 172.21.15.17:0/1301881079 wait complete. 2024-09-15T02:12:42.020 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T02:12:42.020 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.118+0000 7ff4dc0c9640 10 monclient: get_monmap_and_config 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.118+0000 7ff4dc0c9640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.118+0000 7ff4dc0c9640 10 monclient: monmap: 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:42.119124+0000 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:42.119124+0000 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:42.120 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:42.121 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:42.121 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:42.121 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:42.121 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.118+0000 7ff4dc0c9640 1 Processor -- start 2024-09-15T02:12:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 -- start start 2024-09-15T02:12:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient: init 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 0x7ff4d409a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409cf80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4d406ef20 0x7ff4d406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d406ef20 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4d40983b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d4099940 con 0x7ff4d40983b0 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d409c9d0 con 0x7ff4d406ef20 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.126 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40995d0 con 0x7ff4d409cf80 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4dc0c9640 10 monclient(hunting): authenticate will time out at 2022.115619s 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 0x7ff4d409a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4da666640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff4d40983b0 0x7ff4d40987f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59488/0 (socket says 172.21.15.17:59488) 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4da666640 1 -- 172.21.15.17:0/2690671531 learned_addr learned my addr 172.21.15.17:0/2690671531 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): get_auth_request con 0x7ff4d409cf80 auth_method 0 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9664640 1 --2- 172.21.15.17:0/2690671531 >> v2:172.21.15.67:3300/0 conn(0x7ff4d406ef20 0x7ff4d406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.127 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9664640 10 monclient(hunting): get_auth_request con 0x7ff4d406ef20 auth_method 0 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9664640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.119+0000 7ff4d9664640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_done global_id 214589 payload 274 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 1 -- 172.21.15.17:0/2690671531 >> v2:172.21.15.67:3300/0 conn(0x7ff4d406ef20 msgr2=0x7ff4d406f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 1 --2- 172.21.15.17:0/2690671531 >> v2:172.21.15.67:3300/0 conn(0x7ff4d406ef20 0x7ff4d406f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 1 -- 172.21.15.17:0/2690671531 >> v1:172.21.15.17:6789/0 conn(0x7ff4d40983b0 legacy=0x7ff4d40987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 1 monclient: found mon.noname-b 2024-09-15T02:12:42.128 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.120+0000 7ff4d9e65640 1 -- 172.21.15.17:0/2690671531 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4097600 con 0x7ff4d409cf80 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.122+0000 7ff4d9e65640 1 -- 172.21.15.17:0/2690671531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 msgr2=0x7ff4d409a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.122+0000 7ff4d9e65640 1 --2- 172.21.15.17:0/2690671531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 0x7ff4d409a020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009e60 tx=0x7ff4c4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be449dde9ca18fd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 1 -- 172.21.15.17:0/2690671531 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4007760 con 0x7ff4d409cf80 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: dump: 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.129 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 1 -- 172.21.15.17:0/2690671531 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c402d050 con 0x7ff4d409cf80 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 5 monclient: authenticate success, global_id 214589 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 1 -- 172.21.15.17:0/2690671531 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4007af0 con 0x7ff4d409cf80 2024-09-15T02:12:42.130 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 4 set_mon_vals no callback set 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4d8e63640 10 monclient: dump: 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.131 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 10 monclient: shutdown 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/2690671531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 msgr2=0x7ff4d409a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/2690671531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 0x7ff4d409a020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009e60 tx=0x7ff4c4009780 comp rx=0 tx=0).stop 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/2690671531 shutdown_connections 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/2690671531 >> v2:172.21.15.67:3300/0 conn(0x7ff4d406ef20 0x7ff4d406f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/2690671531 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d409cf80 0x7ff4d409a020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/2690671531 >> 172.21.15.17:0/2690671531 conn(0x7ff4d40a1270 msgr2=0x7ff4d40a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/2690671531 shutdown_connections 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.123+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/2690671531 wait complete. 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient: monmap: 2024-09-15T02:12:42.132 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 Processor -- start 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 -- start start 2024-09-15T02:12:42.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient: init 2024-09-15T02:12:42.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:42.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 0x7ff4d412f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d406ef20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 0x7ff4d4119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d40983b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d409cf80 0x7ff4d4119840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409cf80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d409d470 con 0x7ff4d40983b0 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d41307b0 con 0x7ff4d409cf80 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d411cbc0 con 0x7ff4d406ef20 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.124+0000 7ff4dc0c9640 10 monclient(hunting): authenticate will time out at 2022.120986s 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 0x7ff4d412f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 0x7ff4d4119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 0x7ff4d412f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39016/0 (socket says 172.21.15.17:39016) 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 1 -- 172.21.15.17:0/3941582472 learned_addr learned my addr 172.21.15.17:0/3941582472 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): get_auth_request con 0x7ff4d406ef20 auth_method 0 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 0x7ff4d4119300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47124/0 (socket says 172.21.15.17:47124) 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): get_auth_request con 0x7ff4d40983b0 auth_method 0 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d409cf80 0x7ff4d4119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): get_auth_request con 0x7ff4d409cf80 auth_method 0 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4da666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient(hunting): handle_auth_done global_id 254181 payload 824 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 msgr2=0x7ff4d412f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 0x7ff4d412f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d409cf80 msgr2=0x7ff4d4119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d409cf80 0x7ff4d4119840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 monclient: found mon.noname-a 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9664640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4009a20 con 0x7ff4d40983b0 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.125+0000 7ff4d9e65640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff4d406ef20 0x7ff4d412f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d9664640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 msgr2=0x7ff4d4119300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d9664640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 0x7ff4d4119300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800e990 tx=0x7ff4c80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800cb50 con 0x7ff4d40983b0 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: dump: 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.142 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4dc0c9640 5 monclient: authenticate success, global_id 254181 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c800ccb0 con 0x7ff4d40983b0 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4dc0c9640 10 monclient: _renew_subs 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4dc0c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d411adb0 con 0x7ff4d40983b0 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8013470 con 0x7ff4d40983b0 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d1ffb640 10 monclient: dump: 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4dc0c9640 10 monclient: _renew_subs 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.127+0000 7ff4d3fff640 4 set_mon_vals no callback set 2024-09-15T02:12:42.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4dc0c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d41320b0 con 0x7ff4d40983b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4dc0c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff4d40724a0 con 0x7ff4d40983b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.128+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff4c8013860 con 0x7ff4d40983b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4c0038740 0x7ff4c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff4c804c8e0 con 0x7ff4d40983b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7ff4c00416b0 0x7ff4c0043b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4c0044210 con 0x7ff4c00416b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4c80512a0 con 0x7ff4d40983b0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d1ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d9e65640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4c0038740 0x7ff4c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4d9e65640 10 monclient: get_auth_request con 0x7ff4c0038740 auth_method 0 2024-09-15T02:12:42.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4da666640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7ff4c00416b0 0x7ff4c0043b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.129+0000 7ff4da666640 10 monclient: get_auth_request con 0x7ff4c00416b0 auth_method 0 2024-09-15T02:12:42.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.132+0000 7ff4d9e65640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4c0038740 0x7ff4c003abf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40097c0 tx=0x7ff4c4037000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.134+0000 7ff4da666640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7ff4c00416b0 0x7ff4c0043b40 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.134+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== osd.3 v2:172.21.15.67:6816/31880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff4c0044210 con 0x7ff4c00416b0 2024-09-15T02:12:42.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.146+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 --> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff4d412fd00 con 0x7ff4c00416b0 2024-09-15T02:12:42.148 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.146+0000 7ff4d1ffb640 1 -- 172.21.15.17:0/3941582472 <== osd.3 v2:172.21.15.67:6816/31880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff4d412fd00 con 0x7ff4c00416b0 2024-09-15T02:12:42.148 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:42.148 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:42.148 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7ff4c00416b0 msgr2=0x7ff4c0043b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6816/31880,v1:172.21.15.67:6817/31880] conn(0x7ff4c00416b0 0x7ff4c0043b40 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4c0038740 msgr2=0x7ff4c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff4c0038740 0x7ff4c003abf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40097c0 tx=0x7ff4c4037000 comp rx=0 tx=0).stop 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 10 monclient: shutdown 2024-09-15T02:12:42.149 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 msgr2=0x7ff4d4119300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 --2- 172.21.15.17:0/3941582472 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff4d40983b0 0x7ff4d4119300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800e990 tx=0x7ff4c80090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4da666640 1 -- 172.21.15.17:0/3941582472 reap_dead start 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 shutdown_connections 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 >> 172.21.15.17:0/3941582472 conn(0x7ff4d40a1270 msgr2=0x7ff4d409fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 shutdown_connections 2024-09-15T02:12:42.150 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.147+0000 7ff4dc0c9640 1 -- 172.21.15.17:0/3941582472 wait complete. 2024-09-15T02:12:42.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 10 monclient: get_monmap_and_config 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 10 monclient: monmap: 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:42.254992+0000 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:42.254992+0000 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.256 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 1 Processor -- start 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 1 -- start start 2024-09-15T02:12:42.257 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.254+0000 7ff92f6d4640 10 monclient: init 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 0x7ff928132040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9280a58b0 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 0x7ff9280984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9280980a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9280989e0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9280a01e0 con 0x7ff9280989e0 2024-09-15T02:12:42.259 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9280a1580 con 0x7ff9280a58b0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280a5c80 con 0x7ff9280980a0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92f6d4640 10 monclient(hunting): authenticate will time out at 2022.251471s 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 0x7ff928132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 0x7ff9280984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92dc71640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7ff9280989e0 0x7ff928068ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59502/0 (socket says 172.21.15.17:59502) 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92dc71640 1 -- 172.21.15.17:0/1489325899 learned_addr learned my addr 172.21.15.17:0/1489325899 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): get_auth_request con 0x7ff9280a58b0 auth_method 0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): get_auth_request con 0x7ff9280980a0 auth_method 0 2024-09-15T02:12:42.260 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_done global_id 214595 payload 274 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 -- 172.21.15.17:0/1489325899 >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 msgr2=0x7ff928132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 --2- 172.21.15.17:0/1489325899 >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 0x7ff928132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 -- 172.21.15.17:0/1489325899 >> v1:172.21.15.17:6789/0 conn(0x7ff9280989e0 legacy=0x7ff928068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 monclient: found mon.noname-b 2024-09-15T02:12:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92cc6f640 1 -- 172.21.15.17:0/1489325899 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9281326c0 con 0x7ff9280980a0 2024-09-15T02:12:42.262 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.255+0000 7ff92d470640 1 --2- 172.21.15.17:0/1489325899 >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 0x7ff928132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.258+0000 7ff92cc6f640 1 -- 172.21.15.17:0/1489325899 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 msgr2=0x7ff9280984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.258+0000 7ff92cc6f640 1 --2- 172.21.15.17:0/1489325899 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 0x7ff9280984a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009880 tx=0x7ff91c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=464d37d8d5e73cca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.258+0000 7ff927fff640 1 -- 172.21.15.17:0/1489325899 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c02aea0 con 0x7ff9280980a0 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: dump: 2024-09-15T02:12:42.267 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 1 -- 172.21.15.17:0/1489325899 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c032440 con 0x7ff9280980a0 2024-09-15T02:12:42.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 5 monclient: authenticate success, global_id 214595 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 4 set_mon_vals no callback set 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 1 -- 172.21.15.17:0/1489325899 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0327e0 con 0x7ff9280980a0 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff927fff640 10 monclient: dump: 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.269 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 10 monclient: shutdown 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 -- 172.21.15.17:0/1489325899 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 msgr2=0x7ff9280984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/1489325899 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 0x7ff9280984a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009880 tx=0x7ff91c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 -- 172.21.15.17:0/1489325899 shutdown_connections 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/1489325899 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9280980a0 0x7ff9280984a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/1489325899 >> v2:172.21.15.17:3300/0 conn(0x7ff9280a58b0 0x7ff928132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 -- 172.21.15.17:0/1489325899 >> 172.21.15.17:0/1489325899 conn(0x7ff928070ed0 msgr2=0x7ff9280a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 -- 172.21.15.17:0/1489325899 shutdown_connections 2024-09-15T02:12:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.259+0000 7ff92f6d4640 1 -- 172.21.15.17:0/1489325899 wait complete. 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.260+0000 7ff92f6d4640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.260+0000 7ff92f6d4640 10 monclient: monmap: 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:42.271 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.260+0000 7ff92f6d4640 1 Processor -- start 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.260+0000 7ff92f6d4640 1 -- start start 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.260+0000 7ff92f6d4640 10 monclient: init 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 0x7ff92809aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9280980a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9280a58b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9281311a0 0x7ff92811b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): picked mon.noname-c con 0x7ff9281311a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.272 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92811e060 con 0x7ff9280a58b0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92811cc50 con 0x7ff9280980a0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff928098dc0 con 0x7ff9281311a0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92f6d4640 10 monclient(hunting): authenticate will time out at 2022.257122s 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9281311a0 0x7ff92811b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47144/0 (socket says 172.21.15.17:47144) 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 -- 172.21.15.17:0/4271937484 learned_addr learned my addr 172.21.15.17:0/4271937484 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): get_auth_request con 0x7ff9280a58b0 auth_method 0 2024-09-15T02:12:42.273 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): get_auth_request con 0x7ff9281311a0 auth_method 0 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.274 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92dc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient(hunting): handle_auth_done global_id 254187 payload 824 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92d470640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 0x7ff92809aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 -- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9281311a0 msgr2=0x7ff92811b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9281311a0 0x7ff92811b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 -- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 msgr2=0x7ff92809aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 0x7ff92809aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 monclient: found mon.noname-a 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92cc6f640 1 -- 172.21.15.17:0/4271937484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918009a20 con 0x7ff9280a58b0 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92d470640 10 monclient: get_auth_request con 0x7ff9280980a0 auth_method 0 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.261+0000 7ff92d470640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 0x7ff92809aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:12:42.275 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92cc6f640 1 -- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 msgr2=0x7ff928130c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92cc6f640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009360 tx=0x7ff91c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c02aea0 con 0x7ff9280a58b0 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 10 monclient: dump: 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.276 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c031c70 con 0x7ff9280a58b0 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92f6d4640 5 monclient: authenticate success, global_id 254187 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92f6d4640 10 monclient: _renew_subs 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92f6d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.263+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c009390 con 0x7ff9280a58b0 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c03d440 con 0x7ff9280a58b0 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:42.277 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 10 monclient: dump: 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff92f6d4640 10 monclient: _renew_subs 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9277fe640 4 set_mon_vals no callback set 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff92f6d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9281420c0 con 0x7ff9280a58b0 2024-09-15T02:12:42.278 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.264+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff91c039040 con 0x7ff9280a58b0 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92f6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92f6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92f6d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9280724a0 con 0x7ff9280a58b0 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff9257fa640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 0x7ff91403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff91c06c410 con 0x7ff9280a58b0 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92d470640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 0x7ff91403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.279 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.265+0000 7ff92d470640 10 monclient: get_auth_request con 0x7ff914038740 auth_method 0 2024-09-15T02:12:42.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.267+0000 7ff92d470640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 0x7ff91403abf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff9180099f0 tx=0x7ff91800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.269+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff91c02f4a0 con 0x7ff9280a58b0 2024-09-15T02:12:42.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.269+0000 7ff9257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.280 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.269+0000 7ff9257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:42.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.434+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff92811ced0 con 0x7ff914038740 2024-09-15T02:12:42.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.435+0000 7ff9257fa640 1 -- 172.21.15.17:0/4271937484 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff918007e40 con 0x7ff914038740 2024-09-15T02:12:42.436 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T02:12:42.438 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 msgr2=0x7ff91403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 0x7ff91403abf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff9180099f0 tx=0x7ff91800b040 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 10 monclient: shutdown 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 msgr2=0x7ff928130c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009360 tx=0x7ff91c032000 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 shutdown_connections 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff914038740 0x7ff91403abf0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff9280a58b0 0x7ff928130c60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff9281311a0 0x7ff92811b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 --2- 172.21.15.17:0/4271937484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9280980a0 0x7ff92809aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 >> 172.21.15.17:0/4271937484 conn(0x7ff928070ed0 msgr2=0x7ff9280a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.439 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 shutdown_connections 2024-09-15T02:12:42.440 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.437+0000 7ff92f6d4640 1 -- 172.21.15.17:0/4271937484 wait complete. 2024-09-15T02:12:42.449 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T02:12:42.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.547+0000 7faeca39d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.547+0000 7faeca39d640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.547+0000 7faeca39d640 10 monclient: monmap: 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:42.548495+0000 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:42.548495+0000 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.549 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:42.550 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:42.550 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:42.550 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:42.550 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:42.550 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 1 Processor -- start 2024-09-15T02:12:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 1 -- start start 2024-09-15T02:12:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 10 monclient: init 2024-09-15T02:12:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.548+0000 7faeca39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faec406eb30 0x7faec406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-c con 0x7faec406eb30 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-b con 0x7faec406f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faec409c4b0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec409d090 con 0x7faec409c4b0 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec4069cd0 con 0x7faec406eb30 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec406a5e0 con 0x7faec406f440 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faeca39d640 10 monclient(hunting): authenticate will time out at 2022.545071s 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.553 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39036/0 (socket says 172.21.15.17:39036) 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 -- 172.21.15.17:0/2846738669 learned_addr learned my addr 172.21.15.17:0/2846738669 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): get_auth_request con 0x7faec406f440 auth_method 0 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec893a640 1 --1- 172.21.15.17:0/2846738669 >> v1:172.21.15.17:6789/0 conn(0x7faec409c4b0 0x7faec409d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59504/0 (socket says 172.21.15.17:59504) 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 1 --2- 172.21.15.17:0/2846738669 >> v2:172.21.15.67:3300/0 conn(0x7faec406eb30 0x7faec406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): get_auth_request con 0x7faec406eb30 auth_method 0 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.554 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient(hunting): handle_auth_done global_id 214601 payload 274 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 -- 172.21.15.17:0/2846738669 >> v2:172.21.15.67:3300/0 conn(0x7faec406eb30 msgr2=0x7faec406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 --2- 172.21.15.17:0/2846738669 >> v2:172.21.15.67:3300/0 conn(0x7faec406eb30 0x7faec406ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 -- 172.21.15.17:0/2846738669 >> v1:172.21.15.17:6789/0 conn(0x7faec409c4b0 legacy=0x7faec409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 monclient: found mon.noname-b 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.555 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.549+0000 7faec37fe640 1 -- 172.21.15.17:0/2846738669 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec409f740 con 0x7faec406f440 2024-09-15T02:12:42.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec37fe640 1 -- 172.21.15.17:0/2846738669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec409bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec37fe640 1 --2- 172.21.15.17:0/2846738669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009910 tx=0x7faeb80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8565f3433af5eebe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.563 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 1 -- 172.21.15.17:0/2846738669 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8032440 con 0x7faec406f440 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: dump: 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.564 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 1 -- 172.21.15.17:0/2846738669 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb80325a0 con 0x7faec406f440 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 1 -- 172.21.15.17:0/2846738669 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8032980 con 0x7faec406f440 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 5 monclient: authenticate success, global_id 214601 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 4 set_mon_vals no callback set 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.565 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faec2ffd640 10 monclient: dump: 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.566 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 10 monclient: shutdown 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 1 -- 172.21.15.17:0/2846738669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec409bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.552+0000 7faeca39d640 1 --2- 172.21.15.17:0/2846738669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009910 tx=0x7faeb80091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 -- 172.21.15.17:0/2846738669 shutdown_connections 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 --2- 172.21.15.17:0/2846738669 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec409bf70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 --2- 172.21.15.17:0/2846738669 >> v2:172.21.15.67:3300/0 conn(0x7faec406eb30 0x7faec406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 -- 172.21.15.17:0/2846738669 >> 172.21.15.17:0/2846738669 conn(0x7faec40711d0 msgr2=0x7faec40a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 -- 172.21.15.17:0/2846738669 shutdown_connections 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 -- 172.21.15.17:0/2846738669 wait complete. 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 10 monclient: monmap: 2024-09-15T02:12:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.568 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.553+0000 7faeca39d640 1 Processor -- start 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 -- start start 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient: init 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faec4133c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-b con 0x7faec406eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-c con 0x7faec406f440 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faec411df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): picked mon.noname-a con 0x7faec409c4b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.569 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec4120d30 con 0x7faec409c4b0 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec411f7f0 con 0x7faec406eb30 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec409cd60 con 0x7faec406f440 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faeca39d640 10 monclient(hunting): authenticate will time out at 2022.550574s 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39050/0 (socket says 172.21.15.17:39050) 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 learned_addr learned my addr 172.21.15.17:0/1544521573 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): get_auth_request con 0x7faec406f440 auth_method 0 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faec411df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): get_auth_request con 0x7faec409c4b0 auth_method 0 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec3fff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faec4133c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec3fff640 10 monclient(hunting): get_auth_request con 0x7faec406eb30 auth_method 0 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient(hunting): handle_auth_done global_id 214604 payload 824 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 msgr2=0x7faec4133c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faec4133c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 msgr2=0x7faec411df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faec411df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 monclient: found mon.noname-c 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.554+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8009450 con 0x7faec406f440 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec411d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009970 tx=0x7faeb8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.572 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec411d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80319c0 con 0x7faec406f440 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec411d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009970 tx=0x7faeb8032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009970 tx=0x7faeb8032000 comp rx=0 tx=0).stop 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: dump: 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.573 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 msgr2=0x7faec411d9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faec406f440 0x7faec411d9e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faeb00056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.574 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): picked mon.b con 0x7faec406eb30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faeb0006030 0x7faeb00084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): picked mon.c con 0x7faeb0006030 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faeb000afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): picked mon.a con 0x7faec409c4b0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb000b950 con 0x7faec409c4b0 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb000ba90 con 0x7faec406eb30 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb000bbd0 con 0x7faeb0006030 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faeb8031b40 con 0x7faec406f440 2024-09-15T02:12:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeabfff640 10 monclient(hunting): discarding stray monitor message config(0 keys) v1 2024-09-15T02:12:42.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faeb000afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.576 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faeb0006030 0x7faeb00084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): get_auth_request con 0x7faeb0006030 auth_method 0 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): get_auth_request con 0x7faec409c4b0 auth_method 0 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.577 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faeb00056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): get_auth_request con 0x7faec406eb30 auth_method 0 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.578 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient(hunting): handle_auth_done global_id 214604 payload 293 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faeb0006030 msgr2=0x7faeb00084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faeb0006030 0x7faeb00084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 msgr2=0x7faeb00056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.579 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec406eb30 0x7faeb00056b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 monclient: found mon.a 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7faeb000be00 con 0x7faec409c4b0 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec893a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeca39d640 5 monclient: authenticate success, global_id 214604 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7faeb0006030 0x7faeb00084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeca39d640 10 monclient: _renew_subs 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeca39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.580 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.557+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec4134c40 con 0x7faec409c4b0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.558+0000 7faeca39d640 10 monclient: _renew_subs 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.558+0000 7faeca39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.558+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec4141ef0 con 0x7faec409c4b0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.558+0000 7faeca39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.558+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faec411f350 con 0x7faec409c4b0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faec893a640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 msgr2=0x7faeb000afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faec893a640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faeb000afd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faebc007990 tx=0x7faebc004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_get_version(what=osdmap handle=2) v1 -- 0x7faeb8031b40 con 0x7faec409c4b0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc007300 con 0x7faec409c4b0 2024-09-15T02:12:42.581 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 10 monclient: dump: 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.582 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8031b40 con 0x7faec409c4b0 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faeabfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.560+0000 7faec27fc640 4 set_mon_vals no callback set 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faebc01d460 con 0x7faec409c4b0 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faec406f440 0x7faeb0042ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 4 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7faebc04c300 con 0x7faec409c4b0 2024-09-15T02:12:42.583 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7faeb0049770 0x7faeb004bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faeb004c2d0 con 0x7faeb0049770 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 5 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7faebc04c5a0 con 0x7faec409c4b0 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_get_version_reply(handle=2 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7faebc04c7c0 con 0x7faec409c4b0 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faeabfff640 10 monclient: handle_get_version_reply finishing 2 version 257 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faec3fff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faec406f440 0x7faeb0042ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faec3fff640 10 monclient: get_auth_request con 0x7faec406f440 auth_method 0 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7faeb0049770 0x7faeb004bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.561+0000 7faec37fe640 10 monclient: get_auth_request con 0x7faeb0049770 auth_method 0 2024-09-15T02:12:42.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.563+0000 7faec3fff640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faec406f440 0x7faeb0042ea0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009cb0 tx=0x7faeb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.566+0000 7faec37fe640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7faeb0049770 0x7faeb004bc00 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.567+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== osd.4 v2:172.21.15.67:6802/31881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7faeb004c2d0 con 0x7faeb0049770 2024-09-15T02:12:42.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.578+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 --> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faec40724a0 con 0x7faeb0049770 2024-09-15T02:12:42.585 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeabfff640 1 -- 172.21.15.17:0/1544521573 <== osd.4 v2:172.21.15.67:6802/31881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7faec40724a0 con 0x7faeb0049770 2024-09-15T02:12:42.586 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:42.586 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:42.586 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7faeb0049770 msgr2=0x7faeb004bc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6802/31881,v1:172.21.15.67:6804/31881] conn(0x7faeb0049770 0x7faeb004bc00 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faec406f440 msgr2=0x7faeb0042ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7faec406f440 0x7faeb0042ea0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009cb0 tx=0x7faeb4008040 comp rx=0 tx=0).stop 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 10 monclient: shutdown 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 msgr2=0x7faeb000afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 --2- 172.21.15.17:0/1544521573 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7faec409c4b0 0x7faeb000afd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faebc007990 tx=0x7faebc004e20 comp rx=0 tx=0).stop 2024-09-15T02:12:42.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faec893a640 1 -- 172.21.15.17:0/1544521573 reap_dead start 2024-09-15T02:12:42.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 shutdown_connections 2024-09-15T02:12:42.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 >> 172.21.15.17:0/1544521573 conn(0x7faec40711d0 msgr2=0x7faec4134e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 shutdown_connections 2024-09-15T02:12:42.588 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.579+0000 7faeca39d640 1 -- 172.21.15.17:0/1544521573 wait complete. 2024-09-15T02:12:42.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T02:12:42.688 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 10 monclient: get_monmap_and_config 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 10 monclient: monmap: 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:42.687903+0000 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:42.687903+0000 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:42.689 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 1 Processor -- start 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 1 -- start start 2024-09-15T02:12:42.690 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.687+0000 7fb3e26a8640 10 monclient: init 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 0x7fb3dc09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3dc0a0280 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3dc09f1e0 0x7fb3dc138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb3dc09f1e0 addr v2:172.21.15.67:3300/0 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0a0040 con 0x7fb3dc0a0280 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.692 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc09bff0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0a14a0 con 0x7fb3dc131b20 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e26a8640 10 monclient(hunting): authenticate will time out at 2022.684391s 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 0x7fb3dc09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3db7fe640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fb3dc0a0280 0x7fb3dc131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:59520/0 (socket says 172.21.15.17:59520) 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3db7fe640 1 -- 172.21.15.17:0/1800425064 learned_addr learned my addr 172.21.15.17:0/1800425064 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc131b20 auth_method 0 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.693 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e0c45640 1 --2- 172.21.15.17:0/1800425064 >> v2:172.21.15.67:3300/0 conn(0x7fb3dc09f1e0 0x7fb3dc138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e0c45640 10 monclient(hunting): get_auth_request con 0x7fb3dc09f1e0 auth_method 0 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e0c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e0c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3e0c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_done global_id 214607 payload 274 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 -- 172.21.15.17:0/1800425064 >> v2:172.21.15.67:3300/0 conn(0x7fb3dc09f1e0 msgr2=0x7fb3dc138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 --2- 172.21.15.17:0/1800425064 >> v2:172.21.15.67:3300/0 conn(0x7fb3dc09f1e0 0x7fb3dc138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 -- 172.21.15.17:0/1800425064 >> v1:172.21.15.17:6789/0 conn(0x7fb3dc0a0280 legacy=0x7fb3dc131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 monclient: found mon.noname-b 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.694 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.688+0000 7fb3dbfff640 1 -- 172.21.15.17:0/1800425064 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc132600 con 0x7fb3dc131b20 2024-09-15T02:12:42.699 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3dbfff640 1 -- 172.21.15.17:0/1800425064 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 msgr2=0x7fb3dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3dbfff640 1 --2- 172.21.15.17:0/1800425064 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 0x7fb3dc09eca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009850 tx=0x7fb3cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c46b5551b3bec58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 1 -- 172.21.15.17:0/1800425064 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc02adf0 con 0x7fb3dc131b20 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: dump: 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 1 -- 172.21.15.17:0/1800425064 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc032440 con 0x7fb3dc131b20 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3e26a8640 5 monclient: authenticate success, global_id 214607 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3e26a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3e26a8640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 1 -- 172.21.15.17:0/1800425064 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0327e0 con 0x7fb3dc131b20 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.701 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3e26a8640 4 set_mon_vals no callback set 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.691+0000 7fb3daffd640 10 monclient: dump: 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 10 monclient: shutdown 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 -- 172.21.15.17:0/1800425064 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 msgr2=0x7fb3dc09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/1800425064 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 0x7fb3dc09eca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009850 tx=0x7fb3cc0090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 -- 172.21.15.17:0/1800425064 shutdown_connections 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/1800425064 >> v2:172.21.15.67:3300/0 conn(0x7fb3dc09f1e0 0x7fb3dc138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/1800425064 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc131b20 0x7fb3dc09eca0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 -- 172.21.15.17:0/1800425064 >> 172.21.15.17:0/1800425064 conn(0x7fb3dc071270 msgr2=0x7fb3dc0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 -- 172.21.15.17:0/1800425064 shutdown_connections 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 1 -- 172.21.15.17:0/1800425064 wait complete. 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.692+0000 7fb3e26a8640 10 monclient: monmap: 2024-09-15T02:12:42.703 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 Processor -- start 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 -- start start 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient: init 2024-09-15T02:12:42.704 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb3dc09f1e0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb3dc0a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3dc131b20 0x7fb3dc132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc131b20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:42.705 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0a0a20 con 0x7fb3dc0a0280 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc1384a0 con 0x7fb3dc131b20 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc138ea0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3e26a8640 10 monclient(hunting): authenticate will time out at 2022.689860s 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.706 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:47186/0 (socket says 172.21.15.17:47186) 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 1 -- 172.21.15.17:0/683553794 learned_addr learned my addr 172.21.15.17:0/683553794 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 10 monclient(hunting): get_auth_request con 0x7fb3dc0a0280 auth_method 0 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:39066/0 (socket says 172.21.15.17:39066) 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.693+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc09f1e0 auth_method 0 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.707 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3dc131b20 0x7fb3dc132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): get_auth_request con 0x7fb3dc131b20 auth_method 0 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3e0c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.708 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_done global_id 214610 payload 824 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 -- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3dc131b20 msgr2=0x7fb3dc132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3dc131b20 0x7fb3dc132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 -- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 msgr2=0x7fb3dc132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 monclient: found mon.noname-c 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3dbfff640 1 -- 172.21.15.17:0/683553794 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc009390 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.694+0000 7fb3db7fe640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3dbfff640 1 -- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 msgr2=0x7fb3dc136a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3dbfff640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc031000 tx=0x7fb3cc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031bb0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: dump: 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.710 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc031d30 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031ea0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3c3fff640 10 monclient: dump: 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.711 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 5 monclient: authenticate success, global_id 214610 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 10 monclient: _renew_subs 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc1360b0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3da7fc640 4 set_mon_vals no callback set 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 10 monclient: _renew_subs 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.696+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc135e50 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.697+0000 7fb3e26a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.712 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.697+0000 7fb3e26a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.697+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb3cc03e8f0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.697+0000 7fb3e26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.697+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc0724a0 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.698+0000 7fb3c3fff640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 0x7fb3c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.698+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb3cc06c760 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.698+0000 7fb3db7fe640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 0x7fb3c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.713 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.698+0000 7fb3db7fe640 10 monclient: get_auth_request con 0x7fb3c8038740 auth_method 0 2024-09-15T02:12:42.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.701+0000 7fb3db7fe640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 0x7fb3c803abf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006fd0 tx=0x7fb3d0004960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.702+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb3cc030700 con 0x7fb3dc09f1e0 2024-09-15T02:12:42.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.702+0000 7fb3c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:42.714 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.702+0000 7fb3c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:42.867 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.865+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3dc1363f0 con 0x7fb3c8038740 2024-09-15T02:12:42.868 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.866+0000 7fb3c3fff640 1 -- 172.21.15.17:0/683553794 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb3dc1363f0 con 0x7fb3c8038740 2024-09-15T02:12:42.868 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 msgr2=0x7fb3c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 0x7fb3c803abf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006fd0 tx=0x7fb3d0004960 comp rx=0 tx=0).stop 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 10 monclient: shutdown 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 msgr2=0x7fb3dc136a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.870 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc031000 tx=0x7fb3cc032000 comp rx=0 tx=0).stop 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 shutdown_connections 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fb3c8038740 0x7fb3c803abf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb3dc0a0280 0x7fb3dc132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3dc131b20 0x7fb3dc132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 --2- 172.21.15.17:0/683553794 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fb3dc09f1e0 0x7fb3dc136a90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.868+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 >> 172.21.15.17:0/683553794 conn(0x7fb3dc071270 msgr2=0x7fb3dc0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.869+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 shutdown_connections 2024-09-15T02:12:42.871 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.869+0000 7fb3e26a8640 1 -- 172.21.15.17:0/683553794 wait complete. 2024-09-15T02:12:42.881 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T02:12:42.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:12:42.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.982+0000 7f5563caf640 10 monclient: get_monmap_and_config 2024-09-15T02:12:42.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.982+0000 7f5563caf640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.984 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.983+0000 7f5563caf640 10 monclient: monmap: 2024-09-15T02:12:42.984 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:42.983820+0000 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:42.983820+0000 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:42.985 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.983+0000 7f5563caf640 1 Processor -- start 2024-09-15T02:12:42.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.983+0000 7f5563caf640 1 -- start start 2024-09-15T02:12:42.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.983+0000 7f5563caf640 10 monclient: init 2024-09-15T02:12:42.987 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f555c130cc0 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f555c131090 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f555c131490 0x7f555c13a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-a con 0x7f555c131490 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c0a1670 con 0x7f555c130cc0 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c132960 con 0x7f555c131090 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c09fde0 con 0x7f555c131490 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5563caf640 10 monclient(hunting): authenticate will time out at 2022.980296s 2024-09-15T02:12:42.988 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7f555c131490 0x7f555c13a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5561a4b640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f555c130cc0 0x7f555c132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35760/0 (socket says 172.21.15.17:35760) 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f5561a4b640 1 -- 172.21.15.17:0/3184016054 learned_addr learned my addr 172.21.15.17:0/3184016054 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): get_auth_request con 0x7f555c131490 auth_method 0 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient(hunting): handle_auth_done global_id 254196 payload 274 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 1 -- 172.21.15.17:0/3184016054 >> v1:172.21.15.67:6789/0 conn(0x7f555c131090 legacy=0x7f555c136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 1 -- 172.21.15.17:0/3184016054 >> v1:172.21.15.17:6789/0 conn(0x7f555c130cc0 legacy=0x7f555c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:42.989 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient: _finish_hunting 0 2024-09-15T02:12:42.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 1 monclient: found mon.noname-a 2024-09-15T02:12:42.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:42.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:42.990 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.984+0000 7f556224c640 1 -- 172.21.15.17:0/3184016054 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0a1950 con 0x7f555c131490 2024-09-15T02:12:42.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f556224c640 1 -- 172.21.15.17:0/3184016054 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 msgr2=0x7f555c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:42.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f556224c640 1 --2- 172.21.15.17:0/3184016054 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c13a500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f555c0687a0 tx=0x7f55540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1986d5035ffab959 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:42.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 1 -- 172.21.15.17:0/3184016054 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554032440 con 0x7f555c131490 2024-09-15T02:12:42.995 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: dump: 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.996 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: _finish_auth 0 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 1 -- 172.21.15.17:0/3184016054 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55540325a0 con 0x7f555c131490 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 1 -- 172.21.15.17:0/3184016054 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554032940 con 0x7f555c131490 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5563caf640 5 monclient: authenticate success, global_id 254196 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5563caf640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5563caf640 4 set_mon_vals no callback set 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.987+0000 7f5560a49640 10 monclient: dump: 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:42.997 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 10 monclient: shutdown 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 -- 172.21.15.17:0/3184016054 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 msgr2=0x7f555c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 --2- 172.21.15.17:0/3184016054 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c13a500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f555c0687a0 tx=0x7f55540090f0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.998 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 -- 172.21.15.17:0/3184016054 shutdown_connections 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 --2- 172.21.15.17:0/3184016054 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c13a500 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 -- 172.21.15.17:0/3184016054 >> 172.21.15.17:0/3184016054 conn(0x7f555c070fb0 msgr2=0x7f555c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 -- 172.21.15.17:0/3184016054 shutdown_connections 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 1 -- 172.21.15.17:0/3184016054 wait complete. 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 10 monclient: build_initial_monmap 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.988+0000 7f5563caf640 10 monclient: monmap: 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:42.999 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 Processor -- start 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 -- start start 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient: init 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 0x7f555c141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-c con 0x7f555c130cc0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f555c131090 0x7f555c099090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.000 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-b con 0x7f555c131090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c0995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): picked mon.noname-a con 0x7f555c131490 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c11c170 con 0x7f555c131490 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c11be90 con 0x7f555c131090 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c136290 con 0x7f555c130cc0 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5563caf640 10 monclient(hunting): authenticate will time out at 2022.985800s 2024-09-15T02:12:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c0995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5561a4b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 0x7f555c141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c0995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55814/0 (socket says 172.21.15.17:55814) 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 1 -- 172.21.15.17:0/48851998 learned_addr learned my addr 172.21.15.17:0/48851998 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5561a4b640 10 monclient(hunting): get_auth_request con 0x7f555c130cc0 auth_method 0 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5561a4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5561a4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f5561a4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 10 monclient(hunting): get_auth_request con 0x7f555c131490 auth_method 0 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556224c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.002 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.989+0000 7f556124a640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f555c131090 0x7f555c099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556224c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): get_auth_request con 0x7f555c131090 auth_method 0 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.003 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556124a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient(hunting): handle_auth_done global_id 214613 payload 824 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f555c131090 msgr2=0x7f555c099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f555c131090 0x7f555c099090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 msgr2=0x7f555c0995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c0995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient: _finish_hunting 0 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 monclient: found mon.noname-c 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f5561a4b640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554009390 con 0x7f555c130cc0 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.990+0000 7f556224c640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f555c131490 0x7f555c0995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5561a4b640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 msgr2=0x7f555c141630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5561a4b640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 0x7f555c141630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f554c00cbe0 tx=0x7f554c005330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00ce00 con 0x7f555c130cc0 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: dump: 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.005 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00aa60 con 0x7f555c130cc0 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 5 monclient: authenticate success, global_id 214613 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 10 monclient: _renew_subs 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c09b610 con 0x7f555c130cc0 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:43.006 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00ae20 con 0x7f555c130cc0 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5559ffb640 10 monclient: dump: 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.007 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 10 monclient: _renew_subs 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c11b7d0 con 0x7f555c130cc0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.992+0000 7f555bfff640 4 set_mon_vals no callback set 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.993+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f554c007300 con 0x7f555c130cc0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.993+0000 7f5563caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.993+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f555c0724a0 con 0x7f555c130cc0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.993+0000 7f5559ffb640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5548038770 0x7f554803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f554c04c7b0 con 0x7f555c130cc0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f5559ffb640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7f5548041660 0x7f5548043af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55480441c0 con 0x7f5548041660 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (secure 0 0 0) 0x7f554c007790 con 0x7f555c130cc0 2024-09-15T02:12:43.008 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f5559ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f556124a640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5548038770 0x7f554803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f556124a640 10 monclient: get_auth_request con 0x7f5548038770 auth_method 0 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f556224c640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7f5548041660 0x7f5548043af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.994+0000 7f556224c640 10 monclient: get_auth_request con 0x7f5548041660 auth_method 0 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.996+0000 7f556124a640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5548038770 0x7f554803ac20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f555c09c2f0 tx=0x7f5550007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.009 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:42.999+0000 7f556224c640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7f5548041660 0x7f5548043af0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.010 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.000+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== osd.5 v2:172.21.15.67:6800/31882 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f55480441c0 con 0x7f5548041660 2024-09-15T02:12:43.012 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.011+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 --> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f555c141b70 con 0x7f5548041660 2024-09-15T02:12:43.013 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5559ffb640 1 -- 172.21.15.17:0/48851998 <== osd.5 v2:172.21.15.67:6800/31882 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f555c141b70 con 0x7f5548041660 2024-09-15T02:12:43.013 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-15T02:12:43.013 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:12:43.013 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-15T02:12:43.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7f5548041660 msgr2=0x7f5548043af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.014 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6800/31882,v1:172.21.15.67:6801/31882] conn(0x7f5548041660 0x7f5548043af0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5548038770 msgr2=0x7f554803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f5548038770 0x7f554803ac20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f555c09c2f0 tx=0x7f5550007e10 comp rx=0 tx=0).stop 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 10 monclient: shutdown 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 msgr2=0x7f555c141630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 --2- 172.21.15.17:0/48851998 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f555c130cc0 0x7f555c141630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f554c00cbe0 tx=0x7f554c005330 comp rx=0 tx=0).stop 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f556224c640 1 -- 172.21.15.17:0/48851998 reap_dead start 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 shutdown_connections 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 >> 172.21.15.17:0/48851998 conn(0x7f555c070fb0 msgr2=0x7f555c097940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 shutdown_connections 2024-09-15T02:12:43.015 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.012+0000 7f5563caf640 1 -- 172.21.15.17:0/48851998 wait complete. 2024-09-15T02:12:43.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T02:12:43.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 10 monclient: get_monmap_and_config 2024-09-15T02:12:43.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:43.120 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 10 monclient: monmap: 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:43.119879+0000 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:43.119879+0000 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:43.122 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 1 Processor -- start 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 1 -- start start 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.119+0000 7fec0fb9d640 10 monclient: init 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec08131b20 addr v1:172.21.15.17:6789/0 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec080a0280 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fec08137e60 0x7fec0813a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7fec08137e60 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec080a0040 con 0x7fec08131b20 2024-09-15T02:12:43.132 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0809bff0 con 0x7fec080a0280 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec080a14a0 con 0x7fec08137e60 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0fb9d640 10 monclient(hunting): authenticate will time out at 2023.116389s 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0d939640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fec08131b20 0x7fec0809eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:35774/0 (socket says 172.21.15.17:35774) 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fec08137e60 0x7fec0813a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0d939640 1 -- 172.21.15.17:0/155134148 learned_addr learned my addr 172.21.15.17:0/155134148 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): get_auth_request con 0x7fec08137e60 auth_method 0 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient(hunting): handle_auth_done global_id 254202 payload 274 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 1 -- 172.21.15.17:0/155134148 >> v1:172.21.15.67:6789/0 conn(0x7fec080a0280 legacy=0x7fec08131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 1 -- 172.21.15.17:0/155134148 >> v1:172.21.15.17:6789/0 conn(0x7fec08131b20 legacy=0x7fec0809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient: _finish_hunting 0 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 1 monclient: found mon.noname-a 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.120+0000 7fec0e13a640 1 -- 172.21.15.17:0/155134148 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0809f3b0 con 0x7fec08137e60 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0e13a640 1 -- 172.21.15.17:0/155134148 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec08137e60 msgr2=0x7fec0813a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0e13a640 1 --2- 172.21.15.17:0/155134148 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec08137e60 0x7fec0813a240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febf8009e60 tx=0x7febf8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce25aabedd467c28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 1 -- 172.21.15.17:0/155134148 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8007630 con 0x7fec08137e60 2024-09-15T02:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: dump: 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: _finish_auth 0 2024-09-15T02:12:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0fb9d640 5 monclient: authenticate success, global_id 254202 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 1 -- 172.21.15.17:0/155134148 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf802d050 con 0x7fec08137e60 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0fb9d640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0fb9d640 4 set_mon_vals no callback set 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 1 -- 172.21.15.17:0/155134148 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80079c0 con 0x7fec08137e60 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.123+0000 7fec0c937640 10 monclient: dump: 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.136 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 10 monclient: shutdown 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 -- 172.21.15.17:0/155134148 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec08137e60 msgr2=0x7fec0813a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/155134148 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec08137e60 0x7fec0813a240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febf8009e60 tx=0x7febf8009780 comp rx=0 tx=0).stop 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 -- 172.21.15.17:0/155134148 shutdown_connections 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/155134148 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec08137e60 0x7fec0813a240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 -- 172.21.15.17:0/155134148 >> 172.21.15.17:0/155134148 conn(0x7fec08071270 msgr2=0x7fec080a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:43.137 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 -- 172.21.15.17:0/155134148 shutdown_connections 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 1 -- 172.21.15.17:0/155134148 wait complete. 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 10 monclient: build_initial_monmap 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.124+0000 7fec0fb9d640 10 monclient: monmap: 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.138 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 Processor -- start 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 -- start start 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient: init 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7fec080a0280 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fec08131b20 0x7fec0812aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-c con 0x7fec08131b20 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec08137e60 0x7fec0812b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): picked mon.noname-b con 0x7fec08137e60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0812de40 con 0x7fec080a0280 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.139 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0812e120 con 0x7fec08137e60 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0812c260 con 0x7fec08131b20 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0fb9d640 10 monclient(hunting): authenticate will time out at 2023.121794s 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55830/0 (socket says 172.21.15.17:55830) 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 1 -- 172.21.15.17:0/2990835112 learned_addr learned my addr 172.21.15.17:0/2990835112 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): get_auth_request con 0x7fec080a0280 auth_method 0 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d138640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fec08131b20 0x7fec0812aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.140 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.125+0000 7fec0d939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): get_auth_request con 0x7fec08131b20 auth_method 0 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec08137e60 0x7fec0812b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): get_auth_request con 0x7fec08137e60 auth_method 0 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.141 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0e13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 10 monclient(hunting): handle_auth_done global_id 254205 payload 824 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 -- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fec08131b20 msgr2=0x7fec0812aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fec08131b20 0x7fec0812aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 -- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec08137e60 msgr2=0x7fec0812b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec08137e60 0x7fec0812b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 10 monclient: _finish_hunting 0 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 monclient: found mon.noname-a 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.126+0000 7fec0d939640 1 -- 172.21.15.17:0/2990835112 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8009a20 con 0x7fec080a0280 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0d939640 1 -- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 msgr2=0x7fec0809e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0d939640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c5e0 tx=0x7fec00004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00005cb0 con 0x7fec080a0280 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: dump: 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.143 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: _finish_auth 0 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec00005e30 con 0x7fec080a0280 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 5 monclient: authenticate success, global_id 254205 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 10 monclient: _renew_subs 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0812f980 con 0x7fec080a0280 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00007340 con 0x7fec080a0280 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.144 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec05ffb640 10 monclient: dump: 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 10 monclient: _renew_subs 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec07fff640 4 set_mon_vals no callback set 2024-09-15T02:12:43.145 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.128+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0812c4a0 con 0x7fec080a0280 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.129+0000 7fec0fb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.129+0000 7fec0fb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.129+0000 7fec0fb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.129+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fec0001b400 con 0x7fec080a0280 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.129+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec080724a0 con 0x7fec080a0280 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.130+0000 7fec05ffb640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 0x7febf403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.130+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fec0004c140 con 0x7fec080a0280 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.130+0000 7fec0d138640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 0x7febf403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.130+0000 7fec0d138640 10 monclient: get_auth_request con 0x7febf4038770 auth_method 0 2024-09-15T02:12:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.133+0000 7fec0d138640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 0x7febf403ac20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fec080a5cd0 tx=0x7febfc007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.134+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fec0001a720 con 0x7fec080a0280 2024-09-15T02:12:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.134+0000 7fec05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.147 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.134+0000 7fec05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.298+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fec080a09b0 con 0x7febf4038770 2024-09-15T02:12:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.299+0000 7fec05ffb640 1 -- 172.21.15.17:0/2990835112 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fec080a09b0 con 0x7febf4038770 2024-09-15T02:12:43.301 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 msgr2=0x7febf403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 0x7febf403ac20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fec080a5cd0 tx=0x7febfc007e10 comp rx=0 tx=0).stop 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 10 monclient: shutdown 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 msgr2=0x7fec0809e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c5e0 tx=0x7fec00004960 comp rx=0 tx=0).stop 2024-09-15T02:12:43.303 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 shutdown_connections 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7febf4038770 0x7febf403ac20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec08137e60 0x7fec0812b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fec08131b20 0x7fec0812aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 --2- 172.21.15.17:0/2990835112 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fec080a0280 0x7fec0809e1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 >> 172.21.15.17:0/2990835112 conn(0x7fec08071270 msgr2=0x7fec08129b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 shutdown_connections 2024-09-15T02:12:43.304 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.301+0000 7fec0fb9d640 1 -- 172.21.15.17:0/2990835112 wait complete. 2024-09-15T02:12:43.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.407+0000 7ff1b728b640 10 monclient: get_monmap_and_config 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.407+0000 7ff1b728b640 10 monclient: build_initial_monmap 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.407+0000 7ff1b728b640 10 monclient: monmap: 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.409 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:12:43.408431+0000 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:12:43.408431+0000 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 Processor -- start 2024-09-15T02:12:43.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 -- start start 2024-09-15T02:12:43.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient: init 2024-09-15T02:12:43.412 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1b009c0d0 addr v1:172.21.15.67:6789/0 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff1b006eb30 0x7ff1b006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b006eb30 addr v2:172.21.15.17:3300/0 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b00699c0 con 0x7ff1b009c0d0 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b006a5e0 con 0x7ff1b006eb30 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b006a290 con 0x7ff1b009e970 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.408+0000 7ff1b728b640 10 monclient(hunting): authenticate will time out at 2023.404882s 2024-09-15T02:12:43.413 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff1b006eb30 0x7ff1b006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3301/0 says I am v2:172.21.15.17:55554/0 (socket says 172.21.15.17:55554) 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7ff1b006eb30 0x7ff1b006ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55838/0 (socket says 172.21.15.17:55838) 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 1 -- 172.21.15.17:0/2980805099 learned_addr learned my addr 172.21.15.17:0/2980805099 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): get_auth_request con 0x7ff1b009e970 auth_method 0 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): get_auth_request con 0x7ff1b006eb30 auth_method 0 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient(hunting): handle_auth_done global_id 254208 payload 274 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 -- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 msgr2=0x7ff1b009c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 -- 172.21.15.17:0/2980805099 >> v1:172.21.15.67:6789/0 conn(0x7ff1b009c0d0 legacy=0x7ff1b009a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:12:43.415 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient: _finish_hunting 0 2024-09-15T02:12:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 monclient: found mon.noname-a 2024-09-15T02:12:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b5828640 1 -- 172.21.15.17:0/2980805099 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b009ed40 con 0x7ff1b006eb30 2024-09-15T02:12:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.409+0000 7ff1b4826640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b5828640 1 -- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 msgr2=0x7ff1b006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b5828640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b006ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009e60 tx=0x7ff1a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63161210efa3e8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 1 -- 172.21.15.17:0/2980805099 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8007760 con 0x7ff1b006eb30 2024-09-15T02:12:43.422 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: dump: 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.423 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: _finish_auth 0 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 1 -- 172.21.15.17:0/2980805099 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a802d050 con 0x7ff1b006eb30 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 5 monclient: authenticate success, global_id 254208 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 10 monclient: get_monmap_and_config success 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 1 -- 172.21.15.17:0/2980805099 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8007af0 con 0x7ff1b006eb30 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 4 set_mon_vals no callback set 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.424 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1a7fff640 10 monclient: dump: 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.425 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 10 monclient: shutdown 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 -- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 msgr2=0x7ff1b006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b006ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009e60 tx=0x7ff1a8009780 comp rx=0 tx=0).stop 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 -- 172.21.15.17:0/2980805099 shutdown_connections 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b006ef20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 --2- 172.21.15.17:0/2980805099 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b009c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.412+0000 7ff1b728b640 1 -- 172.21.15.17:0/2980805099 >> 172.21.15.17:0/2980805099 conn(0x7ff1b00712f0 msgr2=0x7ff1b00a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 1 -- 172.21.15.17:0/2980805099 shutdown_connections 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 1 -- 172.21.15.17:0/2980805099 wait complete. 2024-09-15T02:12:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 10 monclient: build_initial_monmap 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 10 monclient: monmap: 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 1 Processor -- start 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 1 -- start start 2024-09-15T02:12:43.427 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.413+0000 7ff1b728b640 10 monclient: init 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient: _reopen_session rank -1 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b006eb30 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b009c0d0 0x7ff1b0133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b009c0d0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b011e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b009e970 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b009cd80 con 0x7ff1b006eb30 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0122470 con 0x7ff1b009c0d0 2024-09-15T02:12:43.428 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0141d40 con 0x7ff1b009e970 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): _renew_subs 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b728b640 10 monclient(hunting): authenticate will time out at 2023.410507s 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b011e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:55840/0 (socket says 172.21.15.17:55840) 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 -- 172.21.15.17:0/3547682517 learned_addr learned my addr 172.21.15.17:0/3547682517 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): get_auth_request con 0x7ff1b006eb30 auth_method 0 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): get_auth_request con 0x7ff1b009e970 auth_method 0 2024-09-15T02:12:43.429 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b009c0d0 0x7ff1b0133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): get_auth_request con 0x7ff1b009c0d0 auth_method 0 2024-09-15T02:12:43.430 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b4826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient(hunting): handle_auth_done global_id 254211 payload 824 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 -- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 msgr2=0x7ff1b011e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.431 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b011e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 -- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b009c0d0 msgr2=0x7ff1b0133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b009c0d0 0x7ff1b0133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient: _finish_hunting 0 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 monclient: found mon.noname-a 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5027640 1 -- 172.21.15.17:0/3547682517 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0009450 con 0x7ff1b006eb30 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.414+0000 7ff1b5828640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b011e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b5027640 1 -- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 msgr2=0x7ff1b0140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b5027640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c610 tx=0x7ff1ac007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac012840 con 0x7ff1b006eb30 2024-09-15T02:12:43.432 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: dump: 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: _finish_auth 0 2024-09-15T02:12:43.433 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac0129c0 con 0x7ff1b006eb30 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac012d80 con 0x7ff1b006eb30 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a57fa640 10 monclient: dump: 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:12:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 5 monclient: authenticate success, global_id 254211 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 10 monclient: _renew_subs 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a8009a20 con 0x7ff1b006eb30 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1a77fe640 4 set_mon_vals no callback set 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 10 monclient: _renew_subs 2024-09-15T02:12:43.435 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.417+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b0120ec0 con 0x7ff1b006eb30 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff1ac011590 con 0x7ff1b006eb30 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1a57fa640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 0x7ff19403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7ff1ac04c850 con 0x7ff1b006eb30 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b728b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b728b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b00724a0 con 0x7ff1b006eb30 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b4826640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 0x7ff19403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.418+0000 7ff1b4826640 10 monclient: get_auth_request con 0x7ff194038790 auth_method 0 2024-09-15T02:12:43.436 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.421+0000 7ff1b4826640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 0x7ff19403ac40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff1b009d560 tx=0x7ff1a00091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:12:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.422+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff1ac016030 con 0x7ff1b006eb30 2024-09-15T02:12:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.422+0000 7ff1a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:12:43.437 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.422+0000 7ff1a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:12:43.587 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.586+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1b0121ee0 con 0x7ff194038790 2024-09-15T02:12:43.591 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.590+0000 7ff1a57fa640 1 -- 172.21.15.17:0/3547682517 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97856 (secure 0 0 0) 0x7ff1b0121ee0 con 0x7ff194038790 2024-09-15T02:12:43.592 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:12:43.592 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 msgr2=0x7ff19403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 0x7ff19403ac40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff1b009d560 tx=0x7ff1a00091d0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 10 monclient: shutdown 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 msgr2=0x7ff1b0140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c610 tx=0x7ff1ac007ed0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 shutdown_connections 2024-09-15T02:12:43.594 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7ff194038790 0x7ff19403ac40 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7ff1b009e970 0x7ff1b011e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b009c0d0 0x7ff1b0133cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 --2- 172.21.15.17:0/3547682517 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff1b006eb30 0x7ff1b0140e90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 >> 172.21.15.17:0/3547682517 conn(0x7ff1b00712f0 msgr2=0x7ff1b00a1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 shutdown_connections 2024-09-15T02:12:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:12:43.592+0000 7ff1b728b640 1 -- 172.21.15.17:0/3547682517 wait complete. 2024-09-15T02:12:43.604 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":892,"stamp":"2024-09-15T02:12:43.580118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812796,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418244,"statfs":{"total":579820584960,"available":577964281856,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002665"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":172,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421220+0000","last_change":"2024-09-15T01:58:09.930154+0000","last_active":"2024-09-15T02:00:50.421220+0000","last_peered":"2024-09-15T02:00:50.421220+0000","last_clean":"2024-09-15T02:00:50.421220+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:00:50.421220+0000","last_undegraded":"2024-09-15T02:00:50.421220+0000","last_fullsized":"2024-09-15T02:00:50.421220+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:15:30.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":83,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427638+0000","last_change":"2024-09-15T01:58:09.929368+0000","last_active":"2024-09-15T02:00:51.427638+0000","last_peered":"2024-09-15T02:00:51.427638+0000","last_clean":"2024-09-15T02:00:51.427638+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:00:51.427638+0000","last_undegraded":"2024-09-15T02:00:51.427638+0000","last_fullsized":"2024-09-15T02:00:51.427638+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":150,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.421265+0000","last_change":"2024-09-15T01:58:09.926074+0000","last_active":"2024-09-15T02:00:51.421265+0000","last_peered":"2024-09-15T02:00:51.421265+0000","last_clean":"2024-09-15T02:00:51.421265+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:00:51.421265+0000","last_undegraded":"2024-09-15T02:00:51.421265+0000","last_fullsized":"2024-09-15T02:00:51.421265+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":83,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427663+0000","last_change":"2024-09-15T01:58:09.925557+0000","last_active":"2024-09-15T02:00:51.427663+0000","last_peered":"2024-09-15T02:00:51.427663+0000","last_clean":"2024-09-15T02:00:51.427663+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:00:51.427663+0000","last_undegraded":"2024-09-15T02:00:51.427663+0000","last_fullsized":"2024-09-15T02:00:51.427663+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427734+0000","last_change":"2024-09-15T01:58:11.289109+0000","last_active":"2024-09-15T02:00:51.427734+0000","last_peered":"2024-09-15T02:00:51.427734+0000","last_clean":"2024-09-15T02:00:51.427734+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:00:51.427734+0000","last_undegraded":"2024-09-15T02:00:51.427734+0000","last_fullsized":"2024-09-15T02:00:51.427734+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":104,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421956+0000","last_change":"2024-09-15T01:58:09.929748+0000","last_active":"2024-09-15T02:00:50.421956+0000","last_peered":"2024-09-15T02:00:50.421956+0000","last_clean":"2024-09-15T02:00:50.421956+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:00:50.421956+0000","last_undegraded":"2024-09-15T02:00:50.421956+0000","last_fullsized":"2024-09-15T02:00:50.421956+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:28:34.534341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":80,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.964101+0000","last_change":"2024-09-15T01:58:09.931531+0000","last_active":"2024-09-15T02:00:51.964101+0000","last_peered":"2024-09-15T02:00:51.964101+0000","last_clean":"2024-09-15T02:00:51.964101+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:00:51.964101+0000","last_undegraded":"2024-09-15T02:00:51.964101+0000","last_fullsized":"2024-09-15T02:00:51.964101+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421858+0000","last_change":"2024-09-15T01:58:09.937601+0000","last_active":"2024-09-15T02:00:50.421858+0000","last_peered":"2024-09-15T02:00:50.421858+0000","last_clean":"2024-09-15T02:00:50.421858+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:00:50.421858+0000","last_undegraded":"2024-09-15T02:00:50.421858+0000","last_fullsized":"2024-09-15T02:00:50.421858+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:47:48.897436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":107,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427555+0000","last_change":"2024-09-15T01:58:09.926660+0000","last_active":"2024-09-15T02:00:51.427555+0000","last_peered":"2024-09-15T02:00:51.427555+0000","last_clean":"2024-09-15T02:00:51.427555+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:00:51.427555+0000","last_undegraded":"2024-09-15T02:00:51.427555+0000","last_fullsized":"2024-09-15T02:00:51.427555+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":105,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.964021+0000","last_change":"2024-09-15T01:58:09.925970+0000","last_active":"2024-09-15T02:00:51.964021+0000","last_peered":"2024-09-15T02:00:51.964021+0000","last_clean":"2024-09-15T02:00:51.964021+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:00:51.964021+0000","last_undegraded":"2024-09-15T02:00:51.964021+0000","last_fullsized":"2024-09-15T02:00:51.964021+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":125,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260042+0000","last_change":"2024-09-15T01:58:09.929440+0000","last_active":"2024-09-15T02:00:51.260042+0000","last_peered":"2024-09-15T02:00:51.260042+0000","last_clean":"2024-09-15T02:00:51.260042+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:00:51.260042+0000","last_undegraded":"2024-09-15T02:00:51.260042+0000","last_fullsized":"2024-09-15T02:00:51.260042+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:12.548233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421711+0000","last_change":"2024-09-15T01:58:09.927140+0000","last_active":"2024-09-15T02:00:50.421711+0000","last_peered":"2024-09-15T02:00:50.421711+0000","last_clean":"2024-09-15T02:00:50.421711+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:00:50.421711+0000","last_undegraded":"2024-09-15T02:00:50.421711+0000","last_fullsized":"2024-09-15T02:00:50.421711+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:14:46.532027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":84,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421823+0000","last_change":"2024-09-15T01:58:09.927406+0000","last_active":"2024-09-15T02:00:50.421823+0000","last_peered":"2024-09-15T02:00:50.421823+0000","last_clean":"2024-09-15T02:00:50.421823+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:00:50.421823+0000","last_undegraded":"2024-09-15T02:00:50.421823+0000","last_fullsized":"2024-09-15T02:00:50.421823+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:44:15.914758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":179,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421309+0000","last_change":"2024-09-15T01:58:09.926338+0000","last_active":"2024-09-15T02:00:50.421309+0000","last_peered":"2024-09-15T02:00:50.421309+0000","last_clean":"2024-09-15T02:00:50.421309+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:00:50.421309+0000","last_undegraded":"2024-09-15T02:00:50.421309+0000","last_fullsized":"2024-09-15T02:00:50.421309+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.854672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":83,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.421202+0000","last_change":"2024-09-15T01:58:11.002561+0000","last_active":"2024-09-15T02:00:51.421202+0000","last_peered":"2024-09-15T02:00:51.421202+0000","last_clean":"2024-09-15T02:00:51.421202+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:00:51.421202+0000","last_undegraded":"2024-09-15T02:00:51.421202+0000","last_fullsized":"2024-09-15T02:00:51.421202+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:10.352126+0000","last_clean_scrub_stamp":"2024-09-15T01:56:10.352126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421716+0000","last_change":"2024-09-15T01:58:09.929239+0000","last_active":"2024-09-15T02:00:50.421716+0000","last_peered":"2024-09-15T02:00:50.421716+0000","last_clean":"2024-09-15T02:00:50.421716+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:00:50.421716+0000","last_undegraded":"2024-09-15T02:00:50.421716+0000","last_fullsized":"2024-09-15T02:00:50.421716+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:42:24.014259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":237,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.964047+0000","last_change":"2024-09-15T01:58:09.925695+0000","last_active":"2024-09-15T02:00:51.964047+0000","last_peered":"2024-09-15T02:00:51.964047+0000","last_clean":"2024-09-15T02:00:51.964047+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:00:51.964047+0000","last_undegraded":"2024-09-15T02:00:51.964047+0000","last_fullsized":"2024-09-15T02:00:51.964047+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":231,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427555+0000","last_change":"2024-09-15T01:58:10.926968+0000","last_active":"2024-09-15T02:00:51.427555+0000","last_peered":"2024-09-15T02:00:51.427555+0000","last_clean":"2024-09-15T02:00:51.427555+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:00:51.427555+0000","last_undegraded":"2024-09-15T02:00:51.427555+0000","last_fullsized":"2024-09-15T02:00:51.427555+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":85,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260044+0000","last_change":"2024-09-15T01:58:09.934342+0000","last_active":"2024-09-15T02:00:51.260044+0000","last_peered":"2024-09-15T02:00:51.260044+0000","last_clean":"2024-09-15T02:00:51.260044+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:00:51.260044+0000","last_undegraded":"2024-09-15T02:00:51.260044+0000","last_fullsized":"2024-09-15T02:00:51.260044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:09:06.563997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":89,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421728+0000","last_change":"2024-09-15T01:58:09.937375+0000","last_active":"2024-09-15T02:00:50.421728+0000","last_peered":"2024-09-15T02:00:50.421728+0000","last_clean":"2024-09-15T02:00:50.421728+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:00:50.421728+0000","last_undegraded":"2024-09-15T02:00:50.421728+0000","last_fullsized":"2024-09-15T02:00:50.421728+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:25:49.703445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":364,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.964023+0000","last_change":"2024-09-15T01:58:09.940565+0000","last_active":"2024-09-15T02:00:51.964023+0000","last_peered":"2024-09-15T02:00:51.964023+0000","last_clean":"2024-09-15T02:00:51.964023+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:00:51.964023+0000","last_undegraded":"2024-09-15T02:00:51.964023+0000","last_fullsized":"2024-09-15T02:00:51.964023+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:50.421850+0000","last_change":"2024-09-15T01:58:09.935097+0000","last_active":"2024-09-15T02:00:50.421850+0000","last_peered":"2024-09-15T02:00:50.421850+0000","last_clean":"2024-09-15T02:00:50.421850+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:00:50.421850+0000","last_undegraded":"2024-09-15T02:00:50.421850+0000","last_fullsized":"2024-09-15T02:00:50.421850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:26:20.884430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.260084+0000","last_change":"2024-09-15T01:58:09.921899+0000","last_active":"2024-09-15T02:00:51.260084+0000","last_peered":"2024-09-15T02:00:51.260084+0000","last_clean":"2024-09-15T02:00:51.260084+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:00:51.260084+0000","last_undegraded":"2024-09-15T02:00:51.260084+0000","last_fullsized":"2024-09-15T02:00:51.260084+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:34:26.752431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":79,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.964079+0000","last_change":"2024-09-15T01:58:09.923781+0000","last_active":"2024-09-15T02:00:51.964079+0000","last_peered":"2024-09-15T02:00:51.964079+0000","last_clean":"2024-09-15T02:00:51.964079+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:00:51.964079+0000","last_undegraded":"2024-09-15T02:00:51.964079+0000","last_fullsized":"2024-09-15T02:00:51.964079+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":238,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.427695+0000","last_change":"2024-09-15T01:58:09.929011+0000","last_active":"2024-09-15T02:00:51.427695+0000","last_peered":"2024-09-15T02:00:51.427695+0000","last_clean":"2024-09-15T02:00:51.427695+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:00:51.427695+0000","last_undegraded":"2024-09-15T02:00:51.427695+0000","last_fullsized":"2024-09-15T02:00:51.427695+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640514,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.504,"15min":0.496},"min":{"1min":0.375,"5min":0.315,"15min":0.253},"max":{"1min":0.69299999999999995,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.51800000000000002},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.51300000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.56000000000000005},"min":{"1min":0.45300000000000001,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.626}]},{"osd":2,"last update":"Sun Sep 15 02:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38900000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.58399999999999996}]},{"osd":3,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.501,"5min":0.745,"15min":0.745},"last":0.33200000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41299999999999998,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.44900000000000001,"5min":0.879,"15min":0.879},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.501,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.57399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.41899999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640512,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298500,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073340,"statfs":{"total":96636764160,"available":96331100160,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.62,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.746,"15min":0.80200000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.65700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.438}]},{"osd":2,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.29999999999999999},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.56899999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.66100000000000003},"last":0.377}]},{"osd":5,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.247},"max":{"1min":0.63,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.69799999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.316}]}]},{"osd":3,"up_from":11,"seq":47244640514,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303172,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068668,"statfs":{"total":96636764160,"available":96326316032,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.51900000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68000000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46700000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66000000000000003,"5min":1.022,"15min":1.022},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":1.008,"15min":1.008},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68600000000000005,"5min":0.747,"15min":0.747},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.58999999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39500000000000002,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.45000000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42299999999999999,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.38800000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.29799999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23499999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":0.64400000000000002,"15min":0.64700000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36699999999999999,"15min":0.375},"min":{"1min":0.249,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.27200000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59299999999999997,"15min":0.56499999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.82199999999999995,"15min":0.88100000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53100000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.47599999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.315,"15min":0.32800000000000001},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.45800000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.33800000000000002},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.56899999999999995,"15min":0.66700000000000004},"last":0.44600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73099999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.38700000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.52300000000000002,"5min":0.66800000000000004,"15min":0.75800000000000001},"last":0.497}]},{"osd":3,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.78000000000000003,"15min":0.84999999999999998},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.77800000000000002},"last":0.72199999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.80700000000000005}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.873},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69799999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640515,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302656,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069184,"statfs":{"total":96636764160,"available":96326844416,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.316,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44800000000000001,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.216},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.19400000000000001},"max":{"1min":0.53200000000000003,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.36399999999999999},"min":{"1min":0.23200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":0.56399999999999995,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.29499999999999998}]},{"osd":3,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.40699999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.075},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46000000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":1.2729999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:12:43.609 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.609 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.610 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.610 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.610 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.610 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.610 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.611 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.611 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.611 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.611 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-15T01:56:10.352126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=10, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.612 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:12:43.613 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T02:13:03.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.717+0000 7fe19e314640 10 monclient: get_monmap_and_config 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.717+0000 7fe19e314640 10 monclient: build_initial_monmap 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.717+0000 7fe19e314640 10 monclient: monmap: 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:13:03.719718+0000 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:13:03.719718+0000 2024-09-15T02:13:03.720 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:13:03.721 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.718+0000 7fe19e314640 1 Processor -- start 2024-09-15T02:13:03.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.718+0000 7fe19e314640 1 -- start start 2024-09-15T02:13:03.723 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.718+0000 7fe19e314640 10 monclient: init 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1980708a0 0x7fe19809ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-c con 0x7fe1980708a0 addr v2:172.21.15.67:3300/0 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe19806a510 0x7fe19809b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-a con 0x7fe19806a510 addr v2:172.21.15.17:3300/0 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe19809b8a0 addr v1:172.21.15.17:6789/0 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19809bca0 con 0x7fe19809b8a0 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19809da40 con 0x7fe19806a510 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe198070c70 con 0x7fe1980708a0 2024-09-15T02:13:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19e314640 10 monclient(hunting): authenticate will time out at 2043.716220s 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fe19806a510 0x7fe19809b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19c8b1640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7fe19809b8a0 0x7fe198131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:42116/0 (socket says 172.21.15.17:42116) 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe19c8b1640 1 -- 172.21.15.17:0/3597519658 learned_addr learned my addr 172.21.15.17:0/3597519658 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): get_auth_request con 0x7fe19806a510 auth_method 0 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe197fff640 1 --2- 172.21.15.17:0/3597519658 >> v2:172.21.15.67:3300/0 conn(0x7fe1980708a0 0x7fe19809ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.725 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe197fff640 10 monclient(hunting): get_auth_request con 0x7fe1980708a0 auth_method 0 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe197fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient(hunting): handle_auth_done global_id 254214 payload 274 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 -- 172.21.15.17:0/3597519658 >> v2:172.21.15.67:3300/0 conn(0x7fe1980708a0 msgr2=0x7fe19809ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 --2- 172.21.15.17:0/3597519658 >> v2:172.21.15.67:3300/0 conn(0x7fe1980708a0 0x7fe19809ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 -- 172.21.15.17:0/3597519658 >> v1:172.21.15.17:6789/0 conn(0x7fe19809b8a0 legacy=0x7fe198131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient: _finish_hunting 0 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 monclient: found mon.noname-a 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:13:03.726 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.719+0000 7fe1977fe640 1 -- 172.21.15.17:0/3597519658 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe198070f30 con 0x7fe19806a510 2024-09-15T02:13:03.731 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe1977fe640 1 -- 172.21.15.17:0/3597519658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 msgr2=0x7fe19809b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe1977fe640 1 --2- 172.21.15.17:0/3597519658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19809b360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe198071bb0 tx=0x7fe18c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad2007ef6f6e7f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 1 -- 172.21.15.17:0/3597519658 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c02ae00 con 0x7fe19806a510 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: dump: 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:03.732 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: _finish_auth 0 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 1 -- 172.21.15.17:0/3597519658 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c032440 con 0x7fe19806a510 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 1 -- 172.21.15.17:0/3597519658 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0327e0 con 0x7fe19806a510 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe19e314640 5 monclient: authenticate success, global_id 254214 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe19e314640 10 monclient: get_monmap_and_config success 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe19e314640 4 set_mon_vals no callback set 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.722+0000 7fe196ffd640 10 monclient: dump: 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 10 monclient: shutdown 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:03.734 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 -- 172.21.15.17:0/3597519658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 msgr2=0x7fe19809b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 --2- 172.21.15.17:0/3597519658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19809b360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe198071bb0 tx=0x7fe18c0090f0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 -- 172.21.15.17:0/3597519658 shutdown_connections 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 --2- 172.21.15.17:0/3597519658 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19809b360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 --2- 172.21.15.17:0/3597519658 >> v2:172.21.15.67:3300/0 conn(0x7fe1980708a0 0x7fe19809ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 -- 172.21.15.17:0/3597519658 >> 172.21.15.17:0/3597519658 conn(0x7fe198067ae0 msgr2=0x7fe1980a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 -- 172.21.15.17:0/3597519658 shutdown_connections 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 1 -- 172.21.15.17:0/3597519658 wait complete. 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 10 monclient: build_initial_monmap 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.723+0000 7fe19e314640 10 monclient: monmap: 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:13:03.735 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 1 Processor -- start 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 1 -- start start 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 10 monclient: init 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-a con 0x7fe19806a510 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:13:03.736 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1980708a0 0x7fe19813d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-c con 0x7fe1980708a0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.724+0000 7fe19e314640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 0x7fe198140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): picked mon.noname-b con 0x7fe19809b8a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe198122960 con 0x7fe19806a510 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1981223a0 con 0x7fe19809b8a0 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19811fec0 con 0x7fe1980708a0 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19e314640 10 monclient(hunting): authenticate will time out at 2043.722035s 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:37102/0 (socket says 172.21.15.17:37102) 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1980708a0 0x7fe19813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 -- 172.21.15.17:0/41317123 learned_addr learned my addr 172.21.15.17:0/41317123 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): get_auth_request con 0x7fe19806a510 auth_method 0 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): get_auth_request con 0x7fe1980708a0 auth_method 0 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient(hunting): handle_auth_done global_id 254217 payload 824 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19c8b1640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 0x7fe198140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 -- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1980708a0 msgr2=0x7fe19813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1980708a0 0x7fe19813d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 -- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 msgr2=0x7fe198140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 0x7fe198140ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient: _finish_hunting 0 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 monclient: found mon.noname-a 2024-09-15T02:13:03.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe197fff640 1 -- 172.21.15.17:0/41317123 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c009390 con 0x7fe19806a510 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19c8b1640 10 monclient: get_auth_request con 0x7fe19809b8a0 auth_method 0 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.725+0000 7fe19c8b1640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 0x7fe198140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe197fff640 1 -- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 msgr2=0x7fe19813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe197fff640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d8c0 tx=0x7fe188008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800de50 con 0x7fe19806a510 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: dump: 2024-09-15T02:13:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: _finish_auth 0 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe19e314640 5 monclient: authenticate success, global_id 254217 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18800bae0 con 0x7fe19806a510 2024-09-15T02:13:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800be80 con 0x7fe19806a510 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe19e314640 10 monclient: _renew_subs 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe19e314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1981220c0 con 0x7fe19806a510 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe17ffff640 10 monclient: dump: 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:03.742 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe1967fc640 4 set_mon_vals no callback set 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.727+0000 7fe19e314640 10 monclient: _renew_subs 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.728+0000 7fe19e314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.728+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19813d700 con 0x7fe19806a510 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe188014640 con 0x7fe19806a510 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe19e314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe19e314640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe19e314640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.17:3300/0 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1980724a0 con 0x7fe19806a510 2024-09-15T02:13:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe17ffff640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 0x7fe18403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe18804cc30 con 0x7fe19806a510 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe1977fe640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 0x7fe18403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.729+0000 7fe1977fe640 10 monclient: get_auth_request con 0x7fe184038740 auth_method 0 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.732+0000 7fe1977fe640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 0x7fe18403abf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe19809c8e0 tx=0x7fe18c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.733+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe18801e460 con 0x7fe19806a510 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.733+0000 7fe17ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:03.744 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.733+0000 7fe17ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:13:03.896 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.894+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1980711e0 con 0x7fe184038740 2024-09-15T02:13:03.900 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.897+0000 7fe17ffff640 1 -- 172.21.15.17:0/41317123 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98453 (secure 0 0 0) 0x7fe1980711e0 con 0x7fe184038740 2024-09-15T02:13:03.901 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:13:03.901 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 msgr2=0x7fe18403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 0x7fe18403abf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe19809c8e0 tx=0x7fe18c032000 comp rx=0 tx=0).stop 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 10 monclient: shutdown 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 msgr2=0x7fe19813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:03.903 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d8c0 tx=0x7fe188008040 comp rx=0 tx=0).stop 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 shutdown_connections 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fe184038740 0x7fe18403abf0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe19809b8a0 0x7fe198140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fe1980708a0 0x7fe19813d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 --2- 172.21.15.17:0/41317123 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe19806a510 0x7fe19813d970 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 >> 172.21.15.17:0/41317123 conn(0x7fe198067ae0 msgr2=0x7fe19806caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 shutdown_connections 2024-09-15T02:13:03.904 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:03.899+0000 7fe19e314640 1 -- 172.21.15.17:0/41317123 wait complete. 2024-09-15T02:13:03.913 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":902,"stamp":"2024-09-15T02:13:03.584698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812512,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418528,"statfs":{"total":579820584960,"available":577964572672,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002458"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":183,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:45.296486+0000","last_change":"2024-09-15T02:12:45.296453+0000","last_active":"2024-09-15T02:12:45.296486+0000","last_peered":"2024-09-15T02:12:45.296486+0000","last_clean":"2024-09-15T02:12:45.296486+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:12:45.296486+0000","last_undegraded":"2024-09-15T02:12:45.296486+0000","last_fullsized":"2024-09-15T02:12:45.296486+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:45.296364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:45.296364+0000","last_clean_scrub_stamp":"2024-09-15T02:12:45.296364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.788939+0000","scrub_duration":0.001870989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:55.721821+0000","last_change":"2024-09-15T02:12:55.721792+0000","last_active":"2024-09-15T02:12:55.721821+0000","last_peered":"2024-09-15T02:12:55.721821+0000","last_clean":"2024-09-15T02:12:55.721821+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:12:55.721821+0000","last_undegraded":"2024-09-15T02:12:55.721821+0000","last_fullsized":"2024-09-15T02:12:55.721821+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:55.721687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:55.721687+0000","last_clean_scrub_stamp":"2024-09-15T02:12:55.721687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:18.874128+0000","scrub_duration":0.001553621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":160,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:44.908525+0000","last_change":"2024-09-15T02:12:44.908487+0000","last_active":"2024-09-15T02:12:44.908525+0000","last_peered":"2024-09-15T02:12:44.908525+0000","last_clean":"2024-09-15T02:12:44.908525+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:12:44.908525+0000","last_undegraded":"2024-09-15T02:12:44.908525+0000","last_fullsized":"2024-09-15T02:12:44.908525+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:44.908387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:44.908387+0000","last_clean_scrub_stamp":"2024-09-15T02:12:44.908387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:44:15.588289+0000","scrub_duration":0.0017035729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:50.787929+0000","last_change":"2024-09-15T02:12:50.787899+0000","last_active":"2024-09-15T02:12:50.787929+0000","last_peered":"2024-09-15T02:12:50.787929+0000","last_clean":"2024-09-15T02:12:50.787929+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:12:50.787929+0000","last_undegraded":"2024-09-15T02:12:50.787929+0000","last_fullsized":"2024-09-15T02:12:50.787929+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:50.787810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:50.787810+0000","last_clean_scrub_stamp":"2024-09-15T02:12:50.787810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:32:17.246879+0000","scrub_duration":0.0018925649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":134,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:59.748022+0000","last_change":"2024-09-15T02:12:59.747982+0000","last_active":"2024-09-15T02:12:59.748022+0000","last_peered":"2024-09-15T02:12:59.748022+0000","last_clean":"2024-09-15T02:12:59.748022+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:12:59.748022+0000","last_undegraded":"2024-09-15T02:12:59.748022+0000","last_fullsized":"2024-09-15T02:12:59.748022+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:59.747893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:59.747893+0000","last_clean_scrub_stamp":"2024-09-15T02:12:59.747893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:13:15.130860+0000","scrub_duration":0.0017457309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":115,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:41.992973+0000","last_change":"2024-09-15T02:12:41.992952+0000","last_active":"2024-09-15T02:12:41.992973+0000","last_peered":"2024-09-15T02:12:41.992973+0000","last_clean":"2024-09-15T02:12:41.992973+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:12:41.992973+0000","last_undegraded":"2024-09-15T02:12:41.992973+0000","last_fullsized":"2024-09-15T02:12:41.992973+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:41.992887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:41.992887+0000","last_clean_scrub_stamp":"2024-09-15T02:12:41.992887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:23:22.479818+0000","scrub_duration":0.0014501500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:53.998539+0000","last_change":"2024-09-15T02:12:53.998496+0000","last_active":"2024-09-15T02:12:53.998539+0000","last_peered":"2024-09-15T02:12:53.998539+0000","last_clean":"2024-09-15T02:12:53.998539+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:12:53.998539+0000","last_undegraded":"2024-09-15T02:12:53.998539+0000","last_fullsized":"2024-09-15T02:12:53.998539+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:53.998406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:53.998406+0000","last_clean_scrub_stamp":"2024-09-15T02:12:53.998406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:42:54.384209+0000","scrub_duration":0.001980908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":84,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.982773+0000","last_change":"2024-09-15T01:58:09.937601+0000","last_active":"2024-09-15T02:00:51.982773+0000","last_peered":"2024-09-15T02:00:51.982773+0000","last_clean":"2024-09-15T02:00:51.982773+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:00:51.982773+0000","last_undegraded":"2024-09-15T02:00:51.982773+0000","last_fullsized":"2024-09-15T02:00:51.982773+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":117,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:43.898616+0000","last_change":"2024-09-15T02:12:43.898568+0000","last_active":"2024-09-15T02:12:43.898616+0000","last_peered":"2024-09-15T02:12:43.898616+0000","last_clean":"2024-09-15T02:12:43.898616+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:12:43.898616+0000","last_undegraded":"2024-09-15T02:12:43.898616+0000","last_fullsized":"2024-09-15T02:12:43.898616+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:43.898473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:43.898473+0000","last_clean_scrub_stamp":"2024-09-15T02:12:43.898473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:14:26.471237+0000","scrub_duration":0.001568446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":115,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:51.963239+0000","last_change":"2024-09-15T02:12:51.963201+0000","last_active":"2024-09-15T02:12:51.963239+0000","last_peered":"2024-09-15T02:12:51.963239+0000","last_clean":"2024-09-15T02:12:51.963239+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:12:51.963239+0000","last_undegraded":"2024-09-15T02:12:51.963239+0000","last_fullsized":"2024-09-15T02:12:51.963239+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:51.963104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:51.963104+0000","last_clean_scrub_stamp":"2024-09-15T02:12:51.963104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:57:17.425572+0000","scrub_duration":0.001969972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":136,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:51.783736+0000","last_change":"2024-09-15T02:12:51.783690+0000","last_active":"2024-09-15T02:12:51.783736+0000","last_peered":"2024-09-15T02:12:51.783736+0000","last_clean":"2024-09-15T02:12:51.783736+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:12:51.783736+0000","last_undegraded":"2024-09-15T02:12:51.783736+0000","last_fullsized":"2024-09-15T02:12:51.783736+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:51.783578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:51.783578+0000","last_clean_scrub_stamp":"2024-09-15T02:12:51.783578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:57:09.868285+0000","scrub_duration":0.001808516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":84,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.982782+0000","last_change":"2024-09-15T01:58:09.927140+0000","last_active":"2024-09-15T02:00:51.982782+0000","last_peered":"2024-09-15T02:00:51.982782+0000","last_clean":"2024-09-15T02:00:51.982782+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:00:51.982782+0000","last_undegraded":"2024-09-15T02:00:51.982782+0000","last_fullsized":"2024-09-15T02:00:51.982782+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:56:09.344837+0000","last_clean_scrub_stamp":"2024-09-15T01:56:09.344837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":95,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:56.933937+0000","last_change":"2024-09-15T02:12:56.933900+0000","last_active":"2024-09-15T02:12:56.933937+0000","last_peered":"2024-09-15T02:12:56.933937+0000","last_clean":"2024-09-15T02:12:56.933937+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:12:56.933937+0000","last_undegraded":"2024-09-15T02:12:56.933937+0000","last_fullsized":"2024-09-15T02:12:56.933937+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:56.933793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:56.933793+0000","last_clean_scrub_stamp":"2024-09-15T02:12:56.933793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:09:07.328928+0000","scrub_duration":0.001859426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":190,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:44.304696+0000","last_change":"2024-09-15T02:12:44.304667+0000","last_active":"2024-09-15T02:12:44.304696+0000","last_peered":"2024-09-15T02:12:44.304696+0000","last_clean":"2024-09-15T02:12:44.304696+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:12:44.304696+0000","last_undegraded":"2024-09-15T02:12:44.304696+0000","last_fullsized":"2024-09-15T02:12:44.304696+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:44.304573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:44.304573+0000","last_clean_scrub_stamp":"2024-09-15T02:12:44.304573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:30:54.393137+0000","scrub_duration":0.001302377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:47.933263+0000","last_change":"2024-09-15T02:12:47.933202+0000","last_active":"2024-09-15T02:12:47.933263+0000","last_peered":"2024-09-15T02:12:47.933263+0000","last_clean":"2024-09-15T02:12:47.933263+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:12:47.933263+0000","last_undegraded":"2024-09-15T02:12:47.933263+0000","last_fullsized":"2024-09-15T02:12:47.933263+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:47.933100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:47.933100+0000","last_clean_scrub_stamp":"2024-09-15T02:12:47.933100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:37:11.097929+0000","scrub_duration":0.001854079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:47.946069+0000","last_change":"2024-09-15T02:12:47.946042+0000","last_active":"2024-09-15T02:12:47.946069+0000","last_peered":"2024-09-15T02:12:47.946069+0000","last_clean":"2024-09-15T02:12:47.946069+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:12:47.946069+0000","last_undegraded":"2024-09-15T02:12:47.946069+0000","last_fullsized":"2024-09-15T02:12:47.946069+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:47.945958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:47.945958+0000","last_clean_scrub_stamp":"2024-09-15T02:12:47.945958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:25.464773+0000","scrub_duration":0.0020129380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":247,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:57.871035+0000","last_change":"2024-09-15T02:12:57.870995+0000","last_active":"2024-09-15T02:12:57.871035+0000","last_peered":"2024-09-15T02:12:57.871035+0000","last_clean":"2024-09-15T02:12:57.871035+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:12:57.871035+0000","last_undegraded":"2024-09-15T02:12:57.871035+0000","last_fullsized":"2024-09-15T02:12:57.871035+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:57.870904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:57.870904+0000","last_clean_scrub_stamp":"2024-09-15T02:12:57.870904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:38.069515+0000","scrub_duration":0.0020375380000000002,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":241,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:01.768216+0000","last_change":"2024-09-15T02:13:01.768190+0000","last_active":"2024-09-15T02:13:01.768216+0000","last_peered":"2024-09-15T02:13:01.768216+0000","last_clean":"2024-09-15T02:13:01.768216+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:13:01.768216+0000","last_undegraded":"2024-09-15T02:13:01.768216+0000","last_fullsized":"2024-09-15T02:13:01.768216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:13:01.768094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:13:01.768094+0000","last_clean_scrub_stamp":"2024-09-15T02:13:01.768094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:19:57.268196+0000","scrub_duration":0.0016588079999999999,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"257'3","reported_seq":97,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:52.775551+0000","last_change":"2024-09-15T02:12:52.775524+0000","last_active":"2024-09-15T02:12:52.775551+0000","last_peered":"2024-09-15T02:12:52.775551+0000","last_clean":"2024-09-15T02:12:52.775551+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:12:52.775551+0000","last_undegraded":"2024-09-15T02:12:52.775551+0000","last_fullsized":"2024-09-15T02:12:52.775551+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'3","last_scrub_stamp":"2024-09-15T02:12:52.775442+0000","last_deep_scrub":"257'3","last_deep_scrub_stamp":"2024-09-15T02:12:52.775442+0000","last_clean_scrub_stamp":"2024-09-15T02:12:52.775442+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:51.403534+0000","scrub_duration":0.0029779149999999998,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.982861+0000","last_change":"2024-09-15T01:58:09.937375+0000","last_active":"2024-09-15T02:00:51.982861+0000","last_peered":"2024-09-15T02:00:51.982861+0000","last_clean":"2024-09-15T02:00:51.982861+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:00:51.982861+0000","last_undegraded":"2024-09-15T02:00:51.982861+0000","last_fullsized":"2024-09-15T02:00:51.982861+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"257'74","reported_seq":376,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:45.818132+0000","last_change":"2024-09-15T02:12:45.818099+0000","last_active":"2024-09-15T02:12:45.818132+0000","last_peered":"2024-09-15T02:12:45.818132+0000","last_clean":"2024-09-15T02:12:45.818132+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:12:45.818132+0000","last_undegraded":"2024-09-15T02:12:45.818132+0000","last_fullsized":"2024-09-15T02:12:45.818132+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'74","last_scrub_stamp":"2024-09-15T02:12:45.818006+0000","last_deep_scrub":"257'74","last_deep_scrub_stamp":"2024-09-15T02:12:45.818006+0000","last_clean_scrub_stamp":"2024-09-15T02:12:45.818006+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:26.985216+0000","scrub_duration":0.0045836890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":84,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:00:51.982904+0000","last_change":"2024-09-15T01:58:09.935097+0000","last_active":"2024-09-15T02:00:51.982904+0000","last_peered":"2024-09-15T02:00:51.982904+0000","last_clean":"2024-09-15T02:00:51.982904+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:00:51.982904+0000","last_undegraded":"2024-09-15T02:00:51.982904+0000","last_fullsized":"2024-09-15T02:00:51.982904+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T01:51:29.504854+0000","last_clean_scrub_stamp":"2024-09-15T01:51:29.504854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":92,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:43.760575+0000","last_change":"2024-09-15T02:12:43.760535+0000","last_active":"2024-09-15T02:12:43.760575+0000","last_peered":"2024-09-15T02:12:43.760575+0000","last_clean":"2024-09-15T02:12:43.760575+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:12:43.760575+0000","last_undegraded":"2024-09-15T02:12:43.760575+0000","last_fullsized":"2024-09-15T02:12:43.760575+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:43.760432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:43.760432+0000","last_clean_scrub_stamp":"2024-09-15T02:12:43.760432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:28:01.885095+0000","scrub_duration":0.00167812,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:55.923053+0000","last_change":"2024-09-15T02:12:55.923024+0000","last_active":"2024-09-15T02:12:55.923053+0000","last_peered":"2024-09-15T02:12:55.923053+0000","last_clean":"2024-09-15T02:12:55.923053+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:12:55.923053+0000","last_undegraded":"2024-09-15T02:12:55.923053+0000","last_fullsized":"2024-09-15T02:12:55.923053+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:55.922929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:55.922929+0000","last_clean_scrub_stamp":"2024-09-15T02:12:55.922929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:36.198779+0000","scrub_duration":0.001954744,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":248,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:46.846850+0000","last_change":"2024-09-15T02:12:46.846822+0000","last_active":"2024-09-15T02:12:46.846850+0000","last_peered":"2024-09-15T02:12:46.846850+0000","last_clean":"2024-09-15T02:12:46.846850+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:12:46.846850+0000","last_undegraded":"2024-09-15T02:12:46.846850+0000","last_fullsized":"2024-09-15T02:12:46.846850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:46.846731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:46.846731+0000","last_clean_scrub_stamp":"2024-09-15T02:12:46.846731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:45:40.047718+0000","scrub_duration":0.0019948240000000001,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.497},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53600000000000003,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54300000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.30399999999999999},"max":{"1min":0.60799999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":0.64300000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.505},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38900000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.38300000000000001},"min":{"1min":0.26800000000000002,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.53600000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40400000000000003,"15min":0.38900000000000001},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.60899999999999999,"5min":0.745,"15min":0.745},"last":0.39000000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.57699999999999996,"5min":0.879,"15min":0.879},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.38100000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298448,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073392,"statfs":{"total":96636764160,"available":96331153408,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.378},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.62,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.46800000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.746,"15min":0.80200000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.65700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.502}]},{"osd":2,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.29999999999999999},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.56899999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.66100000000000003},"last":0.32400000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.40600000000000003},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.247},"max":{"1min":0.63,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.45900000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.69799999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41899999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.51900000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28299999999999997},"max":{"1min":0.70199999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68000000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.629}]},{"osd":1,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66000000000000003,"5min":1.022,"15min":1.022},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":1.008,"15min":1.008},"last":0.53600000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68600000000000005,"5min":0.747,"15min":0.747},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.67600000000000005}]},{"osd":4,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39500000000000002,"15min":0.38400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.371}]},{"osd":5,"last update":"Sun Sep 15 02:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40799999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42299999999999999,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.312}]}]},{"osd":2,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.308}]},{"osd":1,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23499999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53900000000000003,"5min":0.64400000000000002,"15min":0.64700000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36699999999999999,"15min":0.375},"min":{"1min":0.249,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.34300000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59299999999999997,"15min":0.56499999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.82199999999999995,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56599999999999995}]},{"osd":5,"last update":"Sun Sep 15 02:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.51100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303128,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068712,"statfs":{"total":96636764160,"available":96326361088,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.315,"15min":0.32800000000000001},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.45800000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.33800000000000002},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.56899999999999995,"15min":0.66700000000000004},"last":0.249}]},{"osd":2,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34799999999999998,"15min":0.36699999999999999},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73099999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.38700000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.52300000000000002,"5min":0.66800000000000004,"15min":0.75800000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.78000000000000003,"15min":0.84999999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.77800000000000002},"last":0.59799999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.58799999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.625}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.316,"15min":0.33000000000000002},"min":{"1min":0.214,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44800000000000001,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.19400000000000001},"max":{"1min":0.53200000000000003,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.38}]},{"osd":2,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.36399999999999999},"min":{"1min":0.23200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":0.56399999999999995,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.40300000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.54600000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.075},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44400000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":1.2729999999999999},"last":0.496},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.79700000000000004,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:13:03.915 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:13:03.916 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-15T01:56:09.344837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=56, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:13:03.916 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:13:03.916 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-15T01:51:29.504854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=1, tm_min=51, tm_sec=29, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=12, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:13:03.916 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T02:13:23.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.026+0000 7f1bf519d640 10 monclient: get_monmap_and_config 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.026+0000 7f1bf519d640 10 monclient: build_initial_monmap 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.026+0000 7f1bf519d640 10 monclient: monmap: 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:13:24.027336+0000 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:13:24.027336+0000 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.028 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:13:24.029 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:13:24.029 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:13:24.029 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:13:24.029 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:13:24.029 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 Processor -- start 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 -- start start 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient: init 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bf006e920 addr v1:172.21.15.17:6789/0 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bf006ecf0 addr v1:172.21.15.67:6789/0 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf0138280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf006f0f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 -- --> v1:172.21.15.17:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf009aef0 con 0x7f1bf006e920 2024-09-15T02:13:24.031 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf006f570 con 0x7f1bf006ecf0 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf009d720 con 0x7f1bf006f0f0 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bf519d640 10 monclient(hunting): authenticate will time out at 2064.023739s 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1beed9d640 1 --1- >> v1:172.21.15.17:6789/0 conn(0x7f1bf006e920 0x7f1bf012fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.17:6789/0 says I am v1:172.21.15.17:45040/0 (socket says 172.21.15.17:45040) 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1beed9d640 1 -- 172.21.15.17:0/326497059 learned_addr learned my addr 172.21.15.17:0/326497059 (peer_addr_for_me v1:172.21.15.17:0/0) 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bef59e640 1 --2- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf0138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bef59e640 10 monclient(hunting): get_auth_request con 0x7f1bf006f0f0 auth_method 0 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bef59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.027+0000 7f1bef59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.032 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient(hunting): handle_auth_done global_id 214628 payload 274 2024-09-15T02:13:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 1 -- 172.21.15.17:0/326497059 >> v1:172.21.15.67:6789/0 conn(0x7f1bf006ecf0 legacy=0x7f1bf0134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 1 -- 172.21.15.17:0/326497059 >> v1:172.21.15.17:6789/0 conn(0x7f1bf006e920 legacy=0x7f1bf012fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient: _finish_hunting 0 2024-09-15T02:13:24.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 1 monclient: found mon.noname-b 2024-09-15T02:13:24.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:24.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.034 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.028+0000 7f1bef59e640 1 -- 172.21.15.17:0/326497059 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf009b0c0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.039 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bef59e640 1 -- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 msgr2=0x7f1bf0138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bef59e640 1 --2- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf0138280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009e90 tx=0x7f1be8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a632bb4e0085c637 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 1 -- 172.21.15.17:0/326497059 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007840 con 0x7f1bf006f0f0 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: dump: 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.040 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: _finish_auth 0 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 1 -- 172.21.15.17:0/326497059 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be802d050 con 0x7f1bf006f0f0 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 5 monclient: authenticate success, global_id 214628 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 10 monclient: get_monmap_and_config success 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 1 -- 172.21.15.17:0/326497059 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007bd0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 4 set_mon_vals no callback set 2024-09-15T02:13:24.041 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bedd9b640 10 monclient: dump: 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 10 monclient: shutdown 2024-09-15T02:13:24.042 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 -- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 msgr2=0x7f1bf0138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 --2- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf0138280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009e90 tx=0x7f1be8009780 comp rx=0 tx=0).stop 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 -- 172.21.15.17:0/326497059 shutdown_connections 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 --2- 172.21.15.17:0/326497059 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf0138280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 -- 172.21.15.17:0/326497059 >> 172.21.15.17:0/326497059 conn(0x7f1bf00a1000 msgr2=0x7f1bf00a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 -- 172.21.15.17:0/326497059 shutdown_connections 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.031+0000 7f1bf519d640 1 -- 172.21.15.17:0/326497059 wait complete. 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.032+0000 7f1bf519d640 10 monclient: build_initial_monmap 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.032+0000 7f1bf519d640 10 monclient: monmap: 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:13:24.043 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.032+0000 7f1bf519d640 1 Processor -- start 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.032+0000 7f1bf519d640 1 -- start start 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.032+0000 7f1bf519d640 10 monclient: init 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf006e920 0x7f1bf0141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.044 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf006e920 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf006ecf0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf01187e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf006f0f0 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf011bdc0 con 0x7f1bf006ecf0 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0099680 con 0x7f1bf006e920 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf011a290 con 0x7f1bf006f0f0 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bf519d640 10 monclient(hunting): authenticate will time out at 2064.029431s 2024-09-15T02:13:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf01187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34962/0 (socket says 172.21.15.17:34962) 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 1 -- 172.21.15.17:0/3663221537 learned_addr learned my addr 172.21.15.17:0/3663221537 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): get_auth_request con 0x7f1bf006ecf0 auth_method 0 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf006e920 0x7f1bf0141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): get_auth_request con 0x7f1bf006f0f0 auth_method 0 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.046 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): get_auth_request con 0x7f1bf006e920 auth_method 0 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.047 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient(hunting): handle_auth_done global_id 214631 payload 824 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 -- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf006e920 msgr2=0x7f1bf0141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf006e920 0x7f1bf0141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 -- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 msgr2=0x7f1bf012eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient: _finish_hunting 0 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 monclient: found mon.noname-c 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bef59e640 1 -- 172.21.15.17:0/3663221537 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8009a20 con 0x7f1bf006f0f0 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.033+0000 7f1bee59c640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bef59e640 1 -- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 msgr2=0x7f1bf01187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bef59e640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf01187e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1be8031000 tx=0x7f1be8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.048 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007ec0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: dump: 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.049 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: _finish_auth 0 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be8040070 con 0x7f1bf006f0f0 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8045430 con 0x7f1bf006f0f0 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 5 monclient: authenticate success, global_id 214631 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1be6ffd640 10 monclient: dump: 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 10 monclient: _renew_subs 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf009bbb0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bed59a640 4 set_mon_vals no callback set 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 10 monclient: _renew_subs 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.036+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf011a750 con 0x7f1bf006f0f0 2024-09-15T02:13:24.051 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1be8042030 con 0x7f1bf006f0f0 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1bf519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1bf519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1bf519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf00724a0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1be6ffd640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 0x7f1bd003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f1be806dff0 con 0x7f1bf006f0f0 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1beed9d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 0x7f1bd003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.037+0000 7f1beed9d640 10 monclient: get_auth_request con 0x7f1bd00387c0 auth_method 0 2024-09-15T02:13:24.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.040+0000 7f1beed9d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 0x7f1bd003ac70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1be0007fd0 tx=0x7f1be0007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.042+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1be8044790 con 0x7f1bf006f0f0 2024-09-15T02:13:24.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.042+0000 7f1be6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.053 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.042+0000 7f1be6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:13:24.203 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.202+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 --> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1bf00724a0 con 0x7f1bd00387c0 2024-09-15T02:13:24.206 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.205+0000 7f1be6ffd640 1 -- 172.21.15.17:0/3663221537 <== mgr.4102 v2:172.21.15.67:6824/31455 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98781 (secure 0 0 0) 0x7f1bf00724a0 con 0x7f1bd00387c0 2024-09-15T02:13:24.207 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-15T02:13:24.208 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-15T02:13:24.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 msgr2=0x7f1bd003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.209 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 0x7f1bd003ac70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1be0007fd0 tx=0x7f1be0007f00 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 10 monclient: shutdown 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 msgr2=0x7f1bf01187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf01187e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1be8031000 tx=0x7f1be8037000 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 shutdown_connections 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7f1bd00387c0 0x7f1bd003ac70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1bf006ecf0 0x7f1bf012eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7f1bf006f0f0 0x7f1bf01187e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 --2- 172.21.15.17:0/3663221537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf006e920 0x7f1bf0141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 >> 172.21.15.17:0/3663221537 conn(0x7f1bf00a1000 msgr2=0x7f1bf009f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 shutdown_connections 2024-09-15T02:13:24.210 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.207+0000 7f1bf519d640 1 -- 172.21.15.17:0/3663221537 wait complete. 2024-09-15T02:13:24.219 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":912,"stamp":"2024-09-15T02:13:23.589384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1057932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812512,"kb_used_data":7600,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564418528,"statfs":{"total":579820584960,"available":577964572672,"internally_reserved":0,"allocated":7782400,"data_stored":6238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":183,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:45.296486+0000","last_change":"2024-09-15T02:12:45.296453+0000","last_active":"2024-09-15T02:12:45.296486+0000","last_peered":"2024-09-15T02:12:45.296486+0000","last_clean":"2024-09-15T02:12:45.296486+0000","last_became_active":"2024-09-15T01:58:09.929074+0000","last_became_peered":"2024-09-15T01:58:09.929074+0000","last_unstale":"2024-09-15T02:12:45.296486+0000","last_undegraded":"2024-09-15T02:12:45.296486+0000","last_fullsized":"2024-09-15T02:12:45.296486+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:45.296364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:45.296364+0000","last_clean_scrub_stamp":"2024-09-15T02:12:45.296364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.788939+0000","scrub_duration":0.001870989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:55.721821+0000","last_change":"2024-09-15T02:12:55.721792+0000","last_active":"2024-09-15T02:12:55.721821+0000","last_peered":"2024-09-15T02:12:55.721821+0000","last_clean":"2024-09-15T02:12:55.721821+0000","last_became_active":"2024-09-15T01:58:09.929009+0000","last_became_peered":"2024-09-15T01:58:09.929009+0000","last_unstale":"2024-09-15T02:12:55.721821+0000","last_undegraded":"2024-09-15T02:12:55.721821+0000","last_fullsized":"2024-09-15T02:12:55.721821+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:55.721687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:55.721687+0000","last_clean_scrub_stamp":"2024-09-15T02:12:55.721687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:18.874128+0000","scrub_duration":0.001553621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":160,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:44.908525+0000","last_change":"2024-09-15T02:12:44.908487+0000","last_active":"2024-09-15T02:12:44.908525+0000","last_peered":"2024-09-15T02:12:44.908525+0000","last_clean":"2024-09-15T02:12:44.908525+0000","last_became_active":"2024-09-15T01:58:09.925426+0000","last_became_peered":"2024-09-15T01:58:09.925426+0000","last_unstale":"2024-09-15T02:12:44.908525+0000","last_undegraded":"2024-09-15T02:12:44.908525+0000","last_fullsized":"2024-09-15T02:12:44.908525+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:44.908387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:44.908387+0000","last_clean_scrub_stamp":"2024-09-15T02:12:44.908387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:44:15.588289+0000","scrub_duration":0.0017035729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:50.787929+0000","last_change":"2024-09-15T02:12:50.787899+0000","last_active":"2024-09-15T02:12:50.787929+0000","last_peered":"2024-09-15T02:12:50.787929+0000","last_clean":"2024-09-15T02:12:50.787929+0000","last_became_active":"2024-09-15T01:58:09.925228+0000","last_became_peered":"2024-09-15T01:58:09.925228+0000","last_unstale":"2024-09-15T02:12:50.787929+0000","last_undegraded":"2024-09-15T02:12:50.787929+0000","last_fullsized":"2024-09-15T02:12:50.787929+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:50.787810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:50.787810+0000","last_clean_scrub_stamp":"2024-09-15T02:12:50.787810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:32:17.246879+0000","scrub_duration":0.0018925649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":134,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:59.748022+0000","last_change":"2024-09-15T02:12:59.747982+0000","last_active":"2024-09-15T02:12:59.748022+0000","last_peered":"2024-09-15T02:12:59.748022+0000","last_clean":"2024-09-15T02:12:59.748022+0000","last_became_active":"2024-09-15T01:58:11.288780+0000","last_became_peered":"2024-09-15T01:58:11.288780+0000","last_unstale":"2024-09-15T02:12:59.748022+0000","last_undegraded":"2024-09-15T02:12:59.748022+0000","last_fullsized":"2024-09-15T02:12:59.748022+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:59.747893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:59.747893+0000","last_clean_scrub_stamp":"2024-09-15T02:12:59.747893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:13:15.130860+0000","scrub_duration":0.0017457309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":115,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:41.992973+0000","last_change":"2024-09-15T02:12:41.992952+0000","last_active":"2024-09-15T02:12:41.992973+0000","last_peered":"2024-09-15T02:12:41.992973+0000","last_clean":"2024-09-15T02:12:41.992973+0000","last_became_active":"2024-09-15T01:58:09.929407+0000","last_became_peered":"2024-09-15T01:58:09.929407+0000","last_unstale":"2024-09-15T02:12:41.992973+0000","last_undegraded":"2024-09-15T02:12:41.992973+0000","last_fullsized":"2024-09-15T02:12:41.992973+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:41.992887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:41.992887+0000","last_clean_scrub_stamp":"2024-09-15T02:12:41.992887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:23:22.479818+0000","scrub_duration":0.0014501500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:53.998539+0000","last_change":"2024-09-15T02:12:53.998496+0000","last_active":"2024-09-15T02:12:53.998539+0000","last_peered":"2024-09-15T02:12:53.998539+0000","last_clean":"2024-09-15T02:12:53.998539+0000","last_became_active":"2024-09-15T01:58:09.931105+0000","last_became_peered":"2024-09-15T01:58:09.931105+0000","last_unstale":"2024-09-15T02:12:53.998539+0000","last_undegraded":"2024-09-15T02:12:53.998539+0000","last_fullsized":"2024-09-15T02:12:53.998539+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:53.998406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:53.998406+0000","last_clean_scrub_stamp":"2024-09-15T02:12:53.998406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:42:54.384209+0000","scrub_duration":0.001980908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:05.791996+0000","last_change":"2024-09-15T02:13:05.791969+0000","last_active":"2024-09-15T02:13:05.791996+0000","last_peered":"2024-09-15T02:13:05.791996+0000","last_clean":"2024-09-15T02:13:05.791996+0000","last_became_active":"2024-09-15T01:58:09.937273+0000","last_became_peered":"2024-09-15T01:58:09.937273+0000","last_unstale":"2024-09-15T02:13:05.791996+0000","last_undegraded":"2024-09-15T02:13:05.791996+0000","last_fullsized":"2024-09-15T02:13:05.791996+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:13:05.791878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:13:05.791878+0000","last_clean_scrub_stamp":"2024-09-15T02:13:05.791878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:11:46.089331+0000","scrub_duration":0.0016984719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":117,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:43.898616+0000","last_change":"2024-09-15T02:12:43.898568+0000","last_active":"2024-09-15T02:12:43.898616+0000","last_peered":"2024-09-15T02:12:43.898616+0000","last_clean":"2024-09-15T02:12:43.898616+0000","last_became_active":"2024-09-15T01:58:09.926317+0000","last_became_peered":"2024-09-15T01:58:09.926317+0000","last_unstale":"2024-09-15T02:12:43.898616+0000","last_undegraded":"2024-09-15T02:12:43.898616+0000","last_fullsized":"2024-09-15T02:12:43.898616+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:43.898473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:43.898473+0000","last_clean_scrub_stamp":"2024-09-15T02:12:43.898473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:14:26.471237+0000","scrub_duration":0.001568446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":115,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:51.963239+0000","last_change":"2024-09-15T02:12:51.963201+0000","last_active":"2024-09-15T02:12:51.963239+0000","last_peered":"2024-09-15T02:12:51.963239+0000","last_clean":"2024-09-15T02:12:51.963239+0000","last_became_active":"2024-09-15T01:58:09.925609+0000","last_became_peered":"2024-09-15T01:58:09.925609+0000","last_unstale":"2024-09-15T02:12:51.963239+0000","last_undegraded":"2024-09-15T02:12:51.963239+0000","last_fullsized":"2024-09-15T02:12:51.963239+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:51.963104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:51.963104+0000","last_clean_scrub_stamp":"2024-09-15T02:12:51.963104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:57:17.425572+0000","scrub_duration":0.001969972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":136,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:51.783736+0000","last_change":"2024-09-15T02:12:51.783690+0000","last_active":"2024-09-15T02:12:51.783736+0000","last_peered":"2024-09-15T02:12:51.783736+0000","last_clean":"2024-09-15T02:12:51.783736+0000","last_became_active":"2024-09-15T01:58:09.929053+0000","last_became_peered":"2024-09-15T01:58:09.929053+0000","last_unstale":"2024-09-15T02:12:51.783736+0000","last_undegraded":"2024-09-15T02:12:51.783736+0000","last_fullsized":"2024-09-15T02:12:51.783736+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:51.783578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:51.783578+0000","last_clean_scrub_stamp":"2024-09-15T02:12:51.783578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:57:09.868285+0000","scrub_duration":0.001808516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:07.812286+0000","last_change":"2024-09-15T02:13:07.812252+0000","last_active":"2024-09-15T02:13:07.812286+0000","last_peered":"2024-09-15T02:13:07.812286+0000","last_clean":"2024-09-15T02:13:07.812286+0000","last_became_active":"2024-09-15T01:58:09.926669+0000","last_became_peered":"2024-09-15T01:58:09.926669+0000","last_unstale":"2024-09-15T02:13:07.812286+0000","last_undegraded":"2024-09-15T02:13:07.812286+0000","last_fullsized":"2024-09-15T02:13:07.812286+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:13:07.812142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:13:07.812142+0000","last_clean_scrub_stamp":"2024-09-15T02:13:07.812142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:05.375128+0000","scrub_duration":0.0015481570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":95,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:56.933937+0000","last_change":"2024-09-15T02:12:56.933900+0000","last_active":"2024-09-15T02:12:56.933937+0000","last_peered":"2024-09-15T02:12:56.933937+0000","last_clean":"2024-09-15T02:12:56.933937+0000","last_became_active":"2024-09-15T01:58:09.927021+0000","last_became_peered":"2024-09-15T01:58:09.927021+0000","last_unstale":"2024-09-15T02:12:56.933937+0000","last_undegraded":"2024-09-15T02:12:56.933937+0000","last_fullsized":"2024-09-15T02:12:56.933937+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:56.933793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:56.933793+0000","last_clean_scrub_stamp":"2024-09-15T02:12:56.933793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:09:07.328928+0000","scrub_duration":0.001859426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":190,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:44.304696+0000","last_change":"2024-09-15T02:12:44.304667+0000","last_active":"2024-09-15T02:12:44.304696+0000","last_peered":"2024-09-15T02:12:44.304696+0000","last_clean":"2024-09-15T02:12:44.304696+0000","last_became_active":"2024-09-15T01:58:09.925634+0000","last_became_peered":"2024-09-15T01:58:09.925634+0000","last_unstale":"2024-09-15T02:12:44.304696+0000","last_undegraded":"2024-09-15T02:12:44.304696+0000","last_fullsized":"2024-09-15T02:12:44.304696+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:44.304573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:44.304573+0000","last_clean_scrub_stamp":"2024-09-15T02:12:44.304573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:30:54.393137+0000","scrub_duration":0.001302377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:47.933263+0000","last_change":"2024-09-15T02:12:47.933202+0000","last_active":"2024-09-15T02:12:47.933263+0000","last_peered":"2024-09-15T02:12:47.933263+0000","last_clean":"2024-09-15T02:12:47.933263+0000","last_became_active":"2024-09-15T01:58:11.002122+0000","last_became_peered":"2024-09-15T01:58:11.002122+0000","last_unstale":"2024-09-15T02:12:47.933263+0000","last_undegraded":"2024-09-15T02:12:47.933263+0000","last_fullsized":"2024-09-15T02:12:47.933263+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:47.933100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:47.933100+0000","last_clean_scrub_stamp":"2024-09-15T02:12:47.933100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:37:11.097929+0000","scrub_duration":0.001854079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:47.946069+0000","last_change":"2024-09-15T02:12:47.946042+0000","last_active":"2024-09-15T02:12:47.946069+0000","last_peered":"2024-09-15T02:12:47.946069+0000","last_clean":"2024-09-15T02:12:47.946069+0000","last_became_active":"2024-09-15T01:58:09.928829+0000","last_became_peered":"2024-09-15T01:58:09.928829+0000","last_unstale":"2024-09-15T02:12:47.946069+0000","last_undegraded":"2024-09-15T02:12:47.946069+0000","last_fullsized":"2024-09-15T02:12:47.946069+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:47.945958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:47.945958+0000","last_clean_scrub_stamp":"2024-09-15T02:12:47.945958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:25.464773+0000","scrub_duration":0.0020129380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":247,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:57.871035+0000","last_change":"2024-09-15T02:12:57.870995+0000","last_active":"2024-09-15T02:12:57.871035+0000","last_peered":"2024-09-15T02:12:57.871035+0000","last_clean":"2024-09-15T02:12:57.871035+0000","last_became_active":"2024-09-15T01:58:09.925404+0000","last_became_peered":"2024-09-15T01:58:09.925404+0000","last_unstale":"2024-09-15T02:12:57.871035+0000","last_undegraded":"2024-09-15T02:12:57.871035+0000","last_fullsized":"2024-09-15T02:12:57.871035+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:57.870904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:57.870904+0000","last_clean_scrub_stamp":"2024-09-15T02:12:57.870904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:38.069515+0000","scrub_duration":0.0020375380000000002,"objects_trimmed":0,"snaptrim_duration":0.00033093700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":241,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:01.768216+0000","last_change":"2024-09-15T02:13:01.768190+0000","last_active":"2024-09-15T02:13:01.768216+0000","last_peered":"2024-09-15T02:13:01.768216+0000","last_clean":"2024-09-15T02:13:01.768216+0000","last_became_active":"2024-09-15T01:58:10.926657+0000","last_became_peered":"2024-09-15T01:58:10.926657+0000","last_unstale":"2024-09-15T02:13:01.768216+0000","last_undegraded":"2024-09-15T02:13:01.768216+0000","last_fullsized":"2024-09-15T02:13:01.768216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:13:01.768094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:13:01.768094+0000","last_clean_scrub_stamp":"2024-09-15T02:13:01.768094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:19:57.268196+0000","scrub_duration":0.0016588079999999999,"objects_trimmed":0,"snaptrim_duration":0.000341689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"257'3","reported_seq":97,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:52.775551+0000","last_change":"2024-09-15T02:12:52.775524+0000","last_active":"2024-09-15T02:12:52.775551+0000","last_peered":"2024-09-15T02:12:52.775551+0000","last_clean":"2024-09-15T02:12:52.775551+0000","last_became_active":"2024-09-15T01:58:09.925453+0000","last_became_peered":"2024-09-15T01:58:09.925453+0000","last_unstale":"2024-09-15T02:12:52.775551+0000","last_undegraded":"2024-09-15T02:12:52.775551+0000","last_fullsized":"2024-09-15T02:12:52.775551+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'3","last_scrub_stamp":"2024-09-15T02:12:52.775442+0000","last_deep_scrub":"257'3","last_deep_scrub_stamp":"2024-09-15T02:12:52.775442+0000","last_clean_scrub_stamp":"2024-09-15T02:12:52.775442+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:51.403534+0000","scrub_duration":0.0029779149999999998,"objects_trimmed":0,"snaptrim_duration":0.00038326300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"257'2","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:04.834045+0000","last_change":"2024-09-15T02:13:04.834012+0000","last_active":"2024-09-15T02:13:04.834045+0000","last_peered":"2024-09-15T02:13:04.834045+0000","last_clean":"2024-09-15T02:13:04.834045+0000","last_became_active":"2024-09-15T01:58:09.929514+0000","last_became_peered":"2024-09-15T01:58:09.929514+0000","last_unstale":"2024-09-15T02:13:04.834045+0000","last_undegraded":"2024-09-15T02:13:04.834045+0000","last_fullsized":"2024-09-15T02:13:04.834045+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'2","last_scrub_stamp":"2024-09-15T02:13:04.833924+0000","last_deep_scrub":"257'2","last_deep_scrub_stamp":"2024-09-15T02:13:04.833924+0000","last_clean_scrub_stamp":"2024-09-15T02:13:04.833924+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:50:41.621185+0000","scrub_duration":0.0034058640000000002,"objects_trimmed":0,"snaptrim_duration":0.00049495300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"257'74","reported_seq":376,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:45.818132+0000","last_change":"2024-09-15T02:12:45.818099+0000","last_active":"2024-09-15T02:12:45.818132+0000","last_peered":"2024-09-15T02:12:45.818132+0000","last_clean":"2024-09-15T02:12:45.818132+0000","last_became_active":"2024-09-15T01:58:09.934656+0000","last_became_peered":"2024-09-15T01:58:09.934656+0000","last_unstale":"2024-09-15T02:12:45.818132+0000","last_undegraded":"2024-09-15T02:12:45.818132+0000","last_fullsized":"2024-09-15T02:12:45.818132+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'74","last_scrub_stamp":"2024-09-15T02:12:45.818006+0000","last_deep_scrub":"257'74","last_deep_scrub_stamp":"2024-09-15T02:12:45.818006+0000","last_clean_scrub_stamp":"2024-09-15T02:12:45.818006+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:19:26.985216+0000","scrub_duration":0.0045836890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:13:02.848100+0000","last_change":"2024-09-15T02:13:02.848062+0000","last_active":"2024-09-15T02:13:02.848100+0000","last_peered":"2024-09-15T02:13:02.848100+0000","last_clean":"2024-09-15T02:13:02.848100+0000","last_became_active":"2024-09-15T01:58:09.934750+0000","last_became_peered":"2024-09-15T01:58:09.934750+0000","last_unstale":"2024-09-15T02:13:02.848100+0000","last_undegraded":"2024-09-15T02:13:02.848100+0000","last_fullsized":"2024-09-15T02:13:02.848100+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:13:02.847966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:13:02.847966+0000","last_clean_scrub_stamp":"2024-09-15T02:13:02.847966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:27.071540+0000","scrub_duration":0.0020878400000000001,"objects_trimmed":0,"snaptrim_duration":0.00043315500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":92,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:43.760575+0000","last_change":"2024-09-15T02:12:43.760535+0000","last_active":"2024-09-15T02:12:43.760575+0000","last_peered":"2024-09-15T02:12:43.760575+0000","last_clean":"2024-09-15T02:12:43.760575+0000","last_became_active":"2024-09-15T01:58:09.921537+0000","last_became_peered":"2024-09-15T01:58:09.921537+0000","last_unstale":"2024-09-15T02:12:43.760575+0000","last_undegraded":"2024-09-15T02:12:43.760575+0000","last_fullsized":"2024-09-15T02:12:43.760575+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:43.760432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:43.760432+0000","last_clean_scrub_stamp":"2024-09-15T02:12:43.760432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:28:01.885095+0000","scrub_duration":0.00167812,"objects_trimmed":0,"snaptrim_duration":0.00040825799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:55.923053+0000","last_change":"2024-09-15T02:12:55.923024+0000","last_active":"2024-09-15T02:12:55.923053+0000","last_peered":"2024-09-15T02:12:55.923053+0000","last_clean":"2024-09-15T02:12:55.923053+0000","last_became_active":"2024-09-15T01:58:09.923416+0000","last_became_peered":"2024-09-15T01:58:09.923416+0000","last_unstale":"2024-09-15T02:12:55.923053+0000","last_undegraded":"2024-09-15T02:12:55.923053+0000","last_fullsized":"2024-09-15T02:12:55.923053+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:55.922929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:55.922929+0000","last_clean_scrub_stamp":"2024-09-15T02:12:55.922929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:43:36.198779+0000","scrub_duration":0.001954744,"objects_trimmed":0,"snaptrim_duration":0.00034811299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":248,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-15T02:12:46.846850+0000","last_change":"2024-09-15T02:12:46.846822+0000","last_active":"2024-09-15T02:12:46.846850+0000","last_peered":"2024-09-15T02:12:46.846850+0000","last_clean":"2024-09-15T02:12:46.846850+0000","last_became_active":"2024-09-15T01:58:09.928519+0000","last_became_peered":"2024-09-15T01:58:09.928519+0000","last_unstale":"2024-09-15T02:12:46.846850+0000","last_undegraded":"2024-09-15T02:12:46.846850+0000","last_fullsized":"2024-09-15T02:12:46.846850+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:12:46.846731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:12:46.846731+0000","last_clean_scrub_stamp":"2024-09-15T02:12:46.846731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:45:40.047718+0000","scrub_duration":0.0019948240000000001,"objects_trimmed":0,"snaptrim_duration":0.00038012899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.497},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.71699999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53600000000000003,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.53300000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.30399999999999999},"max":{"1min":0.60799999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":0.64300000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52700000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.63500000000000001,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.51200000000000001,"15min":0.502},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.28799999999999998},"max":{"1min":0.71099999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.42799999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.38300000000000001},"min":{"1min":0.26800000000000002,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":0.53600000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40400000000000003,"15min":0.38900000000000001},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.60899999999999999,"5min":0.745,"15min":0.745},"last":0.33200000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:12:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.23000000000000001},"max":{"1min":0.57699999999999996,"5min":0.879,"15min":0.879},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.46999999999999997}]}]},{"osd":4,"up_from":11,"seq":47244640520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298448,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073392,"statfs":{"total":96636764160,"available":96331153408,"internally_reserved":0,"allocated":1122304,"data_stored":863404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.34100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67100000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.60699999999999998,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Sun Sep 15 02:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64100000000000001,"5min":0.746,"15min":0.80200000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.32000000000000001},"max":{"1min":0.69299999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51300000000000001,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":0.65300000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.38800000000000001,"5min":0.36399999999999999,"15min":0.28199999999999997},"max":{"1min":0.67700000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.5}]},{"osd":3,"last update":"Sun Sep 15 02:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.23200000000000001},"max":{"1min":0.47399999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.40100000000000002,"15min":0.40200000000000002},"min":{"1min":0.27600000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.502,"5min":0.63,"15min":0.66100000000000003},"last":0.27600000000000002}]},{"osd":5,"last update":"Sun Sep 15 02:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.40699999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.247},"max":{"1min":0.57199999999999995,"5min":0.69099999999999995,"15min":0.71299999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.47099999999999997,"15min":0.45900000000000002},"min":{"1min":0.316,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39000000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1650688,"data_stored":1392313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:13:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.52300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.42399999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71899999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61699999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:13:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.379,"5min":0.254,"15min":0.254},"max":{"1min":0.68700000000000006,"5min":1.022,"15min":1.022},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.34799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80200000000000005,"5min":1.008,"15min":1.008},"last":0.53200000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.374,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71199999999999997,"5min":0.747,"15min":0.747},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67600000000000005,"5min":0.72999999999999998,"15min":0.84899999999999998},"last":0.48099999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:13:04 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39500000000000002,"15min":0.38300000000000001},"min":{"1min":0.28299999999999997,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48199999999999998,"5min":0.61099999999999999,"15min":0.76900000000000002},"last":0.371},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41599999999999998,"15min":0.39900000000000002},"min":{"1min":0.30199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56200000000000006,"5min":0.70199999999999996,"15min":0.80000000000000004},"last":0.41699999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42499999999999999,"15min":0.40999999999999998},"min":{"1min":0.33500000000000002,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":0.58699999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42299999999999999,"15min":0.41399999999999998},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56000000000000005,"5min":0.71699999999999997,"15min":0.77800000000000002},"last":0.39500000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.5,"5min":0.53500000000000003,"15min":0.53800000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.20999999999999999},"max":{"1min":0.51800000000000002,"5min":0.622,"15min":0.622},"last":0.38}]},{"osd":1,"last update":"Sun Sep 15 02:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.39100000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.64700000000000002},"last":0.442},{"interface":"front","average":{"1min":0.38,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.25600000000000001,"5min":0.224,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.59999999999999998,"15min":0.70199999999999996},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.34399999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.88100000000000001},"last":0.63100000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54000000000000004,"15min":0.55000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.753,"5min":0.78200000000000003,"15min":0.82499999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.78400000000000003,"15min":0.80800000000000005},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":0.79500000000000004,"15min":0.91000000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.751,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.59199999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640521,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303128,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068712,"statfs":{"total":96636764160,"available":96326361088,"internally_reserved":0,"allocated":1654784,"data_stored":1392332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.316,"15min":0.32600000000000001},"min":{"1min":0.214,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.60599999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.33600000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19700000000000001},"max":{"1min":0.498,"5min":0.56899999999999995,"15min":0.58399999999999996},"last":0.33700000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.36399999999999999},"min":{"1min":0.22600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.53600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.377,"15min":0.38400000000000001},"min":{"1min":0.27600000000000002,"5min":0.23699999999999999,"15min":0.19500000000000001},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.39000000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.84999999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.72199999999999998,"5min":0.73099999999999998,"15min":0.77700000000000002},"last":0.497}]},{"osd":4,"last update":"Sun Sep 15 02:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79400000000000004,"15min":0.81899999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.48599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":0.84199999999999997,"15min":0.86499999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sun Sep 15 02:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.90300000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60399999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":1118208,"data_stored":863385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.314,"15min":0.32900000000000001},"min":{"1min":0.216,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":0.41799999999999998,"5min":0.49299999999999999,"15min":0.65900000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38200000000000001,"15min":0.377},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.19400000000000001},"max":{"1min":0.48799999999999999,"5min":0.63300000000000001,"15min":0.73899999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.33000000000000002,"15min":0.35999999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.39800000000000002,"5min":0.48499999999999999,"15min":0.79300000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.38200000000000001,"15min":0.36099999999999999},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.20699999999999999},"max":{"1min":0.48099999999999998,"5min":0.57099999999999995,"15min":0.69299999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Sun Sep 15 02:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63900000000000001,"5min":0.80600000000000005,"15min":0.81999999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70599999999999996,"5min":0.81699999999999995,"15min":0.81899999999999995},"last":0.54900000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.53400000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.66000000000000003,"5min":0.76900000000000002,"15min":1.075},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78600000000000003,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.42799999999999999,"5min":0.33400000000000002,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":1.2729999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.35599999999999998},"max":{"1min":0.78400000000000003,"5min":0.81799999999999995,"15min":1.1419999999999999},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:13:24.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T02:13:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 10 monclient: get_monmap_and_config 2024-09-15T02:13:24.321 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 10 monclient: build_initial_monmap 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 10 monclient: monmap: 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T02:13:24.318120+0000 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T02:13:24.318120+0000 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:0: v1:172.21.15.17:6789/0 mon.noname-a-legacy 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:1: v1:172.21.15.67:6789/0 mon.noname-c-legacy 2024-09-15T02:13:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-b 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:3: v2:172.21.15.17:3300/0 mon.noname-a 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:4: v2:172.21.15.67:3300/0 mon.noname-c 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 1 Processor -- start 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 1 -- start start 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.317+0000 7fd02442b640 10 monclient: init 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd01c131a80 addr v1:172.21.15.67:6789/0 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd01c09eba0 0x7fd01c09efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c09eba0 addr v2:172.21.15.17:3300/0 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 0x7fd01c139190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c136db0 addr v2:172.21.15.67:3300/0 2024-09-15T02:13:24.323 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c0a15f0 con 0x7fd01c131a80 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 1 -- --> v2:172.21.15.17:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c132680 con 0x7fd01c09eba0 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c0a1310 con 0x7fd01c136db0 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd02442b640 10 monclient(hunting): authenticate will time out at 2064.314561s 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd01c09eba0 0x7fd01c09efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 --2- >> v2:172.21.15.17:3300/0 conn(0x7fd01c09eba0 0x7fd01c09efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34974/0 (socket says 172.21.15.17:34974) 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 -- 172.21.15.17:0/744476727 learned_addr learned my addr 172.21.15.17:0/744476727 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): get_auth_request con 0x7fd01c09eba0 auth_method 0 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0229c8640 1 --2- 172.21.15.17:0/744476727 >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 0x7fd01c139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient(hunting): handle_auth_done global_id 254223 payload 274 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 -- 172.21.15.17:0/744476727 >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 msgr2=0x7fd01c139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 --2- 172.21.15.17:0/744476727 >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 0x7fd01c139190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 -- 172.21.15.17:0/744476727 >> v1:172.21.15.67:6789/0 conn(0x7fd01c131a80 legacy=0x7fd01c131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient: _finish_hunting 0 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 monclient: found mon.noname-a 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.17:3300/0 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0219c6640 1 -- 172.21.15.17:0/744476727 --> v2:172.21.15.17:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c09f4e0 con 0x7fd01c09eba0 2024-09-15T02:13:24.325 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0229c8640 10 monclient: get_auth_request con 0x7fd01c136db0 auth_method 0 2024-09-15T02:13:24.326 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.318+0000 7fd0229c8640 1 --2- 172.21.15.17:0/744476727 >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 0x7fd01c139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T02:13:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.321+0000 7fd0219c6640 1 -- 172.21.15.17:0/744476727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 msgr2=0x7fd01c09efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.321+0000 7fd0219c6640 1 --2- 172.21.15.17:0/744476727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c09efa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd01c068cd0 tx=0x7fd00c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7878d3c0aa89dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 1 -- 172.21.15.17:0/744476727 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c032440 con 0x7fd01c09eba0 2024-09-15T02:13:24.330 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: dump: 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: _finish_auth 0 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 5 monclient: authenticate success, global_id 254223 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 1 -- 172.21.15.17:0/744476727 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c0325a0 con 0x7fd01c09eba0 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 10 monclient: get_monmap_and_config success 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 4 set_mon_vals no callback set 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 1 -- 172.21.15.17:0/744476727 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c032940 con 0x7fd01c09eba0 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd0211c5640 10 monclient: dump: 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 10 monclient: shutdown 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 -- 172.21.15.17:0/744476727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 msgr2=0x7fd01c09efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 --2- 172.21.15.17:0/744476727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c09efa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd01c068cd0 tx=0x7fd00c0091d0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 -- 172.21.15.17:0/744476727 shutdown_connections 2024-09-15T02:13:24.333 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 --2- 172.21.15.17:0/744476727 >> v2:172.21.15.67:3300/0 conn(0x7fd01c136db0 0x7fd01c139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 --2- 172.21.15.17:0/744476727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c09efa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 -- 172.21.15.17:0/744476727 >> 172.21.15.17:0/744476727 conn(0x7fd01c071320 msgr2=0x7fd01c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 -- 172.21.15.17:0/744476727 shutdown_connections 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.322+0000 7fd02442b640 1 -- 172.21.15.17:0/744476727 wait complete. 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.323+0000 7fd02442b640 10 monclient: build_initial_monmap 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.323+0000 7fd02442b640 10 monclient: monmap: 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 0 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 0.000000 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:created 0.000000 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 0 (unknown) 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.noname-a 2024-09-15T02:13:24.334 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-b 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.noname-c 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.323+0000 7fd02442b640 1 Processor -- start 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.323+0000 7fd02442b640 1 -- start start 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.323+0000 7fd02442b640 10 monclient: init 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient: _reopen_session rank -1 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c09eba0 addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 0x7fd01c097720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c131a80 addr [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c136db0 0x7fd01c097c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.335 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c136db0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c09bec0 con 0x7fd01c09eba0 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c069b00 con 0x7fd01c136db0 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): start opening mon connection 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 1 -- --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c09ab10 con 0x7fd01c131a80 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): _renew_subs 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd02442b640 10 monclient(hunting): authenticate will time out at 2064.320199s 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:34980/0 (socket says 172.21.15.17:34980) 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 --2- >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 0x7fd01c097720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 1 -- 172.21.15.17:0/1894728137 learned_addr learned my addr 172.21.15.17:0/1894728137 (peer_addr_for_me v2:172.21.15.17:0/0) 2024-09-15T02:13:24.336 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): get_auth_request con 0x7fd01c09eba0 auth_method 0 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): get_auth_request con 0x7fd01c131a80 auth_method 0 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c136db0 0x7fd01c097c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): get_auth_request con 0x7fd01c136db0 auth_method 0 2024-09-15T02:13:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0229c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient(hunting): handle_auth_done global_id 214634 payload 824 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 -- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c136db0 msgr2=0x7fd01c097c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c136db0 0x7fd01c097c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 -- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 msgr2=0x7fd01c0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.338 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient: _finish_hunting 0 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 monclient: found mon.noname-c 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0219c6640 1 -- 172.21.15.17:0/1894728137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c009450 con 0x7fd01c131a80 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.324+0000 7fd0221c7640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.326+0000 7fd0219c6640 1 -- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 msgr2=0x7fd01c097720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.326+0000 7fd0219c6640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 0x7fd01c097720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd01c071720 tx=0x7fd00c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c031a90 con 0x7fd01c131a80 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: dump: 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.339 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: _finish_auth 0 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c031bf0 con 0x7fd01c131a80 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 5 monclient: authenticate success, global_id 214634 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 10 monclient: _renew_subs 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.340 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c098d70 con 0x7fd01c131a80 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c03d440 con 0x7fd01c131a80 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0127fc640 10 monclient: dump: 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:epoch 1 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:fsid e3089478-c52a-4174-94a2-258b1876359d 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:last_changed 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:created 2024-09-15T01:50:53.865793+0000 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:min_mon_release 17 (quincy) 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:election_strategy: 1 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T02:13:24.341 INFO:teuthology.orchestra.run.smithi017.stderr:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd0209c4640 4 set_mon_vals no callback set 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 10 monclient: _renew_subs 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.327+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c09a8e0 con 0x7fd01c131a80 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd00c039040 con 0x7fd01c131a80 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd02442b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd02442b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd02442b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01c0724a0 con 0x7fd01c131a80 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd0127fc640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 0x7fd00003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 5 ==== osd_map(257..257 src has 229..257) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fd00c06c0b0 con 0x7fd01c131a80 2024-09-15T02:13:24.342 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd0221c7640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 0x7fd00003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T02:13:24.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.328+0000 7fd0221c7640 10 monclient: get_auth_request con 0x7fd000038740 auth_method 0 2024-09-15T02:13:24.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.331+0000 7fd0221c7640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 0x7fd00003abf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd018009ed0 tx=0x7fd0180090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T02:13:24.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.332+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd00c030940 con 0x7fd01c131a80 2024-09-15T02:13:24.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.332+0000 7fd0127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T02:13:24.343 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.332+0000 7fd0127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T02:13:24.496 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.495+0000 7fd02442b640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:13:24.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.495+0000 7fd02442b640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:13:24.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.495+0000 7fd02442b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.17:3301/0 2024-09-15T02:13:24.497 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.495+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 --> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fd01c09ac80 con 0x7fd01c131a80 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd00c030940 con 0x7fd01c131a80 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0127fc640 10 monclient: handle_config config(1 keys) v1 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0209c4640 4 set_mon_vals no callback set 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0209c4640 10 set_mon_vals mon_health_to_clog = false 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0127fc640 1 -- 172.21.15.17:0/1894728137 <== mon.2 v2:172.21.15.17:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fd00c0370c0 con 0x7fd01c131a80 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T02:13:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.500+0000 7fd0127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 msgr2=0x7fd00003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 0x7fd00003abf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd018009ed0 tx=0x7fd0180090f0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 10 monclient: shutdown 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 msgr2=0x7fd01c097720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T02:13:24.504 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 0x7fd01c097720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd01c071720 tx=0x7fd00c032000 comp rx=0 tx=0).stop 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 shutdown_connections 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:6824/31455,v1:172.21.15.67:6825/31455] conn(0x7fd000038740 0x7fd00003abf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c136db0 0x7fd01c097c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] conn(0x7fd01c131a80 0x7fd01c097720 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 --2- 172.21.15.17:0/1894728137 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd01c09eba0 0x7fd01c0971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 >> 172.21.15.17:0/1894728137 conn(0x7fd01c071320 msgr2=0x7fd01c09d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 shutdown_connections 2024-09-15T02:13:24.505 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-15T02:13:24.502+0000 7fd02442b640 1 -- 172.21.15.17:0/1894728137 wait complete. 2024-09-15T02:13:24.514 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T02:13:24.514 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T02:13:24.515 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T02:13:24.515 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.549 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T02:13:24.549 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T02:13:24.550 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.621 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T02:13:24.622 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T02:13:24.622 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.687 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T02:13:24.687 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T02:13:24.687 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.722 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T02:13:24.723 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T02:13:24.723 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.790 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T02:13:24.790 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T02:13:24.790 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.857 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T02:13:24.857 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T02:13:24.858 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T02:13:24.858 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.928 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T02:13:24.928 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T02:13:24.928 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:13:24.928 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.940 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:13:24.940 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:13:24.940 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:24.994 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:13:24.995 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:13:24.995 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:13:25.009 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:13:25.010 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T02:13:25.010 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-15T02:13:25.069 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T02:13:25.069 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T02:13:25.274 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T02:13:25.275 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T02:13:25.360 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T02:13:25.360 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T02:13:25.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T02:13:25.410 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T02:13:25.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T02:13:25.513 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T02:13:25.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T02:13:25.602 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T02:13:25.659 INFO:tasks.ceph:Archiving mon data... 2024-09-15T02:13:25.661 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/data/mon.a.tgz 2024-09-15T02:13:25.661 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-15T02:13:25.677 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.MjL83RJX6v 2024-09-15T02:13:25.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MjL83RJX6v 2024-09-15T02:13:25.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.MjL83RJX6v 2024-09-15T02:13:25.977 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.MjL83RJX6v is 850KB 2024-09-15T02:13:26.082 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.MjL83RJX6v 2024-09-15T02:13:26.098 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/data/mon.c.tgz 2024-09-15T02:13:26.098 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-15T02:13:26.153 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.oVnE0zDSz8 2024-09-15T02:13:26.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.oVnE0zDSz8 2024-09-15T02:13:26.345 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.oVnE0zDSz8 2024-09-15T02:13:26.429 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.oVnE0zDSz8 is 852KB 2024-09-15T02:13:26.538 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.oVnE0zDSz8 2024-09-15T02:13:26.554 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/data/mon.b.tgz 2024-09-15T02:13:26.554 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-15T02:13:26.573 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.Q20IaYdvIR 2024-09-15T02:13:26.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Q20IaYdvIR 2024-09-15T02:13:26.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.Q20IaYdvIR 2024-09-15T02:13:26.874 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.Q20IaYdvIR is 854KB 2024-09-15T02:13:26.975 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.Q20IaYdvIR 2024-09-15T02:13:26.991 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T02:13:26.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:13:26.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:13:27.055 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T02:13:27.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:13:27.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:13:27.087 INFO:teuthology.orchestra.run.smithi017.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:13:27.123 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:13:27.128 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T02:13:27.130 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi017/crash 2024-09-15T02:13:27.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:13:27.159 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi067/crash 2024-09-15T02:13:27.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:13:27.193 INFO:tasks.ceph:Compressing logs... 2024-09-15T02:13:27.193 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:13:27.204 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:13:27.226 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T02:13:27.228 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45137.log 2024-09-15T02:13:27.228 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37935.log 2024-09-15T02:13:27.228 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.45137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41496.log 2024-09-15T02:13:27.228 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52949.log 2024-09-15T02:13:27.228 INFO:teuthology.orchestra.run.smithi017.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45137.log.gz 2024-09-15T02:13:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43297.log 2024-09-15T02:13:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41496.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37935.log.gz 2024-09-15T02:13:27.229 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51377.log 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52949.log.gz 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41496.log.gz 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43297.log.gz 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48760.log 2024-09-15T02:13:27.255 INFO:teuthology.orchestra.run.smithi017.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51377.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47415.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35388.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48760.log: /var/log/ceph/ceph-client.admin.47415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41767.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35388.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48760.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47625.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47415.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41767.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35388.log.gz 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-09-15T02:13:27.256 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41767.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47625.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45095.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34527.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45095.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51546.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51840.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.45095.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34527.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50265.log 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51546.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2024-09-15T02:13:27.257 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51756.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41683.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51546.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50685.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50265.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40803.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51756.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50685.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.41683.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47373.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52991.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40803.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50685.log.gz 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49096.log 2024-09-15T02:13:27.258 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50349.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47373.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47541.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52991.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43918.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49096.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34687.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50349.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47541.log: /var/log/ceph/ceph-client.admin.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35119.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35295.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47541.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43918.log.gz 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43253.log 2024-09-15T02:13:27.259 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34687.log.gz 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46137.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35119.log.gz 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35295.log.gz 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46137.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.43253.log.gz --verbose 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.44968.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46137.log.gz 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38103.log: gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44968.log.gz /var/log/ceph/ceph-client.admin.44422.log 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.260 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42358.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37706.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45742.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.44422.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35737.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42358.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43423.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38187.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45742.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.35737.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27310.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51125.log 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35737.log.gz 2024-09-15T02:13:27.261 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.tmp-client.admin.27310.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43423.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37243.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27310.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51125.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45179.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37243.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46683.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45179.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41370.log: /var/log/ceph/ceph-client.admin.46683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-09-15T02:13:27.262 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45179.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40845.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53075.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46683.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37159.log: /var/log/ceph/ceph-client.admin.53075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46263.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49970.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53075.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38439.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46263.log.gz 2024-09-15T02:13:27.263 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41118.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49970.log.gz 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38626.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38439.log.gz 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44506.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40257.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz/var/log/ceph/ceph-client.admin.44296.log: 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45240.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44506.log.gz 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38668.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2024-09-15T02:13:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38668.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45240.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40257.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52092.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38668.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52234.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35863.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52092.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52234.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46851.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53201.log 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52529.log: /var/log/ceph/ceph-client.admin.46851.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35863.log.gz 2024-09-15T02:13:27.265 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35821.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46851.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53201.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37748.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35821.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48592.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45952.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40299.log 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37748.log.gz 2024-09-15T02:13:27.266 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51462.log 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45952.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48592.log.gz 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46809.log 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53327.log 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40299.log.gz 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45952.log.gz 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51462.log.gz 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50391.log 2024-09-15T02:13:27.267 INFO:teuthology.orchestra.run.smithi017.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46809.log.gz 2024-09-15T02:13:27.518 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38267.log: /var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39333.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41080.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37279.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38917.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39333.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36531.log 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38917.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39333.log.gz 2024-09-15T02:13:27.519 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37279.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38917.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36531.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38799.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35154.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39284.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2024-09-15T02:13:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36187.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39284.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35916.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36187.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37442.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35916.log.gz 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-09-15T02:13:27.521 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37442.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34823.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37442.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35940.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41176.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34823.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35940.log.gz 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-09-15T02:13:27.522 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41176.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38339.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35296.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41418.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38339.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35296.log.gz 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-09-15T02:13:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39987.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41418.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35318.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36016.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42229.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40156.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35318.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38555.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36016.log.gz 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-09-15T02:13:27.524 INFO:teuthology.orchestra.run.smithi067.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.foo.42229.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36839.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40156.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35362.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38555.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42622.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36839.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37658.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35362.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36743.log 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-09-15T02:13:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39616.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bazar.42122.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36743.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.bar.42622.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37778.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bazar.42122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 88.6% 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39616.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.bazar.42122.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40815.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36937.log 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37778.log.gz 2024-09-15T02:13:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36163.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40815.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36937.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35991.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40936.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36163.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.35991.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38241.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40396.log 2024-09-15T02:13:27.527 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35991.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39694.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40936.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38241.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36767.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40396.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36695.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39694.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38533.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36767.log.gz 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40108.log 2024-09-15T02:13:27.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36695.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37610.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38701.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38533.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35772.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40108.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38313.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37610.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39513.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38701.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37133.log 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35772.log.gz 2024-09-15T02:13:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35724.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38313.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38413.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39513.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37133.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37417.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35724.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42885.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38413.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34801.log 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37417.log.gz 2024-09-15T02:13:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37754.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42885.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42534.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34801.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37754.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35868.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.bar.42534.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-09-15T02:13:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35868.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35471.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41008.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37826.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36507.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35471.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39486.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40035.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2024-09-15T02:13:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39486.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37826.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38291.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39486.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42789.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40035.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35495.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38291.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38653.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42295.log 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.bar.42789.log.gz 2024-09-15T02:13:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35796.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35495.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31709.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38653.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40960.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35796.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39640.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31709.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.foo.42295.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-09-15T02:13:27.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34867.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34959.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35567.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34959.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34867.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36211.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34959.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39115.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35567.log.gz 2024-09-15T02:13:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36211.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39091.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41637.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36211.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34935.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39091.log: /var/log/ceph/ceph-client.admin.39212.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36863.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39091.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34727.log 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36863.log: /var/log/ceph/ceph-client.admin.38775.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34935.log.gz 2024-09-15T02:13:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36863.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37467.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34727.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38022.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35340.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37467.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35340.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38022.log.gz 2024-09-15T02:13:27.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41539.log: /var/log/ceph/ceph-client.admin.41563.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35340.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41685.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36791.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42181.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41685.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41685.log.gz 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-09-15T02:13:27.538 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36791.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37540.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.foo.42181.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31589.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42475.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37540.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.40695.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31589.log: /var/log/ceph/ceph-client.foo.42475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36114.log 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31589.log.gz 2024-09-15T02:13:27.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40695.log: 84.0% -- replaced with /var/log/ceph/ceph-client.foo.42475.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36042.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36114.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42273.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36042.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37634.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40204.log 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.foo.42273.log.gz/var/log/ceph/ceph-client.admin.37634.log: 2024-09-15T02:13:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35210.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35820.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40204.log.gz 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35210.log.gzgzip 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36599.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35820.log: /var/log/ceph/ceph-client.admin.39164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38726.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35820.log.gz 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39819.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36599.log.gz 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38726.log: /var/log/ceph/ceph-client.admin.39819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40839.log 2024-09-15T02:13:27.541 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38726.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39819.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35447.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34776.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40839.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38751.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35447.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37015.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34776.log.gz 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-09-15T02:13:27.542 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38751.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41781.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37015.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35700.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34845.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41781.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40444.log/var/log/ceph/ceph-client.admin.34845.log: 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35700.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34845.log.gz 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37061.log 2024-09-15T02:13:27.543 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42251.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40444.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42251.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37061.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.foo.42251.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37730.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-09-15T02:13:27.544 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37730.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39939.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38943.log: /var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37179.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37111.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37179.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39939.log.gzgzip 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36989.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37179.log.gz 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-09-15T02:13:27.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37111.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36577.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37802.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36989.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37852.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37802.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36577.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36553.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37802.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37588.log 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37852.log.gz 2024-09-15T02:13:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40252.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36553.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35844.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34889.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36719.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37588.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36887.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35844.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42133.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34889.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36719.log.gz 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37347.log 2024-09-15T02:13:27.547 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36887.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42133.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38893.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34703.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38893.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37347.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40084.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40084.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34703.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.34913.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42157.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40084.log.gz 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-09-15T02:13:27.548 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34913.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42157.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37682.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37682.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35965.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37682.log.gz 2024-09-15T02:13:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37253.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35965.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35266.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42578.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37253.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37972.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35266.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.bar.42578.log.gz 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42012.log 2024-09-15T02:13:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38461.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37972.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35676.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42012.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35005.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36139.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36647.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35005.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39567.log 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35005.log.gz 2024-09-15T02:13:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36139.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39891.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36647.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38191.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39567.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38191.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39891.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38191.log.gz 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35184.log 2024-09-15T02:13:27.552 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36090.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35184.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37037.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35184.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42486.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36090.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39018.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38437.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37037.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42486.log.gz 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37998.log 2024-09-15T02:13:27.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39018.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36459.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38437.log.gz/var/log/ceph/ceph-client.admin.37998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39308.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37998.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38070.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36459.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.42861.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40791.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39308.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34981.log 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-09-15T02:13:27.554 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38070.log.gz 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40791.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41940.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34981.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40791.log.gz 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40468.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37301.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39462.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40468.log.gz 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40671.log 2024-09-15T02:13:27.555 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2%/var/log/ceph/ceph-client.admin.35591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37301.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.40671.log: -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.38968.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39540.log: gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40671.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%/var/log/ceph/ceph-client.admin.38968.log: -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36359.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41733.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38968.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36359.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-09-15T02:13:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39438.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36359.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip 88.8% -5 --verbose -- /var/log/ceph/ceph-client.bar.42556.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39867.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38094.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.bar.42556.log: /var/log/ceph/ceph-client.admin.37369.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 88.0% --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38094.log: -- replaced with /var/log/ceph/ceph-client.bar.42556.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38094.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-09-15T02:13:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38389.log: /var/log/ceph/ceph-client.admin.40912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40574.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.35426.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42451.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35748.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35426.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40622.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35426.log.gz 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35104.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35078.log 2024-09-15T02:13:27.558 INFO:teuthology.orchestra.run.smithi067.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35748.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40622.log.gzgzip 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36335.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35078.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35104.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41661.log: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35052.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35078.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36335.log: -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35052.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36335.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35405.log 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35052.log.gz 2024-09-15T02:13:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.b.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gzgzip -5 --verbose 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.39963.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31629.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35405.log: /var/log/ceph/ceph-client.admin.39963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35405.log.gz 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.31629.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36911.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41709.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31629.log.gz 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41916.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36066.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36911.log.gz 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40743.log 2024-09-15T02:13:27.560 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41916.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36409.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36066.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40743.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36066.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40743.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39771.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.41200.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36409.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38143.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.41310.log: --verbose -- /var/log/ceph/ceph-client.admin.39722.log 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39771.log.gz 2024-09-15T02:13:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41757.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38143.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39722.log: /var/log/ceph/ceph-client.admin.41757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36483.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37516.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39722.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41757.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38677.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36483.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31749.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38677.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36483.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39747.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36671.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38677.log.gz 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.39747.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-09-15T02:13:27.562 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31749.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37155.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36671.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39747.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36261.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37564.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36671.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1%/var/log/ceph/ceph-client.admin.36261.log: -- replaced with /var/log/ceph/ceph-client.admin.37155.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38365.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36261.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37564.log.gz 2024-09-15T02:13:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38509.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38365.log.gz 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.37876.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41892.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42600.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31800.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42959.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.bar.42600.log.gz 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35652.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31800.log: 88.6%/var/log/ceph/ceph-client.admin.42959.log: -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35652.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T02:13:27.564 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42959.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31800.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35652.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37325.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.37325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42098.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39843.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37325.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37085.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35519.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37900.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42098.log.gz 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2024-09-15T02:13:27.565 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37085.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37900.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35519.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.35128.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38215.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37900.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39067.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31669.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35128.log: /var/log/ceph/ceph-client.admin.39067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42205.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31669.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35128.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39067.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.foo.42205.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31669.log.gz 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35026.log 2024-09-15T02:13:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40719.log: 88.9% -- replaced with /var/log/ceph/ceph-client.foo.42205.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35026.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42510.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35026.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39411.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.bar.42510.log: /var/log/ceph/ceph-client.admin.37393.log: 88.5% -- replaced with /var/log/ceph/ceph-client.bar.42510.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38581.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41964.log 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38581.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39411.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41964.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38581.log.gz 2024-09-15T02:13:27.567 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41964.log.gz 2024-09-15T02:13:27.570 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T02:13:27.588 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-09-15T02:13:27.588 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53327.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37622.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50391.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47667.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50872.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47667.log: -- /var/log/ceph/ceph-client.admin.45700.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42019.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50872.log: /var/log/ceph/ceph-client.admin.45700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48340.log 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.47667.log.gz 2024-09-15T02:13:27.589 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42019.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50872.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36764.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44464.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42019.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36764.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48340.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46725.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44464.log: /var/log/ceph/ceph-client.admin.46725.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36764.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44464.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44758.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39004.log 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37201.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.46725.log.gz 2024-09-15T02:13:27.590 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50307.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39004.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44758.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51672.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39004.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38313.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50307.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51672.log.gz 2024-09-15T02:13:27.591 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49533.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38313.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45321.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38313.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37811.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49533.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49827.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45321.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42103.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37811.log: 88.7%/var/log/ceph/ceph-client.admin.49827.log: -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45406.log 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42103.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37811.log.gz 2024-09-15T02:13:27.592 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49827.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40007.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49323.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.42103.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45406.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34554.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43549.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49323.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46641.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41454.log 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43549.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.49323.log.gz 2024-09-15T02:13:27.593 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34554.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37035.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48928.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41454.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.43549.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46641.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44842.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41454.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39693.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37035.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42061.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44842.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.48928.log.gz 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.44842.log.gz -5 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2024-09-15T02:13:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42061.log: /var/log/ceph/ceph-client.admin.40952.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39693.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42862.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40593.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44044.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42862.log: /var/log/ceph/ceph-client.admin.40593.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48088.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42862.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42820.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44044.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40593.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44044.log.gz 2024-09-15T02:13:27.595 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44800.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48088.log.gz 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35343.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38920.log: /var/log/ceph/ceph-client.admin.44800.log: gzip 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42820.log.gz -5 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35121.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36386.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4%/var/log/ceph/ceph-client.admin.35121.log: -- replaced with /var/log/ceph/ceph-client.admin.44800.log.gz 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47061.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35343.log.gz 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36031.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35121.log.gz 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53117.log 2024-09-15T02:13:27.596 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36386.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53159.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47061.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53117.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36031.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36722.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44926.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53159.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53117.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44632.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4%/var/log/ceph/ceph-client.admin.44926.log: -- replaced with /var/log/ceph/ceph-client.admin.53159.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36722.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40130.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44926.log.gz 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48172.log 2024-09-15T02:13:27.597 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40130.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44632.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48172.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40130.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34867.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43381.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48172.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46095.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34867.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35905.log 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43381.log.gz 2024-09-15T02:13:27.598 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.39214.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48130.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46095.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35123.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35905.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48130.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35123.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48130.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48508.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49239.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35123.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52319.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48508.log.gz 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48004.log 2024-09-15T02:13:27.599 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49239.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52319.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47878.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41893.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48004.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47878.log: gzip -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46599.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47878.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48718.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38752.log 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34870.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46599.log.gz 2024-09-15T02:13:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48718.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42442.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38752.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51293.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36806.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38544.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36806.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.42442.log.gz 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52613.log 2024-09-15T02:13:27.601 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51293.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38544.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.36806.log.gz -5 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43158.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32011.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52613.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38544.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49659.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52613.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.32011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51630.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43158.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49659.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32011.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46347.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49659.log.gz 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2024-09-15T02:13:27.602 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51630.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53033.log 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.46347.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47331.log 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7%/var/log/ceph/ceph-client.admin.53033.log: -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52361.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47331.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53033.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35527.log 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47331.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35527.log: gzip -5 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41641.log 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52361.log.gz 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43200.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35527.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41641.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43776.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51504.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43200.log: /var/log/ceph/ceph-client.admin.43776.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34864.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43200.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49281.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43776.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46221.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34864.log: 88.8%/var/log/ceph/ceph-client.admin.49281.log: -- replaced with /var/log/ceph/ceph-client.admin.51504.log.gz 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48676.log 2024-09-15T02:13:27.604 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46221.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34864.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.49281.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39819.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44002.log: gzip 90.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.46221.log.gz --verbose 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.35779.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48676.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52823.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35350.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39819.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35611.log 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35350.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35779.log.gz 2024-09-15T02:13:27.605 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48046.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52823.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35611.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35350.log.gz /var/log/ceph/ceph-client.admin.51798.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48046.log: gzip 88.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35611.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.47499.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36304.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51798.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48046.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48298.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36304.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36974.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47499.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48298.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36304.log.gz 2024-09-15T02:13:27.606 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36974.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48298.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36512.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mon.c.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36974.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42990.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47962.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36512.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53243.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42990.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44674.log 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47962.log.gz 2024-09-15T02:13:27.607 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51966.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53243.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31928.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51966.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41977.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44674.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: 88.0%/var/log/ceph/ceph-client.admin.31928.log: -- replaced with /var/log/ceph/ceph-client.admin.51966.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34471.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35989.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31928.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42274.log 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41977.log.gz 2024-09-15T02:13:27.608 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49701.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34471.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53409.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35989.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39903.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42274.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53409.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49701.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38019.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39903.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53409.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47163.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.44128.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45616.log 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47163.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38019.log.gz 2024-09-15T02:13:27.609 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39861.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.47163.log.gz 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31699.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45616.log.gz/var/log/ceph/ceph-client.admin.39861.log: 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47920.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.31699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52697.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39861.log.gz 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.47920.log: -5 --verbose 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31699.log.gz 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.47247.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43116.log 2024-09-15T02:13:27.610 INFO:teuthology.orchestra.run.smithi017.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52697.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47247.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47920.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45053.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47247.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34729.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43116.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37893.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45053.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36890.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49491.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45053.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37893.log.gz 2024-09-15T02:13:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50054.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49491.log: /var/log/ceph/ceph-client.admin.39275.log: gzip -5 --verbose 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36890.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.45868.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50054.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49491.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36638.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50012.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50054.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46515.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.45868.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35695.log 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 89.7%/var/log/ceph/ceph-client.admin.46515.log: -- replaced with /var/log/ceph/ceph-client.admin.36638.log.gz 2024-09-15T02:13:27.612 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52008.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35695.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50012.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34869.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52008.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46515.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35695.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50957.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41412.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34869.log: /var/log/ceph/ceph-client.admin.50957.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.52008.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52571.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34869.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41412.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50957.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37327.log 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41412.log.gz 2024-09-15T02:13:27.613 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48844.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52571.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48844.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49449.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52403.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48844.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49407.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49449.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.52403.log: /var/log/ceph/ceph-client.admin.35118.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: 88.7%/var/log/ceph/ceph-client.admin.49407.log: -- replaced with /var/log/ceph/ceph-client.admin.49449.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47019.log 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52403.log.gz 2024-09-15T02:13:27.614 INFO:teuthology.orchestra.run.smithi017.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49407.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43074.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35118.log: /var/log/ceph/ceph-client.admin.47019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52153.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34871.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47019.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52153.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43074.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38061.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48550.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34871.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52153.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38061.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50433.log 2024-09-15T02:13:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48550.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38061.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51714.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50433.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48550.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52445.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.51714.log: -5 --verbose 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50433.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.34868.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52445.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51714.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34868.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52445.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40425.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.43819.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46389.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34868.log.gz 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T02:13:27.616 INFO:teuthology.orchestra.run.smithi017.stderr: 91.3% 89.8%/var/log/ceph/ceph-client.admin.46389.log: -- replaced with /var/log/ceph/ceph-client.admin.40425.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% 89.0%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46389.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46305.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40173.log: /var/log/ceph/ceph-client.admin.46305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44380.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46305.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:gzip 90.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz --verbose 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.49012.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48382.log 2024-09-15T02:13:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42526.log: /var/log/ceph/ceph-client.admin.49012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48382.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45574.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48382.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36199.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36199.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45574.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50559.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36199.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37977.log 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42947.log: /var/log/ceph/ceph-client.admin.50559.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-09-15T02:13:27.618 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50559.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36241.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37977.log: /var/log/ceph/ceph-client.admin.36241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42652.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47793.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36241.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37977.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35117.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43633.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42652.log: gzip/var/log/ceph/ceph-client.admin.35117.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49785.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43633.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35117.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42652.log.gz 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:gzip 89.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.39567.log 2024-09-15T02:13:27.619 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43633.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49743.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49785.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39567.log: /var/log/ceph/ceph-client.admin.41286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39088.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39567.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49743.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51924.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39088.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49743.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37537.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39088.log.gz 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47835.log 2024-09-15T02:13:27.620 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38836.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51924.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47835.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37537.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38836.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47835.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38836.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35569.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42694.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.35569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48424.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42694.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35569.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48466.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42694.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51335.log 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48466.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48424.log.gz 2024-09-15T02:13:27.621 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46893.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48466.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44884.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46893.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51335.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46893.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44884.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44884.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48970.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45011.log: /var/log/ceph/ceph-client.admin.48970.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48970.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49197.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2024-09-15T02:13:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47583.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39609.log: /var/log/ceph/ceph-client.admin.47583.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47583.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42904.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38878.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42904.log: /var/log/ceph/ceph-client.admin.43695.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42904.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40677.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42778.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36932.log 2024-09-15T02:13:27.623 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39651.log: /var/log/ceph/ceph-client.admin.42778.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42778.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35653.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36932.log: /var/log/ceph/ceph-client.admin.37116.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36932.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35653.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35653.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39945.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35275.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38397.log: /var/log/ceph/ceph-client.admin.39945.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39945.log.gz 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43591.log 2024-09-15T02:13:27.624 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52487.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35275.log: /var/log/ceph/ceph-client.admin.43591.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35275.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.43591.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.53369.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52487.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52487.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48886.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40215.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53369.log: /var/log/ceph/ceph-client.admin.48886.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53369.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48256.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48886.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40215.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40215.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50999.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34645.log 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48256.log: /var/log/ceph/ceph-client.admin.50999.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48256.log.gz 2024-09-15T02:13:27.625 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50999.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49889.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34645.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34645.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49889.log: /var/log/ceph/ceph-client.admin.40383.log: 90.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49889.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43032.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36115.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3%/var/log/ceph/ceph-client.admin.43032.log: -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36848.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43339.log 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43032.log.gz 2024-09-15T02:13:27.626 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36115.log: gzip/var/log/ceph/ceph-client.admin.36848.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50601.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43339.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36848.log.gz 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36115.log.gz 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49617.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43339.log.gz 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47751.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51251.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49617.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46767.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49617.log.gz 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51251.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47751.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42610.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50643.log 2024-09-15T02:13:27.627 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46767.log: /var/log/ceph/ceph-client.admin.42610.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51251.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53285.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42610.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50643.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46767.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42188.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50643.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53285.log: gzip/var/log/ceph/ceph-client.admin.42188.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35947.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45448.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42188.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53285.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50475.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35947.log: -- /var/log/ceph/ceph-client.admin.45826.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44548.log 2024-09-15T02:13:27.628 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50475.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35947.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45826.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50475.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44716.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51882.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44716.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44548.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40551.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.44716.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45658.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51882.log.gz/var/log/ceph/ceph-client.admin.40551.log: 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39735.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40551.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50139.log 2024-09-15T02:13:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51588.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45658.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49575.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50139.log.gz/var/log/ceph/ceph-client.admin.41328.log: 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49575.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.51588.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49575.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38794.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41160.log: /var/log/ceph/ceph-client.admin.38355.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gz 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-09-15T02:13:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39130.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52277.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38794.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47709.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52277.log: /var/log/ceph/ceph-client.admin.43876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47709.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.52277.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49054.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47709.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37285.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47289.log 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49054.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-09-15T02:13:27.631 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49054.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47289.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37285.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38145.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47289.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35115.log: gzip/var/log/ceph/ceph-client.admin.38145.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42146.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45784.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38145.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50223.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52865.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46473.log 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.42146.log.gz 2024-09-15T02:13:27.632 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52865.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50223.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47457.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37453.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52865.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46473.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46013.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.47457.log.gz/var/log/ceph/ceph-client.admin.40761.log: 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37453.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49365.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37580.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49365.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46013.log.gz 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2024-09-15T02:13:27.633 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36680.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49365.log.gz/var/log/ceph/ceph-client.admin.37580.log: 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52781.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37580.log.gz 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36680.log: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44254.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36680.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43507.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52781.log: /var/log/ceph/ceph-client.admin.44254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46557.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52781.log.gz 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43507.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44254.log.gz 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45910.log 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43507.log.gz 2024-09-15T02:13:27.634 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50517.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46557.log: /var/log/ceph/ceph-client.admin.45910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36596.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50517.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46557.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45910.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43960.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50517.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37411.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36596.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43960.log: /var/log/ceph/ceph-client.admin.38710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43465.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37411.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.43960.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37495.log 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37411.log.gz 2024-09-15T02:13:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43465.log.gz 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.37495.log: -- /var/log/ceph/ceph-client.admin.48214.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50830.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38271.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37495.log.gz 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40467.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50830.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48214.log.gz 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50830.log.gz 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50097.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52739.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50097.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.40467.log.gz --verbose 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.48802.log 2024-09-15T02:13:27.636 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52739.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50097.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52739.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48802.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48802.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35416.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50915.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36470.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35416.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51041.log 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr: 90.6%/var/log/ceph/ceph-client.admin.40088.log: -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50915.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-09-15T02:13:27.637 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47205.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52050.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47205.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.51041.log.gz 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50747.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47205.log.gz 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.36428.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50747.log: /var/log/ceph/ceph-client.admin.52050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr: 89.2%/var/log/ceph/ceph-client.admin.36428.log: -- replaced with /var/log/ceph/ceph-client.admin.50747.log.gz 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52050.log.gz 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42232.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3%/var/log/ceph/ceph-client.admin.39525.log: -- replaced with /var/log/ceph/ceph-client.admin.36428.log.gz 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46179.log 2024-09-15T02:13:27.638 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42232.log: gzip/var/log/ceph/ceph-client.admin.41809.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52907.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46179.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42232.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.46179.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44338.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42736.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44338.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51420.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52907.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr: 88.4%/var/log/ceph/ceph-client.admin.42736.log: -- replaced with /var/log/ceph/ceph-client.admin.44338.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51209.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51420.log: 88.2%/var/log/ceph/ceph-client.admin.51209.log: -- replaced with /var/log/ceph/ceph-client.admin.42736.log.gz 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39046.log 2024-09-15T02:13:27.639 INFO:teuthology.orchestra.run.smithi017.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51209.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51420.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46935.log 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39046.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51167.log 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46935.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39046.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36554.log 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46431.log 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46935.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51167.log: /var/log/ceph/ceph-client.admin.36554.log: 88.3%/var/log/ceph/ceph-client.admin.46431.log: -- replaced with /var/log/ceph/ceph-client.admin.51167.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36554.log.gz 2024-09-15T02:13:27.640 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46431.log.gz 2024-09-15T02:13:27.850 INFO:teuthology.orchestra.run.smithi017.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T02:13:27.984 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T02:13:28.259 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T02:13:28.262 INFO:teuthology.orchestra.run.smithi017.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T02:13:28.296 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T02:13:28.354 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T02:13:28.355 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T02:13:28.419 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T02:13:28.420 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T02:13:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m1.171s 2024-09-15T02:13:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m3.974s 2024-09-15T02:13:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.361s 2024-09-15T02:13:28.547 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T02:13:28.549 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-15T02:13:28.549 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m1.332s 2024-09-15T02:13:28.549 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m5.020s 2024-09-15T02:13:28.549 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.496s 2024-09-15T02:13:28.549 INFO:tasks.ceph:Archiving logs... 2024-09-15T02:13:28.550 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi017/log 2024-09-15T02:13:28.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:13:29.629 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi067/log 2024-09-15T02:13:29.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:13:30.511 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T02:13:30.521 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T02:13:30.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:13:30.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:13:30.601 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T02:13:30.610 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T02:13:30.610 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:13:30.613 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:13:30.628 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:13:30.630 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:13:30.633 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:13:30.633 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-15T02:13:30.633 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv01.front.sepia.ceph.com 3 6 377 57 +91us[ +92us] +/- 65ms 2024-09-15T02:13:30.633 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 -238us[ -238us] +/- 49ms 2024-09-15T02:13:30.634 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +84us[ +84us] +/- 88ms 2024-09-15T02:13:30.634 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:13:30.635 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:13:30.635 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T02:13:30.635 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 55 +101us[ +109us] +/- 65ms 2024-09-15T02:13:30.636 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -233us[ -233us] +/- 49ms 2024-09-15T02:13:30.636 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +75us[ +75us] +/- 88ms 2024-09-15T02:13:30.636 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:13:30.636 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T02:13:30.645 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T02:13:30.646 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T02:13:30.655 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T02:13:30.755 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T02:13:30.826 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:13:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:13:30.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T02:13:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:13:30.888 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T02:13:30.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T02:13:30.909 INFO:teuthology.task.internal:Duration was 1717.079696 seconds 2024-09-15T02:13:30.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T02:13:30.917 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T02:13:30.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:13:30.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:13:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:13:30.971 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T02:13:31.322 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T02:13:31.322 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-09-15T02:13:31.322 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:13:31.385 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T02:13:31.386 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:13:31.410 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T02:13:31.411 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:13:31.427 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:13:31.479 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T02:13:31.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:13:31.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:13:31.624 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T02:13:31.634 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T02:13:31.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:13:31.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:13:31.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T02:13:31.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:31.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:31.723 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-09-15T02:13:31.749 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-09-15T02:13:31.763 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:13:31.801 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:13:31.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T02:13:31.830 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T02:13:31.832 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi017 2024-09-15T02:13:31.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:13:31.882 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905072/remote/smithi067 2024-09-15T02:13:31.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:13:31.916 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T02:13:31.917 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:13:31.919 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:13:31.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T02:13:31.978 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T02:13:31.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T02:13:31.987 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T02:13:31.987 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:13:31.990 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:13:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:13 /home/ubuntu/cephtest 2024-09-15T02:13:32.022 INFO:teuthology.orchestra.run.smithi067.stdout: 265704 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:13 /home/ubuntu/cephtest 2024-09-15T02:13:32.023 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T02:13:32.038 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T02:13:32.118 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_workunits} duration: 1717.0796957015991 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T02:13:32.118 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:13:32.206 INFO:teuthology.run:pass